The 5 Step Journey to Becoming a Chatbot Tester

One question we hear a lot is “What’s the best way to start testing a chatbot ?” The answer is surprisingly simple: Build up knowledge by educating yourself, and you will be able to answer this question yourself. This post will walk you through the major milestones that test automation engineers have to take for building this knowledge. We will suggest readings and other material that you might find helpful as well.

The best way to get going is to actually build yourself a sample chatbot. The reason is that some things like natural language processing (NLP) might appear like magic if you are new to the topic. They aren’t. I suggest to take one of these tutorials — don’t worry, all of them have free plans available:

Go the whole way from starting the project to publishing a chatbot on a preview website — everything you need is included in those cloud platforms.

I do not recommend to start with a free, self-hosted stack like Rasa, as it might distract from the journey to become a chatbot tester by tearing your focus more towards infrastructure topics and actual chatbot development.

When dealing with non-trivial chatbots, there is always a machine learning component for natural language processing (NLP) involved. You do not have to get an expert in machine learning to efficiently test them, but you definitely have to understand the taxonomy used in machine learning projects. The central concepts to learn are:

• Intents, Entities, Utterances
• Accuracy, Sensitivity, Specificity, Precision, Recall, F1-Score
• Confidence and Confidence Threshold
• Confusion Matrix

Here are some links to introductory articles:

1. Chatbot Trends Report 2021

2. DO’s and 3 DON’Ts for Training a Chatbot NLP Model

3. Concierge Bot: Handle Multiple Chatbots from One Chat Screen

4. An expert system: Conversational AI Vs Chatbots

You learned how to test websites with Selenium. You learned how to test smartphone apps with Appium. You might be tempted to think A chatbot is basically a website answering questions, why not using Selenium or Appium for testing a chatbot ? Chatbots are a new kind of apps that require new kind of tools to perform effective and high-coverage testing — a goal that cannot be reached with slow, real-time, flaky end-2-end tests.

Wrote about this conflict in one of my previous blog posts.

So invest some days to educate yourself with BotiumScript, an easy-to-learn scripting language for describing chatbot test cases. You can start at the official Botium Docs or watch Shama Ugale in one of her awesome talks:

In kitchen one of the most important phrases is mise en place — it means to put everything into place before getting started with cooking. Now that you have a good understanding of the basics, it is time to get your tool stack in place you can use for your chatbot testing projects.

To get started in Botium Box, read through the Botium Box Wiki with tutorials, howtos, technical background and reference docs.

There are some important differences when testing a chatbot as compared to websites or smartphone apps. What will strike experienced test automation engineers most is that a user can ask a chatbot literally anything (and the users will do as soon as the chatbot is live, believe me). Dealing with non-determinism is a critical topic in software testing. Due to the nature of human language it is impossible for software tests to cover all possible situations.

You can read more in one of my previous blog posts.

Because of this it is even more important to understand the required test levels to meet at least the minimum quality standards in software projects.

• Regression Testing on API level— Identify flaws in the conversation flow before going to production
• NLP Testing — Improve your chatbot understanding
• E2E Testing — Verifying the end-user experience
• Voice Testing — Understand your users on voice channels
• Performance Testing — Ensure your chatbot is responsive under high load
• Security Testing — Making your chatbot secure
• Monitoring — Get notified when problems arise

Read about some important pitfulls here: 4 DO’s and 3 DON’Ts for Chatbot Testing Strategies

The good news is that you can shine and partially re-use your test automation skills gathered in your previous projects in chatbot testing as well.

Wherever you are in your journey, from building your first sample chatbot to running a full-blown continuous chatbot testing pipeline, we’re delighted to have you as part of our Botium community. Please feel free to ask questions on our Botium forum. We look forward to helping you in your journey to become a chatbot testing expert!

Source: https://chatbotslife.com/the-5-step-journey-to-becoming-a-chatbot-tester-14a441e70e8b?source=rss—-a49517e4c30b—4