Quite often we are consulted to design a robust test strategy for a mission-critical enterprise chatbot. How is it possible to test something for all possible unexpected user behaviour in the future ? How can someone confidently make assumptions on the quality if we have no clue what the users will ask the chatbot ?

Testing Chatbots for the Unexpected

Short-Tail vs Long-Tail Topics

While we do not own a magic crystal ball to look into future usage scenarios, from our experience we gained the best results with a systematic approach in a continuous feedback setup. In almost every chatbot project the use cases can be categorized:


One question we hear a lot is “What’s the best way to start testing a chatbot ?” The answer is surprisingly simple: Build up knowledge by educating yourself, and you will be able to answer this question yourself. This post will walk you through the major milestones that test automation engineers have to take for building this knowledge. We will suggest readings and other material that you might find helpful as well.

Five Step Journey

Step One: Build a First Sample Chatbot

The best way to get going is to actually build yourself a sample chatbot. The reason is that some things like natural language processing (NLP) might appear like…


If you have a hammer, every problem looks like a nail.

This is Mjolnir, Thor’s hammer

With Botium, we are currently defining the industry standard for testing chatbots. In our support and developer channels we are regularily receiving questions like:

The conclusion to draw…


A quick summary of 7 important DO’s and DON’Ts when training an NLP model for a chatbot. They are best applied before starting a project, but can also help to build a mindset for quality training data in all chatbot project phases.

DO’s and DON’Ts

✅DO: think in problem space, not in solution space

Users typically think in problem space, not in solution space, and so should you. As a quick example, consider the case of a user who ordered a shirt in an online shop and wants to know when it is expected to arrive. Consider this question:

This is a question from problem space, describing…


Voice platforms like Alexa and Google Assistant make it easy to provide a custom voice experience to your clients, even without going deeper in audio processing — everything is part of the platform. But what if you already invested quite some effort into building a chatbot on SAP Conversational AI ? You certainly don’t want to switch to a totally new platform now.

Voice Testing with Botium and SAP Conversational AI

Head on to my blog article in the SAP Community!


A quick summary of 7 important DO’s and DON’Ts when designing a chatbot testing strategy. We are continuously seeing teams ignoring those actually rather simple rules.

DO’s and DON’Ts

DO plan for iterations

In german we say rome was not built in a day — same applies for your chatbot training data. A robust chatbot is built by multiple iterations, training and testing cycles and by ongoing monitoring and performance tuning: CODE, TEST, DEPLOY, REPEAT

DON’T underestimate the need for constant performance measurement

Without measuring performance with real user conversations, you will never know if your chatbot is really working for your users.


This guide suggests best practices, infrastructure and tools to ensure your voice app continues to deliver outstanding user experience.

Questions When Testing Voice Apps

Application of the suggested practices helps answering the questions:

The Art of Challenging Chatbots

The challenges when testing chatbots, escpecially voice-enabled ones, are different ones than when testing apps with…


Voice platforms like Alexa and Google Assistant make it easy to build your own voice experience, even without going deeper in audio processing — everything is part of the platform. But what if you want to rather go for a solution hosted by yourself, running an assistant on your own website, in your own infrastructure ?

The goal of this article is to show a way how you can build your own voice platform using Open Source tools Rasa and Botium Speech Processing.

An awesome isometric visualization

Rasa is a developer-friendly and extensible chatbot building tool for self-hosting. Botium Speech Processing is a unified…


Botium delivers again — in this case, automated end-2-end testing of WhatsApp chatbots on real or virtual devices. For the first time it is now possible to have a full enterprise-level test strategy for WhatsApp chatbots.

Automated vs Manual Testing

When it comes to testing WhatsApp chatbots up to now there have been mainly two approaches:

Both approaches are valid and no enterprise-level test strategy should miss any of them. But there are two obvious flaws:


This article is pointing out security threats and attack vectors of typical chatbot architectures — based on OWASP Top 10 and adversarial attacks .

The well-known OWASP Top 10 is a list of top security threats for a web application. Most chatbots out there are available over a public web frontend, and as such all the OWASP security risks apply to those chatbot frontends as well. …

Florian Treml

Co-Founder and CTO Botium🤓 — Guitarist 🎸 — 3xFather 🐣

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store