One question we hear a lot is “What’s the best way to start testing a chatbot ?” The answer is surprisingly simple: Build up knowledge by educating yourself, and you will be able to answer this question yourself. This post will walk you through the major milestones that test automation engineers have to take for building this knowledge. We will suggest readings and other material that you might find helpful as well.

Five Step Journey

Step One: Build a First Sample Chatbot

The best way to get going is to actually build yourself a sample chatbot. The reason is that some things like natural language processing (NLP) might appear like…


Si tienes un martillo, cada problema parece un clavo.

Este es Mjolnir, el martillo de Thor.

Con Botium, actualmente estamos definiendo el estándar de la industria para probar chatbots. En nuestros canales de soporte y desarrollo, recibimos regularmente preguntas como:

  • Tengo que probar un chatbot Whatsapp, ¿pueden ayudarme a configurar
    Appium?
  • Para nuestro cliente, tengo que probar un chatbot integrado en su aplicación, ¿puedo probarlo con Botium?
  • Tengo problemas para probar el chatbot de soporte al cliente en nuestro sitio web, Selenium dice <algún código de error aleatorio de Selenium>
  • … y así sucesivamente.

La conclusión que se puede extraer de estas preguntas es: los test engineers o ingenieros de pruebas aprendieron cómo probar sitios web…


Botium ofrece pruebas automatizadas end-to-end de chatbots de WhatsApp en dispositivos reales o virtuales. Por primera vez, es posible tener una estrategia de prueba completa a nivel empresarial para los chatbots de WhatsApp.


If you have a hammer, every problem looks like a nail.

This is Mjolnir, Thor’s hammer

With Botium, we are currently defining the industry standard for testing chatbots. In our support and developer channels we are regularily receiving questions like:

  • I have to test a Whatsapp chatbot, can you help me to set up Appium for it ?
  • For our client I have to test a chatbot embedded in their app, can I test it with Botium ?
  • I have troubles with testing the customer support chatbot on our website, Selenium says <some random Selenium error code>
  • … and so on

The conclusion to draw…


Un resumen rápido de siete puntos importantes a la hora de probar el entrenamiento de tu chatbot. Constantemente vemos equipos que ignoran estas reglas realmente bastante simples.


Esta guía sugiere las mejores prácticas, la infraestructura y las herramientas para garantizar que tu aplicación de voz continúe brindando una experiencia de usuario excepcional.


A quick summary of 7 important DO’s and DON’Ts when training an NLP model for a chatbot. They are best applied before starting a project, but can also help to build a mindset for quality training data in all chatbot project phases.

DO’s and DON’Ts

✅DO: think in problem space, not in solution space

Users typically think in problem space, not in solution space, and so should you. As a quick example, consider the case of a user who ordered a shirt in an online shop and wants to know when it is expected to arrive. Consider this question:

  • when will my shirt arrive

This is a question from problem space, describing…


Voice platforms like Alexa and Google Assistant make it easy to provide a custom voice experience to your clients, even without going deeper in audio processing — everything is part of the platform. But what if you already invested quite some effort into building a chatbot on SAP Conversational AI ? You certainly don’t want to switch to a totally new platform now.

Voice Testing with Botium and SAP Conversational AI

Head on to my blog article in the SAP Community!


A quick summary of 7 important DO’s and DON’Ts when designing a chatbot testing strategy. We are continuously seeing teams ignoring those actually rather simple rules.

DO’s and DON’Ts

DO plan for iterations

In german we say rome was not built in a day — same applies for your chatbot training data. A robust chatbot is built by multiple iterations, training and testing cycles and by ongoing monitoring and performance tuning: CODE, TEST, DEPLOY, REPEAT

DON’T underestimate the need for constant performance measurement

Without measuring performance with real user conversations, you will never know if your chatbot is really working for your users.

Florian Treml

Co-Founder and CTO Botium🤓 — Guitarist 🎸 — 3xFather 🐣

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store