Tutorial: Building a CI Pipeline for Chatbot Developers with Rasa X and Botium Box
UPDATE 2020/11/01: Botium’s free plan is live! With Botium Box Mini you will be able to:
- use multiple chatbot technologies
- set up test automation in a few minutes
- enjoy a new improved user interface
- get the benefits of a hosted, free service
Software developer teams not living in the 90s most likely have a build pipeline in place.
- Source code is managed in a Git repository
- A pipeline tool like Jenkins or Bamboo is triggered on each source code checkin
- Followed by compilation and automated testing of the source code
- Source code is rejected if any problems are detected
The whole process is automated to keep up with multi-team development projects — doing thousands of builds a week is not unusual even for a medium-sized project.
This tutorial shows a way to establish the same methods to chatbot projects, especially for projects focused on training an NLU engine and/or a dialogue engine. The involved tools are:
- Rasa X for chatbot development
- Botium Box for chatbot test automation
- Jenkins as build pipeline
All of them are open source or available in a free edition suitable for small projects
Rasa X
Since version 0.23.0, Rasa X has a feature called Integrated Version Control — when enabled, training data is versioned and managed in a Git repository. So the first step is to install Rasa X and connect it to a Git repository — please follow the instructions in the Rasa documentation.
Rasa X also has a concept called Deployment Environments — a trained model is first published to an experimental or testing stage before deployed to production stage. The testing stage is the server where testing takes place, while the live chatbot is connected to the production stage.
Rasa X comes with only one stage by default, the productions stage. I strongly recommend to add the testing stage by following the instructions in the Rasa X documentation.
Botium Box
Botium Box is the one-stop solution for testing and training of chatbots with a focus an automated testing. It can do dialogue-based test cases as well as NLP analytics. It has been built with CI/CD integration in mind from the beginning and supports every pipeline product out there by providing low-key HTTP/JSON based webhooks for triggering builds and delivering test results.
Botium Box Installation
Follow the installation instructions in the Botium Wiki. Same as with Rasa X, Docker and Docker-Compose is required for installation.
You can also install Botium Box to a Kubernetes Cluster, which is the recommended setup for production usage.
Connecting Botium Box to Rasa
Botium Box includes connectors to most chatbot technologies out there, from the blockbuster cloud services like Dialogflow, IBM Watson or Microsoft LUIS to open source products like Botpress and Rasa to some proprietary chatbot and A.I. platforms from our partners like Wipro Holmes.
With the Botium Box Rasa Connector you now enter the URL and the Rasa Token of your development deployment environment from Rasa X (which is usually the same for all deployment environments).
Test Cases
In Botium Box, test cases are composed in a simple scripting language called BotiumScript. A very simple test case looks like this:
say hi#me
hi bot#bot
hi meat bag
This test case is named say hi, the text hi bot is sent to the chatbot, and the chatbot is expected to answer hi meat bag.
For an introduction to Botium read the Botium in a Nutshell series:
The bad news are, to have a good test coverage for a chatbot, you typically need thousands of test cases. I wrote about the special challenges when testing a chatbot in another blog series:
The good news are: Botium Box can generate the test cases out of the NLU.md from your Rasa model automatically. Click the Import Test Cases from Model NLU File to open the importer. Drag&Drop your NLU.md file and select what kind of test cases should be generated:
- Import Utterances as list only: Import only the utterances as Botium utterances file, without doing any assertions
- Import Utterances for intent assertions: Botium test cases are generated out of user examples and resolved intent assertions
- Import Utterances and Entities for intent and entity assertions: Botium test cases are generated out of user examples and resolved intent as well as detected entities assertions
The so-called low-hanging fruits are to establish a regression testing process :
- The NLU.md is imported into Botium Box as test suite for the current model. As the training data is used for testing, the test suite will have 100% success rate.
- When updating the Rasa project (with new dialogues, or with new user examples from live user feedback, or with new business logic …) and training a new model, the regression testing will tell if the changes actually broke anything from the previous model. This will be done automatically by the build pipeline and Botium Box.
- When having everything fixed and in place on the production environment, the test suite in Botium Box is updated with the latest NLU.md from the deployed Rasa model to start a new generation of regression test cycles.
When running the test cases, the test results are evaluated and asserted against the expected results, including NLU information (intents, entities and confidence).
Botium Box will also calculate a confusion matrix including precision, recall and F1-Score for each of your intents. Read about it here:
Jenkins
The last involved component is Jenkins. In Botium Wiki there is an article how to connect Jenkins to Botium Box — Botium Box provides as powerful as easy CI/CD pipeline integration with HTTP(S) webhooks.
As a starter, configure a Jenkins pipeline:
- Connect to the Git repository holding the Rasa X model. Each checkin to this repository should trigger the build pipeline.
- In a shell task, the pipeline itself triggers the test suite on the Botium Box and downloads the test result.
- E-Mail notification with build results is optional but recommended.
Conclusion
Nowadays, the tools for bringing software development best practices to the chatbot developers world are available, even as open source or as free editions. For a medium skilled DevOps engineer it will take only one working day to deploy an enterprise-grade CI pipeline for your chatbot developers as described in this tutorial.
Give Botium Box a test drive today — start with the free Community Edition, we are happy to hear from you if you find it useful!
Looking for contributors
Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here:
