Tutorial: Building a CI Pipeline for Chatbot Developers with Rasa X and Botium Box

  • use multiple chatbot technologies
  • set up test automation in a few minutes
  • enjoy a new improved user interface
  • get the benefits of a hosted, free service
  • Source code is managed in a Git repository
  • A pipeline tool like Jenkins or Bamboo is triggered on each source code checkin
  • Followed by compilation and automated testing of the source code
  • Source code is rejected if any problems are detected

Rasa X

Botium Box

Botium Box CI/CD Integration

Botium Box Installation

Connecting Botium Box to Rasa

Botium Connector for Rasa

Test Cases

say hi#me 
hi bot
hi meat bag
  • Import Utterances as list only: Import only the utterances as Botium utterances file, without doing any assertions
  • Import Utterances for intent assertions: Botium test cases are generated out of user examples and resolved intent assertions
  • Import Utterances and Entities for intent and entity assertions: Botium test cases are generated out of user examples and resolved intent as well as detected entities assertions
Botium Box NLU.md importer
  • The NLU.md is imported into Botium Box as test suite for the current model. As the training data is used for testing, the test suite will have 100% success rate.
  • When updating the Rasa project (with new dialogues, or with new user examples from live user feedback, or with new business logic …) and training a new model, the regression testing will tell if the changes actually broke anything from the previous model. This will be done automatically by the build pipeline and Botium Box.
  • When having everything fixed and in place on the production environment, the test suite in Botium Box is updated with the latest NLU.md from the deployed Rasa model to start a new generation of regression test cycles.


  • Connect to the Git repository holding the Rasa X model. Each checkin to this repository should trigger the build pipeline.
  • In a shell task, the pipeline itself triggers the test suite on the Botium Box and downloads the test result.
  • E-Mail notification with build results is optional but recommended.
Jenkins project


Looking for contributors




Co-Founder and CTO Botium🤓 — Guitarist 🎸 — 3xFather 🐣

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to check your email with Python

Extreme Programming for Games

C# List.AddRange

Implementing a custom drag event function in JavaScript and THREE.js

A series of vertically stacked networks — drag rotated across the horizontal

Zenity — Linux Command

Leetcode #7 — Reverse Integer (Python)

Setting boundaries for the Player — On the path to Unity development

Second year IT student, I just decided to install ArchLinux.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Florian Treml

Florian Treml

Co-Founder and CTO Botium🤓 — Guitarist 🎸 — 3xFather 🐣

More from Medium

How to contribute to Open source projects on GitHub.

SerpApi’s Google Autocomplete API

Fixing Errors For Flutter + GitActions Automation

How to create a Heroku recovery script