Botium in a Nutshell, Part 3: Automating Chatbot Tests

4 min readNov 9, 2018

This is part 3 of the Botium in a Nutshell series of articles.

Now we are really close to let Botium talk to your chatbot! First we have to tell Botium how to actually connect to your chatbot. Now is the time to install the Botium CLI, the swiss army knife of Botium!

Connecting to your Chatbot

Botium has to somehow build up a communication channel with your chatbot. Usually you have to talk to your development team to provide you the information in a configuration file botium.json — for a chatbot developed with the Microsoft Bot Framework it looks like this:

{
  "botium": {
    "Capabilities": {
      "PROJECTNAME": "Directline3 Plugin Sample",
      "CONTAINERMODE": "directline3",
      "DIRECTLINE3_SECRET": "my-directline-secret",
      "DIRECTLINE3_WEBSOCKET": true,
      "DIRECTLINE3_POLLINGINTERVAL": 1000
    }
  }
}

The configuration items in Botium are called capabilities. They are different for each supported chatbot technology. Botium supports a large variety of chatbot technologies — chatbots built with Dialogflow or Microsoft Bot Framework are supported, as well as chatbots publishing a HTTP/JSON API, or chatbots running on a voice-enabled home automation device, or chatbots running within a chat widget on a website.

The one most important capability to tell Botium what technology to use to connect to your chatbot is CONTAINERMODE. In the example above directline3 is given to instruct Botium to use the Microsoft Bot Framework technology called Directline, Version 3, for a chatbot hosted in Azure. Out-of-the-box, there are already allrelevant chatbot technologies supported:

directline3 for Microsoft Bot Framework Bots hosted on Azure
watson for chatbots developed with IBM Watson
dialogflow for chatbots developed with Google Dialogflow
simplerest for chatbots publishing an HTTP/JSON endpoint
alexa-smapi for Alexa skills (with the Skill Management API)
alexa-avs for Alexa skills on End-To-End-Level (Text-To-Speech and Speech-To-Text conversion)
webdriverio for end-to-end tests on user interface level (chatbots published as website widget)
And more …

Furthermore, with Botiums open interfaces it is possible to plug your own custom chatbot technology into Botium (Node.js developer skills required).

Botium Box includes management of your chatbot capabilities, so you won’t have to write those botium.json files by hand. Instead the configuration is done within a quick 3-step-wizard.

You can validate your configuration settings by using the Botium CLI — it opens a REPL-interface for you, so you can have a chat with your chatbot:

> botium-cli emulator --config path/to/botium.json

The Botium Box includes a “Live Chat” with the same purpose.

Test Case Execution

After following this guide, we now have:

a couple of convo and utterances files, a test set
a valid set of capabilities for connecting to your chatbot

What can you do with it ?

Option 1: Botium CLI

Run Botium CLI like this:

> botium-cli run --config path/to/botium.json --convos path/to/files

Botium CLI will build up a communication channel with your chatbot and run all of your convo files. Status information and a summary are displayed in the command line window.

Option 2: Botium Bindings

Using Botium Bindings is worth a whole article for itself, so please read on here if interested in using a test runner like Mocha, Jest or Jasmine for running Botium test cases.

Option 3: Botium Box Quickstart Wizard

Follow the Get Started With Botium Box tutorial in the Botium Wiki.

Frequently Asked Questions

How can I connect to my chatbot ?

For technical details about supported chatbot technologies and capabilities please consult the Botium Wiki.

My chatbot doesn’t send any response. Please help.

You can find the Troubleshooting Guide in the Botium Documentation. Usually it means that the capabilities in your botium.json are invalid.

Can I integrate Botium with my CI Pipeline from Vendor XYZ / with my costum CI Pipeline?

Yes, absolutely. As long as your CI Pipeline is capable of running a command line script (when using Botium CLI) or calling a HTTP Backend (when using Botium Box), you are fine. Botium Box and Botium CLI support a couple of different test report output formats which usually can be parsed by any pipeline.

I want to see a mega-hipster-responsive-drill-down test report — where is it ?

Botium CLI is not a test report renderer, this task is left to the CI Pipeline and to Botium Box. Nevertheless, Botium CLI supports outputting a mochawsome test report, which may fulfill your needs.

Can Botium send me an E-Mail message when it is ready ?

With Botium CLI, this task is left to the CI Pipeline, or you can write your own shell scripts around Botium CLI to send E-Mails after completion — this is one of the strength of the command line approach, you can use Botium from any shell script.

Botium Box of course includes a notification mechanism.