Botium in a Nutshell, Part 2: Writing Chatbot Tests

This is part 2 of the Botium in a Nutshell series of articles.

While the first part of this series gave an overview of the Botium ecosystem, this part deals with the daily business in software testing and automation, writing test cases.

Hello, World!

  1. submitting a phrase possibly entered by a real user to the chatbot
  2. checking the response of the chatbot with the expected outcome

In Botium, the test cases are described by dialog flows the chatbot is supposed to follow. For a sample “greeting” scenario, the Botium test case looks like this — also known as “BotiumScript”:

hello bot!
Hello, meat bag! How can I help you ?

BotiumScript is an easy-to-use scripting language, which is interpreted by Botium Core in a virtual machine. Don’t worry if you didn’t understand a word of this sentence, it doesn’t really matter from a user perspective.

You can write BotiumScript as:

  • plain text file with Notepad or any other text editor
  • Excel file
  • CSV file (comma separated values)
  • and more …

For details please consult the Botium Documentation.

Convos and Utterances

hi bot!
Hello, meat bag! How can I help you ?

Another user may enter the conversation with “hey dude!”

hey dude
Hello, meat bag! How can I help you ?

And there are plenty of other phrases we can think of. For this most simple use case, there are now at least three or more BotiumScripts to write. So let’s rewrite it.

We name this file hello.convo.txt:

TC01 - Greeting#me
Hello, meat bag! How can I help you ?

You may have noticed the additional lines at the beginning of the BotiumScript.
The first line contains a reference name for the test case to make it easier for you to locate the failing conversation within your test case library.

And we add another file hello_utt.utterances.txt:

hello bot!
hi bot!
hey dude
good evening
hey are you here
anyone at home ?

The first BotiumScript is a convo file — it holds the structure of the conversation you expect the chatbot to follow.

The second BotiumScript is an utterances file — it holds several phrases for greeting someone, and you expect your chatbot to be able to recognize every single one of them as a nice greeting from the user.

Botium will take care that the convo and utterances files are combined to verify every response of your chatbot to every greeting phrase.

So now let’s assume that your chatbot uses several phrases for greeting the user back. In the morning it is:

Good morning, meat bag! How can I help you this early ?

And in the evening it is:

Good evening, meat bag! How can I help you at this late hour ?

Let’s extract the bot responses to another utterances file:

Good evening
Good morning

And now comes the magic, we change the convo file to:


Utterances files can be used to verify chatbot responses as well. To summarize:

  • An utterance referenced in a #me-section means: Botium, send every single phrase to the chatbot and check the response
  • An utterance referenced in a #bot-section means: Botium, my chatbot may use any of these answers, all of them are fine

Where to place BotiumScript files ?

You can even mix plain text files, CSV files and Excel files within a test suite.

If you got so far and understood everything, you are pretty much ready for using Botium. The rest of this article points to some advanced features, but the usage of convos and utterances is the main concept behind BotiumScript.

Even when using Botium Box, you have to understand BotiumScript — Botium Box provides assistance when writing your BotiumScript test cases, but for most efficient use you really have to understand it.

Using Excel Instead of Text Files

First Excel sample

You can download the Excel file to use as a template here.

The A-column corresponds to the #me tag, the B-column corresponds to the #bot tag. For adding more than one convo to a worksheet, just leave one row empty:

Second Excel sample

You can use utterances here as well (and everything else supported by text files):

Using utterances in Excel (Convo)
Using utterances in Excel (Utterances)

All you have to tell Botium is:

  • The sheet names where to look for convos
  • The sheet names where to look for utterances
  • The row and column index where to start looking

Please note: the header rows (in the screenshots above) are not relevant to Botium, they are just for the sake of visual appearance. By default, Botium starts to look for convos and utterances at row number 2 and column 1.

You can spread your convos over as many sheets as you want (and we recommend to do so) — see Botium Documentation for details.

On the plus side, using Excel helps in buildling your convo structure in a very intuitive, visual way. On the negative side, code repositories are way better in handling text files than in handling binary files (like Excel).

And More BotiumScript Features

Scripting Memory

please send me two salami pizza
OK. Your order number is $orderNum
What is the status order $orderNum
Your order will arrive soon

Scripting Memory is disabled by default. See Botium Documentation for details how to activate.

User Interface Elements

please send me two salami pizza
Please select the size of the pizza
BUTTONS Kids|Normal|Family
MEDIA kids_pizza.png|normal_pizza.png|family_pizza.png

The special keywords BUTTONS and MEDIA tell Botium to look out for buttons and media files in the chatbot response.

This concept in BotiumScript is called “asserter”, and you can inject your own custom asserters into BotiumScript (for example, to verify that the chatbot has actually added the order to the order management backend).

Similar behaviour is available on the other end of the conversation as well:

What Pizza do you want ? Please choose an option.

This will simulate a button click or option selection.

Did you like your Pizza ?
No. It was ugly.
MEDIA pizza_ugly.png

And this will simulate a picture attachment.

Button click simulation and media attachments are not supported by all Botium connectors. Please consult the documentation of the connector in question. The default behaviour is to use the button title as message text.

For technical details please consult the Botium Documentation.

Splitting Convos

Here is a file partial_login.pconvo.txt:

Please tell me your customer access code!
Thanks, you are logged in now.

And here another one show_info.convo.txt:

Show my registration info.
OK, here is your registration info:

The conversation steps from the partial convo are inserted at the INCLUDE statement (the partial convo is selected by name).

Frequently Asked Questions

Yes, absolutely, writing test cases takes a lot of time. The Botium ecosystem gives you some tools to speed up the process:

  • A Capture&Replay tool to capture a conversation with your chatbot and generate a convo file out of it
  • Several importers for generating convo files out of chatbot development tools (such as Dialogflow and IBM Watson Assistant)
  • If you have existing conversation logs, they usually can be converted to convo files with reasonable effort
  • The Botium Platform includes hundred thousand of prebuilt test cases for different domains

My chatbot responses include dynamic or alternating text — how to handle this with BotiumScript ?

Apart from separating your testcases in convo and utterances files, it is possible to fine tune how Botium actually verifies the chatbot responses:

  • fixed matching
  • lowercase matching
  • substring matching
  • regular expression matching
  • or you can even implement your own logic and hook it into Botium

For technical details please consult the Botium Wiki.

Can I use BotiumScript for training my chatbot, and how ?

Absolutely. Just send all of your convos to the chatbot and the training begins. How this is actually done depends on the chatbot technology you have choosen. When doing it with IBM Watson Assistant for example, in the Watson Workspace you get a list of the utterances the chatbot did not understand and you are supposed to train them for intents or entities.

For some providers there even are converters available to convert BotiumScript utterances file to the native intent resolution format of the provider.

Can I add the BotiumScript files to a source code repository like git ?

Yes, and we encourage you to do so.

I wrote a lot of BotiumScript files, how can I finally send them to my chatbot ?

See next part of this series.

Co-Founder and CTO Botium🤓 — Guitarist 🎸 — 3xFather 🐣