Tutorial: Using Paraphrasing to Increase Conversational AI Test Coverage

This tutorial provides a step-by-step introduction how to use Botium Box to quickly build a regression test suite for an Amazon Lex chatbot and increase the test coverage by adding paraphrases to the test data.

What we will do:

1. Create an Amazon Lex chatbot and enable programmatic access
2. Connect Botium Box to Lex
3. Automatically generate basic test cases in Botium Box out of the Lex model
4. Use the Botium Box Paraphraser to augment the test cases
5. View the results of a test run

If you don’t know what Botium is about, you should first read about it in the Botium in a Nutshell blog series.

Botium in a Nutshell, Part 1: Overview

1. Create an Amazon Lex Chatbot

Follow the instructions in the official Lex documentation to create a bot using a blueprint (OrderFlowers): https://docs.aws.amazon.com/lex/latest/dg/gs-bp-create-bot.html — You only need step 1 for this tutorial.

Amazon Lex Console

Afterwards, publish a first version of the bot: https://docs.aws.amazon.com/lex/latest/dg/gettingstarted-ex3.html

Now create an IAM user and make sure to enable programmatic access. You will need the access key and the secret later.

Programmatic access

Choose Attach existing policies to user directly to give permissions AmazonLexFullAccess

AmazonLexFullAccess policy

2. Connect Botium Box to Lex

For this tutorial, it is assumed that you already have a working installation of Botium Box.

Botium Box Dashboard

Register a new chatbot (don’t forget to save):

• Select Amazon Lex as Connector/Chatbot Technology
• Enter the IAM Access Key and IAM Secret Key from previous step
• Select the Amazon Region Code, the name of the bot and the alias you used for publishing

Chatbot Registration

Use Say Hello to check your credentials. Your blueprint OrderFlowers bot should answer with something like I didn’t understand.

Botium Box Say Hello

You can also start a quick Live Chat with your Lex bot.

Botium Box Live Chat

3. Generate Test Cases

Now that the infrastructure is set, we are making Botium Box do the hard work for us and let it automatically generate the boilerplate test cases from the Lex model. Click the Import Lex Model as Test Cases button, select a test set name and click OK. Within a few seconds, Botium Box

• downloads the Lex model
• analyzes the structure of the intents and entities
• and generates test cases of all known utterances with intent and entity asserters

Import Lex Model as Test Cases

The boilerplate Lex bot only has one intent with two user examples, so the generated test case is rather thin.

OrderFlowers utterance file

4. Data Augmentation

Clicking on Paraphrase It! will make Botium Box suggest additional user examples based on the two existing ones already defined in the Lex model.

Getting Paraphrases

So the Lex model originally included only two user examples:

I would like to pick up flowers
I would like to order some flowers

After using Paraphrase It! we now have eight utterances with more or less the same meaning — your Lex bot should definitely be able to predict the right intent OrderFlowers for all of these user examples:

I would like to pick up flowers
I would like to order some flowers
I'd like to pick flowers
I'd like to take flowers
I want to pick flowers
I'd like to order flowers
I'd like to order some flowers
I want to order some flowers

5. Test Run

Create a Botium Box Test Project with the chatbot you registered and the test set and let it run. In my case, Lex was unable to predict the correct intent for the user example I’d like to take flowers — all other user examples worked.

Botium Box Test Session

It’s now up to you to decide what to do with this information — train the Lex bot with additional user examples, or remove it from the test set.

Conclusion

With the help of Botium Box it is possible to create a regression test suite for a chatbot and semi-automatically augment the test data within several minutes. You can give it a try immediately.