Tutorial: Using Paraphrasing to Increase Conversational AI Test Coverage
This tutorial provides a step-by-step introduction how to use Botium Box to quickly build a regression test suite for an Amazon Lex chatbot and increase the test coverage by adding paraphrases to the test data.
What we will do:
- Create an Amazon Lex chatbot and enable programmatic access
- Connect Botium Box to Lex
- Automatically generate basic test cases in Botium Box out of the Lex model
- Use the Botium Box Paraphraser to augment the test cases
- View the results of a test run
If you don’t know what Botium is about, you should first read about it in the Botium in a Nutshell blog series.
1. Create an Amazon Lex Chatbot
Follow the instructions in the official Lex documentation to create a bot using a blueprint (OrderFlowers): https://docs.aws.amazon.com/lex/latest/dg/gs-bp-create-bot.html — You only need step 1 for this tutorial.
Afterwards, publish a first version of the bot: https://docs.aws.amazon.com/lex/latest/dg/gettingstarted-ex3.html
Now create an IAM user and make sure to enable programmatic access. You will need the access key and the secret later.
Choose Attach existing policies to user directly to give permissions AmazonLexFullAccess
2. Connect Botium Box to Lex
For this tutorial, it is assumed that you already have a working installation of Botium Box.
Register a new chatbot (don’t forget to save):
- Select Amazon Lex as Connector/Chatbot Technology
- Enter the IAM Access Key and IAM Secret Key from previous step
- Select the Amazon Region Code, the name of the bot and the alias you used for publishing
Use Say Hello to check your credentials. Your blueprint OrderFlowers bot should answer with something like I didn’t understand.
You can also start a quick Live Chat with your Lex bot.
3. Generate Test Cases
Now that the infrastructure is set, we are making Botium Box do the hard work for us and let it automatically generate the boilerplate test cases from the Lex model. Click the Import Lex Model as Test Cases button, select a test set name and click OK. Within a few seconds, Botium Box
- downloads the Lex model
- analyzes the structure of the intents and entities
- and generates test cases of all known utterances with intent and entity asserters
The boilerplate Lex bot only has one intent with two user examples, so the generated test case is rather thin.
4. Data Augmentation
Clicking on Paraphrase It! will make Botium Box suggest additional user examples based on the two existing ones already defined in the Lex model.
So the Lex model originally included only two user examples:
I would like to pick up flowers
I would like to order some flowers
After using Paraphrase It! we now have eight utterances with more or less the same meaning — your Lex bot should definitely be able to predict the right intent OrderFlowers for all of these user examples:
I would like to pick up flowers
I would like to order some flowers
I'd like to pick flowers
I'd like to take flowers
I want to pick flowers
I'd like to order flowers
I'd like to order some flowers
I want to order some flowers
5. Test Run
Create a Botium Box Test Project with the chatbot you registered and the test set and let it run. In my case, Lex was unable to predict the correct intent for the user example I’d like to take flowers — all other user examples worked.
It’s now up to you to decide what to do with this information — train the Lex bot with additional user examples, or remove it from the test set.
Conclusion
With the help of Botium Box it is possible to create a regression test suite for a chatbot and semi-automatically augment the test data within several minutes. You can give it a try immediately.