Regression Testing an NLP model with Microsoft LUIS and Botium
UPDATE 2020/11/01: Botium’s free plan is live! With Botium Box Mini you will be able to:
- use multiple chatbot technologies
- set up test automation in a few minutes
- enjoy a new improved user interface
- get the benefits of a hosted, free service
In the age of agile development, the age of manual regression testing is over. We developed a Botium extension to enable an agile developer team to run automated regression tests on a Microsoft LUIS model within their CI Pipeline.
When doing continuous improvement of an NLP model — in our case, for a chatbot — you have to make sure to not mess the existing model up. Especially when dealing with similar user utterances mapping to closely related intents there is high risk involved on training additional user utterances. Same applies for utterances with closely related entities.
What we need is to make sure that our most important intents and entities are recognized for the most important user utterances. We need regression testing. We just published a Botium connector for doing this with Microsoft LUIS. For an introduction to Botium, the Selenium for Chatbots, please read the “Botium in a Nutshell” blog series:
Botium in a Nutshell, Part 1: Overview
This is part 1 of the Botium in a Nutshell series of articles.
The rest of this article describes the steps required to setup a project for holding the test cases and test runners for your Microsoft LUIS model.
Before you begin, make sure you have a current version of Node.js installed on your workstation.
Botium Installation and Initialization
Botium Box comes with out-of-the-box support for Microsoft LUIS, so you don’t have to manually setup those components.
Run these commands to install Botium and Mocha, to create a Botium project skeleton and run the skeleton project.
> npm install botium-bindings
> npm install botium-connector-luis
> ./node_modules/.bin/botium-bindings init mocha
> npm install && npm run mocha
Follow these steps to change the file botium.json to connect your Microsoft LUIS model to Botium. Create a file lights_on.convo.txt in the spec/convo folder:
set lights concentrate#bot
This BotiumScript asks your Microsoft LUIS model for resolving the user utterance “set lights concentrate” and makes sure that it resolves to the expected intent and entities.
With BotiumScript, you don’t have to write files for all utterances, you can use utterances files to collect all utterances to resolve to the same intent and entity into a single file.
Running Test Cases with Mocha
Running the test case is as simple as:
> npm run mocha
Either trigger the tests manually, or setup your CI Pipeline to run them — it’s up to you now.
Running Test Cases with Botium Box
The Botium Box contains a nice configuration wizard for Microsoft LUIS connector. You can integrate Botium Box into your CI Pipeline as well.
Please see this article for an introduction to Botium Box:
Botium in a Nutshell, Part 5: The Botium Box
This is part 5 of the Botium in a Nutshell series of articles.
Looking for contributors
Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here: