Regression Testing an NLP model with Microsoft LUIS and Botium

UPDATE 2020/11/01: Botium’s free plan is live! With Botium Box Mini you will be able to:

  • use multiple chatbot technologies
  • set up test automation in a few minutes
  • enjoy a new improved user interface
  • get the benefits of a hosted, free service

Take it for a test drive

In the age of agile development, the age of manual regression testing is over. We developed a Botium extension to enable an agile developer team to run automated regression tests on a Microsoft LUIS model within their CI Pipeline.

Why ?

When doing continuous improvement of an NLP model — in our case, for a chatbot — you have to make sure to not mess the existing model up. Especially when dealing with similar user utterances mapping to closely related intents there is high risk involved on training additional user utterances. Same applies for utterances with closely related entities.

What we need is to make sure that our most important intents and entities are recognized for the most important user utterances. We need regression testing. We just published a Botium connector for doing this with Microsoft LUIS. For an introduction to Botium, the Selenium for Chatbots, please read the “Botium in a Nutshell” blog series:

The rest of this article describes the steps required to setup a project for holding the test cases and test runners for your Microsoft LUIS model.

Before you begin, make sure you have a current version of Node.js installed on your workstation.

Botium Installation and Initialization

Botium Box comes with out-of-the-box support for Microsoft LUIS, so you don’t have to manually setup those components.

Run these commands to install Botium and Mocha, to create a Botium project skeleton and run the skeleton project.

> npm install botium-bindings
> npm install botium-connector-luis
> ./node_modules/.bin/botium-bindings init mocha
> npm install && npm run mocha

Follow these steps to change the file botium.json to connect your Microsoft LUIS model to Botium. Create a file lights_on.convo.txt in the spec/convo folder:

lights on#me
set lights concentrate
#bot
INTENT HomeAutomation.TurnOn
ENTITIES HomeAutomation.Operation|...
ENTITY_VALUES concentrate|lights

This BotiumScript asks your Microsoft LUIS model for resolving the user utterance “set lights concentrate” and makes sure that it resolves to the expected intent and entities.

With BotiumScript, you don’t have to write files for all utterances, you can use utterances files to collect all utterances to resolve to the same intent and entity into a single file.

Running Test Cases with Mocha

Running the test case is as simple as:

> npm run mocha

Either trigger the tests manually, or setup your CI Pipeline to run them — it’s up to you now.

Running Test Cases with Botium Box

The Botium Box contains a nice configuration wizard for Microsoft LUIS connector. You can integrate Botium Box into your CI Pipeline as well.

Botium Box Dashboard

Please see this article for an introduction to Botium Box:

Looking for contributors

Please take part in the Botium community to bring chatbots forward! By contributing you help in increasing the quality of chatbots worldwide, leading to increasing end-user acceptance, which again will bring your own chatbot forward! Start here:

Co-Founder and CTO Botium🤓 — Guitarist 🎸 — 3xFather 🐣