No More Excuses: Automated Testing of your Chatbot with “TestMyBot”

TestMyBot is a test automation framework for your chatbot project. It is unopinionated and completely agnostic about any involved development tools. Best of all, it’s free and open source.

TestMyBot is now available on Github.

Requirements

TestMyBot heavily relies on Docker to provide its outstanding capabilities in test automation. Your chatbot is fully transfered into a local docker container, and the API mocks are possible by manipulating the DNS of the docker image. TestMyBot opens a channel to your Chatbot webhook and answers your Chatbots requests to the messenger API, providing everything to your test specs.

Therefore, please install Docker first on your development machines.

Current Requirements on your Chatbot:

  • Developed in Node.js (other languages supported, but with more configuration effort)
  • Developed with Facebook Messenger Platform
  • Facebook Webhook has to listen on all interfaces (listen to 0.0.0.0)
  • Accept self-signed SSL certificates (for Node.js, NODE_TLS_REJECT_UNAUTHORIZED environment variable is set to 0 automatically by TestMyBot)
  • Request verification has to be disabled (for Botkit: validate_requests should be set to false)

Installation

Usually, you won’t install this project on it’s own, but you will include it in your Chatbot projects.

To install it to your chatbot project, type:

$ npm install testmybot --save-dev
$ npm install testmybot-fbmock --save-dev

Please note that you have to install it in your local development directory (not in global registry with -g).

Basic Usage

You can find some sample chatbot projects with rudimentary tests here and here.

With Jasmine, the setup looks like this:

$ npm install testmybot --save-dev
$ npm install testmybot-fbmock --save-dev
$ npm install jasmine --save-dev
$ ./node_modules/.bin/jasmine init

Add a file named “testmybot.json” to your project directory. A very basic configuration, which is always required:

{
"docker": {
"container": {
"testmybot-fbmock": {
"env": {
"TESTMYBOT_FACEBOOK_WEBHOOKPORT": 5000,
"TESTMYBOT_FACEBOOK_WEBHOOKPATH": "webhook"
}
}
}
}
}

You tell TestMyBot that the Facebook Webhook of your chatbot runs on port 5000, and the url path is /webhook. You don’t have to tell the hostname, because it will run in a docker container with a fixed hostname (“testmybot”).

Add a file spec/testmybot.spec.js with a basic test case:

describe('TestMyBot Sample Conversation Test Suite', function() {
var bot = require('testmybot');
beforeAll(function(done) {
bot.beforeAll().then(done);
}, 120000);
beforeEach(function(done) {
bot.beforeEach().then(done);
}, 60000);
afterEach(function(done) {
bot.afterEach().then(done);
}, 60000);
afterAll(function(done) {
bot.afterAll().then(done);
}, 60000);
it('should answer to hello', function(done) { bot.hears('hello'); bot.says().then((msg) => {
expect(msg.messageText).toMatch(/echo/);
done();
}).catch((err) => {
throw new Error(err);
});
});
it('should send a generic payload', function(done) { bot.hears('Generic'); bot.says().then((msg) => {
expect(msg.message.attachment.type).toEqual('template');
expect(msg.message.attachment.payload.template_type).toEqual('generic');
done();
}).catch((err) => {
throw new Error(err);
});
});
});

Take special care for:

  • All test are asynchronous
  • Setup and Teardown has high timeouts, because buildling, running and stopping Docker containers can take some time. Especially on first run, it will take very long. Afterwards, the Docker cache speeds up things.
  • TestMyBot uses Bluebird Promises

In your package.json, define a script for TestMyBot names start_testmybot, which is run in the Docker container.

...
"scripts": {
"start_testmybot": "node index.js",
},
...

You can hand over environment variables to your chatbot here. And finally, run your tests with Jasmine:

$ ./node_modules/.bin/jasmine init

You will see some output from Docker, and in the end, your Jasmine tests should succeed (of course).

Configuration

There are three steps for buildling the configuration:

  • ./node_modules/testmybot/testmybot.default.json
  • ./testmybot.default
  • “config” Parameter to “beforeAll” method

The subsequent steps are overwriting the configuration parameters from the previous steps. The .json-files are converted into a plain object in Node.js. Here are some settings which you may have to adapt in your environment (but in most cases, the defaults will work fine).

Please see ./node_modules/testmybot/testmybot.default.json for reference values.

API

testmybot.beforeAll(config) — builds Docker networking and containers from your configuration

  • config (optional) — you can pass an optional configuration argument (merged into other configuration)
  • returns a bluebird Promise

testmybot.afterAll() — removes Docker networking and containers

  • returns a bluebird Promise

testmybot.beforeEach() — starts Docker containers and waits until online

  • returns a bluebird Promise

testmybot.afterEach() — stops Docker containers

  • returns a bluebird Promise

testmybot.hears(msg, channelId) — send a text (or structured content) to your chatbot

  • msg — text or structured content
  • channelId (optional) — the channel to send your message to (in Facebook: “sender”/”recipient” or user profile id). You can mock parallel conversations with multiple users

testmybot.says(channelId, timeoutMillis) — receive a text (or structured content) from your chatbot

  • channelId (optional) — receive for this channel (or user profile id)
  • timeoutMillis (optional) — timeout when not receiving anything (default: 5000)
  • returns a bluebird Promise, which resolves to the the received message.

Here is an example for a received message. It contains (for brevity) the message text (if applicable), the full original message (in “orig”) and the message body (in “message). For structured messages, there is no messageText.

{
"orig": {
"recipient": {
"id":4440090943
},
"message": {
"text": "Text received, echo: hello"
}
},
"messageText": "Text received, echo: hello",
"message": {
"text": "Text received, echo: hello"
},
"channelId": 4440090943
}

Outlook

There is a long way to go for this library.

  • Support Slack Chatbots
  • Support Wechat Chatbots
  • Support Microsoft Bot Framework Chatbots
  • Support Python Chtabots
  • Define Test Cases by conversation transcripts
  • Run your Tests in live environment with real Endpoints

Looking for Contributors

As all Open Source projects, TestMyBot will get better and more useful the more people there are willing to contribute. Please join the Github project to bring this project further.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Florian Treml

Co-Founder and CTO Botium🤓 — Guitarist 🎸 — 3xFather 🐣