How to Add a New Cognitive Service to Microsoft Chatbot — Part 3

It is usually not enough to use only one cognitive service if you want to create a great user experience for a chatbot. It takes multiple cognitive services such as natural language processing and questions and answer.

So, that brings about the topic of how to add a new cognitive service to your Microsoft chatbot. Sure, there are examples out there in the Microsoft GitHub Bot Samples, but those don’t really tell you how to do it. In this post, I am going to show the process of adding a QnA (question and answer) cognitive service to your chatbot.

Note: The codebase I am using is the basic bot generated code on Azure when you first create a web app bot.

Requirements

You will need the following if you want to follow along:

• MSBot
• Visual Studio 2017
• Bot Framework Emulator version 4
• Microsoft Azure account

Example and Goal

The objective of the example is to manually go through the process to add QnA to the bot. For the QnA knowledge base, we’ll use Microsoft’s FAQ webpage: https://azure.microsoft.com/en-us/services/cognitive-services/qna-maker/faq/.

This means that at the end of the example, you will have a bot that uses language understanding (the generated codebase) to understand greeting and uses QnA service to answer any questions in the FAQ webpage.

Here is how the bot is from the generated codebase:

Here is how the end result will be like:

Creating the QnA App

To create the QnA App from the Microsoft FAQ webpage, you will need to do the following steps:

1. On Azure create a QnA Maker Service
2. Wait for QnA maker service to finish deploying in Azure
3. Go to QnAMaker app page (https://www.qnamaker.ai/) and create a knowledge base by using the Microsoft FAQ (https://azure.microsoft.com/en-us/services/cognitive-services/qna-maker/faq/) as the source
   
4. Once all the configurations are set up, proceed to create the knowledge base
5. Publish the knowledge base so that the bot can talk to the QnA service
6. When you have successfully published the QnA service, you’ll get a popup showing how to talk to the service with cURL or postman.
   

Adding QnA Service to Chatbot Configuration with Msbot

In order for the bot to talk with the QnA service, you will need to add the service to the bot’s configuration file. You can add the QnA service to the bot configuration file with the following steps:

1. Install MSBot

2. npm -g msbot

3. Add the QnA service with msbot

4. msbot connect qna --secret <SECRET KEY> --name "APP NAME" --kbid <KBID> --subscriptionKey <KEY> --endpointKey <ENDPOINT-KEY> --hostname "https://myqna.azurewebsites.net"

If you were worried that you will need to find each of the values to those parameters, you don’t need to worry. I got you covered.

Secret

From the appsettings.json value for the key “secret”. If you don’t have an appsettings.json file you can find the secret in your web app bot application settings in the botFileSecret field.

Name

This is the name of the QnA service. It can be anything you want, but you need to be consistent with this name throughout your bot code.

Subscription Key

The subscription key is from the QnA cognitive service in Azure. You can use either key 1 or 2.

KBID, Endpoint Key, and Hostname

All three of these parameters are within the sample cURL command you get after you have published the QnA app in QnA Maker.  Here is each of the component broken down in the cURL command:

Adding QnA Service in BotServices Code

For the bot to use QnA service, we need to create a singleton of the QnA service in the BotServices.cs file. The generated code already handles LUIS, we need to add in the case for QnA. To do that we need to install the QnA library and then update the code.

We can add the QnA package simply by NuGet package manager in Visual Studio.

Once the package is installed, you will need to update the BotServices.cs file to create a QnA service.

Here is the BotServices.cs file after adding in the QnA portion.

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License

using System;
using System.Collections.Generic;
using Microsoft.Bot.Builder.AI.Luis;
using Microsoft.Bot.Builder.AI.QnA;
using Microsoft.Bot.Configuration;

namespace Microsoft.BotBuilderSamples
{
    /// 

<summary>
    /// Represents references to external services.
    ///
    /// For example, LUIS services are kept here as a singleton.  This external service is configured
    /// using the <see cref="BotConfiguration"/> class.
    /// </summary>


    /// <seealso cref="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/dependency-injection?view=aspnetcore-2.1"/>
    /// <seealso cref="https://www.luis.ai/home"/>
    public class BotServices
    {
        /// 

<summary>
        /// Initializes a new instance of the <see cref="BotServices"/> class.
        /// </summary>


        /// <param name="luisServices">A dictionary of named <see cref="LuisRecognizer"/> instances for usage within the bot.</param>
        public BotServices(BotConfiguration botConfiguration)
        {
            foreach (var service in botConfiguration.Services)
            {
                switch (service.Type)
                {
                    case ServiceTypes.Luis:
                        {
                            var luis = (LuisService)service;
                            if (luis == null)
                            {
                                throw new InvalidOperationException("The LUIS service is not configured correctly in your '.bot' file.");
                            }

                            var app = new LuisApplication(luis.AppId, luis.AuthoringKey, luis.GetEndpoint());
                            var recognizer = new LuisRecognizer(app);
                            this.LuisServices.Add(luis.Name, recognizer);
                            break;
                        }
                    case ServiceTypes.QnA:
                        {
                            if (!(service is QnAMakerService qna))
                            {
                                throw new InvalidOperationException("The QnA service is not configured correctly in your '.bot' file.");
                            }

                            var qnaEndpoint = new QnAMakerEndpoint
                            {
                                KnowledgeBaseId = qna.KbId,
                                EndpointKey = qna.EndpointKey,
                                Host = qna.Hostname
                            };

                            var qnaMaker = new QnAMaker(qnaEndpoint);
                            QnAServices.Add(qna.Name, qnaMaker);
                            break;
                        }
                }
            }
        }

        /// 

<summary>
        /// Gets the set of LUIS Services used.
        /// Given there can be multiple <see cref="LuisRecognizer"/> services used in a single bot,
        /// LuisServices is represented as a dictionary.  This is also modeled in the
        /// ".bot" file since the elements are named.
        /// </summary>


        /// <remarks>The LUIS services collection should not be modified while the bot is running.</remarks>
        /// <value>
        /// A <see cref="LuisRecognizer"/> client instance created based on configuration in the .bot file.
        /// </value>
        public Dictionary<string, LuisRecognizer> LuisServices { get; } = new Dictionary<string, LuisRecognizer>();

        public Dictionary<string, QnAMaker> QnAServices { get; } = new Dictionary<string, QnAMaker>();
    }
}

Updating the Bot to Use QnA Service

We don’t want to completely eliminate the language understanding portion of the bot. So, for this example, the bot will only use the QnA service as a fallback. This means the bot will not reference the QnA knowledge base unless the language understanding portion returns an “I don’t know”.

Here is the BasicBot.cs file with the changes for QnA fallback.

case NoneIntent:
    // try to check with QnA Knowledge base
    var qnaResults = await _services.QnAServices[QnAConfiguration].GetAnswersAsync(dc.Context);
    if (qnaResults.Any())
    {
        await dc.Context.SendActivityAsync(qnaResults.First().Answer, cancellationToken: cancellationToken);
    }
    else 
    {
        await dc.Context.SendActivityAsync("I didn't understand what you just said to me.");
    }
    break;

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

// See https://github.com/microsoft/botbuilder-samples for a more comprehensive list of samples.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Bot.Builder;
using Microsoft.Bot.Builder.Dialogs;
using Microsoft.Bot.Schema;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;

namespace Microsoft.BotBuilderSamples
{
    /// 

<summary>
    /// Main entry point and orchestration for bot.
    /// </summary>


    public class BasicBot : IBot
    {
        // Supported LUIS Intents
        public const string GreetingIntent = "Greeting";
        public const string CancelIntent = "Cancel";
        public const string HelpIntent = "Help";
        public const string NoneIntent = "None";

        /// 

<summary>
        /// Key in the bot config (.bot file) for the LUIS instance.
        /// In the .bot file, multiple instances of LUIS can be configured.
        /// </summary>


        public static readonly string LuisConfiguration = "BasicBotLuisApplication";
        public static readonly string QnAConfiguration = "MS-Faqs";

        private readonly IStatePropertyAccessor<GreetingState> _greetingStateAccessor;
        private readonly IStatePropertyAccessor<DialogState> _dialogStateAccessor;
        private readonly UserState _userState;
        private readonly ConversationState _conversationState;
        private readonly BotServices _services;

        /// 

<summary>
        /// Initializes a new instance of the <see cref="BasicBot"/> class.
        /// </summary>


        /// <param name="botServices">Bot services.</param>
        /// <param name="accessors">Bot State Accessors.</param>
        public BasicBot(BotServices services, UserState userState, ConversationState conversationState, ILoggerFactory loggerFactory)
        {
            _services = services ?? throw new ArgumentNullException(nameof(services));
            _userState = userState ?? throw new ArgumentNullException(nameof(userState));
            _conversationState = conversationState ?? throw new ArgumentNullException(nameof(conversationState));

            _greetingStateAccessor = _userState.CreateProperty<GreetingState>(nameof(GreetingState));
            _dialogStateAccessor = _conversationState.CreateProperty<DialogState>(nameof(DialogState));

            // Verify LUIS configuration.
            if (!_services.LuisServices.ContainsKey(LuisConfiguration))
            {
                throw new InvalidOperationException($"The bot configuration does not contain a service type of `luis` with the id `{LuisConfiguration}`.");
            }

            // Verify QnA configuration
            if (!_services.QnAServices.ContainsKey(QnAConfiguration))
            {
                throw new InvalidOperationException($"The bot configuration does not contain a service type of `qna` with the id `{QnAConfiguration}`.");
            }

            Dialogs = new DialogSet(_dialogStateAccessor);
            Dialogs.Add(new GreetingDialog(_greetingStateAccessor, loggerFactory));
        }

        private DialogSet Dialogs { get; set; }

        /// 

<summary>
        /// Run every turn of the conversation. Handles orchestration of messages.
        /// </summary>


        /// <param name="turnContext">Bot Turn Context.</param>
        /// <param name="cancellationToken">Task CancellationToken.</param>
        /// <returns>A <see cref="Task"/> representing the asynchronous operation.</returns>
        public async Task OnTurnAsync(ITurnContext turnContext, CancellationToken cancellationToken)
        {
            var activity = turnContext.Activity;

            // Create a dialog context
            var dc = await Dialogs.CreateContextAsync(turnContext);

            if (activity.Type == ActivityTypes.Message)
            {
                // Perform a call to LUIS to retrieve results for the current activity message.
                var luisResults = await _services.LuisServices[LuisConfiguration].RecognizeAsync(dc.Context, cancellationToken).ConfigureAwait(false);

                // If any entities were updated, treat as interruption.
                // For example, "no my name is tony" will manifest as an update of the name to be "tony".
                var topScoringIntent = luisResults?.GetTopScoringIntent();

                var topIntent = topScoringIntent.Value.intent;

                // update greeting state with any entities captured
                await UpdateGreetingState(luisResults, dc.Context);

                // Handle conversation interrupts first.
                var interrupted = await IsTurnInterruptedAsync(dc, topIntent);
                if (interrupted)
                {
                    // Bypass the dialog.
                    // Save state before the next turn.
                    await _conversationState.SaveChangesAsync(turnContext);
                    await _userState.SaveChangesAsync(turnContext);
                    return;
                }

                // Continue the current dialog
                var dialogResult = await dc.ContinueDialogAsync();

                // if no one has responded,
                if (!dc.Context.Responded)
                {
                    // examine results from active dialog
                    switch (dialogResult.Status)
                    {
                        case DialogTurnStatus.Empty:
                            switch (topIntent)
                            {
                                case GreetingIntent:
                                    await dc.BeginDialogAsync(nameof(GreetingDialog));
                                    break;

                                case NoneIntent:
                                    // try to check with QnA Knowledge base
                                    var qnaResults = await _services.QnAServices[QnAConfiguration].GetAnswersAsync(dc.Context);
                                    if (qnaResults.Any())
                                    {
                                        await dc.Context.SendActivityAsync(qnaResults.First().Answer, cancellationToken: cancellationToken);
                                    }
                                    else 
                                    {
                                        await dc.Context.SendActivityAsync("I didn't understand what you just said to me.");
                                    }
                                    break;

                                default:
                                    // Help or no intent identified, either way, let's provide some help.
                                    // to the user
                                    await dc.Context.SendActivityAsync("I didn't understand what you just said to me.");
                                    break;
                            }

                            break;

                        case DialogTurnStatus.Waiting:
                            // The active dialog is waiting for a response from the user, so do nothing.
                            break;

                        case DialogTurnStatus.Complete:
                            await dc.EndDialogAsync();
                            break;

                        default:
                            await dc.CancelAllDialogsAsync();
                            break;
                    }
                }
            }
            else if (activity.Type == ActivityTypes.ConversationUpdate)
            {
                if (activity.MembersAdded.Any())
                {
                    // Iterate over all new members added to the conversation.
                    foreach (var member in activity.MembersAdded)
                    {
                        // Greet anyone that was not the target (recipient) of this message.
                        // To learn more about Adaptive Cards, see https://aka.ms/msbot-adaptivecards for more details.
                        if (member.Id != activity.Recipient.Id)
                        {
                            var welcomeCard = CreateAdaptiveCardAttachment();
                            var response = CreateResponse(activity, welcomeCard);
                            await dc.Context.SendActivityAsync(response).ConfigureAwait(false);
                        }
                    }
                }
            }

            await _conversationState.SaveChangesAsync(turnContext);
            await _userState.SaveChangesAsync(turnContext);
        }

        // Determine if an interruption has occured before we dispatch to any active dialog.
        private async Task<bool> IsTurnInterruptedAsync(DialogContext dc, string topIntent)
        {
            // See if there are any conversation interrupts we need to handle.
            if (topIntent.Equals(CancelIntent))
            {
                if (dc.ActiveDialog != null)
                {
                    await dc.CancelAllDialogsAsync();
                    await dc.Context.SendActivityAsync("Ok. I've cancelled our last activity.");
                }
                else
                {
                    await dc.Context.SendActivityAsync("I don't have anything to cancel.");
                }

                return true;        // Handled the interrupt.
            }

            if (topIntent.Equals(HelpIntent))
            {
                await dc.Context.SendActivityAsync("Let me try to provide some help.");
                await dc.Context.SendActivityAsync("I understand greetings, being asked for help, or being asked to cancel what I am doing.");
                if (dc.ActiveDialog != null)
                {
                    await dc.RepromptDialogAsync();
                }

                return true;        // Handled the interrupt.
            }

            return false;           // Did not handle the interrupt.
        }

        // Create an attachment message response.
        private Activity CreateResponse(Activity activity, Attachment attachment)
        {
            var response = activity.CreateReply();
            response.Attachments = new List<Attachment>() { attachment };
            return response;
        }

        // Load attachment from file.
        private Attachment CreateAdaptiveCardAttachment()
        {
            var adaptiveCard = File.ReadAllText($@".{Path.DirectorySeparatorChar}Dialogs{Path.DirectorySeparatorChar}Welcome{Path.DirectorySeparatorChar}Resources{Path.DirectorySeparatorChar}welcomeCard.json");
            return new Attachment()
            {
                ContentType = "application/vnd.microsoft.card.adaptive",
                Content = JsonConvert.DeserializeObject(adaptiveCard),
            };
        }

        /// 

<summary>
        /// Helper function to update greeting state with entities returned by LUIS.
        /// </summary>


        /// <param name="luisResult">LUIS recognizer <see cref="RecognizerResult"/>.</param>
        /// <param name="turnContext">A <see cref="ITurnContext"/> containing all the data needed
        /// for processing this conversation turn.</param>
        /// <returns>A task that represents the work queued to execute.</returns>
        private async Task UpdateGreetingState(RecognizerResult luisResult, ITurnContext turnContext)
        {
            if (luisResult.Entities != null && luisResult.Entities.HasValues)
            {
                // Get latest GreetingState
                var greetingState = await _greetingStateAccessor.GetAsync(turnContext, () => new GreetingState());
                var entities = luisResult.Entities;

                // Supported LUIS Entities
                string[] userNameEntities = { "userName", "userName_paternAny" };
                string[] userLocationEntities = { "userLocation", "userLocation_patternAny" };

                // Update any entities
                // Note: Consider a confirm dialog, instead of just updating.
                foreach (var name in userNameEntities)
                {
                    // Check if we found valid slot values in entities returned from LUIS.
                    if (entities[name] != null)
                    {
                        // Capitalize and set new user name.
                        var newName = (string)entities[name][0];
                        greetingState.Name = char.ToUpper(newName[0]) + newName.Substring(1);
                        break;
                    }
                }

                foreach (var city in userLocationEntities)
                {
                    if (entities[city] != null)
                    {
                        // Captilize and set new city.
                        var newCity = (string)entities[city][0];
                        greetingState.City = char.ToUpper(newCity[0]) + newCity.Substring(1);
                        break;
                    }
                }

                // Set the new values into state.
                await _greetingStateAccessor.SetAsync(turnContext, greetingState);
            }
        }
    }
}

Testing out the QnA Service

At this point, you have added the QnA service that connects to the FAQ knowledge base you created to your bot. To test out the QnA service, build your bot and then run it in Visual Studio.

Open up the bot framework emulator version 4 and begin talking to the bot. If you need to provide a secret, it is the secret value in the appsettings.json file.

Since the bot will only use QnA as a fallback, I recommend you start the conversation with an FAQ question. If the bot is midway in between the greetings dialog it will not respond with the QnA result.

Here is an example conversation that uses QnA and then language understanding to perform greeting:

I hope this post was helpful to you. If you found this post helpful, share it with others so they can benefit too.

What are your experiences working with the bot framework?

To get in touch, you can follow me on Twitter, leave a comment, or send me an email at steven@brightdevelopers.com.

Resources

• Part 1: Setting up a bot framework development environment
• Part 2: Overview of basic bot generated code