Page 29 - MSDN Magazine, July 2018

Figure 1 PersonalAccountsIntent Utterances with Their Confidence Scores
“I would like to get assistance about mortgage rates” “Whom can I speak with regarding mortgages?”
“What is the annual rate for the one-year savings account?” “What terms do you offer for mortgages?”
“Who is responsible for mortgages?”
“What are annual rates for savings accounts?”
“How are your mortgage rates compared to other banks?”
The plan is to use LUIS for natural language understanding of the user requests. One way to go about this is to create two intents and train LUIS to detect them.
Let’s call the first category’s intent PersonalAccountsIntent, and the second category’s intent OtherServicesIntent. You can then use the utterance examples previously listed to train LUIS. It will create a third “catch-all” intent automatically called None for general utter- ances, which should be very different from the first two intents. You can also provide additional examples for the None intent.
Ideally, you want your intents to be distinguishable from one another with a high degree of certainty.
After training, you can publish your model to production. You can also see the utterances along with the confidence scores for the different intents in the LUIS UI, as shown in Figure 1.
The dashboard offers some basic summary statistics about the application. If you look at the dashboard in Figure 1, you’ll notice that the lowest confidence score for PersonalAccountsIntent is 0.59 and is obtained for the utterance, “what are annual rates for my savings accounts?” The confidence score for this utterance to be classified as OtherServicesIntent is pretty close at 0.44. This means that LUIS is not very certain as to how to classify this intent. msdnmagazine.com
Ideally, you want your intents to be distinguishable from one another with a high degree of certainty, that is, to have one intent with a very high confidence score, while other intents have very low scores. If you revisit the utterance lists for both intents, you’ll see there’s another very similar utterance example (“what is the annual rate for the one-year savings account?”) that’s labeled differently as OtherServicesIntent.
Using this insight, you can fine-tune your utterance samples to use different and distinct words.
Here, I’ve presented seven utterance examples for each intent. But what if there were multiple intents (at the time of writing LUIS can classify up to 500 different intents) and many more utterance examples for each intent?
Clearly, a more systematic approach is needed to address such a challenge. In what follows, I’ll show how Scattertext and LIME can help.
Understanding Intent Classification Using Scattertext
Scattertext is an open source tool written in Python by Jason Kessler. You’ll find the source code and a tutorial at bit.ly/2G0DLmp, and a paper entitled “Scattertext: a Browser-Based Tool for Visualizing How Corpora Differ,” which explains the tool in detail, at bit.ly/2G05ow6.
Scattertext was conceived as a tool to visualize the differences and similarities between two collections of text articles, also known as corpora, and has various features you may find useful; for example, it also supports emojis.
In this article, I’m going to leverage the tool to produce a visualiza- tion of the differences and similarities between the utterance examples for the two intents, PersonalAccountsIntent and OtherServicesIntent.
To install Scattertext, which requires Python version 3, follow the installation instructions in the tutorial. I also recommended you install Spacy, an open source Natural Language Processing library (spacy.io) and Pandas (pandas.pydata.org), another open source library that lets you work with tabular data in-memory.
July 2018 23