Discover Card: How we designed an experiment to evaluate conversational experience platforms

Discover Card: How we designed an experiment to evaluate conversational experience platforms

2018-11-08 admin GoogleCloud No comments

[ Editor’s note: Today’s blog post comes from John Coyne, senior software engineer at Discover Financial Services. Discover is developing a conversational experience platform with the help of Dialogflow Enterprise Edition .]

Our mission is to help people spend smarter, manage debt better and save more, by providing banking and credit products that help people achieve their goals. But this mission isn’t possible without customer service that truly meets their needs. So we’re continuously looking for ways to improve customer satisfaction.

One way we’re doing this is by evolving our digital service channels—like our in-app mobile messaging—to serve our more digital-centric consumers. Through AI and conversational automation we hope to improve the speed and experience of these channels. We know time is valuable to our customers, and AI can be a tool that gives more of this valuable currency back to them.

To find the right AI solution to help us evolve our digital service experience, we compared a number of options and ultimately decided on Dialogflow Enterprise Edition as our development platform. In this post, I’ll give you an overview of our methodology for comparing natural language understanding accuracy based on our own use case. Anyone doing a similar comparison should be able to apply or be inspired by the same principles.

How we designed our experiment

Based on our own internal data, we knew the top 10-20 questions that customers ask via our mobile messaging feature, and that many of those interactions could be automated to help them get accurate answers in the fastest amount of time.

It was important to us to thoroughly test the best available platforms for this requirement, which included many of the public cloud providers. To do that, we wanted to build a model that would let us quickly test hundreds of iterations with a feedback loop that would help the testing process. In conjunction with performance testing, we also started designing the solution with security and customer satisfaction as key drivers in ensuring success.

In designing the experiment, we decided to test and record experiments around intent detection. In conversational tech lingo, an intent represents a mapping between the intention behind a user request (e.g. “update address”) and what action should be taken by your bot (e.g. directing the user to the appropriate form or documentation). By making intent detection our main metric for evaluating platforms, we were essentially asserting that if a bot can’t detect intents accurately, then the user experience is guaranteed to be poor.

In evaluating performance for intent detection, four core metrics were generated on a per-intent basis:

• True positives: Expected Intent equals i and Detected Intent equals i

• True negatives: Expected Intent does not equal i and Detected Intent does not equal i

• False positives: Expected Intent does not equal i and Detected Intent equals i

• False negatives: Expected Intent equals i and Detected Intent does not equal i

For our purposes, we wanted to minimize false positives. We would rather have the bot say, “I don’t know what you mean” (or route to an agent), rather than supply an incorrect response.

We used two advanced metrics (both measured from 0.0 – 1.0) to gain more insight from the core metrics described above:

• Precision: Precision can be summarized as: “Of the total number of tests where (i) was detected, what percentage were correct?” We used this metric to determine if we had problems with false positives–i.e., the bot detects an intent, but the wrong one.

• Recall: Recall can be defined as: “Of the total number of tests where (i) was expected, what percentage were detected correctly as (i)?” We used this metric to determine if our intent was too narrowly defined, and was missing messages it was intended to pick up.

These metrics are complementary for the purpose of measuring performance. For example, Precision would be 1.0 with only one true positive as long as no false positives are present. But if Recall is low for the same intent, that means it is missing true positives as well as false positives. In this scenario, the bot does not add much value since it misses most messages entirely, even though it does a good job at avoiding incorrect guesses.

The training process

We started with a labeled dataset and ran hundreds of tests with variations of phrases. These phrases include requests like “I would like to pay my balance” and “What is my balance”. We understood that in being able to correctly identify roughly the top 10-20 intents, most customer requests could be fulfilled. Also, in identifying certain intents that would be more complex to resolve, we could identify when to route the request to a qualified live agent to ensure the best possible customer experience.

The following diagram illustrates an overview of the training process, which is also summarized below.