Chatbots Magazine • Become a Writer • Intro to Chatbots

What kind of analytics do you need to improve your Bot's UX?

(Sandeep) #1

Most of us are used to leveraging both web and mobile analytics to drive data driven improvements to our websites and mobile apps. Chat is a bit different because of the non-linear flow.

How do you use analytics to improve your bots?
What are the top 3 metrics that you look at?
What do you wish you were able to figure out easily?

(Joseph Burchett) #2

I don’t think it’s really all that different. Just like in a web an mobile app you keep track of the parts of the experience that matter, not try and collect data based on everything they do.


If you are building an e-commerce bot then you’d most likely want to know what products people are clicking on the most. So you’d probably only collect data about what product name or button they are pushing the most. Not collecting data on every single action they take… That is just useless data.

(Mike Bailey) #3

We have literally 1 KPI that drives everything we build. User/Bot Interactions (Messages sent/received).

Even though we track tons of analytics, this is really the only one that means success for us. Now for our customers that could be different, it could be video views, link clicks, products sold. However in the end it’s keeping users engaged and interacting that drives their conversion goals as well.

Here are my short answers to your questions.
How do you use analytics to improve your bots? - We use analytics around bounce rates, general bot confusion, so if a user isn’t progressing down a tree we track/fix it.

What are the top 3 metrics that you look at? - Interactions, messages/session, users.

What do you wish you were able to figure out easily? - Figuring out where we went wrong and why the user isn’t coming back is a big focus for us right now and probably the hardest thing to figure out without manually going thru conversations.

(Vinay Chandra) #4

Interaction with Bots should be close to how we interact with humans. Typically, if someone asks a question then based on the response the user asks another question and so on. So for the first main question typically 3 to 4 followup questions are asked by a user. We call them “Nested Bot Interaction”. During this interaction, you will encounter nested questions such as Why or How or others which are very generic questions BUT very contextual to the main question and the earlier followup questions.

Therefore, tracking UNIQUE Nested Bot Interactions is critical for improving user experience with your Bot.

(Philippe Dionne) #5

Is your bot on Slack or on Messenger, Kik, etc? Is your goal to drive transactions? Based on my experience, the type of bot drives the type of metrics, just like the type of website drives the important KPIs.

(josh) #6

You need to understand your bot’s fail rate, to determine where to optimize it. If you get better insight into where and why your bot fails, then you’re in a better position to improve it. is analytics that helps uncover problems in your UX so you can optimize it.

(Sandeep) #7

Vinay - Good point. Today how do you identify when you’re in the nested unique interaction and when the user has moved to the next question?

(Sandeep) #8

Philippe - Yep. So what type of bots are you building currently and what KPIs have you been instrumenting/found important for these bots?

(Sandeep) #9

@Mike - Got it … it sounds like there is a goal that you’re moving the Chatbot users towards in these bots. When you say " if a user isn’t progressing down a tree we track/fix it." What is typically at the end of the tree? i.e. The goal?

(Vinay Chandra) #10

Good question… typically in a Response List if you have gone down the path of “previous” tag multiple times then it is a nested interaction. But identifying unique nested interactions is challenging for sure. Right now for training a bot we identify this manually, which is super time consuming. We haven’t figured it out completely this piece of the puzzle.

(Mike Bailey) #11

@sandeep That really depends. Sometimes it’s a weblink, or a video delivered thru messenger, maybe other times it’s simply to get to the end of the conversation tree, other times it’s been to take a selfie and send it to the bot. But all have one thing in common a conversation funnel that leads to a goal of some sort.

(Philippe Dionne) #12

@mike would graphing these conversation trees be helpful? I guess it would reveal a few nested levels and most of the “traffic” would be concentrated in the main trees.

(antonin) #13

What solution do you guys prefer? Botanalytics? Botmetrics? Why?

(Joseph Burchett) #14

None. The problem with these bot specific analytic services (at least for us) is they don’t support enough platforms fast enough. We are using since it’s open to everything instead of being locked into a particular platform.

We have to do a bit more setup and grunt work but it’s worth it because if we choose to one day start supporting IRC, Discord, Line, HipChat, WhatsApp, Telegram, and many others we can without restriction.

(efrat) #15

You are welcome to try out, we support all platforms, and do a lot more than just analytics. Our focus is marketing automation + user and behavioral insights, and we deal with all the different metrics that were mentioned in this thread. PM me for a personal invite

(Joseph Burchett) #16

@efrat I’d be interested if what you guys are doing is flexible enough to hook into new platforms on the fly.

(efrat) #17

We do, send me your email and I will follow up with an invitation and all the info you need! @jburchett

(Sandeep) #18

@jburchett - Which platforms do you need support for today?

Botmetrics is focused on behavioral engagement on top of analytics. It’s open source (with a 1-click deploy.)

So the data is not locked in and modular so in case we don’t support your entire range of platforms (IRC, Discord, Line, HipChat, WhatsApp, Telegram) it’s easy to extend without restriction. - if you want to chat!