Chatbots Magazine Become a Writer Intro to Chatbots

UX research for chatbots

(Nei Fonseca) #1

So, i was thinking about that. How are you guys conducting user research and validating ideas in the chatbots building process?

(@tomhewitson - conversation designer) #2

Mainly I spend my time sifting the chat logs to see what the user is trying to accomplish and how hard it is for them to succeed. Right now this is a pretty manual process so I just have to do it on a sample of chats - I’d love a tool that let me do this at scale.

(alexis) #3

What about existing analytics / chat logs review tools like Bot Metrics ?

(Nei Fonseca) #4

But are you guys doing previews research like it’s done with APPs?

I’m thinking if also it is possible trough an interview, focus group, quetionary and so predict some keywords, sentences, user behavior that can help build the chatbot.

@tom i’m doing the same.

@alexis thanks for the tip

(Ari Zilnik) #5

Coming from a HCI/UX background, I’ve tried to do a lot of UX research while building out Emoji Salad. We prototype a lot and do task-analyses with users. For example, we often get one of our team members to ‘play computer’ so to speak; ie: play the role of the chatbot before we actually build it out. They follow pretty strict rules, and we ask users to describe their thoughts, feelings, and understanding of the system as they are going through the system and workflow. We find this super helpful. We also are looking to build in a feedback and live support system directly into our app itself, to allow us to diagnose and help users directly in-context.

I’ve found UX research really helpful with building Emoji Salad, and try to react quickly to user feedback. My co-founder @thekevinscott wrote a great piece on usability heuristics applied to bots for Chatbots Magazine, would love to know what you think.

(Nei Fonseca) #6

Hey @ari thanks a lot!

Great article, i thinking that is important start to look at chatbot the way Kevin and you did to understand how to make them really work and create real value for users and costumers.

But i also think the some of the points that he brought up it’s caused by technological limitation, what do you think?

And if a may ask, did you guys while researching did some specially to identify user key words and common phrases that could help write the conversation and bot persona or this was indenfity while testing?

(ashley.e.cribb) #7

@ari Just wanted to say I just starting using Poncho … Love the interaction. It’s one of the ways I plan to do and is a great model on how to interact with your stakeholders.


(Ari Zilnik) #8

Some great questions @nei.fda

Yes some of the issues Kevin identified deal with technical limitations for sure, but in general we as designers try to think through and around those limitations—maybe not all of them can be addressed now but that doesn’t mean we’ll few facing those same issues in the future. What do you think?

Regarding capturing user messages and running analytics on them—yes we do it and yes it’s useful! We were able to detect, for each message sent byour bot, what the most common patterns of response are, and have tweaked our messaging based on that.

(efrat) #9

We have built a tool that tries to tackle this challenge - you can use it to perform advanced conversation filtering, understand the context of specific (desirable or not) events, identify specific patterns that need your attention and see your bot’s interactions with users on a timeline of events. It’s designed to be a powerful companion to any bot owner - If you would like to try it just PM me, I would really appreciate your feedback @nei.fda @tom @ari

(Ari Zilnik) #10

Sure, I’m game. send it over!

(Mark Stephen Meadows) #11

Awesome question - what else is a chatbot if not a user experience? Put the right thing in front of the right person at the right time. We think of the chatbot’s personality as the UX and so it’s super-frikkin’ key to the development of the system. We go through a series of steps to round up the user goals, business needs, and success metrics. Then we research the market, gather target user personas / profiles, do field research, make notes, and look at how people in the field that are in the role of the “Helper” (which is often what the chatbot will do - help) help, then collate that into research findings. That gives us parameters for the personality development. After all, if you don’t know who the user is, and therefor what the use is, you won’t know what you’re building.

Here’s a snapshot of our UX research stuff. We’ve taken this through some dozen builds for 3D characters and so far so good. Small tweaks, but this is the core of the thing:

(Nei Fonseca) #12

hey @ari sorry take so long to answer.

And yeah, i agree with you. I believe that we as designers have to think foward, always trying to stay one step ahead. Everything evolves so fast, that issues will keep happening and changing, so we can’t let this stops us ( so motivational…rs )

(Nei Fonseca) #13

Hi @efrat can i test your tool?

(Nei Fonseca) #14

Great!!! I’ll definitely check your design process closer. :grin: Thanks.

(efrat) #15

Sure @nei.fda - just PMed you!