Chatbots Magazine • Become a Writer • Intro to Chatbots

Segmentation & A/B Testing Dialogue

Has anyone had success with segmentation & A/B testing with bot dialogue with the aim of building user retention?

I got in touch with Mixpanel to see if their platform could support this and it looks like they have some good options to do so (happy to share their answers if anyone finds it useful). It may even be possible to start new conversations based on segments using notifications. While Mixpanel supports email, push, SMS, and in-app messaging in our out-of-the-box mailing system, you can also set up a webhook campaign that delivers a user list to an external server of your choice.

I also took a brief look through Intercom and I’m sure they could do the same.

Whilst this is on the roadmap it’s not yet implemented so I can’t report back any learnings as yet. Interested to hear if anyone else is further down this path!

5 Likes

@hugh please do share.

Hey @hugh try www.getbotmetrics.com supports user segmentation with targeted notifications.

2 Likes

@sandeep - thanks for the tip! I’ll check it out.

@tom - see the Q&A with Mixpanel below:

I’m developing a bot for Slack, see more at button.ai, and I’m curious to understand if Mixpanel has features that can help me with the user experience. Here are some specific questions:

Integration via Segment
I’m using Segment.com to track events and the bot is built in Node.JS using the Microsoft Bot Framework. Given that this is a bot and not a web/mobile app should the Segment tracking include any additional information to ensure Mixpanel has everything it needs to identify teams, users and events?

You can certainly send data to Mixpanel via Segment’s Node library–because this is a server-side library, the data you send in will not include the default properties that are automatically collected by our client-side tracking libraries. This just means that if you’d like to include this data, you’ll need to set these details as properties explicitly as you would any other detail about your events/users.

In the same vein, Mixpanel will not automatically set a distinct_id for your users (as is done by our client-side libraries) when you’re using Segment’s node library, so you’ll need to set this explicitly yourself in order to keep track of unique users. Here’s some more information about controlling you user identity in server-side tracking that may be a helpful reference.

A/B Testing Bot Dialogue
User on boarding and education is our big focus right now. Whilst we have a help centre on the site using the bot to steer users in the right direction is crucial. Currently, bot dialogue is stored within the Node JS app - could Mixpanel help us test different dialogue versions and measure the impact on user retention?

You can indeed use Mixpanel to A/B test different user experiences–the simplest way to do this is to set a property like “Variant ID” with the events sent in by a user in a particular user experience version. For instance, if your user signs up after communicating with the bot’s dialog version A, your track call that captures this could look something like this:

analytics.track({
userId: '019mr8mf4r',
event: 'Sign Up',
properties: {
Dialog Version: 'A'
}
});

Once these different dialog versions are stored as properties of your events, you can apply it segment condition in your reports to see, for example, how many signups are occurring as a result of each version.

Segmented Messages
Mixpanel has great segmentation capabilities for messaging. Chatbots rely on a channel, in this case Slack, to access the user. Would it be possible to segment and message users via a channel using Mixpanel?

While Mixpanel supports email, push, sms, and in-app messaging in our out-of-the-box mailing system, you can also set up a webhook campaign that delivers a user list to an external server of your choice. If you were to make sure that users have a Channel property set on their profiles, and set up a webhook that targets users in a particular channel, you could then configure a receiving server to accept this webhook and direct messages to the users in this channel. Here’s some more information about setting up a webhook campaign.

1 Like

@hugh thanks! Really helpful stuff.

Thanks for sharing this @hugh. Is it me or even though it would work with Mixpanel, it feels a not so suited for the job?