We have been running an experiment in the Open LMS trial experience to measure the effect that in-product onboarding could have on user retention. We are happy to share the results in the community in case you're considering similar approaches or you want to become part of our Onboarding experimentation group
We ran a test for 61 days and 2 cohorts: cohort A was our control group and had no exposure to in-app onboarding, cohort C (yup, we call it AC testing, not AB 😜) was exposed to in-app onboarding elements such as checklists, tours, and tooltips to help users to create their first course.
We tested this with a total of 106 users with 55 of them in cohort A and 51 in cohort C.
We measured multiple components, like time spent in the trial, conversion to various marketing/sales actions but there were 2 components in terms of engagement that we want to highlight because we're convinced that we found an interesting path here.
The first one is the retention rate across the 30 days of the trial experience. We found significant differences in the entire period, but it is especially interesting to see the effect on the first 4 accumulated days on the first week. This is what we call a 4-day product activation rate and the difference is considerable between cohorts (17.54% in cohort C vs 4.74% in A, it may not seem huge but remember, these are trial users and they don't usually stay for long!).
In this graph, cohort C is orange and cohort A is purple
The second metric is the average number of activities that users created during a day. We also found that people exposed to the onboarding flows (cohort C) created on average more activities than those that weren't exposed to it (4.9 vs 1.7).
In this graph, cohort A is orange and cohort C is purple
Limitations and the path forward
Although the results are exciting, we're also aware of the limitations of this test, including the number of samples, the fact that trial users are different from long-term users, or even how we designed the onboarding experience.
That said, we believe the results are pointing us in a direction that we are convinced is important for our customer's experience and that it opens new doors and questions for us to validate in new experiments. Specifically, we would love to partner with clients to design similar tests and experiences for real use cases and long-term users.
If you're interested in participating, join the Onboarding experimentation group or send me a message. I'd be happy to connect!