Customer Cohort Analysis
Analytics is hard. What is harder than analyzing data is coming up with actionable insights to convert more. Most people are confused with the cohort analysis because they know that their reason to analyze data is to gain actionable insights, however, cohort analysis is hard to interpret. In this article, we are going to explain cohort analysis in-depth but simple enough to enable you to interpret your users' actions. Let's start with the definition.
Definition of Cohort Analysis
As a definition, cohort analysis is a technique that focuses on the behavior of specific 'cohorts' of visitors/users overtime to uncover insights about their experience with your website/product. We can say that cohort analysis is a combination of behavioral analysis and user experience.
What makes it different from others and a little difficult to interpret is cohort analysis is dynamic. Instead of summing your session or page activities over a fixed time range, cohort analysis describes behaviors of specific groups of users OVER TIME on your website - which makes the analysis somewhat harder to interpret and turn it into action.
Cohort Analysis is Great For
The cohort is super helpful for a variety of things but it is especially helpful to analyze your visitors' experience and behaviors after changing your website. No matter what you change you change to increase your UX/UI or to add value to your product. With cohort analysis, you can understand whether it worked and how it affected.
You can start your cohort analysis by choosing a group of customers and a time period. You can choose your group by their referrers, age, industry, etc. Let's say you chose SaaS founders living in Estonia and a week of the time period. What you have to do before changing your website is to analyze this group's experience with your product over a week. After you change the website analyze the same group over a week. And compare the results to see the effects of the change you made on SaaS founders living in Estonia and their experience with your product. What you might see can be a decrease/increase in retention, increase/decrease in session time, etc. With this comparison, you will interpret the effectiveness of the change you made. Other examples to analyze with Cohort Analysis:
- Ad content
- Channels
- Campaigns/experiments
- Website redesigns
- New product lines and service offerings
- Sales, discounts, promotion campaigns
Pitfalls in Cohort Analysis
There are some cases that you cannot identify a user as a part of a cohort thus cannot analyze their behaviors and experiences with your product. Some examples are as following:
- Clearing browser cookies
- Visiting site on a different device or browser
- Visiting the site on incognito mode
Other than these factors there might and probably will be confounding variables which are factors that affect the dependent variable, in this case user behavior, other than dependent variable, the change you made. Confounding variables could be the device your user is using to access your website or time of day they use your product. These variables can affect their behaviors. To decrease these factors' effect you have to continuously use Cohort Analysis and compare results between time periods. But always remember to compare the same cohorts.
Another factor that can affect the cohort analysis is a discount. Discounts can affect your users' retention and they must be carefully analyzed. If you make a %50 discount this week, your retention will probably increase. However, you should use cohort analysis and look if other reasons are also increasing your retention such as your new features and design. The only way to be sure is again comparing the results of the same cohort. If you use Cohort Analysis after the discount and see the same retention then you can come to a conclusion.
Vanity Metrics in Cohort Analysis
Vanity Metrics, again. Cohort Analysis is really important but there are still vanity metrics to consider after your analysis to make the best out of your analysis and efforts. Lean Startup has a great example of this. Let's say we wanted to measure your app's engagement by number of photos sent. Each week, we take all the users who joined, and then look at the average number of photos each user send in their first day. We work really hard for 4 weeks, and we hope to see this number rise. Instead, we see this:
| Weeks | Number of photos sent per user | Total photos sent |
|---|---|---|
| Week 1 | 5 | 100 |
| Week 2 | 5 | 200 |
| Week 3 | 5 | 350 |
| Week 4 | 5 | 400 |
Did you notice how total photos sent increases but the metric that should be increasing, the number of photos sent per user is still 5. This is an excellent visualization of a vanity metric. What this example tells us is choosing the right metrics is essential for Cohort Analysis. If you chose total photos as a metric, you cannot grow.
| Weeks | Number of photos sent per user | Total photos sent |
|---|---|---|
| Week 1 | 5 | 100 |
| Week 2 | 6 | 250 |
| Week 3 | 8 | 400 |
| Week 4 | 10 | 600 |
Conclusion
Cohort analysis is a hard to interpret data analysis technique but if you can understand it, it is extremely useful to understand a group of users' behaviors and experiences with your product. We hope that this simple guide helped you to better understand Cohort Analysis.
Stay Optimized!