Incrementality Testing for Retention: How to Measure What Really Lifts Repeat Purchases

Incrementality testing measures the extra revenue or profit an action creates by exposing only a randomly assigned share of customers to it. For retention, tag customers into ten groups by the last digit of their customer ID, apply the tactic to some groups, and compare repeat rate and CLV in a cohort report.

The figures below are an example.

In our previous post on retention economics, I tried to show you that looking at the average CLV is misleading. As in, it doesn’t really show how much extra lifetime value you generate from an incremental second purchase.

Here is the one thing you need to have in place as an eCom brand if you want to take your lifetime value seriously: incrementality testing. Here is what I mean and how you can easily implement it.

What is incrementality testing?

With incrementality testing you can measure the additional revenue/profit you generate from taking a certain action.

Why does incrementality testing matter for retention?

It matters because retention tactics suffer from selection bias and because CLV only shows the result after the fact.

There are two main reasons why incrementality testing is important, especially when it comes to retention:

1. Selection bias: all the things that you do to boost retention, like email, WhatsApp, an app or a loyalty program, are things that your best customers are more inclined to use in the first place. So it won’t be a surprise that the CLV of customers who downloaded the app will be higher.
2. CLV is a lagging indicator: a higher CLV is the outcome of good retention and therefore makes it tough to analyse what led to it.

How do you set up an incrementality test for retention?

Split your customers into randomly assigned groups with customer tags, and expose only some groups to the new retention tactic.

Quite simple. Use customer tags to group your customers randomly into different groups, e.g. 0-9 based on the last digit in their customer ID (this can be done automatically via Shopify Flow, for example). Then whenever you want to implement something that increases retention, you only expose a sub-segment of your customers to this approach. For example, you only invite people in groups 1+2 to download the app over the course of a month but leave out the remaining groups.

Then you have a clear, randomly assigned split of which customers were asked to download the app. You’ll find a more detailed explanation of how to set this up for your shop in the section on customer tags below.

How do you read the results of an incrementality test?

Compare repeat rate and CLV of the exposed groups against the rest in a cohort report; because the split is random, the difference is the incremental effect.

With this split, you can clearly measure the incremental impact of asking people to download the app. Simply use cohort reports in Klar to cluster your customers based on these customer tags and select the date range during which you ran the split. Then you can compare the repeat rate and customer lifetime value of these customers.

In the example, we can see that groups 1+2 have a repeat rate of around 40% whereas the remaining groups are at around 35%. An uplift of 15%. And CLV is also higher at €160 vs €125. An uplift of 28%.

Since customers are placed into the groups completely at random, the only difference between them is whether or not you asked them to download your app. Thus, you can be sure that there is a clear incremental uplift. In this case €35 per customer in contribution margin. At around 20,000 customers this store is acquiring per month, that results in a total additional contribution margin of €700,000.

Now to be honest, this is not rocket science. Yet, I can count the number of brands doing it on one, maybe two hands. In times where profitability is priority number one, two, and three, every investment needs to prove its worth. And randomly created cohort incrementality tests are the best way to do just that.

How do you set up the customer tags in Shopify?

Create a Shopify Flow that tags every new customer with the last digit of their customer ID, and upload the same tag for existing customers.

With Klar, this is simple to do if you have the right customer tags in place, where each customer is randomly assigned to a group and only a specific group or groups are exposed to the new tactic. You can then use the cohort report to see how these customers behave relative to everyone else, and since the groups are created randomly, the difference in CLV will be the incremental impact. So how do you set up these tags? You just need to do two things:

1. Set up a Shopify Flow for new customers

Build a new flow in Shopify where new customers get automatically assigned a tag based on the last digit of the ID in their customer profile. This only takes 5 minutes.

2. Upload tags for old customers

For existing customers, you unfortunately need to do a manual upload. But again, this is simple. Just download a list of your customers and copy this Excel formula to create a new tag based on their customer ID:

=IF(T2="","Incrementality Test - "&RIGHT(A2,1),T2&", Incrementality Test - "&RIGHT(A2,1))

Then reupload your customers and all tags will be added. And that’s it. It takes around 15 minutes to do, but will give you everything you need to get started with incrementality testing.

There’s also a full step-by-step video guide on how to set this up:

What can ecommerce brands learn from Pip Decks' creative collection?

You might not know Pip Decks. But they are the hidden champions of paid social ads, scaling to mid 8-figures with “just” card decks. Card decks for over $100. What stands out in their creative strategy? Diverse creatives and personas. Platform-native content. A benefit-driven approach.

I created an overview of their 32 top-performing ads that are getting the most reach (meaning the most spend) in a Miro board and went over them in a deep-dive video. Get access to the Pip Decks overview.

FAQ

What is incrementality testing in ecommerce?

Incrementality testing measures the additional revenue or profit you generate from taking a certain action. For retention, you compare customers who were exposed to a tactic with a randomly assigned group who were not.

Why is average CLV misleading when you evaluate a retention tactic?

Two reasons: selection bias and timing. Customers who use an app, email or loyalty program are the ones already most inclined to buy again, so their CLV is higher anyway. And CLV is a lagging indicator, so it is hard to tell what caused it.

How do you create random test groups in Shopify?

Build a Shopify Flow that tags each new customer with a group from 0 to 9, based on the last digit of the customer ID. The step takes about 5 minutes. You then expose only some groups, for example groups 1 and 2, to the new tactic.

How do you tag existing customers for an incrementality test?

Download your customer list, use an Excel formula to build a tag from the last digit of each customer ID, and upload the list again. The work takes around 15 minutes, and a video walkthrough is included.

How do you read the result of a retention incrementality test?

Use a cohort report to cluster customers by the test tags and select the date range of the split. Then compare repeat rate and CLV. In the example, groups 1+2 reached about 40% repeat rate against about 35%, and a CLV of €160 against €125.

Which retention tactics can you test this way?

Any tactic you can show to only part of your customers, such as email, WhatsApp, an app or a loyalty program. The example invites only groups 1 and 2 to download the app over one month.

How long does setting up an incrementality test take?

Around 5 minutes for the Shopify Flow plus around 15 minutes for existing customers.

Related reading

• Retention Mindset: Why Average CLV Hides What a Second Purchase Is Worth
• Finding the ideal CLV/CAC ratio for your ecommerce business

About the author

Max Rast is Co-Founder and CEO of Klar. He has built one 8-figure and one 9-figure eCom brand and works with 2,000+ eCom brands at Klar. LinkedIn