Incrementality case study

Measuring for incrementality makes Dunelm’s Facebook Ads 90% more effective ​

Dunelm and Crealytics ran a randomized controlled experiment to compare the attribution system ROAS against incremental impact - with surprising results.

Sign up for our Incrementality Course

Sign Up For Our Incrementality Course

  • This field is for validation purposes and should be left unchanged.

01 Introduction

Incrementality: theory vs practice

Digital Marketers widely understand the importance of incremental performance measurement. A recent survey even shows that the marketing leadership teams of large e-commerce brands consider incremental revenue as the most important KPI in Digital Marketing.
Source: Crealytics and WBR insights retailer survey 2020. n=100
Yet 61 out of 100 companies have not even started to execute incrementality tests. Isn’t that strange for something considered to be the most important KPI?

The explanation for this paradox is that even digitally savvy organizations struggle to bridge the gap between theory and practice.

This case study shows how a home furnishings retailer and Crealytics have bridged this gap together by measuring and improving the incremental return of Facebook campaigns.

About Dunelm

Dunelm is one of the largest homewares retailers in the United Kingdom. It is listed on the London Stock Exchange and a constituent of the FTSE 250 index.

There is more than meets the eye; any attribution model is likely to over- or undervalue the incremental value of any campaign.

Digital Marketers have long depended on ad platform reports and multitouch attribution tools to understand which campaigns drive marketing returns for their business – and they act on the reported numbers by allocating marketing budgets.

But alas, most attribution reports are wrong and can even cause harm. The numbers reported to marketers do not show the true financial impact of marketing channels and campaigns. To understand true causal impact, incrementality testing is needed.
  • In classic holdout tests, a test group gets to see ads while a control group does not. Comparing the resulting buying behavior of both cohorts delivers first estimates on how much revenue would be lost without advertising in that channel. This is what we define as the incremental impact.

  • Knowing incremental campaign value is only the first step. The final goal is to act on insights by re-allocating budgets to maximize the incremental return on ad spend (3).

02 The Challenge

Exploring new ways to move the needle

Dunelm and Crealytics wanted to understand how much they could move the needle by taking incrementality optimization from theory to practice. The chosen approach was to first measure the incremental ROAS of Facebook Ad campaigns, and then to increase it as much as possible – while keeping monthly ad budgets stable. The Facebook campaign structure mimicked a classic e-commerce conversion funnel with two major segments:
  • Prospecting campaigns to target people with no prior visit to the retailer’s website

  • Retargeting campaigns to advertise to past website visitors

Dunelm and Crealytics had two hypotheses in mind:

Prospecting campaigns might be undervalued. We believed that because they initiate new customer journeys, they do serious heavy lifting that may have more impact on revenue than reflected in multitouch attribution.
Retargeting campaigns might be overvalued. Because retargeted people are in the lower funnel, they might buy anyway without being exposed to more ads
If the hypothesis turned out to be true, Crealytics’ Digital Marketing team would re-allocate budgets from where they shine in traditional attribution reporting (Retargeting) to where they truly drive more impact (Prospecting).

But before any execution, these counterintuitive tactics needed proof in data.
03 The approach

Retargeting vs Prospecting

To gain a better understanding of incremental impact, Crealytics ran a randomized controlled experiment, supported by Facebook’s Conversion Lift tool, to compare the attribution system ROAS against incremental impact.​
Facebook's Conversion Lift tool

What is Facebook’s Conversion Lift tool?
Conversion lift compares the actions of real people in randomized test and control groups to measure the additional online, offline or mobile app business driven by Facebook, Instagram and Audience Network ads.

After roughly six weeks of testing, the Conversion Lift Test showed statistically relevant results for both prospecting and remarketing campaigns.
  • During the test, budgets for both segments were balanced to achieve the same nominal ROAS according to client’s Multitouch Attribution. Under a traditional valuation, both campaigns delivered the same advertising efficiency.

Yet, by including the incrementality measurements of the Conversion Lift test, a surprisingly different valuation emerged.
  • The remarketing campaigns incremental ROAS was only 44%, while the acquisition campaigns drove a staggering incremental ROAS of 298%. In other words, the multitouch attribution model underestimated its true value by two-thirds!

To put things into perspective: the Dunelm’s business roughly got a 5.7x higher return from prospecting than retargeting.
04 The Results

A 5.7x higher return from prospecting vs retargeting​

In line with the test, Dunelm decided to move Facebook ad budgets from retargeting to prospecting, gradually increasing its spend share from 55% to 70%.

As a result, the Dunelm’s incremental Facebook ROAS increased by +90%, driving nearly twice as much impact for every advertising dollar. While incrementality optimization can seem daunting at first, it is worth implementing as it is one of the biggest levers to really drive business impact.

Upgrade your customer portfolio

CLV-Centric Advertising: Customer Centricity,
Corporate Valutation, and eCommerce Acquisition

Ready to get started?