Measuring ROI for advertising initiatives is a major challenge in 2026. You launch an advertising initiative, sales go up… but how much of that increase is directly attributable to your action? Without a precise answer, you risk over-investing in low-performing channels or underestimating the power of your best strategies.
This is where incrementality experiments come in. Unlike traditional methods that simply count clicks or conversions, incrementality experiments measure the true effect: the difference between what happens with your initiative and what would happen without it.
This complete guide explains how these tests work, how to set them up, and how to get the most out of them to optimise your advertising approach and maximise your return on investment.
- What is an incrementality experiment?
- Why are incrementality tests essential?
- Definition and principles of incrementality tests
- Differences from other measurement methods
- Examples of applying incrementality tests to advertising initiatives
- How to set up an effective incrementality test
- Best practices for running effective incrementality tests
- Which metrics should you track during an incrementality test?
- How long does it take to see incrementality test results?
- What tools make incrementality testing easier?
- Incrementality tests and Web Push: a particularly effective duo
- Conclusion
What is an incrementality experiment?
An incrementality experiment assesses the additional impact of an advertising initiative. In other words, it answers this question: how many extra sales did my advertising campaign generate?
To do this, the experiment compares two groups:
- Treatment group: users exposed to the campaign
- Control group: similar users but NOT exposed to the campaign
The difference in outcomes between these two sets is the incrementality. It is the most reliable assessment of the real impact of your marketing intervention.
A concrete example
You launch an email initiative to sell shoes. You send the email to 100,000 customers. Result: 5,000 purchases. At first glance, you might think your campaign generated 5,000 sales. But in reality, some of these buyers would have bought anyway, even without your email.
To find out how many sales are truly due to your email, you compare with a control group: 10,000 similar customers who did NOT receive the email. Without the email, only 400 purchases occur (natural conversion of 4%). So your email generated 5,000 – 400 = 4,600 additional sales. That is your incrementality.
Why are incrementality tests essential?
1. Cookies are disappearing, attribution is becoming blurry
For several years now, third-party cookie tracking has been eroding. Without cookies, it is impossible to track a user from ad to conversion with certainty. Incrementality experiments get around this problem by using statistical experimentation instead of cookies.
2. You are optimising on the wrong metrics
Many marketers focus on clicks, impressions, or recorded conversions. But these figures can be misleading. A user who clicks on your ad might have converted anyway. Incrementality tests show you the NET impact.
3. Marketing budgets are tight
With a potential economic downturn, every euro spent on marketing has to justify its existence. Incrementality tests let you reallocate your budget toward the real drivers of business, boosting your marketing profitability.
To go further, check out our analysis of Web Push campaign ROI.
4. Understanding your audience becomes strategic
Knowing WHY your campaign works (or doesn’t) on certain audience segments helps you refine your strategies. Incrementality tests can be segmented by demographics, geography, or behaviour.
Definition and principles of incrementality tests
An incrementality test is a controlled, randomised experiment aimed at measuring the causal impact of a marketing intervention on a measurable outcome (sales, sign-ups, clicks, etc.).
Key characteristics:
- Randomisation: assignment of users to the treatment or control groups must be random
- Isolation: the control group must not be exposed to the intervention being tested
- Identical except for the intervention: the two groups must be as similar as possible before the test
- Sufficient size: you need enough participants to obtain statistically significant conclusions
The statistical principles behind it
Incrementality rests on a simple logic: if two groups are identical at the start, and the only difference is exposure to your campaign, then the difference in outcomes is directly caused by your campaign.
Mathematically: Incrementality = Outcome (treatment group) – Outcome (control group)
For example, if 10% of the exposed group converts and 7% of the control group converts, the incrementality is 3 percentage points.
Differences from other measurement methods
Incrementality vs. attribution
The attribution model (or marketing attribution) assigns credit for a conversion to one or more touchpoints in a campaign. For example, a user sees a paid search ad, then a social media ad. They click, then buy; attribution determines how much credit to give each ad.
Incrementality, on the other hand, asks: would this conversion have happened without ANY of these touchpoints?
Key difference: attribution assumes all touchpoints were necessary. Incrementality assesses whether the entire intervention really changed the outcome.
Incrementality vs. Marketing Mix Modeling (MMM)
MMM (Marketing Mix Modeling) is a statistical method that uses historical data to estimate the impact of each marketing channel on overall sales.
Incrementality is an experimental approach that tests a specific change in real time.
Key difference: MMM looks at the past; incrementality creates the future through experimentation. MMM gives an overall picture; incrementality tests precise hypotheses.
Quick comparison
| Criterion | Attribution | MMM | Incrementality |
| Method | Rules/algorithms | Historical statistics | Controlled experimentation |
| Data required | Tracked events | Aggregated data | Treatment & control groups |
| Timeline | Immediate | Long (months of data) | A few weeks to months |
| Causal reliability | Medium | Good | Very good |
| Cost | Low | Medium | Medium to high |
Examples of applying incrementality tests to advertising initiatives
Example 1: Testing an email campaign
An e-commerce business tests an email campaign to promote its summer sale. It splits its subscriber base into two random groups:
- Treatment group (50,000): receives the promotional email
- Control group (50,000): does not receive the email
Results:
- Treatment group: 2,500 purchases (5% conversion)
- Control group: 1,500 purchases (3% conversion)
- Incrementality: 2 percentage points, or 1,000 additional sales
Insight: the email generated 1,000 additional sales, not 2,500. This is an important finding that refines the ROI calculation.
Example 2: Testing a paid advertising campaign
A cosmetics brand tests an Instagram ad campaign. It exposes one group of users to its ad and randomly excludes another.
Findings:
- Exposed group: 8% visit the website
- Non-exposed group: 6% visit the website (organic traffic or other sources)
- Incrementality: 2 percentage points
Insight: the Instagram campaign generated 2% additional traffic. This helps set a rational budget envelope based on proven results.
Example 3: A geographic test
A restaurant chain tests a local promotion in 50 cities. It activates the promotion in 25 cities (treatment group) and disables it in 25 other cities (control group).
Results:
- Cities with the promotion: 15% increase in visits
- Cities without the promotion: 8% increase in visits (natural growth)
- Incrementality: 7 percentage points
Insight: the promotion truly generates 7% additional traffic after subtracting natural growth.
How to set up an effective incrementality test
Step 1: Clearly define your hypothesis
Before any test, form a precise hypothesis. Examples:
- “This email campaign will increase sales by 3% above baseline”
- “This Facebook ad will generate 10% additional traffic to the site”
- “This sponsored content will increase brand awareness by 5 points”
Step 2: Choose your test population
Make sure your population is:
- Large enough: you need enough participants for statistical significance (usually 10,000 minimum)
- Representative: reflecting your actual audience
- Randomly divisible: you can assign users at random
Step 3: Design your experiment
- Treatment group: exposed to your intervention (email, ad, content, etc.)
- Control group: fully isolated from the intervention
- Test duration: long enough to capture the full impact (2-4 weeks minimum for most campaigns)
Step 4: Track the relevant metrics
Measure what really matters to you:
- Conversions (sales, sign-ups)
- Website traffic
- Engagement (time on site, pages visited)
- Retention (subscription renewed or not)
- Average order value
Step 5: Analyse the results
Once the test is over, compare the two groups. Statistical tools help you determine whether the difference is significant or due to chance.
Significance threshold: generally 95% (p-value < 0.05), meaning you are 95% confident the difference isn’t due to chance.
Step 6: Document and share
Record your findings. This creates a knowledge base on what works and what doesn’t work for your business.
Best practices for running effective incrementality tests
1. Ensure the purity of the control group
The control group must not be exposed to your intervention, even accidentally. Example: if you’re testing a Facebook ad, make sure control group members won’t see the ad through other means.
2. Avoid selection bias
Randomisation is key. Don’t let people “choose” to be in the treatment or control group, as this would introduce bias. Use an algorithm to randomly assign participants.
3. Test with a sufficient sample size
A sample size that’s too small can give misleading results. A statistical power calculator can help you determine the minimum size required.
4. Measure over an adequate duration
Too short, and you’ll miss the effect. Too long, and other variables (seasonality, competition) can pollute the results. 2 to 4 weeks is a typical duration.
5. Segment your results
Don’t settle for a single overall result. Segment by:
- Demographics (age, gender, location)
- Behaviour (new vs. existing customer)
- Context (channel, day of the week)
This often reveals that your campaign performs better on certain segments.
6. Calculate the real ROI
Don’t confuse an increase in conversions with profit. A test that boosts clicks by 10% is only worth something if it improves profit after costs.
7. Document everything
Every test should leave a trace: hypothesis, population, duration, results, insights. This creates a knowledge base for your team.
Which metrics should you track during an incrementality test?
Short-term metrics
- Clicks: increase in traffic to your site
- Impressions: number of times the user saw your message
- Immediate conversions: purchases or sign-ups within hours/days of exposure
- Click-through rate (CTR): percentage of users who click
Medium-term metrics
- Website visits: traffic sent to the site
- Engagement: time on site, pages visited, depth of visit
- Average basket: average value of orders placed
- Number of products per order
Long-term metrics
- Retention: customers who come back to buy again
- Customer Lifetime Value (CLV): total revenue generated by a customer over time
- Adjusted acquisition cost: accounting for the true incremental effect
- Churn rate. To understand how to anticipate and reduce this phenomenon, check out our article on churn rate.
Adrena’tips: choose 1 to 3 main metrics relevant to your business goals, rather than tracking everything.
How long does it take to see incrementality test results?
The duration depends on your sector, your traffic volume, and the size of the effect you’re testing.
Typical timeline
1-2 weeks: initial data, first signals, but not yet statistically significant
2-3 weeks: sufficient for most paid advertising campaign tests
4 weeks or more: necessary for products with long purchase cycles (cars, real estate, B2B software)
Factors that extend duration
- Low traffic volume
- Low baseline conversion rate
- Small expected incremental effect
- Seasonality (results vary depending on the time of year)
Accelerators
- Large user volume
- High conversion rate
- Large expected effect
- Test priority (increasing resources)
What tools make incrementality testing easier?
Tools specialised in incrementality testing
- Google Ads Incrementality Framework: free, built in for Google ads
- Meta Incrementality Suite: for Facebook/Instagram tests
- Booking.com’s AB Testing Tool: reliable for online commerce
- Convoy.ai: a platform dedicated to statistical testing
- Measure Studio: a collaborative tool for testing and documenting
General analytics tools
- Google Analytics 4: lets you create segments and audiences to compare. Check out our guide to measuring your advertising campaigns in GA4.
- Tableau or Looker: for visualising results and comparisons
- Python/R: for advanced statistical analysis
Incrementality tests and Web Push: a particularly effective duo
Web Push Notification is one of the advertising channels best suited to running incrementality tests. Here’s why.
Explicit opt-in that guarantees clean groups
Unlike other channels, Web Push relies on explicit one-click consent via the browser. Every subscriber is precisely identified in the database. This clarity in how the audience is built makes randomisation easier: it’s simple to create an exposed group and a control group without cross-contamination between the two.
Volumes compatible with statistical significance
For a test to be statistically valid, you need a sufficient volume of participants. Adrenalead has an ad network of more than 60 million opt-in subscribers addressable in real time. Even when targeting a precise geographic or behavioural segment, volumes remain well above the recommended threshold of 10,000 participants needed for reliable results.
Native GA4-compatible tracking
Web Push lets you embed UTM parameters directly in notification URLs. Every click is therefore trackable in Google Analytics 4, making it easier to run the differential analysis between the exposed group and the control group. No need for additional third-party tools to measure the incrementality of your Web Push campaigns.
A concrete use case: the abandoned cart test
The abandoned cart scenario is one of the simplest to test for incrementality via Web Push. You expose 50% of your subscribers who abandoned a cart to a follow-up notification. The remaining 50% receive nothing. The difference in conversion rate between the two groups gives you the real incrementality of your follow-up sequence, without confusing natural buyers with buyers genuinely generated by the notification. To go further on this topic, check out our complete guide on abandoned cart in Web Push.
Conclusion
Incrementality tests are the gold standard for measuring the true impact of your marketing initiatives.
Summary of key points:
- Incrementality measures causal impact, not just correlation
- It goes beyond older methods like simplistic attribution
- It requires testing and rigour, but it’s doable
- The results are worth the investment: optimise your marketing ROI
- It’s an ongoing process: test, learn, iterate
Start small: test a simple hypothesis, measure properly, document, and build a culture of decisions based on data rather than assumptions. Your campaigns (and your CFO) will thank you.
How do I know if an incrementality test is needed for my campaign?
Ask yourself this question: “Does this marketing action really generate results beyond what would happen naturally?” If the answer intrigues you, a test is warranted.
This is especially useful for large budgets or strategic decisions — where the stakes of knowing whether a spend truly creates value outweigh the cost of the test itself.
Which metrics should you track during an incrementality test?
It depends on your business goals. For an e-commerce campaign, measure purchases. For a brand campaign, measure awareness or engagement.
The golden rule: choose 1 to 3 key metrics, not ten. Multiplying indicators dilutes the reading of results and complicates decision-making once the test is over.
How long should an incrementality test run?
2 to 4 weeks is enough for most advertising tests. If you sell luxury products or offers with a long purchase cycle, plan for 4 to 8 weeks to capture the real effect.
The goal is to find the right duration: short enough to get actionable results quickly, long enough that the observed effects are representative of your audience’s real behaviour.
How much does an incrementality test cost?
The real cost of an incrementality test corresponds to the campaign budget allocated to the control group — the one that receives nothing. On a 50/50 split with 100,000 users, you “invest” 50% of your budget to gain knowledge rather than immediate conversions.
On a €10,000 campaign, that represents €5,000 of “sacrificed” budget. It’s an investment in knowledge — sometimes fully justified before committing large budgets over time.
Can you run several incrementality tests simultaneously?
Yes, but with caution. If you simultaneously test an email and a Meta ad on the same audience, it becomes difficult to attribute the result to one lever or the other — the effects intertwine and the test’s reliability suffers.
Isolate your tests as much as possible: separate audiences, staggered time periods, or clearly separated variables. A clean test on a single lever is better than two simultaneous tests whose results remain ambiguous.
What tools or platforms make it easier to set up incrementality tests?
Major advertising platforms natively include incrementality testing features: Google Ads (Campaign Experiments), Meta (A/B Tests and Lift Studies). This is often the most accessible starting point.
Specialised tools like Measure Studio or Convoy allow you to go further, particularly for cross-channel analyses or more complex setups. In any case, first check what your current advertising platform already offers natively — the feature is often already available.
