Why Incrementality Testing is the Dumbest Thing You Could Possibly Do in Marketing

In marketing, buzzwords and trendy tactics often dominate the conversation. Incrementality testing is one such tactic that's been heralded as the ultimate solution for measuring the true impact of your marketing campaigns. But is it always the best approach? While incrementality testing can yield valuable insights for some businesses, it could also be the dumbest thing you could possibly do in marketing—if you're not the right size or stage.

Here's why incrementality testing, particularly geo-holdout tests, might not make sense for your business.

1. High Costs Outweigh the Benefits for Smaller Companies

Geo-holdout testing requires significant investments in both time and money, often making it prohibitive for smaller companies.

Why it's Dumb for Smaller Businesses:

• Lost Sales Opportunities: Turning off advertising in key geographic regions can result in substantial revenue loss. For brands generating less than $10m in online sales or $50m in retail sales, these losses can significantly impact overall profitability.
• High Statistical Thresholds: Smaller businesses often lack the necessary sales volume to achieve statistical significance in holdout tests, leading to inconclusive or misleading results.
• Cost of Execution: Designing, executing, and analyzing geo-holdout tests can be resource-intensive. For businesses with limited marketing budgets, this means diverting funds from high-impact channels into testing that may not yield actionable insights.

2. Time-Consuming Process

Incrementality testing, especially geo-holdout tests, is not a quick endeavor. It requires careful planning, execution, and analysis.

Why it's Dumb for Time-Strapped Teams:

• Extended Testing Periods: Tests often need to run for several weeks to capture meaningful trends, especially for campaigns with longer conversion cycles.
• In-Depth Analysis Required: Once the test concludes, the analysis phase requires detailed data aggregation and interpretation to draw meaningful conclusions.

For smaller teams, this time investment could be better spent optimizing existing campaigns rather than on cumbersome tests.

3. Complexity and Interpretation Challenges

Properly conducting incrementality testing involves careful experimental design, segmentation, and statistical analysis.

Why it's Dumb for Non-Data-Driven Teams:

• Experimental Complexity: Setting up control and test groups correctly and ensuring proper randomization can be a complex task.
• Statistical Analysis Barriers: Not every team has access to the statistical expertise needed to interpret test results accurately. Misinterpretation can lead to misguided marketing strategies and wasted resources.

4. Incrementality Testing Is Meant for Larger, Data-Driven Companies

While incrementality testing may not make sense for smaller businesses, it remains invaluable for larger companies that have the necessary scale and resources.

Ideal Businesses for Incrementality Testing:

• $10M+ in Online Sales or $50M+ in Retail Sales: Companies of this size often have sufficient marketing budgets and sales volume to conduct statistically significant tests.
• Data-Driven Marketing Teams: Teams that can design, execute, and analyze tests effectively, often with the help of data scientists or analysts.
• Complex Marketing Portfolios: Businesses with multi-channel campaigns and significant media spend benefit most from understanding the true incremental value of their investments.

Conclusion: Is Incrementality Testing Right for Your Business?

Incrementality testing can offer deep insights into the causal impact of your marketing campaigns, but only if your business has the scale, resources, and data-driven mindset to execute it properly.

• Not Recommended For: Smaller businesses (<$10m in online sales or <$50m in retail sales), time-strapped teams, and non-data-driven marketing departments.
• Recommended For: Larger companies with complex marketing portfolios, sufficient sales volume, and data-driven teams.

If incrementality testing makes sense for your business, consider leveraging Stella as your go-to platform for streamlined, accurate, and actionable incrementality insights. Schedule a demo today and discover how Stella can help you optimize your marketing investments for maximum impact.