When Should a Company Start Incrementality Testing? Understanding Spend, Sales, and Statistical Significance

Incrementality testing is one of the most powerful tools available to marketers today, helping brands determine whether their marketing efforts are genuinely driving new sales or simply taking credit for sales that would have happened anyway. However, the question of when to start incrementality testing can be challenging, especially for mid-market e-commerce brands. At Stella, we believe that while ad spend is important, the number of daily sales is a more crucial factor in determining when to start testing. This focus on sales volume ensures that your incrementality tests can achieve statistical significance, making the insights from these tests more reliable and actionable.

In this guide, we’ll explore the optimal conditions for starting incrementality testing, explain why statistical significance is vital to your testing efforts, and highlight how Stella’s unique approach makes it easier for mid-market e-commerce brands to get started.

Why Statistical Significance is Crucial in Incrementality Testing

Before diving into when you should start incrementality testing, it’s essential to understand the role of statistical significance in the process. Statistical significance helps you determine whether the differences between your test and control groups are due to the marketing intervention you’re testing or if they are simply the result of random chance.

In simple terms, statistical significance ensures that your incrementality test results are valid and that the effects you observe are real and not a fluke. Here’s how it works:

P-value: This is a number that indicates the probability of observing the results you did if there were no true effect (i.e., if the null hypothesis were true). A p-value of 0.05 or less is typically considered statistically significant, meaning there is less than a 5% chance that the observed effect is due to random variation.
Sample Size: Achieving statistical significance requires a large enough sample size. This is why daily sales volume is a key factor when deciding whether you’re ready for incrementality testing. More daily sales mean you can reach statistical significance more quickly.
Effect Size: While statistical significance tells you whether an effect is likely real, effect size tells you how large that effect is. Both are important. For example, a test could show statistical significance but have such a small effect size that the change wouldn’t justify a new strategy.

The Key Indicator: Daily Sales Volume Over Ad Spend

Many marketers mistakenly focus solely on ad spend when deciding to start incrementality testing. However, while ad spend can provide an indication of when you're ready, it is daily sales volume that determines whether your test can reach statistical significance in a timely manner.

Here’s why daily sales matter more:

Sales data allows for enough conversions to be analyzed, which helps achieve statistical significance.
The more daily sales you have, the less time it takes to gather enough data to confidently say that the observed effect is real.

At Stella, we recommend starting incrementality testing if your brand meets one of the following conditions:

You’re spending $85,000 per month or more on a specific channel, such as Google or Meta.
You’re generating 100+ daily orders in a single country or region.

These thresholds ensure you have enough data to run meaningful tests and make informed decisions based on statistically significant results.

The Importance of Incrementality Testing for Mid-Market E-Commerce Brands

Incrementality testing allows brands to see which marketing efforts are truly driving new revenue and which are simply capturing existing demand. Starting this process early in your growth journey can help you:

Optimize budget allocation: By identifying which channels and tactics are truly incremental, you can eliminate wasteful spending and reallocate your budget to the most effective areas.
Refine campaigns: Over time, incrementality testing enables you to test not only channels but also tactic-level adjustments, such as whether "Max Conversion Value" bid strategy on Google is more effective than "Target ROAS" bid strategy, or whether a 7-day click attribution on Meta performs better than a 7-day click + 1-day view attribution setting.
Make data-driven decisions: Incrementality testing provides the concrete data needed to move beyond guesswork and base your marketing strategies on real, measurable results.

Achieving Statistical Significance in Incrementality Testing

For any test to be valid, you need to achieve statistical significance. Here’s how that works in incrementality testing:

Test Group and Control Group: In an incrementality test, you divide your audience into two groups—one that is exposed to the marketing campaign (test group) and another that is not (control group). Statistical significance is achieved when the observed difference between these groups is unlikely to be the result of chance.
Minimum Sales Volume: A critical factor in achieving statistical significance is the number of sales in each group. If you’re not generating enough daily orders, the results of your test may not reach significance, rendering them less reliable.
Proper Experiment Design: At Stella, we help mid-market e-commerce brands achieve statistical significance in as little as 15 days. Our platform analyzes your last 90 days of sales data to identify the regions where holding out a control group will provide the most reliable results, with the lowest investment and in the shortest amount of time.
Effect Size and Confidence Level: Once your test is complete, we analyze whether the results meet the p-value threshold (typically 0.05 or 5%) and whether the effect size is meaningful enough to warrant action.

When Should You Start Incrementality Testing?

So when should your brand begin testing for incrementality? In addition to generating enough daily sales for statistical significance, you should also consider the following thresholds:

$85,000 per month on a single channel (e.g., Google Ads or Meta Ads) ensures you have the spend necessary to run a robust test.
100+ daily orders in a single country provides enough conversion data for the test to yield meaningful, statistically significant results.

If your brand meets either of these criteria, you’re ready to start incrementality testing and begin making smarter, data-driven marketing decisions.

How Stella Simplifies Incrementality Testing for Mid-Market Brands

At Stella, we’ve built a platform designed specifically for mid-market e-commerce brands. Unlike other incrementality tools that cater primarily to enterprise clients, we focus on helping companies that are looking for powerful insights without the enterprise price tag.

Here’s how Stella makes incrementality testing easy:

Custom Data Models: Our platform uses custom data models that are optimized for mid-market brands, allowing you to conduct 15-day experiments while ensuring statistical significance. We analyze your last 90 days of sales data to help you identify the best regions for testing, so you can get actionable results without a long, drawn-out process.
Tactic-Level Testing: With Stella, you’re not limited to channel-level testing. You can also test the incrementality of specific tactics or campaign settings, such as bid strategies or attribution settings, giving you deeper insights into what’s truly driving growth.
Flexible Terms: Unlike other incrementality testing tools that lock you into 6-12 month contracts, Stella offers 30-day and 3-month terms, allowing you to experiment without the pressure of a long-term commitment.

Conclusion: Start Incrementality Testing to Drive Smarter Growth

Incrementality testing isn’t just for enterprise brands with massive budgets. Mid-market e-commerce companies can—and should—start testing as soon as they reach $85,000 per month in ad spend or 100 daily sales in a single country. With the right sales volume and ad spend, you’ll have the data necessary to reach statistical significance, ensuring that your test results are accurate, reliable, and actionable.

By starting incrementality testing early, your brand can build a culture of data-driven decision making, optimize marketing efforts, and ensure that every dollar spent is driving real, measurable growth. With Stella, you have the tools, flexibility, and expertise to make incrementality testing part of your strategy and unlock your company’s full growth potential.

‍