How to Conduct an Incrementality Study for Meta Ads: A Step-by-Step Guide

Incrementality studies are a cornerstone of modern marketing measurement. They help you understand the true impact of your ad spend, isolating results from external factors. For Meta ads, running an incrementality study can provide clear insights into the value your campaigns bring. This guide walks you through every step, focusing on a channel-level analysis using an inverse holdout methodology. We’ll also highlight how Stella, an advanced marketing measurement platform, can simplify and enhance this process.

What Is an Inverse Holdout Study?

An inverse holdout study involves turning off ads in specific regions that were previously live. Unlike a geo holdout (where ads are activated in test regions), this methodology tests the absence of advertising to determine its incremental impact. The results can inform decisions about budget allocation and channel performance.

Step 1: Selecting Holdout Regions

The success of your study hinges on carefully selecting holdout regions. Here’s how:

Map Your Performance Data to Geographic Areas

Use geographic frameworks like commuting zones, Designated Market Areas (DMAs), or Generalized Marketing Areas (GMAs). These structures help account for local economic factors and commuting distances, minimizing the risk of spillover effects, where advertising in one region influences a neighboring area.

• Example:
  • In the U.S., DMAs divide the country into distinct regions based on television market areas.
  • In the UK, GMAs group areas into 82 unique marketing zones.

Analyze Historical Data

• Gather performance data from past campaigns, focusing on metrics like impressions, clicks, conversions, and revenue by region.
• Identify regions with similar characteristics such as:
  • Population size
  • Purchasing behavior
  • Demographics (e.g., age, income levels, interests)

This ensures the regions chosen are representative of your market as a whole.

Create Matched Pairs

• Pair similar regions based on their historical performance data.
  • For instance, pair a region with 1 million people and a 10% conversion rate with another region that has a similar population size and conversion rate.
• Assign one region in each pair to the test group (ads remain active) and the other to the control group (ads are turned off).
• Ensure parity in population size, historical performance, and demographic factors to reduce bias.

Exclude Outliers

• Large metropolitan areas (e.g., New York, Los Angeles, London) or highly unique markets might skew results. Exclude these regions to ensure consistency.

How Stella Helps

Stella’s platform automates the process of selecting holdout locations. Using advanced algorithms, Stella ensures test and control groups are statistically balanced. This eliminates the guesswork and manual effort, providing you with confidence in your study setup.

Step 2: Setting Up the Holdout Experiment

Turning Off Ads in Holdout Regions

‍

1. Access Meta Ads Manager:
   
   
   • Log in to your Meta Ads account.
   • Navigate to Advertising Settings.
   • Select Account Controls, then Audience Controls.
     ‍
2. Exclude Holdout Locations:
   
   
   • Click on My Business Can Only Advertise in Specific Locations.
   • Add your control group regions to the exclusion list. This ensures that ads will not be served in these areas across all campaigns.
     ‍
3. Verify Settings:
   
   
   • Double-check that the exclusions are applied correctly. Run a test by simulating ad delivery to ensure no ads are shown in the designated holdout regions.

Determine the Experiment Duration

• Incrementality studies typically run for 3-4 weeks. However, the exact duration should align with:
  
  
  • Your typical sales cycle to capture delayed conversions.
  • Seasonal trends to avoid skewed results (e.g., avoid Black Friday unless you’re studying that period specifically).
    ‍
• Stella Advantage: Stella calculates the optimal duration and budget required to reach statistical significance, taking the guesswork out of planning.
  
  

Step 3: Running the Experiment

1. Launch the Campaign:
   
   
   • Activate your campaigns with the holdout exclusions in place.
   • Confirm that ads are delivering only to test regions.
     ‍
2. Monitor Performance:
   
   
   • Regularly review your campaign settings and performance metrics to ensure ads are not served in holdout regions.
   • Track any external factors (e.g., competitor activity, market changes) that could influence results.
     ‍
3. Maintain Consistency:
   
   
   • Avoid making significant changes to your ad creatives, budgets, or targeting during the experiment. These changes can introduce variability and compromise the integrity of your results.

Step 4: Collecting and Analyzing Data

Collect Data

To analyze the experiment, you need data from both Meta Ads and your sales platform (e.g., Shopify):

1. Meta Ad Spend Data:
   
   
   • Go to Ads Reporting in Meta Ads Manager.
   • Click the Breakdown menu on the right-hand side.
   • Select Delivery > Region to view performance by state or region.
   • Add a daily breakdown by selecting Time > Day.
   • Include key metrics like Spend, Impressions, and Conversions.
   • Export this data for further analysis.

‍

1. Sales Data (from Shopify):
   
   
   • Log in to your Shopify admin panel.
   • Navigate to Analytics > Reports.
   • Select the Gross sales over time Report.
   • Filter by date and region, ensuring columns for Gross Sales and Orders are included.
   • Export this data as a CSV file.

‍

Combine Data

• Use a spreadsheet tool (e.g., Excel or Google Sheets) to merge Meta’s ad spend data with Shopify’s sales data.
• Ensure the data is organized by date and region, with consistent formatting for key fields.

Calculate Key Metrics

1. Lift: Lift = ((Test Group Revenue - Control Group Revenue) / Control Group Revenue) * 100
   
   
2. Incremental Revenue: Incremental Revenue = Test Group Revenue - Control Group Revenue
   
   
3. iROAS (Incremental Return on Ad Spend): iROAS = Incremental Revenue / Ad Spend in Test Group
   
   
4. iCPO (Incremental Cost Per Order): iCPO = Ad Spend in Test Group / Incremental Orders
   
   

Statistical Significance

• Use tools like Excel’s Data Analysis ToolPak or online calculators to perform a t-test and confirm whether your results are statistically significant.
• Stella Advantage: Stella’s built-in statistical analysis tools automatically check for significance, ensuring accurate and actionable insights.

Step 5: Interpreting Results and Taking Action

Segment Analysis

• Break down results by product categories, customer demographics, or regions to uncover specific insights.

Insights and Next Steps

Since we ran an inverse holdout study, we are actually expecting a negative incremental revenue lift. This is a good outcome in this case, as it confirms that the ads were driving incremental sales when live.

• Negative Lift: Indicates that Meta ads are driving incremental revenue when active, as expected in an inverse holdout study. This confirms the ads’ effectiveness in generating additional sales when live, reducing concerns about potential cannibalization.
• No Significant Lift: Suggests that your ads may not be effective. Reevaluate your targeting, creatives, or messaging.
• Positive Lift: Indicates a concerning outcome in an inverse holdout study. This suggests that turning ads off led to an increase in revenue, indicating potential cannibalization by the ads when they were live. Further analysis is needed to understand and address this issue.

Stella Simplifies Post-Test Analysis

Stella’s AI-generated insights not only summarize the results but also recommend actionable next steps, tailored to your unique findings.

See for yourself and trial our virtual demo below:

Why Use Stella for Incrementality Testing?

While it’s possible to run an incrementality study manually, Stella’s platform significantly enhances the process by:

• Efficient Location Selection: Stella’s algorithms create balanced test and control groups in minutes.
• Accurate Budgeting: Stella determines the spend and time needed to achieve statistical significance.
• Automated Reporting: Generate professional, white-labeled reports that are ready for leadership presentations.
• AI Guidance: Stella’s AI acts as your personal data scientist, answering questions, interpreting results, and providing recommendations.

Conclusion

Conducting an incrementality study for Meta ads using an inverse holdout approach provides invaluable insights into ad effectiveness. While the manual process is feasible, Stella’s advanced platform simplifies every step, making it faster, easier, and more accurate. By integrating Stella into your marketing measurement strategy, you can unlock the true potential of your ad spend and drive sustainable growth.

Ready to simplify your incrementality testing? Discover how Stella can transform your marketing insights today.

‍