How to Conduct an Incrementality Study for Google Ads: A Step-by-Step Guide

Incrementality studies are essential for understanding the true value of your ad spend by isolating the incremental impact of your campaigns. For Google Ads, a channel-level analysis using an inverse holdout methodology can help you uncover actionable insights about the effectiveness of your campaigns. This guide will walk you through the process step-by-step, while also highlighting how you can simplify the process and gain better insights.

What Is an Inverse Holdout Study in Google Ads?

An inverse holdout study involves pausing ads in selected regions where they are usually active to measure the incremental impact of their absence. By comparing the performance of these holdout regions to regions where ads remain active, you can identify the true value your Google Ads campaigns generate.

Step 1: Selecting Holdout Regions for Google Ads

Carefully choosing the right regions for your holdout study is critical to ensuring reliable results.

1. Map Performance Data to Geographic Areas
   Google Ads uses flexible geographic targeting options, such as countries, states, cities, or even zip codes. Consider using geographic frameworks such as Designated Market Areas (DMAs) or commuting zones to account for economic factors and reduce spillover effects.
   Example: In the U.S., DMAs divide the country into distinct regions based on television markets. These regions work well for holdout experiments because of their economic and cultural consistency.
2. Analyze Historical Data
   Collect past performance metrics like impressions, clicks, conversions, and revenue by region. Select regions with similar:
   
   • Population size
   • Purchasing behavior
   • Demographics (e.g., age, income, interests)
3. Create Matched Pairs
   Pair regions with comparable historical performance to form balanced test (ads active) and control (ads paused) groups. Match on metrics such as conversion rates, revenue per visitor, and overall audience size.
4. Exclude Outliers
   Highly unique or populous regions, such as major cities (e.g., New York or London), can distort results. Focus on regions that better represent your typical audience.

How Stella Helps
Stella automates the selection of test and control regions, ensuring statistical balance and eliminating manual effort. This saves time and improves the reliability of your study.

Step 2: Setting Up the Holdout Experiment in Google Ads

For Google Ads, setting up location exclusions requires precision. Here’s how to configure your campaigns for the experiment:

‍

1. Access Location Settings
   
   • Log in to your Google Ads account.
   • Navigate to the "Campaigns" tab.
   • Select the campaign(s) you want to include in the experiment.
2. Exclude Control Regions
   
   • Go to Settings > Locations.
   • Use the Advanced Search option to exclude specific regions.
   • For multiple exclusions, use the bulk upload feature to exclude up to 1,000 locations.
3. Verify Campaign Settings
   
   • Double-check that ads are only running in your test regions. Simulate ad delivery to ensure no spillover into control regions.
4. Determine Experiment Duration
   Most incrementality studies run for 3-4 weeks, allowing sufficient data collection while aligning with your sales cycle and avoiding seasonal biases.

Stella Advantage
Stella determines the optimal study duration and ensures proper campaign setup for statistically valid results.

Step 3: Running the Experiment in Google Ads

1. Launch Your Campaigns
   Ensure that ads are active in your test regions and completely paused in control regions. Monitor performance regularly to confirm the setup is working as planned.
2. Maintain Consistency
   Avoid major changes to budgets, creatives, or targeting during the experiment to preserve data integrity.
3. Monitor External Factors
   Track market conditions, competitor activity, or other external influences that might affect performance during the study period.

Step 4: Collecting and Analyzing Data

‍

1. Collect Ad Spend Data
   
   • Access performance metrics in Google Ads:
     
     • Navigate to Reports > Create report.
     • Add a Daily Breakdown to view cost by region.
     • Export data as a CSV file or Google Sheet file.
2. Collect Sales Data
   
   • Use your sales platform (e.g., Shopify, BigCommerce) to gather revenue and order data by region and date.

1. Combine Data
   
   • Merge Google Ads spend data with sales data in Excel or Google Sheets.
   • Ensure columns align by date and region for accurate analysis.
2. Calculate Key Metrics
   
   • Lift: (Test Revenue−Control Revenue)/Control Revenue×100(\text{Test Revenue} - \text{Control Revenue}) / \text{Control Revenue} \times 100(Test Revenue−Control Revenue)/Control Revenue×100
   • Incremental Revenue: Test Revenue−Control Revenue\text{Test Revenue} - \text{Control Revenue}Test Revenue−Control Revenue
   • iROAS: Incremental Revenue/Test Ad Spend\text{Incremental Revenue} / \text{Test Ad Spend}Incremental Revenue/Test Ad Spend
   • iCPO: Test Ad Spend/Incremental Orders\text{Test Ad Spend} / \text{Incremental Orders}Test Ad Spend/Incremental Orders
3. Confirm Statistical Significance
   Use tools like Excel’s Data Analysis ToolPak or Stella’s built-in significance testing.

Stella Advantage
Stella automates data collection and analysis, delivering clear, actionable insights and statistical validation.

Step 5: Interpreting Results and Taking Action

Since we ran an inverse holdout study, we are actually expecting a negative incremental revenue lift. This is a good outcome in this case, as it confirms that the ads were driving incremental sales when live.

‍

1. Negative Lift: Indicates that Meta ads are driving incremental revenue when active, as expected in an inverse holdout study. This confirms the ads’ effectiveness in generating additional sales when live, reducing concerns about potential cannibalization.
2. No Significant Lift: Suggests that your ads may not be effective. Reevaluate your targeting, creatives, or messaging.
3. Positive Lift: Indicates a concerning outcome in an inverse holdout study. This suggests that turning ads off led to an increase in revenue, indicating potential cannibalization by the ads when they were live. Further analysis is needed to understand and address this issue.
4. Next Steps
   
   • Refine your campaign strategy based on insights.
   • Consider reallocating budgets to higher-performing regions or adjusting creative direction.

Stella Simplifies Post-Test Analysis
Stella provides AI-generated recommendations tailored to your data, helping you optimize campaigns with confidence.

Why Use Stella for Google Ads Incrementality Testing?

While it’s possible to run a manual incrementality study, Stella’s platform offers significant advantages:

• Automated Region Selection: Saves time and reduces errors.
• Optimized Budgets: Recommends the ideal spend and duration for statistically valid results.
• AI-Powered Insights: Provides actionable recommendations based on real-time analysis.
• Dynamic Reporting: Generate ready-to-present reports with clear metrics and insights.

See for yourself and take the interactive virtual demo below:

Conclusion

Running an incrementality study for Google Ads provides powerful insights into your channel's true impact. While the manual process is feasible, Stella makes it faster, simpler, and more accurate, empowering marketers to make data-driven decisions.

Ready to take the guesswork out of incrementality testing? Try Stella and unlock the full potential of your Google Ads campaigns today.

‍