Google measures ad effectiveness with "Ghost Ads"

PROBLEM:  How to measure Ad Effectiveness (paper link)

To measure advertising effectiveness, we want to make a simple comparison, "Did showing the ads change users' behaviour, relative to not showing them?"

This is especially hard to do when algorithms choose ads to show to consumers who are most likely to buy.  This leads to Selection Bias:  one cannot tell whether people purchase because they were shown the ad (the "treatment effect," what we want to measure) or whether people who were going to purchase anyway were shown the ad ("selection bias," what we want to rule out)

To eliminate selection bias, Google uses Ghost Ads to make the experimental ads visible to the ad platform and experimenter but invisible to the control group, exactly as the algorithm would have chosen.  Using the algorithm to select the number and order of ads, but then "ghosting" the ads in the control group, eliminates selection bias.

Figure 4 illustrates how it work.

The treatment group (upper left) sees three Louboutin shoe ads and three ads from other advertisers. The ad platform "selects" the control group (lower right) as those consumers who would have seen an identical quantity and order of the other ads.  To create the control, the researcher simply "ghosts" the ads that would have appeared in the same order and location as the Louboutin shoe ads.

Google also can select users to match the demographics of the treatment and control groups. 

MORAL:  Carefully designed experiments can accurately measure causal relationships.  In this case, the treatment ads lifted website visits by 17% and purchases by 11%.

See simple exposition or the paper.(Ungated: https://courses.cit.cornell.edu/jl2545/adpapers/Randall%20Lewis.pdf)

HT:  Hal