Quantitative Marketing and Economics

, Volume 12, Issue 3, pp 235–266 | Cite as

Online ads and offline sales: measuring the effect of retail advertising via a controlled experiment on Yahoo!

  • Randall A. LewisEmail author
  • David H. Reiley


A randomized experiment with 1.6 million customers measures positive causal effects of online advertising for a major retailer. The advertising profitably increases purchases by 5%. 93% of the increase occurs in brick-and-mortar stores; 78% of the increase derives from consumers who never click the ads. Our large sample reaches the statistical frontier for measuring economically relevant effects. We improve econometric efficiency by supplementing our experimental variation with non-experimental variation caused by consumer browsing behavior. Our experiment provides a specification check for observational difference-in-differences and cross-sectional estimators; the latter exhibits a large negative bias three times the estimated experimental effect.


Online advertising Display advertising Advertising effectiveness Field experiment Difference in differences 

JEL Classification

Codes: C93 - Field Experiments M37 - Advertising D12 - Consumer Economics: Empirical Analysis 



We thank Meredith Gordon, Sergiy Matusevych, and especially Taylor Schreiner for their work on the experiment and data. Yahoo! Inc. provided financial and data assistance and guaranteed academic independence prior to our analysis so that the results could be published no matter how they turned out. We acknowledge the helpful comments of Manuela Angelucci, David Broockman, JP Dubé, Liran Einav, Glenn Ellison, Matt Gentzkow, Jerry Hausman, Kei Hirano, Garrett Johnson, Larry Katz, John List, Preston McAfee, Sendhil Mullainathan, Justin Rao, Paul Ruud, Michael Schwarz, Pai-Ling Yin, and many others, including attendees at many conferences and seminars.


  1. Aaker, D. A., & Carman, J. M. (1982). Are you overadvertising? a review of advertising-sales studies. Journal of Advertising Research, 22(4), 57–70.Google Scholar
  2. Abraham, M. M. (2008). The off-line impact of online ads. Harvard Business Review, 86(4), 28.Google Scholar
  3. Abraham, M. M., & Lodish, L. (1990). Getting the most out of advertising and promotion. Harvard Business Review, 68(3), 50.PubMedGoogle Scholar
  4. Ackerberg, D. (2003). Advertising, learning, and consumer choice in experience good markets: an empirical examination*. International Economic Review, 44(3), 1007–1040.CrossRefGoogle Scholar
  5. Ackerberg, D. A. (2001). Empirically distinguishing informative and prestige effects of advertising. RAND Journal of Economics, 316–333.Google Scholar
  6. Ackoff, R. L., & Emshoff, J. R. (1975). Advertising research at anheuser-busch, inc. (1963-68). Sloan Management Review (pre-1986), 16(2), 1–1. Scholar
  7. Allaire, Y. (1975). A multivariate puzzle: A comment on advertising research at anheuser-busch, inc.(1963-68). Sloan Management Review,(Spring), 91, 94.Google Scholar
  8. Angrist, J. D., Imbens, G. W., Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434), 444–455.zbMATHCrossRefGoogle Scholar
  9. Bagwell, K. (2008). The economic analysis of advertising. Handbook of industrial organization, 3, 1701–1844.CrossRefGoogle Scholar
  10. Berndt, E. R. (1991). The Practice of Econometrics: Classic and Contemporary. MA: Addison-Wesley Reading.Google Scholar
  11. Bertrand, M., Karlan, D., Mullainathan, S., Shafir, E., Zinman, J. (2010). What’s advertising content worth? evidence from a consumer credit marketing field experiment. The Quarterly Journal of Economics, 125(1), 263–306.CrossRefGoogle Scholar
  12. Blake, T., Nosko, C., Tadelis, S. (2013). Consumer heterogeneity and paid search effectiveness: A large scale field experiment. NBER Working Paper, 1–26.Google Scholar
  13. DellaVigna, S., & Gentzkow, M. (2010). Persuasion: Empirical evidence. Annual Review of Economics, 2(1), 643–669.CrossRefGoogle Scholar
  14. Eastlack, J., & Rao, A. (1989). Advertising experiments at the campbell soup company. Marketing Science, 57–71.Google Scholar
  15. Ghose, A., & Yang, S. (2009). An empirical analysis of search engine advertising: Sponsored search in electronic markets. Management Science, 55(10), 1605–1622.CrossRefGoogle Scholar
  16. Holahan C., & Hof R.D. (2007). So many ads, so few clicks, Bloomberg Businessweek.
  17. Hu, Y., Lodish, L. M., Krieger, A. M. (2007). An analysis of real world tv advertising tests: A 15-year update. Journal of Advertising Research, 47(3), 341.CrossRefGoogle Scholar
  18. Levitt, S. D., & List, J. A. (2009). Field experiments in economics: the past, the present, and the future. European Economic Review, 53(1), 1–18.CrossRefGoogle Scholar
  19. Lewis RA, & Rao JM (2013). On the near impossibility of measuring the returns to advertising. Working paper.Google Scholar
  20. Lewis, R. A. & Reiley D.H. (2014). Advertising effectively influences older users, How field experiments can improve measurement and targeting. Review of Industrial Organization forthcoming.
  21. Lewis, R. A., Rao, J. M., Reiley, D. H. (2011). Here, there, and everywhere: correlated online behaviors can lead to overestimates of the effects of advertising. Proceedings of the 20th international conference on World wide web. pp 157-166.Google Scholar
  22. Lewis, R. A., & Reiley, D. H. (2012). Ad attributes and attribution: Large-scale field experiments measure online customer acquisition. Working Paper.Google Scholar
  23. Lodish, L. M., Abraham, M., Kalmenson, S., Livelsberger, J., Lubetkin, B., Richardson, B., Stevens, M. E. (1995a). How tv advertising works: A meta-analysis of 389 real world split cable tv advertising experiments. Journal of Marketing Research, 125–139.Google Scholar
  24. Lodish, L. M., Abraham, M. M., Livelsberger, J., Lubetkin, B., Richardson, B., Stevens, M. E. (1995b). A summary of fifty-five in-market experimental estimates of the long-term effect of tv advertising. Marketing Science, 14(3 supplement), G133—G140.Google Scholar
  25. Meland, M. (1999). Banner click-throughs continue to fall. Forbes.
  26. Shein, E. (2012). Banner ads: Past, present, and... future? CMOcom.
  27. Simester, D., Hu, J., Brynjolfsson, E., Anderson, E. (2009). Dynamics of retail advertising: Evidence from a field experiment. Economic Inquiry, 47(3), 482–499.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Google, Inc.,Mountain ViewUSA

Personalised recommendations