Advertisement

Field Experiments

  • Veronica ValliEmail author
  • Florian Stahl
  • Elea McDonnell Feit
Living reference work entry
  • 294 Downloads

Abstract

Digitalization of value chains and company processes offers new opportunities to measure and control a firm’s activities and to make a business more efficient by better understanding markets, competitors, and consumers’ behaviors. Among other methodologies, field experiments conducted in online and offline environments are rapidly changing the way companies make business decisions. Simple A/B tests as well as more complex multivariate experiments are increasingly employed by managers to inform their marketing decisions.

This chapter explains why field experiments are a reliable way to reveal and to prove that a business action results in a desired outcome and provides guidelines on how to perform such experiments step by step covering issues such as randomization, sample selection, and data analysis. Various practical issues in the design of field experiments are covered with the main focus on causal inference and internal and external validity. We conclude the chapter with a practical case study as well as a brief literature review on recent published articles employing field experiments as a data collection method, providing the reader with a list of examples to consider and to refer to when conducting and designing a field experiment.

Keywords

Field experiment A/B test Randomized experiment Online experiment Digital experiment Business optimization Causal inference Experimental design Internal validity External validity 

References

  1. Aaker, D. A., Kumar, V., Day, G. S., & Leone, R. P. (2011). Marketing research. Hoboken: Wiley.Google Scholar
  2. Anderson, E. T., & Simester, D. (2013). Advertising in a competitive market: The role of product standards, customer learning, and switching costs. Journal of Marketing Research, 50(4), 489–504.CrossRefGoogle Scholar
  3. Andrews, M., Luo, X., Fang, Z., & Ghose, A. (2015). Mobile Ad effectiveness: Hyper-contextual targeting with crowdedness. Marketing Science, 35(2), 1–17.Google Scholar
  4. Ascarza, E., Iyengar, R., & Schleicher, M. (2016). The perils of proactive churn prevention using plan recommendations: Evidence from a field experiment. Journal of Marketing Research, 53(1), 46–60.CrossRefGoogle Scholar
  5. Bawa, K., & Shoemaker, R. (2004). The effects of free sample promotions on incremental brand sales. Marketing Science, 23(3), 345–363.CrossRefGoogle Scholar
  6. Blake, T., Nesko, C., & Tadelis, S. (2015). Consumer heterogeneity and paid search effectiveness: A large scale field experiment. Econometrica, 83(1), 155–174.CrossRefGoogle Scholar
  7. Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54(4), 297–312.CrossRefGoogle Scholar
  8. Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.Google Scholar
  9. Chen, H., Marmorstein, H., Tsiros, M., & Rao, A. R. (2012). When more is less: The impact of base value neglect on consumer preferences for bonus packs over price discounts. Journal of Marketing, 76(4), 64–77.CrossRefGoogle Scholar
  10. Crook, T., Brian, F., Ron, K., & Roger, L. (2009). Seven pitfalls to avoid when running controlled experiments on the web. In Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining.Google Scholar
  11. Dubé, J.-P., Luo, X., & Fang, Z. (2015). Self-signaling and pro-social behavior: A cause marketing experiment. Fox school of business research paper no. 15-079. Available at SSRN: http://ssrn.com/abstract=2635808 or  http://dx.doi.org/10.2139/ssrn.2635808
  12. Dunning, T. (2012). Natural experiments in the social sciences: A design-based approach. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  13. Eisenberg, B., & Quarto-von Tivadar, J. (2009). Always be testing: The complete guide to Google website optimizer. New York: Wiley.Google Scholar
  14. Fisher, R. A. (1925). Statistical methods for research workers. London: Oliver and Boyd.Google Scholar
  15. Gerber, A. S., & Green, D. P. (2008). Field experiments and natural experiments. In J. M. Box-Steffensmeier, H. E. Brady, & D. Collier (Eds.), The Oxford handbook of political methodology, Oxford handbooks online (pp. 357–381). Oxford: Oxford University Press.Google Scholar
  16. Gerber, A. S., & Green, D. P. (2012). Field experiments. Design, analysis, and interpretation. New York: Norton.Google Scholar
  17. Gneezy, A. (2017). Field experimentation in marketing research. Journal of Marketing Research, 46, 140–143.CrossRefGoogle Scholar
  18. Goos, P., & Jones, B. (2011). Optimal design of experiments: A case study approach. New York: Wiley.CrossRefGoogle Scholar
  19. Goldfarb, A., & Tucker, C. E. (2011a). Online display advertising: Targeting and obtrusiveness. Marketing Science, 30(3), 389–404.CrossRefGoogle Scholar
  20. Goldfarb, A., & Tucker, C. E. (2011b). Privacy regulation and online advertising. Management Science, 57(1), 57–71.CrossRefGoogle Scholar
  21. Goldfarb, A., & Tucker, C. E. (2011c). Advertising bans and the substitutability of online and offline advertising. Journal of Marketing Research, 48(2), 207–227.CrossRefGoogle Scholar
  22. Harrison, G. W., & List, J. A. (2004). Field experiments. Journal of Economic Literature, 42(4), 1009–1055.CrossRefGoogle Scholar
  23. Hoban, P. R., & Bucklin, R. E. (2015). Effects of internet display advertising in the purchase funnel: Model-based insights from a randomized field experiment. Journal of Marketing Research, 52(3), 375–393.CrossRefGoogle Scholar
  24. Homburg, C. (2015). Marketingmanagement. Strategie – Instrumente – Umsetzung – Unternehmensführung. Lehrbuch. Wiesbaden: Springer Gabler.Google Scholar
  25. Homburg, C., Kuester, S., & Krohmer, H. (2013). Marketing management. A contemporary perspective. London: McGraw-Hill Higher Education.Google Scholar
  26. Iacobucci, D., & Churchill, G. A. (2010). Marketing research. Methodological foundations. Mason: South-Western Cengage Learning.Google Scholar
  27. Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social and biomedical sciences: An introduction. New York: Cambridge University Press.CrossRefGoogle Scholar
  28. Kalyanam, K., McAteer, J., Marek, J., Hodges, J., & Lin, L. (2015). Cross channel effects of search engine advertising on brick and mortar retail sales: Meta analysis of large scale field experiments on Google.com. Working paper.Google Scholar
  29. Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181.CrossRefGoogle Scholar
  30. Koschate-Fischer, N., & Schandelmeier, S. (2014). A guideline for designing experimental studies in marketing research and a critical discussion of selected problem areas. Journal of Business Economics, 84, 793–826.CrossRefGoogle Scholar
  31. Landsberger, H. A. (1958). Hawthorne revisited. Ithaca: Cornell University.Google Scholar
  32. Lambrecht, A., & Tucker, C. E. (2013). When does retargeting work? Information specificity in online advertising. Journal of Marketing Research, 50(5), 561–576.CrossRefGoogle Scholar
  33. Ledolter, J., & Swersey, A. J. (2007). Testing 1-2-3. Experimental design with applications in marketing and service operations. Standford: Standford University Press.Google Scholar
  34. Levitt, S. D., & List, J. A. (2009). Field experiments in economics: The past, the present, and the future. European Economic Review, 53(1), 1–18.CrossRefGoogle Scholar
  35. Lewis, R. A., & Rao, J. M. (2015). The unfavourable economics of measuring the returns to advertising. The Quarterly Journal of Economics, 1941–1973.Google Scholar
  36. List, J. A. (2004). The nature and extent of discrimination in the marketplace: Evidence from the field. Quarterly Journal of Economics, 119(1), 48–89.CrossRefGoogle Scholar
  37. List, J. A. (2011). Why economists should conduct field experiments and 14 tips for pulling one off. Journal of Economic Perspectives, 25(3), 3–16.CrossRefGoogle Scholar
  38. McFarland, C. (2012). Experiment! Website conversion rate optimization with A/B and multivariate. Berkeley: New Riders.Google Scholar
  39. Montaguti, E., Neslin, S. A., & Valentini, S. (2016). Can marketing campaigns induce multichannel buying and more profitable customers? A field experiment. Marketing Science, 35(2), 201–217.CrossRefGoogle Scholar
  40. Neyman, Jerzy. (1923[1990]). On the application of probability theory to agricultural experiments: Essay on principles. Section 9. Statistical Science, 5 (4), 465–472. Translated by Dabrowska, Dorota M. and Terence P. Speed.Google Scholar
  41. Rush, K. (2012a). Meet the Obama campaign’s $250 million fundraising platform. Blog post 27 Nov 2012.Google Scholar
  42. Rush, K. (2012b). Optimization at the Obama campaign: a/b testing. Blog post 12 Dec 2012.Google Scholar
  43. Sahni, N., Dan, Z., & Pradeep, C. (2015). Do targeted discount offers serve as advertising? Evidence from 70 field experiments. Stanford University Graduate School of Business research paper no. 15-4. Available at SSRN: http://ssrn.com/abstract=2530290 or  http://dx.doi.org/10.2139/ssrn.2530290
  44. Sándor, Z., & Wedel, M. (2001). Designing conjoint choice experiments using managers’ prior beliefs. Journal of Marketing Research, 38(4), 430–444.CrossRefGoogle Scholar
  45. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Belmont: Wadsworth Cengage Learning.Google Scholar
  46. Simonov, A., Nosko, C., & Rao J. M. (2015). Competition and crowd-out for brand keywords in sponsored search. Available at SSRN: http://ssrn.com/abstract=2668265 or  http://dx.doi.org/10.2139/ssrn.2668265
  47. Siroker, D., & Pete K. (2013). A/B testing. Wiley.Google Scholar
  48. Teele, D. L. (2014). Field experiments and their critics: Essays on the uses and abuses of experimentation in the social sciences. New Haven & London: Yale University Press.Google Scholar
  49. Vaver, J., & Koehler, J. (2011). Measuring ad effectiveness using geo experiments. Google Research working paper.Google Scholar
  50. Yang, S., & Ghose, A. (2010). Analyzing the relationship between organic and sponsored search advertising: Positive, negative, or zero interdependence? Marketing Science, 29(4), 602–623.CrossRefGoogle Scholar
  51. Zantedeschi, D., McDonnell Feit, E., & Bradlow, E. T. (2016). Modeling multi-channel advertising response with consumer-level data. Management Science, Articles in Advance. http://pubsonline.informs.org/doi/abs/10.1287/mnsc.2016.2451

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Veronica Valli
    • 1
    Email author
  • Florian Stahl
    • 1
  • Elea McDonnell Feit
    • 2
  1. 1.University of MannheimMannheimGermany
  2. 2.LeBow College of BusinessDrexel UniversityPhiladelphiaUSA

Personalised recommendations