Randomized Experiments to Detect and Estimate Social Influence in Networks

  • Sean J. TaylorEmail author
  • Dean Eckles
Part of the Computational Social Sciences book series (CSS)


Estimation of social influence in networks can be substantially biased in observational studies due to homophily and network correlation in exposure to exogenous events. Randomized experiments, in which the researcher intervenes in the social system and uses randomization to determine how to do so, provide a methodology for credibly estimating of causal effects of social behaviors. In addition to addressing questions central to the social sciences, these estimates can form the basis for effective marketing and public policy. In this review, we discuss the design space of experiments to measure social influence through combinations of interventions and randomizations. We define an experiment as combination of (1) a target population of individuals connected by an observed interaction network, (2) a set of treatments whereby the researcher will intervene in the social system, (3) a randomization strategy which maps individuals or edges to treatments, and (4) a measurement of an outcome of interest after treatment has been assigned. We review experiments that demonstrate potential experimental designs and we evaluate their advantages and tradeoffs for answering different types of causal questions about social influence. We show how randomization also provides a basis for statistical inference when analyzing these experiments.



We would like to thank Lada Adamic, Norberto Andrade, Eytan Bakshy, and George Berry for helpful discussions in preparing this review. Disclosures: D.E. was previously an employee and contractor of Facebook; D.E. has a significant financial interest in Facebook.


  1. 1.
    Anagnostopoulos A, Kumar R, Mahdian M (2008) Influence and correlation in social networks. In: Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, pp 7–15Google Scholar
  2. 2.
    Angelucci M, De Giorgi G (2009) Indirect effects of an aid program: how do cash transfers affect ineligibles’ consumption? Am Econ Rev 99(1):486–508CrossRefGoogle Scholar
  3. 3.
    Angrist JD (2014) The perils of peer effects. Labour Econ 30:98–108CrossRefGoogle Scholar
  4. 4.
    Aplin LM, Farine DR, Morand-Ferron J, Cockburn A, Thornton A, Sheldon BC (2015) Experimentally induced innovations lead to persistent culture via conformity in wild birds. Nature 518(7540):538–541ADSCrossRefGoogle Scholar
  5. 5.
    Aral S (2016) Networked experiments. In: The oxford handbook of the economics of networks. Oxford University Press, OxfordGoogle Scholar
  6. 6.
    Aral S, Walker D (2011) Creating social contagion through viral product design: a randomized trial of peer influence in networks. Manag Sci 57(9):1623–1639CrossRefGoogle Scholar
  7. 7.
    Aral S, Walker D (2012) Identifying influential and susceptible members of social networks. Science 337(6092):337–341ADSMathSciNetCrossRefGoogle Scholar
  8. 8.
    Aral S, Walker D (2014) Tie strength, embeddedness, and social influence: a large-scale networked experiment. Manag Sci 60(6):1352–1370CrossRefGoogle Scholar
  9. 9.
    Aral S, Muchnik L, Sundararajan A (2009) Distinguishing influence-based contagion from homophily-driven diffusion in dynamic networks. Proc Natl Acad Sci 106(51):21544–21549ADSCrossRefGoogle Scholar
  10. 10.
    Aral S, Muchnik L, Sundararajan A (2013) Engineering social contagions: optimal network seeding in the presence of homophily. Netw Sci 1(02):125–153CrossRefGoogle Scholar
  11. 11.
    Aronow PM (2012) A general method for detecting interference between units in randomized experiments. Sociol Methods Res 41(1):3–16ADSMathSciNetCrossRefGoogle Scholar
  12. 12.
    Aronow PM, Samii C (2017) Estimating average causal effects under general interference, with application to a social network experiment. Ann Appl Stat 11(4):1912–1947MathSciNetCrossRefGoogle Scholar
  13. 13.
    Asch SE (1955) Opinions and social pressure. Readings about the social animal. Sci Rep 193:17–26ADSCrossRefGoogle Scholar
  14. 14.
    Athey S, Imbens GW (2015) Machine learning methods for estimating heterogeneous causal effects. Stat 1050(5)Google Scholar
  15. 15.
    Athey S, Imbens GW (2017) The econometrics of randomized experiments. Handb Econ Field Exp 1:73–140Google Scholar
  16. 16.
    Athey S, Eckles D, Imbens GW (2017) Exact p-values for network interference. J Am Stat Assoc
  17. 17.
    Backstrom L, Leskovec J (2011) Supervised random walks: predicting and recommending links in social networks. In: Proceedings of the fourth ACM international conference on Web search and data mining. ACM, New York, pp 635–644Google Scholar
  18. 18.
    Baird S, Bohren JA, McIntosh C, Ozler B (2016) Optimal design of experiments in the presence of interference. working paper
  19. 19.
    Bakshy E, Eckles D (2013) Uncertainty in online experiments with dependent data: an evaluation of bootstrap methods. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, New York, pp 1303–1311Google Scholar
  20. 20.
    Bakshy E, Eckles D, Yan R, Rosenn I (2012) Social influence in social advertising: evidence from field experiments. In: Proceedings of the ACM conference on electronic commerce. ACM, New YorkGoogle Scholar
  21. 21.
    Bakshy E, Rosenn I, Marlow C, Adamic L (2012) The role of social networks in information diffusion. In: Proceedings of the 21st international conference on world wide web. ACM, WWW ’12, pp 519–528Google Scholar
  22. 22.
    Bakshy E, Eckles D, Bernstein MS (2014) Designing and deploying online field experiments. In: Proceedings of the 23rd international conference on world wide web, pp 283–292Google Scholar
  23. 23.
    Bakshy E, Messing S, Adamic LA (2015) Exposure to ideologically diverse news and opinion on facebook. Science 348(6239):1130–1132ADSMathSciNetCrossRefGoogle Scholar
  24. 24.
    Bapna R, Umyarov A (2015) Do your online friends make you pay? A randomized field experiment on peer influence in online social networks. Manag Sci 61(8):1902–1920CrossRefGoogle Scholar
  25. 25.
    Bass FM (1969) A new product growth for model consumer durables. Manag Sci 15(5):215–227. CrossRefGoogle Scholar
  26. 26.
    Basse G, Feller A (2018) Analyzing multilevel experiments in the presence of peer effects. J Am Stat Assoc
  27. 27.
    Basse GW, Airoldi EM (2015) Optimal model-assisted design of experiments for network correlated outcomes suggests new notions of network balance. arXiv preprint arXiv:150700803Google Scholar
  28. 28.
    Beaman L, BenYishay A, Magruder J, Mobarak AM (2015) Can network theory-based targeting increase technology adoption? Working paperGoogle Scholar
  29. 29.
    Beasley A, Mason W (2015) Emotional states vs. emotional words in social media. In: Proceedings of the ACM web science conference. ACM, New York, p 31Google Scholar
  30. 30.
    Berry G, Taylor SJ (2017) Discussion quality diffuses in the digital public square. In: Proceedings of the 26th international conference on world wide web, international world wide web conferences steering committee, pp 1371–1380Google Scholar
  31. 31.
    Bond RM, Fariss CJ, Jones JJ, Kramer AD, Marlow C, Settle JE, Fowler JH (2012) A 61-million-person experiment in social influence and political mobilization. Nature 489(7415):295–298ADSCrossRefGoogle Scholar
  32. 32.
    Bond RM, Settle JE, Fariss CJ, Jones JJ, Fowler JH (2016) Social endorsement cues and political participation. Pol Commun 34:1–21CrossRefGoogle Scholar
  33. 33.
    Bowers J, Fredrickson MM, Panagopoulos C (2013) Reasoning about interference between units: a general framework. Polit Anal 21(1):97–124CrossRefGoogle Scholar
  34. 34.
    Bowers J, Fredrickson MM, Aronow PM (2016) A more powerful test statistic for reasoning about interference between units. Polit Anal 24:mpw018Google Scholar
  35. 35.
    Cai J, De Janvry A, Sadoulet E (2015) Social networks and the decision to insure. Am Econ J Appl Econ 7(2):81–108Google Scholar
  36. 36.
    Cameron AC, Gelbach JB, Miller DL (2011) Robust inference with multiway clustering. J Bus Econ Stat 29(2):238–249MathSciNetCrossRefGoogle Scholar
  37. 37.
    Carrell SE, Fullerton RL, West JE (2009) Does your cohort matter? Measuring peer effects in college achievement. J Labor Econ 27(3):439–464CrossRefGoogle Scholar
  38. 38.
    Carrell SE, Sacerdote BI, West JE (2011) From natural variation to optimal policy? The lucas critique meets peer effects. National Bureau of economic research working paper series no. 16865.
  39. 39.
    Centola D (2010) The spread of behavior in an online social network experiment. Science 329(5996):1194–1197ADSCrossRefGoogle Scholar
  40. 40.
    Choi DS (2017) Estimation of monotone treatment effects in network experiments. J Am Stat Assoc 112(519):1147–1155MathSciNetCrossRefGoogle Scholar
  41. 41.
    Chung E, Romano JP (2013) Exact and asymptotically robust permutation tests. Ann Stat 41(2):484–507MathSciNetCrossRefGoogle Scholar
  42. 42.
    Coey D, Bailey M (2016) People and cookies: imperfect treatment assignment in online experiments. In: Proceedings of the 25th international conference on world wide web, international world wide web conferences steering committee, pp 1103–1111Google Scholar
  43. 43.
    Coppock A, Guess A, Ternovski J (2015) When treatments are tweets: a network mobilization experiment over twitter. Polit Behav 38(1):1–24CrossRefGoogle Scholar
  44. 44.
    Currarini S, Jackson MO, Pin P (2010) Identifying the roles of race-based choice and chance in high school friendship network formation. Proc Natl Acad Sci 107(11):4857–4861ADSCrossRefGoogle Scholar
  45. 45.
    De Choudhury M, Mason WA, Hofman JM, Watts DJ (2010) Inferring relevant social networks from interpersonal communication. In: Proceedings of the 19th international conference on world wide web. ACM, New York, pp 301–310Google Scholar
  46. 46.
    DeLean A, Munson P, Rodbard D (1978) Simultaneous analysis of families of sigmoidal curves: application to bioassay, radioligand assay, and physiological dose-response curves. Am J Physiol Gastrointest Liver Physiol 235(2):G97–102CrossRefGoogle Scholar
  47. 47.
    Dittrich D, Kenneally E et al (2012) The Menlo report: ethical principles guiding information and communication technology research. US Department of Homeland SecurityGoogle Scholar
  48. 48.
    Duflo E, Saez E (2003) The role of information and social interactions in retirement plan decisions: evidence from a randomized experiment. Q J Econ 118(3):815–842CrossRefGoogle Scholar
  49. 49.
    Eckles D, Kizilcec RF, Bakshy E (2016) Estimating peer effects in networks with peer encouragement designs. Proc Natl Acad Sci 113(27):7316–7322CrossRefGoogle Scholar
  50. 50.
    Eckles D, Bakshy, E (2017) Bias and high-dimensional adjustment in observational studies of peer effects, working paper.
  51. 51.
    Eckles D, Karrer B, Ugander J (2017) Design and analysis of experiments in networks: reducing bias from interference. J Causal Inference.
  52. 52.
    Firth JA, Sheldon BC, Farine DR (2016) Pathways of information transmission among wild songbirds follow experimentally imposed changes in social foraging structure. Biol Lett 12(6):20160,144CrossRefGoogle Scholar
  53. 53.
    Fisher RA (1935) The design of experiments. Oliver and Boyd, EdinburghGoogle Scholar
  54. 54.
    Forastiere L, Mealli F, VanderWeele TJ (2015) Identification and estimation of causal mechanisms in clustered encouragement designs: disentangling bed nets using Bayesian principal stratification. J Am Stat Assoc 111(514):510–525MathSciNetCrossRefGoogle Scholar
  55. 55.
    Gerber AS, Green DP (2012) Field experiments: design, analysis, and interpretation. WW Norton, New YorkGoogle Scholar
  56. 56.
    Granovetter M (1978) Threshold models of collective behavior. Am J Sociol 83(6): 1420–1443CrossRefGoogle Scholar
  57. 57.
    Halberstam Y, Knight B (2016) Homophily, group size, and the diffusion of political information in social networks: evidence from twitter. J Public Econ 143:73–88CrossRefGoogle Scholar
  58. 58.
    Halloran ME, Hudgens MG (2016) Dependent happenings: a recent methodological review. Curr Epidemiol Rep 3(4):297–305CrossRefGoogle Scholar
  59. 59.
    Hartmann WR (2010) Demand estimation with social interactions and the implications for targeted marketing. Mark Sci 29(4):585–601CrossRefGoogle Scholar
  60. 60.
    Hasan S, Koning R (2017) Conversational peers and idea generation: evidence from a field experiment Stanford University Graduate School of Business Research Paper No. 17–36.
  61. 61.
    Higgins MJ, Sävje F, Sekhon JS (2016) Improving massive experiments with threshold blocking. Proc Natl Acad Sci 113(27):7369–7376CrossRefGoogle Scholar
  62. 62.
    Huber GA, Malhotra N (2017) Political homophily in social relationships: evidence from online dating behavior. J Polit 79(1):269–283CrossRefGoogle Scholar
  63. 63.
    Imbens GW, Rosenbaum PR (2005) Robust, accurate confidence intervals with a weak instrument: quarter of birth and education. J R Stat Soc Ser A (Stat Soc) 168(1):109–126MathSciNetCrossRefGoogle Scholar
  64. 64.
    Imbens GW, Rubin DB (2015) Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, CambridgeGoogle Scholar
  65. 65.
    Jackman M, Kanerva L (2016) Evolving the IRB: building robust review for industry research. Washington and Lee Law Rev Online 72(3):442Google Scholar
  66. 66.
    Jones JJ, Settle JE, Bond RM, Fariss CJ, Marlow C, Fowler JH (2013) Inferring tie strength from online directed behavior. PloS one 8(1):e52168ADSCrossRefGoogle Scholar
  67. 67.
    Jones JJ, Bond RM, Bakshy E, Eckles D, Fowler JH (2017) Social influence and political mobilization: further evidence from a randomized experiment in the 2012 US presidential election. PLoS One 12(4):e0173851CrossRefGoogle Scholar
  68. 68.
    Karrer B, Newman ME (2011) Stochastic blockmodels and community structure in networks. Phys Rev E 83(1):016107Google Scholar
  69. 69.
    Kearns M, Suri S, Montfort N (2006) An experimental study of the coloring problem on human subject networks. Science 313(5788):824–827ADSCrossRefGoogle Scholar
  70. 70.
    Kempe D, Kleinberg J, Tardos É (2003) Maximizing the spread of influence through a social network. In: Proceedings of the ninth ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, pp 137–146Google Scholar
  71. 71.
    Kim DA, Hwong AR, Stafford D, Hughes DA, O’Malley AJ, Fowler JH, Christakis NA (2015) Social network targeting to maximise population behaviour change: a cluster randomised controlled trial. Lancet 386(9989):145–153CrossRefGoogle Scholar
  72. 72.
    Kohavi R, Deng A, Frasca B, Longbotham R, Walker T, Xu Y (2012) Trustworthy online controlled experiments: five puzzling outcomes explained. In: Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, pp 786–794Google Scholar
  73. 73.
    Kossinets G, Watts DJ (2006) Empirical analysis of an evolving social network. Science 311(5757):88–90ADSMathSciNetCrossRefGoogle Scholar
  74. 74.
    Kramer AD, Guillory JE, Hancock JT (2014) Experimental evidence of massive-scale emotional contagion through social networks. Proc Natl Acad Sci 111(24):8788–8790ADSCrossRefGoogle Scholar
  75. 75.
    Leider S, Möbius MM, Rosenblat T, Do QA (2009) Directed altruism and enforced reciprocity in social networks. Q J Econ 124(4):1815–1851CrossRefGoogle Scholar
  76. 76.
    Leung MP (2017) A weak law for moments of pairwise-stable networks, working paper.
  77. 77.
    Manski CF (1993) Identification of endogenous social effects: the reflection problem. Rev Econ Stud 60(3):531–542MathSciNetCrossRefGoogle Scholar
  78. 78.
    Manski CF (2000) Economic analysis of social interactions. Technical report, National bureau of economic researchGoogle Scholar
  79. 79.
    Manzi J (2012) Uncontrolled: the surprising payoff of trial-and-error for business, politics, and society. Basic Books, New YorkGoogle Scholar
  80. 80.
    Marmaros D, Sacerdote B (2006) How do friendships form? Q J Econ 121(1):79–119zbMATHGoogle Scholar
  81. 81.
    Mason W, Suri S (2012) Conducting behavioral research on amazon’s mechanical turk. Behav Res Methods 44(1):1–23CrossRefGoogle Scholar
  82. 82.
    Mason W, Watts DJ (2012) Collaborative learning in networks. Proc Natl Acad Sci 109(3):764–769ADSCrossRefGoogle Scholar
  83. 83.
    McPherson M, Smith-Lovin L, Cook JM (2001) Birds of a feather: homophily in social networks. Annu Rev Sociol 27(1):415–444CrossRefGoogle Scholar
  84. 84.
    Miguel E, Kremer M (2004) Worms: identifying impacts on education and health in the presence of treatment externalities. Econometrica 72(1):159–217CrossRefGoogle Scholar
  85. 85.
    Miratrix LW, Sekhon JS, Yu B (2013) Adjusting treatment effect estimates by post-stratification in randomized experiments. J R Stat Soc Ser B (Stat Methodol) 75(2):369–396MathSciNetCrossRefGoogle Scholar
  86. 86.
    Moffitt RA et al (2001) Policy interventions, low-level equilibria, and social interactions. Soc Dyn 4:45–82Google Scholar
  87. 87.
    Morris S (2000) Contagion. Rev Econ Stud 67(1):57MathSciNetCrossRefGoogle Scholar
  88. 88.
    Muchnik L, Aral S, Taylor SJ (2013) Social influence bias: a randomized experiment. Science 341(6146):647–651ADSCrossRefGoogle Scholar
  89. 89.
    Narayanan A, Shmatikov V (2010) Myths and fallacies of personally identifiable information. Commun ACM 53(6):24–26CrossRefGoogle Scholar
  90. 90.
    Nickerson DW (2008) Is voting contagious? Evidence from two field experiments. Am Polit Sci. Rev 102(01):49–57CrossRefGoogle Scholar
  91. 91.
    Ohm P (2010) Broken promises of privacy: responding to the surprising failure of anonymization. UCLA L Rev 57:1701–1819Google Scholar
  92. 92.
    Owen AB, Eckles D (2012) Bootstrapping data arrays of arbitrary order. Ann Appl Stat 6(3):895–927MathSciNetCrossRefGoogle Scholar
  93. 93.
    Paluck EL, Shepherd H, Aronow PM (2016) Changing climates of conflict: a social network experiment in 56 schools. Proc Natl Acad Sci 113(3):566–571ADSCrossRefGoogle Scholar
  94. 94.
    Rand DG, Arbesman S, Christakis NA (2011) Dynamic social networks promote cooperation in experiments with humans. Proc Natl Acad Sci 108(48):19193–19198ADSCrossRefGoogle Scholar
  95. 95.
    Rand DG, Nowak MA, Fowler JH, Christakis NA (2014) Static network structure can stabilize human cooperation. Proc Natl Acad Sci 111(48):17093–17098ADSCrossRefGoogle Scholar
  96. 96.
    Resea B, Ryan KJP (1978) The Belmont Report: ethical principles and guidelines for the protection of human subjects of research-the national commission for the protection of human subjects of biomedical and behavioral research. US Government Printing OfficeGoogle Scholar
  97. 97.
    Resnick MD, Bearman PS, Blum RW, Bauman KE, Harris KM, Jones J, Tabor J, Beuhring T, Sieving RE, Shew M et al (1997) Protecting adolescents from harm: findings from the national longitudinal study on adolescent health. J Am Math Assoc 278(10):823–832CrossRefGoogle Scholar
  98. 98.
    Riach PA, Rich J (2004) Deceptive field experiments of discrimination: are they ethical? Kyklos 57(3):457–470CrossRefGoogle Scholar
  99. 99.
    Rock D, Aral S, Taylor SJ (2016) Identification of peer effects in networked panel data. In: Thirty seventh international conference on information systemsGoogle Scholar
  100. 100.
    Rosenbaum PR (1984) Conditional permutation tests and the propensity score in observational studies. J Am Stat Assoc 79(387):565–574MathSciNetCrossRefGoogle Scholar
  101. 101.
    Ryan B, Gross NC (1943) The diffusion of hybrid seed corn in two Iowa communities. Rural Soc 8(1):15Google Scholar
  102. 102.
    Ryan SP, Tucker C (2012) Heterogeneity and the dynamics of technology adoption. Quant Mark Econ 10(1):63–109CrossRefGoogle Scholar
  103. 103.
    Sacerdote B (2001) Peer effects with random assignment: results for dartmouth roommates. Q J Econ 116(2):681–704MathSciNetCrossRefGoogle Scholar
  104. 104.
    Salganik MJ (2017) Bit by bit: social research in the digital age. Princeton University Press, PrincetonGoogle Scholar
  105. 105.
    Saul BC, Hudgens MG, Halloran ME (2017) Causal inference in the study of infectious disease. In: Handbook of statistics. Elsevier, AmsterdamGoogle Scholar
  106. 106.
    Saveski M, Pouget-Abadie J, Saint-Jacques G, Duan W, Ghosh S, Xu Y, Airoldi EM (2017) Detecting network effects: Randomizing over randomized experiments. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp 1027–1035Google Scholar
  107. 107.
    Schelling T (1969) Models of segregation. Am Econ Rev 59(2):488–493Google Scholar
  108. 108.
    Schelling T (1971) Dynamic models of segregation. J Math Sociol 1(2):143–186CrossRefGoogle Scholar
  109. 109.
    Schelling T (1973) Hockey helmets, concealed weapons, and daylight saving: a study of binary choices with externalities. J Confl Resolut 17(3):381–428CrossRefGoogle Scholar
  110. 110.
    Schultz AP, Piepgrass B, Weng CC, Ferrante D, Verma D, Martinazzi P, Alison T, Mao Z (2012) Methods and systems for determining use and content of PYMK based on value model. US Patent App. 13/659,695Google Scholar
  111. 111.
    Shalita A, Karrer B, Kabiljo I, Sharma A, Presta A, Adcock A, Kllapi H, Stumm M (2016) Social hash: an assignment framework for optimizing distributed systems operations on social networks. In: NSDI, pp 455–468Google Scholar
  112. 112.
    Shalizi CR, Thomas AC (2011) Homophily and contagion are generically confounded in observational social network studies. Sociol Methods Res 40(2):211–239MathSciNetCrossRefGoogle Scholar
  113. 113.
    Suri S, Watts DJ (2011) Cooperation and contagion in web-based, networked public goods experiments. PloS one 6(3):e16836ADSCrossRefGoogle Scholar
  114. 114.
    Taylor SJ, Bakshy E, Aral S (2013) Selection effects in online sharing: consequences for peer adoption. In: Proceedings of the fourteenth ACM conference on electronic commerce. ACM, New York, pp 821–836Google Scholar
  115. 115.
    Taylor SJ, Muchnik L, Aral S (2014) Identity and opinion: a randomized experiment, working paper.
  116. 116.
    Tchetgen EJT, Fulcher I, Shpitser I (2017) Auto-G-computation of causal effects on a network. arXiv preprint arXiv:170901577Google Scholar
  117. 117.
    Toulis P, Kao E (2013) Estimation of causal peer influence effects. In: Proceedings of the 30th international conference on machine learning, pp 1489–1497Google Scholar
  118. 118.
    Ugander J, Karrer B, Backstrom L, Kleinberg JM (2013) Graph cluster randomization: network exposure to multiple universes. In: Proceedings of KDD, ACM, New YorkGoogle Scholar
  119. 119.
    Valente TW (1996) Social network thresholds in the diffusion of innovations. Soc Netw 18(1):69–89CrossRefGoogle Scholar
  120. 120.
    van der Laan MJ (2014) Causal inference for a population of causally connected units. J Causal Inference 2:1–6CrossRefGoogle Scholar
  121. 121.
    VanderWeele TJ, Tchetgen EJT, Halloran ME (2012) Components of the indirect effect in vaccine trials: identification of contagion and infectiousness effects. Epidemiology 23(5):751CrossRefGoogle Scholar
  122. 122.
    Walker D, Muchnik L (2014) Design of randomized experiments in networks. Proc IEEE 102(12):1940–1951CrossRefGoogle Scholar
  123. 123.
    Xu Y, Chen N, Fernandez A, Sinno O, Bhasin A (2015) From infrastructure to culture: A/B testing challenges in large scale social networks. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, pp 2227–2236CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.FacebookMenlo ParkUSA
  2. 2.Sloan School of Management and Institute for Data, Systems and SocietyMassachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations