Group Decision and Negotiation

, Volume 27, Issue 2, pp 197–214 | Cite as

Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments

  • Florian Teschner
  • Henner GimpelEmail author


Crowd labor markets such as Amazon Mechanical Turk (MTurk) have emerged as popular platforms where researchers can relatively inexpensively and easily run web-based experiments. Some work even suggests that MTurk can be used to run large-scale field experiments in which groups of participants interact synchronously in real-time such as electronic markets. Besides technical issues, several methodological questions arise and lead to the question of how results from MTurk and laboratory experiments compare. Our data shows comparable results between MTurk and a standard lab setting with student subjects in a controlled environment when running rather simple individual decision tasks. However, our data shows stark differences in results between the experimental settings for a rather complex market experiment. Each experimental setting—lab and MTurk—has its own benefits and drawbacks; which of the two settings is better suited for a specific experiment depends on the theory or artifact to be tested. We discuss potential causes for differences (language understanding, education, cognition and context) that we cannot control for and provide guidance for the selection of the appropriate setting for an experiment. In any case, researchers studying complex artifacts like group decisions or markets should not prematurely adopt MTurk based on extant literature regarding comparable results across experimental settings for rather simple tasks.


Experiments Mechanical turk Electronic markets Information aggregation 

JEL Classification

C9 D8 


  1. Aïmeura E, Lawani O, Dalkir K (2016) When changing the look of privacy policies affects user trust: an experimental study. Comput Hum Behav 58:368–379CrossRefGoogle Scholar
  2. Amir O, Rand DG, Gal YK (2012) Economic games on the internet: the effect of $1 Stakes. PLoS ONE 7(2):e31461CrossRefGoogle Scholar
  3. Barber BM, Odean T (2000) Trading is hazardous to your wealth: the common stock investment performance of individual investors. J Finance 55(2):773–806CrossRefGoogle Scholar
  4. Bennouri M, Gimpel H, Robert J (2011) Measuring the impact of information aggregation mechanisms: an experimental investigation. J Econ Behav Organ 78(3):302–318CrossRefGoogle Scholar
  5. Berg JE, Rietz TA (2003) Prediction markets as decision support systems. Inf Syst Front 5(1):79–93CrossRefGoogle Scholar
  6. Berg JE, Nelson FD, Rietz TA (2008) Prediction market accuracy in the long run. Int J Forecast 24(2):285–300CrossRefGoogle Scholar
  7. Berinsky AJ, Huber GA, Lenz GS (2012) Evaluating online labor markets for experimental research: Amazon. com’s mechanical turk. Polit Anal 20(3):351–368CrossRefGoogle Scholar
  8. Bichler M, Kersten G, Strecker S (2003) Towards a structured design of electronic negotiations. Group Decis Negot 12(4):311–335CrossRefGoogle Scholar
  9. Blohm I, Riedl C, Leimeister JM, Krcmar H (2011) Idea evaluation mechanisms for collective intelligence in open innovation communities: do traders outperform raters?. In: Proceedings of the thirty second international conference on information systems (ICIS 2011), Shanghai, ChinaGoogle Scholar
  10. Breusch TS, Pagan AR (1980) The Lagrange multiplier test and its applications to model specification in econometrics. Rev Econ Stud 47:239–253CrossRefGoogle Scholar
  11. Buhrmester M, Kwang T, Gosling SD (2011) Amazon’s mechanical turk: a new source of inexpensive, yet high-quality data? Perspect Psychol Sci 6(1):3–5CrossRefGoogle Scholar
  12. Casey LS, Jesse Chandler J, Levine AS, Proctor A, Strolovitch DZ (2017) Intertemporal differences among MTurk workers: time-based sample variations and implications for online data collection. SAGE Open.
  13. Chandler D, Kapelner A (2013) Breaking monotony with meaning: motivation in crowdsourcing markets. J Econ Behav Organ 90:123–133CrossRefGoogle Scholar
  14. Chen DL, Horton JJ (2016) Are online labor markets spot markets for tasks?A field experiment on the behavioral response to wage cuts. Inf Syst Res 27(2):403–423CrossRefGoogle Scholar
  15. Chilton LB, Horton JJ, Miller RC, Azenkot S (2010) Task search in a human computation market. In: Proceedings of the ACM SIGKDD workshop on human computation (HCOMP ‘10), New York, NYGoogle Scholar
  16. Djamasbi S, Bengisu B, Loiacono E, Whitefleet-Smith J (2008) Can a reasonable time limit improve the effective usage of a computerized decision aid? Commun Assoc Inf Syst 23:22Google Scholar
  17. Fair RC, Shiller RJ (1989) The informational context of ex-ante forecasts. Rev Econ Stat 71:325–331CrossRefGoogle Scholar
  18. Ferreira A, Antunes P, Herskovic V (2011) Improving group attention: an experiment with synchronous brainstorming. Group Decis Negot 20(5):643–666CrossRefGoogle Scholar
  19. Frederick S (2005) Cognitive reflection and decision making. J Econ Perspect 19(4):25–42CrossRefGoogle Scholar
  20. Graves JT, Acquisti A, Anderson R (2014) Experimental measurement of attitudes regarding cybercrime. In: 13th annual workshop on the economics of information security (WEIS 2014), University Park/State College, PAGoogle Scholar
  21. Hanson R (2003) Combinatorial information market design. Inf Syst Front 5(1):107–119CrossRefGoogle Scholar
  22. Healy PJ, Linardi S, Lowery JR, Ledyard JO (2010) Prediction markets: alternative mechanisms for complex environments with few traders. Manag Sci 56(11):1977–1996CrossRefGoogle Scholar
  23. Horton JJ, Rand DG, Zeckhauser RJ (2011) The online laboratory: conducting experiments in a real labor market. Exp Econ 14(3):399–425CrossRefGoogle Scholar
  24. Jian L, Sami R (2012) Aggregation and manipulation in prediction markets: effects of trading mechanism and information distribution. Manag Sci 58(1):123–140CrossRefGoogle Scholar
  25. Jilke S, Van Ryzin GG, Van de Walle S (2015) Responses to decline in marketized public services: an experimental evaluation of choice overload. J Public Adm Res Theor 26(3):421–432CrossRefGoogle Scholar
  26. Jones JL, Collins RW, Berndt DJ (2009) Information markets: a research landscape. Commun Assoc Inf Syst 25(1):27Google Scholar
  27. Kaufmann N, Schulze T, Veit D (2011) More than fun and money. Worker motivation in crowdsourcing—a study on mechanical turk. In: Proceedings of the 17th Americas conference on information systems (AMCIS 2011), Detroit, MIGoogle Scholar
  28. Kern R, Thies H, Satzger G (2011) Efficient quality management of human-based electronic services leveraging group decision making .In: Proceedings of the 19th European conference on information systems (ECIS 2011), Helsinki, FinlandGoogle Scholar
  29. Kersten G, Noronha S (1999) Negotiation via the world wide web: a cross-cultural study of decision making. Group Decis Negot 8(3):251–279CrossRefGoogle Scholar
  30. Kersten G, Köszegi ST, Vetschera R (2002) The effects of culture in anonymous negotiations: experiment in four countries. In: Proceedings of the 35th Hawaii international conference on system sciences (HICSS-35’02), Big Island, HIGoogle Scholar
  31. Landemore H, Elster J (eds) (2012) Collective wisdom: principles and mechanisms. Cambridge University Press, New YorkGoogle Scholar
  32. Lavoie J (2009) The innovation engine at rite-soluations: lessons from the CEO. J Predict Mark 3:1–11Google Scholar
  33. Ledyard J, Hanson R, Ishikida T (2009) An experimental test of combinatorial information markets. J Econ Behav Organ 69(2):182–189CrossRefGoogle Scholar
  34. Levy Y, Ellis TJ (2011) A guide for novice researchers on experimental and quasi-experimental studies in information systems research. Interdiscip J Inf Knowl Manag 6:151–161Google Scholar
  35. Malone TW, Laubacher R, Dellarocas C (2010) The collective intelligence genome. MIT Sloan Manag Rev 51(3):21–31Google Scholar
  36. Mao A, Chen Y, Gajos KZ, Parkes D, Procaccia AD, Zhang H (2012). TurkServer: enabling synchronous and longitudinal online experiments. In: Proceedings of the fourth workshop on human computation (HCOMP ‘12), Toronto, CanadaGoogle Scholar
  37. Mason W, Suri S (2012) Conducting behavioral research on Amazon’s mechanical turk. Behav Res Methods 44(1):1–23CrossRefGoogle Scholar
  38. Mullinix KJ, Leeper TJ, Druckman JN, Freese J (2015) The generalizability of survey experiments. J Exp Polit Sci 2:109–138CrossRefGoogle Scholar
  39. Nagar Y, Malone TW (2011) Making business predictions by combining human and machine intelligence in prediction markets. In: Proceedings of the thirty second international conference on information systems (ICIS 2011), Shanghai, ChinaGoogle Scholar
  40. Palvia P, Leary D, Mao E, Midha V, Pinjani P, Salam AF (2004) Research methodologies in MIS: an update. Commun Assoc Inf Syst 14:24Google Scholar
  41. Paolacci G, Chandler J, Ipeirotis P (2010) Running experiments on Amazon mechanical turk. Judgm Decis Mak 5(5):411–419Google Scholar
  42. Pilz D, Gewald H (2013) Does money matter? Motivational factors for participation in paid-and non-profit-crowdsourcing communities. In: 11th International conference on Wirtschaftsinformatik, Leipzig, Germany, pp 577–591Google Scholar
  43. Pinsonneault A, Barki H, Gallupe RB, Hoppen N (1999) Electronic brainstorming: the illusion of productivity. Inf Syst Res 10(2):110–133CrossRefGoogle Scholar
  44. Plott CR, Sunder S (1988) Rational expectations and the aggregation of diverse information in laboratory security markets. Econom J Econom Soc 56(5):1085–1118Google Scholar
  45. Qiu L, Rui H, Whinston A (2011) A twitter-based prediction market: social network approach. In: Proceedings of the thirty second international conference on information systems (ICIS 2011), Shanghai, ChinaGoogle Scholar
  46. Ross J, Zaldivar A, Irani L, Tomlinson B (2009) Who are the turkers? Worker demographics in Amazon mechanical turk. Technical report, University of California, Irvine, CAGoogle Scholar
  47. Slamka C, Luckner S, Seemann T, Schröder J (2008) An empirical investigation of the forecast accuracy of play-money prediction markets and professional betting markets. In: Proceedings of the 16th European conference on information systems (ECIS 2008), Galway, Ireland, paper 236Google Scholar
  48. Spann M, Skiera B (2003) Internet-based virtual stock markets for business forecasting. Manag Sci 49(10):1310–1326CrossRefGoogle Scholar
  49. Straub T, Gimpel H, Teschner F, Weinhardt C (2014) Feedback and performance in crowd work: a real effort experiment. In: Proceedings of the 22nd European conference on information systems (ECIS)Google Scholar
  50. Straub T, Gimpel H, Teschner F, Weinhardt C (2015) How (not) to incent crowd workers. Bus Inf Syst Eng 57:167–179CrossRefGoogle Scholar
  51. Teschner F, Mazarakis A, Riordan R, Weinhardt C (2011) Participation, feedback & incentives in a competitive forecasting community. In: Proceedings of the international conference on information systems (ICIS 2011), Shanghai, ChinaGoogle Scholar
  52. Teschner F, Rothschild D, Gimpel H (2017) Manipulation in conditional decision markets. Group Decis Negot. Google Scholar
  53. Wolfers J, Zitzewitz E (2004) Prediction markets. J Econ Perspect 18(2):107–126CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Karlsruhe Institute of Technology (KIT)KarlsruheGermany
  2. 2.University of AugsburgAugsburgGermany

Personalised recommendations