Advertisement

Influencing and Measuring Behaviour in Crowdsourced Activities

  • Sandy J. J. Gould
  • Anna L. Cox
  • Duncan P. Brumby
Chapter
Part of the Human–Computer Interaction Series book series (HCIS)

Abstract

Crowdsourcing psychometric data is common in areas of Human-Computer Interaction (HCI) such as information visualization, text entry, and interface design. In some of the social sciences, crowdsourcing data is now considered routine, and even standard. In this chapter, we explore the collection of data in this manner, beginning by describing the variety of approaches can be used to crowdsource data. Then, we evaluate past literature that has compared the results of these approaches to more traditional data-collection techniques. From this literature, we synthesize a set of design and implementation guidelines for crowdsourcing studies. Finally, we describe how particular analytic techniques can be recruited to aid the analysis of large-scale crowdsourced data. The goal of this chapter it to clearly enumerate the difficulties of crowdsourcing psychometric data and to explore how, with careful planning and execution, these limitations can be overcome.

References

  1. Abbey JD, Meloy MG (2017) Attention by design: using attention checks to detect inattentive respondents and improve data quality. J Oper Manag.  https://doi.org/10.1016/j.jom.2017.06.001
  2. Aitamurto T, Landemore H, Galli JS (2017) Unmasking the crowd: participants’ motivation factors, expectations, and profile in a crowdsourced law reform. Inf Commun Soc 20:1239–1260.  https://doi.org/10.1080/1369118X.2016.1228993 CrossRefGoogle Scholar
  3. Alkhatib A, Bernstein MS, Levi M (2017) Examining crowd work and gig work through the historical lens of piecework. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ‘17. ACM, New York, pp 4599–4616.  https://doi.org/10.1145/3025453.3025974 CrossRefGoogle Scholar
  4. Aubry E, Silverston T, Lahmadi A, Festor O (2014) CrowdOut: A mobile crowdsourcing service for road safety in digital cities, in: 2014 IEEE International Conference on Pervasive Computing and Communication Workshops (PERCOM WORKSHOPS). Presented at the 2014 IEEE International Conference on Pervasive Computing and Communication Workshops (PERCOM WORKSHOPS), pp 86–91  https://doi.org/10.1109/PerComW.2014.6815170
  5. Bailly G, Oulasvirta A, Brumby DP, Howes A (2014) Model of visual search and selection time in linear menus. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 3865–3874.  https://doi.org/10.1145/2556288.2557093 CrossRefGoogle Scholar
  6. Behrend TS, Sharek DJ, Meade AW, Wiebe EN (2011) The viability of crowdsourcing for survey research. Behav Res Methods 43:800–813.  https://doi.org/10.3758/s13428-011-0081-0 CrossRefGoogle Scholar
  7. Biggs J, Ewald N, Valentini A, Gaboriaud C, Dejean T, Griffiths RA, Foster J, Wilkinson JW, Arnell A, Brotherton P, Williams P, Dunn F (2015) Using eDNA to develop a national citizen science-based monitoring programme for the great crested newt (Triturus cristatus). Biol Conserv. Environmental DNA: A powerful new tool for biological conservation 183, 19–28.  https://doi.org/10.1016/j.biocon.2014.11.029
  8. Bødker S (2015) Third-wave HCI, 10 years later—participation and sharing. Interactions 22:24–31.  https://doi.org/10.1145/2804405 CrossRefGoogle Scholar
  9. Bowser A, Shilton K, Preece J, Warrick E (2017) Accounting for privacy in citizen science: ethical research in a context of openness. In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, CSCW ‘17. ACM, New York, pp 2124–2136.  https://doi.org/10.1145/2998181.2998305 CrossRefGoogle Scholar
  10. Brown HR, Zeidman P, Smittenaar P, Adams RA, McNab F, Rutledge RB, Dolan RJ (2014) Crowdsourcing for cognitive science – the utility of smartphones. PLoS One 9:e100662.  https://doi.org/10.1371/journal.pone.0100662 CrossRefGoogle Scholar
  11. Brumby DP, Howes A (2008) Strategies for guiding interactive search: an empirical investigation into the consequences of label relevance for assessment and selection. Human–Computer Interact 23:1–46.  https://doi.org/10.1080/07370020701851078 CrossRefGoogle Scholar
  12. Brumby DP, Zhuang S (2015) Visual grouping in menu interfaces. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 4203–4206.  https://doi.org/10.1145/2702123.2702177 CrossRefGoogle Scholar
  13. Brumby DP, Cox AL, Chung J, Fernandes B (2014) How does knowing what you are looking for change visual search behavior? In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 3895–3898.  https://doi.org/10.1145/2556288.2557064 CrossRefGoogle Scholar
  14. Cardamone C, Schawinski K, Sarzi M, Bamford SP, Bennert N et al (2009) Galaxy zoo green peas: discovery of a class of compact extremely star-forming galaxies. Mon Not R Astron Soc 399:1191–1205.  https://doi.org/10.1111/j.1365-2966.2009.15383.x CrossRefGoogle Scholar
  15. Cecchinato ME, Cox AL, Bird J (2015) Working 9-5?: professional differences in email and boundary management practices. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 3989–3998.  https://doi.org/10.1145/2702123.2702537 CrossRefGoogle Scholar
  16. Cecchinato ME, Cox AL, Bird J (2017) Always on(line)?: user experience of smartwatches and their role within multi-device ecologies. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ‘17. ACM, New York, pp 3557–3568.  https://doi.org/10.1145/3025453.3025538 CrossRefGoogle Scholar
  17. Chandler J, Mueller P, Paolacci G (2014) Nonnaïveté among Amazon mechanical Turk workers: consequences and solutions for behavioral researchers. Behav Res Methods 46:112–130.  https://doi.org/10.3758/s13428-013-0365-7 CrossRefGoogle Scholar
  18. Chandler J, Paolacci G, Peer E, Mueller P, Ratliff K.A (2015) Using nonnaive participants can reduce effect sizes. Psychol Sci 0956797615585115.  https://doi.org/10.1177/0956797615585115, 1131
  19. Chatzimilioudis G, Konstantinidis A, Laoudias C, Zeinalipour-Yazti D (2012) Crowdsourcing with smartphones. IEEE Internet Comput 16:36–44.  https://doi.org/10.1109/MIC.2012.70 CrossRefGoogle Scholar
  20. Cheng J, Teevan J, Iqbal ST, Bernstein MS (2015) Break it down: a comparison of macro- and microtasks. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 4061–4064.  https://doi.org/10.1145/2702123.2702146 CrossRefGoogle Scholar
  21. Colley A, Thebault-Spieker J, Lin AY, Degraen D et al (2017) The geography of PokéMon GO: beneficial and problematic effects on places and movement. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ‘17. ACM, New York, pp 1179–1192.  https://doi.org/10.1145/3025453.3025495 CrossRefGoogle Scholar
  22. Corrigan-Gibbs H, Gupta N, Northcutt C, Cutrell E, Thies W (2015) Deterring cheating in online environments. ACM Trans Comput-Hum Interact 22(28):1–28:23.  https://doi.org/10.1145/2810239 CrossRefGoogle Scholar
  23. Crivellaro C, Taylor A, Vlachokyriakos V, Comber R, Nissen B, Wright P (2016) Re-making places: HCI, “community building” and change. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 2958–2969.  https://doi.org/10.1145/2858036.2858332 CrossRefGoogle Scholar
  24. Dai P, Rzeszotarski JM, Paritosh P, Chi EH (2015) And now for something completely different: improving crowdsourcing workflows with micro-diversions. In: Proceedings of the 18th ACM conference on Computer Supported Cooperative Work & Social Computing, CSCW ‘15. ACM, New York, pp 628–638.  https://doi.org/10.1145/2675133.2675260 CrossRefGoogle Scholar
  25. Dunlop M, Levine J (2012) Multidimensional Pareto optimization of touchscreen keyboards for speed, familiarity and improved spell checking. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘12. ACM, New York, pp 2669–2678.  https://doi.org/10.1145/2207676.2208659 CrossRefGoogle Scholar
  26. Dunning D (2011) The Dunning-Kruger effect: on being ignorant of one’s own ignorance. In: Advances in experimental social psychology. Academic, Cambridge, MAGoogle Scholar
  27. Dunning D, Heath C, Suls JM (2004) Flawed self-assessment: implications for health, education, and the workplace. Psychol Sci Public Interest J Am Psychol Soc 5:69–106.  https://doi.org/10.1111/j.1529-1006.2004.00018.x CrossRefGoogle Scholar
  28. Epstein DA, Caraway M, Johnston C, Ping A, Fogarty J, Munson SA (2016) Beyond abandonment to next steps: understanding and designing for life after personal informatics tool use. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 1109–1113.  https://doi.org/10.1145/2858036.2858045 CrossRefGoogle Scholar
  29. Eveleigh A, Jennett C, Blandford A, Brohan P, Cox AL (2014) Designing for dabblers and deterring drop-outs in citizen science. In: Proceedings of the 32nd annual ACM conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 2985–2994.  https://doi.org/10.1145/2556288.2557262 CrossRefGoogle Scholar
  30. Ferreira D, Kostakos V, Dey AK (2015) AWARE: mobile context instrumentation framework. Front ICT 2.  https://doi.org/10.3389/fict.2015.00006
  31. Findlater L, Zhang J, Froehlich JE, Moffatt K (2017) Differences in crowdsourced vs. lab-based mobile and desktop input performance data. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ‘17. ACM, New York, pp 6813–6824.  https://doi.org/10.1145/3025453.3025820 CrossRefGoogle Scholar
  32. Fox S, Dimond J, Irani L, Hirsch T, Muller M, Bardzell S (2017) Social justice and design: power and oppression in collaborative systems. In: Companion of the 2017 ACM conference on Computer Supported Cooperative Work And Social Computing, CSCW ‘17 companion. ACM, New York, pp 117–122.  https://doi.org/10.1145/3022198.3022201 CrossRefGoogle Scholar
  33. Gadiraju U, Fetahu B, Kawase R, Siehndel P, Dietze S (2017) Using worker self-assessments for competence-based pre-selection in crowdsourcing microtasks. ACM Trans Comput-Hum Interact 24:30:1–30:26.  https://doi.org/10.1145/3119930 CrossRefGoogle Scholar
  34. Germine L, Nakayama K, Duchaine BC, Chabris CF, Chatterjee G, Wilmer JB (2012) Is the web as good as the lab? Comparable performance from web and lab in cognitive/perceptual experiments. Psychon Bull Rev 19:847–857.  https://doi.org/10.3758/s13423-012-0296-9 CrossRefGoogle Scholar
  35. Gitau S, Marsden G, Donner J (2010) After access: challenges facing mobile-only internet users in the developing world. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘10. ACM, New York, pp 2603–2606.  https://doi.org/10.1145/1753326.1753720 CrossRefGoogle Scholar
  36. Gould SJJ, Cox AL, Brumby DP (2013) Frequency and duration of self-initiated task-switching in an online investigation of interrupted performance. In: Human computation and crowdsourcing: works in progress and demonstration abstracts AAAI technical report CR-13-01. Presented at the first AAAI conference on human computation and crowdsourcing, AAAI, pp 22–23Google Scholar
  37. Gould SJJ, Cox AL, Brumby DP (2015a) Task lockouts induce crowdworkers to switch to other activities. Conf Hum Factors Comput Syst Proc 18:1785–1790Google Scholar
  38. Gould SJJ, Cox AL, Brumby DP, Wiseman S (2015b) Home is where the lab is: a comparison of online and lab data from a time-sensitive study of interruption. Hum Comput 2:45–67.  https://doi.org/10.15346/hc.v2i1.4 CrossRefGoogle Scholar
  39. Gould SJJ, Cox AL, Brumby DP (2016a) Diminished control in crowdsourcing: an investigation of crowdworker multitasking behavior. ACM Trans Comput-Hum Interact 23:19:1–19:29.  https://doi.org/10.1145/2928269 CrossRefGoogle Scholar
  40. Gould SJJ, Cox AL, Brumby DP, Wickersham A (2016b) Now check your input: brief task lockouts encourage checking, longer lockouts encourage task switching. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 3311–3323.  https://doi.org/10.1145/2858036.2858067 CrossRefGoogle Scholar
  41. Graber MA, Graber A (2012) Internet-based crowdsourcing and research ethics: the case for IRB review. J Med Ethics medethics-2012-100798  https://doi.org/10.1136/medethics-2012-100798 CrossRefGoogle Scholar
  42. Gray WD (2017) Game-XP: action games as experimental paradigms for cognitive science. Top Cogn Sci 9:289–307.  https://doi.org/10.1111/tops.12260 CrossRefGoogle Scholar
  43. Haklay M (2010) How good is volunteered geographical information? A comparative study of OpenStreetMap and ordnance survey datasets. Environ Plan B Plan Des 37:682–703.  https://doi.org/10.1068/b35097 CrossRefGoogle Scholar
  44. Hara K, Adams A, Milland K, Savage S, Callison-Burch C, Bigham J (2017) A data-driven analysis of workers’ earnings on Amazon Mechanical Turk. ArXiv171205796 CsGoogle Scholar
  45. Harbach M, De Luca A, Egelman S (2016) The anatomy of smartphone unlocking: a field study of android lock screens. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 4806–4817.  https://doi.org/10.1145/2858036.2858267 CrossRefGoogle Scholar
  46. Harris CG (2015) The effects of pay-to-quit incentives on Crowdworker task quality. In: Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, CSCW ‘15. ACM, New York, pp 1801–1812.  https://doi.org/10.1145/2675133.2675185 CrossRefGoogle Scholar
  47. Hauser D.J, Schwarz N (2015) It’s a trap! Instructional manipulation checks prompt systematic thinking on “tricky” tasks. SAGE Open 5, 2158244015584617.  https://doi.org/10.1177/2158244015584617 CrossRefGoogle Scholar
  48. Hoonakker P, Carayon P (2009) Questionnaire survey nonresponse: a comparison of postal mail and internet surveys. Int J Hum Comput Interact 25:348.  https://doi.org/10.1080/10447310902864951 CrossRefGoogle Scholar
  49. Horton JJ, Chilton LB (2010) The labor economics of paid crowdsourcing. In: Proceedings of the 11th ACM conference on electronic commerce, EC ‘10. ACM, New York, pp 209–218.  https://doi.org/10.1145/1807342.1807376 CrossRefGoogle Scholar
  50. Hsieh G, Kraut RE, Hudson SE (2010) Why pay?: exploring how financial incentives are used for question & answer. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘10. ACM, New York, pp 305–314.  https://doi.org/10.1145/1753326.1753373 CrossRefGoogle Scholar
  51. Irani LC, Silberman MS (2013) Turkopticon: interrupting worker invisibility in Amazon Mechanical Turk. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘13. ACM, New York, pp 611–620.  https://doi.org/10.1145/2470654.2470742 CrossRefGoogle Scholar
  52. Jay C, Dunne R, Gelsthorpe D, Vigo M (2016) To sign up, or not to sign up?: maximizing citizen science contribution rates through optional registration. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 1827–1832.  https://doi.org/10.1145/2858036.2858319 CrossRefGoogle Scholar
  53. Jennett C, Furniss DJ, Iacovides I, Wiseman S, Gould SJJ, Cox AL (2014) Exploring citizen psych-science and the motivations of Errordiary volunteers. Hum Comput 1.  https://doi.org/10.15346/hc.v1i2.10
  54. Jennett C, Kloetzer L, Schneider D, Iacovides I, Cox A et al (2016) Motivations, learning and creativity in online citizen science. J Sci Commun 15Google Scholar
  55. Jiang L, Wagner C, Nardi B (2015) Not just in it for the money: a qualitative investigation of workers’ perceived benefits of micro-task crowdsourcing. In: 2015 48th Hawaii International Conference on System Sciences (HICSS). Presented at the 2015 48th Hawaii International Conference on System Sciences (HICSS), pp 773–782  https://doi.org/10.1109/HICSS.2015.98
  56. Jun E, Hsieh G, Reinecke K (2017) Types of motivation affect study selection, attention, and dropouts in online experiments. Proc ACM Hum-Comput Interact 1(56):1–56:15.  https://doi.org/10.1145/3134691 CrossRefGoogle Scholar
  57. Kapelner A, Chandler D (2010) Preventing satisficing in online surveys, in: CrowdConfGoogle Scholar
  58. Kim S, Robson C, Zimmerman T, Pierce J, Haber EM (2011) Creek watch: pairing usefulness and usability for successful citizen science. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘11. ACM, New York, pp 2125–2134.  https://doi.org/10.1145/1978942.1979251 CrossRefGoogle Scholar
  59. Kim Y, Hsu S-H, de Zúñiga HG (2013) Influence of social media use on discussion network heterogeneity and civic engagement: the moderating role of personality traits. J Commun 63:498–516.  https://doi.org/10.1111/jcom.12034 CrossRefGoogle Scholar
  60. Kittur A, Nickerson JV, Bernstein M, Gerber E, Shaw A et al (2013) The future of crowd work. In: Proceedings of the 2013 conference on Computer Supported Cooperative Work, CSCW ‘13. ACM, New York, pp 1301–1318.  https://doi.org/10.1145/2441776.2441923 CrossRefGoogle Scholar
  61. Komarov S, Reinecke K, Gajos KZ (2013) Crowdsourcing performance evaluations of user interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘13. ACM, New York, pp 207–216.  https://doi.org/10.1145/2470654.2470684 CrossRefGoogle Scholar
  62. Land-Zandstra AM, Devilee JLA, Snik F, Buurmeijer F, van den BJM (2015) Citizen science on a smartphone: participants’ motivations and learning. Public Underst Sci:45.  https://doi.org/10.1177/0963662515602406 CrossRefGoogle Scholar
  63. Lane ND, Chon Y, Zhou L, Zhang Y, Li F et al (2013) Piggyback CrowdSensing (PCS): energy efficient crowdsourcing of mobile sensor data by exploiting smartphone app opportunities. In: Proceedings of the 11th ACM conference on embedded networked sensor systems, SenSys ‘13. ACM, New York, pp 7:1–7:14.  https://doi.org/10.1145/2517351.2517372 CrossRefGoogle Scholar
  64. Lasecki WS, Rzeszotarski JM, Marcus A, Bigham JP (2015) The effects of sequence and delay on crowd work. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 1375–1378.  https://doi.org/10.1145/2702123.2702594 CrossRefGoogle Scholar
  65. Lintott CJ, Schawinski K, Slosar A, Land K, Bamford S et al (2008) Galaxy zoo: morphologies derived from visual inspection of galaxies from the Sloan digital sky survey. Mon Not R Astron Soc 389:1179–1189.  https://doi.org/10.1111/j.1365-2966.2008.13689.x CrossRefGoogle Scholar
  66. Mason W, Watts DJ (2010) Financial incentives and the “performance of crowds”. SIGKDD Explor Newsl 11:100–108.  https://doi.org/10.1145/1809400.1809422 CrossRefGoogle Scholar
  67. Mehrotra A, Pejovic V, Vermeulen J, Hendley R, Musolesi M (2016) My phone and me: understanding people’s receptivity to mobile notifications. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 1021–1032.  https://doi.org/10.1145/2858036.2858566 CrossRefGoogle Scholar
  68. Meys HL, Sanderson PM (2013) The effect of individual differences on how people handle interruptions. Proc Hum Factors Ergon Soc Annu Meet 57:868–872.  https://doi.org/10.1177/1541931213571188 CrossRefGoogle Scholar
  69. Moore J, Gay PL, Hogan K, Lintott C, Impey C, Watson C (2011) Facebooking citizen science with the Zooniverse. In: Bulletin of the American Astronomical Society. Presented at the Bulletin of the American Astronomical Society. p 15813Google Scholar
  70. Newell E, Ruths D (2016) How one microtask affects another. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 3155–3166.  https://doi.org/10.1145/2858036.2858490 CrossRefGoogle Scholar
  71. Oppenheimer DM, Meyvis T, Davidenko N (2009) Instructional manipulation checks: detecting satisficing to increase statistical power. J Exp Soc Psychol 45:867–872.  https://doi.org/10.1016/j.jesp.2009.03.009 CrossRefGoogle Scholar
  72. Organisciak P, Twidale MB (2015) Design facets of crowdsourcing. iConference 2015 Proceedings. http://hdl.handle.net/2142/73640
  73. Oulasvirta A, Reichel A, Li W, Zhang Y et al (2013) Improving two-thumb text entry on touchscreen devices. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘13. ACM, New York, pp 2765–2774.  https://doi.org/10.1145/2470654.2481383 CrossRefGoogle Scholar
  74. Peer E, Brandimarte L, Samat S, Acquisti A (2017) Beyond the Turk: alternative platforms for crowdsourcing behavioral research. J Exp Soc Psychol 70:153–163.  https://doi.org/10.1016/j.jesp.2017.01.006 CrossRefGoogle Scholar
  75. Pielot M, Church K, de Oliveira R (2014) An In-situ study of mobile phone notifications, in: Proceedings of the 16th International conference on human-computer interaction with mobile devices & services, MobileHCI ‘14. ACM, New York, pp. 233–242.  https://doi.org/10.1145/2628363.2628364
  76. Preece J (2016) Citizen science: new research challenges for human–computer interaction. Int J Hum Comput Interact 32:585–612.  https://doi.org/10.1080/10447318.2016.1194153 CrossRefGoogle Scholar
  77. Rand DG (2012) The promise of mechanical Turk: how online labor markets can help theorists run behavioral experiments. J Theor Biol Evol Coop 299:172–179.  https://doi.org/10.1016/j.jtbi.2011.03.004 MathSciNetCrossRefGoogle Scholar
  78. Reeves N, Tinati R, Zerr S, Van Kleek M.G, Simperl E (2017) From crowd to community: a survey of online community features in citizen science projects, in: Proceedings of the 2017 ACM conference on Computer Supported Cooperative Work and Social Computing, CSCW ‘17. ACM, New York, pp. 2137–2152.  https://doi.org/10.1145/2998181.2998302
  79. Reinecke K, Gajos KZ (2015) LabintheWild: conducting large-scale online experiments with uncompensated samples. In: Proceedings of the 18th ACM conference on Computer Supported Cooperative Work & Social Computing, CSCW ‘15. ACM, New York, pp 1364–1378.  https://doi.org/10.1145/2675133.2675246 CrossRefGoogle Scholar
  80. Reinecke K, Nguyen MK, Bernstein A, Näf M, Gajos KZ (2013) Doodle around the world: online scheduling behavior reflects cultural differences in time perception and group decision-making. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ‘13. ACM, New York, pp 45–54.  https://doi.org/10.1145/2441776.2441784
  81. Resnik DB, Elliott KC, Miller AK (2015) A framework for addressing ethical issues in citizen science. Environ Sci Pol 54:475–481.  https://doi.org/10.1016/j.envsci.2015.05.008 CrossRefGoogle Scholar
  82. Riegler M, Gaddam V.R, Larson M, Eg R, Halvorsen P, Griwodz C (2016) Crowdsourcing as self-fulfilling prophecy: influence of discarding workers in subjective assessment tasks, in: 2016 14th international workshop on Content-Based Multimedia Indexing (CBMI). Presented at the 2016 14th international workshop on Content-Based Multimedia Indexing (CBMI), pp. 1–6.  https://doi.org/10.1109/CBMI.2016.7500256
  83. Riesch H, Potter C (2014) Citizen science as seen by scientists: methodological, epistemological and ethical dimensions. Public Underst Sci 23:107–120.  https://doi.org/10.1177/0963662513497324 CrossRefGoogle Scholar
  84. Rigby JM, Brumby DP, Gould SJJ, Cox AL (2017) Media multitasking at home: a video observation study of concurrent TV and mobile device usage. In: Proceedings of the 2017 ACM international conference on interactive experiences for TV and online video, TVX ‘17. ACM, New York, pp 3–10.  https://doi.org/10.1145/3077548.3077560 CrossRefGoogle Scholar
  85. Rzeszotarski JM, Kittur A (2011) Instrumenting the crowd: using implicit behavioral measures to predict task performance. In: Proceedings of the 24th annual ACM symposium on User Interface Software and Technology, UIST ‘11. ACM, New York, pp 13–22.  https://doi.org/10.1145/2047196.2047199 CrossRefGoogle Scholar
  86. Rzeszotarski JM, Kittur A (2012) CrowdScape: interactively visualizing user behavior and output. In: Proceedings of the 25th annual ACM symposium on User Interface Software and Technology, UIST ‘12. ACM, New York, pp 55–62.  https://doi.org/10.1145/2380116.2380125 CrossRefGoogle Scholar
  87. Rzeszotarski J.M, Chi E, Paritosh P, Dai P (2013) Inserting micro-breaks into crowdsourcing workflows. In: human computation and crowdsourcing: works in progress and demonstration abstracts AAAI technical report CR-13-01. Presented at the First AAAI Conference on Human Computation and Crowdsourcing, AAAI, pp 62–63Google Scholar
  88. Salehi N, Irani LC, Bernstein MS, Alkhatib A, Ogbe E et al (2015) We are dynamo: overcoming stalling and friction in collective action for crowd workers. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 1621–1630.  https://doi.org/10.1145/2702123.2702508 CrossRefGoogle Scholar
  89. Shaw AD, Horton JJ, Chen DL (2011) Designing incentives for inexpert human raters. In: Proceedings of the ACM 2011 conference on computer supported cooperative work, CSCW ‘11. ACM, New York, pp 275–284.  https://doi.org/10.1145/1958824.1958865 CrossRefGoogle Scholar
  90. Silberman MS, Irani L, Ross J (2010) Ethics and tactics of professional crowdwork. XRDS 17:39–43.  https://doi.org/10.1145/1869086.1869100 CrossRefGoogle Scholar
  91. Silvertown J (2009) A new dawn for citizen science. Trends Ecol Evol 24:467–471.  https://doi.org/10.1016/j.tree.2009.03.017 CrossRefGoogle Scholar
  92. Stawarz K, Cox A.L, Blandford A (2015) Beyond self-tracking and reminders: designing smartphone apps that support habit formation, in: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp. 2653–2662.  https://doi.org/10.1145/2702123.2702230
  93. Stewart N, Chandler J, Paolacci G (2017) Crowdsourcing samples in cognitive science. Trends Cogn Sci 21:736–748.  https://doi.org/10.1016/j.tics.2017.06.007 CrossRefGoogle Scholar
  94. Sullivan BL, Wood CL, Iliff MJ, Bonney RE, Fink D, Kelling S (2009) eBird: a citizen-based bird observation network in the biological sciences. Biol Conserv 142:2282–2292.  https://doi.org/10.1016/j.biocon.2009.05.006 CrossRefGoogle Scholar
  95. Truong KN, Shihipar T, Wigdor DJ (2014) Slide to X: unlocking the potential of smartphone unlocking. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 3635–3644.  https://doi.org/10.1145/2556288.2557044 CrossRefGoogle Scholar
  96. Vaish R, Wyngarden K, Chen J, Cheung B, Bernstein MS (2014) Twitch crowdsourcing: crowd contributions in short bursts of time. In: Proceedings of the 32nd annual ACM conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 3645–3654.  https://doi.org/10.1145/2556288.2556996 CrossRefGoogle Scholar
  97. Vlachokyriakos V, Crivellaro C, Le Dantec CA, Gordon E, Wright P, Olivier P (2016) Digital civics: citizen empowerment with and through technology. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems, CHI EA ‘16. ACM, New York, pp 1096–1099.  https://doi.org/10.1145/2851581.2886436 CrossRefGoogle Scholar
  98. Wiggins A, He Y (2016) Community-based data validation practices in citizen science. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW ‘16. ACM, New York, pp 1548–1559.  https://doi.org/10.1145/2818048.2820063 CrossRefGoogle Scholar
  99. Wiseman S, Cox AL, Gould SJJ, Brumby DP (2017) Exploring the effects of non-monetary reimbursement for participants in HCI research. Hum Comput 4:1–24.  https://doi.org/10.15346/hc.v4i1.1 CrossRefGoogle Scholar
  100. Wright KB (2005) Researching internet-based populations: advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. J Comput-Mediat Commun 10:00–00.  https://doi.org/10.1111/j.1083-6101.2005.tb00259.x CrossRefGoogle Scholar
  101. Xu Q, Xiong J, Huang Q, Yao Y (2013) Robust evaluation for quality of experience in crowdsourcing. In: Proceedings of the 21st ACM international conference on multimedia, MM ‘13. ACM, New York, pp 43–52.  https://doi.org/10.1145/2502081.2502083 CrossRefGoogle Scholar
  102. Ye T, Reinecke K, Robert LP (2017) Personalized feedback versus money: the effect on reliability of subjective data in online experimental platforms. In: Companion of the 2017 ACM conference on Computer Supported Cooperative Work and Social Computing, CSCW ‘17 Companion. ACM, New York, pp 343–346.  https://doi.org/10.1145/3022198.3026339 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Sandy J. J. Gould
    • 1
  • Anna L. Cox
    • 2
  • Duncan P. Brumby
    • 2
  1. 1.University of BirminghamBirminghamUK
  2. 2.University College LondonLondonUK

Personalised recommendations