Skip to main content
SpringerLink
Log in

Influencing and Measuring Behaviour in Crowdsourced Activities

  • Chapter
  • First Online:
New Directions in Third Wave Human-Computer Interaction: Volume 2 - Methodologies

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Crowdsourcing psychometric data is common in areas of Human-Computer Interaction (HCI) such as information visualization, text entry, and interface design. In some of the social sciences, crowdsourcing data is now considered routine, and even standard. In this chapter, we explore the collection of data in this manner, beginning by describing the variety of approaches can be used to crowdsource data. Then, we evaluate past literature that has compared the results of these approaches to more traditional data-collection techniques. From this literature, we synthesize a set of design and implementation guidelines for crowdsourcing studies. Finally, we describe how particular analytic techniques can be recruited to aid the analysis of large-scale crowdsourced data. The goal of this chapter it to clearly enumerate the difficulties of crowdsourcing psychometric data and to explore how, with careful planning and execution, these limitations can be overcome.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Abbey JD, Meloy MG (2017) Attention by design: using attention checks to detect inattentive respondents and improve data quality. J Oper Manag. https://doi.org/10.1016/j.jom.2017.06.001

  • Aitamurto T, Landemore H, Galli JS (2017) Unmasking the crowd: participants’ motivation factors, expectations, and profile in a crowdsourced law reform. Inf Commun Soc 20:1239–1260. https://doi.org/10.1080/1369118X.2016.1228993

    Article  Google Scholar 

  • Alkhatib A, Bernstein MS, Levi M (2017) Examining crowd work and gig work through the historical lens of piecework. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ‘17. ACM, New York, pp 4599–4616. https://doi.org/10.1145/3025453.3025974

    Chapter  Google Scholar 

  • Aubry E, Silverston T, Lahmadi A, Festor O (2014) CrowdOut: A mobile crowdsourcing service for road safety in digital cities, in: 2014 IEEE International Conference on Pervasive Computing and Communication Workshops (PERCOM WORKSHOPS). Presented at the 2014 IEEE International Conference on Pervasive Computing and Communication Workshops (PERCOM WORKSHOPS), pp 86–91 https://doi.org/10.1109/PerComW.2014.6815170

  • Bailly G, Oulasvirta A, Brumby DP, Howes A (2014) Model of visual search and selection time in linear menus. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 3865–3874. https://doi.org/10.1145/2556288.2557093

    Chapter  Google Scholar 

  • Behrend TS, Sharek DJ, Meade AW, Wiebe EN (2011) The viability of crowdsourcing for survey research. Behav Res Methods 43:800–813. https://doi.org/10.3758/s13428-011-0081-0

    Article  Google Scholar 

  • Biggs J, Ewald N, Valentini A, Gaboriaud C, Dejean T, Griffiths RA, Foster J, Wilkinson JW, Arnell A, Brotherton P, Williams P, Dunn F (2015) Using eDNA to develop a national citizen science-based monitoring programme for the great crested newt (Triturus cristatus). Biol Conserv. Environmental DNA: A powerful new tool for biological conservation 183, 19–28. https://doi.org/10.1016/j.biocon.2014.11.029

  • Bødker S (2015) Third-wave HCI, 10 years later—participation and sharing. Interactions 22:24–31. https://doi.org/10.1145/2804405

    Article  Google Scholar 

  • Bowser A, Shilton K, Preece J, Warrick E (2017) Accounting for privacy in citizen science: ethical research in a context of openness. In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, CSCW ‘17. ACM, New York, pp 2124–2136. https://doi.org/10.1145/2998181.2998305

    Chapter  Google Scholar 

  • Brown HR, Zeidman P, Smittenaar P, Adams RA, McNab F, Rutledge RB, Dolan RJ (2014) Crowdsourcing for cognitive science – the utility of smartphones. PLoS One 9:e100662. https://doi.org/10.1371/journal.pone.0100662

    Article  Google Scholar 

  • Brumby DP, Howes A (2008) Strategies for guiding interactive search: an empirical investigation into the consequences of label relevance for assessment and selection. Human–Computer Interact 23:1–46. https://doi.org/10.1080/07370020701851078

    Article  Google Scholar 

  • Brumby DP, Zhuang S (2015) Visual grouping in menu interfaces. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 4203–4206. https://doi.org/10.1145/2702123.2702177

    Chapter  Google Scholar 

  • Brumby DP, Cox AL, Chung J, Fernandes B (2014) How does knowing what you are looking for change visual search behavior? In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 3895–3898. https://doi.org/10.1145/2556288.2557064

    Chapter  Google Scholar 

  • Cardamone C, Schawinski K, Sarzi M, Bamford SP, Bennert N et al (2009) Galaxy zoo green peas: discovery of a class of compact extremely star-forming galaxies. Mon Not R Astron Soc 399:1191–1205. https://doi.org/10.1111/j.1365-2966.2009.15383.x

    Article  Google Scholar 

  • Cecchinato ME, Cox AL, Bird J (2015) Working 9-5?: professional differences in email and boundary management practices. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 3989–3998. https://doi.org/10.1145/2702123.2702537

    Chapter  Google Scholar 

  • Cecchinato ME, Cox AL, Bird J (2017) Always on(line)?: user experience of smartwatches and their role within multi-device ecologies. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ‘17. ACM, New York, pp 3557–3568. https://doi.org/10.1145/3025453.3025538

    Chapter  Google Scholar 

  • Chandler J, Mueller P, Paolacci G (2014) Nonnaïveté among Amazon mechanical Turk workers: consequences and solutions for behavioral researchers. Behav Res Methods 46:112–130. https://doi.org/10.3758/s13428-013-0365-7

    Article  Google Scholar 

  • Chandler J, Paolacci G, Peer E, Mueller P, Ratliff K.A (2015) Using nonnaive participants can reduce effect sizes. Psychol Sci 0956797615585115. https://doi.org/10.1177/0956797615585115, 1131

  • Chatzimilioudis G, Konstantinidis A, Laoudias C, Zeinalipour-Yazti D (2012) Crowdsourcing with smartphones. IEEE Internet Comput 16:36–44. https://doi.org/10.1109/MIC.2012.70

    Article  Google Scholar 

  • Cheng J, Teevan J, Iqbal ST, Bernstein MS (2015) Break it down: a comparison of macro- and microtasks. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 4061–4064. https://doi.org/10.1145/2702123.2702146

    Chapter  Google Scholar 

  • Colley A, Thebault-Spieker J, Lin AY, Degraen D et al (2017) The geography of PokéMon GO: beneficial and problematic effects on places and movement. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ‘17. ACM, New York, pp 1179–1192. https://doi.org/10.1145/3025453.3025495

    Chapter  Google Scholar 

  • Corrigan-Gibbs H, Gupta N, Northcutt C, Cutrell E, Thies W (2015) Deterring cheating in online environments. ACM Trans Comput-Hum Interact 22(28):1–28:23. https://doi.org/10.1145/2810239

    Article  Google Scholar 

  • Crivellaro C, Taylor A, Vlachokyriakos V, Comber R, Nissen B, Wright P (2016) Re-making places: HCI, “community building” and change. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 2958–2969. https://doi.org/10.1145/2858036.2858332

    Chapter  Google Scholar 

  • Dai P, Rzeszotarski JM, Paritosh P, Chi EH (2015) And now for something completely different: improving crowdsourcing workflows with micro-diversions. In: Proceedings of the 18th ACM conference on Computer Supported Cooperative Work & Social Computing, CSCW ‘15. ACM, New York, pp 628–638. https://doi.org/10.1145/2675133.2675260

    Chapter  Google Scholar 

  • Dunlop M, Levine J (2012) Multidimensional Pareto optimization of touchscreen keyboards for speed, familiarity and improved spell checking. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘12. ACM, New York, pp 2669–2678. https://doi.org/10.1145/2207676.2208659

    Chapter  Google Scholar 

  • Dunning D (2011) The Dunning-Kruger effect: on being ignorant of one’s own ignorance. In: Advances in experimental social psychology. Academic, Cambridge, MA

    Google Scholar 

  • Dunning D, Heath C, Suls JM (2004) Flawed self-assessment: implications for health, education, and the workplace. Psychol Sci Public Interest J Am Psychol Soc 5:69–106. https://doi.org/10.1111/j.1529-1006.2004.00018.x

    Article  Google Scholar 

  • Epstein DA, Caraway M, Johnston C, Ping A, Fogarty J, Munson SA (2016) Beyond abandonment to next steps: understanding and designing for life after personal informatics tool use. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 1109–1113. https://doi.org/10.1145/2858036.2858045

    Chapter  Google Scholar 

  • Eveleigh A, Jennett C, Blandford A, Brohan P, Cox AL (2014) Designing for dabblers and deterring drop-outs in citizen science. In: Proceedings of the 32nd annual ACM conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 2985–2994. https://doi.org/10.1145/2556288.2557262

    Chapter  Google Scholar 

  • Ferreira D, Kostakos V, Dey AK (2015) AWARE: mobile context instrumentation framework. Front ICT 2. https://doi.org/10.3389/fict.2015.00006

  • Findlater L, Zhang J, Froehlich JE, Moffatt K (2017) Differences in crowdsourced vs. lab-based mobile and desktop input performance data. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI ‘17. ACM, New York, pp 6813–6824. https://doi.org/10.1145/3025453.3025820

    Chapter  Google Scholar 

  • Fox S, Dimond J, Irani L, Hirsch T, Muller M, Bardzell S (2017) Social justice and design: power and oppression in collaborative systems. In: Companion of the 2017 ACM conference on Computer Supported Cooperative Work And Social Computing, CSCW ‘17 companion. ACM, New York, pp 117–122. https://doi.org/10.1145/3022198.3022201

    Chapter  Google Scholar 

  • Gadiraju U, Fetahu B, Kawase R, Siehndel P, Dietze S (2017) Using worker self-assessments for competence-based pre-selection in crowdsourcing microtasks. ACM Trans Comput-Hum Interact 24:30:1–30:26. https://doi.org/10.1145/3119930

    Article  Google Scholar 

  • Germine L, Nakayama K, Duchaine BC, Chabris CF, Chatterjee G, Wilmer JB (2012) Is the web as good as the lab? Comparable performance from web and lab in cognitive/perceptual experiments. Psychon Bull Rev 19:847–857. https://doi.org/10.3758/s13423-012-0296-9

    Article  Google Scholar 

  • Gitau S, Marsden G, Donner J (2010) After access: challenges facing mobile-only internet users in the developing world. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘10. ACM, New York, pp 2603–2606. https://doi.org/10.1145/1753326.1753720

    Chapter  Google Scholar 

  • Gould SJJ, Cox AL, Brumby DP (2013) Frequency and duration of self-initiated task-switching in an online investigation of interrupted performance. In: Human computation and crowdsourcing: works in progress and demonstration abstracts AAAI technical report CR-13-01. Presented at the first AAAI conference on human computation and crowdsourcing, AAAI, pp 22–23

    Google Scholar 

  • Gould SJJ, Cox AL, Brumby DP (2015a) Task lockouts induce crowdworkers to switch to other activities. Conf Hum Factors Comput Syst Proc 18:1785–1790

    Article  Google Scholar 

  • Gould SJJ, Cox AL, Brumby DP, Wiseman S (2015b) Home is where the lab is: a comparison of online and lab data from a time-sensitive study of interruption. Hum Comput 2:45–67. https://doi.org/10.15346/hc.v2i1.4

    Article  Google Scholar 

  • Gould SJJ, Cox AL, Brumby DP (2016a) Diminished control in crowdsourcing: an investigation of crowdworker multitasking behavior. ACM Trans Comput-Hum Interact 23:19:1–19:29. https://doi.org/10.1145/2928269

    Article  Google Scholar 

  • Gould SJJ, Cox AL, Brumby DP, Wickersham A (2016b) Now check your input: brief task lockouts encourage checking, longer lockouts encourage task switching. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 3311–3323. https://doi.org/10.1145/2858036.2858067

    Chapter  Google Scholar 

  • Graber MA, Graber A (2012) Internet-based crowdsourcing and research ethics: the case for IRB review. J Med Ethics medethics-2012-100798 https://doi.org/10.1136/medethics-2012-100798

  • Gray WD (2017) Game-XP: action games as experimental paradigms for cognitive science. Top Cogn Sci 9:289–307. https://doi.org/10.1111/tops.12260

    Article  Google Scholar 

  • Haklay M (2010) How good is volunteered geographical information? A comparative study of OpenStreetMap and ordnance survey datasets. Environ Plan B Plan Des 37:682–703. https://doi.org/10.1068/b35097

    Article  Google Scholar 

  • Hara K, Adams A, Milland K, Savage S, Callison-Burch C, Bigham J (2017) A data-driven analysis of workers’ earnings on Amazon Mechanical Turk. ArXiv171205796 Cs

    Google Scholar 

  • Harbach M, De Luca A, Egelman S (2016) The anatomy of smartphone unlocking: a field study of android lock screens. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 4806–4817. https://doi.org/10.1145/2858036.2858267

    Chapter  Google Scholar 

  • Harris CG (2015) The effects of pay-to-quit incentives on Crowdworker task quality. In: Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, CSCW ‘15. ACM, New York, pp 1801–1812. https://doi.org/10.1145/2675133.2675185

    Chapter  Google Scholar 

  • Hauser D.J, Schwarz N (2015) It’s a trap! Instructional manipulation checks prompt systematic thinking on “tricky” tasks. SAGE Open 5, 2158244015584617. https://doi.org/10.1177/2158244015584617

  • Hoonakker P, Carayon P (2009) Questionnaire survey nonresponse: a comparison of postal mail and internet surveys. Int J Hum Comput Interact 25:348. https://doi.org/10.1080/10447310902864951

    Article  Google Scholar 

  • Horton JJ, Chilton LB (2010) The labor economics of paid crowdsourcing. In: Proceedings of the 11th ACM conference on electronic commerce, EC ‘10. ACM, New York, pp 209–218. https://doi.org/10.1145/1807342.1807376

    Chapter  Google Scholar 

  • Hsieh G, Kraut RE, Hudson SE (2010) Why pay?: exploring how financial incentives are used for question & answer. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘10. ACM, New York, pp 305–314. https://doi.org/10.1145/1753326.1753373

    Chapter  Google Scholar 

  • Irani LC, Silberman MS (2013) Turkopticon: interrupting worker invisibility in Amazon Mechanical Turk. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘13. ACM, New York, pp 611–620. https://doi.org/10.1145/2470654.2470742

    Chapter  Google Scholar 

  • Jay C, Dunne R, Gelsthorpe D, Vigo M (2016) To sign up, or not to sign up?: maximizing citizen science contribution rates through optional registration. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 1827–1832. https://doi.org/10.1145/2858036.2858319

    Chapter  Google Scholar 

  • Jennett C, Furniss DJ, Iacovides I, Wiseman S, Gould SJJ, Cox AL (2014) Exploring citizen psych-science and the motivations of Errordiary volunteers. Hum Comput 1. https://doi.org/10.15346/hc.v1i2.10

  • Jennett C, Kloetzer L, Schneider D, Iacovides I, Cox A et al (2016) Motivations, learning and creativity in online citizen science. J Sci Commun 15

    Google Scholar 

  • Jiang L, Wagner C, Nardi B (2015) Not just in it for the money: a qualitative investigation of workers’ perceived benefits of micro-task crowdsourcing. In: 2015 48th Hawaii International Conference on System Sciences (HICSS). Presented at the 2015 48th Hawaii International Conference on System Sciences (HICSS), pp 773–782 https://doi.org/10.1109/HICSS.2015.98

  • Jun E, Hsieh G, Reinecke K (2017) Types of motivation affect study selection, attention, and dropouts in online experiments. Proc ACM Hum-Comput Interact 1(56):1–56:15. https://doi.org/10.1145/3134691

    Article  Google Scholar 

  • Kapelner A, Chandler D (2010) Preventing satisficing in online surveys, in: CrowdConf

    Google Scholar 

  • Kim S, Robson C, Zimmerman T, Pierce J, Haber EM (2011) Creek watch: pairing usefulness and usability for successful citizen science. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘11. ACM, New York, pp 2125–2134. https://doi.org/10.1145/1978942.1979251

    Chapter  Google Scholar 

  • Kim Y, Hsu S-H, de Zúñiga HG (2013) Influence of social media use on discussion network heterogeneity and civic engagement: the moderating role of personality traits. J Commun 63:498–516. https://doi.org/10.1111/jcom.12034

    Article  Google Scholar 

  • Kittur A, Nickerson JV, Bernstein M, Gerber E, Shaw A et al (2013) The future of crowd work. In: Proceedings of the 2013 conference on Computer Supported Cooperative Work, CSCW ‘13. ACM, New York, pp 1301–1318. https://doi.org/10.1145/2441776.2441923

    Chapter  Google Scholar 

  • Komarov S, Reinecke K, Gajos KZ (2013) Crowdsourcing performance evaluations of user interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘13. ACM, New York, pp 207–216. https://doi.org/10.1145/2470654.2470684

    Chapter  Google Scholar 

  • Land-Zandstra AM, Devilee JLA, Snik F, Buurmeijer F, van den BJM (2015) Citizen science on a smartphone: participants’ motivations and learning. Public Underst Sci:45. https://doi.org/10.1177/0963662515602406

  • Lane ND, Chon Y, Zhou L, Zhang Y, Li F et al (2013) Piggyback CrowdSensing (PCS): energy efficient crowdsourcing of mobile sensor data by exploiting smartphone app opportunities. In: Proceedings of the 11th ACM conference on embedded networked sensor systems, SenSys ‘13. ACM, New York, pp 7:1–7:14. https://doi.org/10.1145/2517351.2517372

    Chapter  Google Scholar 

  • Lasecki WS, Rzeszotarski JM, Marcus A, Bigham JP (2015) The effects of sequence and delay on crowd work. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 1375–1378. https://doi.org/10.1145/2702123.2702594

    Chapter  Google Scholar 

  • Lintott CJ, Schawinski K, Slosar A, Land K, Bamford S et al (2008) Galaxy zoo: morphologies derived from visual inspection of galaxies from the Sloan digital sky survey. Mon Not R Astron Soc 389:1179–1189. https://doi.org/10.1111/j.1365-2966.2008.13689.x

    Article  Google Scholar 

  • Mason W, Watts DJ (2010) Financial incentives and the “performance of crowds”. SIGKDD Explor Newsl 11:100–108. https://doi.org/10.1145/1809400.1809422

    Article  Google Scholar 

  • Mehrotra A, Pejovic V, Vermeulen J, Hendley R, Musolesi M (2016) My phone and me: understanding people’s receptivity to mobile notifications. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 1021–1032. https://doi.org/10.1145/2858036.2858566

    Chapter  Google Scholar 

  • Meys HL, Sanderson PM (2013) The effect of individual differences on how people handle interruptions. Proc Hum Factors Ergon Soc Annu Meet 57:868–872. https://doi.org/10.1177/1541931213571188

    Article  Google Scholar 

  • Moore J, Gay PL, Hogan K, Lintott C, Impey C, Watson C (2011) Facebooking citizen science with the Zooniverse. In: Bulletin of the American Astronomical Society. Presented at the Bulletin of the American Astronomical Society. p 15813

    Google Scholar 

  • Newell E, Ruths D (2016) How one microtask affects another. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ‘16. ACM, New York, pp 3155–3166. https://doi.org/10.1145/2858036.2858490

    Chapter  Google Scholar 

  • Oppenheimer DM, Meyvis T, Davidenko N (2009) Instructional manipulation checks: detecting satisficing to increase statistical power. J Exp Soc Psychol 45:867–872. https://doi.org/10.1016/j.jesp.2009.03.009

    Article  Google Scholar 

  • Organisciak P, Twidale MB (2015) Design facets of crowdsourcing. iConference 2015 Proceedings. http://hdl.handle.net/2142/73640

  • Oulasvirta A, Reichel A, Li W, Zhang Y et al (2013) Improving two-thumb text entry on touchscreen devices. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘13. ACM, New York, pp 2765–2774. https://doi.org/10.1145/2470654.2481383

    Chapter  Google Scholar 

  • Peer E, Brandimarte L, Samat S, Acquisti A (2017) Beyond the Turk: alternative platforms for crowdsourcing behavioral research. J Exp Soc Psychol 70:153–163. https://doi.org/10.1016/j.jesp.2017.01.006

    Article  Google Scholar 

  • Pielot M, Church K, de Oliveira R (2014) An In-situ study of mobile phone notifications, in: Proceedings of the 16th International conference on human-computer interaction with mobile devices & services, MobileHCI ‘14. ACM, New York, pp. 233–242. https://doi.org/10.1145/2628363.2628364

  • Preece J (2016) Citizen science: new research challenges for human–computer interaction. Int J Hum Comput Interact 32:585–612. https://doi.org/10.1080/10447318.2016.1194153

    Article  Google Scholar 

  • Rand DG (2012) The promise of mechanical Turk: how online labor markets can help theorists run behavioral experiments. J Theor Biol Evol Coop 299:172–179. https://doi.org/10.1016/j.jtbi.2011.03.004

    Article  MathSciNet  Google Scholar 

  • Reeves N, Tinati R, Zerr S, Van Kleek M.G, Simperl E (2017) From crowd to community: a survey of online community features in citizen science projects, in: Proceedings of the 2017 ACM conference on Computer Supported Cooperative Work and Social Computing, CSCW ‘17. ACM, New York, pp. 2137–2152. https://doi.org/10.1145/2998181.2998302

  • Reinecke K, Gajos KZ (2015) LabintheWild: conducting large-scale online experiments with uncompensated samples. In: Proceedings of the 18th ACM conference on Computer Supported Cooperative Work & Social Computing, CSCW ‘15. ACM, New York, pp 1364–1378. https://doi.org/10.1145/2675133.2675246

    Chapter  Google Scholar 

  • Reinecke K, Nguyen MK, Bernstein A, Näf M, Gajos KZ (2013) Doodle around the world: online scheduling behavior reflects cultural differences in time perception and group decision-making. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ‘13. ACM, New York, pp 45–54. https://doi.org/10.1145/2441776.2441784

  • Resnik DB, Elliott KC, Miller AK (2015) A framework for addressing ethical issues in citizen science. Environ Sci Pol 54:475–481. https://doi.org/10.1016/j.envsci.2015.05.008

    Article  Google Scholar 

  • Riegler M, Gaddam V.R, Larson M, Eg R, Halvorsen P, Griwodz C (2016) Crowdsourcing as self-fulfilling prophecy: influence of discarding workers in subjective assessment tasks, in: 2016 14th international workshop on Content-Based Multimedia Indexing (CBMI). Presented at the 2016 14th international workshop on Content-Based Multimedia Indexing (CBMI), pp. 1–6. https://doi.org/10.1109/CBMI.2016.7500256

  • Riesch H, Potter C (2014) Citizen science as seen by scientists: methodological, epistemological and ethical dimensions. Public Underst Sci 23:107–120. https://doi.org/10.1177/0963662513497324

    Article  Google Scholar 

  • Rigby JM, Brumby DP, Gould SJJ, Cox AL (2017) Media multitasking at home: a video observation study of concurrent TV and mobile device usage. In: Proceedings of the 2017 ACM international conference on interactive experiences for TV and online video, TVX ‘17. ACM, New York, pp 3–10. https://doi.org/10.1145/3077548.3077560

    Chapter  Google Scholar 

  • Rzeszotarski JM, Kittur A (2011) Instrumenting the crowd: using implicit behavioral measures to predict task performance. In: Proceedings of the 24th annual ACM symposium on User Interface Software and Technology, UIST ‘11. ACM, New York, pp 13–22. https://doi.org/10.1145/2047196.2047199

    Chapter  Google Scholar 

  • Rzeszotarski JM, Kittur A (2012) CrowdScape: interactively visualizing user behavior and output. In: Proceedings of the 25th annual ACM symposium on User Interface Software and Technology, UIST ‘12. ACM, New York, pp 55–62. https://doi.org/10.1145/2380116.2380125

    Chapter  Google Scholar 

  • Rzeszotarski J.M, Chi E, Paritosh P, Dai P (2013) Inserting micro-breaks into crowdsourcing workflows. In: human computation and crowdsourcing: works in progress and demonstration abstracts AAAI technical report CR-13-01. Presented at the First AAAI Conference on Human Computation and Crowdsourcing, AAAI, pp 62–63

    Google Scholar 

  • Salehi N, Irani LC, Bernstein MS, Alkhatib A, Ogbe E et al (2015) We are dynamo: overcoming stalling and friction in collective action for crowd workers. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp 1621–1630. https://doi.org/10.1145/2702123.2702508

    Chapter  Google Scholar 

  • Shaw AD, Horton JJ, Chen DL (2011) Designing incentives for inexpert human raters. In: Proceedings of the ACM 2011 conference on computer supported cooperative work, CSCW ‘11. ACM, New York, pp 275–284. https://doi.org/10.1145/1958824.1958865

    Chapter  Google Scholar 

  • Silberman MS, Irani L, Ross J (2010) Ethics and tactics of professional crowdwork. XRDS 17:39–43. https://doi.org/10.1145/1869086.1869100

    Article  Google Scholar 

  • Silvertown J (2009) A new dawn for citizen science. Trends Ecol Evol 24:467–471. https://doi.org/10.1016/j.tree.2009.03.017

    Article  Google Scholar 

  • Stawarz K, Cox A.L, Blandford A (2015) Beyond self-tracking and reminders: designing smartphone apps that support habit formation, in: Proceedings of the 33rd annual ACM conference on human factors in computing systems, CHI ‘15. ACM, New York, pp. 2653–2662. https://doi.org/10.1145/2702123.2702230

  • Stewart N, Chandler J, Paolacci G (2017) Crowdsourcing samples in cognitive science. Trends Cogn Sci 21:736–748. https://doi.org/10.1016/j.tics.2017.06.007

    Article  Google Scholar 

  • Sullivan BL, Wood CL, Iliff MJ, Bonney RE, Fink D, Kelling S (2009) eBird: a citizen-based bird observation network in the biological sciences. Biol Conserv 142:2282–2292. https://doi.org/10.1016/j.biocon.2009.05.006

    Article  Google Scholar 

  • Truong KN, Shihipar T, Wigdor DJ (2014) Slide to X: unlocking the potential of smartphone unlocking. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 3635–3644. https://doi.org/10.1145/2556288.2557044

    Chapter  Google Scholar 

  • Vaish R, Wyngarden K, Chen J, Cheung B, Bernstein MS (2014) Twitch crowdsourcing: crowd contributions in short bursts of time. In: Proceedings of the 32nd annual ACM conference on human factors in computing systems, CHI ‘14. ACM, New York, pp 3645–3654. https://doi.org/10.1145/2556288.2556996

    Chapter  Google Scholar 

  • Vlachokyriakos V, Crivellaro C, Le Dantec CA, Gordon E, Wright P, Olivier P (2016) Digital civics: citizen empowerment with and through technology. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems, CHI EA ‘16. ACM, New York, pp 1096–1099. https://doi.org/10.1145/2851581.2886436

    Chapter  Google Scholar 

  • Wiggins A, He Y (2016) Community-based data validation practices in citizen science. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW ‘16. ACM, New York, pp 1548–1559. https://doi.org/10.1145/2818048.2820063

    Chapter  Google Scholar 

  • Wiseman S, Cox AL, Gould SJJ, Brumby DP (2017) Exploring the effects of non-monetary reimbursement for participants in HCI research. Hum Comput 4:1–24. https://doi.org/10.15346/hc.v4i1.1

    Article  Google Scholar 

  • Wright KB (2005) Researching internet-based populations: advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. J Comput-Mediat Commun 10:00–00. https://doi.org/10.1111/j.1083-6101.2005.tb00259.x

    Article  Google Scholar 

  • Xu Q, Xiong J, Huang Q, Yao Y (2013) Robust evaluation for quality of experience in crowdsourcing. In: Proceedings of the 21st ACM international conference on multimedia, MM ‘13. ACM, New York, pp 43–52. https://doi.org/10.1145/2502081.2502083

    Chapter  Google Scholar 

  • Ye T, Reinecke K, Robert LP (2017) Personalized feedback versus money: the effect on reliability of subjective data in online experimental platforms. In: Companion of the 2017 ACM conference on Computer Supported Cooperative Work and Social Computing, CSCW ‘17 Companion. ACM, New York, pp 343–346. https://doi.org/10.1145/3022198.3026339

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Sandy J. J. Gould

  2. University College London, London, UK

    Anna L. Cox & Duncan P. Brumby

Authors
  1. Sandy J. J. Gould
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anna L. Cox
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Duncan P. Brumby
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sandy J. J. Gould .

Editor information

Editors and Affiliations

  1. School of Interactive Arts & Technology, Simon Fraser University, Surrey, BC, Canada

    Michael Filimowicz

  2. School of Communications, Simon Fraser University, Burnaby, BC, Canada

    Veronika Tzankova

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gould, S.J.J., Cox, A.L., Brumby, D.P. (2018). Influencing and Measuring Behaviour in Crowdsourced Activities. In: Filimowicz, M., Tzankova, V. (eds) New Directions in Third Wave Human-Computer Interaction: Volume 2 - Methodologies . Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-319-73374-6_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-73374-6_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-73373-9

  • Online ISBN: 978-3-319-73374-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation