Effect of Cognitive Abilities on Crowdsourcing Task Performance

  • Danula HettiachchiEmail author
  • Niels van Berkel
  • Simo Hosio
  • Vassilis Kostakos
  • Jorge Goncalves
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11746)


Matching crowd workers to suitable tasks is highly desirable as it can enhance task performance, reduce the cost for requesters, and increase worker satisfaction. In this paper, we propose a method that considers workers’ cognitive ability to predict their suitability for a wide range of crowdsourcing tasks. We measure cognitive ability via fast-paced online cognitive tests with a combined average duration of 6.2 min. We then demonstrate that our proposed method can effectively assign or recommend workers to five different popular crowd tasks: Classification, Counting, Proofreading, Sentiment Analysis, and Transcription. Using our approach we demonstrate a significant improvement in the expected overall task accuracy. While previous methods require access to worker history or demographics, our work offers a quick and accurate way to determine which workers are more suitable for which tasks.


Crowdsourcing Cognitive ability Task performance 


  1. 1.
    Alagarai Sampath, H., Rajeshuni, R., Indurkhya, B.: Cognitively inspired task design to improve user performance on crowdsourcing platforms. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2014, pp. 3665–3674. ACM, New York (2014).
  2. 2.
    Bailey, C.E.: Cognitive accuracy and intelligent executive function in the brain and in business. Ann. N. Y. Acad. Sci. 1118, 122–141 (2007). Scholar
  3. 3.
    Borella, E., Carretti, B., Pelegrina, S.: The specific role of inhibition in reading comprehension in good and poor comprehenders. J. Learn. Disabil. 43(6), 541–552 (2010). Scholar
  4. 4.
    Chilton, M.A., Hardgrave, B.C., Armstrong, D.J.: Person-job cognitive style fit for software developers: the effect on strain and performance. J. Manag. Inf. Syst. 22(2), 193–226 (2005). Scholar
  5. 5.
    Clair-Thompson, H.L.S., Gathercole, S.E.: Executive functions and achievements in school: shifting, updating, inhibition, and working memory. Q. J. Exp. Psychol. 59(4), 745–759 (2006). Scholar
  6. 6.
    Crump, M.J.C., McDonnell, J.V., Gureckis, T.M.: Evaluating Amazon’s mechanical Turk as a tool for experimental behavioral research. PLoS ONE 8(3), 1–18 (2013). Scholar
  7. 7.
    Damerau, F.J.: A technique for computer detection and correction of spelling errors. Commun. ACM 7(3), 171–176 (1964)CrossRefGoogle Scholar
  8. 8.
    Deshpande, M., Karypis, G.: Item-based Top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1), 143–177 (2004). Scholar
  9. 9.
    Diamond, A.: Executive functions. Annu. Rev. Psychol. 64(1), 135–168 (2013). Scholar
  10. 10.
    Difallah, D.E., Catasta, M., Demartini, G., Ipeirotis, P.G., Cudré-Mauroux, P.: The dynamics of micro-task crowdsourcing: the case of Amazon MTurk. In: Proceedings of the 24th International Conference on World Wide Web, WWW 2015, pp. 238–247. IW3C2, Switzerland (2015).
  11. 11.
    Difallah, D.E., Demartini, G., Cudré-Mauroux, P.: Pick-a-crowd: tell me what you like, and I’ll tell you what to do. In: Proceedings of the 22nd International Conference on World Wide Web, WWW 2013, pp. 367–374. ACM, New York (2013).
  12. 12.
    Dingler, T., Schmidt, A., Machulla, T.: Building cognition-aware systems: a mobile toolkit for extracting time-of-day fluctuations of cognitive performance. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(3) (2017). Scholar
  13. 13.
    Downs, J.S., Holbrook, M.B., Sheng, S., Cranor, L.F.: Are your participants gaming the system?: Screening mechanical Turk workers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2010, pp. 2399–2402. ACM, New York (2010).
  14. 14.
    Edwards, J.R.: Person-Job Fit: A Conceptual Integration, Literature Review, and Methodological Critique. Wiley, New York (1991)Google Scholar
  15. 15.
    Eickhoff, C.: Cognitive biases in crowdsourcing. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, pp. 162–170. ACM, New York (2018).
  16. 16.
    Ekstrom, R.B., Dermen, D., Harman, H.H.: Manual for Kit of Factor-referenced Cognitive Tests, vol. 102. Educational Testing Service, Princeton (1976)Google Scholar
  17. 17.
    Eriksen, B.A., Eriksen, C.W.: Effects of noise letters upon the identification of a target letter in a nonsearch task. Percept. Psychophys. 16(1), 143–149 (1974)CrossRefGoogle Scholar
  18. 18.
    Fan, J., Li, G., Ooi, B.C., Tan, K.l., Feng, J.: iCrowd: an adaptive crowdsourcing framework. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, pp. 1015–1030. ACM, New York (2015).
  19. 19.
    Federico, P.A., Landis, D.B.: Cognitive styles, abilities, and aptitudes: are they dependent or independent? Contemp. Educ. Psychol. 9(2), 146–161 (1984). Scholar
  20. 20.
    Gadiraju, U., Kawase, R., Dietze, S.: A taxonomy of microtasks on the web. In: Proceedings of the 25th ACM Conference on Hypertext and Social Media, HT 2014, pp. 218–223. ACM, New York (2014)Google Scholar
  21. 21.
    Germine, L., Nakayama, K., Duchaine, B.C., Chabris, C.F., Chatterjee, G., Wilmer, J.B.: Is the web as good as the lab? Comparable performance from web and lab in cognitive/perceptual experiments. Psychon. Bull. Rev. 19(5), 847–857 (2012). Scholar
  22. 22.
    Goncalves, J., Feldman, M., Hu, S., Kostakos, V., Bernstein, A.: Task routing and assignment in crowdsourcing based on cognitive abilities. In: Proceedings of the 26th International Conference on World Wide Web, WWW 2017, pp. 1023–1031. IW3C2, Switzerland (2017).
  23. 23.
    Goncalves, J., et al.: Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2013, pp. 753–762 (2013).
  24. 24.
    Goncalves, J., Hosio, S., van Berkel, N., Ahmed, F., Kostakos, V.: CrowdPickUp: crowdsourcing task pickup in the wild. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(3), 51:1–51:22 (2017). Scholar
  25. 25.
    Goncalves, J., Hosio, S., Rogstadius, J., Karapanos, E., Kostakos, V.: Motivating participation and improving quality of contribution in ubiquitous crowdsourcing. Comput. Netw. 90(C), 34–48 (2015). Scholar
  26. 26.
    Gureckis, T.M., et al.: psiTurk: an open-source framework for conducting replicable behavioral experiments online. Behav. Res. Methods 48(3), 829–842 (2016). Scholar
  27. 27.
    Hair, J., Black, W., Babin, B., Anderson, R.: Multivariate Data Analysis. Prentice-Hall, Upper Saddle River (2010)Google Scholar
  28. 28.
    Han, S., Dai, P., Paritosh, P., Huynh, D.: Crowdsourcing human annotation on web page structure: infrastructure design and behavior-based quality control. ACM Trans. Intell. Syst. Technol. 7(4), 56:1–56:25 (2016)CrossRefGoogle Scholar
  29. 29.
    Hoffman, B.J., Woehr, D.J.: A quantitative review of the relationship between person-organization fit and behavioral outcomes. J. Vocat. Behav. 68(3), 389–399 (2006). Scholar
  30. 30.
    Hosio, S., Goncalves, J., Lehdonvirta, V., Ferreira, D., Kostakos, V.: Situated crowdsourcing using a market model. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST 2014, pp. 55–64. ACM, New York (2014).
  31. 31.
    Jain, A., Sarma, A.D., Parameswaran, A., Widom, J.: Understanding workers, developing effective tasks, and enhancing marketplace dynamics: a study of a large crowdsourcing marketplace. Proc. VLDB Endow. 10(7), 829–840 (2017). Scholar
  32. 32.
    Kazai, G.: In search of quality in crowdsourcing for search engine evaluation. In: Clough, P., et al. (eds.) ECIR 2011. LNCS, vol. 6611, pp. 165–176. Springer, Heidelberg (2011). Scholar
  33. 33.
    Kazai, G., Kamps, J., Milic-Frayling, N.: Worker types and personality traits in crowdsourcing relevance labels. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM 2011, pp. 1941–1944. ACM, New York (2011).
  34. 34.
    Kazai, G., Kamps, J., Milic-Frayling, N.: The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM 2012, pp. 2583–2586. ACM, New York (2012).
  35. 35.
    Kazai, G., Zitouni, I.: Quality management in crowdsourcing using gold judges behavior. In: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, WSDM 2016, pp. 267–276. ACM, New York (2016).
  36. 36.
    Kittur, A., et al.: The future of crowd work. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW 2013, pp. 1301–1318. ACM, New York (2013).
  37. 37.
    Kristof, A.L.: Person-organization fit: an integrative review of its conceptualizations, measurement, and implications. Pers. Psychol. 49(1), 1–49 (1996)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Kristof-Brown, A.L., Zimmerman, R.D., Johnson, E.C.: Consequences of individuals’ fit at work: a meta-analysis of person-job, person-organization, person-group, and person-supervisor fit. Pers. Psychol. 58(2), 281–342 (2005)CrossRefGoogle Scholar
  39. 39.
    de Leeuw, J.R.: jsPsych: a javascript library for creating behavioral experiments in a web browser. Behav. Res. Methods 47(1), 1–12 (2015)CrossRefGoogle Scholar
  40. 40.
    Liu, X., Lu, M., Ooi, B.C., Shen, Y., Wu, S., Zhang, M.: CDAS: a crowdsourcing data analytics system. Proc. VLDB Endow. 5(10), 1040–1051 (2012)CrossRefGoogle Scholar
  41. 41.
    Lykourentzou, I., Antoniou, A., Naudet, Y., Dow, S.P.: Personality matters: balancing for personality types leads to better outcomes for crowd teams. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW 2016, pp. 260–273. ACM, New York (2016)Google Scholar
  42. 42.
    MacLeod, C.M.: Half a century of research on the stroop effect: an integrative review. Psychol. Bull. 109(2), 163 (1991)CrossRefGoogle Scholar
  43. 43.
    Mavridis, P., Gross-Amblard, D., Miklós, Z.: Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing. In: Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Switzerland, pp. 843–853. IW3C2 (2016).
  44. 44.
    McInnis, B., Cosley, D., Nam, C., Leshed, G.: Taking a hit: designing around rejection, mistrust, risk, and workers’ experiences in Amazon mechanical Turk. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 2016, pp. 2271–2282. ACM, New York (2016)Google Scholar
  45. 45.
    Mioshi, E., Dawson, K., Mitchell, J., Arnold, R., Hodges, J.R.: The Addenbrooke’s cognitive examination revised (ACE-R): a brief cognitive test battery for dementia screening. Int. J. Geriatr. Psychiatry 21(11), 1078–1085 (2006)CrossRefGoogle Scholar
  46. 46.
    Mo, K., Zhong, E., Yang, Q.: Cross-task crowdsourcing. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, pp. 677–685. ACM, New York (2013)Google Scholar
  47. 47.
    Monsell, S.: Task switching. Trends Cogn. Sci. 7(3), 134–140 (2003)CrossRefGoogle Scholar
  48. 48.
    Owen, A.M., et al.: Putting brain training to the test. Nature 465, 775 (2010). Scholar
  49. 49.
    Owen, A.M., McMillan, K.M., Laird, A.R., Bullmore, E.: N-back working memory paradigm: a meta-analysis of normative functional neuroimaging studies. Hum. Brain Mapp. 25(1), 46–59 (2005). Scholar
  50. 50.
    Petrides, M., Alivisatos, B., Evans, A.C., Meyer, E.: Dissociation of human mid-dorsolateral from posterior dorsolateral frontal cortex in memory processing. Proc. Natl. Acad. Sci. 90(3), 873–877 (1993)CrossRefGoogle Scholar
  51. 51.
    Robbins, T.W., James, M., Owen, A.M., Sahakian, B.J., McInnes, L., Rabbitt, P.: Cambridge Neuropsychological Test Automated Battery (CANTAB): a factor analytic study of a large sample of normal elderly volunteers. Dement. Geriatr. Cogn. Disord. 5(5), 266–281 (1994). Scholar
  52. 52.
    Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., Vukovic, M.: An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In: Proceedings of the Fifth International AAAI Conference on Web and Social Media, ICWSM, California, USA, vol. 11, pp. 17–21. AAAI (2011)Google Scholar
  53. 53.
    Ruble, T.L., Cosier, R.A.: Effects of cognitive styles and decision setting on performance. Organ. Behav. Hum. Decis. Process. 46(2), 283–295 (1990). Scholar
  54. 54.
    Rzeszotarski, J.M., Kittur, A.: Instrumenting the crowd: using implicit behavioral measures to predict task performance. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST 2011, pp. 13–22. ACM, New York (2011).
  55. 55.
    Schmidt, F.L., Hunter, J.: General mental ability in the world of work: occupational attainment and job performance. J. Pers. Soc. Psychol. 86(1), 162 (2004)CrossRefGoogle Scholar
  56. 56.
    Shaw, A.D., Horton, J.J., Chen, D.L.: Designing incentives for inexpert human raters. In: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, CSCW 2011, pp. 275–284. ACM, New York (2011)Google Scholar
  57. 57.
    Verquer, M.L., Beehr, T.A., Wagner, S.H.: A meta-analysis of relations between person-organization fit and work attitudes. J. Vocat. Behav. 63(3), 473–489 (2003). Scholar
  58. 58.
    Washington, G.: George Washington papers, series 5, financial papers: Copybook of invoices and letters, 1754-1766 (1766).
  59. 59.
    West, R.F., Toplak, M.E., Stanovich, K.E.: Heuristics and biases as measures of critical thinking: associations with cognitive ability and thinking dispositions. J. Educ. Psychol. 100(4), 930 (2008)CrossRefGoogle Scholar
  60. 60.
    Zheng, Y., Wang, J., Li, G., Cheng, R., Feng, J.: QASCA: a quality-aware task assignment system for crowdsourcing applications. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, pp. 1031–1046. ACM, New York (2015)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  1. 1.School of Computing and Information SystemsThe University of MelbourneMelbourneAustralia
  2. 2.Center for Ubiquitous ComputingUniversity of OuluOuluFinland

Personalised recommendations