Advertisement

Effect of Cognitive Abilities on Crowdsourcing Task Performance

  • Danula HettiachchiEmail author
  • Niels van Berkel
  • Simo Hosio
  • Vassilis Kostakos
  • Jorge Goncalves
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11746)

Abstract

Matching crowd workers to suitable tasks is highly desirable as it can enhance task performance, reduce the cost for requesters, and increase worker satisfaction. In this paper, we propose a method that considers workers’ cognitive ability to predict their suitability for a wide range of crowdsourcing tasks. We measure cognitive ability via fast-paced online cognitive tests with a combined average duration of 6.2 min. We then demonstrate that our proposed method can effectively assign or recommend workers to five different popular crowd tasks: Classification, Counting, Proofreading, Sentiment Analysis, and Transcription. Using our approach we demonstrate a significant improvement in the expected overall task accuracy. While previous methods require access to worker history or demographics, our work offers a quick and accurate way to determine which workers are more suitable for which tasks.

Keywords

Crowdsourcing Cognitive ability Task performance 

References

  1. 1.
    Alagarai Sampath, H., Rajeshuni, R., Indurkhya, B.: Cognitively inspired task design to improve user performance on crowdsourcing platforms. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2014, pp. 3665–3674. ACM, New York (2014).  https://doi.org/10.1145/2556288.2557155
  2. 2.
    Bailey, C.E.: Cognitive accuracy and intelligent executive function in the brain and in business. Ann. N. Y. Acad. Sci. 1118, 122–141 (2007).  https://doi.org/10.1196/annals.1412.011CrossRefGoogle Scholar
  3. 3.
    Borella, E., Carretti, B., Pelegrina, S.: The specific role of inhibition in reading comprehension in good and poor comprehenders. J. Learn. Disabil. 43(6), 541–552 (2010).  https://doi.org/10.1177/0022219410371676CrossRefGoogle Scholar
  4. 4.
    Chilton, M.A., Hardgrave, B.C., Armstrong, D.J.: Person-job cognitive style fit for software developers: the effect on strain and performance. J. Manag. Inf. Syst. 22(2), 193–226 (2005).  https://doi.org/10.1080/07421222.2005.11045849CrossRefGoogle Scholar
  5. 5.
    Clair-Thompson, H.L.S., Gathercole, S.E.: Executive functions and achievements in school: shifting, updating, inhibition, and working memory. Q. J. Exp. Psychol. 59(4), 745–759 (2006).  https://doi.org/10.1080/17470210500162854CrossRefGoogle Scholar
  6. 6.
    Crump, M.J.C., McDonnell, J.V., Gureckis, T.M.: Evaluating Amazon’s mechanical Turk as a tool for experimental behavioral research. PLoS ONE 8(3), 1–18 (2013).  https://doi.org/10.1371/journal.pone.0057410CrossRefGoogle Scholar
  7. 7.
    Damerau, F.J.: A technique for computer detection and correction of spelling errors. Commun. ACM 7(3), 171–176 (1964)CrossRefGoogle Scholar
  8. 8.
    Deshpande, M., Karypis, G.: Item-based Top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1), 143–177 (2004).  https://doi.org/10.1145/963770.963776CrossRefGoogle Scholar
  9. 9.
    Diamond, A.: Executive functions. Annu. Rev. Psychol. 64(1), 135–168 (2013).  https://doi.org/10.1146/annurev-psych-113011-143750CrossRefGoogle Scholar
  10. 10.
    Difallah, D.E., Catasta, M., Demartini, G., Ipeirotis, P.G., Cudré-Mauroux, P.: The dynamics of micro-task crowdsourcing: the case of Amazon MTurk. In: Proceedings of the 24th International Conference on World Wide Web, WWW 2015, pp. 238–247. IW3C2, Switzerland (2015).  https://doi.org/10.1145/2736277.2741685
  11. 11.
    Difallah, D.E., Demartini, G., Cudré-Mauroux, P.: Pick-a-crowd: tell me what you like, and I’ll tell you what to do. In: Proceedings of the 22nd International Conference on World Wide Web, WWW 2013, pp. 367–374. ACM, New York (2013).  https://doi.org/10.1145/2488388.2488421
  12. 12.
    Dingler, T., Schmidt, A., Machulla, T.: Building cognition-aware systems: a mobile toolkit for extracting time-of-day fluctuations of cognitive performance. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(3) (2017).  https://doi.org/10.1145/3132025CrossRefGoogle Scholar
  13. 13.
    Downs, J.S., Holbrook, M.B., Sheng, S., Cranor, L.F.: Are your participants gaming the system?: Screening mechanical Turk workers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2010, pp. 2399–2402. ACM, New York (2010).  https://doi.org/10.1145/1753326.1753688
  14. 14.
    Edwards, J.R.: Person-Job Fit: A Conceptual Integration, Literature Review, and Methodological Critique. Wiley, New York (1991)Google Scholar
  15. 15.
    Eickhoff, C.: Cognitive biases in crowdsourcing. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, pp. 162–170. ACM, New York (2018).  https://doi.org/10.1145/3159652.3159654
  16. 16.
    Ekstrom, R.B., Dermen, D., Harman, H.H.: Manual for Kit of Factor-referenced Cognitive Tests, vol. 102. Educational Testing Service, Princeton (1976)Google Scholar
  17. 17.
    Eriksen, B.A., Eriksen, C.W.: Effects of noise letters upon the identification of a target letter in a nonsearch task. Percept. Psychophys. 16(1), 143–149 (1974)CrossRefGoogle Scholar
  18. 18.
    Fan, J., Li, G., Ooi, B.C., Tan, K.l., Feng, J.: iCrowd: an adaptive crowdsourcing framework. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, pp. 1015–1030. ACM, New York (2015).  https://doi.org/10.1145/2723372.2750550
  19. 19.
    Federico, P.A., Landis, D.B.: Cognitive styles, abilities, and aptitudes: are they dependent or independent? Contemp. Educ. Psychol. 9(2), 146–161 (1984).  https://doi.org/10.1016/0361-476X(84)90016-XCrossRefGoogle Scholar
  20. 20.
    Gadiraju, U., Kawase, R., Dietze, S.: A taxonomy of microtasks on the web. In: Proceedings of the 25th ACM Conference on Hypertext and Social Media, HT 2014, pp. 218–223. ACM, New York (2014)Google Scholar
  21. 21.
    Germine, L., Nakayama, K., Duchaine, B.C., Chabris, C.F., Chatterjee, G., Wilmer, J.B.: Is the web as good as the lab? Comparable performance from web and lab in cognitive/perceptual experiments. Psychon. Bull. Rev. 19(5), 847–857 (2012).  https://doi.org/10.3758/s13423-012-0296-9CrossRefGoogle Scholar
  22. 22.
    Goncalves, J., Feldman, M., Hu, S., Kostakos, V., Bernstein, A.: Task routing and assignment in crowdsourcing based on cognitive abilities. In: Proceedings of the 26th International Conference on World Wide Web, WWW 2017, pp. 1023–1031. IW3C2, Switzerland (2017).  https://doi.org/10.1145/3041021.3055128
  23. 23.
    Goncalves, J., et al.: Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp 2013, pp. 753–762 (2013).  https://doi.org/10.1145/2493432.2493481
  24. 24.
    Goncalves, J., Hosio, S., van Berkel, N., Ahmed, F., Kostakos, V.: CrowdPickUp: crowdsourcing task pickup in the wild. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1(3), 51:1–51:22 (2017).  https://doi.org/10.1145/3130916CrossRefGoogle Scholar
  25. 25.
    Goncalves, J., Hosio, S., Rogstadius, J., Karapanos, E., Kostakos, V.: Motivating participation and improving quality of contribution in ubiquitous crowdsourcing. Comput. Netw. 90(C), 34–48 (2015).  https://doi.org/10.1016/j.comnet.2015.07.002CrossRefGoogle Scholar
  26. 26.
    Gureckis, T.M., et al.: psiTurk: an open-source framework for conducting replicable behavioral experiments online. Behav. Res. Methods 48(3), 829–842 (2016).  https://doi.org/10.3758/s13428-015-0642-8CrossRefGoogle Scholar
  27. 27.
    Hair, J., Black, W., Babin, B., Anderson, R.: Multivariate Data Analysis. Prentice-Hall, Upper Saddle River (2010)Google Scholar
  28. 28.
    Han, S., Dai, P., Paritosh, P., Huynh, D.: Crowdsourcing human annotation on web page structure: infrastructure design and behavior-based quality control. ACM Trans. Intell. Syst. Technol. 7(4), 56:1–56:25 (2016)CrossRefGoogle Scholar
  29. 29.
    Hoffman, B.J., Woehr, D.J.: A quantitative review of the relationship between person-organization fit and behavioral outcomes. J. Vocat. Behav. 68(3), 389–399 (2006).  https://doi.org/10.1016/j.jvb.2005.08.003CrossRefGoogle Scholar
  30. 30.
    Hosio, S., Goncalves, J., Lehdonvirta, V., Ferreira, D., Kostakos, V.: Situated crowdsourcing using a market model. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST 2014, pp. 55–64. ACM, New York (2014).  https://doi.org/10.1145/2642918.2647362
  31. 31.
    Jain, A., Sarma, A.D., Parameswaran, A., Widom, J.: Understanding workers, developing effective tasks, and enhancing marketplace dynamics: a study of a large crowdsourcing marketplace. Proc. VLDB Endow. 10(7), 829–840 (2017). https://doi.org/10.14778/3067421.3067431CrossRefGoogle Scholar
  32. 32.
    Kazai, G.: In search of quality in crowdsourcing for search engine evaluation. In: Clough, P., et al. (eds.) ECIR 2011. LNCS, vol. 6611, pp. 165–176. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-20161-5_17CrossRefGoogle Scholar
  33. 33.
    Kazai, G., Kamps, J., Milic-Frayling, N.: Worker types and personality traits in crowdsourcing relevance labels. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM 2011, pp. 1941–1944. ACM, New York (2011).  https://doi.org/10.1145/2063576.2063860
  34. 34.
    Kazai, G., Kamps, J., Milic-Frayling, N.: The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM 2012, pp. 2583–2586. ACM, New York (2012).  https://doi.org/10.1145/2396761.2398697
  35. 35.
    Kazai, G., Zitouni, I.: Quality management in crowdsourcing using gold judges behavior. In: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, WSDM 2016, pp. 267–276. ACM, New York (2016).  https://doi.org/10.1145/2835776.2835835
  36. 36.
    Kittur, A., et al.: The future of crowd work. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW 2013, pp. 1301–1318. ACM, New York (2013).  https://doi.org/10.1145/2441776.2441923
  37. 37.
    Kristof, A.L.: Person-organization fit: an integrative review of its conceptualizations, measurement, and implications. Pers. Psychol. 49(1), 1–49 (1996)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Kristof-Brown, A.L., Zimmerman, R.D., Johnson, E.C.: Consequences of individuals’ fit at work: a meta-analysis of person-job, person-organization, person-group, and person-supervisor fit. Pers. Psychol. 58(2), 281–342 (2005)CrossRefGoogle Scholar
  39. 39.
    de Leeuw, J.R.: jsPsych: a javascript library for creating behavioral experiments in a web browser. Behav. Res. Methods 47(1), 1–12 (2015)CrossRefGoogle Scholar
  40. 40.
    Liu, X., Lu, M., Ooi, B.C., Shen, Y., Wu, S., Zhang, M.: CDAS: a crowdsourcing data analytics system. Proc. VLDB Endow. 5(10), 1040–1051 (2012)CrossRefGoogle Scholar
  41. 41.
    Lykourentzou, I., Antoniou, A., Naudet, Y., Dow, S.P.: Personality matters: balancing for personality types leads to better outcomes for crowd teams. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW 2016, pp. 260–273. ACM, New York (2016)Google Scholar
  42. 42.
    MacLeod, C.M.: Half a century of research on the stroop effect: an integrative review. Psychol. Bull. 109(2), 163 (1991)CrossRefGoogle Scholar
  43. 43.
    Mavridis, P., Gross-Amblard, D., Miklós, Z.: Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing. In: Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Switzerland, pp. 843–853. IW3C2 (2016).  https://doi.org/10.1145/2872427.2883070
  44. 44.
    McInnis, B., Cosley, D., Nam, C., Leshed, G.: Taking a hit: designing around rejection, mistrust, risk, and workers’ experiences in Amazon mechanical Turk. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 2016, pp. 2271–2282. ACM, New York (2016)Google Scholar
  45. 45.
    Mioshi, E., Dawson, K., Mitchell, J., Arnold, R., Hodges, J.R.: The Addenbrooke’s cognitive examination revised (ACE-R): a brief cognitive test battery for dementia screening. Int. J. Geriatr. Psychiatry 21(11), 1078–1085 (2006)CrossRefGoogle Scholar
  46. 46.
    Mo, K., Zhong, E., Yang, Q.: Cross-task crowdsourcing. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, pp. 677–685. ACM, New York (2013)Google Scholar
  47. 47.
    Monsell, S.: Task switching. Trends Cogn. Sci. 7(3), 134–140 (2003)CrossRefGoogle Scholar
  48. 48.
    Owen, A.M., et al.: Putting brain training to the test. Nature 465, 775 (2010). https://doi.org/10.0.4.14/nature09042CrossRefGoogle Scholar
  49. 49.
    Owen, A.M., McMillan, K.M., Laird, A.R., Bullmore, E.: N-back working memory paradigm: a meta-analysis of normative functional neuroimaging studies. Hum. Brain Mapp. 25(1), 46–59 (2005).  https://doi.org/10.1002/hbm.20131CrossRefGoogle Scholar
  50. 50.
    Petrides, M., Alivisatos, B., Evans, A.C., Meyer, E.: Dissociation of human mid-dorsolateral from posterior dorsolateral frontal cortex in memory processing. Proc. Natl. Acad. Sci. 90(3), 873–877 (1993)CrossRefGoogle Scholar
  51. 51.
    Robbins, T.W., James, M., Owen, A.M., Sahakian, B.J., McInnes, L., Rabbitt, P.: Cambridge Neuropsychological Test Automated Battery (CANTAB): a factor analytic study of a large sample of normal elderly volunteers. Dement. Geriatr. Cogn. Disord. 5(5), 266–281 (1994).  https://doi.org/10.1159/000106735CrossRefGoogle Scholar
  52. 52.
    Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., Vukovic, M.: An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In: Proceedings of the Fifth International AAAI Conference on Web and Social Media, ICWSM, California, USA, vol. 11, pp. 17–21. AAAI (2011)Google Scholar
  53. 53.
    Ruble, T.L., Cosier, R.A.: Effects of cognitive styles and decision setting on performance. Organ. Behav. Hum. Decis. Process. 46(2), 283–295 (1990).  https://doi.org/10.1016/0749-5978(90)90033-6CrossRefGoogle Scholar
  54. 54.
    Rzeszotarski, J.M., Kittur, A.: Instrumenting the crowd: using implicit behavioral measures to predict task performance. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST 2011, pp. 13–22. ACM, New York (2011).  https://doi.org/10.1145/2047196.2047199
  55. 55.
    Schmidt, F.L., Hunter, J.: General mental ability in the world of work: occupational attainment and job performance. J. Pers. Soc. Psychol. 86(1), 162 (2004)CrossRefGoogle Scholar
  56. 56.
    Shaw, A.D., Horton, J.J., Chen, D.L.: Designing incentives for inexpert human raters. In: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, CSCW 2011, pp. 275–284. ACM, New York (2011)Google Scholar
  57. 57.
    Verquer, M.L., Beehr, T.A., Wagner, S.H.: A meta-analysis of relations between person-organization fit and work attitudes. J. Vocat. Behav. 63(3), 473–489 (2003).  https://doi.org/10.1016/S0001-8791(02)00036-2CrossRefGoogle Scholar
  58. 58.
    Washington, G.: George Washington papers, series 5, financial papers: Copybook of invoices and letters, 1754-1766 (1766). https://www.loc.gov/item/mgw500003
  59. 59.
    West, R.F., Toplak, M.E., Stanovich, K.E.: Heuristics and biases as measures of critical thinking: associations with cognitive ability and thinking dispositions. J. Educ. Psychol. 100(4), 930 (2008)CrossRefGoogle Scholar
  60. 60.
    Zheng, Y., Wang, J., Li, G., Cheng, R., Feng, J.: QASCA: a quality-aware task assignment system for crowdsourcing applications. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, pp. 1031–1046. ACM, New York (2015)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  1. 1.School of Computing and Information SystemsThe University of MelbourneMelbourneAustralia
  2. 2.Center for Ubiquitous ComputingUniversity of OuluOuluFinland

Personalised recommendations