Utilizing crowdsourcing and machine learning in education: Literature review

  • Hadeel S. AleneziEmail author
  • Maha H. Faisal


For many years, learning continues to be a vital developing field since it is the key measure of the world’s civilization and evolution with its enormous effect on both individuals and societies. Enhancing existing learning activities in general will have a significant impact on literacy rates around the world. One of the crucial activities in education is the assessment method because it is the primary way used to evaluate the student during their studies. The main purpose of this review is to examine the existing learning and e-learning approaches that use either crowdsourcing, machine learning, or both crowdsourcing and machine learning in their proposed solutions. This review will also investigate the addressed applications to identify the existing researches related to the assessment. Identifying all existing applications will assist in finding the unexplored gaps and limitations. This study presents a systematic literature review investigating 30 papers from the following databases: IEEE and ACM Digital Library. After performing the analysis, we found that crowdsourcing is utilized in 47.8% of the investigated learning activities, while each of the machine learning and the hybrid solutions are utilized in 26% of the investigated learning activities. Furthermore, all the existing approaches regarding the exam assessment problem that are using machine learning or crowdsourcing were identified. Some of the existing assessment systems are using the crowdsourcing approach and other systems are using the machine learning, however, none of the approaches provide a hybrid assessment system that uses both crowdsourcing and machine learning. Finally, it is found that using either crowdsourcing or machine learning in the online courses will enhance the interactions between the students. It is concluded that the current learning activities need to be enhanced since it is directly affecting the student’s performance. Moreover, merging both the machine learning to the crowd wisdom will increase the accuracy and the efficiency of education.


Machine learning Crowdsourcing Education E-learning 



  1. Alghamdi, E. A., Aljohani, N. R., Alsaleh, A. N., Bedewi, W., & Basheri, M. (2015). CrowdyQ: A virtual crowdsourcing platform for question items development in higher education. Proceedings of the 17th International Conference on Information Integration and Web-Based Applications &Services - IiWAS ‘15, 1–4.
  2. Barbosa, C. E., Epelbaum, V. J., Antelio, M., Oliveira, J., & Moreira de Souza, J. (2013). Crowdsourcing environments in E-learning scenario: A classification based on educational and collaboration criteria. 2013 IEEE International Conference on Systems, Man, and Cybernetics, 687–692.
  3. Choi, J.-R., Kim, S., & Lim, S.-B. (2017). The feedback block model for an adaptive E-book. Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST ‘17, 127–128.
  4. Corrigan-Gibbs, H., Gupta, N., Northcutt, C., Cutrell, E., & Thies, W. (2015). Deterring cheating in online environments. ACM Transactions on Computer-Human Interaction, 22(6), 1–23. Scholar
  5. Crooks, T. J. (1988). The impact of classroom evaluation practices on students. Review of Educational Research, 58(4), 438–481. Scholar
  6. de Alfaro, L., & Shavlovsky, M. (2014). CrowdGrader: A tool for crowdsourcing the evaluation of homework assignments. Proceedings of the 45th ACM Technical Symposium on Computer Science Education - SIGCSE ‘14, 415–420.
  7. Dow, S., Gerber, E., & Wong, A. (2013). A pilot study of using crowds in the classroom. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ‘13, 227.
  8. Faisal, M. H., AlAmeeri, A. W., & Alsumait, A. A. (2015). An adaptive e-learning framework: Crowdsourcing approach. Proceedings of the 17th International Conference on Information Integration and Web-Based Applications &Services - IiWAS ‘15, 1–5.
  9. Farasat, A., Nikolaev, A., Miller, S., & Gopalsamy, R. (2017). CROWDLEARNING: Towards collaborative problem-posing at scale. Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale - L@S ‘17, 221–224.
  10. Gutierrez, F., Dou, D., Martini, A., Fickas, S., & Zong, H. (2013). Hybrid ontology-based information extraction for automated text grading. 2013 12th International Conference on Machine Learning and Applications, 359–364.
  11. Harlen, W., Crick, R. D., Broadfoot, P., Daugherty, R., Gardner, J., James, M., & Stobart, G. (2002). A systematic review of the impact of summative assessment and tests on students’ motivation for learning. EPPI-Centre, University of London, 151.Google Scholar
  12. Huang, S.-W., Tu, P.-F., Fu, W.-T., & Amamzadeh, M. (2013). Leveraging the crowd to improve feature-sentiment analysis of user reviews. Proceedings of the 2013 International Conference on Intelligent User Interfaces - IUI ‘13, 3.
  13. Kamar, E. (2016). Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence. In IJCAI, 4070-4073.Google Scholar
  14. Kamath, A., Biswas, A., & Balasubramanian, V. (2016). A crowdsourced approach to student engagement recognition in e-learning environments. 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), 1–9.
  15. Kashi, A., Shastri, S., Deshpande, A. R., Doreswamy, J., & Srinivasa, G. (2016). A score recommendation system towards automating assessment in professional courses. 2016 IEEE Eighth International Conference on Technology for Education (T4E), 140–143.
  16. Kim, J., Sterman, S., Cohen, A. A. B., & Bernstein, M. S. (2017). Mechanical novel: Crowdsourcing complex work through reflection and revision. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW ‘17, 233–245.
  17. Kotturi, Y., Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. (2015). Structure and messaging techniques for online peer learning systems that increase stickiness. Proceedings of the Second (2015) ACM Conference on Learning @ Scale - L@S ‘15, 31–38.
  18. Krause, M., Garncarz, T., Song, J., Gerber, E. M., Bailey, B. P., & Dow, S. P. (2017). Critique style guide: Improving Crowdsourced design feedback with a natural language model. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ‘17, 4627–4639.
  19. Kulkarni, C., Cambre, J., Kotturi, Y., Bernstein, M. S., & Klemmer, S. R. (2015a). Talkabout: Making Distance matter with small groups in massive classes. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW ‘15, 1116–1128.
  20. Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015b). PeerStudio: Rapid peer feedback emphasizes revision and improves performance. Proceedings of the Second (2015) ACM Conference on Learning @ Scale - L@S ‘15, 75–84.
  21. Le, L. T., Shah, C., & Choi, E. (2016). Evaluating the quality of educational answers in community question-answering. Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries - JCDL ‘16, 129–138.
  22. Lemley, D., Sudweeks, R., Howell, S., Laws, R. D., & Sawyer, O. (2007). The effects of immediate and delayed feedback on secondary distance learners. Quarterly Review of Distance Education, 8(3), 251.Google Scholar
  23. Li, Y. (2018). An application of EDM: Design of a new Online System for correcting exam paper. 2018 13th International Conference on Computer Science & Education (ICCSE), 1–6.
  24. Li, X., Chang, K., Yuan, Y., & Hauptmann, A. (2015). Massive open online proctor: Protecting the credibility of MOOCs certificates. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW ‘15, 1129–1137.
  25. Luger, S. K. K., & Bowles, J. (2013). An analysis of question quality and user performance in crowdsourced exams. Proceedings of the 2103 Workshop on Data-Driven User Behavioral Modelling and Mining from Social Media - DUBMOD ‘13, 29–32.
  26. Nkoana, T. H. (2016). E-learning: Crowdsourcing as an alternative model to traditional learning. 2016 International Conference on Advances in Computing and Communication Engineering (ICACCE), 423–428.
  27. Oxford Dictionary [Dictionary]. (2020). Retrieved January 1, 2020, from Lexico website: Accessed 1 Jan 2020
  28. Pirttinen, N., Kangas, V., Nikkarinen, I., Nygren, H., Leinonen, J., & Hellas, A. (2018). Crowdsourcing programming assignments with CrowdSorcerer. Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education - ITiCSE 2018, 326–331.
  29. Poetz, M. K., & Schreier, M. (2012). The value of crowdsourcing: Can users really compete with professionals in generating new product ideas?: The value of crowdsourcing. Journal of Product Innovation Management, 29(2), 245–256. Scholar
  30. Schenk, E., & Guittard, C. (2009). Crowdsourcing: What can be Outsourced to the Crowd, and Why. In Workshop on open source innovation, Strasbourg, France 72, 3.Google Scholar
  31. Shetty, B. (2018). Natural Language Processing(NLP) for machine learning. Retrieved November 24, 2018, from towards data science website: Accessed 24 Sept 2019
  32. Shongwe, T., & Zuva, T. (2018). Benefits of crowdsourcing and its growth in the education sector. 2018 International Conference on Advances in Big Data, Computing and Data Communication Systems (IcABCD), 1–6.
  33. Tang, T., Smith, R., Rixner, S., & Warren, J. (2016). Data-driven test case generation for automated programming assessment. Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education - ITiCSE ‘16, 260–265.
  34. Taniguchi, A., & Inoue, S. (2015). A method for automatic assessment of user-generated tests and its evaluation. Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers - UbiComp ‘15, 225–228.
  35. Vaish, R., Goel, S., Davis, J., Bernstein, M. S., Gaikwad, S. (Neil) S., Kovacs, G., … Belongie, S. (2017). Crowd research: Open and scalable university laboratories. Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST ‘17, 829–843.
  36. Vaughan, J. W. (2017). Making better use of the crowd: How crowdsourcing can advance machine learning research. Journal of Machine Learning Research, 18, 193-1.Google Scholar
  37. Watson, P., Ma, T., Tejwani, R., Chang, M., Ahn, J., & Sundararajan, S. (2018). Human-level multiple choice question guessing without domain knowledge: Machine-learning of framing Effects. Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW ‘18, 299–303.
  38. Whitehill, J., & Seltzer, M. (2016). A crowdsourcing approach to collecting tutorial videos—Toward personalized learning-at-scale. ArXiv:1606.09610 [Cs]. Retrieved from Accessed 15 Aug 2019
  39. Williams, J. J., Kim, J., Rafferty, A., Maldonado, S., Gajos, K. Z., Lasecki, W. S., & Heffernan, N. (2016). AXIS: Generating explanations at scale with Learnersourcing and machine learning. Proceedings of the Third (2016) ACM Conference on Learning @ Scale - L@S ‘16, 379–388.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2020

Authors and Affiliations

  1. 1.Department of Computer EngineeringKuwait UniversityKuwait CityKuwait

Personalised recommendations