Skip to main content
SpringerLink
Log in
Book cover

International Conference on Artificial Intelligence in Education

AIED 2021: Artificial Intelligence in Education pp 150–164Cite as

An Approach for Detecting Student Perceptions of the Programming Experience from Interaction Log Data

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12748)

Abstract

Student perceptions of programming can impact their experiences in introductory computer science (CS) courses. For example, some students negatively assess their own ability in response to moments that are natural parts of expert practice, such as using online resources or getting syntax errors. Systems that automatically detect these moments from interaction log data could help us study these moments and intervene when the occur. However, while researchers have analyzed programming log data, few systems detect pre-defined moments, particularly those based on student perceptions. We contribute a new approach and system for detecting programming moments that students perceive as important from interaction log data. We conducted retrospective interviews with 41 CS students in which they identified moments that can prompt negative self-assessments. Then we created a qualitative codebook of the behavioral patterns indicative of each moment, and used this knowledge to build an expert system. We evaluated our system with log data collected from an additional 33 CS students. Our results are promising, with F1 scores ranging from 66% to 98%. We believe that this approach can be applied in many domains to understand and detect student perceptions of learning experiences.

Keywords

  • CS education
  • Detection systems
  • Self-assessment

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Ahadi, A., Lister, R., Haapala, H., Vihavainen, A.: Exploring machine learning methods to automatically identify students in need of assistance. In: Proceedings of the Eleventh Annual International Conference on International Computing Education Research - ICER 2015, pp. 121–130. ACM Press, Omaha (2015). https://doi.org/10.1145/2787622.2787717. http://dl.acm.org/citation.cfm?doid=2787622.2787717

  2. Anderson, J.R., Conrad, F.G., Corbett, A.T.: Skill acquisition and the LISP tutor. Cogn. Sci. 13(4), 467–505 (1989)

    CrossRef  Google Scholar 

  3. Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167–207 (1995)

    CrossRef  Google Scholar 

  4. Bandura, A.: Self-efficacy mechanism in human agency. Am. Psychol. 37(2), 122 (1982)

    CrossRef  Google Scholar 

  5. Bandura, A.: Self-Efficacy: The Exercise of Control. Macmillan, New York (1997)

    Google Scholar 

  6. Bandura, A.: Self-efficacy. In: The Corsini Encyclopedia of Psychology, pp. 1–3. Wiley Online Library (2010)

    Google Scholar 

  7. Berland, M., Martin, T., Benton, T., Petrick Smith, C., Davis, D.: Using learning analytics to understand the learning pathways of novice programmers. J. Learn. Sci. 22(4), 564–599 (2013)

    CrossRef  Google Scholar 

  8. Blikstein, P.: Using learning analytics to assess students’ behavior in open-ended programming tasks. In: Proceedings of the 1st International Conference on Learning Analytics and Knowledge, pp. 110–116. ACM (2011). http://dl.acm.org/citation.cfm?id=2090132

  9. Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., Koller, D.: Programming pluralism: using learning analytics to detect patterns in the learning of computer programming. J. Learn. Sci. 23(4), 561–599 (2014)

    CrossRef  Google Scholar 

  10. Corbett, A.: Cognitive computer tutors: solving the two-sigma problem. In: Bauer, M., Gmytrasiewicz, P.J., Vassileva, J. (eds.) UM 2001. LNCS (LNAI), vol. 2109, pp. 137–147. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44566-8_14

    CrossRef  Google Scholar 

  11. Corbett, A.T., Anderson, J.R.: Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 245–252. ACM (2001)

    Google Scholar 

  12. Cross, J.H., Hendrix, D., Umphress, D.A.: JGRASP: an integrated development environment with visualizations for teaching Java in CS1, CS2, and beyond. In: 34th Annual Frontiers in Education, FIE 2004, pp. 1466–1467. IEEE Computer Society (2004)

    Google Scholar 

  13. Cutts, Q., Cutts, E., Draper, S., O’Donnell, P., Saffrey, P.: Manipulating mindset to positively influence introductory programming performance. In: Proceedings of the 41st ACM Technical Symposium on Computer Science Education, pp. 431–435. ACM (2010). http://dl.acm.org/citation.cfm?id=1734409

  14. Edwards, S., Li, Z.: Towards progress indicators for measuring student programming effort during solution development. In: Proceedings of the 16th Koli Calling International Conference on Computing Education Research - Koli Calling 2016, pp. 31–40. ACM Press, Koli (2016). https://doi.org/10.1145/2999541.2999561. http://dl.acm.org/citation.cfm?doid=2999541.2999561

  15. Ehrlinger, J., Dunning, D.: How chronic self-views influence (and potentially mislead) estimates of performance. J. Pers. Soc. Psychol. 84(1), 5 (2003)

    CrossRef  Google Scholar 

  16. Ericsson, K.A., Simon, H.A.: Protocol Analysis: Verbal Reports as Data. MIT Press, Cambridge (1984)

    Google Scholar 

  17. Felleisen, M., Findler, R.B., Flatt, M., Krishnamurthi, S.: How to Design Programs. MIT Press, Cambridge (2001)

    MATH  Google Scholar 

  18. Fisher, A., Margolis, J.: Unlocking the clubhouse: the Carnegie Mellon experience. ACM SIGCSE Bull. 34(2), 79–83 (2002). http://dl.acm.org/citation.cfm?id=543836

  19. Flanigan, A.E., Peteranetz, M.S., Shell, D.F., Soh, L.K.: Exploring changes in computer science students’ implicit theories of intelligence across the semester. In: Proceedings of the Eleventh Annual International Conference on International Computing Education Research, ICER 2015, Omaha, Nebraska, USA, pp. 161–168. Association for Computing Machinery, New York (2015). https://doi.org/10.1145/2787622.2787722

  20. Fuchs, M., Heckner, M., Raab, F., Wolff, C.: Monitoring students’ mobile app coding behavior data analysis based on IDE and browser interaction logs. In: 2014 IEEE Global Engineering Education Conference (EDUCON). pp. 892–899. IEEE, Istanbul, April 2014. https://doi.org/10.1109/EDUCON.2014.6826202. http://ieeexplore.ieee.org/document/6826202/

  21. Gorson, J., O’Rourke, E.: How do students talk about intelligence? An investigation of motivation, self-efficacy, and mindsets in computer science. In: Proceedings of the 2019 ACM Conference on International Computing Education Research - ICER 2019, pp. 21–29. ACM Press, Toronto ON, Canada (2019). https://doi.org/10.1145/3291279.3339413. http://dl.acm.org/citation.cfm?doid=3291279.3339413

  22. Gorson, J., O’Rourke, E.: Why do CS1 Students Think They’re Bad at Programming? Investigating Self-efficacy and Self-assessments at Three Universities. In: Proceedings of the 2020 ACM Conference on International Computing Education Research, pp. 170–181 (2020)

    Google Scholar 

  23. Jadud, M.C.: Methods and tools for exploring novice compilation behaviour. In: Proceedings of the Second International Workshop on Computing Education Research, Canterbury, United Kingdom, ICER 2006, pp. 73–84. Association for Computing Machinery, New York (2006). https://doi.org/10.1145/1151588.1151600

  24. Kinnunen, P., Simon, B.: Experiencing programming assignments in CS1: the emotional toll. In: Proceedings of the Sixth International Workshop on Computing Education Research, pp. 77–86. ACM (2010)

    Google Scholar 

  25. Kinnunen, P., Simon, B.: CS majors’ self-efficacy perceptions in CS1: results in light of social cognitive theory. In: Proceedings of the Seventh International Workshop on Computing Education Research, pp. 19–26. ACM (2011)

    Google Scholar 

  26. Kinnunen, P., Simon, B.: My program is ok - am I? Computing freshmen’s experiences of doing programming assignments. Comput. Sci. Educ. 22(1), 1–28 (2012). https://doi.org/10.1080/08993408.2012.655091. http://www.tandfonline.com/doi/abs/10.1080/08993408.2012.655091

  27. Köksal, M.F., Başar, R., Üsküdarlı, S.: Screen-replay: a session recording and analysis tool for DrScheme. In: Proceedings of the Scheme and Functional Programming Workshop, Technical Report, California Polytechnic State University, CPSLO-CSC-09, vol. 3, pp. 103–110. Citeseer (2009)

    Google Scholar 

  28. LaToza, T.D., Venolia, G., DeLine, R.: Maintaining mental models: a study of developer work habits. In: Proceeding of the 28th International Conference on Software Engineering - ICSE 2006, p. 492. ACM Press, Shanghai (2006). https://doi.org/10.1145/1134285.1134355. http://portal.acm.org/citation.cfm?doid=1134285.1134355

  29. Lewis, C., Bruno, P., Raygoza, J., Wang, J.: Alignment of goals and perceptions of computing predicts students’ sense of belonging in computing. In: Proceedings of the 2019 ACM Conference on International Computing Education Research - ICER 2019, pp. 11–19. ACM Press, Toronto (2019). https://doi.org/10.1145/3291279.3339426. http://dl.acm.org/citation.cfm?doid=3291279.3339426

  30. Lewis, C.M., Anderson, R.E., Yasuhara, K.: “I Don’t Code All Day”: fitting in computer science when the stereotypes don’t fit. In: Proceedings of the 2016 ACM Conference on International Computing Education Research, ICER 2016, Melbourne, VIC, Australia, pp. 23–32. Association for Computing Machinery, New York (2016). https://doi.org/10.1145/2960310.2960332

  31. Lewis, C.M., Yasuhara, K., Anderson, R.E.: Deciding to major in computer science: a grounded theory of students’ self-assessment of ability. In: Proceedings of the Seventh International Workshop on Computing Education Research, pp. 3–10. ACM (2011)

    Google Scholar 

  32. Lishinski, A., Yadav, A., Enbody, R.: Students’ emotional reactions to programming projects in introduction to programming: measurement approach and influence on learning outcomes. In: Proceedings of the 2017 ACM Conference on International Computing Education Research - ICER 2017, pp. 30–38. ACM Press, Tacoma, Washington (2017). https://doi.org/10.1145/3105726.3106187. http://dl.acm.org/citation.cfm?doid=3105726.3106187

  33. Lishinski, A., Yadav, A., Good, J., Enbody, R.: Learning to program: gender differences and interactive effects of students’ motivation, goals, and self-efficacy on performance. In: Proceedings of the 2016 ACM Conference on International Computing Education Research, pp. 211–220. ACM Press (2016). https://doi.org/10.1145/2960310.2960329. http://dl.acm.org/citation.cfm?doid=2960310.2960329

  34. Loksa, D., Ko, A.J., Jernigan, W., Oleson, A., Mendez, C.J., Burnett, M.M.: Programming, problem solving, and self-awareness: effects of explicit guidance. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 1449–1461. ACM Press (2016). https://doi.org/10.1145/2858036.2858252. http://dl.acm.org/citation.cfm?doid=2858036.2858252

  35. Marwan, S., Gao, G., Fisk, S., Price, T.W., Barnes, T.: Adaptive immediate feedback can improve novice programming engagement and intention to persist in computer science. In: Proceedings of the 2020 ACM Conference on International Computing Education Research, pp. 194–203. ACM, Virtual Event New Zealand, August 2020. https://doi.org/10.1145/3372782.3406264. https://dl.acm.org/doi/10.1145/3372782.3406264

  36. Master, A., Cheryan, S., Meltzoff, A.N.: Computing whether she belongs: stereotypes undermine girls’ interest and sense of belonging in computer science. J. Educ. Psychol. 108(3), 424 (2016)

    CrossRef  Google Scholar 

  37. Miura, I.T.: The relationship of computer self-efficacy expectations to computer interest and course enrollment in college. Sex Roles 16(5–6), 303–311 (1987). https://doi.org/10.1007/BF00289956. http://link.springer.com/10.1007/BF00289956

  38. Munson, J.P., Zitovsky, J.P.: Models for early identification of struggling novice programmers. In: Proceedings of the 49th ACM Technical Symposium on Computer Science Education, pp. 699–704 (2018)

    Google Scholar 

  39. O’Rourke, E., Haimovitz, K., Ballweber, C., Dweck, C., Popović, Z.: Brain points: a growth mindset incentive structure boosts persistence in an educational game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3339–3348. ACM (2014)

    Google Scholar 

  40. O’Reilly, M., Parker, N.: ‘Unsatisfactory Saturation’: a critical exploration of the notion of saturated sample sizes in qualitative research. Qual. Res. 13(2), 190–197 (2013). https://doi.org/10.1177/1468794112446106. http://journals.sagepub.com/doi/10.1177/1468794112446106

  41. Perscheid, M., Siegmund, B., Taeumel, M., Hirschfeld, R.: Studying the advancement in debugging practice of professional software developers. Softw. Qual. J. 25(1), 83–110 (2017). https://doi.org/10.1007/s11219-015-9294-2. http://link.springer.com/10.1007/s11219-015-9294-2

  42. Piech, C., Sahami, M., Koller, D., Cooper, S., Blikstein, P.: Modeling how students learn to program. In: Proceedings of the 43rd ACM Technical Symposium on Computer Science Education, pp. 153–160. ACM (2012)

    Google Scholar 

  43. Ramalingam, V., LaBelle, D., Wiedenbeck, S.: Self-efficacy and mental models in learning to program. In: ACM SIGCSE Bulletin, vol. 36, pp. 171–175. ACM (2004)

    Google Scholar 

  44. Reiser, B.J., Anderson, J.R., Farrell, R.G.: Dynamic student modelling in an intelligent tutor for LISP programming. IJCAI 85, 8–14 (1985)

    Google Scholar 

  45. Relich, J.D., Debus, R.L., Walker, R.: The mediating role of attribution and self-efficacy variables for treatment effects on achievement outcomes. Contemp. Educ. Psychol. 11(3), 195–216 (1986)

    CrossRef  Google Scholar 

  46. Saunders, B., et al.: Saturation in qualitative research: exploring its conceptualization and operationalization. Qual. Quant. 52(4), 1893–1907 (2018). https://doi.org/10.1007/s11135-017-0574-8. http://link.springer.com/10.1007/s11135-017-0574-8

  47. Schunk, D.H.: Self-efficacy, motivation, and performance. J. Appl. Sport Psychol. 7(2), 112–137 (1995). https://doi.org/10.1080/10413209508406961. https://www.tandfonline.com/doi/full/10.1080/10413209508406961

  48. Shapiro, J.R., Williams, A.M.: The role of stereotype threats in undermining girls’ and women’s performance and interest in STEM fields. Sex Roles 66(3–4), 175–183 (2012). https://doi.org/10.1007/s11199-011-0051-0. http://link.springer.com/10.1007/s11199-011-0051-0

  49. Shenton, A.K.: Strategies for ensuring trustworthiness in qualitative research projects. Educ. Inf. 22(2), 63–75 (2004). https://doi.org/10.3233/EFI-2004-22201. https://www.medra.org/servlet/aliasResolver?alias=iospress&doi=10.3233/EFI-2004-22201

  50. Simard, P.Y., et al.: Machine teaching: a new paradigm for building machine learning systems. arXiv preprint arXiv:1707.06742 (2017)

  51. Soloway, E., Bonar, J., Ehrlich, K.: Cognitive strategies and looping constructs: an empirical study. Commu. ACM 26(11), 853–860 (1983). https://doi.org/10.1145/182.358436. https://dl.acm.org/doi/10.1145/182.358436

  52. Sonnentag, S.: Expertise in professional software design: a process study. J. Appl. Psychol. 83(5), 703 (1998)

    CrossRef  Google Scholar 

  53. Steele, C.M., Aronson, J.: Stereotype threat and the intellectual test performance of African Americans. J. Pers. Soc. Psychol. 69(5), 797 (1995)

    CrossRef  Google Scholar 

  54. Veilleux, N., Bates, R., Allendoerfer, C., Jones, D., Crawford, J., Floyd Smith, T.: The relationship between belonging and ability in computer science. In: Proceedings of the 44th ACM Technical Symposium on Computer Science Education, pp. 65–70. ACM (2013)

    Google Scholar 

  55. Watson, C., Li, F.W., Godwin, J.L.: No tests required: comparing traditional and dynamic predictors of programming success. In: Proceedings of the 45th ACM Technical Symposium on Computer Science Education, pp. 469–474 (2014)

    Google Scholar 

  56. Zadeh, L.A.: Fuzzy logic. Computer 21(4), 83–93 (1988)

    CrossRef  Google Scholar 

Download references

Acknowledgement

This work was supported by NSF Grant IIS-1755628. Thank you to Delta Lab.

Author information

Authors and Affiliations

  1. Northwestern University, Evanston, IL, USA

    Jamie Gorson, Nicholas LaGrassa, Cindy Hsinyu Hu, Elise Lee, Ava Marie Robinson & Eleanor O’Rourke

Authors
  1. Jamie Gorson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicholas LaGrassa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cindy Hsinyu Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Elise Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ava Marie Robinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Eleanor O’Rourke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jamie Gorson .

Editor information

Editors and Affiliations

  1. Technion – Israel Institute of Technology, Haifa, Israel

    Ido Roll

  2. Arizona State University, Tempe, AZ, USA

    Danielle McNamara

  3. Utrecht University, Utrecht, The Netherlands

    Sergey Sosnovsky

  4. London Knowledge Lab, London, UK

    Rose Luckin

  5. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gorson, J., LaGrassa, N., Hu, C.H., Lee, E., Robinson, A.M., O’Rourke, E. (2021). An Approach for Detecting Student Perceptions of the Programming Experience from Interaction Log Data. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2021. Lecture Notes in Computer Science(), vol 12748. Springer, Cham. https://doi.org/10.1007/978-3-030-78292-4_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-78292-4_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78291-7

  • Online ISBN: 978-3-030-78292-4

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation