Advertisement

Process Mining of Interactions During Computer-Based Testing for Detecting and Modelling Guessing Behavior

  • Zacharoula Papamitsiou
  • Anastasios A. Economides
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9753)

Abstract

Detecting, recognizing and modelling patterns of observed examinee behaviors during assessment is a topic of great interest for the educational research community. In this paper we investigate the perspectives of process-centric inference of guessing behavior patterns. The underlying idea is to extract knowledge from real processes (i.e., not assumed nor truncated), logged automatically by the assessment environment. We applied a three-step process mining methodology on logged interaction traces from a case study with 259 undergraduate university students. The analysis revealed sequences of interactions in which low goal-orientation students answered quickly and correctly on difficult items, without reviewing them, while they submitted wrong answers on easier items. We assumed that this implies guessing behavior. From the conformance checking and performance analysis we found that the fitness of our process model is almost 85 %. Hence, initial results are encouraging towards modelling guessing behavior. Potential implications and future work plans are also discussed.

Keywords

Assessment analytics Educational data mining Guessing behavior Pattern recognition Process mining Student interaction analysis 

References

  1. 1.
    Papamitsiou, Z., Economides, A.A.: Learning analytics and educational data mining in practice: a systematic literature review of empirical evidence. Educ. Technol. Soc. 17(4), 49–64 (2014)Google Scholar
  2. 2.
    Jordan, B., Henderson, A.: Interaction analysis: foundations and practice. J. Learn. Sci. 4(1), 39–103 (1995)CrossRefGoogle Scholar
  3. 3.
    Donnelly, R.: Interaction analysis in a “Learning by Doing” problem-based professional development context. Comput. Educ. 55, 1357–1366 (2010)CrossRefGoogle Scholar
  4. 4.
    Huang, S., Fang, N.: Predicting student academic performance in an engineering dynamics course: a comparison of four types of predictive mathematical models. Comput. Educ. 61, 133–145 (2013)CrossRefGoogle Scholar
  5. 5.
    Macfadyen, L.P., Dawson, S.: Mining LMS data to develop an “early warning system” for educators: a proof of concept. Comput. Educ. 54(2), 588–599 (2010)CrossRefGoogle Scholar
  6. 6.
    Papamitsiou, Z., Terzis, V., Economides, A.A.: Temporal learning analytics during computer based testing. In: 4th International Conference on Learning Analytics and Knowledge, pp. 31−35. ACM, New York (2014)Google Scholar
  7. 7.
    Lee, Y.-H., Jia, Y.: Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-scale Assessments Educ. 2(1), 1–24 (2014)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Kong, X.J., Wise, S.L., Bhola, D.S.: Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behaviour. Educ. Psychol. Meas. 67(4), 606–619 (2007)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Baker, R.S., Corbett, A.T., Gowda, S.M., Wagner, A.Z., MacLaren, B.A., Kauffman, L.R., Mitchell, A.P., Giguere, S.: Contextual slip and prediction of student performance after use of an intelligent tutor. In: De Bra, P., Kobsa, A., Chin, D. (eds.) UMAP 2010. LNCS, vol. 6075, pp. 52–63. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Schnipke, D.L., Scrams, D.J.: Modeling item response times with a two-state mixture model: a new method of measuring speededness. J. Educ. Meas. 34, 213–232 (1997)CrossRefGoogle Scholar
  11. 11.
    Wise, S.L., Kong, X.: Response time effort: a new measure of examinee motivation in computer-based tests. Appl. Meas. Educ. 16, 163–183 (2005)CrossRefGoogle Scholar
  12. 12.
    Cao, J., Stokes, S.L.: Bayesian IRT guessing models for partial guessing behaviors. Psychometrika 73(2), 209–230 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Baker, R.S.J.d., Corbett, A.T., Aleven, V.: Improving contextual models of guessing and slipping with a truncated training set. In: ‎Baker, R.S.J.d., Barnes, T., Beck, J.E. (eds.) 1st International Conference on Educational Data Mining, pp. 67−76 (2008)Google Scholar
  14. 14.
    Gowda, S.M., Rowe, J.P., Baker, R.S.J.d., Chi, M., Koedinger, K.R.: Improving models of slipping, guessing and moment-by-moment learning with estimates of skill difficulty. In: Pechenizkiy, M., Calders, T., Conati, C., Ventura, S., Romero, C., and Stamper, J. (eds.) 4th International Conference on Educational Data Mining, pp. 199−208 (2011)Google Scholar
  15. 15.
    Trčka, N., Pechenizkiy, M., van der Aalst, W.P.M.: Process mining from educational data (Chapter 9). In: Romero, C., Ventura, S., Pechenizkiy, M., Baker, R.S.J.d. (eds.) Handbook of Educational Data Mining. pp. 123−142. CRC Press, London (2010)Google Scholar
  16. 16.
    van der Aalst, W.P.M.: Process Mining: Discovery, Conformance and Enhancement of Business Processes. Springer, Berlin (2011)CrossRefzbMATHGoogle Scholar
  17. 17.
    van der Aalst, W.P.M.: Process mining: making knowledge discovery process centric. SIGKDD Explor. Newsl. 13(2), 45–49 (2012)CrossRefGoogle Scholar
  18. 18.
    van Dongen, B.F., de Medeiros, A.K.A., Verbeek, H., Weijters, A., van der Aalst, W.M.: The ProM framework: a new era in process mining tool support. In: Ciardo, G., Darondeau, P. (eds.) ICATPN 2005. LNCS, vol. 3536, pp. 444–454. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  19. 19.
    van der Aalst, W.M.: Process mining in the large: a tutorial. In: Zimányi, E. (ed.) eBISS 2013. LNBIP, vol. 172, pp. 33–76. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  20. 20.
    Bannert, M., Reimann, P., Sonnenberg, C.: Process mining techniques for analysing patterns and strategies in students’ self-regulated learning. Metacogn. Learn. 9(2), 161–185 (2014)CrossRefGoogle Scholar
  21. 21.
    Schoor, C., Bannert, M.: Exploring regulatory processes during a CSCL task using process mining. Comput. Hum. Behav. 28(4), 1321–1331 (2012)CrossRefGoogle Scholar
  22. 22.
    Bergenthum, R., Desel, J., Harrer, A., Mauser, S.: Modeling and mining of learnflows. In: Jensen, K., Donatelli, S., Kleijn, J. (eds.) ToPNoC V. LNCS, vol. 6900, pp. 22–50. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  23. 23.
    Southavilay, V., Yacef, K., Calvo, R.A.: Process mining to support students’ collaborative writing. In: Baker, R.S.J.d., Merceron, A., Pavlik, P.I.Jr. (eds.) 3rd International Conference on Educatinal Data Mining, pp. 257−266 (2010)Google Scholar
  24. 24.
    Calvo, R.A., O’Rourke, S.T., Jones, J., Yacef, K., Reimann, P.: Collaborative writing support tools on the cloud. IEEE Trans. Learn. Technol. 4(1), 88–97 (2011)CrossRefGoogle Scholar
  25. 25.
    Pechenizkiy, M., Trčka, N., Vasilyeva, E., van der Aalst, W.M.P., de Bra, P.: Process mining online assessment data. In: Barnes, T., Desmarais, M.C., Romero, C. and Ventura, S. (eds.) 2nd International Conference on Educational Data Mining, pp. 279–288 (2009)Google Scholar
  26. 26.
    Mukala, P., Buijs, J.C.A.M., Leemans, M., van der Aalst, W.M.P.: Learning analytics on Coursera event data: a process mining approach. In: Ceravolo, P., Rinderle-Ma, S. (eds.) 5th SIMPDA, pp. 18−32 (2015)Google Scholar
  27. 27.
    Sedrakyan, G., De Weerdt, J., Snoeck, M.: Process-mining enabled feedback: “tell me what I did wrong” vs. “tell me how to do it right”. Comput. Hum. Behav. 57, 352–376 (2016)CrossRefGoogle Scholar
  28. 28.
    Papamitsiou, Z., Economides, A.A.: A temporal estimation of students’ on-task mental effort and its effect on students’ performance during computer based testing. In: 18th International Conference on Interactive Collaborative Learning, pp. 1136–1144. IEEE (2015)Google Scholar
  29. 29.
    Terzis, V., Economides, A.A.: The acceptance and use of computer based assessment. Comput. Educ. 56(4), 1032–1044 (2011)CrossRefGoogle Scholar
  30. 30.
    van der Aalst, W.M.P.: The application of petri nets to workflow management. J. Circuit. Syst. Comput. 8(1), 21–66 (1998)CrossRefGoogle Scholar
  31. 31.
    Sugar, C.A., James, G.M.: Finding the number of clusters in a dataset: an information-theoretic approach. J. Am. Stat. Assoc. 98, 750–763 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    van der Werf, J.M.E.M., Dongen, B.F., van Hurkens, C.A.J., Serebrenik, A.: Process discovery using integer linear programming. Fundam. Inform. 94(3–4), 387–412 (2009)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Bereby-Meyer, Y., Meyer, J., Flascher, O.M.: Prospect theory analysis of guessing in multiple choice tests. J. Behav. Decis. Making 15(4), 313–327 (2002)CrossRefGoogle Scholar
  34. 34.
    Lamboudis, D., Economides, A.A.: Adaptive exploration of user knowledge in computer based testing. WSEAS Trans. Commun. 3(1), 322–327 (2004)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Zacharoula Papamitsiou
    • 1
  • Anastasios A. Economides
    • 1
  1. 1.IPPS in Information SystemsUniversity of MacedoniaThessalonikiGreece

Personalised recommendations