Advertisement

Learning Analytics and eAssessment—Towards Computational Psychometrics by Combining Psychometrics with Learning Analytics

Chapter
  • 456 Downloads
Part of the Lecture Notes in Educational Technology book series (LNET)

Abstract

From a psychometric point of view, assessment means to infer what a learner knows and can do in the real world from limited evidence observed in a standardized testing situation. From a learning analytics perspective assessment means to observe real behavior in digital learning environments to conclude the learner status with the intent to positively influence the learning process. Although psychometrics and learning analytics share similar goals, for instance, formative assessment, while applying different methods and theories, the two disciplines are so far highly separated. This chapter aims at paving the way for an advanced understanding of assessment by comparing and integrating the learning analytics and the psychometric approach of assessment. We will discuss means to show this new way of assessment of educational concepts such as (meta-) cognition, motivation, and reading comprehension skills that can be addressed either from data-driven approach (learning analytics) or from a theory-driven approach (psychometrics). Finally, we show that radically new ways of assessment are located in the middle space where both disciplines are combined into a new research discipline called ‘Computational Psychometrics’.

Keywords

Psychometrics Learning analytics Formative assessment Multimodal data Process data (Meta-) cognition Motivation Reading comprehension skills 

References

  1. Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2010). Assessment of self-regulation during learning with an intelligent tutoring system. Educational Psychologist, 45, 224–233.CrossRefGoogle Scholar
  2. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76.  https://doi.org/10.1016/j.lindif.2013.01.007.CrossRefGoogle Scholar
  3. Arieli-Attali, M., Ward, S., Thomas, J., Deonovic, B., & von Davier, A. A. (2019). The expanded evidence-centered design (e-ECD) for learning and assessment systems: A framework for incorporating learning goals and processes within assessment design. Frontiers in psychology, 10(853).  https://doi.org/10.3389/fpsyg.2019.00853.
  4. Arnold, K., & Pistilli, M. D. (2012). Course signals at Purdue: using learning analytics to increase student success. In S. Buckingham Shum, D. Gasevic & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK 2012) (pp. 267–270). New York, NY, USA: ACM. http://dx.doi.org/10.1145/2330601.2330666.
  5. Behrens, J. T., & DiCerbo, K. E. (2014). Harnessing the currents of the digital ocean. In J. Larussen & B. White (Eds.), Learning analytics: From research to practice (pp. 39–60). New York: Springer.CrossRefGoogle Scholar
  6. Biedermann, D., Schneider, J. & Drachsler, H. (2018). The learning analytics indicator repository. In 13th European Conference on Technology Enhanced Learning, EC-TEL 2018, Leeds, UK, September 3–5, 2018, Proceedings (Lecture Notes in Computer Science, Vol. 11082, pp. 579–582). Cham: Springer.Google Scholar
  7. Blikstein, P. (2013). Multimodal learning analytics. In Proceedings of the Third International Conference on Learning Analytics and Knowledge—LAK 2013 (pp. 102–106). New York, USA: ACM.  https://doi.org/10.1145/2460296.2460316.
  8. Cukurova, M., Kent, C., & Luckin, R. (2019). The value of multimodal data in classification of social and emotional aspects of tutoring. AIED, 2(2019), 46–51.Google Scholar
  9. DiCerbo, K. E., Shute, V., & Kim, Y. J. (2017). The future of assessment in technology rich environments: Psychometric considerations of ongoing assessment. In J. M. Spector, B. Lockee, & M. Childress (Eds.), Learning, design, and technology: An international compendium of theory, research, practice, and policy (pp. 1–21). New York, NY: Springer.Google Scholar
  10. Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018). From signals to knowledge: A conceptual model for multimodal learning analytics. Journal of Computer Assisted learning, 34, 338–349.CrossRefGoogle Scholar
  11. Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2019). Detecting mistakes in CPR training with multimodal data and neural networks. Sensors, 17, 3099.Google Scholar
  12. Drachsler, H., & Schneider, J. (2018). Special Issue on multimodal learning analytics. Journal of Computer Assisted Learning, 34.Google Scholar
  13. Echeverria, V., Martinez-Maldonado, R., Granda, R., Chiluiza, K., Conati, C., & Shum, S. B. (2018). Driving data storytelling from learning design. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge–LAK ’18 2018 (pp. 131–140).  https://doi.org/10.1145/3170358.3170380.
  14. Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317.CrossRefGoogle Scholar
  15. Goldhammer, F., & Zehner, F. (2017). What to make of and how to interpret process data. Measurement: Interdisciplinary Research and Perspectives, 15, 128–132.Google Scholar
  16. Goldhammer, F., Hahnel, C., & Kroehne, U. (2020). Analyzing log file data from PIAAC. In D. B. Maehler & B. Rammstedt (Eds.), Large-scale cognitive assessment: Analyzing PIAAC data. Cham: Springer.Google Scholar
  17. Gray, J., Chaudhuri, S., Bosworth, A., Layman, A., Reichart, D., Venkatrao, M., et al. (1997). Data cube: A relational aggregation operator generalizing group-by, cross-tab, and sub-totals. Data Mining and Knowledge Discovery, 1(1), 29–53.CrossRefGoogle Scholar
  18. Greller, W., & Drachsler, H. (2012). Translating learning into numbers. Journal of Educational Technology & Society, 15, 42–57.Google Scholar
  19. Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough. Pitfalls of learning analytics dashboards in the educational practice. In 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12–15 September 2017.Google Scholar
  20. Kroehne, U., & Goldhammer, F. (2018). How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items. Behaviormetrika, 45, 527–563.CrossRefGoogle Scholar
  21. Kosinski, M., Stillwell, D., & Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110(15), 5802–5805.  https://doi.org/10.1073/pnas.1218772110.CrossRefGoogle Scholar
  22. Mislevy, R. (2019). On integrating psychometrics and learning analytics in complex assessments. In H. Jiao, R. W. Lissitz, & A. van Wie (Eds.), Data analytics and psychometrics (pp. 1–52). Charlotte, NC, USA: Information Age Publishing.Google Scholar
  23. Mislevy, R., Behrens, J., Dicerbo, K., & Levy, R. (2012). Design and Discovery in Educational Assessment: Evidence-Centered Design, Psychometrics, and Educational Data Mining. Journal of Educational Data Mining, 4(1), 11–48.Google Scholar
  24. Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i–29.  https://doi.org/10.1002/j.2333-8504.2003.tb01908.x.CrossRefGoogle Scholar
  25. Nguyen, Q., Huptych, M., & Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance. Paper presented at the Proceedings of the 8th International Conference on Learning Analytics and Knowledge, Sydney, New South Wales, Australia.Google Scholar
  26. Ochoa, X., & Worsley, M. (2016). Augmenting Learning Analytics with Multimodal Sensory Data. Journal of Learning Analytics, 3, 213–219.  https://doi.org/10.18608/jla.2016.32.10.CrossRefGoogle Scholar
  27. OECD. (2013). OECD skills outlook 2013: First results from the survey of adult skills. Paris: OECD Publishing.CrossRefGoogle Scholar
  28. OECD. (2016). PISA 2015 results (Volume I): Excellence and equity in education. Paris: OECD Publishing.CrossRefGoogle Scholar
  29. Rienties, Bart, & Toetenel, Lisette. (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60, 333–341.CrossRefGoogle Scholar
  30. Scheffel, M., Drachsler, H., de Kraker, J., Kreijns, K., Slootmaker, A., & Specht, M. (2017). Widget, widget on the wall, am I performing well at all? IEEE Transactions on Learning Technologies, 10(1), 42–52.  https://doi.org/10.1109/TLT.2016.2622268.CrossRefGoogle Scholar
  31. Scheffel, M., van Limbeek, E., Joppe, D., van Hooijdonk, J., Kockelkoren, C., Schmitz, M., Ebus, P., Sloep, P., & Drachsler, H. (2019). The means to a blend: A practical model for the redesign of face-to-face education to blended learning. In 14th European Conference on Technology Enhanced Learning, EC-TEL 2019, Delft, 16–19 September 2019, Proceedings (Lecture Notes in Computer Science). Cham: Springer.Google Scholar
  32. Schön, D. (1983). The reflective practitioner. New York: Basic.Google Scholar
  33. Schneider, J., Di Mitri, D., Limbu, B., & Drachsler, H. (2018). Multimodal learning hub: A tool for capturing customizable multimodal learning experiences. In 13th European Conference on Technology Enhanced Learning, EC-TEL 2018, Leeds, UK, 3–5, September 2018, Proceedings (Lecture Notes in Computer Science, Vol. 11082, pp. 45–58). Cham: Springer.Google Scholar
  34. Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., et al. (2013). Personality, gender, and age in the language of social media: The open-vocabulary approach. PLoS ONE, 8(9), e73791.  https://doi.org/10.1371/journal.pone.0073791.CrossRefGoogle Scholar
  35. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.  https://doi.org/10.3102/0034654307313795.CrossRefGoogle Scholar
  36. Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. Computer Games Instruction, 55, 503–524.Google Scholar
  37. Tabuenca, B., Kalz, M., Drachsler, H., & Specht, M. (2015). Time will tell: The role of mobile learning analytics in self-regulated learning. Computers & Education, 89, 53–74.CrossRefGoogle Scholar
  38. Ternier S., Scheffel M., Drachsler H. (2018). Towards a cloud-based big data infrastructure for higher education institutions. In: Spector J. et al. (Eds), Frontiers of cyberlearning. Lecture Notes in Educational Technology. Springer, Singapore.Google Scholar
  39. von Davier, A. A., Chung Wong, P., Polyak, S., & Yudelson, M. (2019). The argument for a “Data Cube” for large-scale psychometric data. Frontiers in Psychology, 4, 00071.  https://doi.org/10.3389/feduc.2019.00071.CrossRefGoogle Scholar
  40. van der Linden, W. J., & Guo, F. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384.CrossRefGoogle Scholar
  41. von Davier, A. A. (2017). Computational psychometrics in support of collaborative educational assessments. Journal of Educational Measurement, 54(1), 3–11.Google Scholar
  42. Weidenbach, M., Drachsler, H., Wild, F., Kreutter, S., Razek, V., Grunst, G., Ender, J., Berlage, T., & Janousek J. (2007). EchoComTEE—A simulator for transoesophageal echocardiography. Anaesthesia, 62(4), 347–357.Google Scholar
  43. Winne, P. H. (2017). Leveraging big data to help each learner upgrade learning and accelerate learning science. Teachers College Record, 119(3).Google Scholar
  44. Wise, S. L., & Kong, X. J. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183.CrossRefGoogle Scholar
  45. Zieky, M. J. (2014). An introduction to the use of evidence-centered design in test development. Psicología Educativa, 20(2), 79–87.  https://doi.org/10.1016/j.pse.2014.11.003.CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  1. 1.DIPF | Leibniz Institute for Research and Information in EducationFrankfurt am MainGermany
  2. 2.Goethe University FrankfurtFrankfurt am MainGermany
  3. 3.Centre for International Student Assessment (ZIB)Frankfurt am MainGermany

Personalised recommendations