Instructional Science

, Volume 43, Issue 1, pp 93–114 | Cite as

Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings

  • Annett SchmeckEmail author
  • Maria Opfermann
  • Tamara van Gog
  • Fred Paas
  • Detlev Leutner


Subjective cognitive load (CL) rating scales are widely used in educational research. However, there are still some open questions regarding the point of time at which such scales should be applied. Whereas some studies apply rating scales directly after each step or task and use an average of these ratings, others assess CL only once after the whole learning or problem-solving phase. To investigate if these two approaches are comparable indicators of experienced CL, two experiments were conducted, in which 168 and 107 teacher education university students, respectively, worked through a sequence of six problems. CL was assessed by means of subjective ratings of mental effort and perceived task difficulty after each problem and after the whole process. Results showed that the delayed ratings of both effort and difficulty were significantly higher than the average of the six ratings made during problem solving. In addition, the problems we assumed to be of higher complexity seemed to be the best predictors for the delayed ratings. Interestingly, for ratings of affective variables, such as interest and motivation, the delayed rating did not differ from the average of immediate ratings.


Cognitive load Problem solving Mental effort Task difficulty Measurement 


  1. Antonenko, P., & Niederhauser, D. (2010). The influence of leads on cognitive load and learning in a hypertext-assisted learning environment. Computers in Human Behavior, 26, 140–150.CrossRefGoogle Scholar
  2. Antonenko, P., Paas, F., Grabner, R., & Van Gog, T. (2010). Using electroencephalography (EEG) to measure cognitive load. Educational Psychology Review, 22, 425–438.CrossRefGoogle Scholar
  3. Ayres, P. (2006). Using subjective measures to detect variations of intrinsic load within problems. Learning and Instruction, 16, 389–400.CrossRefGoogle Scholar
  4. Bratfisch, O., Borg, G., & Dornic, S. (1972). Perceived item difficulty in three tests of intellectual performance capacity. Stockholm: Institute of Applied Psychology, Report No. 29.Google Scholar
  5. Brünken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38, 53–61.CrossRefGoogle Scholar
  6. Brünken, R., Plass, J. L., & Leutner, D. (2004). Assessment of cognitive load in multimedia learning with dual-task methodology: Auditory load and modality effects. Instructional Science, 32, 115–132.CrossRefGoogle Scholar
  7. Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8, 293–332.CrossRefGoogle Scholar
  8. Chandler, P., & Sweller, J. (1992). The split-attention effect as a factor in the design of instruction. British Journal of Educational Psychology, 62, 233–246.CrossRefGoogle Scholar
  9. Chandler, P., & Sweller, J. (1996). Cognitive load while learning to use a computer program. Applied Cognitive Psychology, 10, 1–20.CrossRefGoogle Scholar
  10. de Jong, T. (2010). Cognitive load theory, educational research, and instructional design: Some food for thought. Instructional Science, 38, 105–134.CrossRefGoogle Scholar
  11. Kalyuga, S., Chandler, P., & Sweller, J. (1999). Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology, 13, 351–371.CrossRefGoogle Scholar
  12. Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is superior to studying worked examples. Journal of Educational Psychology, 93, 579–588.CrossRefGoogle Scholar
  13. Kühl, T., Scheiter, K., Gerjets, P., & Edelmann, J. (2011). The influence of text modality on learning with static and dynamic visualizations. Computers in Human Behavior, 27, 29–35.CrossRefGoogle Scholar
  14. Kuusela, H., & Paul, P. (2000). A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology, 113, 387–404.CrossRefGoogle Scholar
  15. Leutner, D., Leopold, C., & Sumfleth, E. (2009). Cognitive load and science text comprehension: Effects of drawing and mentally imagining text content. Computers in Human Behavior, 25, 284–289.CrossRefGoogle Scholar
  16. Logie, R. H. (1995). Visuo-spatial working memory. Hove, UK: Lawrence Erlbaum Associates.Google Scholar
  17. Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of Educational Psychology, 88, 49–63.CrossRefGoogle Scholar
  18. Opfermann, M. (2008). There’s more to it than instructional design: The role of individual learner characteristics for hypermedia learning. Berlin, Germany: Logos.Google Scholar
  19. Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84, 429–434.CrossRefGoogle Scholar
  20. Paas, F., Ayres, P., & Pachman, M. (2008). Assessment of cognitive load in multimedia learning: Theory, methods and applications. In D. Robinson & G. Schraw (Eds.), Recent innovations in educational technology that facilitate student learning (pp. 20–36). Charlotte, NC: Information Age Publishing.Google Scholar
  21. Paas, F., Renkl, A., & Sweller, J. (2003a). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38, 1–4.CrossRefGoogle Scholar
  22. Paas, F., Tuovinen, J., Tabbers, H. K., & Van Gerven, P. W. M. (2003b). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38, 63–71.CrossRefGoogle Scholar
  23. Paas, F., & van Merriënboer, J. J. G. (1994). Variability of worked examples and transfer of geometrical problem solving skills: A cognitive-load approach. Journal of Educational Psychology, 86, 122–133.CrossRefGoogle Scholar
  24. Paas, F., van Merriënboer, J. J. G., & Adam, J. J. (1994). Measurement of cognitive load in instructional research. Perceptual and Motor Skills, 79, 419–430.CrossRefGoogle Scholar
  25. Park, B., Moreno, R., Seufert, T., & Brünken, R. (2011). Does cognitive load moderate the seductive details effect? A multimedia study. Computers in Human Behavior, 27, 5–10.CrossRefGoogle Scholar
  26. Plass, J. L., Moreno, R., & Brünken, R. (Eds.). (2010). Cognitive load: Theory & application. Cambridge: Cambridge University Press.Google Scholar
  27. Renkl, A., Gruber, H., Weber, S., Lerche, T., & Schweizer, K. (2003). Cognitive Load beim Lernen aus Lösungsbeispielen [Cognitive load of learning from worked-out examples]. Zeitschrift für Pädagogische Psychologie, 17, 93–101.CrossRefGoogle Scholar
  28. Rheinberg, F., Vollmeyer, R., & Burns, B. D. (2001). FAM: Ein Fragebogen zur Erfassung aktueller Motivation in Lern- und Leistungssituationen [QCM: A questionnaire to assess current motivation in learning situations]. Diagnostica, 47, 57–66.CrossRefGoogle Scholar
  29. Schwamborn, A., Thillmann, H., Opfermann, M., & Leutner, D. (2011). Cognitive load and instructionally supported learning with provided and learner-generated visualizations. Computer in Human Behavior, 27, 89–93.CrossRefGoogle Scholar
  30. Seufert, T., & Brünken, R. (2006). Cognitive load and the format of instructional aids for coherence formation. Applied Cognitive Psychology, 20, 321–331.CrossRefGoogle Scholar
  31. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285.CrossRefGoogle Scholar
  32. Sweller, J. (1993). Some cognitive processes and their consequences for the organization and presentation of information. Australian Journal of Psychology, 45, 1–8.CrossRefGoogle Scholar
  33. Sweller, J. (2010). Element interactivity and intrinsic, extraneous and germane cognitive load. Educational Psychology Review, 22, 123–138.CrossRefGoogle Scholar
  34. Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. New York: Springer.CrossRefGoogle Scholar
  35. Sweller, J., van Merriënboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296.CrossRefGoogle Scholar
  36. Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. G. (2004). Multimedia instructions and cognitive load theory: Effects of modality and cueing. British Journal of Educational Psychology, 74, 71–81.CrossRefGoogle Scholar
  37. Taylor, K. L., & Dionne, J.-P. (2000). Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology, 92, 413–425.CrossRefGoogle Scholar
  38. Van Gerven, P. W. M., Paas, F., van Merriënboer, J. J. G., & Schmidt, H. G. (2004). Memory load and the cognitive pupillary response in aging. Psychophysiology, 41, 167–174.CrossRefGoogle Scholar
  39. Van Gog, T., & Jarodzka, H. (2013). Eye tracking as a tool to study and enhance cognitive and metacognitive processes in computer-based learning environments. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 143–156). New York: Springer.Google Scholar
  40. Van Gog, T., Kirschner, F., Kester, L., & Paas, F. (2012). Timing and frequency of mental effort measurement: Evidence in favor of repeated measures. Applied Cognitive Psychology, 26, 833–839.CrossRefGoogle Scholar
  41. Van Gog, T., & Paas, F. (2008). Instructional efficiency: Revisiting the original construct in educational research. Educational Psychologist, 43, 16–26.CrossRefGoogle Scholar
  42. Van Gog, T., Paas, F., & van Merriënboer, J. J. G. (2006). Effects of process-oriented worked examples on troubleshooting transfer performance. Learning and Instruction, 16, 154–164.CrossRefGoogle Scholar
  43. Van Merriënboer, J. J. G., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17, 147–177.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  • Annett Schmeck
    • 1
    Email author
  • Maria Opfermann
    • 1
  • Tamara van Gog
    • 2
  • Fred Paas
    • 2
  • Detlev Leutner
    • 1
  1. 1.Department of Instructional Psychology, Faculty for Educational SciencesUniversity of Duisburg-EssenEssenGermany
  2. 2.Capaciteitsgroep Psychologie, Faculteit der Sociale WetenschappenErasmus University RotterdamRotterdamThe Netherlands

Personalised recommendations