Advertisement

Multiple Document Comprehension of University Students

Test Development and Relations to Person and Process Characteristics
  • C. SchoorEmail author
  • C. Hahnel
  • N. Mahlow
  • J. Klagges
  • U. Kroehne
  • F. Goldhammer
  • C. Artelt
Chapter
  • 3 Downloads

Abstract

Multiple document comprehension is the ability to construct an integrated representation of a specific topic based on several sources. It is an important competence for university students; however, there has been so far no established instrument to assess multiple document comprehension in a standardized way. Therefore, we developed a test covering four theory-based cognitive requirements: The corroboration of information across texts, the integration of information across texts, the comparison of sources and source evaluations across texts, and the comparison of source-content links across texts. The developed test comprised 174 items and was empirically examined in a study with 310 university students. Several items had to be excluded due to psychometric misfit and differential item functioning. The resulting final test contains 67 items within 5 units (i.e., test structures of 2–3 texts and related items) and has been shown to fit a unidimensional IRT Rasch model. The test score showed expected relationships to the final school exam grade, the study level (Bachelor/Master), essay performance, sourcing behavior, as well as mental load and mental effort.

Keywords

Multiple document comprehension university students assessment multiple document literacy sourcing mental load mental effort basic computer skills reading frequency 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), pp. 183–198.  https://doi.org/10.1016/j.learninstruc.2006.03.001Google Scholar
  2. Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, pp. 64–76.  https://doi.org/10.1016/j.lindif.2013.01.007Google Scholar
  3. Arbeitskreis Deutscher Qualifikationsrahmen (2011). Deutscher Qualifikationsrahmen für lebenslanges Lernen. Retrieved from http://www.dqr.de/media/content/Der_Deutsche_Qualifikationsrahmen_fue_lebenslanges_Lernen.pdf
  4. Blossfeld, H.-P., Roßbach, H.-G., & Maurice, J. v. (2011). Education as a lifelong process: The German National Educational Panel Study (NEPS) [Special Issue]. Zeitschrift für Erziehungswissenschaften, 14.Google Scholar
  5. Braasch, J. L. G., & Bråten, I. (2017). The discrepancy-induced source comprehension (D-ISC) model: Basic assumptions and preliminary evidence. Educational Psychologist, 52(3), pp. 167–181.  https://doi.org/10.1080/00461520.2017.1323219Google Scholar
  6. Braasch, J. L. G., Bråten, I., Strømsø, H. I., & Anmarkrud, Ø. (2014). Incremental theories of intelligence predict multiple document comprehension. Learning and Individual Differences, 31, pp. 11–20.  https://doi.org/10.1016/j.lindif.2013.12.012Google Scholar
  7. Braasch, J. L. G., Rouet, J.-F., Vibert, N., & Britt, M. A. (2012). Readers’ use of source information in text comprehension. Memory & Cognition, 40(3), pp. 450–465.  https://doi.org/10.3758/s13421-011-0160-6Google Scholar
  8. Bråten, I., Salmerón, L., & Strømsø, H. I. (2016). Who said that? Investigating the Plausibility-Induced Source Focusing assumption with Norwegian undergraduate readers. Contemporary Educational Psychology, 46, pp. 253–262.  https://doi.org/10.1016/j.cedpsych.2016.07.004Google Scholar
  9. Bråten, I., Ferguson, L. E., Strømsø, H. I., & Anmarkrud, Ø. (2014). Students working with multiple conflicting documents on a scientific issue: Relations between epistemic cognition while reading and sourcing and argumentation in essays. British Journal of Educational Psychology, 84(1), pp. 58–85.  https://doi.org/10.1111/bjep.12005Google Scholar
  10. Britt, M. A., & Rouet, J.-F. (2012). Learning with multiple documents: Component skills and their acquisition. In J. R. Kirby & M. J. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276–314). New York: Cambridge University Press.Google Scholar
  11. Britt, M. A., & Sommer, J. (2004). Facilitating textual integration with macro-structure focusing tasks. Reading Psychology, 25(4), pp. 313–339.  https://doi.org/10.1080/02702710490522658Google Scholar
  12. Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20(4), pp. 485–522.  https://doi.org/10.1207/s1532690xci2004_2Google Scholar
  13. Britt, M. A., Perfetti, C. A., Sandak, R., & Rouet, J.-F. (1999). Content integration and source separation in learning from multiple texts. In S. R. Goldman, A. C. Graesser, & P. Van den Broek (Eds.), Narrative, comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 209–233). Mahwah, NJ: Erlbaum.Google Scholar
  14. Cerdán, R., & Vidal-Abarca, E. (2008). The effects of tasks on integrating information from multiple documents. Journal of Educational Psychology, 100(1), pp. 209–222.  https://doi.org/10.1037/0022-0663.100.1.209Google Scholar
  15. Ferguson, L. E. (2015). Epistemic beliefs and their relation to multiple-text comprehension: A Norwegian program of research. Scandinavian Journal of Educational Research, 59(6), pp. 731–752.  https://doi.org/10.1080/00313831.2014.971863Google Scholar
  16. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010a). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35(3), pp. 157–173.  https://doi.org/10.1016/j.cedpsych.2009.11.002Google Scholar
  17. Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010b). Understanding and integrating multiple science texts: Summary tasks are sometimes better than argument tasks. Reading Psychology, 31(1), pp. 30–68.  https://doi.org/10.1080/02702710902733600Google Scholar
  18. Goldhammer, F., Naumann, J., & Keßel, Y. (2013). Assessing individual differences in basic computer skills. Psychometric characteristics of an interactive performance measure. European Journal of Psychological Assessment, 29(4), pp. 263–275.  https://doi.org/10.1027/1015-5759/a000153Google Scholar
  19. Goldman, S. R., & Scardamalia, M. (2013). Managing, understanding, applying, and creating knowledge in the information age: Next-generation challenges and opportunities. Cognition and Instruction, 31(2), pp. 255–269.  https://doi.org/10.1080/10824669.2013.773217Google Scholar
  20. Goldman, S. R., Britt, M. A., Brown, W., Cribb, G., George, M., Greenleaf, C., . . . Project, R. (2016). Disciplinary literacies and learning to read for understanding: A conceptual framework for disciplinary literacy. Educational Psychologist, 51(2), pp. 219–246.  https://doi.org/10.1080/00461520.2016.1168741Google Scholar
  21. Griffin, T. D., Wiley, J., Britt, M. A., & Salas, C. R. (2017). The role of clear thinking in learning science from multiple-document ınquiry tasks. International Electronic Journal of Elementary Education, 5(1), pp. 63–78.Google Scholar
  22. Gruenbaum, E. A. (2012). Common literacy struggles with college students: Using the reciprocal teaching technique. Journal of College Reading and Learning, 42(2), pp. 109–116.  https://doi.org/10.1080/10790195.2012.10850357Google Scholar
  23. Guthrie, J. T., Wigfield, A., Metsala, J. L., & Cox, K. E. (1999). Motivational and cognitive predictors of text comprehension and reading amount. Scientific Studies of Reading, 3(3), pp. 231–256.Google Scholar
  24. Hahnel, C., Kroehne, U., Goldhammer, F., Schoor, C., Mahlow, N., & Artelt, C. (2019). Validating process variables of sourcing in an assessment of multiple document comprehension. British Journal of Educational Psychology, 89(3), pp. 524–537.  https://doi.org/10.1111/bjep.12278
  25. Hahnel, C., Schoor, C., Kröhne, U., Goldhammer, F., Mahlow, N., & Artelt, C. (2019). The role of cognitive load for university students’ comprehension of multiple documents. Zeitschrift für Pädagogische Psychologie, 33(2), pp. 105–118.  https://doi.org/10.1024/1010-0652/a000238Google Scholar
  26. Homann, B. (2000). Das Dynamische Modell der Informationskompetenz (DYMIK) als Grundlage für bibliothekarische Schulungen. In G. Knorz & R. Kuhlen (Eds.), Informationskompetenz – Basiskompetenz in der Informationsgesellschaft. Proceedings des 7. Internationale Symposiums für Informationswissenschaft (ISI 2000), Darmstadt, 8.–10. November 2000 (pp. 195–206). Konstanz: UVK Verlag.Google Scholar
  27. Kammerer, Y., & Gerjets, P. (2014). Quellenbewertungen und Quellenverweise bei Lesen und Zusammenfassen wissensbezogener Informationen aus multiplen Webseiten [Source evaluations and source references when reading and summarizing science-related information from multiple web pages]. Unterrichtswissenschaft, 42(1), pp. 7–23.Google Scholar
  28. Kammerer, Y., Kalbfell, E., & Gerjets, P. (2016). Is this information source commercially biased? How contradictions between web pages stimulate the consideration of source information. Discourse Processes, 53(5–6), pp. 430–456.  https://doi.org/10.1080/0163853x.2016.1169968Google Scholar
  29. Keck, D., Kammerer, Y., & Starauschek, E. (2015). Reading science texts online: Does source information influence the identification of contradictions within texts? Computers & Education, 82, pp. 442–449.  https://doi.org/10.1016/j.compedu.2014.12.005Google Scholar
  30. Kintsch, W. (1998). Comprehension. A paradigm for cognition. Cambridge: Cambridge University Press.Google Scholar
  31. Krell, M. (2015). Evaluating an instrument to measure mental load and mental effort using Item Response Theory. Science Education Review Letters, 2015, pp. 1–6.Google Scholar
  32. Kultusministerkonferenz (2012). Bildungsstandards im Fach Deutsch für die Allgemeine Hochschulreife (Beschluss der Kultusministerkonferenz vom 18.10.2012) [Educational standards in the subject German for the general qualification for university entrance (decision of the Standing Conference of the Ministers of Education and Cultural Affairs of the Länder in the Federal Republic of Germany of 18.10.2012)]. Retrieved from www.kmk.org/fileadmin/veroeffentlichungen_beschluesse/2012/2012_10_18-Bildungsstandards-Deutsch-Abi.pdf
  33. Lau, J. (2006). Guidelines on information literacy for lifelong learning. Retrieved from https://www.archive.ifla.org/VII/s42/pub/IL-Guidelines2006.pdf
  34. List, A., & Alexander, P. A. (2017). Cognitive affective engagement model of multiple source use. Educational Psychologist, 52(3), pp. 182–199.  https://doi.org/10.1080/00461520.2017.1329014Google Scholar
  35. Locher, F. M., & Pfost, M. (2019). Erfassung des Lesevolumens in Large-Scale Studien. Diagnostica, 65(1), pp. 26–36.  https://doi.org/10.1026/0012-1924/a000203Google Scholar
  36. Maier, J., & Richter, T. (2013). Text belief consistency effects in the comprehension of multiple texts with conflicting information. Cognition and Instruction, 31(2), pp. 151–175.  https://doi.org/10.1080/07370008.2013.769997Google Scholar
  37. Mühlen, S. v. d., Richter, T., Schmid, S., Schmidt, E. M., & Berthold, K. (2016). The use of source-related strategies in evaluating multiple psychology texts: A student–scientist comparison. Reading and Writing, 29(8), pp. 1677–1698.  https://doi.org/10.1007/s11145-015-9601-0Google Scholar
  38. Paas, F. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), pp. 429–434.Google Scholar
  39. Paul, J., Macedo-Rouet, M., Rouet, J.-F., & Stadtler, M. (2017). Why attend to source information when reading online? The perspective of ninth grade students from two different countries. Computers & Education, 113, pp. 339–354.  https://doi.org/10.1016/j.compedu.2017.05.020Google Scholar
  40. Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Toward a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The construction of mental representations during reading (pp. 99–122). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  41. Peter, T. (2019, June 18). „Es gibt gravierende Mängel, was die Studierfähigkeit zahlreicher Abiturienten angeht“. Leipziger Volkszeitung. Retrieved from https://www.lvz.de/Nachrichten/Politik/Praesident-der-Hochschulrektorenkonferenz-Es-gibt-gravierende-Maengel-was-die-Studierfaehigkeit-zahlreicher-Abiturienten-angeht
  42. Primor, L., & Katzir, T. (2018). Measuring Multiple Text Integration: A Review. Frontiers in Psychology, 9(2294).  https://doi.org/10.3389/fpsyg.2018.02294
  43. Rölke, H. (2012). The ItemBuilder: A graphical authoring system for complex item development. In T. Bastiaens & G. Marks (Eds.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2012 (Vol. 2012, pp. 344–353). Chesapeake, VA: AACE.Google Scholar
  44. Rouet, J.-F., Britt, M. A., & Durik, A. M. (2017). RESOLV: Readers’ representation of reading contexts and tasks. Educational Psychologist, 52(3), pp. 200–215.  https://doi.org/10.1080/00461520.2017.1329015Google Scholar
  45. Rouet, J.-F., Favart, M., Britt, M. A., & Perfetti, C. A. (1997). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15(1), pp. 85–106.  https://doi.org/10.1207/s1532690xci1501_3.Google Scholar
  46. Salmerón, L., Gil, L., Bråten, I., & Strømsø, H. (2010). Comprehension effects of signalling relationships between documents in search engines. Computers in Human Behavior, 26(3), pp. 419–426.  https://doi.org/10.1016/j.chb.2009.11.013Google Scholar
  47. Scharrer, L., & Salmerón, L. (2016). Sourcing in the reading process [Special issue]. Reading and Writing, 29(8).Google Scholar
  48. Schmalhofer, F., & Glavanov, D. (1986). Three components of understanding a programmer’s manual: Verbatim, propositional, and situational representations. Journal of Memory and Language, 25(3), pp. 279–294.  https://doi.org/10.1016/0749-596x(86)90002-1Google Scholar
  49. Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representation. Learning and Instruction, 13(2), pp. 141–156.Google Scholar
  50. Schoor, C., Hahnel, C., Artelt, C., Reimann, D., Kröhne, U., & Goldhammer, F. (2020). Entwicklung und Skalierung eines Tests zur Erfassung des Verständnisses multipler Dokumente von Studierenden [Developing and scaling a test of multiple document comprehension in university students]. Diagnostica.  https://doi.org/10.1026/0012-1924/a000231
  51. Seufert, T. (2009). Lernen mit multiplen Repräsentationen – Gestaltungs- und Verarbeitungsstrategien [Learning with multiple representations – Design and processing strategies]. In R. Plötzner, T. Leuders, & A. Wichert (Eds.), Lernchance Computer. Strategien für das Lernen mit digitalen Medienverbünden (pp. 45–66). Münster: Waxmann.Google Scholar
  52. Spinath, B., Stiensmeier-Pelster, J., Schöne, C., & Dickhäuser, O. (2002). SELLMO: Skalen zur Erfassung der Lern- und Leistungsmotivation [Learning and Achievement Motivation Scales]. Göttingen: Hogrefe.Google Scholar
  53. Stadtler, M., & Bromme, R. (2014). The content–source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D. N. Rapp & J. L. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). Cambridge, MA: MIT Press.Google Scholar
  54. Stadtler, M., Scharrer, L., Skodzik, T., & Bromme, R. (2014). Comprehending multiple documents on scientific controversies: Effects of reading goals and signaling rhetorical relationships. Discourse Processes, 51(1–2), pp. 93–116.  https://doi.org/10.1080/0163853x.2013.855535Google Scholar
  55. Strømsø, H. I., Bråten, I., Britt, M. A., & Ferguson, L. E. (2013). Spontaneous sourcing among students reading multiple documents. Cognition and Instruction, 31(2), pp. 176–203.  https://doi.org/10.1080/07370008.2013.769994Google Scholar
  56. Sweller, J. (2010). Element interactivity and intrinsic, extraneous, and germane cognitive load. Educational Psychology Review, 22(2), pp. 123–138.  https://doi.org/10.1007/s10648-010-9128-5Google Scholar
  57. Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54(3), pp. 427–450.  https://doi.org/10.1007/bf02294627Google Scholar
  58. Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83(1), pp. 73–87.  https://doi.org/10.1037/0022-0663.83.1.73Google Scholar

Copyright information

© Springer Fachmedien Wiesbaden GmbH, part of Springer Nature 2020

Authors and Affiliations

  • C. Schoor
    • 1
    Email author
  • C. Hahnel
    • 2
  • N. Mahlow
    • 3
  • J. Klagges
    • 2
  • U. Kroehne
    • 2
  • F. Goldhammer
    • 2
  • C. Artelt
    • 3
  1. 1.Universität BambergBambergGermany
  2. 2.DIPF | Leibniz-Institut für Bildungsforschung und BildungsinformationFrankfurt am MainGermany
  3. 3.Leibniz-Institut für BildungsverläufeBambergGermany

Personalised recommendations