How do students process complex formative feedback in question-answering tasks? A think-aloud study

Abstract

The goal of this study was to examine how students process formative feedback that included corrective and elaborative information in online question-answering tasks. Skilled and less-skilled comprehenders in grade 8 read texts and answered comprehension questions. Prior to responding, students were asked to select the textual information relevant to answer each question. Students received formative feedback that included information about the answer correctness and an elaborative message, either automatically delivered or optionally accessed. Students’ actions, as well as verbal protocols during feedback processing, were recorded. The results showed that: (a) students paid more attention to the answer correctness than to elaborative feedback messages; (b) students paid more attention to feedback information after failure than after success; (c) students actively monitored the accuracy of their responses; and (d) differences between skilled and less-skilled comprehenders in processing feedback were very limited. Theoretical and practical implications of these results are discussed.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

Notes

  1. 1.

    PROLEC-SE is a standardized test for the evaluation of Primary and Secondary students’ reading processes in Spanish (Ramos and Cuetos 1999).

References

  1. Ackerman, R., & Goldsmith, M. (2011). Metacognitive regulation of text learning: On screen versus on paper. Journal of Experimental Psychology: Applied, 17(1), 18–32. https://doi.org/10.1037/a0022086.

    Google Scholar 

  2. Ackerman, R., & Lauterman, T. (2012). Taking reading comprehension exams on screen or on paper? A metacognitive analysis of learning texts under time pressure. Computers in Human Behavior, 28(5), 1816–1828. https://doi.org/10.1016/j.chb.2012.04.023.

    Article  Google Scholar 

  3. Anmarkrud, Ø., McCrudden, M. T., Bråten, I., & Strømsø, H. I. (2013). Task-oriented reading of multiple documents: Online comprehension processes and offline products. Instructional Science, 41(5), 873–894. https://doi.org/10.1007/s11251-013-9263-8.

    Article  Google Scholar 

  4. Bohn-Gettler, C. M., & Kendeou, P. (2014). The interplay of reader goals, working memory, and text structure during reading. Contemporary Educational Psychology, 39(3), 206–219. https://doi.org/10.1016/j.cedpsych.2014.05.003.

    Article  Google Scholar 

  5. Cerdán, R., & Vidal-Abarca, E. (2008). The effects of tasks on integrating information from multiple documents. Journal of Educational Psychology, 100(1), 209–222. https://doi.org/10.1037/0022-0663.100.1.209.

    Article  Google Scholar 

  6. Cerdán, R., Gilabert, R., & Vidal-Abarca, E. (2011). Selecting information to answer questions: Strategic individual differences when searching texts. Learning and Individual Differences, 21(2), 201–205. https://doi.org/10.1016/j.lindif.2010.11.007.

    Article  Google Scholar 

  7. Cerdán, R., Gilabert, R., & Vidal-Abarca, E. (2013). Self-generated explanations on the question demands are not always helpful. The Spanish Journal of Psychology, 16(e26), 1–11. https://doi.org/10.1017/sjp.2013.45.

    Google Scholar 

  8. Delgado, P., Vargas, C., Ackerman, R., & Salmerón, L. (2018). Don't throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research Review, 25, 23–38. https://doi.org/10.1016/j.edurev.2018.09.003.

    Article  Google Scholar 

  9. Dunlosky, J., & Metcalfe, J. (2009). Metacognition: A textbook for cognitive, educational, life span and applied psychology. Thousand Oaks: Sage.

    Google Scholar 

  10. Farr, R., Pritchard, R., & Smitten, B. (1990). A description of what happens when an examinee takes a multiple-choice reading comprehension test. Journal of Educational Measurement, 27(3), 209–226. https://doi.org/10.1111/j.1745-3984.1990.tb00744.x.

    Article  Google Scholar 

  11. Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137(2), 316–344. https://doi.org/10.1037/a0021663.

    Article  Google Scholar 

  12. Fox, J.-P., Klein Entink, R., & Timmers, C. (2014). The joint multivariate modeling of multiple mixed response sources: Relating student performances with feedback behavior. Multivariate Behavioral Research, 49(1), 54–66. https://doi.org/10.1080/00273171.2013.843441.

    Article  Google Scholar 

  13. Fyfe, E. R., Rittle-Johnson, B., & DeCaro, M. S. (2012). The effects of feedback during exploratory mathematics problem solving: Prior knowledge matters. Journal of Educational Psychology, 104(4), 1094–1108. https://doi.org/10.1037/a0028389.

    Article  Google Scholar 

  14. Graesser, A. C., & Murachver, T. (1985). Symbolic procedures of question answering. In A. C. Graesser & J. B. Black (Eds.), The psychology of questions (pp. 15–88). Hillsdale: Erlbaum.

    Google Scholar 

  15. Hattie, J., & Gan, M. (2011). Instruction based on feedback. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction (pp. 249–271). New York: Routledge.

    Google Scholar 

  16. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487.

    Article  Google Scholar 

  17. Jordan, S. (2012). Student engagement with assessment and feedback: Some lessons from short-answer free-text e-assessment questions. Computers & Education, 58(2), 818–834. https://doi.org/10.1016/j.compedu.2011.10.007.

    Article  Google Scholar 

  18. Kendeou, P., & O’Brien, E. J. (2016). Prior knowledge: Acquisition and revision. In P. Afflerbach (Ed.), Handbook of individual differences in reading: Reader, text and context (pp. 151–163). New York: Routledge.

    Google Scholar 

  19. Kendeou, P., McMaster, K. L., & Christ, T. J. (2016). Reading comprehension: Core components and processes. Policy Insights From the Behavioral and Brain Sciences, 3(1), 62–69. https://doi.org/10.1177/2372732215624707.

    Article  Google Scholar 

  20. Krause, U.-M., Stark, R., & Mandl, H. (2009). The effects of cooperative learning and feedback on e-learning in statistics. Learning and Instruction, 19(2), 158–170. https://doi.org/10.1016/j.learninstruc.2008.03.003.

    Article  Google Scholar 

  21. Llorens, A. C., Gil, L., Vidal-Abarca, E., Martínez, T., Mañá, A., & Gilabert, R. (2011). Evaluación de la competencia lectora: la prueba de Competencia Lectora para Educación Secundaria (CompLEC). Psicothema, 23(4), 808–817.

    Google Scholar 

  22. Llorens, A. C., Vidal-Abarca, E., Cerdán, R., & Ávila, V. (2015). Does formative feedback on search behavior help students in answering comprehension questions from an available text? Infancia y Aprendizaje, 38(4), 808–841. https://doi.org/10.1080/02103702.2015.1076269.

    Article  Google Scholar 

  23. Llorens, A. C., Vidal-Abarca, E., & Cerdán, R. (2016). Formative feedback to transfer self-regulation of task-oriented reading strategies. Journal of Computer Assisted Learning, 32(4), 314–331. https://doi.org/10.1111/jcal.12134.

    Article  Google Scholar 

  24. Magliano, J. P., & Millis, K. K. (2003). Assessing reading skill with a think-aloud procedure and latent semantic analysis. Cognition and Instruction, 21(3), 251–283. https://doi.org/10.1207/S1532690XCI2103_02.

    Article  Google Scholar 

  25. Maier, U., Wolf, N., & Randler, C. (2016). Effects of a computer-assisted formative assessment intervention based on multiple-tier diagnostic items and different feedback types. Computers & Education, 95, 85–98. https://doi.org/10.1016/j.compedu.2015.12.002.

    Article  Google Scholar 

  26. Máñez, I., Vidal-Abarca, E., & Martínez, T. (2016, July). Accuracy to select relevant text information to answer questions from a text. Paper presented at the 26th annual meeting of the Society for Text and Discourse, Kassel, Germany.

  27. Martínez, T., Vidal-Abarca, E., Sellés, P., & Gilabert, R. (2008). Evaluation of comprehension strategies and processes: Test of comprehension processes (TCP). Infancia y Aprendizaje, 31(3), 319–332. https://doi.org/10.1174/021037008785702956.

    Article  Google Scholar 

  28. Martínez, T., Vidal-Abarca, E., Gil, L., & Gilabert, R. (2009). On-line assessment of comprehension processes. The Spanish Journal of Psychology, 12(1), 308–319. https://doi.org/10.1017/S1138741600001700.

    Article  Google Scholar 

  29. Moreno, R. (2004). Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery-based multimedia. Instructional Science, 32, 99–113. https://doi.org/10.1023/B:TRUC.0000021811.66966.1d.

    Article  Google Scholar 

  30. Mory, E. H. (2004). Feedback research revisited. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (pp. 745–783). Mahwah: Erlbaum.

    Google Scholar 

  31. Narciss, S. (2008). Feedback strategies for interactive learning tasks. In J. M. Spector, M. D. Merril, J. van Merriënboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology (pp. 125–144). New York: Erlbaum.

    Google Scholar 

  32. Narciss, S., & Huth, K. (2004). How to design informative tutoring feedback for multi-media learning. In H. M. Niegemann, D. Leutner, & R. Brünken (Eds.), Instructional design for multimedia learning (pp. 181–195). Münster: Waxmann.

    Google Scholar 

  33. Narciss, S., Sosnovsky, S., Schnaubert, L., Andrès, E., Eichelmann, A., Goguadze, G., & Melis, E. (2014). Exploring feedback and student characteristics relevant for personalizing feedback strategies. Computers & Education, 71, 56–76. https://doi.org/10.1016/j.compedu.2013.09.011.

    Article  Google Scholar 

  34. Ness, M. (2011). Explicit reading comprehension instruction in elementary classrooms: Teacher use of reading comprehension strategies. Journal of Research in Childhood Education, 25(1), 98–117. https://doi.org/10.1080/02568543.2010.531076.

    Article  Google Scholar 

  35. Pekrun, R., Frenzel, A. C., Goetz, T., & Perry, R. P. (2007). The control-value theory of achievement emotions: An integrative approach to emotions in education. In P. A. Schutz & R. Pekrun (Eds.), Emotion in education (pp. 13–36). San Diego: Academic Press.

    Google Scholar 

  36. Perfetti, C. A. (1985). Reading ability. New York: Oxford University Press.

    Google Scholar 

  37. Pressley, M., Wharton-McDonald, R., Mistretta-Hampston, J., & Echevarria, M. (1998). Literacy instruction in 10 fourth-grade classrooms in upstate New York. Scientific Studies of Reading, 2(2), 159–194. https://doi.org/10.1207/s1532799xssr0202_4.

    Article  Google Scholar 

  38. Ramos, J. L., & Cuetos, F. (1999). PROLEC-SE: Evaluación de los procesos lectores en alumnos de tercer ciclo de educación primaria y secundaria. Madrid: TEA Ediciones.

    Google Scholar 

  39. Ramos, L., & Vidal-Abarca, E. (2013). Differences between students with high and low reading literacy skills: A study with think aloud methodology. Cultura y Educación, 25(3), 295–308. https://doi.org/10.1174/113564013807749722.

    Article  Google Scholar 

  40. Roelle, J., & Berthold, K. (2017). Effects of incorporating retrieval into learning tasks: The complexity of the tasks matters. Learning and Instruction, 49(2), 142–156. https://doi.org/10.1016/j.learninstruc.2017.01.008.

    Article  Google Scholar 

  41. Rouet, J.-F., Britt, M. A., & Durik, A. M. (2017). RESOLV: Readers' representation of reading contexts and tasks. Educational Psychologist, 52(3), 200–215. https://doi.org/10.1080/00461520.2017.1329015.

    Article  Google Scholar 

  42. Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441–474. https://doi.org/10.1191/0265532206lt337oa.

    Article  Google Scholar 

  43. Salmerón, L., Strømsø, H. I., Kammerer, Y., Stadtler, M., & van den Broek, P. (2018). Comprehension processes in digital reading. In M. Barzillai, J. Thomson, S. Schroeder, & P. van den Broek (Eds.), Learning to read in a digital world (pp. 91–120). Amsterdam, The Netherlands: John Benjamins.

    Google Scholar 

  44. Schooler, J. W. (2011). Introspecting in the spirit of William James: Comment on Fox, Ericsson, and Best (2011). Psychological Bulletin, 137(2), 345–350. https://doi.org/10.1037/a0022390.

    Article  Google Scholar 

  45. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795.

    Article  Google Scholar 

  46. Timmers, C. F., & Veldkamp, B. P. (2011). Attention paid to feedback provided by a computer-based assessment for learning on information literacy. Computers & Education, 56(3), 923–930. https://doi.org/10.1016/j.compedu.2010.11.007.

    Article  Google Scholar 

  47. Timms, M., DeVelle, S., & Lay, D. (2016). Towards a model of how learners process feedback: A deeper look at learning. Australian Journal of Education, 60(2), 128–145. https://doi.org/10.1177/0004944116652912.

    Article  Google Scholar 

  48. van den Broek, P., Kendeou, P., & White, M. J. (2009). Cognitive processes during reading: Implications for the use of multimedia to foster reading comprehension. In A. G. Bus & S. B. Neuman (Eds.), Multimedia and literacy development: Improving achievement for young learners (pp. 57–73). New York: Routledge.

    Google Scholar 

  49. Van der Kleij, F. M., Eggen, T. J. H. M., Timmers, C. F., & Veldkamp, B. P. (2012). Effects of feedback in a computer-based assessment for learning. Computers & Education, 58(1), 263–272. https://doi.org/10.1016/j.compedu.2011.07.020.

    Article  Google Scholar 

  50. Van der Kleij, F. M., Feskens, R. C. W., & Eggen, T. J. H. M. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes: A meta-analysis. Review of Educational Research, 85(4), 475–511. https://doi.org/10.3102/0034654314564881.

    Article  Google Scholar 

  51. Veenman, M. V. J., Elshout, J. J., & Groen, M. G. M. (1993). Thinking aloud: Does it affect regulatory processes in learning? Tijdschrift voor Onderwijsresearch, 18(6), 322–330.

    Google Scholar 

  52. Vidal-Abarca, E., Gilabert, R., & Abad, N. (2002a). A proposal for good expository text: Toward an expository text technology. Infancia y Aprendizaje, 25(4), 499–514. https://doi.org/10.1174/021037002762064064.

    Article  Google Scholar 

  53. Vidal-Abarca, E., Reyes, H., Gilabert, R., Calpe, J., Soria, E., & Graesser, A. C. (2002b). ETAT: Expository text analysis tool. Behavior Research Methods, Instruments, & Computers, 34(1), 93–107. https://doi.org/10.3758/BF03195428.

    Article  Google Scholar 

  54. Vidal-Abarca, E., Mañá, A., & Gil, L. (2010). Individual differences for self-regulating task-oriented reading activities. Journal of Educational Psychology, 102(4), 817–826. https://doi.org/10.1037/a0020062.

    Article  Google Scholar 

  55. Vidal-Abarca, E., Martínez, T., Salmerón, L., Cerdán, R., Gilabert, R., Gil, L., Mañá, A., Llorens, A. C., & Ferris, R. (2011). Recording online processes in task-oriented reading with Read&Answer. Behavior Research Methods, 43(1), 179–192. https://doi.org/10.3758/s13428-010-0032-1.

    Article  Google Scholar 

  56. Wolfe, M. B., & Goldman, S. R. (2005). Relations between adolescents' text processing and reasoning. Cognition and Instruction, 23(4), 467–502. https://doi.org/10.1207/s1532690xci2304_2.

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the projects FPU014/04646 and EST15/00492, granted by the Spanish Ministry of Education, Culture and Sport to Ignacio Máñez, and the projects EDU2014-55662-R, granted by the Spanish Ministry of Economy and Competitiveness, and PROMETEO/2013/011, granted by Conselleria d’Educació, Cultura i Esport de la Generalitat Valenciana to the Psicotext research group at the University of Valencia.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Ignacio Máñez.

Ethics declarations

Conflict of interest

The authors (Ignacio Máñez, Eduardo Vidal-Abarca, Panayiota Kendeou, Tomás Martínez) declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Máñez, I., Vidal-Abarca, E., Kendeou, P. et al. How do students process complex formative feedback in question-answering tasks? A think-aloud study. Metacognition Learning 14, 65–87 (2019). https://doi.org/10.1007/s11409-019-09192-w

Download citation

Keywords

  • Formative feedback
  • Metacognitive processes
  • Question-answering
  • Reading comprehension
  • Think-aloud