Skip to main content

Dynamics of learning: time-varying feedback effects within the intelligent tutoring system of structure strategy (ITSS)

Abstract

The intelligent tutoring system of structure strategy (ITSS) is a web-based digital tutoring system proven to be effective in helping students recognize and use text structures to comprehend and recall texts. However, little is known about the dynamic learning processes within the ITSS. This study aims to investigate the effects of feedback dosage on lesson mastery throughout the progression of ITSS lessons. We applied a confirmatory factor analysis and extended vector autoregressive model to assess the dynamic relationships among three tasks embedded within the ITSS and found: (1) significant cross-regression effects among the three reading tasks; (2) distinct effects of feedback dosage on the specific reading task; and (3) different effect sizes of feedback across lessons. Results provide helpful insights on ways to design better modules in further development of the ITSS.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. 1.

    The participants were allowed to make up to 6 attempts in the ITSS system. Only data from the first 4 attempts were included in the analysis because most participants exceeded the minimum threshold (scoring at least 80 out of 100) to proceed to the next item by the 4th attempt, thus generating excessive missingness if we were to include data from all 6 attempts.

References

  1. Baddeley, A., Papagno, C., & Vallar, G. (1988). When long-term learning depends on short-term storage. Journal of Memory and Language, 27(5), 586–595.

    Article  Google Scholar 

  2. Bar-Shalom, Y., Li, X. R., & Kirubarajan, T. (2001). Estimation with applications to tracking and navigation: {T}heory algorithms and software. Wiley.

    Book  Google Scholar 

  3. Bringmann, L. F., Ferrer, E., Hamaker, E. L., Borsboom, D., & Tuerlinckx, F. (2018). Modeling nonstationary emotion dynamics in dyads using a time-varying vector-autoregressive model. Multivariate Behavioral Research, 3171, 1–22.

    Google Scholar 

  4. Chen, M., Chow, S.-M., Hammal, Z., Messinger, D. S., & Cohn, J. F. (2020). A person-and time-varying vector autoregressive model to capture interactive infant-mother head movement dynamics. Multivariate Behavioral Research. https://doi.org/10.1080/00273171.2020.1762065

    Article  Google Scholar 

  5. Chen, M., Chow, S.-M., & Hunter, M. D. (2018). Stochastic differential equation models with time-varying parameters. Continuous time modeling in the behavioral and related sciences (pp. 205–238). Springer.

    Chapter  Google Scholar 

  6. Chow, S. M., Grimm, K. J., Filteau, G., Dolan, C. V., & McArdle, J. J. (2013). Regime-switching bivariate dual change score model. Multivariate Behavioral Research, 48(4), 463–502. https://doi.org/10.1080/00273171.2013.787870

    Article  Google Scholar 

  7. Chow, S.-M., Haltigan, J. D., & Messinger, D. S. (2010). Dynamic patterns of infant-parent interactions during face-to-face and still-face episodes. Emotion, 10(1), 101–114.

    Article  Google Scholar 

  8. Chow, S.-M., Hamagami, F., & Nesselroade, J. R. (2007). Age differences in dynamical cognition-emotion linkages. Psychology and Aging, 22(4), 765–780.

    Article  Google Scholar 

  9. Chow, S. M., Zu, J., Shifren, K., & Zhang, G. (2011). Dynamic factor analysis models with time-varying parameters. Multivariate Behavioral Research, 46(2), 303–339. https://doi.org/10.1080/00273171.2011.563697

    Article  Google Scholar 

  10. Clarke, B., Baker, S., Smolkowski, K., & Chard, D. J. (2008). An analysis of early numeracy curriculum-based measurement: Examining the role of growth in student outcomes. Remedial and Special Education, 29(1), 46–57. https://doi.org/10.1177/0741932507309694

    Article  Google Scholar 

  11. Dalton, B., & Proctor, C. P. (2007). Reading as thinking: Integrating strategy instruction in a universally designed digital literacy environment. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 421–440). Erlbaum.

    Google Scholar 

  12. De Bot, K. (2008). Introduction: Second language development as a dynamic process. The Modern Language Journal, 92, 166–178. https://doi.org/10.1111/j.1540-4781.2008.00712.x

    Article  Google Scholar 

  13. Du Toit, S. H. C., & Browne, M. W. (2007). Structural equation modeling of multivariate time series. Multivariate Behavioral Research, 42, 67–101.

    Article  Google Scholar 

  14. Fan, J., & Zhang, W. (2008). Statistical methods with varying coefficient models. Statistics and Its Interface, 1, 2.

    Google Scholar 

  15. Goegebeur, Y., De Boeck, P., Wollack, J. A., & Cohen, A. S. (2008). A speeded item response model with gradual process change. Psychometrika, 73(1), 65–87. https://doi.org/10.1007/s11336-007-9031-2

    Article  Google Scholar 

  16. Hamilton, J. D. (1994). Time series analysis. Princeton University Press.

    Book  Google Scholar 

  17. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research., 77, 81–112.

    Article  Google Scholar 

  18. Horn, J. L., & Little, K. B. (1966). Isolating change and invariance in patterns of behavior. Multivariate Behavioral Research, 1, 219–222.

    Article  Google Scholar 

  19. Hung, L. F., & Wang, W. C. (2012). The generalized multilevel facets model for longitudinal data. Journal of Educational and Behavioral Statistics, 37(2), 231–255. https://doi.org/10.3102/1076998611402503

    Article  Google Scholar 

  20. Ji, L., & Chow, S.-M. (2018). Methodological issues and extensions to the latent difference score framework 1. In E. Ferrer, S. M. Boker, & K. J. Grimm (Eds.), Longitudinal multivariate psychology (pp. 9–37). Routledge.

    Google Scholar 

  21. Jöreskog, K. G. (1969). A general approach to confirmatory maximum likelihood factor analysis. Psychometrika, 34(2), 183–202. https://doi.org/10.1007/BF02289343

    Article  Google Scholar 

  22. Joshi, R. M., & Aaron, P. G. (2000). The component model of reading: Simple view of reading made a little more complex. Reading Psychology, 21(2), 85–97. https://doi.org/10.1080/02702710050084428

    Article  Google Scholar 

  23. Kim, S., & Camilli, G. (2014). An item response theory approach to longitudinal analysis with application to summer setback in preschool language/literacy. Large-Scale Assessments in Education, 2(1), 1–17. https://doi.org/10.1186/2196-0739-2-1

    Article  Google Scholar 

  24. Krause, U., Stark, R., & Mandl, H. (2009). The effects of cooperative learning and feedback on e-learning in statistics. Learning and Instruction, 19(2), 158–170.

    Article  Google Scholar 

  25. Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of Educational Research, 86(1), 42–78. https://doi.org/10.3102/0034654315581420

    Article  Google Scholar 

  26. Lemke, J. L. (2002). Language development and identity: Multiple timescales in the social ecology of learning (pp. 68–87). Language Acquisition and Language Socialization.

    Google Scholar 

  27. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Addison-Welsley Publishing Company.

    Google Scholar 

  28. Lütkepohl, H. (2005). Introduction to multiple time series analysis (2nd ed.). Springer-Verlag.

    Book  Google Scholar 

  29. Lysakowski, R. S., & Walberg, H. J. (1982). Instructional effects of cues, participation, and corrective feedback: A quantitative synthesis. American Educational Research Journal, 19(4), 559–572. https://doi.org/10.3102/00028312019004559

    Article  Google Scholar 

  30. Mason, L. H. (2013). Teaching students who struggle with learning to think before, while, and after reading: Effects of SRSD instruction. Reading and Writing Quarterly, 29, 124–144.

    Article  Google Scholar 

  31. Maulana, R., Opdenakker, M. C., Stroet, K., & Bosker, R. (2013). Changes in teachers’ involvement versus rejection and links with academic motivation during the first year of secondary education: A multilevel growth curve analysis. Journal of Youth and Adolescence, 42(9), 1348–1371. https://doi.org/10.1007/s10964-013-9921-9

    Article  Google Scholar 

  32. Meyer, B. J. F. (1975). The organization of prose and its effects on memory. North—Holland Press.

    Google Scholar 

  33. Meyer, B. J. F., & Poon, L. W. (2001). Effects of the structure strategy and signaling on recall of text. Journal of Educational Psychology, 93, 141–159.

    Article  Google Scholar 

  34. Meyer, B. J. F., Wijekumar, K., & Lei, P. (2018). Comparative signaling generated for expository texts by 4th–8th graders: Variations by text structure strategy instruction, comprehension skill, and signal word. Reading and Writing, 31(9), 1937–1968. https://doi.org/10.1007/s11145-018-9871-4

    Article  Google Scholar 

  35. Meyer, B. J. F., Wijekumar, K., Middlemiss, W., Higley, K., Lei, P.-W., Meier, C., & Spielvogel, J. (2010). Web-based tutoring of the structure strategy with or without elaborated feedback or choice for fifth-and seventh-grade readers. Reading Research Quarterly, 45(1), 62–92.

    Article  Google Scholar 

  36. Meyer, B. J. F., Young, C. J., & Bartlett, B. J. (1989). Memory improved: Reading and memory enhancement across the life span through strategic text structures. Erlbaum.

    Google Scholar 

  37. Molenaar, P. C. M. (1994). Dynamic factor analysis of psychophysiological signals. In J. R. Jennings, P. K. Ackles, & M. G. H. Coles (Eds.), Advances in psychophysiology: A research annual (Vol. 5, pp. 229–302). Jessica Kingsley Publishers.

    Google Scholar 

  38. Molenaar, P. C. M., De Gooijer, J. G., & Schmitz, B. (1992). Dynamic factor analysis of nonstationary multivariate time series. Psychometrika, 57(3), 333–349. https://doi.org/10.1007/BF02295422

    Article  Google Scholar 

  39. Molenaar, P., Sinclair, K. O., Rovine, M. J., Ram, N., & Corneal, S. E. (2009). Analyzing developmental processes on an individual level using nonstationary time series modeling. Developmental Psychology, 45(1), 260.

    Article  Google Scholar 

  40. Moreno, R. (2004). Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery-based multimedia. Instructional Science, 32, 99–113.

    Article  Google Scholar 

  41. Murphy, P. (2007). Reading comprehension exercises online: The effects of feedback, proficiency and interaction. Language Learning & Technology, 11(3), 107–129.

    Google Scholar 

  42. National Assessment of Educational Progress (NAEP). (2007). Retrieved December 20, 2008, from https://www.nationsreportcard.gov/reading_2007/r0003.asp

  43. Nicol, D. (2010). From monologue to dialogue: Improving written feedback processes in mass higher education. Assessment & Evaluation in Higher Education., 35, 501–517. https://doi.org/10.1080/02602931003786559

    Article  Google Scholar 

  44. Ou, L., Chow, S.-M., Ji, L., & Molenaar, P. C. M. (2017). (Re)evaluating the implications of the autoregressive latent trajectory model through likelihood ratio tests of its initial conditions. Multivariate Behavioral Research, 52, 178–199.

    Article  Google Scholar 

  45. Ou, L., Hunter, M. D., & Chow, S. M. (2019). What’s for dynr: A package for linear and nonlinear dynamic modeling in R. The R Journal, 11(1), 91–111.

    Article  Google Scholar 

  46. Pagan, A. (1980). Some identification and estimation results for regression models with stochastically varying coefficients. Journal of Econometrics, 13, 341–363. https://doi.org/10.1016/0304-4076(80)90084-6

    Article  Google Scholar 

  47. Perfetti, C., & Stafura, J. (2014). Word knowledge in a theory of reading comprehension. Scientific Studies of Reading, 18, 22–37. https://doi.org/10.1080/10888438.2013.827687

    Article  Google Scholar 

  48. Raphael, T. E., & Kirschner, B. M. (1985). The effects of instruction in compare/contrast text structure on sixth-grade students' reading comprehension and writing products. Research Series No. 161.

  49. Rijmen, F., De Boeck, P., & Van Der Maas, H. L. J. (2005). An IRT model with a parameter-driven process for change. Psychometrika, 70(4), 651–669. https://doi.org/10.1007/s11336-002-1047-z

    Article  Google Scholar 

  50. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36.

    Article  Google Scholar 

  51. Shevlin, M., & Millar, R. (2006). Career education: An application of latent growth curve modelling to career information-seeking behaviour of school pupils. British Journal of Educational Psychology, 76(1), 141–153. https://doi.org/10.1348/000709904x22386

    Article  Google Scholar 

  52. Stoel, R. D., Peetsma, T. T. D., & Roeleveld, J. (2003). Relations between the development of school investment, self-confidence, and language achievement in elementary education: A multivariate latent growth curve approach. Learning and Individual Differences, 13(4), 313–333. https://doi.org/10.1016/S1041-6080(03)00017-7

    Article  Google Scholar 

  53. Tenenbaum, G., & Goldring, E. (1989). A meta-analysis of the effect of enhanced instruction: Cues, participation, reinforcement and feedback and correctives on motor skill learning. Journal of Research and Development in Education, 22, 53–64.

    Google Scholar 

  54. Van de Gaer, E., De Fraine, B., Pustjens, H., Van Damme, J., De Munter, A., & Onghena, P. (2009). School effects on the development of motivation toward learning tasks and the development of academic self-concept in secondary education: A multivariate latent growth curve approach. School Effectiveness and School Improvement, 20(2), 235–253.

    Article  Google Scholar 

  55. Van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. Academic Press.

    Google Scholar 

  56. Vollmeyer, R., & Rheinberg, F. (2005). A surprising effect of feedback on learning. Learning and Instruction, 15, 589–602. https://doi.org/10.1016/j.learninstruc.2005.08.001

    Article  Google Scholar 

  57. Wang, Q., Molenaar, P., Harsh, S., Freeman, K., Xie, J., Gold, C., Rovine, M., & Ulbrecht, J. (2014). Personalized state-space modeling of glucose dynamics for type 1 diabetes using continuously monitored glucose, insulin dose, and meal intake: An extended Kalman filter approach. Journal of Diabetes Science and Technology, 8(2), 331–345. https://doi.org/10.1177/1932296814524080

    Article  Google Scholar 

  58. Waninge, F., Dörnyei, Z., & De Bot, K. (2014). Motivational dynamics in language learning: Change, stability, and context. The Modern Language Journal. https://doi.org/10.1111/j.1540-4781.2014.12118.x

    Article  Google Scholar 

  59. Wiederholt, J. L., & Blalock, G. (2000). Gray silent reading tests. Pro-Ed.

    Google Scholar 

  60. Wijekumar, K., Meyer, B. J. F., & Lei, P. (2012). Large-scale randomized controlled trial with 4th graders using intelligent tutoring of the structure strategy to improve nonfiction reading comprehension. Journal of Educational Technology Research and Development., 60, 987–1013.

    Article  Google Scholar 

  61. Wijekumar, K., Meyer, B. J. F., & Lei, P. (2017a). Web-based text structure strategy instruction improves seventh graders’ content area reading comprehension. Journal of Educational Psychology, 109(6), 741–760.

    Article  Google Scholar 

  62. Wijekumar, K., Meyer, B. J. F., Lei, P., Cheng, W., Ji, X., & Joshi, R. M. (2017b). Evidence of an intelligent tutoring system as a mindtool to promote strategic memory of expository texts and comprehension with children in grades 4 and 5. Journal of Educational Computing Research, 55(7), 1022–1048. https://doi.org/10.1177/0735633117696909

    Article  Google Scholar 

  63. Wijekumar, K., Meyer, B. J. F., Lei, P., Lin, Y., Johnson, L. A., Spielvogel, J. A., et al. (2014). Multisite randomized controlled trial examining intelligent tutoring of structure strategy for fifth-grade. Journal of Research on Educational Effectiveness, 7, 331–357. https://doi.org/10.1080/19345747.2013.853333

    Article  Google Scholar 

  64. Wijekumar, K., Zhang, S., Joshi, R. M., & PetiStantic, A. (2021). Introduction to the special issue: Textbook content and organization—why it matters to reading comprehension in elementary grades? Technology, Knowledge and Learning, 26(2), 243–249.

    Article  Google Scholar 

  65. Williams, J. P., Hall, K. M., Lauer, K. D., DeSisto, L. A., deCani, J. S., & Stafford, K. B. (2005). Expository text comprehension in the primary grade classroom. Journal of Educational Psychology, 97(4), 538–550. https://doi.org/10.1037/0022-0663.97.4.538

    Article  Google Scholar 

  66. Woolf, B. P. (2009). Building intelligent interactive tutors: Student centered strategies for revolutionizing e-learning. Elsevier.

    Google Scholar 

Download references

Acknowledgements

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education [grant number R305A080133] and National Science Foundation [grant number IGE-1806874]. The opinions expressed are those of the authors and do not represent views of the Institutes or the U.S. Department of Education.

Funding

This study was funded by the Institute of Education Sciences, U.S. Department of Education (grant number R305A080133) and and National Science Foundation [grant number IGE-1806874].

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jungmin Lee.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee (Penn State IRB #28763). This was a secondary data analysis with de-identified data from computer log files.

Informed consent

Informed consent was obtained from all individual participants included in the study. Consent was obtained for the original study under the IRB approval 28763.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, J., Chow, SM., Lei, P. et al. Dynamics of learning: time-varying feedback effects within the intelligent tutoring system of structure strategy (ITSS). Education Tech Research Dev 69, 2963–2984 (2021). https://doi.org/10.1007/s11423-021-10049-w

Download citation

Keywords

  • Reading comprehension
  • Human–computer interaction
  • Feedback effectiveness
  • Evaluation methodologies
  • Dynamical systems