Assessing Students’ Problem Solving Ability and Cognitive Regulation with Learning Trajectories

  • Ron StevensEmail author
  • Carole R. Beal
  • Marcia Sprang
Part of the Springer International Handbooks of Education book series (SIHE, volume 28)


Learning trajectories have been developed for 1650 students who solved a series of online chemistry problem solving simulations using quantitative measures of the efficiency and the effectiveness of their problem solving approaches. These analyses showed that the poorer problem solvers, as determined by item response theory analysis, were modifying their strategic efficiency as rapidly as the better students, but did not converge on effective outcomes. This trend was also observed at the classroom level with the more successful classes simultaneously improving both their problem solving efficiency and effectiveness. A strong teacher effect was observed, with multiple classes of the same teacher showing consistently high or low problem solving performance.

The analytic approach was then used to better understand how interventions designed to improve problem solving exerted their effects. Placing students in collaborative groups increased both the efficiency and effectiveness of the problem solving process, while providing pedagogical text messages increased problem solving effectiveness, but at the expense of problem solving efficiency.


Artificial Neural Network Hide Markov Modeling Item Response Theory Intelligent Tutor System Learn Trajectory 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



Supported in part by National Science Foundation Grants DUE 0512203 and ROLE 0528840 and by a grant from the US Department of Education’s Institute of Education Sciences (R305H050052).


  1. Alberts, B. (2009). Redefining science education. Science, 323(5913), 437.CrossRefGoogle Scholar
  2. Anderson, J. R., Carter, C., & Koedinger, K. R. (2000). Tracking the course of mathematics problems. National Science Foundation, ROLE Award, Carnegie Mellon University.CrossRefGoogle Scholar
  3. Ayersman, D. J. (1995). Effects of knowledge representation format and hypermedia instruction on metacognitive accuracy. Computers in Human Behavior, 11(3–4), 533–555.Google Scholar
  4. Beal, C. R., Mitra, S., & Cohen, P. R. (2007). Modeling learning patterns of students with a tutoring system using Hidden Markov Models. Proceedings of the13th International Conference on Artificial Intelligence in Education. Amsterdam: IOS press.Google Scholar
  5. Beal, C.R., & Stevens, R. (2011). Improving students’ problem solving in a web-based chemistry simulation through embedded metacognitive messages. Tech­nology Instruction, Cognition and Learning, 8(3–4) 255–271.Google Scholar
  6. Case, J., Gunstone, R., & Lewis, A. (2001). Students’ metacognitive development in an innovative second year chemical engineering course. Research in Science Education, 31, 313–335.CrossRefGoogle Scholar
  7. Case, E., Stevens, R., & Cooper, M. M. (2007). Is collaborative grouping an effective instructional strategy? Using IMMEX to find new answers to an old question. Journal of College Science Teaching, 36(6), 42.Google Scholar
  8. Chung, G. K. W. K., deVries, L. F., Cheak, A. M., Stevens, R. H., & Bewley, W. L. (2002). Cognitive process validation of an online problem solving assessment. Computers and Human Behavior, 18(6), 669–684.Google Scholar
  9. Conati, C., & Zhao, X. (2004). Building and evaluating an intelligent pedagogical agent to improve the effectiveness of an educational game. Proceedings of the 9th International Conference on Intelligent user Interfaces (pp. 6–13). Funchal, Madeira, Portugal.Google Scholar
  10. Cooper, M. M., Cox, C. T., Jr., Nammouz, M., Case, E., & Stevens, R. H. (2008). An assessment of the effect of collaborative groups on students’ problem solving strategies and abilities. Journal of Chemical Education, 85(6), 866–872.CrossRefGoogle Scholar
  11. Cooper, M. M., & Sandi-Urena, S. (2009). Design and validation of an instrument to assess metacognitive skillfulness in chemistry problem solving. Journal of Chemical Education, 86(2), 240–245.CrossRefGoogle Scholar
  12. Desimone, L. (2002). How can comprehensive school reform models be successfully implemented? Review of Educational Research, 72(3), 433–479.CrossRefGoogle Scholar
  13. Frederiksen, N. (1984). Implications of cognitive theory for instruction in problem solving. Review of Educational Research, 54(3), 363–407.CrossRefGoogle Scholar
  14. Georghiades, P. (2006). The role of metacognitive activities in the contextual use of primary pupils’ conceptions of science. Research in Science Education, 36, 29–49. doi: 10.1007/s11165-004-3954-8.CrossRefGoogle Scholar
  15. Haider, H., & Frensch, P. A. (1996). The role of information reduction in skill acquisition. Cognitive Psychology, 30(3), 304–337.CrossRefGoogle Scholar
  16. Hausmann, R. G., Chi, M. T. H., & Roy, M. (2004). Learning from collaborative problem solving: An analysis of three hypothesized mechanisms. 26th Annual Conference of the Cognitive Science Society (pp. 547–552). Chicago, IL.Google Scholar
  17. Heffernan, N. T., & Koedinger, K. R. (2002). An intelligent tutoring system incorporating a model of an experienced human tutor. Proceedings of the Sixth International Conference on Intelligent Tutoring Systems, Biarritz, France.Google Scholar
  18. Koedinger, K. R., Corbett, A. T., Ritter, S., & Shapiro, L. J. (2000). Carnegie learning’s cognitive tutor: Summary research results. White paper. Pittsburgh: Carnegie Learning.Google Scholar
  19. Lajoie, S. P. (2003). Transitions and trajectories for studies of expertise. Educational Researcher, 32(8), 21–25.CrossRefGoogle Scholar
  20. Lawless, K. A., & Pellegrino, J. W. (2007). Professional development in integrating technology into teaching and learning: knowns, unknowns, and ways to pursue better questions and answers. AERA Review of Educational Research, 77(4), 575–614.CrossRefGoogle Scholar
  21. Lepper, M. R., Woolverton, M., Mumme, D., & Gurtner, J. (1993). Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. In S. P. Lajoie & S. J. Derry (Eds.), Computers as cognitive tools (pp. 75–105). Hillsdale: Erlbaum.Google Scholar
  22. Linacre, J. M. (2004). WINSTEPS Rasch measurement computer program. Chicago: Scholar
  23. Loehle, C. (2009). A guide to increased creativity in research—Inspiration or perspiration. Bioscience, 40, 123–129.Google Scholar
  24. Mislevy, R. J., Almond, R. G., Yan, D., & Steinberg, L. S. (1999). Bayes nets in educational assessment: Where do the numbers come from? In K. B. Laskey & H. Prade (Eds.), Proceedings of the fifteenth conference on uncertainty in artificial intelligence (pp. 437–446). San Francisco: Morgan Kaufmann.Google Scholar
  25. Moreno, R., & Duran, R. (2004). Do multiple representations need explanations? The role of verbal guidance and individual differences in multimedia mathematics learning. Journal of Educational Psychology, 96, 492–503.CrossRefGoogle Scholar
  26. Murray, R. C. & VanLehn, K. (2000). A decision-theoretic, dynamic approach for optimal selection of tutorial actions. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Intelligent Tutoring Systems, Fifth International Conference, ITS 2000, Montreal, Canada (pp. 153–162). New York: Springer.Google Scholar
  27. Partnership for 21st Century Skills. Retrieved February 8, 2013,
  28. Quellmalz, E. S., & Pellegrino, J. W. (2009). Technology and testing. Science, 323(5910), 75–79.CrossRefGoogle Scholar
  29. Sandi-Urena, S., Cooper, M. M., & Stevens, R. H. (2010). Enhancement of metacognition use and awareness by means of a collaborative intervention. International Journal of Science Education. doi:10.1080/ 09500690903452922. Retrieved from. First published on: 2 February 2010 (iFirst).CrossRefGoogle Scholar
  30. Schauble, L. (1990). Belief revision in children. Journal of Experimental Child Psychology, 49, 31–57.CrossRefGoogle Scholar
  31. Schraw, G. (2001). Promoting general metacognitive awareness. In H. J. Hartman (Ed.), Metacognition in learning and instruction (pp. 3–16). Dordrecht: Kluwer.CrossRefGoogle Scholar
  32. Schraw, G., Brooks, D. W., & Crippen, K. J. (2005). Using an interactive, compensatory model of learning to improve chemistry teaching. Journal of Chemical Education, 82(4), 637–640.CrossRefGoogle Scholar
  33. Schraw, G., Crippen, K. J., & Hartley, K. (2006). Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education, 36, 111–139.CrossRefGoogle Scholar
  34. Schwarz, C. V., & White, B. Y. (2005). Metamodeling knowledge: Developing students’ understanding of scientific modeling. Cognition and Instruction, 23(2), 165–205.CrossRefGoogle Scholar
  35. Soller, A., & Stevens, R. H. (2007). Applications of stochastic analyses for collaborative learning and cognitive assessment. In G. Hancock & K. Samuelson (Eds.), Advances in latent variable mixture models. Charlotte: Information Age Publishing.Google Scholar
  36. Spillane, J. P., Reiser, B. J., & Reimer, T. (2002). Policy implementation and cognition: Reframing and refocusing implementation research. Review of Educational Research, 72(3), 387–431.CrossRefGoogle Scholar
  37. Stark, R., Mandl, H., Gruber, H., & Renkl, A. (1999). Instructional means to overcome transfer problems in the domain of economics: Empirical studies. International Journal of Educational Research, 31, 591–609.CrossRefGoogle Scholar
  38. Stevens, R. H., & Palacio-Cayetano, J. (2003). Design and performance frameworks for constructing problem-solving simulations. Cell Biology Education, 2(3), 162–179.CrossRefGoogle Scholar
  39. Stevens, R. H., Soller, A., Cooper, M., & Sprang, M. (2004). Modeling the development of problem solving skills in chemistry with a web-based tutor. In J. C. Lester, R. M. Vicari, & F. Paraguaca (Eds.), Intelligent Tutoring Systems, 7th International Conference Proceedings (pp. 580–591). Berlin: Springer.Google Scholar
  40. Stevens, R. H., Wang, P., & Lopo, A. (1996). Artificial neural networks can distinguish novice and expert strategies during complex problem solving. Journal of the American Medical Informatics Association, 3(2), 131–138.CrossRefGoogle Scholar
  41. Stevens, R. H. (2007). A value-based approach for quantifying student’s scientific problem solving efficiency and effectiveness within and across educational systems. In R. W. Lissitz (Ed.), Assessing and modeling cognitive development in school (pp. 217–240). Maple Grove: JAM.Google Scholar
  42. Stevens, R. H., & Thadani, V. (2007). Quantifying student’s scientific problem solving efficiency and effectiveness. Technology, Instruction, Cognition and Learning, 5(2–3–4), 325–337.Google Scholar
  43. Stevens, R. H., & Casillas, A. (2006). Artificial neural networks. In R. E. Mislevy, D. M. Williamson, & I. Bejar (Eds.), Automated scoring of complex tasks in computer based testing: An introduction (pp. 259–312). Mahwah: Lawrence Erlbaum.Google Scholar
  44. Swanson, H. L. (1990). Influence of metacognitive knowledge and aptitude on problem solving. Journal of Educational Psychology, 82(2), 306–314.Google Scholar
  45. Thadani, V., Stevens, R. H., & Tao, A. (2009). Measuring complex features of science instruction: developing tools to investigate the link between teaching and learning. Journal of the Learning Sciences, 18(2), 285–322.CrossRefGoogle Scholar
  46. Walles, R., Beal, C. R., Arroyo, I., & Woolf, B. P. (2005, April). Cognitive predictors of response to web-based tutoring. Accepted for presentation at the biennial meeting of the Society for Research in Child Development, Atlanta, GA.Google Scholar
  47. Williamson, D. M., Mislevy, R. J., & Bejar, I. I. (Eds.). (2006). Automated scoring of complex tasks in computer based testing. Mahwah: Erlbaum Associates.Google Scholar
  48. Zimmerman, C. (2007). The development of scientific thinking skills in elementary and middle school. Developmental Review, 27(2), 172–223.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.UCLA IMMEX ProjectBrain Research Institute, UCLA School of MedicineCulver CityUSA
  2. 2.School of Information ScienceUniversity of ArizonaTucsonUSA
  3. 3.Placentia-Yorba Linda Unified School DistrictAnaheimUSA

Personalised recommendations