Assessing Students’ Problem Solving Ability and Cognitive Regulation with Learning Trajectories
Learning trajectories have been developed for 1650 students who solved a series of online chemistry problem solving simulations using quantitative measures of the efficiency and the effectiveness of their problem solving approaches. These analyses showed that the poorer problem solvers, as determined by item response theory analysis, were modifying their strategic efficiency as rapidly as the better students, but did not converge on effective outcomes. This trend was also observed at the classroom level with the more successful classes simultaneously improving both their problem solving efficiency and effectiveness. A strong teacher effect was observed, with multiple classes of the same teacher showing consistently high or low problem solving performance.
The analytic approach was then used to better understand how interventions designed to improve problem solving exerted their effects. Placing students in collaborative groups increased both the efficiency and effectiveness of the problem solving process, while providing pedagogical text messages increased problem solving effectiveness, but at the expense of problem solving efficiency.
KeywordsArtificial Neural Network Hide Markov Modeling Item Response Theory Intelligent Tutor System Learn Trajectory
Supported in part by National Science Foundation Grants DUE 0512203 and ROLE 0528840 and by a grant from the US Department of Education’s Institute of Education Sciences (R305H050052).
- Ayersman, D. J. (1995). Effects of knowledge representation format and hypermedia instruction on metacognitive accuracy. Computers in Human Behavior, 11(3–4), 533–555.Google Scholar
- Beal, C. R., Mitra, S., & Cohen, P. R. (2007). Modeling learning patterns of students with a tutoring system using Hidden Markov Models. Proceedings of the13th International Conference on Artificial Intelligence in Education. Amsterdam: IOS press.Google Scholar
- Beal, C.R., & Stevens, R. (2011). Improving students’ problem solving in a web-based chemistry simulation through embedded metacognitive messages. Technology Instruction, Cognition and Learning, 8(3–4) 255–271.Google Scholar
- Case, E., Stevens, R., & Cooper, M. M. (2007). Is collaborative grouping an effective instructional strategy? Using IMMEX to find new answers to an old question. Journal of College Science Teaching, 36(6), 42.Google Scholar
- Chung, G. K. W. K., deVries, L. F., Cheak, A. M., Stevens, R. H., & Bewley, W. L. (2002). Cognitive process validation of an online problem solving assessment. Computers and Human Behavior, 18(6), 669–684.Google Scholar
- Conati, C., & Zhao, X. (2004). Building and evaluating an intelligent pedagogical agent to improve the effectiveness of an educational game. Proceedings of the 9th International Conference on Intelligent user Interfaces (pp. 6–13). Funchal, Madeira, Portugal.Google Scholar
- Hausmann, R. G., Chi, M. T. H., & Roy, M. (2004). Learning from collaborative problem solving: An analysis of three hypothesized mechanisms. 26th Annual Conference of the Cognitive Science Society (pp. 547–552). Chicago, IL.Google Scholar
- Heffernan, N. T., & Koedinger, K. R. (2002). An intelligent tutoring system incorporating a model of an experienced human tutor. Proceedings of the Sixth International Conference on Intelligent Tutoring Systems, Biarritz, France.Google Scholar
- Koedinger, K. R., Corbett, A. T., Ritter, S., & Shapiro, L. J. (2000). Carnegie learning’s cognitive tutor: Summary research results. White paper. Pittsburgh: Carnegie Learning.Google Scholar
- Lepper, M. R., Woolverton, M., Mumme, D., & Gurtner, J. (1993). Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. In S. P. Lajoie & S. J. Derry (Eds.), Computers as cognitive tools (pp. 75–105). Hillsdale: Erlbaum.Google Scholar
- Linacre, J. M. (2004). WINSTEPS Rasch measurement computer program. Chicago: Winsteps.com.Google Scholar
- Loehle, C. (2009). A guide to increased creativity in research—Inspiration or perspiration. Bioscience, 40, 123–129.Google Scholar
- Mislevy, R. J., Almond, R. G., Yan, D., & Steinberg, L. S. (1999). Bayes nets in educational assessment: Where do the numbers come from? In K. B. Laskey & H. Prade (Eds.), Proceedings of the fifteenth conference on uncertainty in artificial intelligence (pp. 437–446). San Francisco: Morgan Kaufmann.Google Scholar
- Murray, R. C. & VanLehn, K. (2000). A decision-theoretic, dynamic approach for optimal selection of tutorial actions. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Intelligent Tutoring Systems, Fifth International Conference, ITS 2000, Montreal, Canada (pp. 153–162). New York: Springer.Google Scholar
- Partnership for 21st Century Skills. Retrieved February 8, 2013, http://www.p21.org.
- Sandi-Urena, S., Cooper, M. M., & Stevens, R. H. (2010). Enhancement of metacognition use and awareness by means of a collaborative intervention. International Journal of Science Education. doi:10.1080/ 09500690903452922. Retrieved from. http://dx.doi.org/10.1080/09500690903452922. First published on: 2 February 2010 (iFirst).CrossRefGoogle Scholar
- Soller, A., & Stevens, R. H. (2007). Applications of stochastic analyses for collaborative learning and cognitive assessment. In G. Hancock & K. Samuelson (Eds.), Advances in latent variable mixture models. Charlotte: Information Age Publishing.Google Scholar
- Stevens, R. H., Soller, A., Cooper, M., & Sprang, M. (2004). Modeling the development of problem solving skills in chemistry with a web-based tutor. In J. C. Lester, R. M. Vicari, & F. Paraguaca (Eds.), Intelligent Tutoring Systems, 7th International Conference Proceedings (pp. 580–591). Berlin: Springer.Google Scholar
- Stevens, R. H. (2007). A value-based approach for quantifying student’s scientific problem solving efficiency and effectiveness within and across educational systems. In R. W. Lissitz (Ed.), Assessing and modeling cognitive development in school (pp. 217–240). Maple Grove: JAM.Google Scholar
- Stevens, R. H., & Thadani, V. (2007). Quantifying student’s scientific problem solving efficiency and effectiveness. Technology, Instruction, Cognition and Learning, 5(2–3–4), 325–337.Google Scholar
- Stevens, R. H., & Casillas, A. (2006). Artificial neural networks. In R. E. Mislevy, D. M. Williamson, & I. Bejar (Eds.), Automated scoring of complex tasks in computer based testing: An introduction (pp. 259–312). Mahwah: Lawrence Erlbaum.Google Scholar
- Swanson, H. L. (1990). Influence of metacognitive knowledge and aptitude on problem solving. Journal of Educational Psychology, 82(2), 306–314.Google Scholar
- Walles, R., Beal, C. R., Arroyo, I., & Woolf, B. P. (2005, April). Cognitive predictors of response to web-based tutoring. Accepted for presentation at the biennial meeting of the Society for Research in Child Development, Atlanta, GA.Google Scholar
- Williamson, D. M., Mislevy, R. J., & Bejar, I. I. (Eds.). (2006). Automated scoring of complex tasks in computer based testing. Mahwah: Erlbaum Associates.Google Scholar