Instructional Science

, Volume 23, Issue 5–6, pp 433–452 | Cite as

Modeling the student in intelligent tutoring sytems: The promise of a new psychometrics

  • Howard T. Everson
Articles

Abstract

This paper reviews a number of relatively new and promising psychometric approaches to the problem of modeling student achievement (the student model) within intelligent tutoring systems (ITS). A shared characteristic of most ITSs is their need to estimate a model of the student's understanding of the domain, and use this model to modify and adapt subsequent instructional content and sequence. Sound cognitive diagnosis and the need to advance ITS technology require the development of student models that are integrated with cognitive theory and instructional science. A number of cognitively oriented psychometric approaches — including latent-trait models, statistical pattern recognition methods, and causal probabilistic networks — are described and discussed within the current ITS framework. As measurement-based student models are refined, we anticipate their compatibility with future generations of intelligent tutoring systems.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Andersen, J. R. (1993).Rules of the Mind. Hillsdale, NJ: Erlbaum Associates.Google Scholar
  2. Barr, A. & Feigenbaum, E. A. (1982).The Handbook of Artificial Intelligence, 2. Los Altos: Kaufman.Google Scholar
  3. Bélan, A. & Mislevy, R. J. (1992). Probability-based inference in a domain of proportional reasoning tasks.ETS Research Report 92-15-ONR. Princeton, NJ: Educational Testing Service.Google Scholar
  4. Bennett, R. E. (1993). Toward intelligent assessment: An integration of constructed-response testing, artificial intelligence, and model-based measurement, in N. Frederiksen, R. J. Mislevy & I. Bejar, eds.,Test Theory for a New Generation of Tests. Hiilsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  5. Bennett, R. E., Enright, M. & Tatsuoka, K. (1991).Developing Measurement Approaches for Constructed-Response Formats. ETS Technical Report, Princeton, NJ: Educational Testing Service.Google Scholar
  6. Bennett, R. E. & Ward, W. C. (eds.) (1993).Construction Versus Choice in Cognitive Measurement: Issues in Constructed Response, Performance Testing, and Portfolio Assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  7. Birenbaum, M., Kelly, A. E. & Tatsuoka, K. (1992).Diagnosing Knowledge States in Algebra Using the Rule Space Model (ETS Tech. Rep. RR-92-57-ONR). Princeton, NJ: Educational Testing Service.Google Scholar
  8. Burton, R. R. (1982). Diagnosing bugs in a simple procedural skill, in D. H. Sleeman & J. S. Brown, eds.,Intelligent Tutoring Systems (pp. 157–184). London: Academic Press.Google Scholar
  9. Caudill, M. & Butler, C. (1992).Understanding Neural Networks, Vol. I. MA: Massachusetts Institute of Technology.Google Scholar
  10. Chance, D., Cheung, J. & Pagan, J. (1992).Short Term Forecasting by Neural Networks While Incorporating Known Characteristics. Paper submitted for presentation to the International Conference on Neural Networks.Google Scholar
  11. Dickieson, J. & Gollub, L. (1992).Artificial Neural Networks and Training. Independent research and independent exploratory development programs: FY91 annual report. San Diego, CA: Navy Personnel Research and Development Center.Google Scholar
  12. Doignon, J. P. & Falmagne, J. C. (1985). Spaces for the assessment of knowledge.International Journal of Man-Machine Studies 23: 175–196.Google Scholar
  13. Embretson, S. E. (1985). Multicomponent latent trait models for test design, in S. E. Embretson, ed.,Test Design: Developments inPsychology and Psychometrics (pp. 195–218). Orlando, FL: Academic Press.Google Scholar
  14. Embretson, S. E. (1990). Diagnostic testing by measuring learning processes: Psychometric considerations for dynamic testing, in N. Frederiksen, R. L. Glaser, A. M. Lesgold & M. G. Shafto, eds.,Diagnostic Monitoring of Skill and Knowledge Acquisition. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  15. Everson, H. T., Chance, D. & Lykins, S. (1994).Exploring the Use of Artificial Neural Networks in Educational Research. Paper presented at the American Educational Research Association meeting, New Orleans, LA.Google Scholar
  16. Centner, D. & Gentner, D. R. (1983). Flowing waters or teeming crowds: Mental models of electricity, in D. Gentner & A. Stevens, eds.,Mental models (pp. 99–129). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  17. Gitomer, D. H., Steinberg, L. S. & Mislevy, R. J. (1995). Diagnostic assessment of trouble-shooting skill in an intelligent tutoring system, in P. Nichols, S. Chipman & R. Brennan, eds.,Cognitive Diagnostic Assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  18. Gitomer, D. H. & Rock, D. (1993). Addressing process variable in test analysis, in N. Frederiksen, R. J. Mislevy & I. Bejar, eds.,Test Theory for a New Generation of Tests. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  19. Gitomer, D. H. & Yamamoto, K. (1991). Performance modeling that integrates latent trait and latent class theory.Journal of Educational Measurement 28: 173–189.Google Scholar
  20. Hambleton, R. K. & Swaminathan, H. (1984).Item Response Theory: Principles and Applications. Boston, MA: Kluwer Academic Publishers.Google Scholar
  21. Harp, S. A., Samad, T. & Villano, M. (1993).Modeling Student Knowledge with Self-Organizing Feature Maps (Armstrong Laboratory Tech. Rep. AL-TR-1992-0114). TX: U.S. Air Force.Google Scholar
  22. Johnson, W. L. & Soloway, E. M. (1984). PROUST: Knowledge-based program debugging.Proceedings of the Seventh International Software Engineering Conference, Orlando, FL.Google Scholar
  23. Katz, I. R., Martinez, M. E., Sheehan, K. M. & Tatsuoka, K. (1993).Extending the Rule Space Model to a Semantically Rich Domain: Diagnostic Assessment in Architecture (ETS Tech. Rep. RR-93-42-ONR). Princeton, NJ: Educational Testing Service.Google Scholar
  24. Kohonen, T. E. (1984).Self-Organization and Associative Memory NY: Springer-Verlag.Google Scholar
  25. Lauritzen, S. L. & Spiegelhalter, D. J. (1988). Local computations with probabilities on graphical structures and their application to expert systems.Journal of the Royal Statistical Society, Series B 50: 157–224.Google Scholar
  26. Lord, F. M. & Novick, M. R. (1968).Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.Google Scholar
  27. Lykins, S. & Chance, D. (1992).Comparing Artificial Neural Networks and Multiple Regression for Predictive Application. Paper presented at Southwestern Psychological Society, Austin, TX.Google Scholar
  28. Masters, G. & Mislevy, R. J. (1993). New views of student learning: Implications for educational measurement, in N. Frederiksen, R. J. Mislevy & I. Bejar, eds.,Test Theory for a New Generation of Tests. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  29. Martin, J. D. & VanLehn, K. (1993). OLEA: Progress toward a multi-activity, Bayesian student modeler, in S. P. Brna, S. Ohlsson & H. Pain, eds.,Artificial Intelligence in Education: Proceedings of the International Conference on Artificial Intelligence in Education (pp. 41–417). Charlottesville, VA: Association for the Advancement in Computing in Education.Google Scholar
  30. Martin, J. D. & VanLehn, K. (in press). Student assessment using Bayesian nets. To appear in theProceedings of the Association of Computing Machines.Google Scholar
  31. Mislevy, R. J. (1993). Foundations of a new test theory, in N. Frederiksen, R. J. Mislevy & I. Bejar, eds.,Test Theory for a New Generation of Tests. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  32. Mislevy, R. J. (1994). Evidence and inference in educational assessment.Psychometrika 59(4): 439–584.Google Scholar
  33. Mislevy, R. J. (1995). Probability-based inference in cognitive diagnosis, in P. Nichols, S. Chipman & R. Brennan, eds.,Cognitive Diagnostic Assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  34. Mislevy, R. J. & Verhelst, N. (1990). Modeling item responses when different subjects employ different solution strategies.Psychometrika 55: 195–215.Google Scholar
  35. Mislevy, R. J., Wingersky, M. S., Irvine, S. H. & Dann, P. L. (1991). Resolving mixtures of strategies in spatial visualization tasks.British Journal of Mathematical and Statistical Psychology 44: 265–288.Google Scholar
  36. National Council of Teachers of Mathematics (1989).Curriculum and Evaluation Standards for School Mathematics. Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  37. Nelson, M. M. & Illgworth, W. T. (1991).A Practical Guide to Neural Nets. Reading, MA: Addison-Wesley.Google Scholar
  38. Nichols, P., Chipman, S. & Brennan, R. (eds.) (1995).Cognitive Diagnostic Assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  39. Nunnally, J. C. & Bernstein, I. H. (1994).Psychometric Theory. NY: McGraw-Hill.Google Scholar
  40. Nwana, H. S. (1990). Intelligent tutoring systems: An overview.Artificial Intelligence Review 4(4): 251–257.Google Scholar
  41. Ohlsson, S. (1987). Some principles of intelligent tutoring, in R. W. Lawler & M. Yazdani, eds.,Artificial Intelligence in Education (Vol. 1) (pp. 203–237). Norwood, NJ: Ablex.Google Scholar
  42. Pearl, J. (1988).Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Mateo, CA: Morgan Kaufmann.Google Scholar
  43. Reckase, M. D. (1985). The difficulty of test items that measure more than one ability.Applied Psychological Measurement 9: 401–412.Google Scholar
  44. Samejima, F. (1973). Homogenous case of the continuous response model.Psychometrika 38: 203–219.Google Scholar
  45. Samejima, F. (1983).A Latent Trait Model for Differential Strategies in Cognitive Processes (Technical Report ONR/RR-83-1). Knoxville, TN: University of Tennessee.Google Scholar
  46. Sands, W. A. (1991).Artificial Neural Networks: A Tool for Psychologists. Paper presented at the 33rd Annual Conference of the Military Testing Association, San Antonio, TX.Google Scholar
  47. Sands, W. A. & Wilkins, C. A. (1991).Artificial Neural Networks for Personal Selection. Paper presented at the 33rd Annual Conference of the Military Testing Association, San Antonio, TX.Google Scholar
  48. Sharda, R. & Patil, R. B. (1990).Neural Networks as Forecasting Experts: An Empirical Test. Proceedings of the International Joint Conference on Neural Networks (pp. 491–494). Washington, DC.Google Scholar
  49. Sheehan, K. M., Tatsuoka, K. K. & Lewis, C. (1993).A Diagnostic Classification Model for Document Processing Skills (ETS Tech Rep. RR-93-39-ONR). Princeton, NJ: Educational Testing Service.Google Scholar
  50. Simon, H. (1987). Guest foreward, in S. C. Shapiro, ed.,Encyclopedia of Artificial Intelligence- Vol. I. NY: John Wiley & Sons.Google Scholar
  51. Snow, R. E. & Lohman, D. F. (1989). Implications of cognitive psychology for educational measurement, in R. L. Linn, ed.,Educational Measurement (3rd ed.) (pp. 263–331). NY: Macmillian.Google Scholar
  52. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory.Journal of Educational Measurement 20(4): 345–354.Google Scholar
  53. Tatsuoka, K. K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach.Journal of Educational Statistics 12(1): 55–73.Google Scholar
  54. Tatsuoka, K. K. (1990). Toward an integration of item-response theory and cognitive error diagnoses, in N. Frederiksen, R. L. Glaser, A. M. Lesgold & M. G. Shafto, eds.,Diagnostic Monitoring of Skill and Knowledge Acquisition. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  55. Tatsuoka, K. K. (1993). Item construction and psychometric models appropriate for constructed responses, in R. E. Bennett & W. C. Ward, eds.,Construction Versus Choice in Cognitive Measurement: Issues in Constructed Response, Performance Testing, and Portfolio Assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  56. Tatsuoka, K. K. & Tatsuoka, M. M. (1983). Bug distribution and statistical pattern classification.Psychometrika 2: 193–206.Google Scholar
  57. Tatsuoka, K. K. & Tatsuoka, M. M. (1992).A Psychometrically Sound Cognitive Diagnostic Model: Effect of Remediation as Empirical Validity (ETS Research Report). Princeton, NJ: Educational Testing Service.Google Scholar
  58. VanLehn, K. (1988). Student modeling, in M. C. Poison & J. J. Richardson, eds.,Foundations of Intelligent Tutoring Systems (pp. 55–78). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  59. VanLehn, K. (1990).Mind Bugs: The Origins of Procedural Misconceptions. Cambridge, MA: MIT Press.Google Scholar
  60. Villano, M. (in press).Probabilistic Student Models: Bayesian Belief Networks and Knowledge Space Theory. Proceedings of the Second International Conference on Intelligent Tutoring Systems. NY: Springer-Verlag, Lecture Notes in Computer Science.Google Scholar
  61. Yamamoto, K. (1987).A Model that Combines IRT and Latent Class Models. Unpublished doctoral dissertation. University of Illinois, Champaign-Urbana.Google Scholar
  62. Yamamoto, K. & Gitomer, D. H. (1993). Application of a HYBRID model to at test of cognitive skill representation, in N. Frederiksen, R. J. Mislevy & I. Bejar, eds.,Test Theory for a New Generation of Tests. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar

Copyright information

© Kluwer Academic Publishers 1996

Authors and Affiliations

  • Howard T. Everson
    • 1
  1. 1.The College BoardNew York

Personalised recommendations