Skip to main content
SpringerLink
Log in

Adaptive testing for hierarchical student models

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

This paper presents an approach to student modeling in which knowledge is represented by means of probability distributions associated to a tree of concepts. A diagnosis procedure which uses adaptive testing is part of this approach. Adaptive tests provide well-founded and accurate diagnosis thanks to the underlying probabilistic theory, i.e., the Item Response Theory. Most adaptive testing proposals are based on dichotomous models, where he student answer can only be considered either correct or incorrect. In the work described here, a polytomous model has been used, i.e., answers can be given partial credits. Thus, models are more informative and diagnosis is more efficient. This paper also presents an algorithm for estimating question characteristic curves, which are necessary in order to apply the Item Response Theory to a given domain and hence must be inferred before testing begins. Most prior estimation procedures need huge sets of data. We have modified preexisting procedures in such a way that data requirements are significantly reduced. Finally, this paper presents the results of some controlled evaluations that have been carried out in order to analyze the feasibility and advantages of this approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Barbero, M.I.: Gestión informatizada de bancos de ítems. In: Olea, J., Ponsoda, V., Prieto, G. (eds.) Tests Informatizados: Fundamentos y aplicaciones, pp. 63–83. Pirámide (1999)

  • Birnbaum, A.: Some latent trait models and their use in inferring an examinee’s mental ability’. In: Statistical Theories of Mental Test Scores. Addison-Wesley: Reading, MA (1968)

  • Bock R.D. (1997) The nominal categories model. In: van der Linden W.J., Hambleton R.K., (eds) Handbook of Modern Item Response Theory. Springer, Verlag New York, pp. 33–49

    Google Scholar 

  • Burton, R.: Diagnosing bugs in a simple procedural skill. In: Sleeman, D., Brown, J. (eds.) Intelligent Tutoring Systems, pp. 57–183. Academic Press (1982)

  • Carver C.A., Howard R.A., Lane W.D. (1999) Enhancing student learning through hypermedia courseware and incorporation of student learning styles. IEEE Trans. Educ. 42(1):33–38

    Article  Google Scholar 

  • Castillo, E., Gutiérrez, J.M., Hadi, A.S.: Expert Systems and Probabilistic Network Models. Monographs in Computer Science. Springer-Verlag (1997)

  • Chickering, D.M., Paek, T.: Personalizing influence diagrams: applying online learning strategies to dialogue management. User Modeling and User-Adapted Interaction (in this issue) (2007)

  • Collins, J.A.: Adaptive testing with granularity. Master’s thesis, University of Saskatchewan (1996)

  • Collins J.A., Greer J.E., Huang S.X. (1996) Adaptive assessment using granularity hierarchies and bayesian nets. In: Frasson C., Gauthier G., Lesgold A. (eds) Proceedings of the 3rd International Conference on Intelligent Tutoring Systems ITS 1996. Springer Verlag, New York, pp. 569–577, Lecture Notes in Computer Science, No. 1086

    Google Scholar 

  • Conati C. (2002) Probabilistic assessment of user’s emotions in educational games. App. Artif. Intell. 16, 555–575

    Article  Google Scholar 

  • Conejo R., Guzmán E., Millán E., Trella M., Pérez de la Cruz J.L., Ríos A. (2004) SIETTE: a Web-based tool for adaptive testing. J Artif. Intell. Educ. 14, 29–61

    Google Scholar 

  • Desmarais M.C., Pu X. (2005) a bayesian inference adaptive testing framework and its comparison with item response theory. Int. J Artif. Intell. Educ. 15, 291–323

    Google Scholar 

  • Dodd B.G., De Ayala R.J., Koch W.R. (1995) Computerized adaptive testing with polytomous items. Appl. Psychol. Measure. 19(1):5–22

    Google Scholar 

  • Domshlak, C., Joachmis, T.: Efficient and non-parametric reasoning over user preferences. User Model. User-adap. Interac. (in this issue) (2007)

  • Douglas J., Cohen A. (2001) Nonparametric item response function estimation for assessing parametric model fit. Appl. Psychol. Measure. 25(3):234–243

    Article  MathSciNet  Google Scholar 

  • Embretson S.E. (1991) A multidimensional latent trait model for measuring learning and change. Psychometrika 56(3):495–515

    Article  MATH  Google Scholar 

  • Embretson, S.E., Reise, S.P.: Item Response Theory for Psychologists. Lawrence Erlbaum, Mahwah, NJ (2000)

  • Gentner D., Stevens A.L. (eds) (1983) Mental Models. Lawrence Erlbaum, Hillsdale, NJ

    Google Scholar 

  • Greer J.E., McCalla G. (1994) Granularity-based reasoning and belief revision in student models. In: Greer J.E., McCalla G. (eds) Student Modelling: The Key to Individualized Knowledge-Based Instruction, vol. 125. Springer Verlag, New York, pp. 39–62

    Google Scholar 

  • Guzmán, E.: Un modelo de evaluación cognitiva basado en Tests Adaptativos Informatizados para el diagnóstico en Sistemas Tutores Inteligentes. Unpublished doctoral dissertation, Dpto. Lenguajes y Ciencias de la Computación. E.T.S.I. Informática., Universidad de Málaga (2005)

  • Guzmán, E., Conejo, R. Simultaneous evaluation of multiple topics in SIETTE. In: Cerri, S., , G., Paraguacu, F. (eds.) Proceedings of the 6th International Conference on Intelligent Tutoring Systems (ITS 2002). Lecture Notes in Computer Science, No. 2363, pp. 739–748. Springer Verlag, New York (2002)

  • Guzmán E., Conejo R. (2004a) A library of templates for exercise construction in an adaptive assessment system. Technol. Instruct. Cogn. Learning (TICL) 2(1–2), 21–43

    Google Scholar 

  • Guzmán E., Conejo R. (2005b) A model for student knowledge diagnosis through adaptive testing. In: Lester J.C., Vicari R.M., Paraguau F. (eds) Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004). Springer Verlag, New York, pp. 12–21, Lecture Notes in Computer Science, No. 3220

    Google Scholar 

  • Guzmán E., Conejo R. (2005a) Self-assessment in a feasible, adaptive web-based testing system. IEEE Trans. Educ. 48(4):688–695

    Article  Google Scholar 

  • Guzmán E., Conejo R. (2005b) Towards efficient item calibration in adaptive testing. In: Ardissono L., Brna P., Mitrovic A. (eds) Proceedings of the 10th International Conference on User Modeling (UM 2005). Springer Verlag, New York, pp. 414–418, Lecture Notes in Artificial Intelligence, No. 3538

    Google Scholar 

  • Habing B. (2001) Nonparametric regression and the parametric bootstrap for local dependence assessment. Appl. Psychol. Measure. 25(3):221–233

    Article  MathSciNet  Google Scholar 

  • Härdle W. (1992) Applied Nonparametric Regression. University Press, Cambridge

    Google Scholar 

  • Holt P., Dubs S., Jones M., Greer J. (1994) The state of student modelling. In: Greer J.E., McCalla G. (eds) Student Modelling: The Key to Individualized Knowledge-Based Instruction, vol. 125. Springer Verlag, New York, pp. 3–35

    Google Scholar 

  • Hontangas P., Ponsoda V., Olea J., Abad F. (2000) Los test adaptativos informatizados en la frontera del siglo XXI: una revisión. Metodología de las Ciencias del Comportamiento 2(2):183–216

    Google Scholar 

  • Horvitz, E., Paek, T.: Complementary computing: policies for transferring callers from dialog systems to human receptionists. User Model. User-Adapted Interact. (in this issue) (2007)

  • Huang, S.X.: A content-balanced adaptive testing algorithm for computer-based training systems. In: Frasson, C., Gauthier, G., Lesgold, A. (eds.). Lecture Notes in Computer Science 1086. Proceedings of the 3rd International Conference on Intelligent Tutoring Systems. ITS 1996, pp. 306–314. Springer Verlag New York (1996)

  • Junker B.W., Sijtsma K. (2001) Nonparametric item response theory in action: an overview of the Special Issue. Appl. Psychol. Measure. 25(3):211–220

    Article  MathSciNet  Google Scholar 

  • Kingsbury, G.G., Weiss, D.J.: An adaptive testing strategy for mastery decision. Psychometric Method Program Research Report 79-5, Department of Psychology. University of Minnesota (1979)

  • Lilley M., Barker T., Britton C. (2004) The development and evaluation of a software prototype for computer-adaptive testing. Comp. Educ. 43, 109–123

    Article  Google Scholar 

  • Lord F.M. (1980) Applications of item response theory to practical testing problems. Lawrence Erlbaum Associates, Hillsdale, NJ

    Google Scholar 

  • Mayo M., Mitrovic A. (2001) Optimising ITS behaviour with bayesian networks and decision theory. Int. J. Artif. Intell. Educ. 12, 124–153

    Google Scholar 

  • Melis E., Andres E., Bndenbender J., Frischauf A., Goguadze G., Libbrecht P., Pollet M., Ullrich C. (2001) ACTIVEMATH: a generic and adaptive web-based learning environment. Int. J. Artif. Intell. Educ. 12, 385–407

    Google Scholar 

  • Millán E., Pérez de la Cruz J.L. (2002) Diagnosis algorithm for student modeling diagnosis and its evaluation. User Model. User-adapted Interact. 12(2–3):281–330

    Article  MATH  Google Scholar 

  • Mislevy, R.J., Steinberg, L.S., Almond, R.G.: On the roles of task model variables in assessment design. Technical Report CSE Technical Report 500, Education Testing Service (1999)

  • Mislevy, R.J., Steinberg, L.S., Almond, R.G.: Evidence-centered assessment design. Technical Report A Submission for the NCME Award for Technical or Scientific Contributions to the Field of Educational Measurement, Education Testing Service (2000)

  • Muraki E. (1992) A generalized partial credit model: Application to an EM algorithm. Appl. Psychol. Measure. 16, 159–176

    Google Scholar 

  • Murray, T.: Formative qualitative evaluation for ‘Exploratory’ ITS research. Int. Artif. Intell. Educ.: Special Issue on Evaluation 4(2/3), 179–207 (1993)

    Google Scholar 

  • Nanas, N., Uren, V.: Exploting term dependencies for multi-topic information filtering with single user profile. User Model. User-Adapted Interact. (in this issue) (2007)

  • Owen, R.J.: A Bayesian approach to tailored testing. Research Report 69-92, Educational Testing Service (1969)

  • Owen R.J. (1975) A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. J. Am. Stat. Associ. 70(350):351–371

    Article  MATH  MathSciNet  Google Scholar 

  • Paek, T., Chickering, D.M.: Improving command and control speech recognition on mobile devices: using predictive user models for language modeling. User Model. User-Adapted Interact. (in this issue) (2007)

  • Papanikolaou K.A., Grigoriadou M., Kornikolakis H., Magoulas G.D. (2003) Personalizing the interaction in a web-based educational hypermedia system: the case of INSPIRE. User Model. User-Adapted Interact. 13, 213–267

    Article  Google Scholar 

  • Ramsay J.O. (1991) Kernel smoothing approaches to nonparametric item characteristic curve estimation. Psychometrika 56, 611–630

    Article  MATH  MathSciNet  Google Scholar 

  • Reye J. (2002) A belief net backbone for student modelling. In: Cerri S.A., Gouardres G., Paraguacu F. (eds) Proceedings of the 6th International Conference on Intelligent Tutoring Systems (ITS 2002). Springer Verlag, New York, pp. 596–604, Lecture Notes in Computer Science, No. 2363.

    Google Scholar 

  • Samejima F. (1997) The graded response model. In: van der Linden W.J., Hambleton R.K. (eds) Handbook of Modern Item Response Theory. Springer Verlag, New York, pp. 85–100

    Google Scholar 

  • Schank R.C., Cleary C. (1994) Engines for Education. Lawrence Erlbaum Associates, Hillscale, NJ

    Google Scholar 

  • Scriven M. (1967) The methodology of evaluation. In: Stake R.E. (eds) Perspectives of Curriculum Evaluation. Rand McNally, Chicago, IL

    Google Scholar 

  • Segall D.O. (1996) Multidimensional adaptive testing. Psychometrika 61(1):331–354

    Article  MATH  MathSciNet  Google Scholar 

  • Self J.A. (1994) Formal approaches to student modeling. In: Greer J.E., McCalla G. (eds) Student Modeling: The Key to Individualized Knowledge-Based Instruction, vol. 125. Springer Verlag, New York, pp. 295–352

    Google Scholar 

  • Shute, V.J., Graf, E.A., Hansen, E.G.: Technology-Based Education: Bringing Researchers and Practicioners Together, Chapt. Design Adaptive, Diagnostic Math Assessments for Sighted and Visually Disabled Students, pp. 169–202. Information Age Publishing (2005)

  • Simonoff J.S. (1996) Smoothing Methods in Statistics. Springer-Verlag, New York

    MATH  Google Scholar 

  • Stout W. (2002) Psychometrics: from practice to theory and back. Psychometrika 67, 485–518

    Article  MathSciNet  Google Scholar 

  • Tam, S.S.: A comparison of methods for adaptive estimation of a multidimensional trait. Ph.D. thesis, Graduate School of Arts and Science, Columbia University. Order number: 9221219 (1992)

  • Tatsuoka K. (1985) A probabilistic model for diagnosing misconceptions in the pattern classification approach. J. Educ. Stat. 12(1):55–73

    Article  Google Scholar 

  • Thissen D., Steinberg L. (1997) A response model for multiple choice items. In: van der Linden W.J., Hambleton R.K. (eds) Handbook of modern item response theory. Springer Verlag, New York, pp. 51–65

    Google Scholar 

  • Twidale, M.: Redressing the balance: the advantages of informal evaluation techniques for intelligent learning environments. Int. J. Artif. Intell. Educ. Special Issue on Evaluation 4(2/3), 155–178 (1993)

  • van der Linden, W.J., Glas, C.A.W.: Computerized Adaptive Testing: Theory and Practice. Kluwer Academic Publishers (2000)

  • van der Linden, W.J., Pashley, P.J.: Item selection and ability estimation in Adaptive Testing. In: van der Linden, W.J., Glas, C.A.W. (eds.) Computerized Adaptive Testing: Theory and Practice, pp. 1–26. Kluwer Academic Publisher, Dordrecht

  • VanLehn, K., Niu, Z., Siler, S., Gertner, A.S.: student modeling from conventional test data: a Bayesian approach without priors. In: Goettl, B., Redfield, C.L., Halff, H.M., Shute, V.J. (eds.) Proceedings of 4th International Conference on Intelligent Tutoring Systems. ITS’98. Lecture Notes in Computer Science, vol. 1452. pp. 434–443 (1998)

  • Vassileva, J.: Dynamic course generation on the WWW. In: du Bolay, B., Mizoguchi, R. (eds.) Knowledge and Media in Learning Systems. Proceedings of the 8th World Conference on Artificial Intelligence in Education AIED’97. pp. 498–505 (1997)

  • Wand M.P., Jones M.C. (1995) Kernel Smoothing. Chapman and Hall, London

    MATH  Google Scholar 

  • Wang W.C., Chen P.H. (2004) Implementation and measurement efficiency of multidimensional computerized adaptive testing. Appl. Psychol. Measure. 28(5):295–316

    Article  Google Scholar 

  • Weber G., Brusilovsky P. (2001) ELM-ART: an adaptive versatile system for web-based instruction. Int. J. Artif. Intell. Educ. 12, 351–383

    Google Scholar 

  • Zukerman I., Albrecht D. (2001) Predictive statistical models for user modeling. User Model. User-Adapted Interact. 11, 5–18

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dpto. de Lenguajes y Ciencias de la Computación, Universidad de Málaga, Bulevar Louis Pasteur, 35, Campus de Teatinos, 29071, Málaga, Spain

    Eduardo Guzmán, Ricardo Conejo & José-Luis Pérez-de-la-Cruz

Authors
  1. Eduardo Guzmán
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ricardo Conejo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. José-Luis Pérez-de-la-Cruz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eduardo Guzmán.

Additional information

This paper (or a similar version) is not currently under review by a journal or conference, nor will it be submitted to such within the next three months.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Guzmán, E., Conejo, R. & Pérez-de-la-Cruz, JL. Adaptive testing for hierarchical student models. User Model User-Adap Inter 17, 119–157 (2007). https://doi.org/10.1007/s11257-006-9018-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-006-9018-1

Keywords

Search

Navigation