Highly integrated model assessment technology and tools

  • Pablo Pirnay-Dummer
  • Dirk Ifenthaler
  • J. Michael Spector
Development Article


Effective and efficient measurement of the development of skill and knowledge, especially in domains of human activity that involve complex and challenging problems, is important with regard to workplace and academic performance. However, there has been little progress in the area of practical measurement and assessment, due in part to the lack of automated tools that are appropriate for assessing the acquisition and development of complex cognitive skills. In the last 2 years, an international team of researchers has developed and validated an integrated set of assessment tools called highly integrated model assessment technology and tools (HIMATT) which addresses this deficiency. HIMATT is web-based and has been shown to scale up for practical use in educational and workplace settings, unlike many of the research tools developed solely to study basic issues in human learning and performance. In this paper, we describe the functions of HIMATT and demonstrate several applications for its use. Additionally, we present two studies on the quality and usability of HIMATT. We conclude with research suggestions for the use of HIMATT and for its further development.


Mental models Automated assessment Knowledge representation Cognitive structure 


  1. Al-Diban, S. (2002). Diagnose mentaler Modelle. Hamburg: Verlag Dr. Kovac.Google Scholar
  2. Bonato, M. (1990). Wissensstrukturierung mittels Struktur-Lege-Techniken. Eine graphentheoretische Analyse von Wissensnetzen.Google Scholar
  3. Bollobàs, B. (1998). Modern graph theory. New York: Springer.Google Scholar
  4. Collins, L. M., & Sayer, A. G. (Eds.). (2001). New methods for the analysis of change. Washington, DC: American Psychological Association.Google Scholar
  5. Davis, E. (1990). Representations of commonsense knowledge. San Mateo, CA: Morgan Kaufmann.Google Scholar
  6. Ding, Y. (2001). A review of ontologies with the semantic Web in view. Journal of Information Science, 27(6), 377–384. doi: 10.1177/016555150102700603.CrossRefGoogle Scholar
  7. Dummer, P., & Ifenthaler, D. (2005). Planning and assessing navigation in model-centered learning environments. Why learners often do not follow the path laid out for them. In G. Chiazzese, M. Allegra, A. Chifari, & S. Ottaviano (Eds.), Methods and technologies for learning (pp. 327–334). Sothhampton: WIT Press.Google Scholar
  8. Ellson, J., Gansner, E. R., Koutsofios, E., North, S. C., & Woodhull, G. (2003). GraphViz and Dynagraph. Static and dynamic graph drawing tools. Florham Park, NJ: AT&T Labs.Google Scholar
  9. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.Google Scholar
  10. Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life. Mind, Culture, and Activity, 5(3), 178–186. doi: 10.1207/s15327884mca0503_3.CrossRefGoogle Scholar
  11. Frazier, L. (1999). On sentence interpretation. Dordrecht: Kluwer.Google Scholar
  12. Harary, F. (1974). Graphentheorie. München: Oldenbourg.Google Scholar
  13. Harris, C. W. (Ed.). (1963). Problems in measuring change. Madison, WI: The University of Wisconsin Press.Google Scholar
  14. Hietaniemi, J. (2008). Graph-0.84. Retrieved May 6 2008 from
  15. Ifenthaler, D. (2006). Diagnose lernabhängiger Veränderung mentaler Modelle. Entwicklung der SMD-Technologie als methodologisches Verfahren zur relationalen, strukturellen und semantischen Analyse individueller Modellkonstruktionen. Freiburg: FreiDok.Google Scholar
  16. Ifenthaler, D. (2007). Relational, structural, and semantic analysis of graphical representations and concept maps. Paper presented at the Annual Convention of the AECT, Anaheim, CA.Google Scholar
  17. Ifenthaler, D. (2008). Practical solutions for the diagnosis of progressing mental models. In D. Ifenthaler, P. Pirnay-Dummer, & J. M. Spector (Eds.), Understanding models for learning and instruction. Essays in honor of Norbert M. Seel (pp. 43–61). New York: Springer.CrossRefGoogle Scholar
  18. Ifenthaler, D., Masduki, I., & Seel, N. M. (2008). Tracking the development of cognitive structures over time. Paper presented at the AREA 2008, New York.Google Scholar
  19. Ifenthaler, D., Pirnay-Dummer, P., & Seel, N. M. (2007). The role of cognitive learning strategies and intellectual abilities in mental model building processes. Technology, Instruction, Cognition and Learning, 5, 353–366.Google Scholar
  20. Ifenthaler, D., & Seel, N. M. (2005). The measurement of change: Learning-dependent progression of mental models. Technology, Instruction Cognition and Learning, 2(4), 317–336.Google Scholar
  21. Jackendoff, R. (1983). Semantics and cognition. Cambridge, MA: MIT Press.Google Scholar
  22. Jech, T. (2007). Set theory. New York: Springer.Google Scholar
  23. Johnson, T. E., O’Connor, D. L., Spector, J. M., Ifenthaler, D., & Pirnay-Dummer, P. (2006). Comparative study of mental model research methods: Relationships among ACSMM, SMD, MITOCAR & DEEP methodologies. In A. J. Cañas & J. D. Novak (Eds.), Concept maps: Theory, methodology, technology. Proceedings of the second international conference on concept Mapping (Vol. 1, pp. 87–94). San José: Universidad de Costa Rica.Google Scholar
  24. Johnson-Laird, P. N. (1983). Mental models. Towards a cognitive science of language, inference, and consciousness. Cambridge, UK: Cambridge University Press.Google Scholar
  25. Johnson-Laird, P. N., & Byrne, R. (1991). Deduction. Hove: Lawrence Erlbaum.Google Scholar
  26. Kruskal, J. (1964). Nonmetric multidimensional scaling: A numerical method. Psychometric Monographes, 29, 115–129. doi: 10.1007/BF02289694.CrossRefGoogle Scholar
  27. Lewin, K. (1922). Das Problem der Wissensmessung und das Grundgesetz der Assoziation. Teil 1. Psychologische Forschung, 1, 191–302. doi: 10.1007/BF00410391.CrossRefGoogle Scholar
  28. McCoon, G., & Ratcliff, R. (1992). Inference during reading. Psychological Review, 99(3), 440–466. doi: 10.1037/0033-295X.99.3.440.CrossRefGoogle Scholar
  29. McNamara, T. P. (1992). Priming and constraints it places on theories of memory and retrieval. Psychological Review, 99(4), 650–662. doi: 10.1037/0033-295X.99.4.650.CrossRefGoogle Scholar
  30. McNamara, T. P. (1994). Priming and theories of memory: A reply to Ratcliff and McCoon. Psychological Review, 101(1), 185–187. doi: 10.1037/0033-295X.101.1.185.CrossRefGoogle Scholar
  31. Pirnay-Dummer, P. (2006). Expertise und Modellbildung: MITOCAR. Freiburg: FreiDok.Google Scholar
  32. Pirnay-Dummer, P. (2007). Model inspection trace of concepts and relations. A heuristic approach to language-oriented model assessment. Paper presented at the AREA 2007, Chicago, IL.Google Scholar
  33. Pirnay-Dummer, P., Ifenthaler, D., & Johnson, T. E. (2008). Reading with the guide of automated graphical representations. How model based text visualizations facilitate learning in reading comprehension tasks. Paper presented at the AREA 2008, New York.Google Scholar
  34. Pirnay-Dummer, P., & Lachner, A. (2008). Towards model based knowledge management. A new approach to the assessment and development of organizational knowledge. Paper accepted for presentation at the Annual Convention of the AECT, Orlando, FL.Google Scholar
  35. Pirnay-Dummer, P., & Nußbickel, M. (2008). New ways to find out what is needed to know. Using the latest tools for knowledge elicitation in the processes of needs assessment. Paper presented at the AREA 2008, New York.Google Scholar
  36. Pirnay-Dummer, P., & Rauh, K. (2008). Annotations for knowledge structures. Quantitative measurability of propositions in concept maps and new approaches to mental model assessment. Paper presented at the AREA 2008, New York.Google Scholar
  37. Pirnay-Dummer, P., & Spector, J. M. (2008). Language, association, and model re-representation. How features of language and human association can be utilized for automated knowledge assessment. Paper presented at the AREA 2008, New York.Google Scholar
  38. Pirnay-Dummer, P., & Walter, S. (2008). Bridging the world’s knowledge to individual knowledge. Using latent semantic analysis and Web ontologies to implement classical and new knowledge assessment technologies. Paper presented at the AREA 2008, New York.Google Scholar
  39. Pollio, H. R. (1966). The structural basis of word association behavior. The Hague: Mouton.Google Scholar
  40. Ratcliff, R., & McCoon, G. (1994). Retrieving information from memory: Spreading-activation theories versus compound-cue theories. Psychological Review, 101(1), 177–184.CrossRefGoogle Scholar
  41. Rothmaler, P. (2000). Introduction to model theory. Amsterdam: Gordon & Breach Science Publishers.Google Scholar
  42. Russel, W. A., & Jenkins, J. J. (1954). The complete Minnesota norms for responses to 100 words from the Kent-Rosanoff word association test. Technological Report 11, University of Minnesota.Google Scholar
  43. Scheele, B., & Groeben, N. (1984). Die Heidelberger Struktur-Lege-Technik (SLT). Eine Dialog-Konsens-Methode zur Erhebung subjektiver Theorien mittlerer Reichweite. Weinheim: Beltz.Google Scholar
  44. Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge organization. Norwood, NJ: Ablex Publishing Corporation.Google Scholar
  45. Seel, N. M. (1991). Weltwissen und mentale Modelle. Göttingen: Hogrefe.Google Scholar
  46. Spector, J. M. (2006a). A methodology for assessing learning in complex and ill-structured task domains. Innovations in Education and Teaching International, 43(2), 109–120.CrossRefGoogle Scholar
  47. Spector, J. M. (2006b). Introduction to the special issue on models, simulations and learning in complex domains. Technology, Instruction, Cognition and Learning, 3(3–4), 199–204.Google Scholar
  48. Spector, J. M., & Koszalka, T. A. (2004). The DEEP methodology for assessing learning in complex domains (Final report to the National Science Foundation Evaluative Research and Evaluation Capacity Building). Syracuse, NY: Syracuse University.Google Scholar
  49. Stachowiak, F. J. (1979). Zur semantischen Struktur des subjektiven Lexikons. München: Wilhelm Fink Verlag.Google Scholar
  50. Tutte, W. T. (2001). Graph theory. Cambridge, UK: Cambridge University Press.Google Scholar
  51. Tversky, A. (1977). Features of similarity. Psychological Review, 84, 327–352.CrossRefGoogle Scholar

Copyright information

© Association for Educational Communications and Technology 2009

Authors and Affiliations

  • Pablo Pirnay-Dummer
    • 1
  • Dirk Ifenthaler
    • 1
  • J. Michael Spector
    • 2
  1. 1.Department for Educational ScienceAlbert-Ludwigs-UniversityFreiburgGermany
  2. 2.University of GeorgiaAthensUSA

Personalised recommendations