Skip to main content

Advertisement

SpringerLink
Log in

Highly integrated model assessment technology and tools

  • Development Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

Effective and efficient measurement of the development of skill and knowledge, especially in domains of human activity that involve complex and challenging problems, is important with regard to workplace and academic performance. However, there has been little progress in the area of practical measurement and assessment, due in part to the lack of automated tools that are appropriate for assessing the acquisition and development of complex cognitive skills. In the last 2 years, an international team of researchers has developed and validated an integrated set of assessment tools called highly integrated model assessment technology and tools (HIMATT) which addresses this deficiency. HIMATT is web-based and has been shown to scale up for practical use in educational and workplace settings, unlike many of the research tools developed solely to study basic issues in human learning and performance. In this paper, we describe the functions of HIMATT and demonstrate several applications for its use. Additionally, we present two studies on the quality and usability of HIMATT. We conclude with research suggestions for the use of HIMATT and for its further development.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Al-Diban, S. (2002). Diagnose mentaler Modelle. Hamburg: Verlag Dr. Kovac.

  • Bonato, M. (1990). Wissensstrukturierung mittels Struktur-Lege-Techniken. Eine graphentheoretische Analyse von Wissensnetzen.

  • Bollobàs, B. (1998). Modern graph theory. New York: Springer.

    Google Scholar 

  • Collins, L. M., & Sayer, A. G. (Eds.). (2001). New methods for the analysis of change. Washington, DC: American Psychological Association.

    Google Scholar 

  • Davis, E. (1990). Representations of commonsense knowledge. San Mateo, CA: Morgan Kaufmann.

    Google Scholar 

  • Ding, Y. (2001). A review of ontologies with the semantic Web in view. Journal of Information Science, 27(6), 377–384. doi:10.1177/016555150102700603.

    Article  Google Scholar 

  • Dummer, P., & Ifenthaler, D. (2005). Planning and assessing navigation in model-centered learning environments. Why learners often do not follow the path laid out for them. In G. Chiazzese, M. Allegra, A. Chifari, & S. Ottaviano (Eds.), Methods and technologies for learning (pp. 327–334). Sothhampton: WIT Press.

    Google Scholar 

  • Ellson, J., Gansner, E. R., Koutsofios, E., North, S. C., & Woodhull, G. (2003). GraphViz and Dynagraph. Static and dynamic graph drawing tools. Florham Park, NJ: AT&T Labs.

    Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.

    Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life. Mind, Culture, and Activity, 5(3), 178–186. doi:10.1207/s15327884mca0503_3.

    Article  Google Scholar 

  • Frazier, L. (1999). On sentence interpretation. Dordrecht: Kluwer.

    Google Scholar 

  • Harary, F. (1974). Graphentheorie. München: Oldenbourg.

    Google Scholar 

  • Harris, C. W. (Ed.). (1963). Problems in measuring change. Madison, WI: The University of Wisconsin Press.

    Google Scholar 

  • Hietaniemi, J. (2008). Graph-0.84. Retrieved May 6 2008 from http://search.cpan.org/~jhi/Graph-0.84/lib/Graph.pod.

  • Ifenthaler, D. (2006). Diagnose lernabhängiger Veränderung mentaler Modelle. Entwicklung der SMD-Technologie als methodologisches Verfahren zur relationalen, strukturellen und semantischen Analyse individueller Modellkonstruktionen. Freiburg: FreiDok.

    Google Scholar 

  • Ifenthaler, D. (2007). Relational, structural, and semantic analysis of graphical representations and concept maps. Paper presented at the Annual Convention of the AECT, Anaheim, CA.

  • Ifenthaler, D. (2008). Practical solutions for the diagnosis of progressing mental models. In D. Ifenthaler, P. Pirnay-Dummer, & J. M. Spector (Eds.), Understanding models for learning and instruction. Essays in honor of Norbert M. Seel (pp. 43–61). New York: Springer.

    Chapter  Google Scholar 

  • Ifenthaler, D., Masduki, I., & Seel, N. M. (2008). Tracking the development of cognitive structures over time. Paper presented at the AREA 2008, New York.

  • Ifenthaler, D., Pirnay-Dummer, P., & Seel, N. M. (2007). The role of cognitive learning strategies and intellectual abilities in mental model building processes. Technology, Instruction, Cognition and Learning, 5, 353–366.

    Google Scholar 

  • Ifenthaler, D., & Seel, N. M. (2005). The measurement of change: Learning-dependent progression of mental models. Technology, Instruction Cognition and Learning, 2(4), 317–336.

    Google Scholar 

  • Jackendoff, R. (1983). Semantics and cognition. Cambridge, MA: MIT Press.

    Google Scholar 

  • Jech, T. (2007). Set theory. New York: Springer.

    Google Scholar 

  • Johnson, T. E., O’Connor, D. L., Spector, J. M., Ifenthaler, D., & Pirnay-Dummer, P. (2006). Comparative study of mental model research methods: Relationships among ACSMM, SMD, MITOCAR & DEEP methodologies. In A. J. Cañas & J. D. Novak (Eds.), Concept maps: Theory, methodology, technology. Proceedings of the second international conference on concept Mapping (Vol. 1, pp. 87–94). San José: Universidad de Costa Rica.

  • Johnson-Laird, P. N. (1983). Mental models. Towards a cognitive science of language, inference, and consciousness. Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Johnson-Laird, P. N., & Byrne, R. (1991). Deduction. Hove: Lawrence Erlbaum.

    Google Scholar 

  • Kruskal, J. (1964). Nonmetric multidimensional scaling: A numerical method. Psychometric Monographes, 29, 115–129. doi:10.1007/BF02289694.

    Article  Google Scholar 

  • Lewin, K. (1922). Das Problem der Wissensmessung und das Grundgesetz der Assoziation. Teil 1. Psychologische Forschung, 1, 191–302. doi:10.1007/BF00410391.

    Article  Google Scholar 

  • McCoon, G., & Ratcliff, R. (1992). Inference during reading. Psychological Review, 99(3), 440–466. doi:10.1037/0033-295X.99.3.440.

    Article  Google Scholar 

  • McNamara, T. P. (1992). Priming and constraints it places on theories of memory and retrieval. Psychological Review, 99(4), 650–662. doi:10.1037/0033-295X.99.4.650.

    Article  Google Scholar 

  • McNamara, T. P. (1994). Priming and theories of memory: A reply to Ratcliff and McCoon. Psychological Review, 101(1), 185–187. doi:10.1037/0033-295X.101.1.185.

    Article  Google Scholar 

  • Pirnay-Dummer, P. (2006). Expertise und Modellbildung: MITOCAR. Freiburg: FreiDok.

    Google Scholar 

  • Pirnay-Dummer, P. (2007). Model inspection trace of concepts and relations. A heuristic approach to language-oriented model assessment. Paper presented at the AREA 2007, Chicago, IL.

  • Pirnay-Dummer, P., Ifenthaler, D., & Johnson, T. E. (2008). Reading with the guide of automated graphical representations. How model based text visualizations facilitate learning in reading comprehension tasks. Paper presented at the AREA 2008, New York.

  • Pirnay-Dummer, P., & Lachner, A. (2008). Towards model based knowledge management. A new approach to the assessment and development of organizational knowledge. Paper accepted for presentation at the Annual Convention of the AECT, Orlando, FL.

  • Pirnay-Dummer, P., & Nußbickel, M. (2008). New ways to find out what is needed to know. Using the latest tools for knowledge elicitation in the processes of needs assessment. Paper presented at the AREA 2008, New York.

  • Pirnay-Dummer, P., & Rauh, K. (2008). Annotations for knowledge structures. Quantitative measurability of propositions in concept maps and new approaches to mental model assessment. Paper presented at the AREA 2008, New York.

  • Pirnay-Dummer, P., & Spector, J. M. (2008). Language, association, and model re-representation. How features of language and human association can be utilized for automated knowledge assessment. Paper presented at the AREA 2008, New York.

  • Pirnay-Dummer, P., & Walter, S. (2008). Bridging the world’s knowledge to individual knowledge. Using latent semantic analysis and Web ontologies to implement classical and new knowledge assessment technologies. Paper presented at the AREA 2008, New York.

  • Pollio, H. R. (1966). The structural basis of word association behavior. The Hague: Mouton.

    Google Scholar 

  • Ratcliff, R., & McCoon, G. (1994). Retrieving information from memory: Spreading-activation theories versus compound-cue theories. Psychological Review, 101(1), 177–184.

    Article  Google Scholar 

  • Rothmaler, P. (2000). Introduction to model theory. Amsterdam: Gordon & Breach Science Publishers.

    Google Scholar 

  • Russel, W. A., & Jenkins, J. J. (1954). The complete Minnesota norms for responses to 100 words from the Kent-Rosanoff word association test. Technological Report 11, University of Minnesota.

  • Scheele, B., & Groeben, N. (1984). Die Heidelberger Struktur-Lege-Technik (SLT). Eine Dialog-Konsens-Methode zur Erhebung subjektiver Theorien mittlerer Reichweite. Weinheim: Beltz.

    Google Scholar 

  • Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge organization. Norwood, NJ: Ablex Publishing Corporation.

    Google Scholar 

  • Seel, N. M. (1991). Weltwissen und mentale Modelle. Göttingen: Hogrefe.

    Google Scholar 

  • Spector, J. M. (2006a). A methodology for assessing learning in complex and ill-structured task domains. Innovations in Education and Teaching International, 43(2), 109–120.

    Article  Google Scholar 

  • Spector, J. M. (2006b). Introduction to the special issue on models, simulations and learning in complex domains. Technology, Instruction, Cognition and Learning, 3(3–4), 199–204.

    Google Scholar 

  • Spector, J. M., & Koszalka, T. A. (2004). The DEEP methodology for assessing learning in complex domains (Final report to the National Science Foundation Evaluative Research and Evaluation Capacity Building). Syracuse, NY: Syracuse University.

  • Stachowiak, F. J. (1979). Zur semantischen Struktur des subjektiven Lexikons. München: Wilhelm Fink Verlag.

    Google Scholar 

  • Tutte, W. T. (2001). Graph theory. Cambridge, UK: Cambridge University Press.

    Google Scholar 

  • Tversky, A. (1977). Features of similarity. Psychological Review, 84, 327–352.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department for Educational Science, Albert-Ludwigs-University, Rempartstr. 11a, 79114, Freiburg, Germany

    Pablo Pirnay-Dummer & Dirk Ifenthaler

  2. University of Georgia, Athens, GA, USA

    J. Michael Spector

Authors
  1. Pablo Pirnay-Dummer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dirk Ifenthaler
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. Michael Spector
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pablo Pirnay-Dummer.

Appendix 1

Appendix 1

See Table 4.

Table 4 Original items of the usability questionnaire and corresponding translations
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pirnay-Dummer, P., Ifenthaler, D. & Spector, J.M. Highly integrated model assessment technology and tools. Education Tech Research Dev 58, 3–18 (2010). https://doi.org/10.1007/s11423-009-9119-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-009-9119-8

Keywords

Search

Navigation