Skip to main content
Log in

Evaluating mental models in mathematics: a comparison of methods

Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

Cognitive scientists investigate mental models (how humans organize and structure knowledge in their minds) so as to understand human understanding of and interactions with the world. Cognitive and mental model research is concerned with internal conceptual systems that are not easily or directly observable. The goal of this research was to investigate the use of Evaluation of Mental Models (EMM) to assess the mental models of individuals and groups in solving complex problems and to compare novices and experts models as bases for providing feedback to learners. This study tested a qualified web-based assessment tool kit, Highly Interactive Model-based Assessment Tools and Technologies (HIMATT), in an as yet untested domain—mathematics. In this study, university students and their mathematics instructors used two tools in HIMATT, Dynamic Evaluation of Enhanced Problem Solving (DEEP) and Text-Model Inspection Trace of Concepts and Relations (T-MITOCAR). The research questions include: Do novice participants exhibit common patterns of thoughts when they conceptualize complex mathematical problems? Do novices conceptualize complex mathematical problems differently from experts? What differences in DEEP and T-MITOCAR patterns and responses exist according to the measures of HIMATT? Findings suggest that EMM and HIMATT could effectively support formative assessment in a complex mathematical domain. Finally, this study confirms a common assumption of cognitive scientists that the tool being used could affect the tool user’s understanding of the problem being solved. In this case, while DEEP and T-MITOCAR led to somewhat different expert models, both tools prove useful in support of formative assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Alexander, C. (1963). Notes on the synthesis of form. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Andrews, G., & Halford, G. S. (2002). A cognitive complexity metric applied to cognitive development. Cognitive Psychology, 45(2), 153–219.

    Article  Google Scholar 

  • Ariely, D. (2009). Predictably irrational: The hidden forces that shape our decisions. New York: Harper Collins Publishers Ltd.

    Google Scholar 

  • Boyce, W. E., & DiPrima, R. C. (2005). Elementary differential equations and boundary value problems (8th ed.). Hoboken, NJ: Wiley.

    Google Scholar 

  • Dabbagh, N. H., Jonassen, D. H., Yueh, H. P., & Samouilova, M. (2000). Assessing a problem-based learning approach to an introductory instructional design course: A case study. Performance Improvement Quarterly, 13(3), 60–83.

    Article  Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (revised ed.). Cambridge, MA: Bradford Books/MIT Press.

    Google Scholar 

  • Ericsson, K. A., & Smith, J. (1991). Prospects and limits in the empirical study of expertise: An introduction. In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 1–38). Cambridge, MA: Cambridge University Press.

    Google Scholar 

  • Fisher, K. M. (2000). SemNet® semantic networking. In K. M. Fisher, J. H. Wandersee, & D. Moody (Eds.), Mapping biology knowledge (pp. 143–166). Dordrecht, The Netherlands: Kluwer.

    Google Scholar 

  • Gentner, D., & Stevens, A. L. (1983). Mental models. Mahwah, NJ: Lawrence Erlbaum Associates Inc.

    Google Scholar 

  • Glaser, R. (1996). Changing the agency for learning: Acquiring expert performance. In K. A. Ericsson (Ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports, and games (pp. 303–311). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Gogus, A. (2009). Assessment of learning in complex domains. The First International Congress of Educational Technology, “Trends and Issues in Educational Research”, 1-3 May 2009, Canakkale, Turkey.

  • Gogus, A. (2012a). Evaluation of mental models: Using highly Interactive Model-Based Assessment Tools and Technologies (HIMATT) in mathematics domain. Technology Instruction, Cognition and Learning (TICL), 9(1), 31–50.

    Google Scholar 

  • Gogus, A. (2012b). Model-based assessment of conceptual representations. Journal of Instructional Psychology (accepted).

  • Gogus, A., & Gogus, N. G. (2009, November). Evaluation of Mental Models (EMM) in mathematics domain. Proceedings of IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2009), 19-22, 2009, Italy.

  • Gogus, A., Koszalka, T. A., & Spector, J. M. (2009). Assessing conceptual representations of ill-structured problems. Technology Instruction, Cognition and Learning, 7(1), 1–20.

    Google Scholar 

  • Herl, H. E., O’Neil, H. F, Jr, Chung, G. L., & Schacter, J. (1999). Reliability and validity of a computer-based knowledge mapping system to measure content understanding. Computers in Human Behavior, 15(3–4), 315–333.

    Article  Google Scholar 

  • Ifenthaler, D. (2006). Diagnosis of the learning-dependent progression of mental models. Development of the SMD Technology as a methodology for assessing individual models on relational, structural and semantic levels. (Dissertation, Freiburg: Universitäts, 2006).

  • Ifenthaler, D. (2009, November). Bridging the gap between expert-novice differences: The model-based feedback approach. IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2009), Rome, Italy.

  • Ifenthaler, D. (2010). Relational, structural, and semantic analysis of graphical representations and concept maps. Educational Technology Research and Development, 58(1), 81–97. doi:10.1007/s11423-008-9087-4.

    Article  Google Scholar 

  • Ifenthaler, D., Masduki, I., & Seel, N. M. (2009). The mystery of cognitive structure and how we can detect it. Tracking the development of cognitive structures over time. Instructional Science. doi:10.1007/s11251-009-9097-6.

    Google Scholar 

  • Ifenthaler, D., Pirnay-Dummer, P., & Seel, N. M. (2007). The role of cognitive learning strategies and intellectual abilities in mental model building processes. Technology, Instruction, Cognition and Learning, 5, 353–366.

    Google Scholar 

  • Ifenthaler, D., & Seel, N. M. (2005). The measurement of change: Learning-dependent progression of mental models. Technology, Instruction, Cognition, and Learning, 2(4), 317–336.

    Google Scholar 

  • Jacobson, M. J. (2000). Problem solving about complex systems: Difference between experts and novices. In B. Fishman & S. O’Connor-Divelbiss (Eds.), Fourth international conference of the learning sciences (pp. 14–21). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

    Google Scholar 

  • Johnson, T.E., O’Connor, D.L., Pirnay-Dummer, P.N., Ifenthaler, D., Spector, J.M., & Seel, N. (2006). Comparative study of mental model research methods: relationships among ACSMM, SMD, MITOCAR & DEEP methodologies. Paper presented at the Second International Conference on Concept Mapping, San Jose, Costa Rica.

  • Johnson-Laird, P. N. (1983). Mental models. Cambridge: Cambridge University Press.

    Google Scholar 

  • Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65–95.

    Article  Google Scholar 

  • Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85.

    Article  Google Scholar 

  • Jonassen, D. H., Beissner, K., & Yacci, M. (1993). Structural knowledge: Techniques for assessing, conveying, and acquiring structural knowledge. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

    Google Scholar 

  • Kim, H. (2008). An investigation of the effects of model-centered instruction in individual and collaborative contexts: The case of acquiring instructional design expertise. Unpublished dissertation, Educational Psychology & Learning Systems, Florida State University, Tallahassee, FL.

  • Kim, A. (2010). The effects of initial mental model construction on mental model progression, performance, intrinsic motivation, and metacognition in learning financial literacy. Unpublished dissertation, Educational Psychology & Learning Systems, Florida State University, Tallahassee, FL.

  • Kim, M. K. (2012). Assessment technologies for adaptive instruction: Diagnosis of stage-sequential learning progress in problem solving contexts. Unpublished dissertation, Educational Psychology & Instructional Technology, University of Georgia, Athens, GA.

  • Lakoff, G., & Johnson, M. (1981). Metaphors we live by. Chicago, IL: Chicago University Press.

    Google Scholar 

  • Lee, J. (2008). Effects of model-centered instruction and levels of learner expertise on ill-structured problem solving. Unpublihed dissertation, Educational Psychology & Learning Systems, Florida State University, Tallahassee, FL.

  • Liu, X., & Hichey, M. (1996). The internal consistency of a concept mapping scoring scheme and its effect on prediction validity. International Journal of Science Education, 18(8), 921–937.

    Article  Google Scholar 

  • Markham, K. M., Mintzes, J. J., & Jones, M. G. (1994). The concept map as a research and evaluation tool: Further evidence of validity. Journal of Research in Science Teaching, 31(1), 91–101.

    Article  Google Scholar 

  • McClure, J., Sonak, B., & Suen, H. (1999). Concept map assessment of classroom learning: Reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36, 475–492.

    Article  Google Scholar 

  • McKeown, J. O. (2008). Using annotated concept map assessments as predictors of performance and understanding of complex problems for teacher technology integration. Unpublished dissertation, Educational Psychology and Learning Systems, Florida State University, Tallahassee, FL.

  • Milrad, M., Spector, J. M., & Davidsen, P. I. (2003). Model Facilitated Learning. In S. Naidu (Ed.), Learning and teaching with technology: Principles and practices (pp. 13–27). London: Kogan Page.

    Google Scholar 

  • Payne, S. J. (1991). A descriptive study of mental models. Behaviour and Information Technology, 10(1), 3–21.

    Article  Google Scholar 

  • Perez, R. S., Johnson, J. F., & Emery, C. D. (1995). Instructional design expertise: a cognitive model of design. Instructional Science, 23(5–6), 321–349.

    Article  Google Scholar 

  • Pirnay-Dummer, P. (2007). Model inspection trace of concepts and relations. A heuristic approach to language-oriented model assessment. AREA 2007, Chicago, IL.

  • Pirnay-Dummer, P. (2010). Complete structure comparison. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (pp. 235–258). New York: Springer.

    Chapter  Google Scholar 

  • Pirnay-Dummer, P., & Ifenthaler, D. (2010). Automated knowledge visualization and assessment. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (pp. 77–115). New York: Springer.

    Chapter  Google Scholar 

  • Pirnay-Dummer, P., Ifenthaler, D., & Spector, J. M. (2008, October). Highly integrated model assessment technology and tools. Paper presented at the CELDA International Conference, Freiburg, Germany.

  • Pirnay-Dummer, P., Ifenthaler, D., & Spector, J. M. (2010). Highly integrated model assessment technology and tools. Educational Technology Research and Development, 58(1), 3–18.

    Article  Google Scholar 

  • Polk, T. A., & Seifert, C. M. (2002). Cognitive modelling. Cambridge, MA: MIT Press.

    Google Scholar 

  • Ruiz-Primo, M. A., & Shavelson, R. J. (1997). Concept-map based assessment: On possible sources of sampling variability. ERIC Document Reproduction Service No. ED422403.

  • Seel, N. M. (1991). Weltwissen und mentale Modelle. Göttingen: Hogrefe.

    Google Scholar 

  • Seel, N. M. (1999). Educational diagnosis of mental models: Assessment problems and technology-based solutions. Journal of Structural Learning and Intelligent Systems, 14(2), 153–185.

    Google Scholar 

  • Seel, N. M. (2003). Model centered learning and instruction. Technology, Instruction, Cognition and Learning, 1(1), 59–85.

    Google Scholar 

  • Seel, N. M. (2004). Model-centered learning environments: Theory, instructional design, and effects. In N. M. Seel & S. Dijkstra (Eds.), Curriculum, plans, and processes in instructional design: International perspectives. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

    Google Scholar 

  • Shin, N., Jonassen, D. H., & McGee, S. (2003). Predictors of well-structured and ill-structured problem solving in an astronomy simulation. Journal of Research in Science Teaching, 40(1), 6–33.

    Article  Google Scholar 

  • Shute, V., Torreano, L., & Willis, R. (2000). DNA: Toward an automated knowledge elicitation and organization tool. In S. P. Lajoie (Ed.), Computers as cognitive tools, Vol. two: No more walls (pp. 309–338). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

    Google Scholar 

  • Spector, J. M. (2010, October). Assessing progress of learning in complex domains. Invited paper presentation at the 11th International Conference on Education Research, Seoul, South Korea.

  • Spector, J. M., Dennen, V. P., & Koszalka, T. A. (2005). Causal maps, mental models and assessing acquisition of expertise. Technology, Instruction, Cognition and Learning, 3, 167–183.

    Google Scholar 

  • Spector, J. M., & Koszalka, T. A. (2004). The DEEP methodology for assessing learning in complex domains. Technical Report No. NSF-03-542. Syracuse, NY: Syracuse University, Instructional Design Development, and Evaluation (IDD&E).

  • Taleb, N. N. (2008). The black swan: The impact of the highly improbable. New York: Penguin.

    Google Scholar 

  • Taricani, E. M., & Clariana, R. B. (2006). A technique for automatically scoring open ended concept maps. Educational Technology Research and Development, 54(1), 61–78.

    Article  Google Scholar 

  • van Merriënboer, J. J. G. (1997). Training complex cognitive skills: A four-component instructional design model for technical training. Englewood Cliffs, NJ: Educational Technology Publications.

    Google Scholar 

  • Young, I. (2008). Mental models: Aligning design strategy with human behavior. New York: Rosenfeld Media.

    Google Scholar 

Download references

Acknowledgments

This research study is the main part of a Grant project called “Evaluation of Mental Models” supported by the Scientific & Technological Research Council of Turkey (TUBİTAK). Some parts of the results of the research study were presented and published as shown in the author’s cited references that are Gogus (2009, 2012a, 2012b), and Gogus and Gogus (2009). This is an expanded version of the paper, Gogus (2012a).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Aytac Gogus.

Appendices

Appendix 1

The first problem scenario: the hanging chain (Boyce and DiPrima 2005)

Imagine a flexible steel chain, attached firmly at equal height at both ends, hanging under its own weight (see Fig. 12). What shape will it describe as it hangs? This is a classical problem of mechanical engineering, and its analytical solution involves calculus, elementary physics, and differential equations. We describe it here.

figure 12

Fig. 12

We analyze a portion of the chain between points A and B, as shown in Fig. 13, where A is the lowest point of the chain and B = (x, y) is a variable point. We let T1 be the horizontal tension at A; T2 be the component of tension tangent to the chain at B; w be the weight of the chain per unit of length. Here T1, T2, w are numbers. Figure 14 exhibits these quantities.

figure 13

Fig. 13

figure 14

Fig. 14

Notice that if s is the length of the chain between two given points, then sw is the downward force of gravity on this portion of the chain; this is indicated in the figure. We use the symbol θ to denote the angle that the tangent to the chain at B makes with the horizontal.

Problem

Think of the hanging chain as the graph of a function: y (the height of the wire from the ground) is a function of x. Find an equation of y(x) in terms of y(0) (the height at the point A), T1 and w.

Problem instruction

Your task is to reflect on the problem scenario and then indicate how you conceptualize the problem and how you would go about solving the problem. Here, the main purpose is NOT to solve the problem. The primary interest is in understanding what you believe in relevant to developing an acceptable solution.

Appendix 2

Solution of the problem 1

Concept map was drawn (see Fig. 1). Description of each concept and links were written as below:

Concepts

Equilibrium of forces, modeling, obtaining differential equation (DE, Pythagoras, trigonometry, derivative, tangent line, slope, initial value, solving of the differential equation (DE), slope of tangent, tangent line, slope.

Descriptions of concepts

  1. 1.

    Equilibrium of forces: This is an engineering problem; one assumes that the cable has the shape of a curve y = f(x) and identifies forces acting at a particular point of the cable. In this step engineering common sense is needed. To derive the DE that governs this phenomenon one has to use vector calculus and the physical notion of equilibrium of the three acting forces. Horizontal and vertical components of force acting on a body are equal. Newton’s first law, tension, physics can be used instead of equilibrium of forces.

  2. 2.

    Modeling: Expressing a real world problem in terms of mathematics. Writing a differential equation.

  3. 3.

    Differential Equation (DE): After obtaining differential equation, one identifies the type of the DE and solves it. In this case by the method of separation of the variables.

  4. 4.

    Solving of DE: Integration is used to obtain the solution of differential equation.

  5. 5.

    Derivative: differential equation consists of a function and its derivatives. There will be a need to express the arc length variable by x and y, and this is achieved by differentiating the equation obtained from the equilibrium of the acting forces with respect to x and using the formula for the derivative of arc length in terms of dy/dx.

  6. 6.

    Tangent line—slope: The slope of the tangent line gives the derivative of the function.

  7. 7.

    Integral: derivative or differentiation is reverse operation of integration or integral.

  8. 8.

    Pythagoras—trigonometry: There will be a need to express the arc length variable by x and y. Elementary trigonometry is used for modeling.

Description of links

  1. 1.

    Step 1: This is an engineering problem; one assumes that the cable has the shape of a curve y = f(x) and identifies forces acting at a particular point of the cable. In this step engineering common sense is needed. To derive the DE that governs this phenomenon one has to use vector calculus and the physical notion of equilibrium of the three acting forces. Horizontal and vertical components of force acting on a body are equal. Newton’s first law, tension, physics can be used instead of equilibrium of forces.

  2. 2.

    Step 2: Expressing a real world problem in terms of mathematics. Writing a differential equation. Obtaining DE.

  3. 3.

    Step 3: After obtaining differential equation, one identifies the type of the DE and solve it. In this case by the method of separation of the variables.

  4. 4.

    Integration is used to obtain the solution of differential equation.

  5. 5.

    Differential equation consists of a function and its derivatives. There will be a need to express the arc length variable by x and y, and this is achieved by differentiating the equation obtained from the equilibrium of the acting forces with respect to x and using the formula for the derivative of arc length in terms of dy/dx.

  6. 6.

    Derivative or differentiation is reverse operation of integration or integral.

  7. 7.

    Elementary trigonometry is used for modeling.

  8. 8.

    Slope of tangent line equals to derivative or differentiation.

Appendix 3

Sample DEEP models from an expert participant and a novice participant

See Figs. 15 and 16.

Fig. 15
figure 15

Sample for one of the experts’ DEEP model

Fig. 16
figure 16

Sample for one of the novices’ DEEP model

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gogus, A. Evaluating mental models in mathematics: a comparison of methods. Education Tech Research Dev 61, 171–195 (2013). https://doi.org/10.1007/s11423-012-9281-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-012-9281-2

Keywords

Navigation