Relational, structural, and semantic analysis of graphical representations and concept maps
- First Online:
- Cite this article as:
- Ifenthaler, D. Education Tech Research Dev (2010) 58: 81. doi:10.1007/s11423-008-9087-4
- 511 Views
The demand for good instructional environments presupposes valid and reliable analytical instruments for educational research. This paper introduces the SMD Technology (Surface, Matching, Deep Structure), which measures relational, structural, and semantic levels of graphical representations and concept maps. The reliability and validity of the computer-based and automated SMD Technology was tested in three experimental studies with 106 participants. The findings indicate a high reliability and validity. The discussion focuses on the development and realization of the three levels of the SMD Technology and applications for research, learning and instruction.
KeywordsSMD technologyAssessmentMental modelsConcept mapsKnowledge representation
The demand for good instructional environments presupposes valid and reliable tools, instruments and methodologies for educational research. However, many of them are developed with little or no theoretical justification, which leads to doubtful findings and no contribution to the improvement of learning environments (Novak 1998). Accordingly, the development of new tools, instruments and methodologies to capture key latent variables associated with human learning and cognition requires a solid theoretical foundation.
One central interest of psychological and educational research is internal cognitive processes and systems, which are described by theoretical constructs such as mental models and schemata (see Seel 1991). However, mental models and schemata are theoretical scientific constructs which are not directly observable. Accordingly, researchers can only learn about mental models or schemata if (1) individuals communicate their internal systems (Seel 1991) and if (2) valid and reliable instruments and methodologies are used to analyze them (Seel 1999). A wide variety of empirical approaches for the analysis of external representations of mental models and schemata exist (see Al-Diban 2002), but they often lack a solid theoretical foundation and their analysis is considered to be very time consuming (Ifenthaler 2006). On the other hand, new technologies such as concept mapping tools are being introduced into learning environments, but the analysis of data collected with such new technologies still places a huge demand on methodologies.
The purpose of this paper is to introduce the computer-based and automated SMD Technology for relational, structural, and semantic analysis of graphical representations and concept maps. We first introduce the theoretical constructs of mental models and schemata as a key concept for understanding human learning and problem solving processes. Second, the complex processes of externalizing internal knowledge representations (re-representation) will be discussed. Third, we introduce our own SMD Technology, which enables us to measure graphical representations and concept maps with three different quantitative indices. We then focus on the empirical reliability and validity testing of the SMD Technology. Finally, we introduce a broad field of applications for the SMD Technology within the field of research, learning, and instruction. The article ends with a conclusion and further perspectives.
Mental models are dynamic ad hoc constructions of individuals that provide subjective plausible explanations on the basis of restricted domain-specific information. Johnson-Laird (1983) describes the model building process as a step-by-step reconstruction of an initial mental model (fleshing out). Additionally, the reduction to absurdity (Seel 1991) is used to test whether the activated mental model can be replaced by another mental model. However, as long as an activated mental model provides enough subjective plausibility to meet the requirements of a phenomenon to be explained, there is no need for the construction of a new mental model. Seel (1991) assigns mental models four general functions, (1) simplification, (2) envisioning, (3) analogical reasoning, and (4) mental simulation. Depending on the objective of the model-building person, one of the four functions is used for the mental model building process. In comparison to the activation of an available schema, the mental effort for the construction of a mental model is higher and more time consuming (Seel 2008).
Accordingly, learning, reasoning, and problem solving involve the construction of mental models and schemata. In order to support successful learning, reasoning, and problem solving, it is necessary to investigate the mental model building process precisely. However, as it is not possible to measure internal representations of knowledge directly (e.g. schemata, mental models), the following paragraph will focus on the complex processes of externalizing internal knowledge representations.
Externalization of internal knowledge structures
Theoretical constructs such as the mental models and schemata discussed above are used by cognitive and educational researchers to explain the complex phenomenon of human learning, reasoning, and problem solving. As long as these internal knowledge structures are not directly observable, researchers require adequate tools, instruments, and methodologies to allow people to externalize them. According to Scandura (2007), there exist various possibilities how to construct such knowledge representations. We consider the process of externalization as a conscious process of communicating mental models or schemata using adequate sign and symbol systems (see Le Ny 1993; Ifenthaler 2006). Hence, externalization can be realized through speaking out aloud, writing a text, drawing a picture, or constructing a diagram, graphic, or concept map (see Hanke 2006).
Second, the process of externalization enables a person to communicate his or her understanding of phenomena in the world. This perspective is the only way in which researches can learn more about a person’s internal representations. Accordingly, adequate tools, instruments, and methodologies for the analysis of mental models or schemata can only be developed with a clear understanding of the complex processes of internalization and externalization. Although it appears to be possible to assess internal representations through their externalized re-representations, we need to keep in mind that the re-representations might be biased through the lack of communication skills, the use of inadequate sign and symbol systems or the use of insufficient research instruments.
Therefore we argue that instruments used for the analysis of such constructs must have a strong theoretical foundation and be tested for reliability and validity (Seel 1999; Ifenthaler and Seel 2005). A detailed review of methodologies for the assessment of graphical representations revealed a huge demand for an automated and computer-based tool (see Ifenthaler 2006). As a result, we developed our own SMD Technology.
Based on the theory of mental models (Johnson-Laird 1983; Seel 1991) and graph theory (Harary 1974; Chartrand 1977; Bonato 1990; Tittman 2003), the computer-based and automated SMD Technology (Surface, Matching, Deep Structure) uses (a) graphical representations such as concept maps or (b) natural language expressions to analyze individual processes in persons solving complex problems at single time points or multiple intervals over time. In the following, we define the externalized knowledge structures as a model M.
Raw data of a model stored pairwise (as propositions)
After the raw data has been transformed into the standardized format (see Table 1), it is stored on a SQL (structured query language) database. However, the transformation process of paper and pencil models (e.g. Structure Formation Technique) is very time consuming. Therefore, we recommend the use of computer-based elicitation techniques which already support the standardized format (e.g. DEEP; CMap; MITOCAR) in order to guarantee a more economical analysis and additionally a highly reliable transformation process (see Ifenthaler 2006).
The relational structure of each individual model Mi is represented on the Surface Structure. This simple and easily calculable indicator is computed as the sum of all propositions Pi in a model Mi.
The structural property of a model Mi is displayed on the Matching Structure. The second level of the SMD Technology indicates the range and complexity of a model Mi.
A diameter is defined as the quantity of links of the shortest path between the most distant nodes. For the calculation of the Matching Structure index, the spanning tree is transformed into a distance matrix D.
The change in range or complexity of a person’s model Mi is our second key indicator for the analysis of learning and problem solving processes (see Seel 1991; Ifenthaler 2006). Further graph theoretical such as maximum circumference (all possible relations), ruggedness (quantity of submodels which are independent or not linked), linking density (quotient of actual amount of relations and the total amount of possible relations), or node centrality (weight of a single node within a model) can be used to describe and analyze the structure of a model Mi in more detail.
The semantic composition of a model Mi is measured on the Deep Structure. The Deep Structure is calculated with the help of the similarity measure (Tversky 1977) as the semantic similarity between an individual model Mi and a reference model Mr. A reference model Mr is defined as a subject domain-specific model (e.g. expert solution; another subject’s model; the same subject’s model constructed at a different time point).
In contrast to the graph theory-based calculation of the Surface and Matching Structure, model analysis on the Deep Structure is realized through a similarity calculation between a model Mi and a domain-dependent reference model Mr. Hence, a reference model Mr of high quality is a necessary precondition for a comprehensive analysis of the Deep Structure.
The similarity between A and B is related to their commonality. The more commonality they share, the more similar they are.
The similarity between A and B is related to the differences between them. The more differences they have, the less similar they are.
The maximum similarity between A and B is reached when A and B are identical, no matter how much commonality they share.
Accordingly, the smallest similarity between two objects A and B is given if no common features exist. In this case, the two objects are completely different and the similarity measure is 0. The similarity measure increases with a rise in the number of common features. A complete similarity of all features results in a similarity measure of 1.
A and B are the amount of propositions of a model comparison. The function f(M) corresponds to the number of elements in the amount M. The parameters α and β control the weighting of similar and different features. Both similar and different features are considered in the calculation if the weighting of α and β is equal (α = β = 0.5). The value of the Deep Structure index δ is defined between 0 (no semantic similarity between the models) and 1 (absolute similarity between the models).
Thus, the semantic similarity between model M6and reference model Mris δ = 0.57 or 57%. The quantitative measures of the Surface, Matching, and Deep Structure can be used for further statistical analysis. A qualitative analysis is made possible with the standardized re-representations of the SMD Technology.
Experimental validation of the SMD Technology
To investigate the objectivity, reliability, and validity of the computer-based and automated SMD Technology, we conducted three quasi-experimental studies. The objectivity of the SMD Technology was guaranteed by the computer-based and automated realization of the instrument. In the following section we report our results for reliability and validity of the SMD Technology.
Three quasi-experimental studies (Studies 1, 2, and 3) were conducted with 106 subjects (70 female and 36 male) at the University of Freiburg. Their mean age was 18.3 years (SD = 4.6). The subject domain of Study 1 was geology and that of Studies 2 and 3 was geophysics. The subjects spent five hours on successive days working on complex problems with a multimedia discovery-learning environment.
The multimedia discovery-learning environment consisted of four modules. The modules could be divided into declarative and heuristic modules. The declarative modules contained all information needed to solve the phenomenon in question, while the heuristic modules primarily supported the model building process (see Dummer and Ifenthaler 2005).
Starting from the problem & learning task area, the subjects solve complex tasks from specific subject domains (Study 1: geology; Studies 2 and 3: geophysics). The subjects can navigate through different topics of the subject domain within the curriculum module. Additional information about the subject domain is provided in the form of various text documents, pictures, and audio recordings in the knowledge archive. The Model Building Kit (MoBuKi) provides the subjects with information about models, model building, and analogical reasoning. It contains three levels of abstraction of the material provided: (1) knowledge level; (2) procedural level; and (3) examples level. The toolbox is used to elicit the subjects’ understanding of the phenomenon in question constructing open concept maps.
The three quasi-experiments took place in the computer laboratory at the University of Freiburg. Subjects had to solve a complex problem while working with a multimedia discovery-learning environment. The problem solution had to be elicited on six subsequent measurement points as an open concept map. Every subject was given an introduction to the use and construction of open concept maps.
Pretest: Before the subjects were able to access the multimedia discovery-learning environment, a pretest was conducted which included: (a) the domain specific knowledge test; (b) elicitation of the preconception of the complex problem to be solved as an open concept map; (c) a test on cognitive learning strategies (LIST-Test); (d) a test on intellectual abilities (BIS-Test).
Model building process: During the quasi-experimental session, the subjects were asked to solve a complex problem while working within the multimedia discovery-learning environment. At five measurement points, the subjects had to elicit their understanding of the complex problem in question as an open concept map.
Posttest: The individual learning outputs were captured with: (a) a domain specific declarative knowledge test; (b) elicitation of the final solution to the complex problem as an open concept map.
The primary interest of the empirical investigation in this article is the experimental validation of the SMD Technology. Therefore, we focus in the following section on reliability and validity tests. However, details on the learning-dependent progression of externalised models and treatment effects during the three quasi-experiments are reported in detail by Ifenthaler (2006) and Ifenthaler et al. (2007).
For the computation of the test–retest reliability (Spearman’s rank correlation), the Surface, Matching, and DeepStructure indices of measurement points three and four (control group) were used.
Test–retest reliability of the SMD Technology
Especially with newly designed and developed instruments (e.g. SMD Technology), it is necessary to map theory based characteristics to measurable criteria. The goal of the construct validation is to determine from a theoretical point of view what the instrument really measures. For this purpose, several methodological best practices1 are available (see Lienert and Raatz 1994, p. 226). A comprehensive analysis of the theory of mental models (Johnson-Laird 1983; Seel 1991) and available instruments for the assessment of models constitutes the basis for the theory-based development of the SMD Technology. From an empirical point of view, the validity of the SMD Technology is identified with the outside criterion (1) MITOCAR, and (2) domain specific knowledge.
Pirnay-Dummer (2006) developed the instrument MITOCAR (Model Inspection Trace Of Concepts And Relations), which enables a structural and conceptual analysis of natural language expressions. The raw data of the third quasi-experimental study (N = 47) was analysed with the MITOCAR software, which was tested for reliability and validity (see Pirnay-Dummer 2006, p. 209). In the following, we use the results of the MITOCAR analysis for validity tests of the SMD Technology.
Correlation between the SMD Technology and MITOCAR (N = 47)
MITOCAR (Concept and structure)
MITOCAR (concept and structure)
Correlation between the SMD Technology and the declarative knowledge test (N = 47)
Applications for research, learning, and instruction
The use of different computer-based tools for re-representing knowledge structures (e.g. concept mapping software) has become increasingly accepted for research, learning, and instruction (Jonassen et al. 1997). In various research projects, concept maps have been used for analyzing learning outcomes, learners’ knowledge structures, and for self-assessment (see Mansfield and Happs 1991; Eckert 2000; Stracke 2004). In the field of learning and instruction, concept maps have been used for providing feedback and advance organizers and for facilitating problem solving tasks (see Jonassen et al. 1997; Stoyanova and Kommers 2002; Al-Diban 2002; Ifenthaler 2006). However, a large number of the available tools do not support automated feedback and analysis features. Accordingly, the development of the computer-based and automated SMD Technology opens up a broad field of applications for research, learning, and instruction.
SMD Technology and research
Re-representations of knowledge structures are often analyzed by raters using diverse scoring approaches (see Taricani and Clariana 2006; Jonassen et al. 1997). Depending on the research question, the raters focus on the quantity and quality of nodes and links, causal relationships, semantic content, direction and strength of links, hierarchy, or other visual arrangements. However, measuring the diverse information of individual concept maps by hand is very time consuming, and almost impossible for larger sets of data. Additionally, to guarantee high reliability and validity, every human rater must be an expert in the subject domain in question and in the application of quantitative and qualitative assessment strategies (Taricani and Clariana 2006). Therefore, the automated analysis procedure of the SMD Technology calculates quantitative indicators of concept maps, which then can be used for further statistical computations.
So far, the SMD Technology has been applied in different fields of mental model research. Ifenthaler (2006) investigated the trajectory of mental models constructed by subjects working on complex problem solving tasks. An HLM analysis of three quasi-experimental studies (N = 106) showed a significant increase of propositions when subjects worked for five hours in a multimedia learning environment (Surface Structure). Accordingly, as long as new information is subjective plausible it will be added to a person’s knowledge structure. Further results indicate a significant increase in the diameter of the externalized knowledge structures (Matching Structure). Consequently, we found not only a significant learning-dependent increase in the number of propositions, but also a significant learning-dependent increase in structural complexity.
In order to investigate the learning-dependent progression of novices’ mental models to more expert-like models, Ifenthaler (2006) compared the semantic similarity of externalized knowledge structures of novices with expert knowledge structures in different subject domains. The results of the Deep Structure indicator of the SMD Technology revealed a significant increase in similarity between novice and expert models. However, further HLM analysis indicated that the learning time of five hours was not long enough to integrate all information provided and consequently to gain higher similarity to an expert’s solution of a problem. Predictions about novice’s problem solving skills to become more expert like are also possible (e.g. Ifenthaler et al. 2007). Additionally, the provided learning materials and feedback could be improved for further experiments.
Ifenthaler et al. (2007) investigated the role of cognitive learning strategies and intellectual abilities in mental model building processes using the Deep Structure indicator of the SMD Technology. The results indicate that the training of mental model building skills is a complex problem which should be investigated further with regard to the roles of conditions based on the theory of mental models (Seel 1991).
Additionally, the SMD Technology has been used to investigate sharedness among team members (see Johnson et al. 2006). The focus on individually constructed concept maps and team re-representations can help to identify problems of team performance and lead to a better understanding of the complex performance processes within teams. Thanks to the flexibility of the SMD Technology, other indicators can be easily implemented in order to produce specific measures for a large number of research questions.
SMD Technology and learning and instruction
In the following, we will focus on the application of the SMD Technology for knowledge diagnosis, self assessment, and knowledge management. Other applications in the field of learning and instruction, such as analysis of navigation paths in learning environments (see Dummer and Ifenthaler 2005), could be discussed on another occasion.
In order to provide learners with the best possible learning materials, the instructor or an Intelligent Tutoring System (ITS) must be aware of their state of knowledge. In general, knowledge diagnosis is applied by collecting necessary information about the learner with the help of various tests. By integrating the SMD Technology or parts of it (graphical re-representation; quantitative indicators) either into a computer-based learning environment or other instructional settings, it can easily be applied for individual knowledge diagnosis. The SMD Technology has been implemented as a cross-platform application which enables an easy integration into a computer-based learning environment. Therefore, the instructional designer may choose which components of the SMD Technology should be applied for an adequate knowledge diagnosis. The quantitative indicators could provide instant longitudinal information about the individual learning process. The indicators (Surface, Matching, and Deep) provide multiple information about changes in the knowledge structure and domain-specific knowledge acquisition. Depending on the results of the SMD Technology, the learning environments will provide specific feedback or other instructional materials to foster future learning processes (see Ifenthaler 2006). On the other hand, the graphical re-representation of the SMD Technology can be easily applied for individual feedback on specific tasks. The instructor could use the re-representation at a specific point during the learning phase to discuss the strength and weaknesses of a learner’s learning process. Additionally, the similarity and contrast model provide further feedback materials.
Another use of the SMD Technology in the field of learning and instruction could be various fields of self assessment. As self assessment has the ambitious goal of making judgments about a learner’s own learning process, the feedback of an automated system should be very sensible to changes in the learner’s knowledge structure. As discussed above, the quantitative indicators and/or graphical re-representations of the SMD Technology could be applied for self assessment. A learner could receive quantitative information about his or her learning progress after working for a defined period with a computer-based learning environment. Additionally, the graphical re-representation could provide descriptive information about the learner’s knowledge structure. Furthermore, the similarity and contrast representation could elicit differences between previous points during the learning process or other learners or experts. This feature could therefore easily help to avoid the construction of misconceptions during self assessment phases. The major advantage of the SMD Technology for self assessment is the automated and instant generation of desired results. When learners receive the results of self assessment directly, their motivation to continue with the learning environment may be obtained longer than with other options of self assessment.
Finally, the SMD Technology could be applied for analysis of knowledge management processes. Individuals may use the quantitative indicators and or the graphical re-representations to compare it with other team members while working on a project. Also, the affordances of a task could be compared with the individual understanding of the task and gaps could be identified to solve it effectively. Another application of the SMD Technology for knowledge management could be the communication of individual or group knowledge for better cooperation and understanding with other members or groups of a project team. Further applications could include knowledge identification, knowledge use, and knowledge generation (see Tergan 2003).
Conclusion and further perspectives
The new developed SMD Technology is based on the theory of mental models (Seel 1991) and graph theory (Harary 1974) and captures key latent variables associated with human learning and cognition. Graphical representations such as concept maps or natural language expression can be analyzed on three different levels. These levels help to describe individual knowledge structures from a relational, structural, and semantic point of view. Additionally, graphical re-representations of the SMD Technology provide further information regarding the externalized knowledge structures of a person.
The objectivity, reliability, and validity of the computer-based and automated SMD Technology were investigated in three quasi-experimental studies. The results show a high reliability and validity in all indicators. Based on our findings, we developed further ideas for developing new features for the SMD Technology. These developments will include a tool for constructing concept maps, new techniques for describing the constructed models, and automated statistical reports.
Nevertheless, the SMD Technology or parts of it (graphical re-representation; quantitative indicators) can be easily integrated into various applications. The tool can be used not only in mental model research, but also in various fields of learning and instruction. Beyond this, such computer-based and automated instruments could also prove to be beneficial in a wide span of other fields of research on technology and instructional development.
Correlation of a test with several outside criteria; Correlation with tests with similar validation requirements; correlation with tests that assess other criteria; analysis of inter- and intraindividual differences in test results; factorial analysis (see Lienert and Raatz 1994).
The Deep Structure index δ of the SMD Technology compares the semantic similarity between a model and a reference model. This feature is not available with MITOCAR. Accordingly, the calculation of correlations between the Deep Structure and the MITOCAR indices is not necessary.