Educational Technology Research and Development

, Volume 58, Issue 1, pp 81–97

Relational, structural, and semantic analysis of graphical representations and concept maps

Authors

    • Department of Educational ScienceAlbert-Ludwigs-University Freiburg
Development Article

DOI: 10.1007/s11423-008-9087-4

Cite this article as:
Ifenthaler, D. Education Tech Research Dev (2010) 58: 81. doi:10.1007/s11423-008-9087-4

Abstract

The demand for good instructional environments presupposes valid and reliable analytical instruments for educational research. This paper introduces the SMD Technology (Surface, Matching, Deep Structure), which measures relational, structural, and semantic levels of graphical representations and concept maps. The reliability and validity of the computer-based and automated SMD Technology was tested in three experimental studies with 106 participants. The findings indicate a high reliability and validity. The discussion focuses on the development and realization of the three levels of the SMD Technology and applications for research, learning and instruction.

Keywords

SMD technologyAssessmentMental modelsConcept mapsKnowledge representation

Introduction

The demand for good instructional environments presupposes valid and reliable tools, instruments and methodologies for educational research. However, many of them are developed with little or no theoretical justification, which leads to doubtful findings and no contribution to the improvement of learning environments (Novak 1998). Accordingly, the development of new tools, instruments and methodologies to capture key latent variables associated with human learning and cognition requires a solid theoretical foundation.

One central interest of psychological and educational research is internal cognitive processes and systems, which are described by theoretical constructs such as mental models and schemata (see Seel 1991). However, mental models and schemata are theoretical scientific constructs which are not directly observable. Accordingly, researchers can only learn about mental models or schemata if (1) individuals communicate their internal systems (Seel 1991) and if (2) valid and reliable instruments and methodologies are used to analyze them (Seel 1999). A wide variety of empirical approaches for the analysis of external representations of mental models and schemata exist (see Al-Diban 2002), but they often lack a solid theoretical foundation and their analysis is considered to be very time consuming (Ifenthaler 2006). On the other hand, new technologies such as concept mapping tools are being introduced into learning environments, but the analysis of data collected with such new technologies still places a huge demand on methodologies.

The purpose of this paper is to introduce the computer-based and automated SMD Technology for relational, structural, and semantic analysis of graphical representations and concept maps. We first introduce the theoretical constructs of mental models and schemata as a key concept for understanding human learning and problem solving processes. Second, the complex processes of externalizing internal knowledge representations (re-representation) will be discussed. Third, we introduce our own SMD Technology, which enables us to measure graphical representations and concept maps with three different quantitative indices. We then focus on the empirical reliability and validity testing of the SMD Technology. Finally, we introduce a broad field of applications for the SMD Technology within the field of research, learning, and instruction. The article ends with a conclusion and further perspectives.

Background

Mental models and schemata are theoretical constructs for understanding human learning and problem solving processes. Following the verdict of Piaget (1950, 1976), we argue that new information is processed by the complimentary processes of assimilation and accommodation. According to Seel (1991), a person can assimilate new information as long as an adequate schema can be activated. If the activated schema does not match exactly, it can be adjusted by means of accretion, tuning, or reorganization. The accretion process is defined as an accumulation of new information to the existing schema. Tuning can be described as a change of single components within the activated schema. The result of a successful adjustment of a schema is a subjective plausible solution of a problem or the understanding of new information. However, if the processes of accretion and tuning are not successful or if no schema is available at all, new information can only be accommodated by the process of reorganization. According to Seel (1991), the process of reorganization is realized by constructing a mental model (see Fig. 1).
https://static-content.springer.com/image/art%3A10.1007%2Fs11423-008-9087-4/MediaObjects/11423_2008_9087_Fig1_HTML.gif
Fig. 1

The process of assimilation and accommodation

Mental models are dynamic ad hoc constructions of individuals that provide subjective plausible explanations on the basis of restricted domain-specific information. Johnson-Laird (1983) describes the model building process as a step-by-step reconstruction of an initial mental model (fleshing out). Additionally, the reduction to absurdity (Seel 1991) is used to test whether the activated mental model can be replaced by another mental model. However, as long as an activated mental model provides enough subjective plausibility to meet the requirements of a phenomenon to be explained, there is no need for the construction of a new mental model. Seel (1991) assigns mental models four general functions, (1) simplification, (2) envisioning, (3) analogical reasoning, and (4) mental simulation. Depending on the objective of the model-building person, one of the four functions is used for the mental model building process. In comparison to the activation of an available schema, the mental effort for the construction of a mental model is higher and more time consuming (Seel 2008).

Accordingly, learning, reasoning, and problem solving involve the construction of mental models and schemata. In order to support successful learning, reasoning, and problem solving, it is necessary to investigate the mental model building process precisely. However, as it is not possible to measure internal representations of knowledge directly (e.g. schemata, mental models), the following paragraph will focus on the complex processes of externalizing internal knowledge representations.

Externalization of internal knowledge structures

Theoretical constructs such as the mental models and schemata discussed above are used by cognitive and educational researchers to explain the complex phenomenon of human learning, reasoning, and problem solving. As long as these internal knowledge structures are not directly observable, researchers require adequate tools, instruments, and methodologies to allow people to externalize them. According to Scandura (2007), there exist various possibilities how to construct such knowledge representations. We consider the process of externalization as a conscious process of communicating mental models or schemata using adequate sign and symbol systems (see Le Ny 1993; Ifenthaler 2006). Hence, externalization can be realized through speaking out aloud, writing a text, drawing a picture, or constructing a diagram, graphic, or concept map (see Hanke 2006).

As shown in Fig. 2, we are able to distinguish between internal representations (e.g. mental models, schemata) and external re-representations (communicated using adequate sign and symbol systems). Furthermore, we argue that these two types of model representations are interrelated. First, through the process of internalization, a person is able to construct a mental model or activate an available schema. From the point of view of instructional design, the process of internalization is where we can systematically influence the construction of mental models by providing well-designed external re-representations (e.g. learning materials, feedback, etc.) of phenomena to be explained (e.g. Norman 1983).
https://static-content.springer.com/image/art%3A10.1007%2Fs11423-008-9087-4/MediaObjects/11423_2008_9087_Fig2_HTML.gif
Fig. 2

Interrelation of internal and external representations

Second, the process of externalization enables a person to communicate his or her understanding of phenomena in the world. This perspective is the only way in which researches can learn more about a person’s internal representations. Accordingly, adequate tools, instruments, and methodologies for the analysis of mental models or schemata can only be developed with a clear understanding of the complex processes of internalization and externalization. Although it appears to be possible to assess internal representations through their externalized re-representations, we need to keep in mind that the re-representations might be biased through the lack of communication skills, the use of inadequate sign and symbol systems or the use of insufficient research instruments.

Therefore we argue that instruments used for the analysis of such constructs must have a strong theoretical foundation and be tested for reliability and validity (Seel 1999; Ifenthaler and Seel 2005). A detailed review of methodologies for the assessment of graphical representations revealed a huge demand for an automated and computer-based tool (see Ifenthaler 2006). As a result, we developed our own SMD Technology.

SMD Technology

Based on the theory of mental models (Johnson-Laird 1983; Seel 1991) and graph theory (Harary 1974; Chartrand 1977; Bonato 1990; Tittman 2003), the computer-based and automated SMD Technology (Surface, Matching, Deep Structure) uses (a) graphical representations such as concept maps or (b) natural language expressions to analyze individual processes in persons solving complex problems at single time points or multiple intervals over time. In the following, we define the externalized knowledge structures as a model M.

Depending on the elicitation process (e.g. using the Structure Formation Technique [paper and pencil]; concept mapping tools [computer-based]; natural language statements [computer-based or paper and pencil]), the raw data should be stored pairwise (as propositions Pi) including (a) the modelnumber as an indicator of which model a proposition belongs to, (b) node1 as the first node of the proposition, (c) node2, which is connected to the first node, and (d) a link which describes the link between the two nodes (see Fig. 3 and Table 1).
https://static-content.springer.com/image/art%3A10.1007%2Fs11423-008-9087-4/MediaObjects/11423_2008_9087_Fig3_HTML.gif
Fig. 3

Model M3 composed of two propositions Pi

Table 1

Raw data of a model stored pairwise (as propositions)

Model number

Node1

Node2

Link

003

Cells

Animal cells

Consists of

003

Cells

Plant cells

Consists of

   

After the raw data has been transformed into the standardized format (see Table 1), it is stored on a SQL (structured query language) database. However, the transformation process of paper and pencil models (e.g. Structure Formation Technique) is very time consuming. Therefore, we recommend the use of computer-based elicitation techniques which already support the standardized format (e.g. DEEP; CMap; MITOCAR) in order to guarantee a more economical analysis and additionally a highly reliable transformation process (see Ifenthaler 2006).

The automated analysis process of the SMD Technology will be started by the researcher through the User Interface, where all stored models in the SQL database can be selected (see Fig. 4). After selecting the models Mi for the analysis process, the system will automatically calculate three numerical indicators out of all nodes and links—Surface, Matching, and Deep Structure—and generate standardized graphical re-representations for each individual model Mi(Ifenthaler 2006).
https://static-content.springer.com/image/art%3A10.1007%2Fs11423-008-9087-4/MediaObjects/11423_2008_9087_Fig4_HTML.gif
Fig. 4

User interface of the SMD Technology

Surface structure

The relational structure of each individual model Mi is represented on the Surface Structure. This simple and easily calculable indicator is computed as the sum of all propositions Pi in a model Mi.

$$ \theta = {\sum\limits_{i = 0}^n {P_{i} } } $$
(1)
θ is defined as a value between 0 (no proposition = no model) and n (n propositions Pi of a model Mi). The Surface Structure of model M3, represented in Fig. 3, would result in θ = 2. According to the theory of mental models (Seel 1991), the number of nodes and links or propositions a person uses is a key indicator for the investigation of the progression of knowledge over time in the course of problem solving processes (see Scandura 1988). However, although this first indicator enables a rapid and economical analysis of the relational structure of a model Mi, additional indicators are required for a more detailed analysis.

Matching structure

The structural property of a model Mi is displayed on the Matching Structure. The second level of the SMD Technology indicates the range and complexity of a model Mi.

$$ \mu = {\mathop {\max }\limits_{i,j} }{\left\{ {d{\left( {i,j} \right)}} \right\}} $$
(2)
μ is computed as the diameter of the spanning tree of a model Miand can lie between 0 (no links) and n. In accordance with graph theory, every model Mi contains a spanning tree. Spanning trees include all nodes of a model Miand are acyclic (see Harary 1974; Tittman 2003). Figure 5 illustrates model M5 and its corresponding spanning tree.
https://static-content.springer.com/image/art%3A10.1007%2Fs11423-008-9087-4/MediaObjects/11423_2008_9087_Fig5_HTML.gif
Fig. 5

Model M5 and its corresponding spanning tree

A diameter is defined as the quantity of links of the shortest path between the most distant nodes. For the calculation of the Matching Structure index, the spanning tree is transformed into a distance matrix D.

$$ D = {\left[ \begin{aligned}{} & 0\quad 1\quad 2\quad 3\quad 4 \\ & 1\quad 0\quad 1\quad 2\quad 3 \\ & 2\quad 1\quad 0\quad 1\quad 2 \\ & 3\quad 2\quad 1\quad 0\quad 1 \\ & 4\quad 3\quad 2\quad 1\quad 0 \\ \end{aligned} \right]}, $$
(3)
The Matching Structure index is calculated as the maximum value of all entries in the distance matrix D. The diameter or Matching Structure of the spanning tree in Fig. 5 is calculated as follows:
$$ \mu = {\mathop {\max }\limits_{i,j} }{\left\{ {d{\left( {i,j} \right)}} \right\}} = 4 $$
(4)

The change in range or complexity of a person’s model Mi is our second key indicator for the analysis of learning and problem solving processes (see Seel 1991; Ifenthaler 2006). Further graph theoretical such as maximum circumference (all possible relations), ruggedness (quantity of submodels which are independent or not linked), linking density (quotient of actual amount of relations and the total amount of possible relations), or node centrality (weight of a single node within a model) can be used to describe and analyze the structure of a model Mi in more detail.

Deep structure

The semantic composition of a model Mi is measured on the Deep Structure. The Deep Structure is calculated with the help of the similarity measure (Tversky 1977) as the semantic similarity between an individual model Mi and a reference model Mr. A reference model Mr is defined as a subject domain-specific model (e.g. expert solution; another subject’s model; the same subject’s model constructed at a different time point).

In contrast to the graph theory-based calculation of the Surface and Matching Structure, model analysis on the Deep Structure is realized through a similarity calculation between a model Mi and a domain-dependent reference model Mr. Hence, a reference model Mr of high quality is a necessary precondition for a comprehensive analysis of the Deep Structure.

A similarity measure describes the degree of similarity between two objects, represented by a number between 0 and 1. Decisive for a similarity measure are objects with similar and different features. Tversky (1977) considered an object as an amount of features. The identification of a similarity between two objects is realized through a comparison of their features. The similarity formula takes not only the amount of similar features into account, but also the amount of different features. Lin (1998) defines similarity with the following three statements:
  • The similarity between A and B is related to their commonality. The more commonality they share, the more similar they are.

  • The similarity between A and B is related to the differences between them. The more differences they have, the less similar they are.

  • The maximum similarity between A and B is reached when A and B are identical, no matter how much commonality they share.

Accordingly, the smallest similarity between two objects A and B is given if no common features exist. In this case, the two objects are completely different and the similarity measure is 0. The similarity measure increases with a rise in the number of common features. A complete similarity of all features results in a similarity measure of 1.

The similarity of models on the Deep Structure is identified through the feature “proposition”—the semantic characteristic of the proposition. The Deep Structure index δ is defined as the Tversky similarity between a model Mi and a reference model Mr. In general, we calculate:
$$ \delta = \frac{{f{\left( {A \cap B} \right)}}} {{f{\left( {A \cap B} \right)} + \alpha \cdot f{\left( {A - B} \right)} + \beta \cdot f{\left( {B - A} \right)}}} $$
(5)

A and B are the amount of propositions of a model comparison. The function f(M) corresponds to the number of elements in the amount M. The parameters α and β control the weighting of similar and different features. Both similar and different features are considered in the calculation if the weighting of α and β is equal (α = β = 0.5). The value of the Deep Structure index δ is defined between 0 (no semantic similarity between the models) and 1 (absolute similarity between the models).

The Deep Structure or semantic similarity between model M6and reference model Mris calculated in an automated iterative process. Every proposition in model M6is analysed for similarity with every proposition in the reference model Mr. The Deep Structure index is calculated as follows:
$$ \delta = 0.57 $$
(6)

Thus, the semantic similarity between model M6and reference model Mris δ = 0.57 or 57%. The quantitative measures of the Surface, Matching, and Deep Structure can be used for further statistical analysis. A qualitative analysis is made possible with the standardized re-representations of the SMD Technology.

Standardized re-representations

The standardized graphical re-representation of the subject’s data is constructed as an undirected or directed graph with named nodes and links. This automated feature of the SMD Technology is realized with the help of the open source graph visualization software GraphViz (Ellson et al. 2003). For every single analysis, four standardized PNG (Portable Network Graphics) images are generated. Images (1) and (2) are the re-representations of model Miand reference model Mr (for an example see Fig. 6). Image (3) represents the similarity model, including only the nodes and links which are semantically similar between model Miand reference model Mr (see Fig. 7).
https://static-content.springer.com/image/art%3A10.1007%2Fs11423-008-9087-4/MediaObjects/11423_2008_9087_Fig6_HTML.gif
Fig. 6

Model M6 and reference model Mr

https://static-content.springer.com/image/art%3A10.1007%2Fs11423-008-9087-4/MediaObjects/11423_2008_9087_Fig7_HTML.gif
Fig. 7

Similarity re-representation of model M6 and reference model Mr

Image (4) is defined as the contrast model. It includes only nodes and links which have no semantic similarity within model Miand reference model Mr (see Fig. 8).
https://static-content.springer.com/image/art%3A10.1007%2Fs11423-008-9087-4/MediaObjects/11423_2008_9087_Fig8_HTML.gif
Fig. 8

Contrast re-representation of model M6 and reference model Mr

Experimental validation of the SMD Technology

To investigate the objectivity, reliability, and validity of the computer-based and automated SMD Technology, we conducted three quasi-experimental studies. The objectivity of the SMD Technology was guaranteed by the computer-based and automated realization of the instrument. In the following section we report our results for reliability and validity of the SMD Technology.

Subjects

Three quasi-experimental studies (Studies 1, 2, and 3) were conducted with 106 subjects (70 female and 36 male) at the University of Freiburg. Their mean age was 18.3 years (SD = 4.6). The subject domain of Study 1 was geology and that of Studies 2 and 3 was geophysics. The subjects spent five hours on successive days working on complex problems with a multimedia discovery-learning environment.

Learning environment

The multimedia discovery-learning environment consisted of four modules. The modules could be divided into declarative and heuristic modules. The declarative modules contained all information needed to solve the phenomenon in question, while the heuristic modules primarily supported the model building process (see Dummer and Ifenthaler 2005).

Starting from the problem & learning task area, the subjects solve complex tasks from specific subject domains (Study 1: geology; Studies 2 and 3: geophysics). The subjects can navigate through different topics of the subject domain within the curriculum module. Additional information about the subject domain is provided in the form of various text documents, pictures, and audio recordings in the knowledge archive. The Model Building Kit (MoBuKi) provides the subjects with information about models, model building, and analogical reasoning. It contains three levels of abstraction of the material provided: (1) knowledge level; (2) procedural level; and (3) examples level. The toolbox is used to elicit the subjects’ understanding of the phenomenon in question constructing open concept maps.

Procedure

The three quasi-experiments took place in the computer laboratory at the University of Freiburg. Subjects had to solve a complex problem while working with a multimedia discovery-learning environment. The problem solution had to be elicited on six subsequent measurement points as an open concept map. Every subject was given an introduction to the use and construction of open concept maps.

All subjects were randomly assigned to three types of treatments. The groups were distributed as (a) scaffolding-based learning, (b) self-guided learning, and (c) control group. The subjects in group (a) received detailed feedback concerning their concept map during the model building process, subjects in group (b) received no feedback, and subjects in group (c) received no feedback and worked within a multimedia discovery-learning environment whose content was not linked to the complex problem to be solved. The quasi-experimental procedure consisted of three main parts:
  1. (1)

    Pretest: Before the subjects were able to access the multimedia discovery-learning environment, a pretest was conducted which included: (a) the domain specific knowledge test; (b) elicitation of the preconception of the complex problem to be solved as an open concept map; (c) a test on cognitive learning strategies (LIST-Test); (d) a test on intellectual abilities (BIS-Test).

     
  2. (2)

    Model building process: During the quasi-experimental session, the subjects were asked to solve a complex problem while working within the multimedia discovery-learning environment. At five measurement points, the subjects had to elicit their understanding of the complex problem in question as an open concept map.

     
  3. (3)

    Posttest: The individual learning outputs were captured with: (a) a domain specific declarative knowledge test; (b) elicitation of the final solution to the complex problem as an open concept map.

     

The primary interest of the empirical investigation in this article is the experimental validation of the SMD Technology. Therefore, we focus in the following section on reliability and validity tests. However, details on the learning-dependent progression of externalised models and treatment effects during the three quasi-experiments are reported in detail by Ifenthaler (2006) and Ifenthaler et al. (2007).

Reliability test

For the computation of the test–retest reliability (Spearman’s rank correlation), the Surface, Matching, and DeepStructure indices of measurement points three and four (control group) were used.

The results in Table 2 show a high significant correlation between the indices (Surface, Matching, and Deep Structure). Accordingly, this result is a broad hint for the reliability of the quasi-experimental study. On the other hand, we want to point out that mental models are individual ad hoc constructions (see Seel 1991), and therefore standard reliability tests, e.g. Test–Retest-, Split-Half- or Odd-Even-Method (see Rost 2005), have only limited validity as they consider the latent variable to be stable. However, the detailed research design of the three quasi-experimental studies and the applied learning environment guarantee at least an exact repeatability of the experiments.
Table 2

Test–retest reliability of the SMD Technology

 

Test–retest reliability

Surface structure

.824*

Matching structure

.815*

Deep structure

.901*

* P < .01 (two-sided significance)

Validity test

Especially with newly designed and developed instruments (e.g. SMD Technology), it is necessary to map theory based characteristics to measurable criteria. The goal of the construct validation is to determine from a theoretical point of view what the instrument really measures. For this purpose, several methodological best practices1 are available (see Lienert and Raatz 1994, p. 226). A comprehensive analysis of the theory of mental models (Johnson-Laird 1983; Seel 1991) and available instruments for the assessment of models constitutes the basis for the theory-based development of the SMD Technology. From an empirical point of view, the validity of the SMD Technology is identified with the outside criterion (1) MITOCAR, and (2) domain specific knowledge.

Pirnay-Dummer (2006) developed the instrument MITOCAR (Model Inspection Trace Of Concepts And Relations), which enables a structural and conceptual analysis of natural language expressions. The raw data of the third quasi-experimental study (N = 47) was analysed with the MITOCAR software, which was tested for reliability and validity (see Pirnay-Dummer 2006, p. 209). In the following, we use the results of the MITOCAR analysis for validity tests of the SMD Technology.

The results in Table 3 show significant correlations between the outside criterion MITOCAR and the Surface and Matching Structure of the SMD Technology.2 After verifying convergent validity of the SMD Technology, we want to test the SMD Technology with another outside criterion. This second validity test is for divergent validity on the basis of a valid and reliable domain specific knowledge test consisting out of 19 multiple-choice questions (see Couné et al. 2004). We assume that there is no correlation between the Surface and Matching Structure of the SMD Technology and the declarative knowledge measure. Further, we assume a correlation between the Deep Structure and the declarative knowledge.
Table 3

Correlation between the SMD Technology and MITOCAR (N = 47)

 

MITOCAR (Concept and structure)

Surface structure

Matching structure

MITOCAR (concept and structure)

.610**,a

.527**,a

Surface structure

 

.766**,a

Matching structure

  

** P < .01 (two-sided significance)

aPearson’s correlation

The results in Table 4 show no correlations between the declarative knowledge and the Surface and Matching Structure. This is consistent with the theoretical and methodological assumptions of the SMD Technology—the indices of the Surface and Matching Structure have no direct connection to the subject domain. The significant correlation between the declarative knowledge and the Deep Structure confirms the assumptions of the SMD Technology—we assume that persons with high declarative knowledge in a specific subject domain will also have a high Deep Structure index δ. To sum up, the empirical analysis revealed convergent and divergent validity with regard to the outside criterion. Additionally, the SMD Technology was part of a series of comparative studies of different quantitative and qualitative methodologies conducted in order to determine the methodologies’ strength and unique characteristics and to report collective validity (see Johnson et al. 2006).
Table 4

Correlation between the SMD Technology and the declarative knowledge test (N = 47)

 

Declarative knowledge

Surface structure

Matching structure

Deep structure

Declarative knowledge

.273a

.112a

.355*,b

Surface structure

 

.766**,a

.089b

Matching structure

  

.166b

Deep structure

   

* P < .05; ** P < .01 (two-sided significance)

aPearson’s correlation; b Spearman’s correlation

Applications for research, learning, and instruction

The use of different computer-based tools for re-representing knowledge structures (e.g. concept mapping software) has become increasingly accepted for research, learning, and instruction (Jonassen et al. 1997). In various research projects, concept maps have been used for analyzing learning outcomes, learners’ knowledge structures, and for self-assessment (see Mansfield and Happs 1991; Eckert 2000; Stracke 2004). In the field of learning and instruction, concept maps have been used for providing feedback and advance organizers and for facilitating problem solving tasks (see Jonassen et al. 1997; Stoyanova and Kommers 2002; Al-Diban 2002; Ifenthaler 2006). However, a large number of the available tools do not support automated feedback and analysis features. Accordingly, the development of the computer-based and automated SMD Technology opens up a broad field of applications for research, learning, and instruction.

SMD Technology and research

Re-representations of knowledge structures are often analyzed by raters using diverse scoring approaches (see Taricani and Clariana 2006; Jonassen et al. 1997). Depending on the research question, the raters focus on the quantity and quality of nodes and links, causal relationships, semantic content, direction and strength of links, hierarchy, or other visual arrangements. However, measuring the diverse information of individual concept maps by hand is very time consuming, and almost impossible for larger sets of data. Additionally, to guarantee high reliability and validity, every human rater must be an expert in the subject domain in question and in the application of quantitative and qualitative assessment strategies (Taricani and Clariana 2006). Therefore, the automated analysis procedure of the SMD Technology calculates quantitative indicators of concept maps, which then can be used for further statistical computations.

So far, the SMD Technology has been applied in different fields of mental model research. Ifenthaler (2006) investigated the trajectory of mental models constructed by subjects working on complex problem solving tasks. An HLM analysis of three quasi-experimental studies (N = 106) showed a significant increase of propositions when subjects worked for five hours in a multimedia learning environment (Surface Structure). Accordingly, as long as new information is subjective plausible it will be added to a person’s knowledge structure. Further results indicate a significant increase in the diameter of the externalized knowledge structures (Matching Structure). Consequently, we found not only a significant learning-dependent increase in the number of propositions, but also a significant learning-dependent increase in structural complexity.

In order to investigate the learning-dependent progression of novices’ mental models to more expert-like models, Ifenthaler (2006) compared the semantic similarity of externalized knowledge structures of novices with expert knowledge structures in different subject domains. The results of the Deep Structure indicator of the SMD Technology revealed a significant increase in similarity between novice and expert models. However, further HLM analysis indicated that the learning time of five hours was not long enough to integrate all information provided and consequently to gain higher similarity to an expert’s solution of a problem. Predictions about novice’s problem solving skills to become more expert like are also possible (e.g. Ifenthaler et al. 2007). Additionally, the provided learning materials and feedback could be improved for further experiments.

Ifenthaler et al. (2007) investigated the role of cognitive learning strategies and intellectual abilities in mental model building processes using the Deep Structure indicator of the SMD Technology. The results indicate that the training of mental model building skills is a complex problem which should be investigated further with regard to the roles of conditions based on the theory of mental models (Seel 1991).

Additionally, the SMD Technology has been used to investigate sharedness among team members (see Johnson et al. 2006). The focus on individually constructed concept maps and team re-representations can help to identify problems of team performance and lead to a better understanding of the complex performance processes within teams. Thanks to the flexibility of the SMD Technology, other indicators can be easily implemented in order to produce specific measures for a large number of research questions.

SMD Technology and learning and instruction

In the following, we will focus on the application of the SMD Technology for knowledge diagnosis, self assessment, and knowledge management. Other applications in the field of learning and instruction, such as analysis of navigation paths in learning environments (see Dummer and Ifenthaler 2005), could be discussed on another occasion.

In order to provide learners with the best possible learning materials, the instructor or an Intelligent Tutoring System (ITS) must be aware of their state of knowledge. In general, knowledge diagnosis is applied by collecting necessary information about the learner with the help of various tests. By integrating the SMD Technology or parts of it (graphical re-representation; quantitative indicators) either into a computer-based learning environment or other instructional settings, it can easily be applied for individual knowledge diagnosis. The SMD Technology has been implemented as a cross-platform application which enables an easy integration into a computer-based learning environment. Therefore, the instructional designer may choose which components of the SMD Technology should be applied for an adequate knowledge diagnosis. The quantitative indicators could provide instant longitudinal information about the individual learning process. The indicators (Surface, Matching, and Deep) provide multiple information about changes in the knowledge structure and domain-specific knowledge acquisition. Depending on the results of the SMD Technology, the learning environments will provide specific feedback or other instructional materials to foster future learning processes (see Ifenthaler 2006). On the other hand, the graphical re-representation of the SMD Technology can be easily applied for individual feedback on specific tasks. The instructor could use the re-representation at a specific point during the learning phase to discuss the strength and weaknesses of a learner’s learning process. Additionally, the similarity and contrast model provide further feedback materials.

Another use of the SMD Technology in the field of learning and instruction could be various fields of self assessment. As self assessment has the ambitious goal of making judgments about a learner’s own learning process, the feedback of an automated system should be very sensible to changes in the learner’s knowledge structure. As discussed above, the quantitative indicators and/or graphical re-representations of the SMD Technology could be applied for self assessment. A learner could receive quantitative information about his or her learning progress after working for a defined period with a computer-based learning environment. Additionally, the graphical re-representation could provide descriptive information about the learner’s knowledge structure. Furthermore, the similarity and contrast representation could elicit differences between previous points during the learning process or other learners or experts. This feature could therefore easily help to avoid the construction of misconceptions during self assessment phases. The major advantage of the SMD Technology for self assessment is the automated and instant generation of desired results. When learners receive the results of self assessment directly, their motivation to continue with the learning environment may be obtained longer than with other options of self assessment.

Finally, the SMD Technology could be applied for analysis of knowledge management processes. Individuals may use the quantitative indicators and or the graphical re-representations to compare it with other team members while working on a project. Also, the affordances of a task could be compared with the individual understanding of the task and gaps could be identified to solve it effectively. Another application of the SMD Technology for knowledge management could be the communication of individual or group knowledge for better cooperation and understanding with other members or groups of a project team. Further applications could include knowledge identification, knowledge use, and knowledge generation (see Tergan 2003).

Conclusion and further perspectives

The new developed SMD Technology is based on the theory of mental models (Seel 1991) and graph theory (Harary 1974) and captures key latent variables associated with human learning and cognition. Graphical representations such as concept maps or natural language expression can be analyzed on three different levels. These levels help to describe individual knowledge structures from a relational, structural, and semantic point of view. Additionally, graphical re-representations of the SMD Technology provide further information regarding the externalized knowledge structures of a person.

The objectivity, reliability, and validity of the computer-based and automated SMD Technology were investigated in three quasi-experimental studies. The results show a high reliability and validity in all indicators. Based on our findings, we developed further ideas for developing new features for the SMD Technology. These developments will include a tool for constructing concept maps, new techniques for describing the constructed models, and automated statistical reports.

Nevertheless, the SMD Technology or parts of it (graphical re-representation; quantitative indicators) can be easily integrated into various applications. The tool can be used not only in mental model research, but also in various fields of learning and instruction. Beyond this, such computer-based and automated instruments could also prove to be beneficial in a wide span of other fields of research on technology and instructional development.

Footnotes
1

Correlation of a test with several outside criteria; Correlation with tests with similar validation requirements; correlation with tests that assess other criteria; analysis of inter- and intraindividual differences in test results; factorial analysis (see Lienert and Raatz 1994).

 
2

The Deep Structure index δ of the SMD Technology compares the semantic similarity between a model and a reference model. This feature is not available with MITOCAR. Accordingly, the calculation of correlations between the Deep Structure and the MITOCAR indices is not necessary.

 

Copyright information

© Association for Educational Communications and Technology 2008