A class of kmodes algorithms for extracting knowledge structures from data
 456 Downloads
 2 Citations
Abstract
One of the most crucial issues in knowledge space theory is the construction of the socalled knowledge structures. In the present paper, a new datadriven procedure for large data sets is described, which overcomes some of the drawbacks of the already existing methods. The procedure, called kstates, is an incremental extension of the kmodes algorithm, which generates a sequence of locally optimal knowledge structures of increasing size, among which a “best” model is selected. The performance of kstates is compared to other two procedures in both a simulation study and an empirical application. In the former, kstates displays a better accuracy in reconstructing knowledge structures; in the latter, the structure extracted by kstates obtained a better fit.
Keywords
Knowledge structures kmodes Datadriven proceduresIntroduction
Knowledge space theory (KST; Doignon and Falmagne, 1985, 1999; Falmagne and Doignon, 2011), is a theory developed in the field of mathematical psychology and fruitfully applied in adaptive knowledge assessment. In this theory, individual knowledge is represented by the socalled knowledge state, which is the subset K of all problems in a given domain Q that the individual is capable of solving. A knowledge structure is a pair \((Q, \mathcal {K})\) where \(\mathcal {K}\) is a collection of knowledge states containing at least ∅ and Q itself. Some particular kinds of structures are defined on the basis of the specific closure they satisfy: whenever a knowledge structure is closed under union (i.e., every union of states is a state in the structure), it is named a knowledge space; a structure closed under intersection (i.e., every intersection of states is a state in the structure) is a closure space; finally, a structure closed under both union and intersection is a quasi ordinal space.
This article explores one of the more critical issues in KST, that is the construction of the socalled knowledge structures. This issue has been deeply explored within KST, in which three main categories of methods exist. The first one refers to the query to experts (Dowling 1993; Kambouri et al. 1994; Koppen 1993; Koppen and Doignon 1990; Müller 1989; Schrepp and Held 1995). The second method is based on skill map construction (Heller et al. 2013; Albert and Lukas 1999; Doignon 1994; Lukas and Albert 1993). The third category is that of datadriven approaches (Falmagne et al., 2013; Robusto and Stefanutti, 2014; Sargin and Ünlü , 2009; Schrepp, 1999a, b, 2003; Spoto et al., 2015; Villano, 1991).
Datadriven methods can be further classified into two categories. By imposing specific properties to the knowledge structure underlying the data, methods of the former category are capable of inferring knowledge states that are never observed in a data set (Sargin and Ünlü, 2009; Schrepp, 1999b, 2003; Spoto et al., 2015). On the contrary, methods of the second category do not impose any restrictions to the underlying knowledge structure, but they cannot infer the existence of states that have never been observed (Robusto and Stefanutti 2014; Schrepp 1999a).
In the present paper, a procedure for extracting a knowledge structure out a set of observed data is described. The procedure aims at constructing a knowledge structure by neither imposing restrictions on it, nor assuming that only observed patterns could be states. It is an incremental extension of the kmodes algorithm (Chaturvedi et al. 2001; Huang and Ng 1999) to knowledge structure extraction.
The paper is organized as follows. After presenting in some more details the KST datadriven methodologies (“Datadriven methodologies in KST”), the kmodes algorithm is introduced (“The kmodes algorithm”). Then, an adaptation of kmodes to the KST framework is presented (“A kmodes approach to knowledge structure extraction”) along with its incremental extension (“Incremental extensions of kmodes”). This last version of the algorithm, called kstates, is then tested through a simulation study (“Simulation study”) and an empirical application (“Empirical application”) in which its performance is compared with those of other two KST datadriven methodologies developed by Schrepp (1999a, 2003). The paper concludes with a discussion (“Discussion”).
Datadriven methodologies in KST
It has to be stressed that the datadriven methods are aimed at building a knowledge structure through an empirical approach, without any prior theoretical assumptions about either the relations among items, or the skills needed to solve them. This aspect clearly distinguishes them from both the expert query and the skillmap approaches.
All procedures described in this section apply to a data set consisting of a collection of Nresponse patterns, each of which is represented by the subset R⊆Q of those items receiving a correct response.
All the datadriven methods proposed so far refer to two main categories (Falmagne et al. 2013): (a) the Boolean analysis of questionnaires methods, aimed at building an implication relation among the items of a questionnaire (Schrepp 1999b; Sargin and Ünlü 2009); and (b) the methods to derive structures directly from data (e.g.; Schrepp, 1999a, b; Desmarais et al., 1995). Examples of the former class of methods are the Item Tree Analysis (ITA; Ünlü and Albert, 2004; Van Leeuwe, 1974) and the Inductive Item Tree Analysis (IITA; Schrepp, 2003, 2002), while examples of the latter can be found in Schrepp (1999a, b). Furthermore, some “hybrid” procedures have been proposed that put together a datadriven approach and either skill maps (DSMEP Spoto et al., 2015) or query to experts procedure (Cosyn and Thiéry 2000). All methodologies follow a threestep procedure: (a) constructing a set of knowledge structures; (b) testing the models according to a set of fitting criteria; and (c) selecting the best fitting model as the representation of the latent structure.
Since the knowledge states are extracted on the basis of the empirical evidence, existing datadriven procedures operate by either: (1) simply assuming that all knowledge states are observable (Robusto and Stefanutti, 2014; Schrepp, 1999a, b) or (2) imposing specific properties to the structure, that allow to infer the unobserved states. For example ITA and IITA are conceived to build quasi ordinal spaces, whereas DSMEP builds closure spaces, knowledge spaces or quasi ordinal spaces.
Both ways described in 1 and 2 have drawbacks: on the one hand, the assumption that states must have a positive observed frequency could be false in finite samples. For instance, the observed response pattern could differ from the knowledge state underlying it because of careless errors or lucky guesses to some of the items. On the other hand, there might be empirical situations in which the assumptions on the properties of the extracted knowledge structure are too strict or false. For this reason, the procedure itself could lack generality.
The kmodes algorithm
The kmodes algorithm represents an extension of the kmeans (Hartigan and Wong 1979) paradigm to categorical data. Such extension is based on three fundamental characterizations: (a) kmodes uses a simple matching dissimilarity measure; (b) it refers to modes statistics instead of means to center clusters; and (c) it updates modes on the basis of observed frequencies. In the next paragraphs, an overview of these three crucial issues is provided.
It is now possible to summarize the main issues involved in any of the different algorithms used to implement the kmodes approach (e.g., Ng et al., 2007; San and Huynh, 2004; Chaturvedi et al., 2001; Huang and Ng, 1999; Huang, 1998). First, the algorithm assigns each object to the cluster whose mode is the nearest according to a discrepancy measure and, then, it recomputes the mode on the basis of the objects included in each cluster; second, it reallocates objects into clusters until a certain criterion (e.g., no objects left to reallocate into different clusters, or the discrepancy within each cluster is below a specific value) is satisfied. All the main procedures that apply kmodes follow these fundamental steps and differ one another with respect to specific features, aimed at improving the efficiency of the algorithm or its accuracy.
In the next sections, an adaptation of the standard kmodes algorithm, called kstates, is described. Its accuracy and efficiency are tested through both a simulation study and a practical application to real data. Finally, the perspectives and improvements of the procedure are discussed.
A kmodes approach to knowledge structure extraction
 (KM1)
given knowledge structure \(\mathcal {K}_{i}\), classify the N observed response patterns into \(\mathcal {K}_{i}\) different clusters, each of which is uniquely represented by a knowledge state \(K \in \mathcal {K}_{i}\);
 (KM2)
adjust each knowledge state \(K \in \mathcal {K}_{i}\) so that the mean discrepancy between K and the patterns in the class represented by K is minimized. Let \(\mathcal {K}_{i+1}\) be the collection of the adjusted knowledge states.
 (C1)
f(R, K)≥0 for all \(R \in \mathcal {R}\) and \(K \in \mathcal {K}\),
 (C2)
\(\sum \nolimits _{K \in \mathcal {K}} f(R,K) = F(R)\) for all \(R \in \mathcal {R}\).
The partition function f is interpreted in the following way: given \(K \in \mathcal {K}\) and \(R \in \mathcal {R}\), f assigns f(R, K) out of F(R) occurrences of response pattern R to the class represented by the knowledge state K. Condition (C2) assures that every occurrence of R is assigned to some state K. It should be observed that the two conditions (C1) and (C2) do not prevent that some of the classes may be empty. That is the possibility that \({\sum }_{R \in \mathcal {R}} f(R,K)=0\) for some \(K \in \mathcal {K}\) can not be excluded. In this situation, indeed, all the observed response patterns are assigned to a strict subset of \(\mathcal {K}\).
Proposition 1
Proof
Proposition 2
Proof

if \(\theta _{K_{i},q} > 1/2\) then q∈K _{ i+1},

if \(\theta _{K_{i},q} < 1/2\) then q∉K _{ i+1},

if \(\theta _{K_{i},q} = 1/2\) then q∈K _{ i+1} with probability 1/2.
Proposition 3
Let \(\mathcal {K}_{i}\) be any knowledge structure on the set Q, and \(f_{i}:\mathcal {R} \times \mathcal {K}_{i} \to \Re \) be any minimum discrepancy partition function for \(\mathcal {K}_{i}\) . If \(\mathcal {K}_{i+1}\) is the knowledge structure obtained from \(\mathcal {K}_{i}\) by an application of the state adjustment rule, then \(D_{f_{i}}(\mathcal {R},\mathcal {K}_{i+1})\) is minimal in the sense that there is no knowledge structure \(\mathcal {K}^{\prime }\) on Q, with \(\mathcal {K}^{\prime } = \mathcal {K}_{i+1}\) such that \(D_{f_{i}}(\mathcal {R},\mathcal {K}^{\prime }) < D_{f_{i}}(\mathcal {R},\mathcal {K}_{i+1})\).
Proof
Thus in both steps (KM1) and (KM2) of the kmodes algorithm the overall discrepancy \(D_{f}(\mathcal {R},\mathcal {K})\) is minimized. In particular, at each iteration i>0, in step (KM1) the knowledge structure \(\mathcal {K}_{i}\) is set fixed and the partition function f _{ i−1} is replaced by a minimum discrepancy partition function f _{ i } for \(\mathcal {K}_{i}\). This gives \(D_{f_{i}}(\mathcal {R},\mathcal {K}_{i}) \le D_{f_{i1}}(\mathcal {R},\mathcal {K}_{i})\). In step (KM2) the partition function f _{ i } is set fixed and the knowledge structure \(\mathcal {K}_{i}\) is adjusted so that \(D_{f_{i}}(\mathcal {R},\mathcal {K}_{i+1}) \le D_{f_{i}}(\mathcal {R},\mathcal {K}_{i})\). Thus, the difference \(D_{f_{i1}}(\mathcal {R},\mathcal {K}_{i}) D_{f_{i}}(\mathcal {R},\mathcal {K}_{i+1})\) is nonnegative at each iteration i>0. The algorithm terminates when this difference is zero, or below some tolerance value.
It has been stated at the beginning of the section that a partition function cannot prevent empty classes. This remains true also for a minimum discrepancy partition function. If \(d(R,K) > {d_{\min }}(R,\mathcal {K})\) happens to be true for all \(R \in \mathcal {R}\), and a given \(K \in \mathcal {K}\), then \({\sum }_{R \in \mathcal {R}} f(R,K)\) will be zero, meaning that the class of K is empty. Suppose that, for m>0, \(\mathcal {K}_{m}\) is the knowledge structure obtained at the last iteration of the kmodes algorithm. States in \(\mathcal {K}_{m}\) representing empty classes play no role in the classification of the observed response patterns and, for this reason, they can be removed from \(\mathcal {K}_{m}\) with no harm. An obvious exception is represented by the empty set and Q since, by definition, a knowledge structure always contains these two subsets.
Incremental extensions of kmodes
It is well known that kmodes type algorithms are local minimizers (see, e.g., Chaturvedi et al., 2001). In the KST context this means that the proposed kmodes algorithm will generally converge to a local minimum of the discrepancy \(D_{f}(\mathcal {R},\mathcal {K})\) that strictly depends on the input knowledge structure \(\mathcal {K}\). Therefore, different input knowledge structures of the same size may lead to different local minima and thus to different solutions.
Proposition 4
For any knowledge structure \(\mathcal {K} \subseteq 2^{Q}\) , the equality \(D_{f}(\mathcal {R},\mathcal {K}) = 0\) holds true if and only if \(\mathcal {K}^{\top } \subseteq \mathcal {K}\).
Proof
If \(\mathcal {K}^{\top } \subseteq \mathcal {K}\) then \(d_{\min }(R,\mathcal {K})=0\) for all \(R \in \mathcal {R}\) such that F(R)>0. Thus \(D_{f}(\mathcal {R},\mathcal {K}) = {\sum }_{R \in \mathcal {R}}F(R){d_{\min }}(R,\mathcal {K})=0\). If \(\mathcal {K}^{\top } \not \subseteq \mathcal {K}\) then there is \(R \in \mathcal {R}\) such that F(R)>0 and \({d_{\min }}(R,\mathcal {K})>0\). Thus \(D_{f}(\mathcal {R},\mathcal {K})\) cannot be zero. □
In this section an incremental extension of the kmodes algorithm is considered that generates a sequence of m>0 locally optimal knowledge structures \(\mathcal {K}_{0}^{*},\mathcal {K}_{1}^{*},\ldots ,\mathcal {K}_{m1}^{*}\) of increasing size, where the smallest structure is \(\mathcal {K}_{0}^{*} = \{\emptyset ,Q\}\).
 1.
let \(\mathcal {K}_{0} = \{\emptyset ,Q\}\) be the initial knowledge structure;
 2.
At each new iteration j≥0, apply kmodes to \(\mathcal {K}_{j}\), thus obtaining \(\mathcal {K}_{j}^{*}\);
 3.
if \(D_{f}(\mathcal {R},\mathcal {K}_{j}^{*})=0\) then terminate;
 4.
else choose a new arbitrary subset \(K \in \mathcal {R} \setminus \mathcal {K}_{j}^{*}\), form the new knowledge structure \(\mathcal {K}_{j+1} = \mathcal {K}_{j}^{*} \cup \{K\}\) and repeat from step 2.
Proposition 5
Proof
 1.
let \(\mathcal {K}_{0} = \{\emptyset ,Q\}\) be the initial knowledge structure;
 2.
At each new iteration j≥0, apply kmodes to \(\mathcal {K}_{j}\), thus obtaining \(\mathcal {K}_{j}^{*}\);
 3.
if \(D_{f}(\mathcal {R},\mathcal {K}_{j}^{*})=0\) then terminate;
 4.
else choose an optimal improvement \(\hat {K} \in \mathcal {R} \setminus \mathcal {K}_{j}^{*}\), form the new knowledge structure \(\mathcal {K}_{j+1} = \mathcal {K}_{j}^{*} \cup \{\hat {K}\}\) and repeat from step 2.
When the set of items is not small (say with more than 15 items) the algorithm described above could become rather expensive from a computational point of view, since at every iteration it requires a search in the collection \(\mathcal {R} \setminus \mathcal {K}_{j}^{*}\). To improve efficiency of the algorithm, it could be useful to restrict the search to some smaller subset of \(\mathcal {R} \setminus \mathcal {K}_{j}^{*}\). The following corollary states that, if the search is confined to the collection \(\mathcal {K}^{\top } \setminus \mathcal {K}_{j}^{*}\) of observed response patterns, then improvement is guaranteed, although it is not known whether it will be optimal.
Corollary 1
In the trivial incremental extension of kmodes, if \(\mathcal {K}_{j+1} = \mathcal {K}_{j}^{*} \cup \{K\}\) , with \(K \in \mathcal {K}^{\top } \setminus \mathcal {K}_{j}^{*}\) then strict inequality \(D_{f}(\mathcal {R},\mathcal {K}_{j+1}^{*}) < D_{f}(\mathcal {R},\mathcal {K}_{j}^{*})\) holds true.
Proof
From \(K \notin \mathcal {K}_{j}^{*}\) it follows that \(d_{\min }(K,\mathcal {K}_{j}^{*}) > 0\). Since K is such that \(K \in \mathcal {R}\), F(K)>0 and \(d(K,K)=0 < d_{\min }(K,\mathcal {K}_{j}^{*})\), the result immediately follows from Proposition 5. □
Selecting a “best” knowledge structure
Having available the whole set \(\{\mathcal {K}_{0}^{*},\mathcal {K}_{1}^{*},\ldots ,\mathcal {K}_{m1}^{*}\}\) of locally optimal knowledge structures of increasing size, the question is now to select a “best” one. If the observed data \((\mathcal {R},F)\) had been generated by some true, though unknown, knowledge structure \(\mathcal {K}_{\text {true}}\) through some probabilistic process (the basic local independence model – BLIM – described by Falmagne and Doignon (1988a) is an example of one such process), then one could seek to find the structure \(\mathcal {K}_{j}^{*}\) that best approximates the true knowledge structure \(\mathcal {K}_{\text {true}}\). Standard model selection criteria like, for instance, the Akaike information criterion (AIC), the Bayesian information criterion (BIC), or still other criteria exist for this purpose. However, to be applicable, they all require a probabilistic framework, which is not established here. It would certainly be possible to provide one: the BLIM itself is an example. Nonetheless this route will not be pursued here for three different reasons.
First, both the AIC and the BIC tend to perform poorly when the sample size is “small” compared to number of parameters (Claeskens and Hjort 2008; Giraud 2014). In many probabilistic models for knowledge structures the number of parameters is proportional to the size of the knowledge structure, which could be very large in concrete applications (e.g., thousands of states) even with a moderate number of items (e.g., 20). In this situation, even 1000 would be “small” as a sample size.
The second reason is the efficiency of the extraction procedure, which would be heavily affected by the need of estimating (and reestimating many times, if local maxima are an issue) model parameters for each of the competing knowledge structures \(\mathcal {K}_{j}^{*}\).
Thirdly, the choice of any family of parametric models would put, on top of the assumptions of the proposed procedure, all the assumptions of the chosen family. This would make the procedure dependent on the family of parametric models that one chooses. Since the purpose is selecting a best model, the selection process would unavoidably be shaped by the chosen family.
It has been shown in the previous section that the discrepancy \(D_{f}(\mathcal {R},\mathcal {K}_{j}^{*})\) strictly decreases as \(\mathcal {K}_{j}^{*}\) increases. Given this, a tradeoff exists between discrepancy (fit to data) on the one side, and number of knowledge states (model complexity) on the other. Therefore, adhering to a parsimony principle, we can aim at selecting the knowledge structure that displays the “best tradeoff” between size \(\mathcal {K}_{j}^{*}\) and discrepancy \(D_{f}(\mathcal {R},\mathcal {K}_{j}^{*})\). Clearly, this approach comes with no guarantees that the selected model will be the one that best approximates any true knowledge structure. Albeit, the simulation study described in “Simulation study” is aimed at investigating this issue systematically, in a number of different conditions.
Simulation study
In this section a simulation study is described in which a comparison among three different procedures for datadriven knowledge structure construction was carried out. In particular, the aim was to compare the performance of kstates with Inductive Item Three Analysis (IITA; Schrepp2003) and the procedure proposed by Schrepp (1999a) (in the sequel we refer to this last procedure as the appbased procedure).
Having available a knowledge structure, the response patterns of N students can be simulated by using the BLIM. First, a knowledge states K is sampled from the structure with probability π _{ K }. Then, for every item q∈Q, random lucky guesses and careless errors are produced with probabilities β _{ q } and η _{ q }.
Before presenting the simulation design, a brief description of the two procedures proposed by Schrepp are given.
Competing procedures
The two procedures proposed by Schrepp (1999a, 2003) are datadriven methodologies for generating a knowledge structure from a set of observed data. These methods share the goal to derive a knowledge structure from data, but differ in the algorithms and in the assumptions they define to reach this goal.
 (i)
β _{ q } and η _{ q } probabilities are assumed to be equal across items;
 (ii)
the probability distribution on the knowledge states is uniform.
 1.
sorting the observed response patterns from the one having the highest observed frequency F(R) to the one having the smallest frequency;
 2.computing the \(app(\mathcal {K}_{L},\mathcal {R})\) distance for every possible value of the cutoff L, where:F _{ L }(R) is the expected frequency of response pattern R obtained by an application of the BLIM equations to the knowledge structure \(\mathcal {K}_{L}\) (with the restrictions (i) and (ii) described above); \(\mathcal {R}=2^{Q}\) is the power set on the set Q of items; \(\mathcal {K}_{L}\) is a knowledge structure obtained by collecting all response patterns having an observed frequency greater than L. The smaller the app the better the approximation of the model to the data;$$app(\mathcal{K}_{L}, \mathcal{R}) =\sum\limits_{R \in \mathcal{R}} \frac{(F(R)F_{L}(R))^{2}}{\mathcal{R}}; $$
 3.
selecting the \(\mathcal {K}_{L}\) for which the app is the smallest one.
The latter methodology is IITA (Schrepp 2003). This procedure was developed out of ITA (Bart and Krus 1973; Van Leeuwe 1974), and it is aimed at uncovering the logical implications among the items in Q. Such logical implications form a quasiorder (i.e., a reflexive and transitive binary relation) on the set of items. The first crucial assumption of IITA is that the true structure is a quasiordinal knowledge space, that is a knowledge structure closed under both union and intersection.
In real data sets some noise is always present, thus, even if a specific implication between two items q and r holds, a certain number b _{ qr } of counterexamples of this implication will potentially be included in the response patterns. Let L be the number of counterexamples of an implication from q to r observed in a sample of size m. The main task of IITA is to define quasiorders \(\sqsubseteq _{L}\) (L = 0,1,…, m) for Q. For instance, the relation \(q \sqsubseteq _{0} r\) involves all those item pairs for which b _{ qr }=0. One of the core issues in IITA is the computation of the expected number of counterexamples \(b^{*}_{qr}\), which has to take into account the estimate of a random error probability γ _{ qL } for each item q∈Q and each number L of counterexamples. A fundamental assumption is that γ _{ qL } = γ _{ L } is constant across items.
In the present simulation study we used the modified version of IITA presented by Sargin and Ünlü (2009). In their article, the authors highlight some of the main criticisms of IITA and propose some solutions for correctly addressing the computation of \(b^{*}_{qr}\) in the case in which \(q \not \sqsubseteq _{L} r\). More specifically, they recognize that when \(q \not \sqsubseteq _{L} r\) two different configurations can hold: in the first one \(r \not \sqsubseteq _{L} q\); in the second one \(r \sqsubseteq _{L} q\). In the former case independence holds between q and r, thus \(b^{*}_{qr}= (1p_{q})p_{r} m\). In the latter case independence cannot be assumed. Authors present the following correction of the estimate for this case: \(b^{*}_{qr}= (p_{r}  p_{q}(1 \gamma _{L})) m\) (Sargin and Ünlü 2009). Furthermore, they introduced an improvement of the procedure that minimizes the diff coefficient with respect to the error probability γ _{ L }.
These modifications have been implemented into the DAKS package for R (Unlü and Sargin 2010), that has been used in this simulation study.
Simulation Design
Simulation design of the study. In column 1, the ten different conditions are displayed.
Condition  Q  \(\mathcal {K}\)  N 

1  10  35  500 
2  10  35  1000 
3  10  70  500 
4  10  70  1000 
5  15  150  1000 
6  15  150  2000 
7  15  300  1000 
8  15  300  2000 
9  15  300  3000 
10  15  300  4000 

number of items q∈Q: 10 or 15;

number of knowledge states \(K \in \mathcal {K}\): 35, 70, 150 or 300;

sample size N: 500, 1000, 2000, 3000 or 4000.
In the whole, 4 different knowledge structures were considered: (1) \(\mathcal {K}_{1}\) for conditions 1 and 2; (2) \(\mathcal {K}_{2}\) for conditions 3 and 4; (3) \(\mathcal {K}_{3}\) for conditions 5 and 6; (4) \(\mathcal {K}_{4}\) for conditions 7, 8, 9 and 10. These four knowledge structures were obtained by computing \(\{\emptyset ,Q\} \cup \mathcal {P}\), where \(\mathcal {P}\) was generated at random, using a sampling without replacement on the collection 2^{ Q }∖{∅, Q}.
In each condition, 100 simulated data sets of size N were generated by the BLIM, in which the β _{ q } and η _{ q } parameters of the items were generated by using a uniform distribution in the interval (0,.1]. The probabilities π _{ K } of the knowledge states \(K \in \mathcal {K}_{n}\), where n∈{1,2,3,4}, were generated by using a uniform distribution in the interval [.4,.6], and then they were normalized to sum up to 1. The β _{ q } and η _{ q } parameter values and the knowledge state probabilities were kept constant across simulation conditions using the same knowledge structure. In the whole 10×100=1,000 data sets were generated.
Comparison among the three procedures: performance indexes
 1.The true positive rate (TPR) is the proportion of true knowledge states \(K \in \mathcal {K}\) belonging to the extracted knowledge structure \(\mathcal {K}_{e}\). Formally:$$ \text{TPR}=\frac{\mathcal{K}_{e} \cap \mathcal{K}}{\mathcal{K}} $$(4)
 2.The false positive rate (FPR) is the proportion of knowledge states \(K \in \mathcal {K}_{e}\) not belonging to the true knowledge structure \(\mathcal {K}\). Formally:$$ \text{FPR}=\frac{\mathcal{K}_{e} \setminus \mathcal{K}}{\mathcal{K}_{e}} $$(5)
 3.The average discrepancy between \(\mathcal {K}_{e}\) and \(\mathcal {K}\):where \(d_{\min }(K,\mathcal {K})=\min _{K^{\prime } \in \mathcal {K}} d(K,K^{\prime })\) is the minimum discrepancy between the knowledge state \(K \in \mathcal {K}_{e}\) and the true knowledge structure \(\mathcal {K}\).$$ D(\mathcal{K}_{e},\mathcal{K})=\frac{1}{\mathcal{K}_{e}}\sum\limits_{K \in \mathcal{K}_{e}} d_{\min}(K,\mathcal{K}), $$(6)
 4.The average discrepancy between K and \(\mathcal {K}_{e}\):where \(d_{\min }(K,\mathcal {K}_{e})=\min _{K^{\prime } \in \mathcal {K}_{e}} d(K,K^{\prime })\) is the minimum symmetric distance between the knowledge state \(K \in \mathcal {K}\) and the knowledge structure \(\mathcal {K}_{e}\).$$ D(\mathcal{K},\mathcal{K}_{e})=\frac{1}{\mathcal{K}}\sum\limits_{K \in \mathcal{K}} d{\min}(K,\mathcal{K}_{e}), $$(7)
 5.The Cohen’s κ, computed for the following observed frequencies:The number of false positives plus the number of false negatives gives the total number of observed disagreements.

number of positive agreements: \(\mathcal {K}_{e} \cap \mathcal {K}\);

number of negative agreements: \( 2^{Q}\mathcal {K} \cup K_{e}\);

number of false positives: \(\mathcal {K}_{e} \setminus \mathcal {K}\);

number of false negatives: \(\mathcal {K} \setminus \mathcal {K}_{e}\).

Results
Comparison among the performance of kstates (top panel), appbased (mid panel) and IITA (bottom panel) procedures in their goodness of recovery.
Condition  R  \(\mathcal {K}_{e}\)  TPR  FPR  \(D(\mathcal {K}_{e},\mathcal {K})\)  \(D(\mathcal {K},\mathcal {K}_{e})\)  κ 

kstates  
1  197.71  33.34  0.93  0.02  0.08  0.02  0.96 
2  293.41  35.94  1.00  0.03  <0.01  0.03  0.99 
3  237.95  44.79  0.57  0.11  0.52  0.11  0.68 
4  354.45  58.05  0.79  0.05  0.23  0.05  0.86 
5  595.09  110.79  0.68  0.08  0.61  0.09  0.78 
6  985.80  142.28  0.94  0.01  0.09  0.01  0.96 
7  712.66  127.40  0.32  0.25  1.40  0.27  0.44 
8  1181.50  196.66  0.58  0.12  0.73  0.12  0.69 
9  1577.40  242.64  0.76  0.06  0.37  0.06  0.84 
10  1933.46  270.07  0.88  0.03  0.17  0.03  0.92 
appbased  
Condition  R  \(\mathcal {K}_{e}\)  TPR  FPR  \(D(\mathcal {K}_{e},\mathcal {K})\)  \(D(\mathcal {K},\mathcal {K}_{e})\)  κ 
1  197.71  70.81  0.99  0.50  0.01  0.52  0.65 
2  293.41  62.23  1.00  0.41  <0.01  0.41  0.72 
3  237.95  101.17  0.91  0.34  0.11  0.36  0.74 
4  354.45  141.42  0.99  0.49  0.01  0.51  0.63 
5  595.09  561.02  0.95  0.75  0.06  0.99  0.40 
6  985.80  519.94  1.00  0.71  <0.01  0.75  0.44 
7  712.66  676.19  0.78  0.65  0.29  0.84  0.47 
8  1181.50  1181.50  0.96  0.76  0.04  0.98  0.38 
9  1577.40  1577.40  0.99  0.81  0.01  1.06  0.31 
10  1933.46  609.29  0.99  0.51  0.02  0.55  0.65 
IITA  
Condition  R  \(\mathcal {K}_{e}\)  TPR  FPR  \(D(\mathcal {K}_{e},\mathcal {K})\)  \(D(\mathcal {K},\mathcal {K}_{e})\)  κ 
1  197.71  186.22  0.39  0.93  0.82  1.43  0.07 
2  293.41  190.14  0.41  0.92  0.79  1.42  0.07 
3  237.95  144.63  0.20  0.90  1.20  1.25  0.04 
4  354.45  152.68  0.21  0.90  1.14  1.25  0.04 
5  595.09  4278.23  0.23  0.99  1.44  2.32  0.01 
6  985.80  4226.91  0.22  0.99  1.47  2.31  0.01 
7  712.66  2456.79  0.11  0.99  1.88  2.01  0.01 
8  1181.50  2371.85  0.11  0.99  1.81  2.00  0.01 
9  1577.40  2347.10  0.11  0.99  1.77  1.99  0.01 
10  1933.46  2322.70  0.11  0.99  1.73  1.99  0.01 
It can be observed that the cardinalities of the extracted knowledge structures for the appbased and the IITA procedures is systematically greater than both \(\mathcal {K}_{e}\) extracted by kstates and the true knowledge structure. This happen irrespectively of the number of items, the sample and knowledge structure sizes. Except for conditions in which the number of items was 10, the cardinality of the \(\mathcal {K}_{e}\) extracted by the appbased procedure approaches the number of the simulated response patterns. This means that almost all the response patterns are included in \(\mathcal {K}_{e}\). Whereas, knowledge structures extracted by IITA far outweigh the cardinality of the true structures, doubling in most cases the number of the simulated response patterns.
On the one hand, this result has a positive effect on the TPR index for the appbased and kstates procedures: It is always higher for the appbased procedure, meaning that it extracts a greater number of true knowledge states. On the other hand, this result has a negative effect on the FPR index: It is higher for appbased procedure, meaning that its extracted knowledge structures contain many false states (in some conditions more than 70 %). Conversely, kstates seems to be more parsimonious, preferring the choice of few but true knowledge states. In fact, the FPR index of kstates is systematically the lowest one, with percentages that are much smaller than those of the other two procedures. Concerning IITA, in all conditions the TPR index is much smaller than the FPR, that in many cases approaches 1.
All these results are reflected by the two discrepancies \(D(\mathcal {K}_{e},\mathcal {K})\) and \(D(\mathcal {K},\mathcal {K}_{e})\): The former is smaller for the appbased procedure, the latter is smaller for kstates. Furthermore, irrespectively of the considered extraction method, the higher the TPR, the lower the discrepancy \(D(\mathcal {K}_{e},\mathcal {K})\), whereas the higher the FPR, the higher \(D(\mathcal {K},\mathcal {K}_{e})\).
To provide a more synthetic index for comparing the accuracy of the two methods, we computed also Cohen’s κ (last column of Table 2), which takes concurrently into account both the information on TPR and FPR. It can been noted that, in 8 out of 10 conditions, Cohen’s κ is higher for the knowledge structures extracted by kstates, approaching in some conditions (1, 2 and 6) the upper bound 1.
The worsening of the performance of kstates in conditions number 3 and 7 could be due to an inadequate sample size compared to the number of states in the structure. With 10 items and 70 knowledge states, a sample size of 500 might be inadequate for recovering the true knowledge structure by using kstates. Indeed, by increasing the sample size to 1000 (condition 4), also kstates’ performance improves. The same reasoning applies to condition 7, where a sample size of 1000 seems to be inadequate with 15 items and 150 states.
These results suggest that increasing the amount of empirical evidence positively affects the kstates’ performance: the TPR percentage increases while that of the FPR decreases in most cases. On the contrary, in the appbased procedure the FPR is systematically very high irrespectively of the sample size. Concluding, an increase of the sample size improves kstates’ performance, but paradoxically, it worsens that of the appbased procedure. Concerning IITA, the increase of sample size seems having no effect on its performance.
Empirical application
In order to apply kstates to real data, an empirical application was carried out. The aim was to (1) apply all three competing procedures to a real data set, thus obtaining three (possibly) different structures; (2) fit the BLIM to the data for each of the extracted structures.
Methods
The application was carried out starting from a set of answers to the reduced form of the Maudsley ObsessionalCompulsive Questionnaire (MOCQR; Sanavio and Vidotto, 1985), a questionnaire investigating the obsessive and compulsive symptoms included in the wide spectrum clinical assessment battery CBA 2.0 (Sanavio et al. 2008). The questionnaire presents a dichotomous answer format and it is composed of 21 items divided into three subscales investigating three of the main dimensions of the Obsessive Compulsive Disorder (OCD): “Checking”, “Cleaning” and “DoubtingRuminating”. More specifically, the application was focused on the first two subscales containing a total of 16 items. The sample was composed of 4412 individuals and it was used in previous research (Spoto et al. 2010, 2012); the questionnaire was administered during a wider assessment procedure. Participants signed the informed consent and were asked to answer to all the items of the questionnaire. No time limit was imposed. A number of 4297 out of 4412 filled questionnaires were used for the analysis, while incomplete questionnaires were excluded. The three procedures were applied to these data in order to extract a structure for the 16 items.
In order to test the goodness of fit of the obtained structures, a sample of 59 patients with diagnosis of OCD (formulated by experts in Cognitive Behavioral Therapy) was used. Patients filled the MOCQR during the assessment phase of their treatment. The BLIM was then fitted to the clinical data set for each of the three extracted structures. This analysis was aimed at comparing the goodness of fit of the structures obtained through each procedure.
The goodnessoffit of the BLIM was tested by the Pearson Chisquare statistic. A parametric bootstrap procedure (Efron 1979) over 1000 replications was used to compute the pvalue of the Chisquare. This is necessary because the approximation to the asymptotic distribution of the Chisquare statistic lacks accuracy for large and sparse data matrices and this was the case of the present empirical application.
Parametric bootstrap was performed in the following way: (i) The parameter estimates obtained by fitting the model to the data were used to generate 1000 data sets of the same size of the observed sample (N = 59); (ii) the model was fitted to each of the 1000 simulated data sets; and (iii) the proportion of replications in which the model obtained a Chisquare greater than the observed one was the bootstrapped pvalue.
Results
The appbased procedure extracted a structure containing only the empty set and the total set: An overly simple model. It is likely that the restrictions at the basis of this procedure are too strong, leading the app selection criterion to be excessively conservative. Needless to say, the fit of the BLIM to data led to a strong rejection.
Concerning IITA, it extracted a structure of 103 states, but the pvalue of the BLIM’s Chisquare computed for this structure was p = .07 (which, considering a first type error probability α = .10, leads to a rejection).
Finally, kstates extracted a structure composed of 246 states. Identifiability of the BLIM for the extracted structure was tested by the BLIMIT function (Stefanutti et al. 2012). No identifiability issues were found. The pvalue of the BLIM’s Chisquare, computed for this structure, was p = .23, meaning that the model predicts the data quite well. Also the values of the error parameter estimates \(\hat {\beta }_{q}\) and \(\hat {\eta }_{q}\) of the items suggest a very good fit of the model to the data. Indeed, the following average values across items were obtained: \(\bar {\beta }_{q}=.11\) (SD =.09) and \(\bar {\eta }_{q}=.13\) (SD =0.16).
Discussion
In the present paper a new datadriven procedure to build knowledge structures, called kstates, was presented. The development of the procedure drawn upon the area of data mining and, in particular, to the kmodes clustering (Chaturvedi et al. 2001; Huang 1998). The proposed algorithm is an incremental extension of kmodes that generates a sequence of locally optimal knowledge structures of increasing size, among which a “best” model is selected.
In order to test the applicability of the kstates algorithm, a simulation study and an empirical application were carried out. In the former study the aim was to compare the performance of kstates with that of the appbased (Schrepp 1999a) and the IITA (Schrepp 2003) procedures, in different simulation conditions. In the comparison of the knowledge structures extracted by the three procedures with those used for generating the data, kstates performed better in most cases. Despite kstates performs quite well in all the simulation conditions, it extracts knowledge structures with cardinality systematically smaller than that of the “true” structure. This suggests that the selection criterion is rather conservative and needs improvements. Further studies should investigate other types of selection criteria that allow to improve the proportion of “true” states, while keeping low the proportion of “false” states contained in the resulting structure.
Concerning the empirical application, each of the three procedures extracted a structure from an existing data set of 4297 respondents to 16 items of the MOCQR (MOCQR; Sanavio and Vidotto, 1985). The only structure for which the BLIM obtained an acceptable fit was the one extracted by kstates.
Concluding, the strengths of kstates, compared to the other procedures, can be summarized as follows: (1) unlike ITA and IITA, kstates does not assume any restriction on the properties of the knowledge structure; (2) Unlike the appbased procedure, it does not require that the collection of knowledge states is a subset of the observed response patterns; (3) In selecting the states that have to be included in the structure, kstates seems to be more parsimonious than the appbased procedure, preferring few but correct knowledge states.
Notes
Acknowledgments
The research developed in this article was carried out under the research project CPDA149902, funded by the University of Padua. The authors would like to acknowledge Giorgio Bertolotti and Salvatore Maugeri Foundation for providing the data set used in the empirical example.
References
 Albert, D., & Lukas, J. (Eds.) (1999). Knowledge Spaces: Theories, Empirical Research, and Applications Knowledge spaces: Theories, empirical research and applications: Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
 Bart, W.M., & Krus, D.J. (1973). An orderingtheoretic method to determine hierarchies among items. Educational and psychological measurement.Google Scholar
 Chaturvedi, A., Green, P.E., & Caroll, J.D. (2001). Kmodes clustering Kmodes clustering. Journal of Classification, 18(1), 35– 55.CrossRefGoogle Scholar
 de Chiusole, D., Stefanutti, L., Anselmi, P., & Robusto, E. (2013). Assessing Parameter Invariance in the BLIM: Bipartition Models Assessing parameter invariance in the BLIM: Bipartition Models. Psychometrika, 78(4), 710–724.CrossRefPubMedGoogle Scholar
 de Chiusole, D., Stefanutti, L., Anselmi, P., & Robusto, E. (2015). Modeling missing data in knowledge space theory. Modeling missing data in knowledge space theory. Psychological Methods, 20(4), 506.CrossRefPubMedGoogle Scholar
 Claeskens, G., & Hjort, N.L. (2008). Model selection and model averaging Model selection and model averaging (Vol 330). Cambridge University Press, Cambridge.Google Scholar
 Cosyn, E., & Thiéry, N. (2000). A practical procedure to build a knowledge structure A practical procedure to build a knowledge structure. Journal of mathematical psychology, 44(3), 383– 407.CrossRefPubMedGoogle Scholar
 Desmarais, M.C., Maluf, A., & Liu, J. (1995). Userexpertise modeling with empirically derived probabilistic implication networks Userexpertise modeling with empirically derived probabilistic implication networks. User modeling and useradapted interaction, 5(34), 283–315.CrossRefGoogle Scholar
 Doignon, J.P. (1994). Knowledge spaces and skill assignments Knowledge spaces and skill assignments. In Fischer, G., & Laming, D. (Eds.) Contributions to Mathematical Psychology, Psychometrics and Methodology Contributions to mathematical psychology, psychometrics and methodology (p. 111121). New York SpringerVerlag.Google Scholar
 Doignon, J.P., & Falmagne, J.C. (1985). Spaces for the assessment of knowledge Spaces for the assessment of knowledge. International Journal of ManMachine Studies, 23, 175–196.CrossRefGoogle Scholar
 Doignon, J.P., & Falmagne, J.C. (1999). Knowledge spaces Knowledge spaces. New York: Springer.CrossRefGoogle Scholar
 Dowling, C.E. (1993). Applying the basis of a knowledge space for controlling the questioning of an expert Applying the basis of a knowledge space for controlling the questioning of an expert. Journal of Mathematical Psychology, 37(1), 21–48.CrossRefGoogle Scholar
 Efron, B. (1979). Bootstrap methods: Another look at jackknife. Annual Statistics, 7(2), 1–2.CrossRefGoogle Scholar
 Falmagne, J.C., Albert, D., Doble, C., Eppstein, D., & Hu, X. (2013). Knowledge spaces: Applications in education Knowledge spaces. Springer Science & Business Media.Google Scholar
 Falmagne, J.C., & Doignon, J.P. (1988a). A class of stochastic procedures for the assessment of knowledge A class of stochastic procedures for the assessment of knowledge. British Journal of Mathematical and Statistical Psychology, 41, 1– 23.CrossRefGoogle Scholar
 Falmagne, J.C., & Doignon, J.P. (1988b). A Markovian procedure for assessing the state of a system A Markovian procedure for assessing the state of a system. Journal of Mathematical Psychology, 32, 232–258.CrossRefGoogle Scholar
 Falmagne, J.C., & Doignon, J.P. (2011). Learning spaces Learning spaces. New York: Springer.CrossRefGoogle Scholar
 Giraud, C. (2014). Introduction to highdimensional statistics Introduction to highdimensional statistics (Vol. 138): CRC Press.Google Scholar
 Hartigan, J.A., & Wong, M.A. (1979). Algorithm AS 136: A kmeans clustering algorithm Algorithm as 136: A kmeans clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1), 100–108.Google Scholar
 Heller, J., Augustin, T., Hockemeyer, C., Stefanutti, L., & Albert, D. (2013). Recent Developments in Competencebased Knowledge Space Theory Recent developments in competencebased knowledge space theory. Knowledge Spaces Knowledge spaces (p. 243–286): Springer.Google Scholar
 Heller, J., & Wickelmaier, F. (2013). Minimum Discrepancy Estimation in Probabilistic Knowledge Structures Minimum discrepancy estimation in probabilistic knowledge structures. Electronic Notes in Discrete Mathematics, 42 (4), 49–56.CrossRefGoogle Scholar
 Huang, Z. (1998). Extensions to the kmeans algorithm for clustering large data sets with categorical values Extensions to the kmeans algorithm for clustering large data sets with categorical values. Data mining and knowledge discovery, 2(3), 283–304.CrossRefGoogle Scholar
 Huang, Z., & Ng, M.K. (1999). A fuzzy kmodes algorithm for clustering categorical data A fuzzy kmodes algorithm for clustering categorical data. IEEE Transactions on Fuzzy Systems, 7(4), 446–452.CrossRefGoogle Scholar
 Kambouri, M., Koppen, M., Villano, M., & Falmagne, J.C. (1994). Knowledge assessment: Tapping human expertise by the QUERY routine Knowledge assessment: Tapping human expertise by the query routine. International Journal of HumanComputer Studies, 40(1), 119–151.CrossRefGoogle Scholar
 Koppen, M. (1993). Extracting human expertise for constructing knowledge spaces: An algorithm Extracting human expertise for constructing knowledge spaces: An algorithm. Journal of mathematical psychology, 37(1), 1–20.CrossRefGoogle Scholar
 Koppen, M., & Doignon, J.P. (1990). How to build a knowledge space by querying an expert How to build a knowledge space by querying an expert. Journal of Mathematical Psychology, 34(3), 311–331.CrossRefGoogle Scholar
 Lukas, J., & Albert, D. (1993). Knowledge Assessment Based on Skill Assignment and Psychological Task Analysis Knowledge assessment based on skill assignment and psychological task analysis. In Strube, G., & Wender, K. (Eds.) The Cognitive Psychology of Knowledge The cognitive psychology of knowledge (p. 139160). Amsterdam NorthHolland.Google Scholar
 Müller, C.E. (1989). A procedure for facilitating an experts judgements on a set of rules A procedure for facilitating an experts judgements on a set of rules. Mathematical psychology in progress Mathematical psychology in progress (pp. 157–170): Springer.Google Scholar
 Ng, M.K., Li, M.J., Huang, J.Z., & He, Z. (2007). On the impact of dissimilarity measure in kmodes clustering algorithm On the impact of dissimilarity measure in kmodes clustering algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(3), 503–507.CrossRefPubMedGoogle Scholar
 Robusto, E., & Stefanutti, L. (2014). Extracting a knowledge structure from the data by a maximum residuals method Extracting a knowledge structure from the data by a maximum residuals method. TPM: Testing, Psychometrics, Methodology in Applied Psychology.Google Scholar
 San, O M., & Huynh, N. (2004). An alternative extension of the kmeans algorithm for clustering categorical data An alternative extension of the kmeans algorithm for clustering categorical data. International Journal of Applied Mathematics and Computer Science, 14(2), 241–248.Google Scholar
 Sanavio, E., Bertolotti, G., Michielin, P., Vidotto, G., & Zotti, A. (2008). CBA2.0 scale primarie: Manuale. Una batteria ad ampio spettro per l’assessment psicologico. Cba2.0 scale primarie: Manuale. una batteria ad ampio spettro per l’assessment psicologico. Firenze: Organizzazioni Speciali.Google Scholar
 Sanavio, E., & Vidotto, G. (1985). The components of the Maudsley obsessionalcompulsive questionnaire The components of the Maudsley obsessionalcompulsive questionnaire. Behaviour Research and Therapy, 23(6), 659–662.CrossRefPubMedGoogle Scholar
 Sargin, A., & Ünlü, A (2009). Inductive item tree analysis: Corrections, improvements, and comparisons Inductive item tree analysis: Corrections, improvements, and comparisons. Mathematical Social Sciences, 58(3), 376–392.CrossRefGoogle Scholar
 Schrepp, M. (1999a). Extracting Knowledge Structures from Observed Data Extracting Knowledge Structures from Observed Data. British Journal of Mathematical and Statistical Psychology, 52, 213–224.CrossRefGoogle Scholar
 Schrepp, M. (1999b). On the empirical construction of implications between bivalued test items On the empirical construction of implications between bivalued test items. Mathematical social sciences, 38(3), 361–375.CrossRefGoogle Scholar
 Schrepp, M. (2002). Explorative analysis of empirical data by Boolean analysis of questionnaires. Zeitschrift für Psychologie mit Zeitschrift für angewandte Psychologie.Google Scholar
 Schrepp, M. (2003). A method for the analysis of hierarchical dependencies between items of a questionnaire A method for the analysis of hierarchical dependencies between items of a questionnaire. Methods of Psychological Research Online, 19, 43–79.Google Scholar
 Schrepp, M., & Held, T. (1995). A simulation study concerning the effect of errors on the establishment of knowledge spaces by querying experts A simulation study concerning the effect of errors on the establishment of knowledge spaces by querying experts. Journal of Mathematical Psychology, 39(4), 376–382.CrossRefGoogle Scholar
 Spoto, A., Stefanutti, L., & Vidotto, G. (2010). Knowledge space theory, formal concept analysis, and computerized psychological assessment Knowledge space theory, formal concept analysis, and computerized psychological assessment. Behavior Research Methods, 42(1), 342–350.CrossRefPubMedGoogle Scholar
 Spoto, A., Stefanutti, L., & Vidotto, G. (2012). On the unidentifiability of a certain class of skill multi map based probabilistic knowledge structures On the unidentifiability of a certain class of skill multi map based probabilistic knowledge structures. Journal of Mathematical Psychology, 56(4), 248–255.CrossRefGoogle Scholar
 Spoto, A., Stefanutti, L., & Vidotto, G. (2015). An iterative procedure for extracting skill maps from data An iterative procedure for extracting skill maps from data. Behavior research methods, 1– 13.Google Scholar
 Stefanutti, L., Heller, J., Anselmi, P., & Robusto, E. (2012). Assessing the local identifiability of probabilistic knowledge structures Assessing the local identifiability of probabilistic knowledge structures. Behaviour Research Methods, 44, 1197–1211.CrossRefGoogle Scholar
 Ünlü, A., & Albert, D (2004). The Correlational Agreement Coefficient CA (≤, D) a mathematical analysis of a descriptive goodnessoffit measure The correlational agreement coefficient ca (≤, d) a mathematical analysis of a descriptive goodnessoffit measure. Mathematical Social Sciences, 48(3), 281–314.CrossRefGoogle Scholar
 Ünlü, A., & Sargin, A (2010). DAKS: An R package for data analysis methods in knowledge space theory Daks: An R package for data analysis methods in knowledge space theory. Journal of Statistical Software, 37(2), 1–31.CrossRefGoogle Scholar
 Van Leeuwe, J.F. (1974). Item tree analysis. Item tree analysis. Nederlands Tijdschrift voor de Psychologie en haar Grensgebieden.Google Scholar
 Villano, M. (1991). Computerized knowledge assessment: Building the knowledge structure and calibrating the assessment routine. Unpublished doctoral dissertation.Google Scholar