An iterative procedure for extracting skill maps from data
 457 Downloads
 1 Citations
Abstract
The methodologies for the construction of a knowledge structure mainly refer to the query to experts, the skill maps, and the datadriven approaches. This last method is of growing interest in recent literature. In this paper, an iterative procedure for building a skill map from a set of data is introduced. This procedure is based on the minimization of the distance between the knowledge structure delineated by a given skill map and the data. The accuracy of the proposed method is tested through a number of simulation studies where the amount of noise in the data is manipulated as well as the kind of structure to be reconstructed. Results show that the procedure is accurate and that its performance tends to be sufficiently stable even with high error rates. The procedure is compared to two alreadyexisting methodologies to derive knowledge structures from a set of data. The use of the corrected Akaike Information Criterion (AICc) as a stopping criterion of the iterative reconstruction procedure is tested against the app criterion introduced by Schrepp. Moreover, two empirical applications on clinical data are reported, and their results show the applicability of the procedure.
Keywords
Skill maps Knowledge space theory Datadriven construction methods Conjunctive model Disjunctive modelIntroduction
The present paper aims to introduce a new methodology within the framework of the Knowledge Space Theory (KST; Doignon & Falmagne, 1985, 1999; Falmagne & Doignon, 2011) to derive a skill map from a data set. A skill map is a function that assigns to each item the set of abilities needed to solve it. KST has been mainly applied in the assessment of knowledge of students. One of its most successful applications is the software ALEKS (Assessment and LEarning in Knowledge Spaces; Doignon & Falmagne, 1999; Falmagne, Doble, Albert, Eppstein, & Hu, 2013; Falmagne & Doignon, 2011). Nevertheless, in recent years, the KST approach has been applied in clinical contexts in order to carry out an effective and adaptive assessment of the clinical state of patients (Bottesi et al. 2014). Such applications have shown both that the KST approach could adequately fit clinical requirements, and that the possibility to further theoretically develop these innovative applications deserves a great attention.
In KST, some basic concepts are defined: a knowledge domain is the entire set Q of items that can be asked about a specific topic; a knowledge state is the subset K of the knowledge domain that a person masters; a knowledge structure is a pair \((Q, \mathcal {K})\) where \(\mathcal {K}\) is the collection of all knowledge states existing in a population, containing at least the empty set (∅) and the total set (Q).
 1.
86+14= ?
 2.
100−14= ?
 3.
(86+14)/4= ?
 4.
(100−14)/2= ?

a) sum two integer numbers;

b) subtract two integer numbers;

c) divide two integer numbers.
It can be argued that item 1 requires skill a, item 2 requires skill b, whereas item 3 requires both skills a and c, and item 4 requires both skills b and c. Starting from such assignment, it is possible to deduce which knowledge states appear in the structure (as explained in Section 2). One of the crucial aspects to be considered in performing the itemskill association is the rule, here named model, about the relation between items and skills. A skill map is a triple (Q,S,τ) where Q is a nonempty set of items, S is a nonempty set of skills, and τ is a mapping from the set Q of items to the power set of S, assigning to each item a nonempty set of skills. Three different models are usually taken into account: the disjunctive, the conjunctive, and the competency model (Doignon & Falmagne, 1999; Korossy, 1997). In the disjunctive model, in order to master q, it is sufficient to have one of the skills in τ(q). In the conjunctive model, on the contrary, all the skills assigned to an item are needed to master it, this model applies, for instance, to the example given above. For a detailed explanation of these two models as well as of the competency model, which is beyond the scope of this article, refer to, e.g., Doignon and Falmagne 1999. The datadriven methodology refers to the possibility to generate a knowledge structure starting from a set of collected data. All the datadriven methods available at the present for the construction of knowledge structures can be subdivided into two main categories (Falmagne et al. 2013): on one hand, we have the Boolean analysis of questionnaires methods, aimed at building an implication relation among the items of a questionnaire (Sargin & Ünlü, 2009; Schrepp, 1999b); on the other hand, the second group of methods derives structures directly from data (e.g.; Desmarais & Maluf, 1996; Schrepp, 1999a). Both methodologies follow approximately the same threestep procedure: the first step consists of constructing a set of relations/knowledge structures; the second step tests such models according to a set of fitting criteria; in the last step, the best fitting model is selected as the best representation of the latent structure. Examples of the first class of methods are the Item Tree Analysis (ITA; Ünlü & Albert, 2004; vanLeeuwe, 1974) and the Inductive Item Tree Analysis (IITA; Schrepp, 2002, 2003). Examples of the second class of methods can be found, for example, in Schrepp (1999a). In the present paper, we propose a method to derive a skill map from an empirical data set. The method, which is the first attempt in literature to carry out this task, consists of first deriving a knowledge structure from data, and then deriving the skill map that corresponds to that structure. The next sections present some further details on the main concepts of KST, illustrate the proposed procedure, and, finally review a simulation study and two practical applications on real data to show how the procedure works and its pros and cons.
Extracting knowledge structures from skill maps
Skill map used in the example (see the text)
Item (q)  τ(q) 

1  {a} 
2  {b} 
3  {a,c} 
4  {b,c} 
To summarize, the knowledge space \(\mathcal {K}\) delineated by a skill map τ via the disjunctive model and the closure space \(\mathcal {C}\) delineated by the same skill map via the conjunctive model are dual of one another.
Deriving skill maps from knowledge structures
The material in the previous section was based on the assumption that a skill map can be constructed starting from theoretical considerations about the items. The knowledge structure on the set of items is then derived from the skill map. In this section, we consider the opposite situation: a knowledge structure is available and a skill map delineating this structure has to be identified. This is possible if the knowledge structure is closed under either union, intersection or both. The former case is considered first.
Theorem 6.3.8 of Falmagne and Doignon (2011) asserts that any knowledge space is delineated by some (minimal) skill map if and only if it has a basis whose cardinality equals that of the set of skills. A strong relationship exists between the basis of a knowledge space and the smallest skill map delineating it.
Corollary 1
Assume that \(\mathcal {K}\) is a space with basis \(\mathcal {B}\) , and S an arbitrary set such that there is a bijective correspondence \(f : S \rightarrow \mathcal {B}\) . Then the space delineated by the disjunctive skill map τ(q)={s∈S:q∈f(s)} is \(\mathcal {K}\).
Proof
First, it can be seen that \(\tau (q)= \{s\in S : q \in f(s)\} \Leftrightarrow s \in \tau (q) \Leftrightarrow q \in f(s)\). Then, by the disjunctive model, we know that \(K = \{q \in Q : \tau (q) \cap T \neq \emptyset \}\), i.e., Eq. 3. By applying Eq. 3 to the singleton {s} we have \(\{q \in Q : \tau (q) \cap \{s\} \neq \emptyset \} = \{q \in Q : s \in \tau (q)\} = \{q \in Q : q \in f(s)\}\). By applying the same procedure to all the singletons, all the element of the basis \(\mathcal {B}\) are reproduced. Then, \(\mathcal {K}\) is obtained by closure under union of \(\mathcal {B}\). □
To give an example of the application of Corollary 1, let S={a,b,c}, and define the bijection \(f: S \rightarrow \mathcal {B}_{\mathcal {K}}\) such that f(a)={1,3}, f(b)={2,4}, and f(c)={3,4}. Now, setting τ(q)={s∈S:q∈f(s)} for each item q∈Q the skill map in Table 1 is obtained. We recall that this is exactly the skill map delineating \(\mathcal {K}\) under the disjunctive model.
 1.
the dual \(\mathcal {K}\) of \(\mathcal {C}\) is computed;
 2.
the basis \(\mathcal {B_{K}}\) of \(\mathcal {K}\) is obtained;
 3.
a set S of abstract skills is defined such that there is a bijection \(f : S \rightarrow \mathcal {B}\);
 4.
the skill map τ is defined such that τ(q)={s∈S:q∈f(s)} for each q∈Q.
For the reasons explained above, the skill map τ delineates the structure \(\mathcal {C}\) via the conjunctive model. To give an example, suppose that the knowledge structure is as in Eq. 2. Since \(\mathcal {C}\) is closed under intersection, the procedure described above can be applied for deriving the corresponding skill map. To do so, it is necessary to compute first the dual of \(\mathcal {C}\), which is already defined in Eq. 4 and its basis \(\mathcal {B_{K}}\). Therefore, since \(\mathcal {B_{K}}\) contains three states, the set S of skills will also contain three skills that we name a, b, and c. Now, by applying step 4 of the procedure, the skill map τ displayed in Table 1 is obtained. Of course the interpretation of the skill map is in this case conjunctive.
The basic local independence model
The introduction of these notions about KST allows considering in more detail some of the alreadyexisting procedures to build a knowledge structure from a set of data. More specifically, in the next subsection, the critical elements of both Boolean analysis of questionnaires and IITA are explored in order to clarify the rationale for looking for a new method able to cope with all these limitations.
Limitations of existing datadriven methods
In general, from a cognitive perspective, the most important limitation of both query and datadriven procedures is that they work at the items level, leaving out any information concerning the skills level. However, what people learn is skills, not items. Items (regarded as questions or problems) are means for ascertaining whether skills are mastered or not by individuals. Shifting the attention from the behavioral level of the items to the competence level of the skills offers a possible cognitive interpretation of the response behavior of an individual. Referring to the example proposed in the Introduction, both query and datadriven procedures would reconstruct the structure, but do not formally/explicitly provide information about the cognitive aspects that make item 1 less demanding than item 3. Importantly, a great amount of research concerning the role of the skills or abilities to master items has been conducted in the area of cognitive diagnosis models (CDM; e.g., DeLaTorre & Douglas, 2008; Templin & Henson, 2006), and it has also been deeply investigated in KST (see e.g., Doignon, 1994; Düntsch & Gediga, 1995; Heller et al., 2013; Korossy, 1997). Furthermore, considering the main datadriven approaches, some other considerations have to be taken into account. In his paper, Schrepp (1999a) indicates at least two main critical aspects of his procedure based on sorting the observed response patterns by decreasing frequency and then adding to the reconstructed structure one pattern at a time. At each step, a fit index is computed and the model that displays the minimum value of such index is selected as the correct knowledge structure. The two main criticisms of the procedure introduced by Schrepp are: the number of observed patterns necessary to reliably reconstruct the structure increases heavily with the cardinality of Q, thus, a researcher who wants to use such a procedure may have to face the impossibility of collecting such immense data sets; if the states of the underlying knowledge structure do not have equal probability, the accuracy of the procedure is influenced negatively. Furthermore, the procedure does not account for any state that is not observed in terms of response pattern, and this could be a problem. In fact, suppose a population is characterized by a knowledge structure \(\mathcal {K}\) on the set Q of items, and a sample of response patterns \(R \subseteq Q\) is available. Some of the states \(K \in \mathcal {K}\) might be not contained in the sample for at least two different reasons: (1) π _{ K } is small, and for this reason K does not belong to the sample at hand, or (2), due to lucky guesses and careless errors, none of the response patterns in the sample equals K. If either of these two cases occurs, Schrepp’s procedure fails to reconstruct the “true” structure \(\mathcal {K}\). In other words, if the response pattern corresponding to a latent state of the knowledge structure is not observed in the sample, it will never be included in the structure. Moreover, given that the introduction of a pattern into the structure follows the frequency criterion, the procedure does not guarantee to produce a knowledge structure (since either the total set Q or the empty set ∅ may have a very low frequency in the sample at hand). Another important criticism of this procedure is that it both underestimates the error rates of the items and assumes that such values are equal for all the items of the domain (Schrepp, 1999a).
The IITA procedure, on the other hand, is conceived to build knowledge structures that are closed under both set union and set intersection, that is, it systematically produces quasiordinal knowledge spaces (Falmagne & Doignon, 2011). In this respect, one main criticism arises: the assumption of having a structure both \(\bigcap \) and \(\bigcup \)closed may be too strong in several applicative contexts. Given the limitations of the existing procedures, we aimed at proposing a new approach able to cope with most of the critical issues described above. More specifically, we aimed at obtaining a structure that could be traced back to a skill map (i.e., it has to be closed under intersection, union, or both) and does not present IITA strong assumptions. In the next section, the procedure used to derive the knowledge structure and the skill map is described.
Procedure

i) to build a chain of nested closure spaces on the basis of a given criterion (as described below);

ii) to select one of the closure spaces according to an appropriate criterion.
The procedure would then end up with the best fitting closure space from which, following the principles described in the previous sections, a conjunctive skill map is obtained. Let \(\mathcal {R}= 2^{Q}\) be the power set on Q. The data set consists of N response patterns \(R \in \mathcal {R}\). We assume that for every theoretically observable response pattern R, the observed frequency F(R) is known (with \({\sum }_{R \in \mathcal {R}} F(R) = N\)).
Such an index refers to the distance between the reconstructed knowledge structure \(\mathcal {K}_{L}\), at each step of the procedure, and the set of observed response patterns \(\mathcal {R}\). This distance is the ratio between the squared difference of relative frequencies F(R) of the patterns and theoretical frequency F _{ L }(R) expected assuming \(\mathcal {K}_{L}\) to be the correct structure, and the cardinality of \(\mathcal {R}\). The smaller the value of app, the better the approximation of the model to the data is. The step in which app reaches its minimum corresponds to the best fitting model. In our application, we selected this index as a stopping criterion. Its computation requires that the theoretical frequencies F _{ L }(R) are available. These expectations can be obtained by estimating the BLIM for the knowledge structure \(\mathcal {K}_{L}\).
Once the structure is selected, since it is a closure space, the basis of its dual can be obtained, and, from this basis, the corresponding conjunctive skill map can be derived. Like Schrepp’s procedure, the proposed one is carried out in a number of steps. In each step i a knowledge structure is specified, which is a strict superset of the one obtained in step i−1. A stopping criterion is used for deciding when the procedure should terminate. The proposed procedure differs from the Schrepp’s one with respect to three basic aspects: (1) the next response patterns to be added to the structure are selected by a minimum distance, rather than by the maximum frequency criterion; (2) the structure is closed under intersection in every single step; in the proposed procedure the app criterion is based on the careless error and lucky guess parameter estimates obtained by fitting the BLIM to the data, meaning that β and η parameters can be different for different items.
So far, the procedure was described for extracting a conjunctive skill map from data. Nevertheless, if the aim is to extract a disjunctive skill map, this procedure can be adapted with minimal effort. In fact, it is sufficient to replace closure under intersection by closure under union at each step n of the procedure. The remaining parts of the procedure are exactly the same. Of course, there is the possibility of performing both closures at each step, thus obtaining, as the final result, a quasiordinal knowledge space as in the IITA procedure.
Simulation study
In this section, we introduce a simulation study carried out to test the efficiency and accuracy of the iterative procedure to build a closure space starting from a set of observed data. The performance of the Data Driven Skill Map Extraction Procedure (DSMEP) is compared to that of two of the main alternative procedures, i.e., the Schrepp’s method and the Inductive Item Tree Analysis (IITA).
Method
 i)
A number of 100 random samples each containing 1000 response patterns were generated by parametric bootstrap (Efron & Tibshirani, 1993), where the probabilistic model was the BLIM. In all replications, the same fixed knowledge structures \(\mathcal {K}_{t}\) on eight items containing 50 states for the conditions (1) and (2) and containing 48 states for the condition (3), were used as the true structure.
 ii)
The three iterative procedures were applied to every single data set, obtaining a structure for each of the 100 data sets in each of the nine conditions.
 iii)With respect to the new reconstruction procedure, at every step both the corrected AIC (Brockwell & Davis, 1991; Burnham & Anderson, 2002), the app index (Schrepp, 1999a) and a Distance from the True Structure (DTS) were computed. The DTS was obtained through the following:where \(\mathcal {C}_{n}\) is the structure obtained at step n of the procedure. The DTS attains the value 0 if an only if \(\mathcal {K}_{t} = \mathcal {C}_{n}\). On the other hand, given that by definition both the empty set (∅) and the total set Q belong to both \(\mathcal {K}_{t}\) and \(\mathcal {C}_{n}\), the maximum possible value of DTS is equal to Q/2. In fact, the maximum value of the minimum distance between a given state in \(\mathcal {K}_{t}\) and a state in \(\mathcal {C}_{n}\) is half the number of items in Q. Moreover, if the DTS between the two structures \(\mathcal {K}_{t}\) and \(\mathcal {C}_{n}\) is d, then on the average any given state \(K \in \mathcal {K}_{t}\) differs from its closest state \(C_{i} \in \mathcal {C}_{n}\) by d items.$$\begin{array}{@{}rcl@{}} \text{DTS}(\mathcal{K}_{t}, \mathcal{C}_{n})&=& d(\mathcal{K}_{t}, \mathcal{C}_{n})+ d(\mathcal{C}_{n},\mathcal{K}_{t}) \\&=& \frac{1}{\mathcal{K}_{t}} \sum\limits_{K \in \mathcal{K}_{t}}d_{min}(K, \mathcal{C}_{n})\\&&+ \frac{1}{\mathcal{C}_{n}} \sum\limits_{C \in \mathcal{C}_{n}}d_{min}(C, \mathcal{K}_{t}) \end{array} $$
With respect to the other two methodologies (IITA and Schrepp), for each condition, all the steps of the procedures were carried out in order to obtain a structure according to the assumptions of each method.
 iv)The following additional statistics were computed for each of the three procedures, in each of the nine conditions:
 a)
The average proportion of states of the “true” structure contained in the reconstructed structure (true positive states).
 b)
The average proportion of states not contained in the “true” structure, but contained in the reconstructed structure (false positive states).
 c)
The average distance \(d(\mathcal {K}_{t}, \mathcal {C}_{n})\) between the “true” structure and the reconstructed one.
 d)
The average distance \(d(\mathcal {C}_{n}, \mathcal {K}_{t})\) between the reconstructed structure and the “true” one. It should be observed that, in general \(d(\mathcal {C}_{n}, \mathcal {K}_{t}) \neq d(\mathcal {K}_{t}, \mathcal {C}_{n})\) (see e.g., Falmagne & Doignon, 2011).
 e)
The average DTS.
 a)
Concerning Schrepp’s procedure, all these indexes were computed on the structures obtained at the iteration in which the minimum app was obtained; the IITA procedure does not allow to compute the app index. Therefore, for this procedure, these statistics were computed on the obtained model; in DSMEP these statistics were computed on the structures obtained when the app reached its own minimum. Additionally, for the proposed procedure, the statistics were computed also on the structure corresponding to the minimum of the AICc.
Results
The performances of the three procedures in reconstructing a structure neither closed under union nor intersection. The table displays the proportion of true states included in the reconstructed structure (True Pos), the proportion of the false states in the reconstructed structure (False Pos), the average distance between the true structure and the reconstructed structure (\(d(\mathcal {K}_{t}, \mathcal {C}_{n})\)), the average distance between the reconstructed structure and the true structure (\(d(\mathcal {C}_{n}, \mathcal {K}_{t})\)), and the DTS (notice that each of the three distances take on values in the interval [0,Q/2]). For our procedure, we reported these statistics with respect to two different model selection criteria: the AICc (DSMEP AICc) and the app index (DSMEP app)
Error  Method  True  False  \(d(\mathcal {K}_{t}, \mathcal {C}_{n})\)  \(d(\mathcal {C}_{n}, \mathcal {K}_{t})\)  DTS 

rate  Pos  Pos  
Schrepp  0.997  0.168  0.003  0.168  0.171  
IITA  0.320  0.693  0.879  0.697  1.576  
.10  DSMEP AICc  0.979  0.788  0.021  0.935  0.956 
DSMEP app  0.980  0.786  0.020  0.932  0.953  
Schrepp  0.990  0.483  0.010  0.491  0.501  
IITA  0.389  0.693  0.775  0.723  1.498  
.20  DSMEP AICc  0.789  0.752  0.216  0.871  1.087 
DSMEP app  0.985  0.789  0.015  0.939  0.954  
Schrepp  0.952  0.680  0.048  0.732  0.779  
IITA  0.564  0.711  0.559  0.777  1.336  
.30  DSMEP AICc  0.274  0.602  1.004  0.632  1.636 
DSMEP app  0.999  0.804  0.001  0.963  0.964 
The performances of the three procedures in reconstructing a structure closed under both union and intersection (notice that \(d(\mathcal {K}_{t}, \mathcal {C}_{n})\), \(d(\mathcal {C}_{n}, \mathcal {K}_{t})\) and DTS take on values in the interval [0,Q/2])
Error  Method  True  False  \(d(\mathcal {K}_{t}, \mathcal {C}_{n})\)  \(d(\mathcal {C}_{n}, \mathcal {K}_{t})\)  DTS 

Rate  Pos  Pos  
Schrepp  0.991  0.004  0.009  0.004  0.012  
IITA  1  0.001  0  0.001  0.001  
.10  DSMEP AICc  0.393  0.184  0.699  0.184  0.883 
DSMEP app  0.976  0.439  0.024  0.457  0.481  
Schrepp  0.987  0.082  0.013  0.082  0.094  
IITA  0.935  0.046  0.065  0.046  0.111  
.20  DSMEP AICc  0.482  0.156  0.542  0.161  0.704 
DSMEP app  0.977  0.414  0.024  0.462  0.485  
Schrepp  0.982  0.424  0.012  0.440  0.459  
IITA  0.992  0.376  0.001  0.428  0.428  
.30  DSMEP AICc  0.421  0.138  0.735  0.139  0.873 
DSMEP app  0.991  0.566  0.009  0.642  0.651 
The performances of the three procedures in reconstructing a structure closed under intersection (notice that \(d(\mathcal {K}_{t}, \mathcal {C}_{n})\), \(d(\mathcal {C}_{n}, \mathcal {K}_{t})\) and DTS take on values in the interval [0,Q/2])
Error  Method  True  False  \(d(\mathcal {K}_{t}, \mathcal {C}_{n})\)  \(d(\mathcal {C}_{n}, \mathcal {K}_{t})\)  DTS 

Rate  Pos  Pos  
Schrepp  0.994  0.016  0.006  0.016  0.022  
IITA  0.588  0.492  0.492  0.568  1.060  
.10  DSMEP AICc  0.981  0.405  0.019  0.431  0.450 
DSMEP app  0.965  0.361  0.035  0.383  0.418  
Schrepp  0.991  0.408  0.009  0.415  0.424  
IITA  0.459  0.636  0.634  0.805  1.439  
.20  DSMEP AICc  0.798  0.343  0.209  0.369  0.575 
DSMEP app  0.969  0.599  0.031  0.699  0.730  
Schrepp  0.969  0.604  0.033  0.677  0.709  
IITA  0.604  0.584  0.525  0.676  1.201  
.30  DSMEP AICc  0.483  0.273  0.627  0.316  0.944 
DSMEP app  0.966  0.728  0.034  0.926  0.960 
Table 2 displays the results obtained in reconstructing a structure neither closed to union nor to intersection at increasing error rates. In these conditions, the best performing procedure is the one proposed by Schrepp. In fact, it maximizes the proportion of true states, minimizes the amount of false states in the reconstructed structure as well as the distances between the true and the reconstructed models. In this set of conditions, the underlying knowledge structure cannot be obtained through any skill map (since no closure is present). Nevertheless, it has to be stressed that the proposed method (using the app index as a stopping criterion) performs almost as well as the Schrepp’s one. Thus, it produces a structure that is adequately close to the true one, and it can be conveniently described by a skill map. Our procedure provides adequate results in all the three error rate conditions when the app is used as a stopping criterion, while the use of AICc seems to be much more critical when the error tends to increase. Concerning Tables 3 and 4, analogous conclusions can be drawn. For this reasons, it was decided to use the app as a stopping criterion. With respect to the performance of the IITA procedure, it has to be stressed that its performance is systematically lower when the true model is not a quasiordinal knowledge space. On the other hand, it is the best performing method when the underlying true structure satisfies this condition. It is important to stress that both Schrepp’s method and DSMEP do provide adequate results in a quite stable way across different true model and error rates conditions. This consideration seems to be further supported by the low value (i.e., always less than 1 item) of the DTS even with high error rates.
Discussion
In general, the results of the simulations support the goodness of the proposed reconstruction procedure, since the proportion of true and false states in the reconstructed structure as well as the average DTS are adequate in all the simulated conditions. The procedure has a better performance when the app index is used as a stopping criterion. Furthermore, as it could have been easily predicted, the procedure tends to perform better when careless error and lucky guess probabilities are relatively small. However, this difference in performance is not dramatic, allowing to trustfully apply the method even with high error rate.
The results suggest that the AICc criterion works adequately in some situations (namely those with low error rates), but fails in others, pointing that the app can be a better performing stopping criterion. Schrepp’s procedure does perform very well in all conditions. Finally, the IITA procedure seems to work very well when the underlying model satisfies both closures, while it underperforms in all other cases.
With respect to DSMEP, some criticisms have emerged around algorithm efficiency. For instance, the time required to carry out a single replication on rather small structures was about 500 seconds on a standard machine. This time is expected to increase very rapidly with the number of items. This indicates the need to improve the optimization of the computational steps.
Finally, in comparing DSMEP with Schrepp’s procedure, it could be useful to consider that they differ by three factors: (1) the selection method (minimum distance vs. maximum frequency); (2) closure under intersection vs. no closure; (3) parameter estimation method used for the computation of the app index (iterative vs. noniterative). This distinction would lead to a 2 (selection methods) ×2 (closure vs. no closure) ×2 (iterative app vs. non iterative app) factorial design. This would lead to the assessment of the other six procedures, all different from both DSMEP and the Schrepp’s one. Taking into account that differences in performance between DSMEP and the Schrepp’s procedure are fairly small in all simulations, it is not sure that this factorial study would help understanding such small differences.
In the next section, two empirical applications on clinical data are described in order to provide some sort of a tutorial about how to apply DSMEP.
Empirical applications
The eight items of the “Cleaning” and “Checking” subscales of the MOCQR
Item  Cleaning  Checking 

i _{1}  If I touch an animal, I feel contaminated  I frequently have to check things (e.g., gas or water taps, doors, etc.) 
i _{2}  I use too much soap to wash myself  In my job I am often late because I spend time to do the same thing over and over again 
i _{3}  I am not excessively concerned about cleanliness  I have to check a letter over and over again before post it 
i _{4}  I am not unduly concerned about germs  
and diseases  One of my major problems is that I pay too much attention to detail  
i _{5}  My hands feel dirty after touching money  My major problem is repeated checking 
i _{6}  I take rather a long time to complete my  
washing in the morning  I tend to check things more than once  
i _{7}  I use a great deal of detergents and antiseptics  I stick to a very strict routine when doing ordinary things 
i _{8}  Hanging and folding my clothes at night  
takes up a long time  I spend a lot of time every day checking things over and over again 
These empirical applications have been selected since KST has recently been applied to clinical assessment through the Formal Psychological Assessment (FPA; Bottesi et al., 2014; Spoto, Bottesi, Sanavio, & Vidotto, 2013). In FPA, the main concepts of KST are formulated according to clinical psychologyspecific issues: items are clinical questions included in surveys, skills are named attributes and refer to specific symptoms investigated by each item, careless error and lucky guess are, respectively, falsenegative and falsepositive rates. What remains unchanged is the methodology to obtain a structure starting from an attribute assignment through the conjunctive or disjunctive model and the probabilistic model related to the structures. One of the crucial issues in FPA is the conceptualization of the skills as the attributes, i.e., the clinical symptoms, diagnostic criteria, or characteristics that an individual has to present in order to affirmatively answer to a clinical item. In this perspective, the response pattern of a patient is directly related to the set of clinical symptoms he/she presents.
Methods
The questionnaires were administered to participants after they signed the informed consent form. No time limit was imposed, and participants were asked to answer all items accurately. After the administration of the tests, data were collected, and the frequencies of the response patterns were computed. The data consisted of a set of response patterns \(R \subseteq Q\) (where Q is the collection of items), with each response pattern with its own observed frequency F(R). The three procedures for structure construction were then applied. In the proposed procedure, for each iteration, the app index was computed, and all the BLIM’s parameters were estimated. The model with the minimum value of the app index was then selected and, by applying the procedure described in Section 2, the basis and the corresponding skill map were obtained. With respect to IITA and Schrepp’s procedures, they were applied using the same steps described for the simulation study. Three different structures were obtained and each of them was fitted to data by an application of the BLIM. The three fitted models were then compared through their Chisquare and AIC values.
Results
The performances of the three procedures in reconstructing the knowledge structure for the “Checking” and “Cleaning” subscales
Checking  

N states  χ ^{2}  df  Bootstrap p  AIC  
DSMEP  56  518.35  184  0.0008  35,675 
IITA  7  1098.50  233  <0.0001  36,049 
Schrepp  3  1368.70  237  <0.0001  36,246 
Cleaning  
DSMEP  34  359.48  206  0.7697  36,208 
IITA  39  301.54  201  0.9851  36,164 
Schrepp  4  497.72  236  0.0045  36,276 
Binary representation of the skill map obtained for the empirical application with respect to the “Checking” subscale
Attributes  

Items  a  b  c  d  e  f  g  h 
i _{1}  0  0  0  0  0  1  0  0 
i _{2}  0  1  0  1  1  1  1  1 
i _{3}  0  0  0  0  1  0  0  0 
i _{4}  0  0  0  0  0  0  1  0 
i _{5}  0  0  1  0  0  0  0  1 
i _{6}  0  0  0  0  0  0  0  1 
i _{7}  1  0  0  0  0  0  0  0 
i _{8}  0  0  0  1  1  1  1  1 
Estimated BLIM’s parameters β and η for each item of the “Checking” and “Cleaning” subscales of the MOCQR (n.i. means that the parameter was not identifiable)
Structures  

obtained by  i _{1}  i _{2}  i _{3}  i _{4}  i _{5}  i _{6}  i _{7}  i _{8}  
Checking  
DSMEP  n.i.  0.057  n.i.  n.i.  0.027  n.i.  n.i.  0.013  
η  IITA  0.183  0.052  n.i.  n.i.  n.i.  n.i.  0.053  n.i. 
Schrepp  0.187  0.051  n.i.  0.284  0.051  0.354  0.277  0.006  
DSMEP  0.258  n.i.  0.346  0.184  n.i.  0.050  n.i.  0.381  
β  IITA  n.i.  0.638  0.356  0.226  0.138  0.044  n.i.  0.398 
Schrepp  0.304  0.757  0.465  0.219  0.141  0.036  0.198  0.630  
Cleaning  
DSMEP  0.065  0.025  0.010  n.i.  n.i.  n.i.  0.009  n.i.  
η  IITA  0.037  0.028  0.139  0  0.081  0  n.i.  0.064 
Schrepp  0.054  0.020  0.055  0  0.258  0.132  n.i.  0.347  
DSMEP  0.719  0.525  0.233  0.449  0.335  0.315  0.568  0.331  
β  IITA  0.514  0.518  0.250  0.409  0.316  0.284  0.191  0.345 
Schrepp  0.764  0.666  0.254  0.453  0.324  0.349  0.689  0.312 
Binary representation of the skill map obtained for the empirical application with respect to the “Cleaning” subscale
Attributes  

Items  a  b  c  d  e  f 
i _{1}  1  1  1  1  1  1 
i _{2}  1  1  1  1  1  1 
i _{3}  1  0  0  0  0  0 
i _{4}  0  0  0  0  1  0 
i _{5}  0  0  0  0  0  1 
i _{6}  0  0  0  1  0  0 
i _{7}  0  1  1  1  1  1 
i _{8}  0  0  1  0  0  0 
From the table it emerges that items i _{1} and i _{2} have the same set of attributes, thus they are equally informative, i.e., they are systematically present in the same states of the structure. Table 8 displays the error rate estimates. As in the previous application, some of the parameters are not identifiable. In this application, the best performing procedure happened to be the IITA that reconstructed a structure containing 39 states with a better fit of the BLIM (χ ^{2}(201)=301.54; bootstrap p=.99; AIC= 36,164). Notice that even in this case, DSMEP did not greatly differed from the best performing one. Once again, the Schrepp’s procedure reconstructed a model containing only four states and with an inadequate fit to the BLIM (χ ^{2}(236)=497.72; bootstrap p=.005; AIC = 36,276).
Discussion
The results of the empirical applications of the procedure show both the applicability of the methodology and its good performance compared to the alreadyexisting methodologies to reconstruct a knowledge structure starting from a set of data. The applications provide also some information about the possibility of interpreting the derived skill map in clinical context. More specifically, the application to the “Checking” subscale presents some interesting aspects that are noteworthy. First, the obtained structure counts only 56 out of the 256 potential response patterns (i.e., the powerset on the set of eight items). Furthermore, a remark has to be referred to the fact that the reconstructed skill map has as many skills as the number of items, although this does not mean that items are independent. For instance, if items i _{5} and i _{6} are considered, it can be argued that if no error is taken into account, an affirmative response to the former implies an affirmative answer to the latter since the set of attributes referred to i _{6} is strictly included in the set of attributes of i _{5}.
The application to the “Cleaning” subscale, on the other hand, depicts a case in which the number of attributes is less than the number of items. Moreover, the proposed procedure identifies two items that need the same set of attributes. The critical aspect about this application is referred to the high values of the error rate estimates obtained with all three reconstruction procedures. Response errors might have a quite different interpretation in a clinical questionnaire. They may reflect the tendency to simulate or dissimulate a symptom, depending on social desirability or other similar issues. This can lead to substantially greater error probabilities. It is noteworthy how Schrepp’s procedure fails even in this application, while the IITA reconstructs a model that better fits the BLIM. These results seem to be in contradiction with those obtained in the simulation study. A possible issue could be the characteristics of the simulated models. To verify this, 100 data sets were simulated by the BLIM, using as “true states” the knowledge structure reconstructed by IITA and as “true parameters” those estimated by fitting the BLIM to data with the knowledge structure obtained by IITA. Schrepp’s procedure was then applied to each of the 100 data sets and the DTS, TPR, and FPR statistics were computed. The mean DTS was 1.11, the mean TPR was .079, the mean FPR was 0, the mean number of states in the reconstructed structure was 3.07. These results (especially the TPR and the number of reconstructed states) show that the apparent contradiction between simulations and empirical application depend in all probability on the specific choice that was made in the simulation study, concerning structures and parameter values. A choice that could have facilitated, somehow Schrepp’s procedure.
Finally, it has to be stressed that all the obtained models present some identification problems. Such problems, as demonstrated in recent literature (Spoto et al., 2012, 2013), can be detected a priori from the obtained skill maps (when the proposed procedure is applied), or from the reconstructed structure.
With respect to the interpretability of the obtained skill maps, it emerged that two items investigates the same set of attributes, thus they can be considered as equivalent in conveying diagnostic information about cleaning rituals. Furthermore, a number of items investigate only one clinical attribute indicating that their information represents a specific element useful for a precise case formulation. Finally, item 7 provides information about a number of attributes that are investigated also by other items. This last issue could be fruitfully implemented into an automatic adaptive procedure that could allow for a reduction of the time needed to administer the scale.
General discussion
In this article, a procedure to construct a skill map starting from a data set has been described. This procedure is mainly based on the possibility of deriving a conjunctive skill map from a closure space or a disjunctive skill map from a knowledge space. The procedure begins, in the former case, with the minimal closure space (i.e., the one including only the empty set and the total set) and iteratively tries to improve such structure by minimizing the distance between the reconstructed closure space and the observed data. One of the most critical aspects of the procedure is the stopping criterion. In the present paper, the use of AICc has been tested and compared to the use of the app criterion developed by Schrepp for his procedure (Schrepp, 1999a) through a number of simulation studies. The results highlighted a better performance of this last index. In general, the procedure accurately reproduces the structure even when the error rate of the data (measured by parameters η and β) are relatively high (i.e., up to 0.30).
In order to test the applicability of the procedure in a real setting, two empirical studies were conducted on clinical data. These studies suggested the procedure is concretely applicable and that the obtained results are interpretable. It must be stressed that in this article, the procedure has been tested on relatively small structures. This was due to the high time consumption of the procedure to carry out the reconstruction on rather small domains. In order to apply the procedure to larger structures, the algorithm must be improved, since, in the present version, it appears to be too computationally expensive. Nevertheless, both the basic ideas and the method proposed in this study look very promising for a new way to build structures and skill maps starting from filled questionnaires.
The proposed procedure applies to conjunctive skill maps but can be trivially adapted to the disjunctive model. In that case, each structure in the chain will be closed under union, and the disjunctive skill map will simply correspond to the basis of the obtained space.
Lastly, the empirical application shows that the skills in the conjunctive skill map extracted from the data may then be interpreted by some experts in the field. This suggests the possibility of constructing hybrid procedures that rely on both the expert’s opinion about the relationships between items and skills and the information contained in a data set. For instance, the expert may provide a rough initial skill map that is then further improved by an extraction method like the one presented in this article.
To summarize, the DSMEP procedure presents a number of pros and some cons to be taken into account. The main cons are referred to the rather poor efficiency of the procedure in terms of computational time and to the stopping criterion. With respect to the former, it has to be stressed how a major improvement could be obtained by removing the closure constrain. In fact, if no closure is required, then the procedure may improve much in efficiency. Concerning the latter, further studies are needed. On the other hand, DSMEP successfully copes with some of the main limitations of the alreadyexisting procedures. First, the proposed procedure allows building an either conjunctive or disjunctive skill map starting from a set of observed data. DSMEP could be modulated in order to obtain either a knowledge space or a closure space, or a quasiordinal knowledge space. Second, differently form Schrepp’s procedure, it allows reconstructing states that are not observed in the sample at hand, provided that they are the intersection (or union) of observed states. Moreover, again in contrast with Schrepp’s procedure, DSMEP does not assume any equality constraints neither on the state probabilities nor on the η and β parameters. Finally, it proved to be robust even with high error rates.
References
 Bottesi, G., Spoto, A., Sanavio, E., & Vidotto, G. (2014). Beyond the score: clinical evaluation through formal psychological assessment. Journal of Personality Assessment. doi: 10.1080/00223891.2014.958846.
 Brockwell, P.J., & Davis, R.A. (1991). Time series: theory and methods, 2nd edn. New York: Springer.Google Scholar
 Burnham, K,P., & Anderson, D.R. (2002). Model selection and multimodel inference. A practical informationtheoretic approach, 2nd edn. New York: Springer.Google Scholar
 Cosyn, E., & Thiéry, N. (2000). A practical procedure to build a knowledge structure. Journal of Mathematical Psychology, 44, 383–407.CrossRefPubMedGoogle Scholar
 DeLaTorre, J., & Douglas, J. (2008). Model evaluation and multiple strategies in cognitive diagnosis: an analysis of fraction subtraction data. Psychometrika, 73, 595–624.CrossRefGoogle Scholar
 Desmarais, M.C., & Maluf, A. (1996). Userexpertise modeling with empirically derived probabilistic implication networks. User Modeling and UserAdapted Interaction, 5, 283–315.CrossRefGoogle Scholar
 Doignon, J.P. (1994). Knowledge spaces and skill assignments. In: Fischer, G. H., & Laming, D. (Eds.) Contributions to mathematical psychology, psychometrics, and methodology. (pp. 111–121). New York: SpringerVerlag.Google Scholar
 Doignon, J.P., & Falmagne, J.C. (1985). Spaces for the assessment of knowledge. International Journal of ManMachine Studies, 23, 175–196.CrossRefGoogle Scholar
 Doignon, J.P., & Falmagne, J.C. (1999). Knowledge spaces. Berlin and Heidelberg: SpringerVerlag.CrossRefGoogle Scholar
 Düntsch, I., & Gediga, G. (1995). Skills and knowledge structures. British Journal of Mathematical Psychology, 48, 9–27.CrossRefGoogle Scholar
 Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap Boca Raton, FL Chapman & Hall CRC.Google Scholar
 Falmagne, J.C., Doble, C., Albert, D., Eppstein, D., & Hu, X. (2013). Knowledge spaces: Applications in education. Berlin and Heidelberg: SpringerVerlag.CrossRefGoogle Scholar
 Falmagne, J.C., & Doignon, J.P. (1988a). A class of stochastic procedures for the assessment of knowledge. British Journal of Mathematical and Statistical Psychology, 41, 1–23.CrossRefGoogle Scholar
 Falmagne, J.C., & Doignon, J.P. (1988b). A Markovian procedure for assessing the state of a system. Journal of Mathematical Psychology, 32, 232–258.CrossRefGoogle Scholar
 Falmagne, J.C., & Doignon, J.P. (2011). Learning spaces. Berlin and Heidelberg: SpringerVerlag.CrossRefGoogle Scholar
 Falmagne, J.C., Koppen, M., Villano, M., Doignon, J.P., & Johannesen, L. (1990). Introduction to knowledge spaces: how to build, test and search them. Psychological Review, 97, 201–224.CrossRefGoogle Scholar
 Heller, J., Ünlü, A., & Albert, D. (2013). Skills, competencies and knowledge structures. In: Falmagne, J. C., Doble, C., Albert, D., Eppstein, D., & Hu, X. (Eds.) Knowledge spaces: Applications in education. (pp. 229–242). Berlin and Heidelberg: SpringerVerlag.Google Scholar
 Kambouri, M., Koppen, M., Villano, M., & Falmagne, J.C. (1993). Knowledge assessment: tapping human expertise by the QUERY routine. International Journal of Human Computer Studies, 40, 119–151.CrossRefGoogle Scholar
 Koppen, M. (1993). Extracting human expertise for constructing knowledge spaces: an algorithm. Journal of Mathematical Psychology, 37, 1–20.CrossRefGoogle Scholar
 Koppen, M., & Doignon, J.P. (1990). How to build a knowledge space by querying an expert. Journal of Mathematical Psychology, 34, 311–331.CrossRefGoogle Scholar
 Korossy, K. (1997). Extending the theory of knowledge spaces: a competenceperformance approach. Zeitschrift für Psychologie, 205, 53–82.Google Scholar
 Sanavio, E., & Vidotto, G. (1985). The components of the Maudsley ObsessionalCompulsive Questionnaire. Behaviour Research and Therapy, 23, 659–662.CrossRefPubMedGoogle Scholar
 Sargin, A., & Ünlü, A. (2009). Inductive item tree analysis: corrections improvements and comparisons. Mathematical Social Sciences, 58, 376–392.CrossRefGoogle Scholar
 Schrepp, M. (1999a). Extracting knowledge structures from observed data. British Journal of Mathematical and Statistical Psychology, 52, 213–224.CrossRefGoogle Scholar
 Schrepp, M. (1999b). On the empirical construction of implications between bivalued test items. Mathematical Social Sciences, 38, 361–375.CrossRefGoogle Scholar
 Schrepp, M. (2002). Explorative analysis of empirical data by Boolean analysis of questionnaires. Zeitschrift für Psychologie, 210(2), 99–109.CrossRefGoogle Scholar
 Schrepp, M. (2003). A method for the analysis of the hierarchical dependencies between items of a questionnaire. Methods of Psychological Research  Online, 19, 43–79.Google Scholar
 Smith, N.A., & Tromble, R.W. (2004). Sampling uniformly from the unit simplex, Technical Report, John Hopkins University.Google Scholar
 Spoto, A., Bottesi, G., Sanavio, E., & Vidotto, G. (2013). Theoretical foundations and clinical implications of formal psychological assessment. Psychotherapy and Psychosomatics, 82(3), 197–199.CrossRefPubMedGoogle Scholar
 Spoto, A., Stefanutti, L., & Vidotto, G. (2010). Knowledge space theory, formal concept analysis, and computerized psychological assessment. Behavior Research Methods, 42, 342–350.CrossRefPubMedGoogle Scholar
 Spoto, A., Stefanutti, L., & Vidotto, G. (2012). On the unidentifiability of a certain class of skill mapbased probabilistic knowledge structures. Journal of Mathematical Psychology, 56(4), 248–255.CrossRefGoogle Scholar
 Spoto, A., Stefanutti, L., & Vidotto, G. (2013). Considerations about the identification of forward and backwardgraded knowledge structures. Journal of Mathematical Psychology, 57(4), 249–254.CrossRefGoogle Scholar
 Stefanutti, L. (2006). A logistic approach to knowledge structures. Journal of Mathematical Psychology, 50, 545–561.CrossRefGoogle Scholar
 Stefanutti, L., & Koppen, M. (2003). A procedure for the incremental construction of a knowledge space. Journal of Mathematical Psychology, 47, 265–277.CrossRefGoogle Scholar
 Templin, J.L., & Henson, R.A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.CrossRefPubMedGoogle Scholar
 Ünlü, A., & Albert, D. (2004). The correlational agreement coefficient CA(≤,D)—a mathematical analysis of a descriptive goodnessoffit measure. Mathematical Social Sciences, 48, 281–314.CrossRefGoogle Scholar
 Ünlü, A., Schrepp, M., Heller, J., Hockemeyer, C., Wesiak, G., & Albert, D. (2013). Recent developments in performancebased knowledge space theory. In: Falmagne, J. C., Doble, C., Albert, D., Eppstein, D., & Hu, X. (Eds.) Knowledge spaces: Applications in education. (pp. 147–192). Berlin and Heidelberg: SpringerVerlag.Google Scholar
 vanLeeuwe, J.F.J. (1974). Item tree analysis. Nederlands Tijdschrift voor de Psychologie, 29, 475–484.Google Scholar