Latent Feature Extraction for Process Data via Multidimensional Scaling

Abstract

Computer-based interactive items have become prevalent in recent educational assessments. In such items, detailed human–computer interactive process, known as response process, is recorded in a log file. The recorded response processes provide great opportunities to understand individuals’ problem solving processes. However, difficulties exist in analyzing these data as they are high-dimensional sequences in a nonstandard format. This paper aims at extracting useful information from response processes. In particular, we consider an exploratory analysis that extracts latent variables from process data through a multidimensional scaling framework. A dissimilarity measure is described to quantify the discrepancy between two response processes. The proposed method is applied to both simulated data and real process data from 14 PSTRE items in PIAAC 2012. A prediction procedure is used to examine the information contained in the extracted latent variables. We find that the extracted latent variables preserve a substantial amount of information in the process and have reasonable interpretability. We also empirically prove that process data contains more information than classic binary item responses in terms of out-of-sample prediction of many variables.

Appendix

To compare the prediction performance of features extracted from different dissimilarity measures, we compute the Levenshtein distance matrix of the action sequences for each item and extracted features using Procedure 1 with the number of features K chosen by fivefold cross-validation. With these newly extracted features, we repeat the experiment of score prediction using multiple items (Section 4.5.2). All the settings are the same as before. A comparison of the prediction performance with the results in the main text is presented in Figure 12. Although the $\text {OSR}^2$ for the Levenshtein distance features is lower than that for the features extracted previously, it is still higher than that from the baseline model and the general trend of $\text {OSR}^2$ as the number of items increases is similar.

Fig. 12

Comparison of score prediction for features extracted based on different dissimilarity measures. “OSS” and “L” stand for the measure used in the main text and the Levenshtein distance, respectively.