Accurate Assessment via Process Data

Abstract

Accurate assessment of a student’s ability is the key task of a test. Assessments based on final responses are the standard. As the infrastructure advances, substantially more information is observed. One of such instances is the process data that is collected by computer-based interactive items and contain a student’s detailed interactive processes. In this paper, we show both theoretically and with simulated and empirical data that appropriately including such information in the assessment will substantially improve relevant assessment precision.

Appendix: Proofs of Theorem 1 and Theorem 2

To prove Theorem 1, we establish the following lemma.

Lemma 1

Let X be a nonconstant random variable, and $f(\cdot )$ and $g(\cdot )$ be strictly increasing functions. Suppose that f(X) and g(X) have finite second moments. Then, ${{\,\mathrm{Cov}\,}}\left( f(X), g(X) \right) >0$ .

Proof of lemma 1

Let Y be an independent and identically distributed (i.i.d.) copy of X. It is easy to verify the following identity

$$\begin{aligned} {{\,\mathrm{Cov}\,}}\left( f(X), g(X) \right) = \frac{1}{2} E \left[ \left( f(X) -f(Y) \right) \left( g(X) - g(Y) \right) \right] . \end{aligned}$$

(10)

Clearly, for any x and y, $(f(x) -f(y) ) (g(x) - g(y))\ge 0$, and $``=''$ holds if and only if $x=y$. Since $P(X\not =Y)>0$, the right-hand side of equation (10) must be positive. $\square$

Proof of Theorem 1

By Assumption A2 (local independence),

$$\begin{aligned} T_{{\mathbf {X}}_{-j}} = E \left[ {{\hat{\theta }}}_{Y_j} | {\mathbf {X}}_{-j}\right] = E \left[ E \left[ {{\hat{\theta }}}_{Y_j} | {\mathbf {X}}_{-j} ,\theta \right] | {\mathbf {X}}_{-j} \right] = E \left[ E \left[ {{\hat{\theta }}}_{Y_j} | \theta \right] | \mathbf{X}_{-j} \right] = E \left[ m_j (\theta ) | {\mathbf {X}}_{-j} \right] . \end{aligned}$$

Due to Assumption A3 (exponential family), the posterior distribution of $\theta$ given ${\mathbf {X}}_{-j}$ depends on $\mathbf{X}_{-j}$ only through the sufficient statistic $T_j({\mathbf {X}}_{-j})$. In fact,

$$\begin{aligned} T_{{\mathbf {X}}_{-j}} = E \left[ m_j (\theta ) | {\mathbf {X}}_{-j} \right] = G_j(T_j({\mathbf {X}}_{-j})), \end{aligned}$$

where $G_j(t) =E \left[ m_j (\theta ) |T_j({\mathbf {X}}_{-j})=t\right]$. Furthermore, by making use of the exponential family form in Assumption A3 and the simple exchange of order of differentiation and integration, we can show that

$$\begin{aligned} G_j'(t) = {{\,\mathrm{Cov}\,}}\left[ m_j(\theta ),\eta _j(\theta ) | T_j(\mathbf{X}_{-j})=t \right] . \end{aligned}$$

Since both $m_j$ and $\eta _j$ are strictly monotone, Lemma 1 implies that $G_j'(t)$ is strictly positive or negative for all t and, therefore, $G_j$ is strictly monotone. In other words, there is a one-to-one mapping between $T_{{\mathbf {X}}_j}$ and $T_j({\mathbf {X}}_{-j})$. $\square$

Proof of Theorem 2

From Theorem 1, we know that $T_{\mathbf{X}_{-j}}$ is a sufficient statistic of ${\mathbf {X}}_{-j}$ for each j. Since ${{\hat{\theta }}}_{{\mathbf {Y}}}$ is a function of ${\mathbf {Y}}$ and $\sigma ({\mathbf {Y}}_{-j}) \subseteq \sigma ({\mathbf {X}}_{-j})$, the conditional distribution ${{\hat{\theta }}}_{{\mathbf {Y}}} | T_{\mathbf{X}_{-j}}, Y_j$ is free of $\theta$. Therefore, we have $E[{{\hat{\theta }}}_{{\mathbf {Y}}} | T_{{\mathbf {X}}_{-j}}, Y_j, \theta ] = E[{{\hat{\theta }}}_{{\mathbf {Y}}} | T_{{\mathbf {X}}_{-j}}, Y_j ] = \hat{\theta }_{{\mathbf {X}}_{-j}}.$ It follows from the well-known Rao–Blackwell theorem (Casella & Berger, 2002) that ${{\hat{\theta }}}_{{\mathbf {X}}_{-j}}$ reduces the conditional variance and

$$\begin{aligned} E[({{\hat{\theta }}}_{{\mathbf {X}}_{-j}} - \theta )^2 | \theta ] \le E[({{\hat{\theta }}}_{{\mathbf {Y}}} - \theta )^2 | \theta ] \end{aligned}$$

holds for every j and $\theta$.

By Cauchy–Schwarz inequality, we get

$$\begin{aligned} E[({{\hat{\theta }}}_{{\mathbf {X}}} - \theta )^2 | \theta ] \le E \left[ \frac{1}{J} \sum _{j=1}^J ({{\hat{\theta }}}_{{\mathbf {X}}_{-j}} - \theta )^2 | \theta ] \le E[({{\hat{\theta }}}_{{\mathbf {Y}}} - \theta )^2 | \theta \right] . \end{aligned}$$