Polytomous Effectiveness Indicators in Complex Problem-Solving Tasks and Their Applications in Developing Measurement Model

Abstract

Recent years have witnessed the emergence of measurement models for analyzing action sequences in computer-based problem-solving interactive tasks. The cutting-edge psychometrics process models require pre-specification of the effectiveness of state transitions often simplifying them into dichotomous indicators. However, the dichotomous effectiveness becomes impractical when dealing with complex tasks that involve multiple optimal paths and numerous state transitions. Building on the concept of problem-solving, we introduce polytomous indicators to assess the effectiveness of problem states $d_{s}$ and state-to-state transitions ${\mathrm {\Delta }d}_{\mathrm {s\rightarrow s'}}$. The three-step evaluation method for these two types of indicators is proposed and illustrated across two real problem-solving tasks. We further present a novel psychometrics process model, the sequential response model with polytomous effectiveness indicators (SRM-PEI), which is tailored to encompass a broader range of problem-solving tasks. Monte Carlo simulations indicated that SRM-PEI performed well in the estimation of latent ability and transition tendency parameters across different conditions. Empirical studies conducted on two real tasks supported the better fit of SRM-PEI over previous models such as SRM and SRMM, providing rational and interpretable estimates of latent abilities and transition tendencies through effectiveness indicators. The paper concludes by outlining potential avenues for the further application and enhancement of polytomous effectiveness indicators and SRM-PEI.

Appendix A. Algorithm for Automatically Calculating State Effectiveness in the Balance Beam Task

Fig. 5

The interface of the initial state in the Chinese version of the Balance Beam task.

Fig. 6

The diagram for the four types of transitions that can occur when a weight moves among ten possible positions in the Balance Beam task.

In the Balance Beam task, the ten potential positions for each weight are categorized into four groups: (1) Positions 1–4: Positioned on side A of the beam; (2) Position 5: Not suspended on side A; (3) Position 6: Not suspended on side B; (4) Positions 7–10: Positioned on side B of the beam. Figure 5 illustrates the transition of each weight among ten positions through four types of operations: (1) removing a weight from the beam; (2) hanging an unhung weight; (3) passing a weight to the other student; and (4) shifting the position of a weight on the same side. Each arrow represents an operation that can lead to a transition. Through this figure, we can easily find the minimum number of transitions between any two positions for one weight. Since an operation can only alter the position of one weight once, the shortest distance between states s and $s'$ equals the sum of the minimum number of operations required for each of the four weights to change its position from state s to $s'$. Then, we can quickly and accurately calculate the shortest distance $d_{s}^{(k)}$ between a state s and the target state $s_{target}^{(k)}$ using the state code and rules to change the position according to Fig. 6. Finally, we select the minimum distance $d_{s}=\min \left( d_{s}^{(1)}, d_{s}^{(2)}, \ldots , d_{s}^{(k)}\right)$ as the effectiveness indicator $d_{s}$ of the state s

During the process of programming the calculations mentioned above, the position of each weight can be assigned a unique number from one to ten. Therefore, any given state in the Balance Beam task can be encoded by a sequence of four numbers, a representation we refer to as the state code. For one weight, calculating the shortest distance between any two positions can be simplified by several rules. The R code for evaluating the effectiveness of states for the Balance Beam task that requires the use of two weights to achieve balance is available at https://osf.io/fw82q/.

In the example of the code, the four positions for hanging weights on the balance beam on student A’s side are coded as 1 to 4, and the four positions on student B’s side are coded as $-1$ to $-4$. The unhung weights are coded as 0.5 when in student A’s hand and $-$0.5 when in student B’s hand. In the initial state, all four weights are in the hand of A, and the state code is (0.5, 0.5, 0.5, 0.5). The effectiveness of the initial state is equal to 3, which means that the balance state using two weights can be achieved after a minimum of three transitions. Another example is that Student B holds the 50 g and 100 g weights and Student A has hung the 300 g weight at position 1 and the 500 g weight at position 2. This state is at a minimum distance of 2 from the balance state.