Incremental validity of placekeeping as a predictor of multitasking

Abstract

Multitasking is ubiquitous in everyday life, which means there is value in developing measures that predict successful multitasking performance. In a large sample (N = 404 contributing data), we examined the predictive and incremental validity of placekeeping, which is the ability to perform a sequence of operations in a certain order without omissions or repetitions. In the context of multitasking, placekeeping should play a role in the performance of procedural subtasks and the interleaving of subtasks that interrupt each other. Regression analyses revealed that placekeeping ability accounted for 11% of the variance in multitasking performance, and had incremental validity relative to each of a diverse set of cognitive abilities (working memory capacity, fluid intelligence, perceptual speed, and crystallized intelligence). The predictive validity of placekeeping for multitasking was stable across samples of performance and robust to placekeeping practice. Broader measures of performance on our placekeeping task accounted for 21% of the variance in multitasking performance and had incremental validity relative to an estimate of psychometric g. The results provide evidence that placekeeping is a distinct cognitive ability with its own specific role to play in multitasking, and raise the possibility that measures of placekeeping ability could have utility in selecting personnel for occupations that require certain kinds of multitasking, such as interleaving of procedures.

Appendix

Here we explore the predictive and incremental validity of the UNRAVEL task as a whole, looking beyond placekeeping measures. Like many complex tasks, UNRAVEL affords multiple measures of behavior that collectively reflect a range of different cognitive processes. From a practical perspective, when testing time is costly or limited it makes sense to wring the most possible from each instrument in a battery, and we wanted to assess what predictive value the UNRAVEL task would have for multitasking if we used every predictive piece of it.

We tested seven measures selected to span all aspects of task performance during an UNRAVEL session. The measures are exhaustive but also exclusive, in that they are recorded from non-overlapping sets of events (e.g., from distinct sets of trials). Descriptive statistics and reliabilities are presented in Table 5. Six of the seven measures had acceptable reliability (coefficient alpha > .70). The seventh (introduction duration, described below) did not have repeated measures from which to compute reliability. Correlations among the measures and between the measures and multitasking are presented in Table 6. Six of seven measures correlated significantly with multitasking in the expected (negative) direction, and the seventh trended in that direction.

Table 5 Descriptive statistics for expanded set of UNRAVEL measures

Table 6 Correlations for multitasking and expanded set of UNRAVEL measures

We tested three new measures: the choice-rule error rate, introduction duration, and interruption duration. The choice-rule error rate is the proportion of trials on which the participant selects the correct step but chooses the wrong response according to the rule for that step (see Fig. 1). Choice-rule errors and placekeeping errors thus occur on different trials, and neither is included in response time, which is recorded on correct trials only. Together, choice-rule errors, placekeeping errors, and response times measure behavior on all trials during test blocks. Introduction duration is the time spent on the introductory phase that occurs before the test blocks in an experimental session. During this phase, participants learn the task and receive practice trials and interruptions. Introduction duration incorporates effects of errors, because the computer requires correct responses during this phase, so an error costs time as the participant must respond again. Finally, interruption duration is the time spent typing the “codes” presented during interruptions in test blocks. This measure similarly incorporates effects of errors, because an incorrectly typed code costs time as the participant must try again.

The four remaining measures were related to placekeeping. Here we divided the placekeeping error rate and response time each into two variants according to a distinction between post-interruption trials, which immediately follow an interruption, and non-interruption trials, which immediately follow another trial. Post-interruption trials generate more placekeeping errors and longer response times than non-interruption trials, reflecting the effect of interruptions on memory for the most recently performed step (Altmann et al., 2017). In the body of the paper, we combined the two trial types because the underlying placekeeping operations are hypothetically the same, differing only in the age of memory for recently performed trials (Altmann & Trafton, 2015). Here we separate them to explore the effect of their empirical differences on predictive validity. We do not separate choice-rule errors by trial type, because empirically they do not differ by trial type (Altmann et al., 2014).

Predictive validity

We performed a regression analysis to test the predictive validity of these seven measures for multitasking performance. The results are presented in Table 7.

Table 7 Regression analysis testing predictive validity of UNRAVEL measures for multitasking performance

There are two main findings. First, the full model explained 21% of the variance in multitasking performance. This is nearly double the 11% of variance explained when we restricted our analysis to placekeeping measures (Table 3), indicating that there is more to UNRAVEL performance than placekeeping, and that some of the additional processes also play a role in multitasking. Twenty-one percent trends larger than the variance explained by working memory capacity and perceptual speed (20% and 17%, respectively; Table 4), and is substantially more than the variance explained by crystallized intelligence (7%), but is substantially less than the variance explained by fluid intelligence (36%).

Second, four measures—introduction duration, interruption duration, the post-interruption placekeeping error rate, and non-interruption response time—were significant predictors, suggesting that they captured systematic variance that is worth trying to interpret in theoretical terms. Introduction duration may measure a person’s ability to acquire a procedure quickly, and for multitasking may predict the ability to develop procedures or strategies for interleaving subtasks. Interruption duration measures speed and accuracy of typing and presumably of the perceptual processes needed to correctly encode a string of randomly-ordered letters, processes that seem basic to many tasks. Interruption duration is also a predictor of the placekeeping error rate in our cognitive model (Altmann, et al., 2017) because it affects memory for the step performed before the interruption, but any placekeeping-related variance it explains in multitasking may be mediated by the placekeeping error rate.

Finally, the post-interruption placekeeping error rate and non-interruption response time are two of the four variants of our placekeeping measures. The other two variants—the non-interruption placekeeping error rate and post-interruption response time—were not significant predictors. In statistical terms, the reason is probably that, with reference to Table 6, each non-significant measure correlated strongly with its significant counterpart (e.g., r = .71 for non-interruption and post-interruption placekeeping errors), but correlated somewhat less strongly with multitasking than did its counterpart (e.g., r = − .22 for non-interruption placekeeping errors and multitasking vs. r = − .30 for post-interruption placekeeping errors and multitasking) and thus was dominated in the model by its significant counterpart. The strong correlations between the two placekeeping error variants and between the two response time variants support our assumption earlier that the same underlying mechanisms account for performance of the two trial types. Whether the somewhat different correlations with multitasking have a theoretical basis or merely reflect sampling variability is an interesting question for future work.

Incremental validity

We performed hierarchical regression analyses to test the incremental validity of the four significant UNRAVEL predictors relative to the measure of psychometric g we reported in the body of the paper. In Step 1, we entered psychometric g, and in Step 2, we entered the four significant UNRAVEL predictors. We performed this analysis with the UNRAVEL measures aggregated over the full session and also for Block 1 alone to assess the possibility of using a reduced version of the task to save testing time.

The UNRAVEL measures entered in Step 2 explained an additional 1.6% of variance in multitasking performance, F(4, 385) = 2.90, p = .022 for the full-session analysis and an additional 1.5% of variance, F(4, 385) = 2.66, p = .033 for the Block 1 analysis. Thus, an expanded set of UNRAVEL measures showed incremental validity for multitasking relative to psychometric g, whether they were collected from the whole session or only from Block 1. In terms of testing time, there would be considerable savings from using just Block 1. The duration of the introductory phase plus Block 1 (M = 18 m, SD = 3.8 m) was substantially shorter than the duration of the full session (M = 43 m, SD = 8.5 m), and is similar to the total duration of the three fluid intelligence indicators (about 20 m).

An additional 1.6% of criterion variance explained may seem a trivial amount, but increments in R² underestimate the practical significance of incremental validity (Taylor & Russell, 1939; see also Hambrick, Burgoyne, & Oswald, 2019). For example, per Taylor and Russell’s tables, an additional 1.0% of criterion variance explained (i.e., r = .10) increases the proportion of new employees considered satisfactory by 5%, for a job in which 50% of existing employees are considered satisfactory and the selection ratio for new employees is 30%. For jobs that require extensive training, or in which procedural or placekeeping errors are especially costly, the benefits of the extra test could outweigh the costs. For purposes of this analysis, we suppose that if a given instrument—UNRAVEL, here, with all significant predictors used—explains criterion variance above and beyond psychometric g, it may also explain criterion variance above and beyond any individual aptitude test, given that specific aptitudes rarely explain more variance than g (Schmidt et al., 1992). In general, other complex tasks may add predictive value also if task performance is measured in a similarly comprehensive manner.