Individual Differences and Motivational Effects
Why do some people improve on untrained tasks following cognitive training while others do not? One possibility is that there are individual difference factors that play a key role in cognitive training outcomes. The present chapter examines a range of these factors, including baseline performance, age, personality, and motivation. Some of these factors, such as baseline performance and age, have long been examined in the context of cognitive training, and extant research provides evidence that they contribute to the outcome of both training-related improvements as well as transfer gains. Other factors, including personality and motivation, remain largely unexamined in the context of cognitive training, but preliminary research indicates that they may play a substantial role in the success of these interventions. We suggest that researchers ignore these factors at their peril and that future cognitive training studies should incorporate measures of individual differences in studies well powered enough to examine them. Furthermore, it is possible that for training interventions to be broadly successful for large populations, they must be personalized to take these factors into account.
KeywordsWorking memory Motivation Age Personality Baseline performance Transfer
Reproducibility is an essential feature of high-quality psychological research, and the subfield of cognitive training has produced no shortage of replications and follow-up studies. For example, one intervention developed by two of the authors of this chapter, n-back training, has been used in no less than fourteen studies from other research groups that also examined training in the context of transfer to fluid intelligence (Au et al. 2014). The outcome of these studies, however, is inconsistent: While some studies find improvements in untrained tasks following the intervention, others do not. Although these subsequent studies may not adhere to the level of fidelity with the original research that was attempted in the recent Reproducibility Project (Open Science Collaboration 2015), similar or identical training paradigms and transfer tasks were used. How does one make sense of not only a single failed or successful replication but an entire corpus of divergent results?
One potential answer to this conundrum is the use of meta-analytic techniques, as these allow researchers to determine not only whether an intervention has a meaningful effect over a large body of studies but also to reveal potential moderators—such as demographic makeup, pre-existing individual differences, or training dosage—that may influence the outcome of the intervention. However, even the handful of extant meta-analyses (Au et al. 2014; Karbach and Verhaeghen 2014; Melby-Lervåg and Hulme 2013; Schwaighofer et al. 2015) arrive at different conclusions about the efficacy of cognitive training. A brief interrogation of these quantitative reviews makes this outcome unsurprising, considering that each meta-analysis relied on different selection criteria for including studies and somewhat different methodologies to calculate effect sizes. Some debate has occurred regarding the specific procedures used in each of these meta-analyses; however, it is reasonable to suggest that, as long as sensible alternatives exist in methodology that deliver divergent results, meta-analyses of existing studies alone may be insufficient for reaching consensus about the efficacy of cognitive training.
In this chapter we suggest an alternate solution to the issues facing cognitive training research. There is compelling evidence that cognitive training is not equally effective for all participants across all studies. Rather, it is likely that certain individual difference factors such as age, baseline performance, socioeconomic status, personality, experience with games, and motivation, among many others, may impact the outcome of the intervention for any individual participant. These differences have significant implications not only for our ability to improve our theoretical understanding of cognitive training but also for the real-world efficacy of any individual intervention. However, many extant cognitive training studies do not examine these factors. Furthermore, most use sample sizes that are too small to adequately account for them individually, let alone the extent they may interact with each other. Conducting larger, better-powered studies that allow scientists to understand the effects of these differences may help to explain the inconsistency across studies thus far. This chapter focuses on the evidence that certain individual difference factors may influence the outcome of cognitive training, with the hope that researchers may examine them more closely in future studies. Additionally, given the importance of transferability to untrained tasks in cognitive training research, the present chapter focuses primarily on the contribution of these individual difference factors to improvements on transfer tasks, although these factors are also discussed with respect to training gains when relevant.
Individual Difference Factors that May Influence Cognitive Training Outcomes
A full discussion of all the individual difference factors that might influence the outcome of cognitive training would require a book by itself. For a comprehensive review of how individual difference factors impact cognition more generally, the Handbook of Individual Differences in Cognition (Gruszka et al. 2012) provides a detailed discussion. For the purposes of this introduction to the topic, however, we focus on individual difference factors that have been examined in previous working memory and executive function training research. This list, perhaps unsurprisingly given the previous paragraphs, is fairly short—only a handful of the many cognitive training studies published thus far have explicitly examined individual difference factors. At present, these studies consider age, baseline performance, personality factors, and motivation. Readers well versed in the study of individual differences may notice other factors that may be meaningful predictors in other contexts—such as gender or cultural factors1—missing from this chapter. While these certainly merit further study, we exclude them here simply because extant research does not yet suggest that they make a significant contribution to the outcome of a cognitive training paradigm.
The potential contribution of baseline performance (either on the training task itself or on the set of cognitive tests used at pretest) to improvements on untrained assessments merits primacy in a discussion of individual differences. While many individual difference factors have not been specifically studied in the context of cognitive training, most of these have been examined in the context of baseline performance on a variety of cognitive abilities, such as working memory or executive function. If baseline performance impacts the outcome of a training intervention, it is reasonable to suggest that the other individual difference factors that influence baseline performance merit further investigation; the influence of baseline performance in a domain on its trainability has long been a focus of cognitive training research (Verhaeghen et al. 1992; 1992; Willis 1989; see also Snow 1991). Baseline performance on working memory tasks may influence the outcome of cognitive training in one of two directions. One possibility is that those who perform worse prior to the intervention have more room to improve following training and thus may experience greater gains. Alternatively, those with higher baseline performance may be better able to benefit from completing a cognitive training regimen—they may perform better at the training task over the course of the intervention and thus also experience greater improvements; these participants may also be more likely to complete the entire intervention and not drop out of a study (Jaeggi et al. 2014). One factor to keep in mind is that the source of individual differences in baseline performance may differ across studies: in some studies lower baseline individuals may have less experience, be younger or older, and so forth. Thus, it is not surprising that baseline performance may have different effects across studies. It is also possible that different training paradigms may result in different patterns of performance across high- and low-baseline participants. For example, process-based training often results in higher gains for individuals with lower baseline performance, while strategy-based training programs often result in greater gains for high-baseline individuals (Karbach and Unger 2014). Thus, a sensible approach to resolving these issues is to focus on the underlying individual differences that may influence baseline performance as well as training paradigm characteristics.
Of the few studies examining baseline performance and cognitive training, a number support the former possibility—that is, that those who start with lower levels of performance experience greater gains. In two studies by Zinke et al. (2012, 2014) individuals who performed worse at baseline, across multiple training paradigms of both working memory and executive control, experienced larger gains on the training task. Although Zinke et al. did not directly examine how baseline performance on untrained WM or fluid intelligence measures impacts transfer gains, they did find that, like Jaeggi et al. (2011), Schmiedek et al. (2010), and Chein and Morrison (2010), the amount of improvement on the training task does contribute to the amount of transfer gains on certain executive control and verbal WM tasks. These studies suggest that individuals who begin with lower baseline performance on the trained tasks may have stood to improve more at both the training and related transfer tasks. One possibility, as Zinke and colleagues discuss, is that individuals with higher baseline performance may be closer to ceiling performance at the task. If improvement in the task is a necessary precursor to transfer gains, it is also possible that modifying the task to permit high performers to continue improving beyond present ceiling levels might also permit them to experience greater transfer gain.
Few studies have specifically looked at how pretest performance on the transfer tasks may influence transfer gain, although consistent with the previously mentioned small studies, one recent, large-scale study found that individuals who performed worse at pretest on the set of transfer tasks also showed greater improvements on these tasks following training than those with higher pretest scores (Hardy et al. 2015). This finding is also consistent with research conducted on the ACTIVE training project with older adults (Willis and Caskie 2013) that found that lower performance on certain baseline measures was correlated with greater improvement after a period of cognitive training. While these studies provide some evidence that lower-performing individuals may stand to benefit more from the training than those who are closer to ceiling, the relationship between baseline ability and transfer may be fairly complex and might also be influenced by methodological differences, such as the design of the intervention or the adaptivity algorithms used to increase or decrease the difficulty of training. And finally, some of the outcome measures might not be sensitive enough to detect changes at the upper end of the scale, which could also contribute to the relatively smaller improvements observed in high-ability samples.
A substantial body of research provides evidence for the effects of age on cognitive plasticity across the lifespan (Guye et al. of this volume); it should be unsurprising that age has often been linked to differences in transfer improvements following cognitive training. Several studies have found that improvements on untrained tasks are smaller for older adults than younger adults (Zinke et al. 2014; Brehmer et al. 2012; Schmiedek et al. 2010) and even smaller for old-old adults, when compared to young-old individuals (Borella et al. 2014). However, meta-analytic work has revealed inconsistent findings on this issue: While one recent meta-analysis found no difference between younger and older adults in transfer improvements (Karbach and Verhaeghen 2014), another found that younger adults improved more on these tasks than older adults (Wass et al. 2012). Given that these meta-analyses include different sets of studies based on differing parameters (e.g., Wass et al. include a larger range of ages than Karbach and Verhaeghen), it is difficult to compare them to each other.
Given the extent to which age impacts baseline performance on a wide variety of cognitive tasks (Salthouse 1996), there is a reasonable impetus for examining the effects of age in cognitive training research. Furthermore, if training mitigates the effects of age-related cognitive decline, older adult populations may benefit the most from cognitive training (Richmond et al. 2011). Perhaps most problematic, from the perspective of critiquing extant research, is that age effects have generally been examined only in the context of older versus younger adults, or in children, with the exception of Borella et al. (2014) mentioned above. Little is known about how age may impact transfer in individuals who are older than college age but younger than retirement, and until a truly comprehensive study is made that includes the entire lifespan—from children to young adults to middle age to older adults—a significant gap remains in our understanding of age as an individual difference factor relevant to cognitive training research.
A new line of research on cognitive training has begun to examine how individual differences in personality and temperament may moderate training gains and transfer effects. The findings from these studies seem to suggest that emotional stability is an important factor moderating the efficacy of cognitive training.
Conscientiousness is one moderating factor that has been investigated. A highly conscientious person tends to be persistent, hardworking, self-disciplined, and competitive. Not surprisingly, individuals with high levels of conscientiousness tend to have high training scores and improvements in near transfer tasks (Studer-Luethi et al. 2012). A surprising finding, however, is that participants with high levels of conscientiousness tend to have lower levels of improvement in far transfer measures. This finding suggests that conscientious individuals are able to develop successful, though nontransferable, task-specific skills (Studer-Luethi et al. 2012). However, given that this was a fairly small study sample, the range of personality factors was somewhat restricted. Thus, the results are considered preliminary, and further investigation is needed.
Related to conscientiousness is a factor called effortful control, which describes individual differences in emotional stability and the ability to self-regulate one’s behavior depending on current and future goals. For example, children with high levels of effortful control are better able to overcome negative effect in order to achieve a goal. Consistent with the findings from (Studer-Luethi et al. 2012; Studer-Luethi et al. 2015), found that high effortful control and low neuroticism, a characteristic describing anxious and emotionally unstable individuals, best predicted transfer effects and suggest that cognitive training is most effective when children are emotionally stable and able to sufficiently self-regulate their emotions (Studer-Luethi et al. 2015).
This hypothesis is in line with the finding by Urbánek and Marček (2015) that individuals who scored high on the rhapsodic Personality Styles and Disorders Inventory, indicating an extravagantly emotional personality type, as well as schizotypal individuals who are typically highly anxious in social situations, were less likely to perform well on transfer tasks after training (Urbánek and Marček 2015). Similarly, neuroticism has been associated with lower training scores and lower performance on transfer tasks (Studer-Luethi et al. 2012). These personality types likely have a negative influence on training outcomes due to processing capacity being limited by disadvantageous arousal levels, intrusive thoughts, and negative emotions (Studer-Luethi et al. 2015).
Based on this work, emotional stability appears to be an important personality factor to take into account for cognitive training. Future work should consider the underlying mechanisms behind personality differences as well as other personality differences, such as openness to experience, which has not only been correlated with cognitive ability (Schaie et al. 2004) but has also shown to be changed as a function of cognitive training (Jackson et al. 2012).
Despite being an entire subfield of psychological research, few studies examine how a participant’s motivation, either to complete the intervention or to improve their cognitive capacity, impacts training and transfer. A number of previous training studies inform participants that they may improve their intelligence or cognitive function during the study (Jaeggi et al. 2008; Klingberg et al. 2005), while other studies only mention practicing computerized tasks (Redick et al. 2013). One study, from the authors of this chapter, suggests that personal beliefs about the malleability of intelligence may contribute to the amount of transfer after a cognitive training intervention (Jaeggi et al. 2014). Individuals who believed that intelligence could be improved experienced larger transfer gains following training. Although the beliefs-by-intervention interaction was not significant in this instance, it does suggest that personal beliefs about whether one is able to improve cognition—itself a major factor in how motivated one might be to complete cognitive training—could have a substantial impact on the outcome of training. Additionally, the use of payment or other forms of extrinsic motivators as a means of incentive in training studies may also undermine the outcome of the intervention. The sole meta-analysis that examined compensation levels in the context of transfer gain points toward a negative impact of renumeration on transfer improvements (Au et al. 2014). However, findings from a recent study conducted in our laboratory specifically designed to measure the impact of renumeration suggests that payment may not necessarily have an undermining influence when participants also have intrinsic motivation to complete the training (Katz et al. submitted for publication).
One curious point related to motivation is the inclusion of “game-like” elements in cognitive training paradigms that are meant to motivate or engage participants. A number of cognitive training programs have been designed to mimic the motivational elements of video games (Jaeggi et al. 2011; Klingberg et al. 2005), while others do not include these game-like features, such as scoring, feedback, or animations. There is some evidence that game elements may influence performance on the tasks involved (Katz et al. 2014), although this one study from our laboratory suggests that adding game-like features may undermine training and transfer if they distract a participant from the core task. Interestingly, this difference has appeared most sharply in training targeted toward children versus training targeted toward adults, as if there is an expectation by researchers that game-like features are more motivating for younger individuals than older ones. Clearly, more work is needed to better understand the role of motivation in cognitive training as well as what researchers can do to best motivate their participants.
It may be somewhat disconcerting that extant research cannot yet pinpoint all individual differences that play a significant role in the outcome of cognitive training. We suggest that, at present, it is enough simply to illustrate that many of these factors can play a role in the outcome of training. Whether or not each individual difference factor is meaningful, one thing can be concluded for certain: We cannot know the extent to which they matter if we do not measure them. Furthermore, there is likely to be multicollinearity of variables when multiple individual difference measures are assessed, and it may be difficult to identify the primary factors. Some factors, like age, are often examined in cognitive training studies, while others, such as motivation, personality, socioeconomic status, and psychopathology, are generally not considered in cognitive training research. Given that most cognitive training studies only include a relatively small number of participants, this is a significant oversight—in an underpowered study, each of these factors could have an oversized effect. For example, given that Au and colleagues’ (2014) meta-analysis suggests an effect of one particular motivational factor—renumeration—on transfer, it is possible that the other meta-analyses discussed here, none of which examine motivation or renumeration, may have reached different conclusions were they to examine these factors. But it is important to keep in mind that even the most carefully conducted meta-analysis is only as strong as the sum of the studies it includes. Future cognitive training studies should include these factors as variables of interest, and furthermore, they must be adequately powered to fully examine them.
Finally, if some of these factors have a significant impact on transfer improvements, they may need to be incorporated into training paradigms themselves. If older adults experience smaller training gains than younger adults and the amount of training improvement predicted transfer, would it make sense to offer them the same training paradigm? A physical trainer does not offer the same training regimen to two individuals with different levels of baseline fitness nor does a clinical psychologist offer identical courses of therapy to individuals facing different mental health issues—or even the same mental health issues; for example, aptitude-treatment interaction methods have been used to take individual differences into account in psychotherapy interventions with considerable success (Snow 1991). There is no “one-size-fits-all” in human intervention research. And yet this is precisely what is often done in cognitive training studies: Individuals who may have vastly different motivations, baseline performance, and educational backgrounds find themselves facing identical training paradigms that, even if they are adaptive, do not take into account these individual differences. Better measurement and larger sample sizes may help us understand how these factors contribute to the outcome of training, but personalized training programs, which build these factors into the training paradigm itself, may ultimately help researchers create cognitive training regimens that are more effective—for all participants.
- Gruszka, A., Matthews, G., & Szymura, B. (Eds.). (2012). Handbook of individual differences in cognition: Attention, memory, and executive control (pp. 295–320). New York: Springer. doi:10.1007/978-1-4419-1210-7.
- Jackson, J. J., Hill, P. L., Payne, B. R., Roberts, B. W., & Stine-Morrow, E. A. (2012). Can an old dog learn (and want to experience) new tricks? Cognitive training increases openness to experience in older adults. Psychology and Aging, 27(2), 286–292. doi:10.1037/a0025918.CrossRefPubMedPubMedCentralGoogle Scholar
- Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008). Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences of the United States of America, 105(19), 6829–6833. doi:10.1073/pnas.0801268105.CrossRefPubMedPubMedCentralGoogle Scholar
- Katz, B., Jaeggi, S., Buschkuehl, M., Shah, P., & Jonides, J. (submitted for publication). The effect of monetary compensation on cognitive training.Google Scholar
- Klingberg, T., Fernell, E., Olesen, P. J., Johnson, M., Gustafsson, P., Dahlström, K., et al. (2005). Computerized training of working memory in children with ADHD—A randomized, controlled trial. Journal of the American Academy of Child & Adolescent Psychiatry, 44(2), 177–186. doi:10.1097/00004583-200502000-00010.CrossRefGoogle Scholar
- Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., et al. (2013). No evidence of intelligence improvement after working memory training: A randomized, placebo-controlled study. Journal of Experimental Psychology: General, 142(2), 359–379. doi:10.1037/a0029082.CrossRefGoogle Scholar
- Urbánek, T., & Marček, V. (2015). Investigating the effectiveness of working memory training in the context of Personality Systems Interaction theory. Psychological Research, 1–12. doi:10.1007/s00426-015-0687-4.
- Willis, S. L. (1989). Improvement with cognitive training: Which old dogs learn what tricks? In Poon, L. W., Rubin, D. C., & Wilson, B. A. (Eds.), Everyday cognition in adulthood and late life (pp. 545–569). New York, NY: Cambridge University Press; xii, 708 pp. doi:10.1017/CBO9780511759390.
- Verhaeghen, P., Marcoen, A., & Goossens, L. (1992). Improving memory performance in the aged through mnemonic training: a meta-analytic study. Psychology and aging, 7(2), 242.doi:10.1037/0882-79188.8.131.52.
- Wass, S. V., Scerif, G., & Johnson, M. H. (2012). Training attentional control and working memory–Is younger, better?. Developmental Review, 32(4), 360-387.doi:10.1016/j.dr.2012.07.001.