Skip to main content

What counts as evidence for working memory training? Problems with correlated gains and dichotomization

Abstract

The question of whether computerized cognitive training leads to generalized improvements of intellectual abilities has been a popular, yet contentious, topic within both the psychological and neurocognitive literatures. Evidence for the effective transfer of cognitive training to nontrained measures of cognitive abilities is mixed, with some studies showing apparent successful transfer, while others have failed to obtain this effect. At the same time, several authors have made claims about both successful and unsuccessful transfer effects on the basis of a form of responder analysis, an analysis technique that shows that those who gain the most on training show the greatest gains on transfer tasks. Through a series of Monte Carlo experiments and mathematical analyses, we demonstrate that the apparent transfer effects observed through responder analysis are illusory and are independent of the effectiveness of cognitive training. We argue that responder analysis can be used neither to support nor to refute hypotheses related to whether cognitive training is a useful intervention to obtain generalized cognitive benefits. We end by discussing several proposed alternative analysis techniques that incorporate training gain scores and argue that none of these methods are appropriate for testing hypotheses regarding the effectiveness of cognitive training.

This is a preview of subscription content, access via your institution.

Fig. 1

Notes

  1. 1.

    Google Scholar citations as of 25 October, 2013.

References

  1. Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8, 445–454.

    Article  Google Scholar 

  2. Chein, J. M., & Morrison, A. B. (2010). Expanding the mind’s workspace: training and transfer effects with a complex working memory span task. Psychonomic Bulletin & Review, 17, 193–199. doi:10.3758/PBR.17.2.193

    Article  Google Scholar 

  3. Dimitrov, D. M., & Rumrill, P. D. (2003). Pretest-posttest designs and measurement of change. Work, 20, 159–165.

    PubMed  Google Scholar 

  4. Dowle, M., Short, T., & Lianoglou, S. (2013). data.table: Extension of data.frame for fast indexing, fast ordered joins, fast assignment, fast grouping and list columns (R package version 1.8.8) [Computer software]. Retrieved from http://cran.r-project.org/web/packages/data.table/index.html

  5. Engle, R. (2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11, 19–23. doi:10.1111/1467-8721.00160

    Article  Google Scholar 

  6. Fisher, R. A. (1915). Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika, 10, 507–521.

    Google Scholar 

  7. Gallistel, C. R. (2009). The importance of proving the null. Psychological Review, 116, 439–453. doi:10.1037/a0015251

    PubMed Central  PubMed  Article  Google Scholar 

  8. Gibson, B. S., Gondoli, D. M., Johnson, A. C., & Robison, M. K. (2013). Recall initiation strategies must be controlled in training studies that use immediate free recall tasks to measure the components of working memory capacity across time. Child Neuropsychology. doi:10.1080/09297049.2013.826185

    PubMed  Google Scholar 

  9. Huck, S. W., & McLean, R. A. (1975). Using a repeated measures ANOVA to analyze the data from a pretest–posttest design: A potentially confusing task. Psychological Bulletin, 82, 511–518. doi:10.1037/h0076767

    Article  Google Scholar 

  10. Hurley, D. (2012, Nov 4). The brain trainers. New York Times, p. ED18

  11. Hussey, E. K., & Novick, J. M. (2012). The benefits of executive control training and the implications for language processing. Frontiers in Psychology, 3, 158. doi:10.3389/fpsyg.2012.00158

    PubMed Central  PubMed  Article  Google Scholar 

  12. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008). Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences, 105, 6829–6833. doi:10.1073/pnas.0801268105

    Article  Google Scholar 

  13. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Shah, P. (2011). Short- and long-term benefits of cognitive training. Proceedings of the National Academy of Sciences, 108, 10081–10086. doi:10.1073/pnas.1103228108

    Article  Google Scholar 

  14. Klingberg, T. (2010). Training and plasticity of working memory. Trends in Cognitive Sciences, 14, 317–24. doi:10.1016/j.tics.2010.05.002

    PubMed  Article  Google Scholar 

  15. Kundu, B., Sutterer, D. W., Emrich, S. M., & Postle, B. R. (2013). Strengthened effective connectivity underlies transfer of working memory training to tests of short-term memory and attention. Journal of Neuroscience, 33, 8705–8715. doi:10.1523/JNEUROSCI.5565-12.2013

    PubMed Central  PubMed  Article  Google Scholar 

  16. MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40. doi:10.1037/1082-989X.7.1.19

    PubMed  Article  Google Scholar 

  17. McClelland, G., & Irwin, J. (2003). Negative consequences of dichotomizing continuous predictor variables. Journal of Marketing Research, XL, 366–371.

    Google Scholar 

  18. McNab, F., Varrone, A., Farde, L., Jucaite, A., Bystritsky, P., Forssberg, H., & Klingberg, T. (2009). Changes in cortical dopamine D1 receptor binding associated with cognitive training. Science, 323, 800–802. doi:10.1126/science.1166102

    PubMed  Article  Google Scholar 

  19. Melby-Lervåg, M., & Hulme, C. (2013). Is working memory training effective? A meta-analytic review. Developmental Psychology, 49, 270–291. doi:10.1037/a0028228

    PubMed  Article  Google Scholar 

  20. Miaskowski, C., Dodd, M., West, C., Paul, S. M., Schumacher, K., Tripathy, D., & Koo, P. (2007). The use of a responder analysis to identify differences in patient outcomes following a self-care intervention to improve cancer pain management. Pain, 129, 55–63. doi:10.1016/j.pain.2006.09.031

    PubMed Central  PubMed  Article  Google Scholar 

  21. Morrison, A. B., & Chein, J. M. (2011). Does working memory training work? The promise and challenges of enhancing cognition by training working memory. Psychonomic Bulletin & Review, 18, 46–60. doi:10.3758/s13423-010-0034-0

    Article  Google Scholar 

  22. Novick, J. M., Hussey, E., Teubner-Rhodes, S., Harbison, J. I., & Bunting, M. F. (2013). Clearing the garden-path: Improving sentence processing through cognitive control training. Language and Cognitive Processes, 1–44. doi:10.1080/01690965.2012.758297

  23. Owen, A. M., Hampshire, A., Grahn, J. A., Stenton, R., Dajani, S., Burns, A. S., Howard, R.J., & Ballard, C.G. (2010). Putting brain training to the test. Nature, 465, 775–8. doi:10.1038/nature09042

    PubMed Central  PubMed  Article  Google Scholar 

  24. Protzko, J., Aronson, J., & Blair, C. (2013). How to make a young child smarter: Evidence from the database of raising intelligence. Perspectives on Psychological Science, 8, 25–40. doi:10.1177/1745691612462585

    Article  Google Scholar 

  25. R Development Core Team. (2013). R: A language and environment for statistical computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from www.R-project.org

  26. Raven, J. (2000). The Raven’s Progressive Matrices: Change and stability over culture and time. Cognitive Psychology, 41, 1–48. doi:10.1006/cogp.1999.0735

    PubMed  Article  Google Scholar 

  27. Reddy, S. (2013, May 14). When computer games may keep the brain nimble. Wall Street Journal, p. D1.

  28. Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., Kane, M. J., & Engle, R. W. (2013). No evidence of intelligence improvement after working memory training: A randomized, placebo-controlled study. Journal of Experimental Psychology. General, 142, 359–79. doi:10.1037/a0029082

    PubMed  Article  Google Scholar 

  29. Revolution Analytics. (2013). doMC: Foreach parallel adaptor for the multicore package (R package version 1.3.0) [Computer Software]. Palo Alto, CA. Retrieved fromhttp://cran.r-project.org/web/packages/doMC/index.html

  30. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237. doi:10.3758/PBR.16.2.225

    Article  Google Scholar 

  31. Rudebeck, S. R., Bor, D., Ormond, A., O’Reilly, J. X., & Lee, A. C. H. (2012). A potential spatial working memory training task to improve both episodic memory and fluid intelligence. PLoS ONE, 7, e50431. doi:10.1371/journal.pone.0050431

    PubMed Central  PubMed  Article  Google Scholar 

  32. Schmiedek, F., Lövdén, M., & Lindenberger, U. (2010). Hundred days of cognitive training enhance broad cognitive abilities in adulthood: Findings from the COGITO study. Frontiers in Aging Neuroscience, 2(27), 1–10. doi:10.3389/fnagi.2010.00027

    Google Scholar 

  33. Schweizer, S., Grahn, J., Hampshire, A., Mobbs, D., & Dalgleish, T. (2013). Training the emotional brain: Improving affective control through emotional working memory training. Journal of Neuroscience, 33, 5301–5311. doi:10.1523/JNEUROSCI.2593-12.2013

    PubMed  Article  Google Scholar 

  34. Senn, S., & Julious, S. (2009). Measurement in clinical trials: A neglected issue for statisticians? Statistics in Medicine, 28, 3189–3209. doi:10.1002/sim.3603

    PubMed  Article  Google Scholar 

  35. Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training effective? Psychological Bulletin, 138, 628–654. doi:10.1037/a0027473

    PubMed  Article  Google Scholar 

  36. Sprenger, A. M., Atkins, S. M., Bolger, D. J., Harbison, J. I., Novick, J. M., Weems, S. A., Chrabaszcz, J. S., Smith, V., Bobb, S., Bunting, M. F., & Dougherty, M. R. (2013). Training working memory: Limits of transfer. Intelligence, 41, 638–663. doi:10.1016/j.intell.2013.07.013

    Article  Google Scholar 

  37. Sukel, K. (2013, Dec 9). Digital “brain health” market predicted to boom. Retrieved from http://bigthink.com/world-in-mind/digital-brain-health-market-predicted-to-boom

  38. Thompson, T. W., Waskom, M. L., Garel, K.L. A., Cardenas-Iniguez, C., Reynolds, G. O., Winter, R., Chang, P., Pollard, K., Lala, N., Alvarez, G.A., & Gabrieli, J. (2013). Failure of working memory training to enhance cognition or intelligence. PLoS ONE, 8, e63614. doi:10.1371/journal.pone.0063614

    PubMed Central  PubMed  Article  Google Scholar 

  39. Venables, W. N., & Ripley, B. D. (2002). Modern applied statistics with S (4th ed.). New York, NY: Springer.

    Book  Google Scholar 

  40. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291–298. doi:10.1177/1745691611406923

    Article  Google Scholar 

  41. Wickham, H. (2007). Reshaping data with the reshape package. Journal of Statistical Software, 21(12), 1–20.

    Google Scholar 

  42. Zinke, K., Zeintl, M., Rose, N. S., Putzmann, J., Pydde, A., & Kliegel, M. (2013). Working memory training and transfer in older adults: Effects of age, baseline performance, and training gains. Developmental Psychology. doi:10.1037/a0032982

    PubMed  Google Scholar 

Download references

Author Note

This work was supported by the Office of Naval Research, Grant No. N000141010605, awarded to M.R.D. The authors thank Thomas Wallsten for comments on a prior version of the manuscript.

Author information

Affiliations

Authors

Corresponding authors

Correspondence to Joe W. Tidwell or Michael R. Dougherty.

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1

(PDF 36 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Tidwell, J.W., Dougherty, M.R., Chrabaszcz, J.R. et al. What counts as evidence for working memory training? Problems with correlated gains and dichotomization. Psychon Bull Rev 21, 620–628 (2014). https://doi.org/10.3758/s13423-013-0560-7

Download citation

Keywords

  • Cognitive training
  • Human memory and learning
  • Individual differences
  • Memory capacity
  • Statistical inference