Psychonomic Bulletin & Review

, Volume 21, Issue 3, pp 620–628 | Cite as

What counts as evidence for working memory training? Problems with correlated gains and dichotomization

  • Joe W. Tidwell
  • Michael R. Dougherty
  • Jeffrey R. Chrabaszcz
  • Rick P. Thomas
  • Jorge L. Mendoza
Theoretical Review


The question of whether computerized cognitive training leads to generalized improvements of intellectual abilities has been a popular, yet contentious, topic within both the psychological and neurocognitive literatures. Evidence for the effective transfer of cognitive training to nontrained measures of cognitive abilities is mixed, with some studies showing apparent successful transfer, while others have failed to obtain this effect. At the same time, several authors have made claims about both successful and unsuccessful transfer effects on the basis of a form of responder analysis, an analysis technique that shows that those who gain the most on training show the greatest gains on transfer tasks. Through a series of Monte Carlo experiments and mathematical analyses, we demonstrate that the apparent transfer effects observed through responder analysis are illusory and are independent of the effectiveness of cognitive training. We argue that responder analysis can be used neither to support nor to refute hypotheses related to whether cognitive training is a useful intervention to obtain generalized cognitive benefits. We end by discussing several proposed alternative analysis techniques that incorporate training gain scores and argue that none of these methods are appropriate for testing hypotheses regarding the effectiveness of cognitive training.


Cognitive training Human memory and learning Individual differences Memory capacity Statistical inference 


Author Note

This work was supported by the Office of Naval Research, Grant No. N000141010605, awarded to M.R.D. The authors thank Thomas Wallsten for comments on a prior version of the manuscript.

Supplementary material

13423_2013_560_MOESM1_ESM.pdf (37 kb)
ESM 1 (PDF 36 kb)


  1. Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8, 445–454.CrossRefGoogle Scholar
  2. Chein, J. M., & Morrison, A. B. (2010). Expanding the mind’s workspace: training and transfer effects with a complex working memory span task. Psychonomic Bulletin & Review, 17, 193–199. doi: 10.3758/PBR.17.2.193 CrossRefGoogle Scholar
  3. Dimitrov, D. M., & Rumrill, P. D. (2003). Pretest-posttest designs and measurement of change. Work, 20, 159–165.PubMedGoogle Scholar
  4. Dowle, M., Short, T., & Lianoglou, S. (2013). data.table: Extension of data.frame for fast indexing, fast ordered joins, fast assignment, fast grouping and list columns (R package version 1.8.8) [Computer software]. Retrieved from
  5. Engle, R. (2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11, 19–23. doi: 10.1111/1467-8721.00160 CrossRefGoogle Scholar
  6. Fisher, R. A. (1915). Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika, 10, 507–521.Google Scholar
  7. Gallistel, C. R. (2009). The importance of proving the null. Psychological Review, 116, 439–453. doi: 10.1037/a0015251 PubMedCentralPubMedCrossRefGoogle Scholar
  8. Gibson, B. S., Gondoli, D. M., Johnson, A. C., & Robison, M. K. (2013). Recall initiation strategies must be controlled in training studies that use immediate free recall tasks to measure the components of working memory capacity across time. Child Neuropsychology. doi: 10.1080/09297049.2013.826185 PubMedGoogle Scholar
  9. Huck, S. W., & McLean, R. A. (1975). Using a repeated measures ANOVA to analyze the data from a pretest–posttest design: A potentially confusing task. Psychological Bulletin, 82, 511–518. doi: 10.1037/h0076767 CrossRefGoogle Scholar
  10. Hurley, D. (2012, Nov 4). The brain trainers. New York Times, p. ED18Google Scholar
  11. Hussey, E. K., & Novick, J. M. (2012). The benefits of executive control training and the implications for language processing. Frontiers in Psychology, 3, 158. doi: 10.3389/fpsyg.2012.00158 PubMedCentralPubMedCrossRefGoogle Scholar
  12. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008). Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences, 105, 6829–6833. doi: 10.1073/pnas.0801268105 CrossRefGoogle Scholar
  13. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Shah, P. (2011). Short- and long-term benefits of cognitive training. Proceedings of the National Academy of Sciences, 108, 10081–10086. doi: 10.1073/pnas.1103228108 CrossRefGoogle Scholar
  14. Klingberg, T. (2010). Training and plasticity of working memory. Trends in Cognitive Sciences, 14, 317–24. doi: 10.1016/j.tics.2010.05.002 PubMedCrossRefGoogle Scholar
  15. Kundu, B., Sutterer, D. W., Emrich, S. M., & Postle, B. R. (2013). Strengthened effective connectivity underlies transfer of working memory training to tests of short-term memory and attention. Journal of Neuroscience, 33, 8705–8715. doi: 10.1523/JNEUROSCI.5565-12.2013 PubMedCentralPubMedCrossRefGoogle Scholar
  16. MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40. doi: 10.1037/1082-989X.7.1.19 PubMedCrossRefGoogle Scholar
  17. McClelland, G., & Irwin, J. (2003). Negative consequences of dichotomizing continuous predictor variables. Journal of Marketing Research, XL, 366–371.Google Scholar
  18. McNab, F., Varrone, A., Farde, L., Jucaite, A., Bystritsky, P., Forssberg, H., & Klingberg, T. (2009). Changes in cortical dopamine D1 receptor binding associated with cognitive training. Science, 323, 800–802. doi: 10.1126/science.1166102 PubMedCrossRefGoogle Scholar
  19. Melby-Lervåg, M., & Hulme, C. (2013). Is working memory training effective? A meta-analytic review. Developmental Psychology, 49, 270–291. doi: 10.1037/a0028228 PubMedCrossRefGoogle Scholar
  20. Miaskowski, C., Dodd, M., West, C., Paul, S. M., Schumacher, K., Tripathy, D., & Koo, P. (2007). The use of a responder analysis to identify differences in patient outcomes following a self-care intervention to improve cancer pain management. Pain, 129, 55–63. doi: 10.1016/j.pain.2006.09.031 PubMedCentralPubMedCrossRefGoogle Scholar
  21. Morrison, A. B., & Chein, J. M. (2011). Does working memory training work? The promise and challenges of enhancing cognition by training working memory. Psychonomic Bulletin & Review, 18, 46–60. doi: 10.3758/s13423-010-0034-0 CrossRefGoogle Scholar
  22. Novick, J. M., Hussey, E., Teubner-Rhodes, S., Harbison, J. I., & Bunting, M. F. (2013). Clearing the garden-path: Improving sentence processing through cognitive control training. Language and Cognitive Processes, 1–44. doi: 10.1080/01690965.2012.758297
  23. Owen, A. M., Hampshire, A., Grahn, J. A., Stenton, R., Dajani, S., Burns, A. S., Howard, R.J., & Ballard, C.G. (2010). Putting brain training to the test. Nature, 465, 775–8. doi: 10.1038/nature09042 PubMedCentralPubMedCrossRefGoogle Scholar
  24. Protzko, J., Aronson, J., & Blair, C. (2013). How to make a young child smarter: Evidence from the database of raising intelligence. Perspectives on Psychological Science, 8, 25–40. doi: 10.1177/1745691612462585 CrossRefGoogle Scholar
  25. R Development Core Team. (2013). R: A language and environment for statistical computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from
  26. Raven, J. (2000). The Raven’s Progressive Matrices: Change and stability over culture and time. Cognitive Psychology, 41, 1–48. doi: 10.1006/cogp.1999.0735 PubMedCrossRefGoogle Scholar
  27. Reddy, S. (2013, May 14). When computer games may keep the brain nimble. Wall Street Journal, p. D1.Google Scholar
  28. Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., Kane, M. J., & Engle, R. W. (2013). No evidence of intelligence improvement after working memory training: A randomized, placebo-controlled study. Journal of Experimental Psychology. General, 142, 359–79. doi: 10.1037/a0029082 PubMedCrossRefGoogle Scholar
  29. Revolution Analytics. (2013). doMC: Foreach parallel adaptor for the multicore package (R package version 1.3.0) [Computer Software]. Palo Alto, CA. Retrieved from
  30. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237. doi: 10.3758/PBR.16.2.225 CrossRefGoogle Scholar
  31. Rudebeck, S. R., Bor, D., Ormond, A., O’Reilly, J. X., & Lee, A. C. H. (2012). A potential spatial working memory training task to improve both episodic memory and fluid intelligence. PLoS ONE, 7, e50431. doi: 10.1371/journal.pone.0050431 PubMedCentralPubMedCrossRefGoogle Scholar
  32. Schmiedek, F., Lövdén, M., & Lindenberger, U. (2010). Hundred days of cognitive training enhance broad cognitive abilities in adulthood: Findings from the COGITO study. Frontiers in Aging Neuroscience, 2(27), 1–10. doi: 10.3389/fnagi.2010.00027 Google Scholar
  33. Schweizer, S., Grahn, J., Hampshire, A., Mobbs, D., & Dalgleish, T. (2013). Training the emotional brain: Improving affective control through emotional working memory training. Journal of Neuroscience, 33, 5301–5311. doi: 10.1523/JNEUROSCI.2593-12.2013 PubMedCrossRefGoogle Scholar
  34. Senn, S., & Julious, S. (2009). Measurement in clinical trials: A neglected issue for statisticians? Statistics in Medicine, 28, 3189–3209. doi: 10.1002/sim.3603 PubMedCrossRefGoogle Scholar
  35. Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training effective? Psychological Bulletin, 138, 628–654. doi: 10.1037/a0027473 PubMedCrossRefGoogle Scholar
  36. Sprenger, A. M., Atkins, S. M., Bolger, D. J., Harbison, J. I., Novick, J. M., Weems, S. A., Chrabaszcz, J. S., Smith, V., Bobb, S., Bunting, M. F., & Dougherty, M. R. (2013). Training working memory: Limits of transfer. Intelligence, 41, 638–663. doi: 10.1016/j.intell.2013.07.013 CrossRefGoogle Scholar
  37. Sukel, K. (2013, Dec 9). Digital “brain health” market predicted to boom. Retrieved from
  38. Thompson, T. W., Waskom, M. L., Garel, K.L. A., Cardenas-Iniguez, C., Reynolds, G. O., Winter, R., Chang, P., Pollard, K., Lala, N., Alvarez, G.A., & Gabrieli, J. (2013). Failure of working memory training to enhance cognition or intelligence. PLoS ONE, 8, e63614. doi: 10.1371/journal.pone.0063614 PubMedCentralPubMedCrossRefGoogle Scholar
  39. Venables, W. N., & Ripley, B. D. (2002). Modern applied statistics with S (4th ed.). New York, NY: Springer.CrossRefGoogle Scholar
  40. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291–298. doi: 10.1177/1745691611406923 CrossRefGoogle Scholar
  41. Wickham, H. (2007). Reshaping data with the reshape package. Journal of Statistical Software, 21(12), 1–20.Google Scholar
  42. Zinke, K., Zeintl, M., Rose, N. S., Putzmann, J., Pydde, A., & Kliegel, M. (2013). Working memory training and transfer in older adults: Effects of age, baseline performance, and training gains. Developmental Psychology. doi: 10.1037/a0032982 PubMedGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2013

Authors and Affiliations

  • Joe W. Tidwell
    • 1
    • 3
  • Michael R. Dougherty
    • 1
    • 3
  • Jeffrey R. Chrabaszcz
    • 1
  • Rick P. Thomas
    • 2
  • Jorge L. Mendoza
    • 2
  1. 1.University of MarylandCollege ParkUSA
  2. 2.University of OklahomaNormanUSA
  3. 3.Department of PsychologyUniversity of MarylandCollege ParkUSA

Personalised recommendations