Improving Methodological Standards in Behavioral Interventions for Cognitive Enhancement

Abstract

There is substantial interest in the possibility that cognitive skills can be improved by dedicated behavioral training. Yet despite the large amount of work being conducted in this domain, there is not an explicit and widely agreed upon consensus around the best methodological practices. This document seeks to fill this gap. We start from the perspective that there are many types of studies that are important in this domain—e.g., feasibility, mechanistic, efficacy, and effectiveness. These studies have fundamentally different goals, and, as such, the best-practice methods to meet those goals will also differ. We thus make suggestions in topics ranging from the design and implementation of control groups, to reporting of results, to dissemination and communication, taking the perspective that the best practices are not necessarily uniform across all study types. We also explicitly recognize and discuss the fact that there are methodological issues around which we currently lack the theoretical and/or empirical foundation to determine best practices (e.g., as pertains to assessing participant expectations). For these, we suggest important routes forward, including greater interdisciplinary collaboration with individuals from domains that face related concerns. Our hope is that these recommendations will greatly increase the rate at which science in this domain advances.

This is a preview of subscription content, log in to check access.

References

  1. Acosta, A., Adams, R.B., Jr., Albohn, D.N., Allard, E.S., Beek, T., Benning, S. D., … Zwaan, R. A. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917–928. doi:https://doi.org/10.1177/1745691616674458.

  2. Andrews, G. (1999). Efficacy, effectiveness and efficiency in mental health service delivery. The Australian and New Zealand Journal of Psychiatry, 33(3), 316–322. https://doi.org/10.1046/j.1440-1614.1999.00581.x.

    Article  PubMed  Google Scholar 

  3. Anguera, J.A., Boccanfuso, J., Rintoul, J.L., Al-Hashimi, O., Faraji, F., Janowich, J., … Gazzaley, A. (2013). Video game training enhances cognitive control in older adults. Nature, 501(7465), 97–101. doi:https://doi.org/10.1038/nature12486.

  4. Au, J., Sheehan, E., Tsai, N., Duncan, G. J., Buschkuehl, M., & Jaeggi, S. M. (2015). Improving fluid intelligence with training on working memory: a meta-analysis. Psychonomic Bulletin & Review, 22(2), 366–377. https://doi.org/10.3758/s13423-014-0699-x.

    Article  Google Scholar 

  5. Ball, K., Berch, D. B., Helmers, K. F., Jobe, J. B., Leveck, M. D., Marsiske, M., … Group, A. S. (2002). Effects of cognitive training interventions with older adults: a randomized controlled trial. JAMA, 288(18), 2271–2281.

  6. Baniqued, P., Allen, C.M., Kranz, M.B., Johnson, K., Sipolins, A., Dickens, C., …, Kramer, A.F. (2015). Working memory, reasoning, and task switching training: transfer effects, limitations, and great expectations?. PLoS One. doi:https://doi.org/10.1371/journal.pone.0142169.

  7. Barnett, S. M., & Ceci, S. J. (2002). When and where do we apply what we learn?: A taxonomy for far transfer. Psychological Bulletin, 128(4), 612–637.

    Article  Google Scholar 

  8. Barry, A. E., Szucs, L. E., Reyes, J. V., Ji, Q., Wilson, K. L., & Thompson, B. (2016). Failure to report effect sizes: the handling of quantitative results in published health education and behavior research. Health Education & Behavior, 43(5), 518–527. https://doi.org/10.1177/1090198116669521.

    Article  Google Scholar 

  9. Basak, C., Boot, W. R., Voss, M. W., & Kramer, A. F. (2008). Can training in a real-time strategy video game attenuate cognitive decline in older adults. Psychology and Aging, 23(4), 765–777.

    Article  Google Scholar 

  10. Bavelier, D., & Davidson, R. J. (2013). Brain training: games to do you good. Nature, 494(7438), 425–426. https://doi.org/10.1038/494425a.

    Article  PubMed  Google Scholar 

  11. Bayraktar, S. (2001). A meta-analysis of the effectiveness of computer-assisted instruction in science education. Journal of Research on Technology in Education, 34(2), 173–188.

    Article  Google Scholar 

  12. Beaumont, J. L., Havlik, R., Cook, K. F., Hays, R. D., Wallner-Allen, K., Korper, S. P., …, Gershon, R. (2013). Norming plans for the NIH toolbox. Neurology, 80(11 Suppl 3), S87–92. doi:https://doi.org/10.1212/WNL.0b013e3182872e70.

  13. Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018). Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills. Psychological Bulletin, 144(1), 77–110. https://doi.org/10.1037/bul0000130.

    Article  PubMed  Google Scholar 

  14. Biagianti, B., & Vinogradov, S. (2013). Computerized cognitive training targeting brain plasticity in schizophrenia. Progress in Brain Research, 207, 301–326. https://doi.org/10.1016/B978-0-444-63327-9.00011-4.

    Article  PubMed  Google Scholar 

  15. Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology: why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8(4), 445–454. https://doi.org/10.1177/1745691613491271.

    Article  PubMed  Google Scholar 

  16. Bryant, D. C. N., & Deluca, J. (2004). Objective measurement of cognitive fatigue in multiple sclerosis. Rehabilitation Psychology, 49(2), 114–122.

    Article  Google Scholar 

  17. Carvalho, C., Caetano, J. M., Cunha, L., Rebouta, P., Kaptchuk, T. J., & Kirsch, I. (2016). Open-label placebo treatment in chronic low back pain: a randomized controlled trial. Pain, 157(12), 2766–2772. https://doi.org/10.1097/j.pain.0000000000000700.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Chen, L. H., & Lee, W. C. (2011). Two-way minimization: a novel treatment allocation method for small trials. PLoS One, 6(12), e28604.

    Article  Google Scholar 

  19. Coburn, K. M., & Vevea, J. L. (2015). Publication bias as a function of study characteristics. Psychological Methods, 20(3), 310–330. https://doi.org/10.1037/met0000046.

    Article  PubMed  Google Scholar 

  20. Colloca, L., & Benedetti, F. (2006). How prior experience shapes placebo analgesia. Pain, 124(1–2), 126–133. https://doi.org/10.1016/j.pain.2006.04.005.

    Article  PubMed  Google Scholar 

  21. Colloca, L., Klinger, R., Flor, H., & Bingel, U. (2013). Placebo analgesia: psychological and neurobiological mechanisms. Pain, 154(4), 511–514. https://doi.org/10.1016/j.pain.2013.02.002.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Colzato, L. S., van den Wildenberg, W. P., & Hommel, B. (2014). Cognitive control and the COMT Val(1)(5)(8)Met polymorphism: genetic modulation of videogame training and transfer to task-switching efficiency. Psychological Research, 78(5), 670–678. https://doi.org/10.1007/s00426-013-0514-8.

    Article  PubMed  Google Scholar 

  23. Davidson, R. J., & Dahl, C. J. (2017). Varieties of contemplative practice. JAMA Psychiatry, 74(2), 121–123. https://doi.org/10.1001/jamapsychiatry.2016.3469.

    Article  PubMed  Google Scholar 

  24. Davidson, R. J., & Dahl, C. J. (2018). Outstanding challenges in scientific research on mindfulness and meditation. Perspectives on Psychological Science, 13(1), 62–65. https://doi.org/10.1177/1745691617718358.

    Article  PubMed  Google Scholar 

  25. Deveau, J., Jaeggi, S. M., Zordan, V., Phung, C., & Seitz, A. R. (2014a). How to build better memory training games. Frontiers in Systems Neuroscience, 8, 243. https://doi.org/10.3389/fnsys.2014.00243.

    Article  PubMed  Google Scholar 

  26. Deveau, J., Ozer, D. J., & Seitz, A. R. (2014b). Improved vision and on-field performance in baseball through perceptual learning. Current Biology, 24(4), R146–R147. https://doi.org/10.1016/j.cub.2014.01.004.

    Article  PubMed  Google Scholar 

  27. Diao, D., Wright, J. M., Cundiff, D. K., & Gueyffier, F. (2012). Pharmacotherapy for mild hypertension. Cochrane Database of Systematic Reviews(8). doi:https://doi.org/10.1002/14651858.CD006742.pub2.

  28. Dweck, C. (2006). Mindset: the new psychology of success. New York: Random House.

    Google Scholar 

  29. Eldridge, S. M., Lancaster, G. A., Campbell, M. J., Thabane, L., Hopewell, S., Coleman, C. L., & Bond, C. M. (2016). Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One, 11(3), e0150205. https://doi.org/10.1371/journal.pone.0150205.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Engle, R. W., Tuholski, S. W., Laughlin, J. E., & Conway, A. R. A. (1999). Working memory, short-term memory, and general fluid intelligence: a latent variable approach. Journal of Experimental Psychology: General, 128, 309–331.

    Article  Google Scholar 

  31. Erickson, K. I., Voss, M. W., Prakash, R. S., Basak, C., Szabo, A., Chaddock, L., et al. (2011). Exercise training increases size of hippocampus and improves memory. Proceedings of the National Academy of Sciences of the United States of America, 108(7), 3017–3022. https://doi.org/10.1073/pnas.1015950108.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Fassler, M., Meissner, K., Kleijnen, J., Hrobjartsson, A., & Linde, K. (2015). A systematic review found no consistent difference in effect between more and less intensive placebo interventions. Journal of Clinical Epidemiology, 68(4), 442–451. https://doi.org/10.1016/j.jclinepi.2014.11.018.

    Article  PubMed  Google Scholar 

  33. FEDERAL TRADE COMMISSION, (2016) Plaintiff, V.. Lumos Labs, Inc., a corporation d/b/a Lumosity and Kunal Sarkar, and Michael Scanlon, Individually and as officers of LumoS Labs, Inc. Defendants. , No. Case No. 3:16-cv-00001-sk (United StateS DistricT Court for the Northern District of California, San Francisco Division.

  34. Fergusson, D., Glass, K. C., Waring, D., & Shapiro, S. (2004). Turning a blind eye: the success of blinding reported in a random sample of randomised placebo controlled trials. BMJ, 328(7437), 432. https://doi.org/10.1136/bmj.37952.631667.EE.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Foroughi, C. K., Monfort, S. S., Paczynski, M., McKnight, P. E., & Greenwood, P. M. (2016). Placebo effects in cognitive training. PNAS, 113, 7470–7474. https://doi.org/10.1073/pnas.1601243113.

    Article  PubMed  Google Scholar 

  36. Franceschini, S., Gori, S., Ruffino, M., Viola, S., Molteni, M., & Facoetti, A. (2013). Action video games make dyslexic children read better. Current Biology, 23(6), 462–466. https://doi.org/10.1016/j.cub.2013.01.044.

    Article  PubMed  Google Scholar 

  37. Fritz, J. M., & Cleland, J. (2003). Effectiveness versus efficacy: more than a debate over language. The Journal of Orthopaedic and Sports Physical Therapy, 33(4), 163–165. https://doi.org/10.2519/jospt.2003.33.4.163.

    Article  PubMed  Google Scholar 

  38. Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention. Nature, 423(6939), 534–537.

    Article  Google Scholar 

  39. Green, C. S., & Bavelier, D. (2012). Learning, attentional control and action video games. Current Biology, 22, R197–R206.

    Article  Google Scholar 

  40. Green, C. S., Pouget, A., & Bavelier, D. (2010). Improved probabilistic inference as a general mechanism for learning with action video games. Current Biology, 23, 1573–1579.

    Article  Google Scholar 

  41. Green, C. S., Strobach, T., & Schubert, T. (2014). On methodological standards in training and transfer experiments. Psychological Research, 78(6), 756–772. https://doi.org/10.1007/s00426-013-0535-3.

    Article  PubMed  Google Scholar 

  42. Greitemeyer, T., Osswald, S., & Brauer, M. (2010). Playing prosocial video games increases empathy and decreases schadenfreude. Emotion, 10(6), 796–802. https://doi.org/10.1037/a0020194.

    Article  PubMed  Google Scholar 

  43. Hallock, H., Collins, D., Lampit, A., Deol, K., Fleming, J., & Valenzuela, M. (2016). Cognitive training for post-acute traumatic brain injury: a systematic review and meta-analysis. Frontiers in Human Neuroscience, 10, 537. https://doi.org/10.3389/fnhum.2016.00537.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Hawes, Z., Moss, J., Caswell, B., Naqvi, S., & MacKinnon, S. (2017). Enhancing children’s spatial and numerical skills through a dynamic spatial approach to early geometry instruction: effects of a 32-week intervention. Cognition and Instruction, 35(3), 236–264.

    Article  Google Scholar 

  45. Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), e1002106. https://doi.org/10.1371/journal.pbio.1002106.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Health, N. I. o. (2014). Notice of Revised NIH Definition of “Clinical Trial”. (NOT-OD-15-015). Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-015.html. Accessed 1 Nov 2018.

  47. Hendershot, T., Pan, H., Haines, J., Harlan, W. R., Marazita, M. L., McCarty, C. A., …, Hamilton, C. M.. (2015). Using the PhenX toolkit to add standard measures to a study. Curr Protoc Hum Genet, 86, 1 21 21–17. https://doi.org/10.1002/0471142905.hg0121s86.

  48. Hillman, C. H., Erickson, K. I., & Kramer, A. F. (2008). Be smart, exercise your heart: exercise effects on brain and cognition. Nature Reviews Neuroscience, 9, 58–65.

    Article  Google Scholar 

  49. Hills, M., & Armitage, P. (2004). The two-period cross-over clinical trial. 1979. British Journal of Clinical Pharmacology, 58(7), S703–S716; discussion S717-709. https://doi.org/10.1111/j.1365-2125.2004.02275.x.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Hodes, R. J., Insel, T. R., Landis, S. C., & Research, N. I. H. B. f. N. (2013). The NIH toolbox: setting a standard for biomedical research. Neurology, 80(11 Suppl 3), S1. https://doi.org/10.1212/WNL.0b013e3182872e90.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Holtzer, R., Shuman, M., Mahoney, J. R., Lipton, R., & Verghese, J. (2011). Cognitive fatigue defined in the context of attention networks. Neuropsychology, Development, and Cognition. Section B, Aging, Neuropsychology and Cognition, 18(1), 108–128. https://doi.org/10.1080/13825585.2010.517826.

    Article  PubMed  Google Scholar 

  52. Hommel, B., Colzato, L. S., Fischer, R., & Christoffels, I. K. (2011). Bilingualism and creativity: benefits in convergent thinking come with losses in divergent thinking. Frontiers in Psychology, 2, 273. https://doi.org/10.3389/fpsyg.2011.00273.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Howard, J. (2016). Do brain-training exercises really work? CNN. Retrieved from https://www.cnn.com/2016/10/20/health/brain-training-exercises/index.html. Accessed 1 Nov 2018.

  54. Hrobjartsson, A., Forfang, E., Haahr, M. T., Als-Nielsen, B., & Brorson, S. (2007). Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding. International Journal of Epidemiology, 36(3), 654–663. https://doi.org/10.1093/ije/dym020.

    Article  PubMed  Google Scholar 

  55. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008). Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences of the United States of America, 105(19), 6829–6833.

    Article  Google Scholar 

  56. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Shah, P. (2011). Short- and long-term benefits of cognitive training. Proceedings of the National Academy of Sciences of the United States of America, 108, 10081–10086.

    Article  Google Scholar 

  57. Jones, R. N., Marsiske, M., Ball, K., Rebok, G., Willis, S. L., Morris, J. N., & Tennstedt, S. L. (2013). The ACTIVE cognitive training interventions and trajectories of performance among older adults. Journal of Aging and Health, 25(8 Suppl), 186S–208S. https://doi.org/10.1177/0898264312461938.

    Article  PubMed  Google Scholar 

  58. Kaptchuk, T. J. (2001). The double-blind, randomized, placebo-controlled trial: gold standard or golden calf? Journal of Clinical Epidemiology, 54(6), 541–549.

    Article  Google Scholar 

  59. Kaptchuk, T. J., & Miller, F. G. (2015). Placebo effects in medicine. The New England Journal of Medicine, 373(1), 8–9. https://doi.org/10.1056/NEJMp1504023.

    Article  PubMed  Google Scholar 

  60. Kaptchuk, T. J., Friedlander, E., Kelley, J. M., Sanchez, M. N., Kokkotou, E., Singer, J. P., et al. (2010). Placebos without deception: a randomized controlled trial in irritable bowel syndrome. PLoS One, 5(12), e15591. https://doi.org/10.1371/journal.pone.0015591.

    Article  PubMed  PubMed Central  Google Scholar 

  61. Karbach, J., & Unger, K. (2014). Executive control training from middle childhood to adolescence. Frontiers in Psychology, 5, 390. https://doi.org/10.3389/fpsyg.2014.00390.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Katz, B., Jaeggi, S. M., Buschkuehl, M., Shah, P., & Jonides, J. (2018). The effect of monetary compensation on cognitive training outcomes. Learning and Motivation, 63(77–90).

  63. Kelley, J. M., Kaptchuk, T. J., Cusin, C., Lipkin, S., & Fava, M. (2012). Open-label placebo for major depressive disorder: a pilot randomized controlled trial. Psychotherapy and Psychosomatics, 81(5), 312–314. https://doi.org/10.1159/000337053.

    Article  PubMed  Google Scholar 

  64. Kirsch, I. (2005). Placebo psychotherapy: synonym or oxymoron? Journal of Clinical Psychology, 61(7), 791–803. https://doi.org/10.1002/jclp.20126.

    Article  PubMed  Google Scholar 

  65. Klingberg, T., Fernell, E., Olesen, P. J., Johnson, M., Gustafsson, P., Dahlstrom, K., et al. (2005). Computerized training of working memory in children with ADHD—a randomized, controlled trial. J Am Acad Child Adolesc Psychiatry, 44(2), 177–186.

    Article  Google Scholar 

  66. Kolahi, J., Bang, H., & Park, J. (2009). Towards a proposal for assessment of blinding success in clinical trials: up-to-date review. Community Dentistry and Oral Epidemiology, 37(6), 477–484. https://doi.org/10.1111/j.1600-0528.2009.00494.x.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Kramer, A. F., Larish, J., & Strayer, D. L. (1995). Training for attentional control in dual-task settings: a comparison of young and old adults. Journal of Experimental Psychology: Applied, 1, 50–76.

    Google Scholar 

  68. Lutz, A., Jha, A. P., Dunne, J. D., & Saron, C. D. (2015). Investigating the phenomenological matrix of mindfulness-related practices from a neurocognitive perspective. The American Psychologist, 70(7), 632–658. https://doi.org/10.1037/a0039585.

    Article  PubMed  PubMed Central  Google Scholar 

  69. Marchand, E., Stice, E., Rohde, P., & Becker, C. B. (2011). Moving from efficacy to effectiveness trials in prevention research. Behaviour Research and Therapy, 49(1), 32–41. https://doi.org/10.1016/j.brat.2010.10.008.

    Article  PubMed  Google Scholar 

  70. Mayer, R. E. (Ed.). (2014). Computer games for learning: an evidence-based approach. Cambridge: MIT Press.

    Google Scholar 

  71. Melby-Lervag, M., & Hulme, C. (2013). Is working memory training effective? A meta-analytic review. Developmental Psychology, 49(2), 270–291.

    Article  Google Scholar 

  72. Merzenich, M. M., Nahum, M., & Van Vleet, T. M. (2013). Neuroplasticity: introduction. Progress in Brain Research, 207, xxi–xxvi. https://doi.org/10.1016/B978-0-444-63327-9.10000-1.

    Article  PubMed  Google Scholar 

  73. Mohr, D. C., Schueller, S. M., Riley, W. T., Brown, C. H., Cuijpers, P., Duan, N., et al. (2015). Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. Journal of Medical Internet Research, 17(7), e166. https://doi.org/10.2196/jmir.4391.

    Article  PubMed  PubMed Central  Google Scholar 

  74. Morey, R. D., Romeijn, J.-W., & Rouder, J. N. (2016). The philosophy of Bayes factors and the quantification of statistical evidence. Journal of Mathematical Psychology, 72, 6–18.

    Article  Google Scholar 

  75. Nahum, M., Lee, H., & Merzenich, M. M. (2013). Principles of neuroplasticity-based rehabilitation. Progress in Brain Research, 207, 141–171. https://doi.org/10.1016/B978-0-444-63327-9.00009-6.

    Article  PubMed  Google Scholar 

  76. Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: investigating participant demand characteristics. The Journal of General Psychology, 135(2), 151–165.

    Article  Google Scholar 

  77. Noah, T., Yaacov, S., & Mayo, R. (2018). When both the original study and its failed replication are correct: feeling observed eliminates the facial-feedback effect. Journal of Personality and Social Psychology, 114(5), 657–664.

    Article  Google Scholar 

  78. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017). The preregistration revolution. OSF Preprints. doi:https://doi.org/10.17605/OSF.IO/2DXU5.

  79. O’Leary, K. D., Rosenbaum, A., & Hughes, P. C. (1978). Direct and systematic replication: a rejoinder. Journal of Abnormal Child Psychology, 6(3), 295–297.

    Article  Google Scholar 

  80. Onken, L. S., Carroll, K. M., Shoham, V., Cuthbert, B. N., & Riddle, M. (2014). Reenvisioning clinical science: unifying the discipline to improve the public health. Clinical Psychological Science: A Journal of the Association for Psychological Science, 2(1), 22–34. https://doi.org/10.1177/2167702613497932.

    Article  Google Scholar 

  81. Open Science, C. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657–660. https://doi.org/10.1177/1745691612462588.

    Article  Google Scholar 

  82. Orne, M. T. (1962). On the social psychology of the psychological experiment: with particular reference to demand characteristics and their implications. American Psychologist, 17, 776–783.

    Article  Google Scholar 

  83. Owen, A. M., Hampshire, A., Grahn, J. A., Stenton, R., Dajani, S., Burns, A. S., … Ballard, C. G. (2010). Putting brain training to the test. Nature, 465(7299), 775–778.

  84. Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536. https://doi.org/10.1177/1745691612463401.

    Article  Google Scholar 

  85. Pek, J., & Flora, D. B. (2017). Reporting effect sizes in original psychological research: a discussion and tutorial. Psychological Methods. doi:https://doi.org/10.1037/met0000126.

  86. Prakash, R. S., De Leon, A. A., Patterson, B., Schirda, B. L., & Janssen, A. L. (2014). Mindfulness and the aging brain: a proposed paradigm shift. Frontiers in Aging Neuroscience, 6, 120. https://doi.org/10.3389/fnagi.2014.00120.

    Article  PubMed  PubMed Central  Google Scholar 

  87. Rebok, G. W., Ball, K., Guey, L. T., Jones, R. N., Kim, H. Y., King, J. W., … Group, A. S. (2014). Ten-year effects of the advanced cognitive training for independent and vital elderly cognitive training trial on cognition and everyday functioning in older adults. Journal of the American Geriatrics Society, 62(1), 16–24. doi:https://doi.org/10.1111/jgs.12607.

  88. Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., … Engle, R. W. (2013). No evidence of intelligence improvement after working memory training: a randomized, placebo-controlled study. Journal of Experimental Psychology: General, 142(2), 359–379.

  89. Redick, T. S., Shipstead, Z., Wiemers, E. A., Melby-Lervag, M., & Hulme, C. (2015). What’s working in working memory training? An educational perspective. Educational Psychology Review, 27(4), 617–633. https://doi.org/10.1007/s10648-015-9314-6.

    Article  PubMed  PubMed Central  Google Scholar 

  90. Roberts, G., Quach, J., Spencer-Smith, M., Anderson, P. J., Gathercole, S., Gold, L., … Wake, M. (2016). Academic outcomes 2 years after working memory training for children with low working memory: a randomized clinical trial. JAMA Pediatrics, 170(5), e154568. doi:https://doi.org/10.1001/jamapediatrics.2015.4568.

  91. Rohde, T. E., & Thompson, L. A. (2007). Predicting academic achievement with cognitive ability. Intelligence, 35(1), 83–92.

    Article  Google Scholar 

  92. Rosenthal, R., & Jacobson, L. (1968). Pygmalion in the classroom: teacher expectation and pupils’ intellectual development. New York: Holt, Rinehart and Winston.

    Google Scholar 

  93. Ross, L. A., Edwards, J. D., O’Connor, M. L., Ball, K. K., Wadley, V. G., & Vance, D. E. (2016). The transfer of cognitive speed of processing training to older adults’ driving mobility across 5 years. The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 71(1), 87–97. https://doi.org/10.1093/geronb/gbv022.

    Article  PubMed  Google Scholar 

  94. Rotello, C. M., Heit, E., & Dube, C. (2015). When more data steer us wrong: replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin & Review, 22(4), 944–954. https://doi.org/10.3758/s13423-014-0759-2.

    Article  Google Scholar 

  95. Rothbaum, B. O., Price, M., Jovanovic, T., Norrholm, S. D., Gerardi, M., Dunlop, B., … Ressler, K. J. (2014). A randomized, double-blind evaluation of D-cycloserine or alprazolam combined with virtual reality exposure therapy for posttraumatic stress disorder in Iraq and Afghanistan War veterans. The American Journal of Psychiatry, 171(6), 640–648. doi:https://doi.org/10.1176/appi.ajp.2014.13121625.

  96. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237. https://doi.org/10.3758/PBR.16.2.225.

    Article  Google Scholar 

  97. Rubin, M. (2016). The Perceived Awareness of the Research Hypothesis Scale: assessing the influence of demand characteristics. In.

  98. Rutherford, B. R., Sneed, J. R., & Roose, S. P. (2009). Does study design influence outcome?. The effects of placebo control and treatment duration in antidepressant trials. Psychotherapy and Psychosomatics, 78(3), 172–181. https://doi.org/10.1159/000209348.

    Article  PubMed  PubMed Central  Google Scholar 

  99. Saghaei, M. (2011). An overview of randomization and minimization programs for randomized clinical trials. Journal of Medical Signals and Sensors, 1(1), 55–61.

    PubMed  PubMed Central  Google Scholar 

  100. Sandler, A. D., & Bodfish, J. W. (2008). Open-label use of placebos in the treatment of ADHD: a pilot study. Child: Care, Health and Development, 34(1), 104–110. https://doi.org/10.1111/j.1365-2214.2007.00797.x.

    Article  Google Scholar 

  101. Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: introduction. Educational Psychologist, 39(4), 199–201.

    Article  Google Scholar 

  102. Schellenberg, E. G. (2004). Music lessions enhance IQ. Psychological Science, 15(8), 511–514.

    Article  Google Scholar 

  103. Schlickum, M. K., Hedman, L., Enochsson, L., Kjellin, A., & Fellander-Tsai, L. (2009). Systematic video game training in surgical novices improves performance in virtual reality endoscopic surgical simulators: a prospective randomized study. World Journal of Surgery, 33(11), 2360–2367.

    Article  Google Scholar 

  104. Schmiedek, F., Lövdén, M., & Lindenberger, U. (2010). Hundred days of cognitive training enhance broad abilities in adulthood: findings from the COGITO study. Frontiers in Aging Neuroscience, 2.

  105. Schönbrodt, F. D., Wagenmakers, E. J., Zehetleitner, M., & Perugini, M. (2017). Sequential hypothesis testing with Bayes factors: efficiently testing mean differences. Psychological Methods, 22(2), 322–339.

    Article  Google Scholar 

  106. Schubert, T., & Strobach, T. (2012). Video game experience and optimized executive control skills—on false positives and false negatives: reply to Boot and Simons (2012). Acta Psychologica, 141(2), 278–280.

    Article  Google Scholar 

  107. Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The landscape and lexicon of blinding in randomized trials. Annals of Internal Medicine, 136(3), 254–259.

    Article  Google Scholar 

  108. Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32.

    Article  Google Scholar 

  109. Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training effective? Psychological Bulletin, 138(4), 623–654.

    Article  Google Scholar 

  110. Sidman, M. (1966). Tactics of scientific research: evaluating experimental data in psychology. Oxford: Basic Books.

    Google Scholar 

  111. Simons, D. J., Boot, W. R., Charness, N., Gathercole, S. E., Chabris, C. F., Hambrick, D. Z., & Stine-Morrow, E. A. (2016). Do “Brain-Training” programs work? Psychological Science in the Public Interest, 17(3), 103–186. https://doi.org/10.1177/1529100616661983.

    Article  PubMed  Google Scholar 

  112. Singal, A. G., Higgins, P. D., & Waljee, A. K. (2014). A primer on effectiveness and efficacy trials. Clinical Translational Gastroenterology, 5, e45. https://doi.org/10.1038/ctg.2013.13.

    Article  PubMed  Google Scholar 

  113. Smith, G. E., Housen, P., Yaffe, K., Ruff, R., Kennison, R. F., Mahncke, H. W., & Zelinski, E. M. (2009). A cognitive training program based on principles of brain plasticity: results from Improvement in Memory with Plasticity-based Adaptive cognitive Training (IMPACT) study. Journal of the American Geriatrics Society, 57(4), 594–603.

    Article  Google Scholar 

  114. Stieff, M., & Uttal, D. (2015). How much can spatial training improve STEM achievement. Educational Psychology Review, 27(4), 607–615.

    Article  Google Scholar 

  115. Stierlin, A. S., Herder, K., Helmbrecht, M. J., Prinz, S., Walendzik, J., Holzmann, M., … Kilian, R. (2014). Effectiveness and efficiency of integrated mental health care programmes in Germany: study protocol of an observational controlled trial. BMC Psychiatry, 14, 163. doi:https://doi.org/10.1186/1471-244X-14-163.

  116. Strack, F., Martin, L., & Stepper, S. (1988). Inhibiting and facilitating conditions of the human smile: a nonobtrusive test of the facial feedback hypothesis. Journal of Personality and Social Psychology, 54(5), 768–777.

    Article  Google Scholar 

  117. Strobach, T., & Karbach, J. (Eds.). (2016). Cognitive training: an overview of features and applications. New York: Springer.

    Google Scholar 

  118. Strobach, T., Frensch, P. A., & Schubert, T. (2012). Video game practice optimizes executive control skills in dual-task and task switching situations. Acta Psychologica, 140(1), 13–24.

    Article  Google Scholar 

  119. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59–71. https://doi.org/10.1177/1745691613514450.

    Article  Google Scholar 

  120. Subramaniam, K., Luks, T. L., Garrett, C., Chung, C., Fisher, M., Nagarajan, S., & Vinogradov, S. (2014). Intensive cognitive training in schizophrenia enhances working memory and associated prefrontal cortical efficiency in a manner that drives long-term functional gains. NeuroImage, 99, 281–292. https://doi.org/10.1016/j.neuroimage.2014.05.057.

    Article  PubMed  PubMed Central  Google Scholar 

  121. Sullivan, G. M., & Feinn, R. (2012). Using effect size-or why the P value is not enough. Journal of Graduate Medical Education, 4(3), 279–282. https://doi.org/10.4300/JGME-D-12-00156.1.

    Article  PubMed  PubMed Central  Google Scholar 

  122. Tang, Y. Y., Ma, Y., Wang, J., Fan, Y., Feng, S., Lu, Q., et al. (2007). Short-term meditation training improves attention and self-regulation. Proceedings of the National Academy of Sciences of the United States of America, 104(43), 17152–17156. https://doi.org/10.1073/pnas.0707678104.

    Article  PubMed  PubMed Central  Google Scholar 

  123. Taves, D. R. (1974). Minimization: a new method of assigning patients to treatment and control groups. Clinical Pharmacology and Therapeutics, 15, 443–453.

    Article  Google Scholar 

  124. Terlecki, M. S., Newcombe, N. S., & Little, M. (2008). Durable and generalized effects of spatial experience on mental rotation: gender differences in growth patterns. Applied Cognitive Psychology, 22, 996–1013.

    Article  Google Scholar 

  125. Tickle-Degnen, L. (2013). Nuts and bolts of conducting feasibility studies. The American Journal of Occupational Therapy, 67(2), 171–176. https://doi.org/10.5014/ajot.2013.006270.

    Article  PubMed  PubMed Central  Google Scholar 

  126. Tsai, N., Buschkuehl, M., Kamarsu, S., Shah, P., Jonides, J., & Jaeggi, S. M. (2018). (Un)Great expectations: the role of placebo effects in cognitive training. Journal of Applied Research in Memory and Cognition, 7(4), 564–573. https://doi.org/10.1016/j.jarmac.2018.06.001.

  127. Valdes, E.G., Andel, R., Lister, J.J., Gamaldo, A., & Edwards, J.D. (2017). Can cognitive speed of processing training improve everyday functioning among older adults with psychometrically defined mild cognitive impairment? Journal of Aging and Health, 898264317738828. https://doi.org/10.1177/0898264317738828.

  128. Van Dam, N. T., van Vugt, M. K., Vago, D. R., Schmalzl, L., Saron, C. D., Olendzki, A., et al. (2018). Mind the hype: a critical evaluation and prescriptive agenda for research on mindfulness and meditation. Perspectives on Psychological Science, 13(1), 36–61. https://doi.org/10.1177/1745691617709589.

    Article  PubMed  Google Scholar 

  129. Voss, M.W., Prakash, R.S., Erickson, K.I., Basak, C., Chaddock, L., Kim, J. S., … Kramer, A. F. (2010). Plasticity of brain networks in a randomized intervention trial of exercise training in older adults. Frontiers in Aging Neuroscience, 2. https://doi.org/10.3389/fnagi.2010.00032.

  130. Voudouris, N. J., Peck, C. L., & Coleman, G. (1985). Conditioned placebo responses. Journal of Personality and Social Psychology, 48(1), 47–53.

    Article  Google Scholar 

  131. Walton, A.G. (2016). Do brain training games work, or is it the placebo effect? Forbes.com . Retrieved from https://www.forbes.com/sites/alicegwalton/2016/06/21/does-brain-training-work-or-is-it-all-placebo/#3b654dc67497. Accessed 1 Nov 2018.

  132. Weintraub, S., Dikmen, S.S., Heaton, R.K., Tulsky, D.S., Zelazo, P.D., Bauer, P.J., … Gershon, R.C. (2013). Cognition assessment using the NIH toolbox. Neurology, 80(11 Suppl 3), S54–64. https://doi.org/10.1212/WNL.0b013e3182872ded.

  133. Wexler, B.E., Iseli, M., Leon, S., Zaggle, W., Rush, C., Goodman, A., … Bo, E. (2016). Cognitive priming and cognitive training: immediate and far transfer to academic skills in children. Scientific Reports, 6, 32859. doi:https://doi.org/10.1038/srep32859.

  134. Whitehead, A. L., Sully, B. G., & Campbell, M. J. (2014). Pilot and feasibility studies: is there a difference from each other and from a randomised controlled trial? Contemporary Clinical Trials, 38(1), 130–133. https://doi.org/10.1016/j.cct.2014.04.001.

    Article  PubMed  Google Scholar 

  135. Whitlock, L. A., McLaughlin, A. C., & Allaire, J. C. (2012). Individual differences in response to cognitive training: using a multi-modal, attentionally demanding game-based intervention for older adults. Computers in Human Behavior, 28(4), 1091–1096.

    Article  Google Scholar 

  136. Willis, S.L., Tennstedt, S.L., Marsiske, M., Ball, K., Elias, J., Koepke, K.M., … Group, A.S. (2006). Long-term effects of cognitive training on everyday functional outcomes in older adults. JAMA, 296(23), 2805–2814.

  137. Wolfe, J. M., & Kanwisher, N. G. (2018). Not your parent’s NIH clinical trial. Nature Human Behaviour, 2, 107–109.

    Article  Google Scholar 

  138. Wright, R., Thompson, W. L., Ganis, G., Newcombe, N. S., & Kosslyn, S. M. (2008). Training generalized spatial skills. Psychonomic Bulletin and Review, 15(4), 763–771.

    Article  Google Scholar 

  139. Zhao, W., Hill, M.D., & Palesch, Y. (2012). Minimal sufficient balance—a new strategy to balance baseline covariates and preserve randomness of treatment allocation. Statistical Methods in Medical Research https://doi.org/10.1177/0962280212436447.

  140. Zwaan, R.A., Etz, A., Lucas, R.E., & Donnellan, M. B. (2017). Making replication mainstream. The Behavioral and Brain Sciences, 1–50. https://doi.org/10.1017/S0140525X17001972.

Download references

Funding

National Science Foundation (DRL-1641280) to Dr C. Shawn Green; Office of Naval Research grant to Dr. Daphne Bavelier.

Author information

Affiliations

Authors

Corresponding author

Correspondence to C. Shawn Green.

Ethics declarations

Conflict of Interest

The following authors have declared conflict(s) of interest. Bavelier is a founding partner and on the scientific advisory board of Akili Interactive, Boston; Vinogradov is a consultant for Posit Science Corp, Alkermes, Inc., and Mindstrong, Inc.; Ball owns stock in the Visual Awareness Research Group (formerly Visual Awareness, Inc.) and Posit Science, Inc., the companies that market the Useful Field of View Test and speed of processing training software (now the Double Decision exercise in BrainHQ), and is a member of the Posit Science Scientific Advisory Board; Gazzaley is a co-founder, scientific advisor, and BOD member for Akili Interactive Lab and has several patents filed at UCSF for video game enhancement technologies; Jaeggi has an indirect financial interest in the MIND Research Institute, Irvine, CA, whose interests are related to this work; Levi is a member of the Scientific Advisory Board of NovaSight; Morris is on the scientific advisory boards of Neurotrack and of the AARP Global Council on Brain Health; Nahum is a paid consultant for Posit Science; Panizzutti is the founder of NeuroForma LTDA, a company with a financial interest in computerized cognitive training; Seitz is a founder and stakeholder in Carrot Neurotechnology, a company that sells a vision brain game called ULTIMEYES. The other authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Shawn Green, C., Bavelier, D., Kramer, A.F. et al. Improving Methodological Standards in Behavioral Interventions for Cognitive Enhancement. J Cogn Enhanc 3, 2–29 (2019). https://doi.org/10.1007/s41465-018-0115-y

Download citation

Keywords

  • Cognitive enhancement
  • Behavioral intervention methodology