There is substantial interest in the possibility that cognitive skills can be improved by dedicated behavioral training. Yet despite the large amount of work being conducted in this domain, there is not an explicit and widely agreed upon consensus around the best methodological practices. This document seeks to fill this gap. We start from the perspective that there are many types of studies that are important in this domain—e.g., feasibility, mechanistic, efficacy, and effectiveness. These studies have fundamentally different goals, and, as such, the best-practice methods to meet those goals will also differ. We thus make suggestions in topics ranging from the design and implementation of control groups, to reporting of results, to dissemination and communication, taking the perspective that the best practices are not necessarily uniform across all study types. We also explicitly recognize and discuss the fact that there are methodological issues around which we currently lack the theoretical and/or empirical foundation to determine best practices (e.g., as pertains to assessing participant expectations). For these, we suggest important routes forward, including greater interdisciplinary collaboration with individuals from domains that face related concerns. Our hope is that these recommendations will greatly increase the rate at which science in this domain advances.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Acosta, A., Adams, R.B., Jr., Albohn, D.N., Allard, E.S., Beek, T., Benning, S. D., … Zwaan, R. A. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917–928. doi:https://doi.org/10.1177/1745691616674458.
Andrews, G. (1999). Efficacy, effectiveness and efficiency in mental health service delivery. The Australian and New Zealand Journal of Psychiatry, 33(3), 316–322. https://doi.org/10.1046/j.1440-1614.1999.00581.x.
Anguera, J.A., Boccanfuso, J., Rintoul, J.L., Al-Hashimi, O., Faraji, F., Janowich, J., … Gazzaley, A. (2013). Video game training enhances cognitive control in older adults. Nature, 501(7465), 97–101. doi:https://doi.org/10.1038/nature12486.
Au, J., Sheehan, E., Tsai, N., Duncan, G. J., Buschkuehl, M., & Jaeggi, S. M. (2015). Improving fluid intelligence with training on working memory: a meta-analysis. Psychonomic Bulletin & Review, 22(2), 366–377. https://doi.org/10.3758/s13423-014-0699-x.
Ball, K., Berch, D. B., Helmers, K. F., Jobe, J. B., Leveck, M. D., Marsiske, M., … Group, A. S. (2002). Effects of cognitive training interventions with older adults: a randomized controlled trial. JAMA, 288(18), 2271–2281.
Baniqued, P., Allen, C.M., Kranz, M.B., Johnson, K., Sipolins, A., Dickens, C., …, Kramer, A.F. (2015). Working memory, reasoning, and task switching training: transfer effects, limitations, and great expectations?. PLoS One. doi:https://doi.org/10.1371/journal.pone.0142169.
Barnett, S. M., & Ceci, S. J. (2002). When and where do we apply what we learn?: A taxonomy for far transfer. Psychological Bulletin, 128(4), 612–637.
Barry, A. E., Szucs, L. E., Reyes, J. V., Ji, Q., Wilson, K. L., & Thompson, B. (2016). Failure to report effect sizes: the handling of quantitative results in published health education and behavior research. Health Education & Behavior, 43(5), 518–527. https://doi.org/10.1177/1090198116669521.
Basak, C., Boot, W. R., Voss, M. W., & Kramer, A. F. (2008). Can training in a real-time strategy video game attenuate cognitive decline in older adults. Psychology and Aging, 23(4), 765–777.
Bavelier, D., & Davidson, R. J. (2013). Brain training: games to do you good. Nature, 494(7438), 425–426. https://doi.org/10.1038/494425a.
Bayraktar, S. (2001). A meta-analysis of the effectiveness of computer-assisted instruction in science education. Journal of Research on Technology in Education, 34(2), 173–188.
Beaumont, J. L., Havlik, R., Cook, K. F., Hays, R. D., Wallner-Allen, K., Korper, S. P., …, Gershon, R. (2013). Norming plans for the NIH toolbox. Neurology, 80(11 Suppl 3), S87–92. doi:https://doi.org/10.1212/WNL.0b013e3182872e70.
Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018). Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills. Psychological Bulletin, 144(1), 77–110. https://doi.org/10.1037/bul0000130.
Biagianti, B., & Vinogradov, S. (2013). Computerized cognitive training targeting brain plasticity in schizophrenia. Progress in Brain Research, 207, 301–326. https://doi.org/10.1016/B978-0-444-63327-9.00011-4.
Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology: why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8(4), 445–454. https://doi.org/10.1177/1745691613491271.
Bryant, D. C. N., & Deluca, J. (2004). Objective measurement of cognitive fatigue in multiple sclerosis. Rehabilitation Psychology, 49(2), 114–122.
Carvalho, C., Caetano, J. M., Cunha, L., Rebouta, P., Kaptchuk, T. J., & Kirsch, I. (2016). Open-label placebo treatment in chronic low back pain: a randomized controlled trial. Pain, 157(12), 2766–2772. https://doi.org/10.1097/j.pain.0000000000000700.
Chen, L. H., & Lee, W. C. (2011). Two-way minimization: a novel treatment allocation method for small trials. PLoS One, 6(12), e28604.
Coburn, K. M., & Vevea, J. L. (2015). Publication bias as a function of study characteristics. Psychological Methods, 20(3), 310–330. https://doi.org/10.1037/met0000046.
Colloca, L., & Benedetti, F. (2006). How prior experience shapes placebo analgesia. Pain, 124(1–2), 126–133. https://doi.org/10.1016/j.pain.2006.04.005.
Colloca, L., Klinger, R., Flor, H., & Bingel, U. (2013). Placebo analgesia: psychological and neurobiological mechanisms. Pain, 154(4), 511–514. https://doi.org/10.1016/j.pain.2013.02.002.
Colzato, L. S., van den Wildenberg, W. P., & Hommel, B. (2014). Cognitive control and the COMT Val(1)(5)(8)Met polymorphism: genetic modulation of videogame training and transfer to task-switching efficiency. Psychological Research, 78(5), 670–678. https://doi.org/10.1007/s00426-013-0514-8.
Davidson, R. J., & Dahl, C. J. (2017). Varieties of contemplative practice. JAMA Psychiatry, 74(2), 121–123. https://doi.org/10.1001/jamapsychiatry.2016.3469.
Davidson, R. J., & Dahl, C. J. (2018). Outstanding challenges in scientific research on mindfulness and meditation. Perspectives on Psychological Science, 13(1), 62–65. https://doi.org/10.1177/1745691617718358.
Deveau, J., Jaeggi, S. M., Zordan, V., Phung, C., & Seitz, A. R. (2014a). How to build better memory training games. Frontiers in Systems Neuroscience, 8, 243. https://doi.org/10.3389/fnsys.2014.00243.
Deveau, J., Ozer, D. J., & Seitz, A. R. (2014b). Improved vision and on-field performance in baseball through perceptual learning. Current Biology, 24(4), R146–R147. https://doi.org/10.1016/j.cub.2014.01.004.
Diao, D., Wright, J. M., Cundiff, D. K., & Gueyffier, F. (2012). Pharmacotherapy for mild hypertension. Cochrane Database of Systematic Reviews(8). doi:https://doi.org/10.1002/14651858.CD006742.pub2.
Dweck, C. (2006). Mindset: the new psychology of success. New York: Random House.
Eldridge, S. M., Lancaster, G. A., Campbell, M. J., Thabane, L., Hopewell, S., Coleman, C. L., & Bond, C. M. (2016). Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One, 11(3), e0150205. https://doi.org/10.1371/journal.pone.0150205.
Engle, R. W., Tuholski, S. W., Laughlin, J. E., & Conway, A. R. A. (1999). Working memory, short-term memory, and general fluid intelligence: a latent variable approach. Journal of Experimental Psychology: General, 128, 309–331.
Erickson, K. I., Voss, M. W., Prakash, R. S., Basak, C., Szabo, A., Chaddock, L., et al. (2011). Exercise training increases size of hippocampus and improves memory. Proceedings of the National Academy of Sciences of the United States of America, 108(7), 3017–3022. https://doi.org/10.1073/pnas.1015950108.
Fassler, M., Meissner, K., Kleijnen, J., Hrobjartsson, A., & Linde, K. (2015). A systematic review found no consistent difference in effect between more and less intensive placebo interventions. Journal of Clinical Epidemiology, 68(4), 442–451. https://doi.org/10.1016/j.jclinepi.2014.11.018.
FEDERAL TRADE COMMISSION, (2016) Plaintiff, V.. Lumos Labs, Inc., a corporation d/b/a Lumosity and Kunal Sarkar, and Michael Scanlon, Individually and as officers of LumoS Labs, Inc. Defendants. , No. Case No. 3:16-cv-00001-sk (United StateS DistricT Court for the Northern District of California, San Francisco Division.
Fergusson, D., Glass, K. C., Waring, D., & Shapiro, S. (2004). Turning a blind eye: the success of blinding reported in a random sample of randomised placebo controlled trials. BMJ, 328(7437), 432. https://doi.org/10.1136/bmj.37952.631667.EE.
Foroughi, C. K., Monfort, S. S., Paczynski, M., McKnight, P. E., & Greenwood, P. M. (2016). Placebo effects in cognitive training. PNAS, 113, 7470–7474. https://doi.org/10.1073/pnas.1601243113.
Franceschini, S., Gori, S., Ruffino, M., Viola, S., Molteni, M., & Facoetti, A. (2013). Action video games make dyslexic children read better. Current Biology, 23(6), 462–466. https://doi.org/10.1016/j.cub.2013.01.044.
Fritz, J. M., & Cleland, J. (2003). Effectiveness versus efficacy: more than a debate over language. The Journal of Orthopaedic and Sports Physical Therapy, 33(4), 163–165. https://doi.org/10.2519/jospt.2003.33.4.163.
Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention. Nature, 423(6939), 534–537.
Green, C. S., & Bavelier, D. (2012). Learning, attentional control and action video games. Current Biology, 22, R197–R206.
Green, C. S., Pouget, A., & Bavelier, D. (2010). Improved probabilistic inference as a general mechanism for learning with action video games. Current Biology, 23, 1573–1579.
Green, C. S., Strobach, T., & Schubert, T. (2014). On methodological standards in training and transfer experiments. Psychological Research, 78(6), 756–772. https://doi.org/10.1007/s00426-013-0535-3.
Greitemeyer, T., Osswald, S., & Brauer, M. (2010). Playing prosocial video games increases empathy and decreases schadenfreude. Emotion, 10(6), 796–802. https://doi.org/10.1037/a0020194.
Hallock, H., Collins, D., Lampit, A., Deol, K., Fleming, J., & Valenzuela, M. (2016). Cognitive training for post-acute traumatic brain injury: a systematic review and meta-analysis. Frontiers in Human Neuroscience, 10, 537. https://doi.org/10.3389/fnhum.2016.00537.
Hawes, Z., Moss, J., Caswell, B., Naqvi, S., & MacKinnon, S. (2017). Enhancing children’s spatial and numerical skills through a dynamic spatial approach to early geometry instruction: effects of a 32-week intervention. Cognition and Instruction, 35(3), 236–264.
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), e1002106. https://doi.org/10.1371/journal.pbio.1002106.
Health, N. I. o. (2014). Notice of Revised NIH Definition of “Clinical Trial”. (NOT-OD-15-015). Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-015.html. Accessed 1 Nov 2018.
Hendershot, T., Pan, H., Haines, J., Harlan, W. R., Marazita, M. L., McCarty, C. A., …, Hamilton, C. M.. (2015). Using the PhenX toolkit to add standard measures to a study. Curr Protoc Hum Genet, 86, 1 21 21–17. https://doi.org/10.1002/0471142905.hg0121s86.
Hillman, C. H., Erickson, K. I., & Kramer, A. F. (2008). Be smart, exercise your heart: exercise effects on brain and cognition. Nature Reviews Neuroscience, 9, 58–65.
Hills, M., & Armitage, P. (2004). The two-period cross-over clinical trial. 1979. British Journal of Clinical Pharmacology, 58(7), S703–S716; discussion S717-709. https://doi.org/10.1111/j.1365-2125.2004.02275.x.
Hodes, R. J., Insel, T. R., Landis, S. C., & Research, N. I. H. B. f. N. (2013). The NIH toolbox: setting a standard for biomedical research. Neurology, 80(11 Suppl 3), S1. https://doi.org/10.1212/WNL.0b013e3182872e90.
Holtzer, R., Shuman, M., Mahoney, J. R., Lipton, R., & Verghese, J. (2011). Cognitive fatigue defined in the context of attention networks. Neuropsychology, Development, and Cognition. Section B, Aging, Neuropsychology and Cognition, 18(1), 108–128. https://doi.org/10.1080/13825585.2010.517826.
Hommel, B., Colzato, L. S., Fischer, R., & Christoffels, I. K. (2011). Bilingualism and creativity: benefits in convergent thinking come with losses in divergent thinking. Frontiers in Psychology, 2, 273. https://doi.org/10.3389/fpsyg.2011.00273.
Howard, J. (2016). Do brain-training exercises really work? CNN. Retrieved from https://www.cnn.com/2016/10/20/health/brain-training-exercises/index.html. Accessed 1 Nov 2018.
Hrobjartsson, A., Forfang, E., Haahr, M. T., Als-Nielsen, B., & Brorson, S. (2007). Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding. International Journal of Epidemiology, 36(3), 654–663. https://doi.org/10.1093/ije/dym020.
Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008). Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences of the United States of America, 105(19), 6829–6833.
Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Shah, P. (2011). Short- and long-term benefits of cognitive training. Proceedings of the National Academy of Sciences of the United States of America, 108, 10081–10086.
Jones, R. N., Marsiske, M., Ball, K., Rebok, G., Willis, S. L., Morris, J. N., & Tennstedt, S. L. (2013). The ACTIVE cognitive training interventions and trajectories of performance among older adults. Journal of Aging and Health, 25(8 Suppl), 186S–208S. https://doi.org/10.1177/0898264312461938.
Kaptchuk, T. J. (2001). The double-blind, randomized, placebo-controlled trial: gold standard or golden calf? Journal of Clinical Epidemiology, 54(6), 541–549.
Kaptchuk, T. J., & Miller, F. G. (2015). Placebo effects in medicine. The New England Journal of Medicine, 373(1), 8–9. https://doi.org/10.1056/NEJMp1504023.
Kaptchuk, T. J., Friedlander, E., Kelley, J. M., Sanchez, M. N., Kokkotou, E., Singer, J. P., et al. (2010). Placebos without deception: a randomized controlled trial in irritable bowel syndrome. PLoS One, 5(12), e15591. https://doi.org/10.1371/journal.pone.0015591.
Karbach, J., & Unger, K. (2014). Executive control training from middle childhood to adolescence. Frontiers in Psychology, 5, 390. https://doi.org/10.3389/fpsyg.2014.00390.
Katz, B., Jaeggi, S. M., Buschkuehl, M., Shah, P., & Jonides, J. (2018). The effect of monetary compensation on cognitive training outcomes. Learning and Motivation, 63(77–90).
Kelley, J. M., Kaptchuk, T. J., Cusin, C., Lipkin, S., & Fava, M. (2012). Open-label placebo for major depressive disorder: a pilot randomized controlled trial. Psychotherapy and Psychosomatics, 81(5), 312–314. https://doi.org/10.1159/000337053.
Kirsch, I. (2005). Placebo psychotherapy: synonym or oxymoron? Journal of Clinical Psychology, 61(7), 791–803. https://doi.org/10.1002/jclp.20126.
Klingberg, T., Fernell, E., Olesen, P. J., Johnson, M., Gustafsson, P., Dahlstrom, K., et al. (2005). Computerized training of working memory in children with ADHD—a randomized, controlled trial. J Am Acad Child Adolesc Psychiatry, 44(2), 177–186.
Kolahi, J., Bang, H., & Park, J. (2009). Towards a proposal for assessment of blinding success in clinical trials: up-to-date review. Community Dentistry and Oral Epidemiology, 37(6), 477–484. https://doi.org/10.1111/j.1600-0528.2009.00494.x.
Kramer, A. F., Larish, J., & Strayer, D. L. (1995). Training for attentional control in dual-task settings: a comparison of young and old adults. Journal of Experimental Psychology: Applied, 1, 50–76.
Lutz, A., Jha, A. P., Dunne, J. D., & Saron, C. D. (2015). Investigating the phenomenological matrix of mindfulness-related practices from a neurocognitive perspective. The American Psychologist, 70(7), 632–658. https://doi.org/10.1037/a0039585.
Marchand, E., Stice, E., Rohde, P., & Becker, C. B. (2011). Moving from efficacy to effectiveness trials in prevention research. Behaviour Research and Therapy, 49(1), 32–41. https://doi.org/10.1016/j.brat.2010.10.008.
Mayer, R. E. (Ed.). (2014). Computer games for learning: an evidence-based approach. Cambridge: MIT Press.
Melby-Lervag, M., & Hulme, C. (2013). Is working memory training effective? A meta-analytic review. Developmental Psychology, 49(2), 270–291.
Merzenich, M. M., Nahum, M., & Van Vleet, T. M. (2013). Neuroplasticity: introduction. Progress in Brain Research, 207, xxi–xxvi. https://doi.org/10.1016/B978-0-444-63327-9.10000-1.
Mohr, D. C., Schueller, S. M., Riley, W. T., Brown, C. H., Cuijpers, P., Duan, N., et al. (2015). Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. Journal of Medical Internet Research, 17(7), e166. https://doi.org/10.2196/jmir.4391.
Morey, R. D., Romeijn, J.-W., & Rouder, J. N. (2016). The philosophy of Bayes factors and the quantification of statistical evidence. Journal of Mathematical Psychology, 72, 6–18.
Nahum, M., Lee, H., & Merzenich, M. M. (2013). Principles of neuroplasticity-based rehabilitation. Progress in Brain Research, 207, 141–171. https://doi.org/10.1016/B978-0-444-63327-9.00009-6.
Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: investigating participant demand characteristics. The Journal of General Psychology, 135(2), 151–165.
Noah, T., Yaacov, S., & Mayo, R. (2018). When both the original study and its failed replication are correct: feeling observed eliminates the facial-feedback effect. Journal of Personality and Social Psychology, 114(5), 657–664.
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017). The preregistration revolution. OSF Preprints. doi:https://doi.org/10.17605/OSF.IO/2DXU5.
O’Leary, K. D., Rosenbaum, A., & Hughes, P. C. (1978). Direct and systematic replication: a rejoinder. Journal of Abnormal Child Psychology, 6(3), 295–297.
Onken, L. S., Carroll, K. M., Shoham, V., Cuthbert, B. N., & Riddle, M. (2014). Reenvisioning clinical science: unifying the discipline to improve the public health. Clinical Psychological Science: A Journal of the Association for Psychological Science, 2(1), 22–34. https://doi.org/10.1177/2167702613497932.
Open Science, C. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657–660. https://doi.org/10.1177/1745691612462588.
Orne, M. T. (1962). On the social psychology of the psychological experiment: with particular reference to demand characteristics and their implications. American Psychologist, 17, 776–783.
Owen, A. M., Hampshire, A., Grahn, J. A., Stenton, R., Dajani, S., Burns, A. S., … Ballard, C. G. (2010). Putting brain training to the test. Nature, 465(7299), 775–778.
Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536. https://doi.org/10.1177/1745691612463401.
Pek, J., & Flora, D. B. (2017). Reporting effect sizes in original psychological research: a discussion and tutorial. Psychological Methods. doi:https://doi.org/10.1037/met0000126.
Prakash, R. S., De Leon, A. A., Patterson, B., Schirda, B. L., & Janssen, A. L. (2014). Mindfulness and the aging brain: a proposed paradigm shift. Frontiers in Aging Neuroscience, 6, 120. https://doi.org/10.3389/fnagi.2014.00120.
Rebok, G. W., Ball, K., Guey, L. T., Jones, R. N., Kim, H. Y., King, J. W., … Group, A. S. (2014). Ten-year effects of the advanced cognitive training for independent and vital elderly cognitive training trial on cognition and everyday functioning in older adults. Journal of the American Geriatrics Society, 62(1), 16–24. doi:https://doi.org/10.1111/jgs.12607.
Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., … Engle, R. W. (2013). No evidence of intelligence improvement after working memory training: a randomized, placebo-controlled study. Journal of Experimental Psychology: General, 142(2), 359–379.
Redick, T. S., Shipstead, Z., Wiemers, E. A., Melby-Lervag, M., & Hulme, C. (2015). What’s working in working memory training? An educational perspective. Educational Psychology Review, 27(4), 617–633. https://doi.org/10.1007/s10648-015-9314-6.
Roberts, G., Quach, J., Spencer-Smith, M., Anderson, P. J., Gathercole, S., Gold, L., … Wake, M. (2016). Academic outcomes 2 years after working memory training for children with low working memory: a randomized clinical trial. JAMA Pediatrics, 170(5), e154568. doi:https://doi.org/10.1001/jamapediatrics.2015.4568.
Rohde, T. E., & Thompson, L. A. (2007). Predicting academic achievement with cognitive ability. Intelligence, 35(1), 83–92.
Rosenthal, R., & Jacobson, L. (1968). Pygmalion in the classroom: teacher expectation and pupils’ intellectual development. New York: Holt, Rinehart and Winston.
Ross, L. A., Edwards, J. D., O’Connor, M. L., Ball, K. K., Wadley, V. G., & Vance, D. E. (2016). The transfer of cognitive speed of processing training to older adults’ driving mobility across 5 years. The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 71(1), 87–97. https://doi.org/10.1093/geronb/gbv022.
Rotello, C. M., Heit, E., & Dube, C. (2015). When more data steer us wrong: replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin & Review, 22(4), 944–954. https://doi.org/10.3758/s13423-014-0759-2.
Rothbaum, B. O., Price, M., Jovanovic, T., Norrholm, S. D., Gerardi, M., Dunlop, B., … Ressler, K. J. (2014). A randomized, double-blind evaluation of D-cycloserine or alprazolam combined with virtual reality exposure therapy for posttraumatic stress disorder in Iraq and Afghanistan War veterans. The American Journal of Psychiatry, 171(6), 640–648. doi:https://doi.org/10.1176/appi.ajp.2014.13121625.
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237. https://doi.org/10.3758/PBR.16.2.225.
Rubin, M. (2016). The Perceived Awareness of the Research Hypothesis Scale: assessing the influence of demand characteristics. In.
Rutherford, B. R., Sneed, J. R., & Roose, S. P. (2009). Does study design influence outcome?. The effects of placebo control and treatment duration in antidepressant trials. Psychotherapy and Psychosomatics, 78(3), 172–181. https://doi.org/10.1159/000209348.
Saghaei, M. (2011). An overview of randomization and minimization programs for randomized clinical trials. Journal of Medical Signals and Sensors, 1(1), 55–61.
Sandler, A. D., & Bodfish, J. W. (2008). Open-label use of placebos in the treatment of ADHD: a pilot study. Child: Care, Health and Development, 34(1), 104–110. https://doi.org/10.1111/j.1365-2214.2007.00797.x.
Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: introduction. Educational Psychologist, 39(4), 199–201.
Schellenberg, E. G. (2004). Music lessions enhance IQ. Psychological Science, 15(8), 511–514.
Schlickum, M. K., Hedman, L., Enochsson, L., Kjellin, A., & Fellander-Tsai, L. (2009). Systematic video game training in surgical novices improves performance in virtual reality endoscopic surgical simulators: a prospective randomized study. World Journal of Surgery, 33(11), 2360–2367.
Schmiedek, F., Lövdén, M., & Lindenberger, U. (2010). Hundred days of cognitive training enhance broad abilities in adulthood: findings from the COGITO study. Frontiers in Aging Neuroscience, 2.
Schönbrodt, F. D., Wagenmakers, E. J., Zehetleitner, M., & Perugini, M. (2017). Sequential hypothesis testing with Bayes factors: efficiently testing mean differences. Psychological Methods, 22(2), 322–339.
Schubert, T., & Strobach, T. (2012). Video game experience and optimized executive control skills—on false positives and false negatives: reply to Boot and Simons (2012). Acta Psychologica, 141(2), 278–280.
Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The landscape and lexicon of blinding in randomized trials. Annals of Internal Medicine, 136(3), 254–259.
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32.
Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training effective? Psychological Bulletin, 138(4), 623–654.
Sidman, M. (1966). Tactics of scientific research: evaluating experimental data in psychology. Oxford: Basic Books.
Simons, D. J., Boot, W. R., Charness, N., Gathercole, S. E., Chabris, C. F., Hambrick, D. Z., & Stine-Morrow, E. A. (2016). Do “Brain-Training” programs work? Psychological Science in the Public Interest, 17(3), 103–186. https://doi.org/10.1177/1529100616661983.
Singal, A. G., Higgins, P. D., & Waljee, A. K. (2014). A primer on effectiveness and efficacy trials. Clinical Translational Gastroenterology, 5, e45. https://doi.org/10.1038/ctg.2013.13.
Smith, G. E., Housen, P., Yaffe, K., Ruff, R., Kennison, R. F., Mahncke, H. W., & Zelinski, E. M. (2009). A cognitive training program based on principles of brain plasticity: results from Improvement in Memory with Plasticity-based Adaptive cognitive Training (IMPACT) study. Journal of the American Geriatrics Society, 57(4), 594–603.
Stieff, M., & Uttal, D. (2015). How much can spatial training improve STEM achievement. Educational Psychology Review, 27(4), 607–615.
Stierlin, A. S., Herder, K., Helmbrecht, M. J., Prinz, S., Walendzik, J., Holzmann, M., … Kilian, R. (2014). Effectiveness and efficiency of integrated mental health care programmes in Germany: study protocol of an observational controlled trial. BMC Psychiatry, 14, 163. doi:https://doi.org/10.1186/1471-244X-14-163.
Strack, F., Martin, L., & Stepper, S. (1988). Inhibiting and facilitating conditions of the human smile: a nonobtrusive test of the facial feedback hypothesis. Journal of Personality and Social Psychology, 54(5), 768–777.
Strobach, T., & Karbach, J. (Eds.). (2016). Cognitive training: an overview of features and applications. New York: Springer.
Strobach, T., Frensch, P. A., & Schubert, T. (2012). Video game practice optimizes executive control skills in dual-task and task switching situations. Acta Psychologica, 140(1), 13–24.
Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59–71. https://doi.org/10.1177/1745691613514450.
Subramaniam, K., Luks, T. L., Garrett, C., Chung, C., Fisher, M., Nagarajan, S., & Vinogradov, S. (2014). Intensive cognitive training in schizophrenia enhances working memory and associated prefrontal cortical efficiency in a manner that drives long-term functional gains. NeuroImage, 99, 281–292. https://doi.org/10.1016/j.neuroimage.2014.05.057.
Sullivan, G. M., & Feinn, R. (2012). Using effect size-or why the P value is not enough. Journal of Graduate Medical Education, 4(3), 279–282. https://doi.org/10.4300/JGME-D-12-00156.1.
Tang, Y. Y., Ma, Y., Wang, J., Fan, Y., Feng, S., Lu, Q., et al. (2007). Short-term meditation training improves attention and self-regulation. Proceedings of the National Academy of Sciences of the United States of America, 104(43), 17152–17156. https://doi.org/10.1073/pnas.0707678104.
Taves, D. R. (1974). Minimization: a new method of assigning patients to treatment and control groups. Clinical Pharmacology and Therapeutics, 15, 443–453.
Terlecki, M. S., Newcombe, N. S., & Little, M. (2008). Durable and generalized effects of spatial experience on mental rotation: gender differences in growth patterns. Applied Cognitive Psychology, 22, 996–1013.
Tickle-Degnen, L. (2013). Nuts and bolts of conducting feasibility studies. The American Journal of Occupational Therapy, 67(2), 171–176. https://doi.org/10.5014/ajot.2013.006270.
Tsai, N., Buschkuehl, M., Kamarsu, S., Shah, P., Jonides, J., & Jaeggi, S. M. (2018). (Un)Great expectations: the role of placebo effects in cognitive training. Journal of Applied Research in Memory and Cognition, 7(4), 564–573. https://doi.org/10.1016/j.jarmac.2018.06.001.
Valdes, E.G., Andel, R., Lister, J.J., Gamaldo, A., & Edwards, J.D. (2017). Can cognitive speed of processing training improve everyday functioning among older adults with psychometrically defined mild cognitive impairment? Journal of Aging and Health, 898264317738828. https://doi.org/10.1177/0898264317738828.
Van Dam, N. T., van Vugt, M. K., Vago, D. R., Schmalzl, L., Saron, C. D., Olendzki, A., et al. (2018). Mind the hype: a critical evaluation and prescriptive agenda for research on mindfulness and meditation. Perspectives on Psychological Science, 13(1), 36–61. https://doi.org/10.1177/1745691617709589.
Voss, M.W., Prakash, R.S., Erickson, K.I., Basak, C., Chaddock, L., Kim, J. S., … Kramer, A. F. (2010). Plasticity of brain networks in a randomized intervention trial of exercise training in older adults. Frontiers in Aging Neuroscience, 2. https://doi.org/10.3389/fnagi.2010.00032.
Voudouris, N. J., Peck, C. L., & Coleman, G. (1985). Conditioned placebo responses. Journal of Personality and Social Psychology, 48(1), 47–53.
Walton, A.G. (2016). Do brain training games work, or is it the placebo effect? Forbes.com . Retrieved from https://www.forbes.com/sites/alicegwalton/2016/06/21/does-brain-training-work-or-is-it-all-placebo/#3b654dc67497. Accessed 1 Nov 2018.
Weintraub, S., Dikmen, S.S., Heaton, R.K., Tulsky, D.S., Zelazo, P.D., Bauer, P.J., … Gershon, R.C. (2013). Cognition assessment using the NIH toolbox. Neurology, 80(11 Suppl 3), S54–64. https://doi.org/10.1212/WNL.0b013e3182872ded.
Wexler, B.E., Iseli, M., Leon, S., Zaggle, W., Rush, C., Goodman, A., … Bo, E. (2016). Cognitive priming and cognitive training: immediate and far transfer to academic skills in children. Scientific Reports, 6, 32859. doi:https://doi.org/10.1038/srep32859.
Whitehead, A. L., Sully, B. G., & Campbell, M. J. (2014). Pilot and feasibility studies: is there a difference from each other and from a randomised controlled trial? Contemporary Clinical Trials, 38(1), 130–133. https://doi.org/10.1016/j.cct.2014.04.001.
Whitlock, L. A., McLaughlin, A. C., & Allaire, J. C. (2012). Individual differences in response to cognitive training: using a multi-modal, attentionally demanding game-based intervention for older adults. Computers in Human Behavior, 28(4), 1091–1096.
Willis, S.L., Tennstedt, S.L., Marsiske, M., Ball, K., Elias, J., Koepke, K.M., … Group, A.S. (2006). Long-term effects of cognitive training on everyday functional outcomes in older adults. JAMA, 296(23), 2805–2814.
Wolfe, J. M., & Kanwisher, N. G. (2018). Not your parent’s NIH clinical trial. Nature Human Behaviour, 2, 107–109.
Wright, R., Thompson, W. L., Ganis, G., Newcombe, N. S., & Kosslyn, S. M. (2008). Training generalized spatial skills. Psychonomic Bulletin and Review, 15(4), 763–771.
Zhao, W., Hill, M.D., & Palesch, Y. (2012). Minimal sufficient balance—a new strategy to balance baseline covariates and preserve randomness of treatment allocation. Statistical Methods in Medical Research https://doi.org/10.1177/0962280212436447.
Zwaan, R.A., Etz, A., Lucas, R.E., & Donnellan, M. B. (2017). Making replication mainstream. The Behavioral and Brain Sciences, 1–50. https://doi.org/10.1017/S0140525X17001972.
National Science Foundation (DRL-1641280) to Dr C. Shawn Green; Office of Naval Research grant to Dr. Daphne Bavelier.
Conflict of Interest
The following authors have declared conflict(s) of interest. Bavelier is a founding partner and on the scientific advisory board of Akili Interactive, Boston; Vinogradov is a consultant for Posit Science Corp, Alkermes, Inc., and Mindstrong, Inc.; Ball owns stock in the Visual Awareness Research Group (formerly Visual Awareness, Inc.) and Posit Science, Inc., the companies that market the Useful Field of View Test and speed of processing training software (now the Double Decision exercise in BrainHQ), and is a member of the Posit Science Scientific Advisory Board; Gazzaley is a co-founder, scientific advisor, and BOD member for Akili Interactive Lab and has several patents filed at UCSF for video game enhancement technologies; Jaeggi has an indirect financial interest in the MIND Research Institute, Irvine, CA, whose interests are related to this work; Levi is a member of the Scientific Advisory Board of NovaSight; Morris is on the scientific advisory boards of Neurotrack and of the AARP Global Council on Brain Health; Nahum is a paid consultant for Posit Science; Panizzutti is the founder of NeuroForma LTDA, a company with a financial interest in computerized cognitive training; Seitz is a founder and stakeholder in Carrot Neurotechnology, a company that sells a vision brain game called ULTIMEYES. The other authors declare that they have no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Shawn Green, C., Bavelier, D., Kramer, A.F. et al. Improving Methodological Standards in Behavioral Interventions for Cognitive Enhancement. J Cogn Enhanc 3, 2–29 (2019). https://doi.org/10.1007/s41465-018-0115-y
- Cognitive enhancement
- Behavioral intervention methodology