Advertisement

Improving Methodological Standards in Behavioral Interventions for Cognitive Enhancement

  • C. Shawn GreenEmail author
  • Daphne Bavelier
  • Arthur F. Kramer
  • Sophia Vinogradov
  • Ulrich Ansorge
  • Karlene K. Ball
  • Ulrike Bingel
  • Jason M. Chein
  • Lorenza S. Colzato
  • Jerri D. Edwards
  • Andrea Facoetti
  • Adam Gazzaley
  • Susan E. Gathercole
  • Paolo Ghisletta
  • Simone Gori
  • Isabela Granic
  • Charles H. Hillman
  • Bernhard Hommel
  • Susanne M. Jaeggi
  • Philipp Kanske
  • Julia Karbach
  • Alan Kingstone
  • Matthias Kliegel
  • Torkel Klingberg
  • Simone Kühn
  • Dennis M. Levi
  • Richard E. Mayer
  • Anne Collins McLaughlin
  • Danielle S. McNamara
  • Martha Clare Morris
  • Mor Nahum
  • Nora S. Newcombe
  • Rogerio Panizzutti
  • Ruchika Shaurya Prakash
  • Albert Rizzo
  • Torsten Schubert
  • Aaron R. Seitz
  • Sarah J. Short
  • Ilina Singh
  • James D. Slotta
  • Tilo Strobach
  • Michael S. C. Thomas
  • Elizabeth Tipton
  • Xin Tong
  • Haley A. Vlach
  • Julie Loebach Wetherell
  • Anna Wexler
  • Claudia M. Witt
Review
First Online:

Abstract

There is substantial interest in the possibility that cognitive skills can be improved by dedicated behavioral training. Yet despite the large amount of work being conducted in this domain, there is not an explicit and widely agreed upon consensus around the best methodological practices. This document seeks to fill this gap. We start from the perspective that there are many types of studies that are important in this domain—e.g., feasibility, mechanistic, efficacy, and effectiveness. These studies have fundamentally different goals, and, as such, the best-practice methods to meet those goals will also differ. We thus make suggestions in topics ranging from the design and implementation of control groups, to reporting of results, to dissemination and communication, taking the perspective that the best practices are not necessarily uniform across all study types. We also explicitly recognize and discuss the fact that there are methodological issues around which we currently lack the theoretical and/or empirical foundation to determine best practices (e.g., as pertains to assessing participant expectations). For these, we suggest important routes forward, including greater interdisciplinary collaboration with individuals from domains that face related concerns. Our hope is that these recommendations will greatly increase the rate at which science in this domain advances.

Keywords

Cognitive enhancement Behavioral intervention methodology 
This is a preview of subscription content, log in to check access.

Notes

Funding

National Science Foundation (DRL-1641280) to Dr C. Shawn Green; Office of Naval Research grant to Dr. Daphne Bavelier.

Compliance with Ethical Standards

Conflict of Interest

The following authors have declared conflict(s) of interest. Bavelier is a founding partner and on the scientific advisory board of Akili Interactive, Boston; Vinogradov is a consultant for Posit Science Corp, Alkermes, Inc., and Mindstrong, Inc.; Ball owns stock in the Visual Awareness Research Group (formerly Visual Awareness, Inc.) and Posit Science, Inc., the companies that market the Useful Field of View Test and speed of processing training software (now the Double Decision exercise in BrainHQ), and is a member of the Posit Science Scientific Advisory Board; Gazzaley is a co-founder, scientific advisor, and BOD member for Akili Interactive Lab and has several patents filed at UCSF for video game enhancement technologies; Jaeggi has an indirect financial interest in the MIND Research Institute, Irvine, CA, whose interests are related to this work; Levi is a member of the Scientific Advisory Board of NovaSight; Morris is on the scientific advisory boards of Neurotrack and of the AARP Global Council on Brain Health; Nahum is a paid consultant for Posit Science; Panizzutti is the founder of NeuroForma LTDA, a company with a financial interest in computerized cognitive training; Seitz is a founder and stakeholder in Carrot Neurotechnology, a company that sells a vision brain game called ULTIMEYES. The other authors declare that they have no conflict of interest.

References

  1. Acosta, A., Adams, R.B., Jr., Albohn, D.N., Allard, E.S., Beek, T., Benning, S. D., … Zwaan, R. A. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917–928. doi: https://doi.org/10.1177/1745691616674458.
  2. Andrews, G. (1999). Efficacy, effectiveness and efficiency in mental health service delivery. The Australian and New Zealand Journal of Psychiatry, 33(3), 316–322.  https://doi.org/10.1046/j.1440-1614.1999.00581.x.CrossRefPubMedGoogle Scholar
  3. Anguera, J.A., Boccanfuso, J., Rintoul, J.L., Al-Hashimi, O., Faraji, F., Janowich, J., … Gazzaley, A. (2013). Video game training enhances cognitive control in older adults. Nature, 501(7465), 97–101. doi: https://doi.org/10.1038/nature12486.
  4. Au, J., Sheehan, E., Tsai, N., Duncan, G. J., Buschkuehl, M., & Jaeggi, S. M. (2015). Improving fluid intelligence with training on working memory: a meta-analysis. Psychonomic Bulletin & Review, 22(2), 366–377.  https://doi.org/10.3758/s13423-014-0699-x.CrossRefGoogle Scholar
  5. Ball, K., Berch, D. B., Helmers, K. F., Jobe, J. B., Leveck, M. D., Marsiske, M., … Group, A. S. (2002). Effects of cognitive training interventions with older adults: a randomized controlled trial. JAMA, 288(18), 2271–2281.Google Scholar
  6. Baniqued, P., Allen, C.M., Kranz, M.B., Johnson, K., Sipolins, A., Dickens, C., …, Kramer, A.F. (2015). Working memory, reasoning, and task switching training: transfer effects, limitations, and great expectations?. PLoS One. doi: https://doi.org/10.1371/journal.pone.0142169.
  7. Barnett, S. M., & Ceci, S. J. (2002). When and where do we apply what we learn?: A taxonomy for far transfer. Psychological Bulletin, 128(4), 612–637.CrossRefGoogle Scholar
  8. Barry, A. E., Szucs, L. E., Reyes, J. V., Ji, Q., Wilson, K. L., & Thompson, B. (2016). Failure to report effect sizes: the handling of quantitative results in published health education and behavior research. Health Education & Behavior, 43(5), 518–527.  https://doi.org/10.1177/1090198116669521.CrossRefGoogle Scholar
  9. Basak, C., Boot, W. R., Voss, M. W., & Kramer, A. F. (2008). Can training in a real-time strategy video game attenuate cognitive decline in older adults. Psychology and Aging, 23(4), 765–777.CrossRefGoogle Scholar
  10. Bavelier, D., & Davidson, R. J. (2013). Brain training: games to do you good. Nature, 494(7438), 425–426.  https://doi.org/10.1038/494425a.CrossRefPubMedGoogle Scholar
  11. Bayraktar, S. (2001). A meta-analysis of the effectiveness of computer-assisted instruction in science education. Journal of Research on Technology in Education, 34(2), 173–188.CrossRefGoogle Scholar
  12. Beaumont, J. L., Havlik, R., Cook, K. F., Hays, R. D., Wallner-Allen, K., Korper, S. P., …, Gershon, R. (2013). Norming plans for the NIH toolbox. Neurology, 80(11 Suppl 3), S87–92. doi: https://doi.org/10.1212/WNL.0b013e3182872e70.
  13. Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018). Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills. Psychological Bulletin, 144(1), 77–110.  https://doi.org/10.1037/bul0000130.CrossRefPubMedGoogle Scholar
  14. Biagianti, B., & Vinogradov, S. (2013). Computerized cognitive training targeting brain plasticity in schizophrenia. Progress in Brain Research, 207, 301–326.  https://doi.org/10.1016/B978-0-444-63327-9.00011-4.CrossRefPubMedGoogle Scholar
  15. Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology: why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8(4), 445–454.  https://doi.org/10.1177/1745691613491271.CrossRefPubMedGoogle Scholar
  16. Bryant, D. C. N., & Deluca, J. (2004). Objective measurement of cognitive fatigue in multiple sclerosis. Rehabilitation Psychology, 49(2), 114–122.CrossRefGoogle Scholar
  17. Carvalho, C., Caetano, J. M., Cunha, L., Rebouta, P., Kaptchuk, T. J., & Kirsch, I. (2016). Open-label placebo treatment in chronic low back pain: a randomized controlled trial. Pain, 157(12), 2766–2772.  https://doi.org/10.1097/j.pain.0000000000000700.CrossRefPubMedPubMedCentralGoogle Scholar
  18. Chen, L. H., & Lee, W. C. (2011). Two-way minimization: a novel treatment allocation method for small trials. PLoS One, 6(12), e28604.CrossRefGoogle Scholar
  19. Coburn, K. M., & Vevea, J. L. (2015). Publication bias as a function of study characteristics. Psychological Methods, 20(3), 310–330.  https://doi.org/10.1037/met0000046.CrossRefPubMedGoogle Scholar
  20. Colloca, L., & Benedetti, F. (2006). How prior experience shapes placebo analgesia. Pain, 124(1–2), 126–133.  https://doi.org/10.1016/j.pain.2006.04.005.CrossRefPubMedGoogle Scholar
  21. Colloca, L., Klinger, R., Flor, H., & Bingel, U. (2013). Placebo analgesia: psychological and neurobiological mechanisms. Pain, 154(4), 511–514.  https://doi.org/10.1016/j.pain.2013.02.002.CrossRefPubMedPubMedCentralGoogle Scholar
  22. Colzato, L. S., van den Wildenberg, W. P., & Hommel, B. (2014). Cognitive control and the COMT Val(1)(5)(8)Met polymorphism: genetic modulation of videogame training and transfer to task-switching efficiency. Psychological Research, 78(5), 670–678.  https://doi.org/10.1007/s00426-013-0514-8.CrossRefPubMedGoogle Scholar
  23. Davidson, R. J., & Dahl, C. J. (2017). Varieties of contemplative practice. JAMA Psychiatry, 74(2), 121–123.  https://doi.org/10.1001/jamapsychiatry.2016.3469.CrossRefPubMedGoogle Scholar
  24. Davidson, R. J., & Dahl, C. J. (2018). Outstanding challenges in scientific research on mindfulness and meditation. Perspectives on Psychological Science, 13(1), 62–65.  https://doi.org/10.1177/1745691617718358.CrossRefPubMedGoogle Scholar
  25. Deveau, J., Jaeggi, S. M., Zordan, V., Phung, C., & Seitz, A. R. (2014a). How to build better memory training games. Frontiers in Systems Neuroscience, 8, 243.  https://doi.org/10.3389/fnsys.2014.00243.CrossRefPubMedGoogle Scholar
  26. Deveau, J., Ozer, D. J., & Seitz, A. R. (2014b). Improved vision and on-field performance in baseball through perceptual learning. Current Biology, 24(4), R146–R147.  https://doi.org/10.1016/j.cub.2014.01.004.CrossRefPubMedGoogle Scholar
  27. Diao, D., Wright, J. M., Cundiff, D. K., & Gueyffier, F. (2012). Pharmacotherapy for mild hypertension. Cochrane Database of Systematic Reviews(8). doi: https://doi.org/10.1002/14651858.CD006742.pub2.
  28. Dweck, C. (2006). Mindset: the new psychology of success. New York: Random House.Google Scholar
  29. Eldridge, S. M., Lancaster, G. A., Campbell, M. J., Thabane, L., Hopewell, S., Coleman, C. L., & Bond, C. M. (2016). Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One, 11(3), e0150205.  https://doi.org/10.1371/journal.pone.0150205.CrossRefPubMedPubMedCentralGoogle Scholar
  30. Engle, R. W., Tuholski, S. W., Laughlin, J. E., & Conway, A. R. A. (1999). Working memory, short-term memory, and general fluid intelligence: a latent variable approach. Journal of Experimental Psychology: General, 128, 309–331.CrossRefGoogle Scholar
  31. Erickson, K. I., Voss, M. W., Prakash, R. S., Basak, C., Szabo, A., Chaddock, L., et al. (2011). Exercise training increases size of hippocampus and improves memory. Proceedings of the National Academy of Sciences of the United States of America, 108(7), 3017–3022.  https://doi.org/10.1073/pnas.1015950108.CrossRefPubMedPubMedCentralGoogle Scholar
  32. Fassler, M., Meissner, K., Kleijnen, J., Hrobjartsson, A., & Linde, K. (2015). A systematic review found no consistent difference in effect between more and less intensive placebo interventions. Journal of Clinical Epidemiology, 68(4), 442–451.  https://doi.org/10.1016/j.jclinepi.2014.11.018.CrossRefPubMedGoogle Scholar
  33. FEDERAL TRADE COMMISSION, (2016) Plaintiff, V.. Lumos Labs, Inc., a corporation d/b/a Lumosity and Kunal Sarkar, and Michael Scanlon, Individually and as officers of LumoS Labs, Inc. Defendants. , No. Case No. 3:16-cv-00001-sk (United StateS DistricT Court for the Northern District of California, San Francisco Division.Google Scholar
  34. Fergusson, D., Glass, K. C., Waring, D., & Shapiro, S. (2004). Turning a blind eye: the success of blinding reported in a random sample of randomised placebo controlled trials. BMJ, 328(7437), 432.  https://doi.org/10.1136/bmj.37952.631667.EE.CrossRefPubMedPubMedCentralGoogle Scholar
  35. Foroughi, C. K., Monfort, S. S., Paczynski, M., McKnight, P. E., & Greenwood, P. M. (2016). Placebo effects in cognitive training. PNAS, 113, 7470–7474.  https://doi.org/10.1073/pnas.1601243113.CrossRefPubMedGoogle Scholar
  36. Franceschini, S., Gori, S., Ruffino, M., Viola, S., Molteni, M., & Facoetti, A. (2013). Action video games make dyslexic children read better. Current Biology, 23(6), 462–466.  https://doi.org/10.1016/j.cub.2013.01.044.CrossRefPubMedGoogle Scholar
  37. Fritz, J. M., & Cleland, J. (2003). Effectiveness versus efficacy: more than a debate over language. The Journal of Orthopaedic and Sports Physical Therapy, 33(4), 163–165.  https://doi.org/10.2519/jospt.2003.33.4.163.CrossRefPubMedGoogle Scholar
  38. Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention. Nature, 423(6939), 534–537.CrossRefGoogle Scholar
  39. Green, C. S., & Bavelier, D. (2012). Learning, attentional control and action video games. Current Biology, 22, R197–R206.CrossRefGoogle Scholar
  40. Green, C. S., Pouget, A., & Bavelier, D. (2010). Improved probabilistic inference as a general mechanism for learning with action video games. Current Biology, 23, 1573–1579.CrossRefGoogle Scholar
  41. Green, C. S., Strobach, T., & Schubert, T. (2014). On methodological standards in training and transfer experiments. Psychological Research, 78(6), 756–772.  https://doi.org/10.1007/s00426-013-0535-3.CrossRefPubMedGoogle Scholar
  42. Greitemeyer, T., Osswald, S., & Brauer, M. (2010). Playing prosocial video games increases empathy and decreases schadenfreude. Emotion, 10(6), 796–802.  https://doi.org/10.1037/a0020194.CrossRefPubMedGoogle Scholar
  43. Hallock, H., Collins, D., Lampit, A., Deol, K., Fleming, J., & Valenzuela, M. (2016). Cognitive training for post-acute traumatic brain injury: a systematic review and meta-analysis. Frontiers in Human Neuroscience, 10, 537.  https://doi.org/10.3389/fnhum.2016.00537.CrossRefPubMedPubMedCentralGoogle Scholar
  44. Hawes, Z., Moss, J., Caswell, B., Naqvi, S., & MacKinnon, S. (2017). Enhancing children’s spatial and numerical skills through a dynamic spatial approach to early geometry instruction: effects of a 32-week intervention. Cognition and Instruction, 35(3), 236–264.CrossRefGoogle Scholar
  45. Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), e1002106.  https://doi.org/10.1371/journal.pbio.1002106.CrossRefPubMedPubMedCentralGoogle Scholar
  46. Health, N. I. o. (2014). Notice of Revised NIH Definition of “Clinical Trial”. (NOT-OD-15-015). Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-015.html. Accessed 1 Nov 2018.
  47. Hendershot, T., Pan, H., Haines, J., Harlan, W. R., Marazita, M. L., McCarty, C. A., …, Hamilton, C. M.. (2015). Using the PhenX toolkit to add standard measures to a study. Curr Protoc Hum Genet, 86, 1 21 21–17.  https://doi.org/10.1002/0471142905.hg0121s86.
  48. Hillman, C. H., Erickson, K. I., & Kramer, A. F. (2008). Be smart, exercise your heart: exercise effects on brain and cognition. Nature Reviews Neuroscience, 9, 58–65.CrossRefGoogle Scholar
  49. Hills, M., & Armitage, P. (2004). The two-period cross-over clinical trial. 1979. British Journal of Clinical Pharmacology, 58(7), S703–S716; discussion S717-709.  https://doi.org/10.1111/j.1365-2125.2004.02275.x.CrossRefPubMedPubMedCentralGoogle Scholar
  50. Hodes, R. J., Insel, T. R., Landis, S. C., & Research, N. I. H. B. f. N. (2013). The NIH toolbox: setting a standard for biomedical research. Neurology, 80(11 Suppl 3), S1.  https://doi.org/10.1212/WNL.0b013e3182872e90.CrossRefPubMedPubMedCentralGoogle Scholar
  51. Holtzer, R., Shuman, M., Mahoney, J. R., Lipton, R., & Verghese, J. (2011). Cognitive fatigue defined in the context of attention networks. Neuropsychology, Development, and Cognition. Section B, Aging, Neuropsychology and Cognition, 18(1), 108–128.  https://doi.org/10.1080/13825585.2010.517826.CrossRefPubMedGoogle Scholar
  52. Hommel, B., Colzato, L. S., Fischer, R., & Christoffels, I. K. (2011). Bilingualism and creativity: benefits in convergent thinking come with losses in divergent thinking. Frontiers in Psychology, 2, 273.  https://doi.org/10.3389/fpsyg.2011.00273.CrossRefPubMedPubMedCentralGoogle Scholar
  53. Howard, J. (2016). Do brain-training exercises really work? CNN. Retrieved from https://www.cnn.com/2016/10/20/health/brain-training-exercises/index.html. Accessed 1 Nov 2018.
  54. Hrobjartsson, A., Forfang, E., Haahr, M. T., Als-Nielsen, B., & Brorson, S. (2007). Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding. International Journal of Epidemiology, 36(3), 654–663.  https://doi.org/10.1093/ije/dym020.CrossRefPubMedGoogle Scholar
  55. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008). Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences of the United States of America, 105(19), 6829–6833.CrossRefGoogle Scholar
  56. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Shah, P. (2011). Short- and long-term benefits of cognitive training. Proceedings of the National Academy of Sciences of the United States of America, 108, 10081–10086.CrossRefGoogle Scholar
  57. Jones, R. N., Marsiske, M., Ball, K., Rebok, G., Willis, S. L., Morris, J. N., & Tennstedt, S. L. (2013). The ACTIVE cognitive training interventions and trajectories of performance among older adults. Journal of Aging and Health, 25(8 Suppl), 186S–208S.  https://doi.org/10.1177/0898264312461938.CrossRefPubMedGoogle Scholar
  58. Kaptchuk, T. J. (2001). The double-blind, randomized, placebo-controlled trial: gold standard or golden calf? Journal of Clinical Epidemiology, 54(6), 541–549.CrossRefGoogle Scholar
  59. Kaptchuk, T. J., & Miller, F. G. (2015). Placebo effects in medicine. The New England Journal of Medicine, 373(1), 8–9.  https://doi.org/10.1056/NEJMp1504023.CrossRefPubMedGoogle Scholar
  60. Kaptchuk, T. J., Friedlander, E., Kelley, J. M., Sanchez, M. N., Kokkotou, E., Singer, J. P., et al. (2010). Placebos without deception: a randomized controlled trial in irritable bowel syndrome. PLoS One, 5(12), e15591.  https://doi.org/10.1371/journal.pone.0015591.CrossRefPubMedPubMedCentralGoogle Scholar
  61. Karbach, J., & Unger, K. (2014). Executive control training from middle childhood to adolescence. Frontiers in Psychology, 5, 390.  https://doi.org/10.3389/fpsyg.2014.00390.CrossRefPubMedPubMedCentralGoogle Scholar
  62. Katz, B., Jaeggi, S. M., Buschkuehl, M., Shah, P., & Jonides, J. (2018). The effect of monetary compensation on cognitive training outcomes. Learning and Motivation, 63(77–90).Google Scholar
  63. Kelley, J. M., Kaptchuk, T. J., Cusin, C., Lipkin, S., & Fava, M. (2012). Open-label placebo for major depressive disorder: a pilot randomized controlled trial. Psychotherapy and Psychosomatics, 81(5), 312–314.  https://doi.org/10.1159/000337053.CrossRefPubMedGoogle Scholar
  64. Kirsch, I. (2005). Placebo psychotherapy: synonym or oxymoron? Journal of Clinical Psychology, 61(7), 791–803.  https://doi.org/10.1002/jclp.20126.CrossRefPubMedGoogle Scholar
  65. Klingberg, T., Fernell, E., Olesen, P. J., Johnson, M., Gustafsson, P., Dahlstrom, K., et al. (2005). Computerized training of working memory in children with ADHD—a randomized, controlled trial. J Am Acad Child Adolesc Psychiatry, 44(2), 177–186.CrossRefGoogle Scholar
  66. Kolahi, J., Bang, H., & Park, J. (2009). Towards a proposal for assessment of blinding success in clinical trials: up-to-date review. Community Dentistry and Oral Epidemiology, 37(6), 477–484.  https://doi.org/10.1111/j.1600-0528.2009.00494.x.CrossRefPubMedPubMedCentralGoogle Scholar
  67. Kramer, A. F., Larish, J., & Strayer, D. L. (1995). Training for attentional control in dual-task settings: a comparison of young and old adults. Journal of Experimental Psychology: Applied, 1, 50–76.Google Scholar
  68. Lutz, A., Jha, A. P., Dunne, J. D., & Saron, C. D. (2015). Investigating the phenomenological matrix of mindfulness-related practices from a neurocognitive perspective. The American Psychologist, 70(7), 632–658.  https://doi.org/10.1037/a0039585.CrossRefPubMedPubMedCentralGoogle Scholar
  69. Marchand, E., Stice, E., Rohde, P., & Becker, C. B. (2011). Moving from efficacy to effectiveness trials in prevention research. Behaviour Research and Therapy, 49(1), 32–41.  https://doi.org/10.1016/j.brat.2010.10.008.CrossRefPubMedGoogle Scholar
  70. Mayer, R. E. (Ed.). (2014). Computer games for learning: an evidence-based approach. Cambridge: MIT Press.Google Scholar
  71. Melby-Lervag, M., & Hulme, C. (2013). Is working memory training effective? A meta-analytic review. Developmental Psychology, 49(2), 270–291.CrossRefGoogle Scholar
  72. Merzenich, M. M., Nahum, M., & Van Vleet, T. M. (2013). Neuroplasticity: introduction. Progress in Brain Research, 207, xxi–xxvi.  https://doi.org/10.1016/B978-0-444-63327-9.10000-1.CrossRefPubMedGoogle Scholar
  73. Mohr, D. C., Schueller, S. M., Riley, W. T., Brown, C. H., Cuijpers, P., Duan, N., et al. (2015). Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. Journal of Medical Internet Research, 17(7), e166.  https://doi.org/10.2196/jmir.4391.CrossRefPubMedPubMedCentralGoogle Scholar
  74. Morey, R. D., Romeijn, J.-W., & Rouder, J. N. (2016). The philosophy of Bayes factors and the quantification of statistical evidence. Journal of Mathematical Psychology, 72, 6–18.CrossRefGoogle Scholar
  75. Nahum, M., Lee, H., & Merzenich, M. M. (2013). Principles of neuroplasticity-based rehabilitation. Progress in Brain Research, 207, 141–171.  https://doi.org/10.1016/B978-0-444-63327-9.00009-6.CrossRefPubMedGoogle Scholar
  76. Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: investigating participant demand characteristics. The Journal of General Psychology, 135(2), 151–165.CrossRefGoogle Scholar
  77. Noah, T., Yaacov, S., & Mayo, R. (2018). When both the original study and its failed replication are correct: feeling observed eliminates the facial-feedback effect. Journal of Personality and Social Psychology, 114(5), 657–664.CrossRefGoogle Scholar
  78. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017). The preregistration revolution. OSF Preprints. doi: https://doi.org/10.17605/OSF.IO/2DXU5.
  79. O’Leary, K. D., Rosenbaum, A., & Hughes, P. C. (1978). Direct and systematic replication: a rejoinder. Journal of Abnormal Child Psychology, 6(3), 295–297.CrossRefGoogle Scholar
  80. Onken, L. S., Carroll, K. M., Shoham, V., Cuthbert, B. N., & Riddle, M. (2014). Reenvisioning clinical science: unifying the discipline to improve the public health. Clinical Psychological Science: A Journal of the Association for Psychological Science, 2(1), 22–34.  https://doi.org/10.1177/2167702613497932.CrossRefGoogle Scholar
  81. Open Science, C. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657–660.  https://doi.org/10.1177/1745691612462588.CrossRefGoogle Scholar
  82. Orne, M. T. (1962). On the social psychology of the psychological experiment: with particular reference to demand characteristics and their implications. American Psychologist, 17, 776–783.CrossRefGoogle Scholar
  83. Owen, A. M., Hampshire, A., Grahn, J. A., Stenton, R., Dajani, S., Burns, A. S., … Ballard, C. G. (2010). Putting brain training to the test. Nature, 465(7299), 775–778.Google Scholar
  84. Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536.  https://doi.org/10.1177/1745691612463401.CrossRefPubMedGoogle Scholar
  85. Pek, J., & Flora, D. B. (2017). Reporting effect sizes in original psychological research: a discussion and tutorial. Psychological Methods. doi: https://doi.org/10.1037/met0000126.
  86. Prakash, R. S., De Leon, A. A., Patterson, B., Schirda, B. L., & Janssen, A. L. (2014). Mindfulness and the aging brain: a proposed paradigm shift. Frontiers in Aging Neuroscience, 6, 120.  https://doi.org/10.3389/fnagi.2014.00120.CrossRefPubMedPubMedCentralGoogle Scholar
  87. Rebok, G. W., Ball, K., Guey, L. T., Jones, R. N., Kim, H. Y., King, J. W., … Group, A. S. (2014). Ten-year effects of the advanced cognitive training for independent and vital elderly cognitive training trial on cognition and everyday functioning in older adults. Journal of the American Geriatrics Society, 62(1), 16–24. doi: https://doi.org/10.1111/jgs.12607.
  88. Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., … Engle, R. W. (2013). No evidence of intelligence improvement after working memory training: a randomized, placebo-controlled study. Journal of Experimental Psychology: General, 142(2), 359–379.Google Scholar
  89. Redick, T. S., Shipstead, Z., Wiemers, E. A., Melby-Lervag, M., & Hulme, C. (2015). What’s working in working memory training? An educational perspective. Educational Psychology Review, 27(4), 617–633.  https://doi.org/10.1007/s10648-015-9314-6.CrossRefPubMedPubMedCentralGoogle Scholar
  90. Roberts, G., Quach, J., Spencer-Smith, M., Anderson, P. J., Gathercole, S., Gold, L., … Wake, M. (2016). Academic outcomes 2 years after working memory training for children with low working memory: a randomized clinical trial. JAMA Pediatrics, 170(5), e154568. doi: https://doi.org/10.1001/jamapediatrics.2015.4568.
  91. Rohde, T. E., & Thompson, L. A. (2007). Predicting academic achievement with cognitive ability. Intelligence, 35(1), 83–92.CrossRefGoogle Scholar
  92. Rosenthal, R., & Jacobson, L. (1968). Pygmalion in the classroom: teacher expectation and pupils’ intellectual development. New York: Holt, Rinehart and Winston.CrossRefGoogle Scholar
  93. Ross, L. A., Edwards, J. D., O’Connor, M. L., Ball, K. K., Wadley, V. G., & Vance, D. E. (2016). The transfer of cognitive speed of processing training to older adults’ driving mobility across 5 years. The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 71(1), 87–97.  https://doi.org/10.1093/geronb/gbv022.CrossRefPubMedGoogle Scholar
  94. Rotello, C. M., Heit, E., & Dube, C. (2015). When more data steer us wrong: replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin & Review, 22(4), 944–954.  https://doi.org/10.3758/s13423-014-0759-2.CrossRefGoogle Scholar
  95. Rothbaum, B. O., Price, M., Jovanovic, T., Norrholm, S. D., Gerardi, M., Dunlop, B., … Ressler, K. J. (2014). A randomized, double-blind evaluation of D-cycloserine or alprazolam combined with virtual reality exposure therapy for posttraumatic stress disorder in Iraq and Afghanistan War veterans. The American Journal of Psychiatry, 171(6), 640–648. doi: https://doi.org/10.1176/appi.ajp.2014.13121625.
  96. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237.  https://doi.org/10.3758/PBR.16.2.225.CrossRefGoogle Scholar
  97. Rubin, M. (2016). The Perceived Awareness of the Research Hypothesis Scale: assessing the influence of demand characteristics. In.Google Scholar
  98. Rutherford, B. R., Sneed, J. R., & Roose, S. P. (2009). Does study design influence outcome?. The effects of placebo control and treatment duration in antidepressant trials. Psychotherapy and Psychosomatics, 78(3), 172–181.  https://doi.org/10.1159/000209348.CrossRefPubMedPubMedCentralGoogle Scholar
  99. Saghaei, M. (2011). An overview of randomization and minimization programs for randomized clinical trials. Journal of Medical Signals and Sensors, 1(1), 55–61.PubMedPubMedCentralGoogle Scholar
  100. Sandler, A. D., & Bodfish, J. W. (2008). Open-label use of placebos in the treatment of ADHD: a pilot study. Child: Care, Health and Development, 34(1), 104–110.  https://doi.org/10.1111/j.1365-2214.2007.00797.x.CrossRefGoogle Scholar
  101. Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: introduction. Educational Psychologist, 39(4), 199–201.CrossRefGoogle Scholar
  102. Schellenberg, E. G. (2004). Music lessions enhance IQ. Psychological Science, 15(8), 511–514.CrossRefGoogle Scholar
  103. Schlickum, M. K., Hedman, L., Enochsson, L., Kjellin, A., & Fellander-Tsai, L. (2009). Systematic video game training in surgical novices improves performance in virtual reality endoscopic surgical simulators: a prospective randomized study. World Journal of Surgery, 33(11), 2360–2367.CrossRefGoogle Scholar
  104. Schmiedek, F., Lövdén, M., & Lindenberger, U. (2010). Hundred days of cognitive training enhance broad abilities in adulthood: findings from the COGITO study. Frontiers in Aging Neuroscience, 2.Google Scholar
  105. Schönbrodt, F. D., Wagenmakers, E. J., Zehetleitner, M., & Perugini, M. (2017). Sequential hypothesis testing with Bayes factors: efficiently testing mean differences. Psychological Methods, 22(2), 322–339.CrossRefGoogle Scholar
  106. Schubert, T., & Strobach, T. (2012). Video game experience and optimized executive control skills—on false positives and false negatives: reply to Boot and Simons (2012). Acta Psychologica, 141(2), 278–280.CrossRefGoogle Scholar
  107. Schulz, K. F., Chalmers, I., & Altman, D. G. (2002). The landscape and lexicon of blinding in randomized trials. Annals of Internal Medicine, 136(3), 254–259.CrossRefGoogle Scholar
  108. Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32.CrossRefGoogle Scholar
  109. Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training effective? Psychological Bulletin, 138(4), 623–654.CrossRefGoogle Scholar
  110. Sidman, M. (1966). Tactics of scientific research: evaluating experimental data in psychology. Oxford: Basic Books.Google Scholar
  111. Simons, D. J., Boot, W. R., Charness, N., Gathercole, S. E., Chabris, C. F., Hambrick, D. Z., & Stine-Morrow, E. A. (2016). Do “Brain-Training” programs work? Psychological Science in the Public Interest, 17(3), 103–186.  https://doi.org/10.1177/1529100616661983.CrossRefPubMedGoogle Scholar
  112. Singal, A. G., Higgins, P. D., & Waljee, A. K. (2014). A primer on effectiveness and efficacy trials. Clinical Translational Gastroenterology, 5, e45.  https://doi.org/10.1038/ctg.2013.13.CrossRefPubMedGoogle Scholar
  113. Smith, G. E., Housen, P., Yaffe, K., Ruff, R., Kennison, R. F., Mahncke, H. W., & Zelinski, E. M. (2009). A cognitive training program based on principles of brain plasticity: results from Improvement in Memory with Plasticity-based Adaptive cognitive Training (IMPACT) study. Journal of the American Geriatrics Society, 57(4), 594–603.CrossRefGoogle Scholar
  114. Stieff, M., & Uttal, D. (2015). How much can spatial training improve STEM achievement. Educational Psychology Review, 27(4), 607–615.CrossRefGoogle Scholar
  115. Stierlin, A. S., Herder, K., Helmbrecht, M. J., Prinz, S., Walendzik, J., Holzmann, M., … Kilian, R. (2014). Effectiveness and efficiency of integrated mental health care programmes in Germany: study protocol of an observational controlled trial. BMC Psychiatry, 14, 163. doi: https://doi.org/10.1186/1471-244X-14-163.
  116. Strack, F., Martin, L., & Stepper, S. (1988). Inhibiting and facilitating conditions of the human smile: a nonobtrusive test of the facial feedback hypothesis. Journal of Personality and Social Psychology, 54(5), 768–777.CrossRefGoogle Scholar
  117. Strobach, T., & Karbach, J. (Eds.). (2016). Cognitive training: an overview of features and applications. New York: Springer.Google Scholar
  118. Strobach, T., Frensch, P. A., & Schubert, T. (2012). Video game practice optimizes executive control skills in dual-task and task switching situations. Acta Psychologica, 140(1), 13–24.CrossRefGoogle Scholar
  119. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59–71.  https://doi.org/10.1177/1745691613514450.CrossRefPubMedGoogle Scholar
  120. Subramaniam, K., Luks, T. L., Garrett, C., Chung, C., Fisher, M., Nagarajan, S., & Vinogradov, S. (2014). Intensive cognitive training in schizophrenia enhances working memory and associated prefrontal cortical efficiency in a manner that drives long-term functional gains. NeuroImage, 99, 281–292.  https://doi.org/10.1016/j.neuroimage.2014.05.057.CrossRefPubMedPubMedCentralGoogle Scholar
  121. Sullivan, G. M., & Feinn, R. (2012). Using effect size-or why the P value is not enough. Journal of Graduate Medical Education, 4(3), 279–282.  https://doi.org/10.4300/JGME-D-12-00156.1.CrossRefPubMedPubMedCentralGoogle Scholar
  122. Tang, Y. Y., Ma, Y., Wang, J., Fan, Y., Feng, S., Lu, Q., et al. (2007). Short-term meditation training improves attention and self-regulation. Proceedings of the National Academy of Sciences of the United States of America, 104(43), 17152–17156.  https://doi.org/10.1073/pnas.0707678104.CrossRefPubMedPubMedCentralGoogle Scholar
  123. Taves, D. R. (1974). Minimization: a new method of assigning patients to treatment and control groups. Clinical Pharmacology and Therapeutics, 15, 443–453.CrossRefGoogle Scholar
  124. Terlecki, M. S., Newcombe, N. S., & Little, M. (2008). Durable and generalized effects of spatial experience on mental rotation: gender differences in growth patterns. Applied Cognitive Psychology, 22, 996–1013.CrossRefGoogle Scholar
  125. Tickle-Degnen, L. (2013). Nuts and bolts of conducting feasibility studies. The American Journal of Occupational Therapy, 67(2), 171–176.  https://doi.org/10.5014/ajot.2013.006270.CrossRefPubMedPubMedCentralGoogle Scholar
  126. Tsai, N., Buschkuehl, M., Kamarsu, S., Shah, P., Jonides, J., & Jaeggi, S. M. (2018). (Un)Great expectations: the role of placebo effects in cognitive training. Journal of Applied Research in Memory and Cognition, 7(4), 564–573.  https://doi.org/10.1016/j.jarmac.2018.06.001.
  127. Valdes, E.G., Andel, R., Lister, J.J., Gamaldo, A., & Edwards, J.D. (2017). Can cognitive speed of processing training improve everyday functioning among older adults with psychometrically defined mild cognitive impairment? Journal of Aging and Health, 898264317738828.  https://doi.org/10.1177/0898264317738828.
  128. Van Dam, N. T., van Vugt, M. K., Vago, D. R., Schmalzl, L., Saron, C. D., Olendzki, A., et al. (2018). Mind the hype: a critical evaluation and prescriptive agenda for research on mindfulness and meditation. Perspectives on Psychological Science, 13(1), 36–61.  https://doi.org/10.1177/1745691617709589.CrossRefPubMedGoogle Scholar
  129. Voss, M.W., Prakash, R.S., Erickson, K.I., Basak, C., Chaddock, L., Kim, J. S., … Kramer, A. F. (2010). Plasticity of brain networks in a randomized intervention trial of exercise training in older adults. Frontiers in Aging Neuroscience, 2.  https://doi.org/10.3389/fnagi.2010.00032.
  130. Voudouris, N. J., Peck, C. L., & Coleman, G. (1985). Conditioned placebo responses. Journal of Personality and Social Psychology, 48(1), 47–53.CrossRefGoogle Scholar
  131. Walton, A.G. (2016). Do brain training games work, or is it the placebo effect? Forbes.com. Retrieved from https://www.forbes.com/sites/alicegwalton/2016/06/21/does-brain-training-work-or-is-it-all-placebo/#3b654dc67497. Accessed 1 Nov 2018.
  132. Weintraub, S., Dikmen, S.S., Heaton, R.K., Tulsky, D.S., Zelazo, P.D., Bauer, P.J., … Gershon, R.C. (2013). Cognition assessment using the NIH toolbox. Neurology, 80(11 Suppl 3), S54–64.  https://doi.org/10.1212/WNL.0b013e3182872ded.
  133. Wexler, B.E., Iseli, M., Leon, S., Zaggle, W., Rush, C., Goodman, A., … Bo, E. (2016). Cognitive priming and cognitive training: immediate and far transfer to academic skills in children. Scientific Reports, 6, 32859. doi: https://doi.org/10.1038/srep32859.
  134. Whitehead, A. L., Sully, B. G., & Campbell, M. J. (2014). Pilot and feasibility studies: is there a difference from each other and from a randomised controlled trial? Contemporary Clinical Trials, 38(1), 130–133.  https://doi.org/10.1016/j.cct.2014.04.001.CrossRefPubMedGoogle Scholar
  135. Whitlock, L. A., McLaughlin, A. C., & Allaire, J. C. (2012). Individual differences in response to cognitive training: using a multi-modal, attentionally demanding game-based intervention for older adults. Computers in Human Behavior, 28(4), 1091–1096.CrossRefGoogle Scholar
  136. Willis, S.L., Tennstedt, S.L., Marsiske, M., Ball, K., Elias, J., Koepke, K.M., … Group, A.S. (2006). Long-term effects of cognitive training on everyday functional outcomes in older adults. JAMA, 296(23), 2805–2814.Google Scholar
  137. Wolfe, J. M., & Kanwisher, N. G. (2018). Not your parent’s NIH clinical trial. Nature Human Behaviour, 2, 107–109.CrossRefGoogle Scholar
  138. Wright, R., Thompson, W. L., Ganis, G., Newcombe, N. S., & Kosslyn, S. M. (2008). Training generalized spatial skills. Psychonomic Bulletin and Review, 15(4), 763–771.CrossRefGoogle Scholar
  139. Zhao, W., Hill, M.D., & Palesch, Y. (2012). Minimal sufficient balance—a new strategy to balance baseline covariates and preserve randomness of treatment allocation. Statistical Methods in Medical Research  https://doi.org/10.1177/0962280212436447.
  140. Zwaan, R.A., Etz, A., Lucas, R.E., & Donnellan, M. B. (2017). Making replication mainstream. The Behavioral and Brain Sciences, 1–50.  https://doi.org/10.1017/S0140525X17001972.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • C. Shawn Green
    • 1
    Email author
  • Daphne Bavelier
    • 2
  • Arthur F. Kramer
    • 3
    • 4
  • Sophia Vinogradov
    • 5
    • 6
    • 7
  • Ulrich Ansorge
    • 8
  • Karlene K. Ball
    • 9
  • Ulrike Bingel
    • 10
  • Jason M. Chein
    • 11
  • Lorenza S. Colzato
    • 12
    • 13
  • Jerri D. Edwards
    • 14
  • Andrea Facoetti
    • 15
  • Adam Gazzaley
    • 16
  • Susan E. Gathercole
    • 17
  • Paolo Ghisletta
    • 18
    • 19
    • 20
  • Simone Gori
    • 21
  • Isabela Granic
    • 22
  • Charles H. Hillman
    • 23
  • Bernhard Hommel
    • 24
  • Susanne M. Jaeggi
    • 25
  • Philipp Kanske
    • 26
  • Julia Karbach
    • 27
    • 28
  • Alan Kingstone
    • 29
  • Matthias Kliegel
    • 30
  • Torkel Klingberg
    • 31
  • Simone Kühn
    • 32
  • Dennis M. Levi
    • 33
  • Richard E. Mayer
    • 34
  • Anne Collins McLaughlin
    • 35
  • Danielle S. McNamara
    • 36
  • Martha Clare Morris
    • 37
  • Mor Nahum
    • 38
  • Nora S. Newcombe
    • 39
  • Rogerio Panizzutti
    • 40
    • 41
  • Ruchika Shaurya Prakash
    • 42
  • Albert Rizzo
    • 43
  • Torsten Schubert
    • 44
  • Aaron R. Seitz
    • 45
  • Sarah J. Short
    • 46
  • Ilina Singh
    • 47
  • James D. Slotta
    • 48
  • Tilo Strobach
    • 49
  • Michael S. C. Thomas
    • 50
  • Elizabeth Tipton
    • 51
    • 52
  • Xin Tong
    • 53
  • Haley A. Vlach
    • 54
  • Julie Loebach Wetherell
    • 55
    • 56
  • Anna Wexler
    • 57
  • Claudia M. Witt
    • 58
  1. 1.Department of PsychologyUniversity of WisconsinMadisonUSA
  2. 2.Psychology and Education Sciences (FPSE) & Campus BiotechUniversity of GenevaGenevaSwitzerland
  3. 3.Department of PsychologyNortheastern UniversityBostonUSA
  4. 4.Beckman InstituteUniversity of Illinois at Urbana-ChampaignChampaignUSA
  5. 5.Department of PsychiatryUniversity of MinnesotaMinneapolisUSA
  6. 6.University of Minnesota Medical SchoolMinneapolisUSA
  7. 7.University of Minnesota PhysiciansMinneapolisUSA
  8. 8.Faculty of PsychologyUniversity of ViennaViennaAustria
  9. 9.Center for Research on Applied Gerontology & Department of PsychologyUniversity of Alabama at BirminghamBirminghamUSA
  10. 10.Department of NeurologyUniversity Hospital Essen, University Duisburg-EssenDuisburgGermany
  11. 11.Department of PsychologyTemple UniversityPhiladelphiaUSA
  12. 12.Cognitive Psychology Unit & Leiden Institute for Brain and CognitionLeiden UniversityLeidenNetherlands
  13. 13.Department of Cognitive Psychology, Institute of Cognitive Neuroscience, Faculty of PsychologyRuhr University BochumBochumGermany
  14. 14.Department of Psychiatry and Behavioral NeurosciencesUniversity of South FloridaTampaUSA
  15. 15.Developmental and Cognitive Neuroscience Laboratory, Department of General PsychologyUniversity of PaduaPaduaItaly
  16. 16.Departments of Neurology, Physiology & PsychiatryUniversity of CaliforniaSan FranciscoUSA
  17. 17.MRC Cognition and Brain Sciences UnitUniversity of CambridgeCambridgeEngland
  18. 18.Faculty of Psychology and Educational SciencesUniversity of GenevaGenevaSwitzerland
  19. 19.Swiss Distance Learning UniversityBrigSwitzerland
  20. 20.Swiss National Centre of Competence in Research LIVES – Overcoming vulnerability: Life course perspectivesUniversity of Lausanne, Laussane, Switzerland; University of GenevaGenevaSwitzerland
  21. 21.Department of Human and Social SciencesUniversity of BergamoBergamoItaly
  22. 22.Behavioural Science InstituteRadboud UniversityNijmegenNetherlands
  23. 23.Department of Psychology & Department of Physical Therapy, Movement & Rehabilitation SciencesNortheastern UniversityBostonUSA
  24. 24.Institute of PsychologyLeiden UniversityLeidenNetherlands
  25. 25.School of Education and School of Social Sciences, Department of Cognitive SciencesUniversity of CaliforniaIrvineUSA
  26. 26.Faculty of Psychology, Clinical Psychology and Behavioral NeuroscienceTechnische Universität DresdenDresdenGermany
  27. 27.Department of PsychologyUniversity of Koblenz-LandauLandauGermany
  28. 28.Center for Research on Individual Development and Adaptive Education of Children at Risk (IDeA)FrankfurtGermany
  29. 29.Department of PsychologyUniversity of British ColumbiaVancouverCanada
  30. 30.Department of Psychology & Center for the Interdisciplinary Study of Gerontology and VulnerabilityUniversity of GenevaGenevaSwitzerland
  31. 31.Department of NeuroscienceKarolinska InstitutetSolnaSweden
  32. 32.Clinic for Psychiatry and PsychotherapyUniversity Clinic Hamburg-EppendorfHamburgGermany
  33. 33.School of Optometry & Graduate Group in Vision ScienceUniversity of CaliforniaBerkeleyUSA
  34. 34.Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraUSA
  35. 35.Department of PsychologyNorth Carolina State UniversityRaleighUSA
  36. 36.Department of PsychologyArizona State UniversityTempeUSA
  37. 37.Rush Institute for Healthy AgingRush UniversityChicagoUSA
  38. 38.School of Occupational Therapy, Faculty of MedicineHebrew University of JerusalemJerusalemIsrael
  39. 39.Department of PsychologyTemple UniversityPhiladelphiaUSA
  40. 40.Instituto de Ciencias Biomedicas & Instituto de PsiquiatriaUniversidade Federal do Rio de JaneiroRio de JaneiroBrazil
  41. 41.Global Brain Health InstituteTrinity College DublinDublinIreland
  42. 42.Department of PsychologyThe Ohio State UniversityColumbusUSA
  43. 43.Institute for Creative TechnologiesUniversity of Southern CaliforniaLos AngelesUSA
  44. 44.Institute of PsychologyMartin-Luther-University Halle-WittenbergHalleGermany
  45. 45.Department of Psychology & Brain Game CenterUniversity of CaliforniaRiversideUSA
  46. 46.Department of Educational Psychology & Center for Healthy MindsUniversity of WisconsinMadisonUSA
  47. 47.Department of Psychiatry and Wellcome Centre for Ethics and the HumanitiesUniversity of OxfordOxfordUK
  48. 48.Department of Curriculum, Teaching and LearningBoston CollegeChestnut HillUSA
  49. 49.Department of PsychologyMedical School HamburgHamburgGermany
  50. 50.Centre for Educational Neuroscience, Department of Psychological SciencesBirkbeck, University of LondonLondonUK
  51. 51.Teachers CollegeColumbia UniversityNew YorkUSA
  52. 52.Department of Statistics and Institute for Policy ResearchNorthwestern UniversityEvanstonUSA
  53. 53.Department of PsychologyUniversity of VirginiaCharlottesvilleUSA
  54. 54.Department of Educational PsychologyUniversity of WisconsinMadisonUSA
  55. 55.VA San Diego Healthcare SystemSan DiegoUSA
  56. 56.University of California-San DiegoSan DiegoUSA
  57. 57.Department of Medical Ethics & Health PolicyUniversity of PennsylvaniaPhiladelphiaUSA
  58. 58.Institute for Complementary and Integrative MedicineUniversity Hospital and University of ZurichZürichSwitzerland

Personalised recommendations

Cite article

Buy options