Skip to main content
SpringerLink
Log in

A New Framework of Design and Continuous Evaluation to Improve Brain Training

  • Theory & Hypothesis
  • Published:
Journal of Cognitive Enhancement Aims and scope Submit manuscript

Abstract

Effective teaching critically relies upon effective evaluation because evaluation is required to gain understanding of existing abilities and, in turn, determine learning outcomes. Methods of effective evaluation are surprisingly elusive in many fields, and this limits our understanding of which training methods are truly effective. In the present article, issues of effective evaluation are discussed in the context of “brain training,” an exciting but much criticized field. Problems in test validity in the field of brain training parallel those in many other fields, such as deficiencies in test reliability, teaching to the test, expectation effects, as well as statistical rigor. Here we review these issues and discuss how commonalities between the goals of evaluation and adaptive training procedures suggest a new paradigm that synthesizes evaluation and training. We suggest that continuous evaluation, where testing is integrated into the training, may provide a path towards greater reliability of skill evaluation, through longitudinal sampling, and validity, through better alignment of evaluation activities in respect to learning objectives.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aberg, K. C., Tartaglia, E. M., & Herzog, M. H. (2009). Perceptual learning with Chevrons requires a minimal number of trials, transfers to untrained directions, but does not require sleep. Vision Research, 49(16), 2087–2094.

    Article  PubMed  Google Scholar 

  • Ahissar, M., & Hochstein, S. (1997). Task difficulty and the specificity of perceptual learning. Nature, 387(6631), 401–406.

    Article  PubMed  Google Scholar 

  • Anguera, J. A., Boccanfuso, J., Rintoul, J. L., Al-Hashimi, O., Faraji, F., Janowich, J., et al. (2013). Video game training enhances cognitive control in older adults. Nature, 501(7465), 97–101.

    Article  PubMed Central  PubMed  Google Scholar 

  • Au, J., Sheehan, E., Tsai, N., Duncan, G. J., Buschkuehl, M., & Jaeggi, S. M. (2015). Improving fluid intelligence with training on working memory: a meta-analysis. Psychonomic Bulletin & Review, 22(2), 366–377.

    Article  Google Scholar 

  • Baker, C. I., Peli, E., Knouf, N., & Kanwisher, N. G. (2005). Reorganization of visual processing in macular degeneration. The Journal of Neuroscience, 25(3), 614–618.

    Article  PubMed  Google Scholar 

  • Ball, K., & Sekuler, R. (1981). Adaptive processing of visual motion. Journal of Experimental Psychology. Human Perception and Performance, 7(4), 780–794.

    Article  PubMed  Google Scholar 

  • Beste, C., Wascher, E., Gunturkun, O., & Dinse, H. R. (2011). Improvement and impairment of visually guided behavior through LTP- and LTD-like exposure-based visual learning. Current Biology, 21(10), 876–882.

    Article  PubMed  Google Scholar 

  • Boot, W. R., Blakely, D. P., & Simons, D. J. (2011). Do action video games improve perception and cognition? Frontiers in Psychology, 2, 226.

    Article  PubMed Central  PubMed  Google Scholar 

  • Bul, K. C., Franken, I. H., Van der Oord, S., Kato, P. M., Danckaerts, M., Vreeke, L. J., et al. (2015). Development and user satisfaction of “Plan-It Commander,” a serious game for children with ADHD. Games Health J, 4(6), 502–512.

    Article  PubMed  Google Scholar 

  • Bul, K. C., Kato, K. M., Van der Oord, S., Danckaerts, M., Vreeke, L. J., Willems, A., et al. (2016). Behavioral outcome effects of serious gaming as an adjunct to treatment for children with attention-deficit/hyperactivity disorder: a randomized controlled trial. Journal of Medical Internet Research, 18(21), 1–18.

    Google Scholar 

  • Chung, S. T. (2011). Improving reading speed for people with central vision loss through perceptual learning. Investigative Ophthalmology & Visual Science, 52(2), 1164–1170.

    Article  Google Scholar 

  • Crist, R. E., Kapadia, M. K., Westheimer, G., & Gilbert, C. D. (1997). Perceptual learning of spatial localization: specificity for orientation, position, and context. Journal of Neurophysiology, 78(6), 2889–2894.

    Article  PubMed  Google Scholar 

  • Crist, R. E., Li, W., & Gilbert, C. D. (2001). Learning to see: experience and attention in primary visual cortex. Nature Neuroscience, 4(5), 519–525.

    Article  PubMed  Google Scholar 

  • Csikszentmihalyi, M., & Rathunde, K. (1992). The measurement of flow in everyday life: toward a theory of emergent motivation. Nebraska Symposium on Motivation, 40, 57–97.

    PubMed  Google Scholar 

  • Dahlin, E., Nyberg, L., Backman, L., & Neely, A. S. (2008). Plasticity of executive functioning in young and older adults: immediate training gains, transfer, and long-term maintenance. Psychology and Aging, 23(4), 720–730.

    Article  PubMed  Google Scholar 

  • Das, A., Tadin, D., & Huxlin, K. R. (2014). Beyond blindsight: properties of visual relearning in cortically blind fields. The Journal of Neuroscience, 34(35), 11652–11664.

    Article  PubMed Central  PubMed  Google Scholar 

  • de Jonge, M., Tabbers, H. K., Pecher, D., & Zeelenberg, R. (2012). The effect of study time distribution on learning and retention: a Goldilocks principle for presentation rate. Journal of Experimental Psychology. Learning, Memory, and Cognition, 38(2), 405–412.

    Article  PubMed  Google Scholar 

  • DeLoss, D. J., Watanabe, T., & Andersen, G. J. (2015). Improving vision among older adults: behavioral training to improve sight. Psychological Science, 26(4), 456–466.

    Article  PubMed Central  PubMed  Google Scholar 

  • Deveau, J., Jaeggi, S., Zordan, V., Phung, C., & Seitz, A. R. (2015). How to build better memory training games. Frontiers in Systems Neuroscience, 8(243), 1–7.

    Google Scholar 

  • Deveau, J., Lovcik, G., & Seitz, A. R. (2014a). Broad-based visual benefits from training with an integrated perceptual-learning video game. Vision Research, 99, 134–140.

  • Deveau, J., Ozer, D. J., & Seitz, A. R. (2014b). Improved vision and on-field performance in baseball through perceptual learning. Current Biology, 24(4), R146–R147.

    Article  PubMed Central  PubMed  Google Scholar 

  • Deveau, J., & Seitz, A. R. (2014). Applying perceptual learning to achieve practical changes in vision. Frontiers in Psychology, 5, 1166.

    Article  PubMed Central  PubMed  Google Scholar 

  • Diziol, D., Walker, E., Rummel, N., & Koedinger, K. R. (2010). Using intelligent tutor technology to implement adaptive support for student collaboration. Educational Psychology Review, 22(1), 89–102.

    Article  Google Scholar 

  • Dovis, S., Van der Oord, S., Wiers, R. W., & Prins, P. J. (2015). Improving executive functioning in children with ADHD: training multiple executive functions within the context of a computer game. a randomized double-blind placebo controlled trial. PLoS One, 10(4), e0121651.

    Article  PubMed Central  PubMed  Google Scholar 

  • Drew, T., Vo, M. L., & Wolfe, J. M. (2013). The invisible gorilla strikes again: sustained inattentional blindness in expert observers. Psychological Science, 24(9), 1848–1853.

    Article  PubMed Central  PubMed  Google Scholar 

  • Eagle, N., & Pendland, A. S. (2005). Reality mining: sensing complex social systems. Personal and Ubiquitous Computing, 10(4), 255–268.

    Article  Google Scholar 

  • Fahle, M. (2005). Perceptual learning: specificity versus generalization. Current Opinion in Neurobiology, 15(2), 154–160.

    Article  PubMed  Google Scholar 

  • Fahle, M., & Poggio, T. (2002). Perceptual learning. Cambridge: MIT Press.

    Google Scholar 

  • Fiorentini, A., & Berardi, N. (1980). Perceptual learning specific for orientation and spatial frequency. Nature, 287(5777), 43–44.

    Article  PubMed  Google Scholar 

  • Franceschini, S., Gori, S., Ruffino, M., Viola, S., Molteni, M., & Facoetti, A. (2013). Action video games make dyslexic children read better. Current Biology, 23(6), 462–466.

    Article  PubMed  Google Scholar 

  • Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention. Nature, 423(6939), 534–537.

    Article  PubMed  Google Scholar 

  • Green, C. S., & Bavelier, D. (2007). Action-video-game experience alters the spatial resolution of vision. Psychological Science, 18(1), 88–94.

    Article  PubMed Central  PubMed  Google Scholar 

  • Green, C. S., Strobach, T., & Schubert, T. (2014). On methodological standards in training and transfer experiments. Psychological Research Psychologische Forschung, 78(6), 756–772.

    Article  PubMed  Google Scholar 

  • Hardy, J. L., Nelson, R. A., Thomason, M. E., Sternberg, D. A., Katovich, K., Farzin, F., & Scanlon, M. (2015). Enhancing cognitive abilities with comprehensive training: a large, online, randomized, active-controlled trial. PLoS One, 10(9), e0134467.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hikosaka, O., Nakamura, K., Sakai, K., & Nakahara, H. (2002). Central mechanisms of motor skill learning. Current Opinion in Neurobiology, 12(2), 217–222.

    Article  PubMed  Google Scholar 

  • Hofmann, W., Schmeichel, B. J., & Baddeley, A. D. (2012). Executive functions and self-regulation. Trends in Cognitive Sciences, 16(3), 174–180.

    Article  PubMed  Google Scholar 

  • Huang, C. B., Zhou, Y., & Lu, Z. L. (2008). Broad bandwidth of perceptual learning in the visual system of adults with anisometropic amblyopia. Proceedings of the National Academy of Sciences of the United States of America, 105(10), 4068–4073.

    Article  PubMed Central  PubMed  Google Scholar 

  • Hung, S. C., & Seitz, A. R. (2014). Prolonged training at threshold promotes robust retinotopic specificity in perceptual learning. The Journal of Neuroscience, 34(25), 8423–8431.

    Article  PubMed Central  PubMed  Google Scholar 

  • Huxlin, K. R., Martin, T., Kelly, K., Riley, M., Friedman, D. I., Burgin, W. S., & Hayhoe, M. (2009). Perceptual relearning of complex visual motion after V1 damage in humans. The Journal of Neuroscience, 29(13), 3981–3991.

    Article  PubMed Central  PubMed  Google Scholar 

  • Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008). Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences of the United States of America, 105(19), 6829–6833.

    Article  PubMed Central  PubMed  Google Scholar 

  • Jeter, P. E., Dosher, B. A., Liu, S. H., & Lu, Z. L. (2010). Specificity of perceptual learning increases with increased training. Vision Research, 50(19), 1928–1940.

    Article  PubMed Central  PubMed  Google Scholar 

  • Jeter, P. E., Dosher, B. A., Petrov, A., & Lu, Z. L. (2009). Task precision at transfer determines specificity of perceptual learning. Journal of Vision, 9(3), 11–13.

    Article  Google Scholar 

  • Klingberg, T., Fernell, E., Olesen, P. J., Johnson, M., Gustafsson, P., Dahlstrom, K., et al. (2005). Computerized training of working memory in children with ADHD—a randomized, controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 44(2), 177–186.

    Article  PubMed  Google Scholar 

  • Klingberg, T., Forssberg, H., & Westerberg, H. (2002). Training of working memory in children with ADHD. Journal of Clinical and Experimental Neuropsychology, 24(6), 781–791.

    Article  PubMed  Google Scholar 

  • Li, J., Thompson, B., Deng, D., Chan, L. Y., Yu, M., & Hess, R. F. (2013). Dichoptic training enables the adult amblyopic brain to learn. Current Biology, 23(8), R308–R309.

    Article  PubMed  Google Scholar 

  • Li, R. W., Ngo, C., Nguyen, J., & Levi, D. M. (2011). Video-game play induces plasticity in the visual system of adults with amblyopia. PLoS Biology, 9(8), e1001135.

    Article  PubMed Central  PubMed  Google Scholar 

  • Merzenich, M. M., Jenkins, W. M., Johnston, P., Schreiner, C., Miller, S. L., & Tallal, P. (1996). Temporal processing deficits of language-learning impaired children ameliorated by training. Science, 271(5245), 77–81.

    Article  PubMed  Google Scholar 

  • Mishra, J., de Villers-Sidani, E., Merzenich, M., & Gazzaley, A. (2014). Adaptive training diminishes distractibility in aging across species. Neuron, 84(5), 1091–1103.

    Article  PubMed Central  PubMed  Google Scholar 

  • Mislevy, R. J. (2013). Evidence-centered design for simulation-based assessment. Military Medicine, 178(10 Suppl), 107–114.

    Article  PubMed  Google Scholar 

  • O’Rourke, E., Andersen, E., Gulwani, S., & Popović, Z. (2015). A framework for automatically generating interactive instructional scaffolding. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp 1545–1554). Seoul: ACM.

  • Owen, A. M., Hampshire, A., Grahn, J. A., Stenton, R., Dajani, S., Burns, A. S., et al. (2010). Putting brain training to the test. Nature, 465(7299), 775–778.

    Article  PubMed Central  PubMed  Google Scholar 

  • Piaget, J. (1952). The origins of intelligence in children. New York: International Universities Press.

    Book  Google Scholar 

  • Polat, U. (2009). Making perceptual learning practical to improve visual functions. Vision Research, 49(21), 2566–2573.

  • Polat, U., Ma-Naim, T., Belkin, M., & Sagi, D. (2004). Improving vision in adult amblyopia by perceptual learning. Proceedings of the National Academy of Sciences of the United States of America, 101(17), 6692–6697.

    Article  PubMed Central  PubMed  Google Scholar 

  • Polat, U., Schor, C., Tong, J. L., Zomet, A., Lev, M., Yehezkel, O., et al. (2012). Training the brain to overcome the effect of aging on the human eye. Scientific Reports, 2, 278.

    Article  PubMed Central  PubMed  Google Scholar 

  • Popov, T., Jordanov, T., Rockstroh, B., Elbert, T., Merzenich, M. M., & Miller, G. A. (2011). Specific cognitive training normalizes auditory sensory gating in schizophrenia: a randomized trial. Biological Psychiatry, 69(5), 465–471.

    Article  PubMed  Google Scholar 

  • Raiguel, S., Vogels, R., Mysore, S. G., & Orban, G. A. (2006). Learning to see the difference specifically alters the most informative V4 neurons. The Journal of Neuroscience, 26(24), 6589–6602.

    Article  PubMed  Google Scholar 

  • Sagi, D., & Tanne, D. (1994). Perceptual learning: learning to see. Current Opinion in Neurobiology, 4(2), 195–199.

    Article  PubMed  Google Scholar 

  • Schoups, A., Vogels, R., Qian, N., & Orban, G. (2001). Practising orientation identification improves orientation coding in V1 neurons. Nature, 412(6846), 549–553.

    Article  PubMed  Google Scholar 

  • Schoups, A. A., Vogels, R., & Orban, G. A. (1995). Human perceptual learning in identifying the oblique orientation: retinotopy, orientation specificity and monocularity. The Journal of Physiology, 483(Pt 3), 797–810.

    Article  PubMed Central  PubMed  Google Scholar 

  • Seitz AR (2017) Perceptual learning. Current Biology, 27, R631–R636.

  • Seitz, A. R., Kim, R., & Shams, L. (2006). Sound facilitates visual learning. Current Biology, 16(14), 1422–1427.

    Article  PubMed  Google Scholar 

  • Seitz, A. R., & Watanabe, T. (2009). The phenomenon of task-irrelevant perceptual learning. Vision Research, 49(21), 2604–2610.

    Article  PubMed Central  PubMed  Google Scholar 

  • Shams, L., & Seitz, A. R. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12, 411–417.

  • Shipstead, Z., Redick, T. S., & Engle, R. W. (2012). Is working memory training effective? Psychological Bulletin, 138(4), 628–654.

    Article  PubMed  Google Scholar 

  • Shute, J. V., Leighton, J. P., Jang, E. E., & Chu, M. W. (2016). Advances in the science of assessment. Educational Assessment, 21(1), 34–59.

    Article  Google Scholar 

  • Shute, V., Ke, F., & Wang, L. (2017). Assessment and adaptation in games. In P. Wouters & H. V. Oostendrop (Eds.), Instructional techniques to facilitate learning and motivation of serious games (pp. 59–78). New York, NY: Springer.

  • Simons, D. J., Boot, W. R., Charness, N., Gathercole, S. E., Chabris, C. F., Hambrick, D. Z., & Stine-Morrow, E. A. (2016). Do “brain-training” programs work? Psychological Science in the Public Interest, 17(3), 103–186.

    Article  PubMed  Google Scholar 

  • Smith, G. E., Housen, P., Yaffe, K., Ruff, R., Kennison, R. F., Mahncke, H. W., & Zelinski, E. M. (2009). A cognitive training program based on principles of brain plasticity: results from the Improvement in Memory with Plasticity-based Adaptive Cognitive Training (IMPACT) study. Journal of the American Geriatrics Society, 57(4), 594–603.

    Article  PubMed Central  PubMed  Google Scholar 

  • Szpiro, S. F., Wright, B. A., & Carrasco, M. (2014). Learning one task by interleaving practice with another task. Vision Research, 101, 118–124.

    Article  PubMed Central  PubMed  Google Scholar 

  • Thorndike, E. L. (1913). Educational psychology. New York: Teachers college, Columbia University.

    Google Scholar 

  • Titz, C., & Karbach, J. (2014). Working memory and executive functions: effects of training on academic achievement. Psychological Research, 78(6), 852–868.

    Article  PubMed  Google Scholar 

  • Vlahou, E. L., Protopapas, A., & Seitz, A. R. (2012). Implicit training of nonnative speech stimuli. Journal of Experimental Psychology. General, 141(2), 363–381.

    Article  PubMed  Google Scholar 

  • Volante, L. (2004). Teaching to the test: what every educator and policy-maker should know. Canadian Journal of Educational Administration and Policy, 35, 6

  • Watanabe, T., Nanez Sr., J. E., Koyama, S., Mukai, I., Liederman, J., & Sasaki, Y. (2002). Greater plasticity in lower-level than higher-level visual motion processing in a passive perceptual learning task. Nature Neuroscience, 5(10), 1003–1009.

    Article  PubMed  Google Scholar 

  • Wexler, B. E., Iseli, M., Leon, S., Zaggle, W., Rush, C., Goodman, A., et al. (2016). Cognitive priming and cognitive training: immediate and far transfer to academic skills in children. Scientific Reports, 6, 32859.

    Article  PubMed Central  PubMed  Google Scholar 

  • Whitton, J. P., Hancock, K. E., & Polley, D. B. (2014). Immersive audiomotor game play enhances neural and perceptual salience of weak signals in noise. Proceedings of the National Academy of Sciences of the United States of America, 111(25), E2606–E2615.

    Article  PubMed Central  PubMed  Google Scholar 

  • Xiao, L. Q., Zhang, J. Y., Wang, R., Klein, S. A., Levi, D. M., & Yu, C. (2008). Complete transfer of perceptual learning across retinal locations enabled by double training. Current Biology, 18(24), 1922–1926.

    Article  PubMed Central  PubMed  Google Scholar 

  • Zhou, J., Zhang, Y., Dai, Y., Zhao, H., Liu, R., Hou, F., et al. (2012). The eye limits the brain’s learning potential. Scientific Reports, 2, 364.

    Article  PubMed Central  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of California, Riverside, Riverside, CA, USA

    Aaron R. Seitz

Authors
  1. Aaron R. Seitz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aaron R. Seitz.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Seitz, A.R. A New Framework of Design and Continuous Evaluation to Improve Brain Training. J Cogn Enhanc 2, 78–87 (2018). https://doi.org/10.1007/s41465-017-0058-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41465-017-0058-8

Keywords

Search

Navigation