A Game-Based Repeated Assessment for Cognitive Monitoring: Initial Usability and Adherence Study in a Summer Camp Setting

  • Rachel M. FlynnEmail author
  • Nirmaliz Colón-Acosta
  • Jimmy Zhou
  • Jeffrey Bower


The current feasibility study examined the adherence, reliability, and assessment potential of an evidence-based game-like mobile Monitoring Tool (Akili Interactive Labs), to monitor 100 participants’ cognition for eight sessions at a summer camp for children with special needs. A validated measure of attention was administered at baseline. In the last session, participants completed an exit questionnaire. The Monitoring Tool was found to be enjoyable, and showed a high rate of adherence. No Monitor-related adverse events were reported. Monitor metrics showed good reliability across repeated measurements, indicating it is stable over long-term cognitive monitoring. There was evidence that the Monitoring Tool was able to detect differences in cognition between the children diagnosed with attention deficit hyperactivity disorder and autism spectrum disorders.


Autism spectrum disorders Attention deficit hyperactivity disorder Learning disorders Cognition Clinical assessment Video game 



We would like to recognize the children, staff and leadership at Ramapo for Children for their participation. We would especially like to recognize the research assistants who conducted the study Laurel Kiyabu Morgan, Juliana Demicco, Gabi Verdoni, Adriana Gonzalez, Beatrice Baker, Angela Law, Dottie Bradshaw, Reeti Pal, Brianna Morales, and Gabbi Gaddy. In addition, we would like to thank Dr. Mary McKay for her mentorship on this project. This work was presented as a poster in October 2017 at the annual meeting for the American Academy of Child & Adolescent Psychiatry, Washington, D.C.

Author Contributions

RF conceived of the study, planned its design, coordinated data collection, participated in data interpretation, and drafted the manuscript; NC participated in the study design, coordinated data collection and helped draft the manuscript; JB planned the study design, performed the statistical analysis, and participated in data interpretation; JZ performed the statistical analysis. All authors read, edited and approved the final manuscript.

Compliance with Ethical Standards

Conflict of interest

Rachel Flynn and Nirmaliz Colón-Acosta both were employees of Ramapo for Children, the data collection site, at various points of time. Jeffrey Bower is an employee of Akili Interactive whose product was used. In addition, Jimmy Zhou was an employee of Akili Interactive.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.


  1. Aggarwal, A., & Lillystone, D. (2000). A follow up pilot study of objective measures in children with attention deficit hyperactivity disorder. Journal of Pediatrics and Child Health, 36(2), 134–138. Scholar
  2. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th edn.). Arlington: American Psychiatric Publishing.CrossRefGoogle Scholar
  3. Anderson-Hanley, C., Tureck, K., & Schneiderman, R. (2011). Autism and exergaming: Effects on repetitive behaviors and cognition. Psychology Research and Behavior Management, 4, 129–137. Scholar
  4. Anguera, J. A., Boccanfuso, J., Rintoul, J. L., Al-Hashimi, O., Faraji, F., Janowich, J., Kong, E., Larraburo, Y., Johnston, E., & Gazzaley, A. (2013). Video game training enhances cognitive control in older adults. Nature, 501(7465), 97–101. Scholar
  5. Anguera, J. A., Brandes-Aitken, A. N., Rolle, C. E., Skinner, S. N., Desai, S. S., Bower, J. D., & Marco, E. J. (2016). Characterizing cognitive control abilities in children with 16p11. 2 deletion using adaptive ‘video game’ technology: a pilot study. Translational Psychiatry, 6(9), e893.CrossRefPubMedCentralGoogle Scholar
  6. Anguera, J. A., Jordan, J. T., Castaneda, D., Gazzaley, A., & Areán, P. A. (2016). Conducting a fully mobile and randomized clinical trial for depression: Access, engagement and expense. BMJ Innovations, 2(1), 14–21. Scholar
  7. Atlis, M. M., Hahn, J., & Butcher, J. N. (2006). Computer-based assessment with the MMPI-2. In J. N. Butcher (Ed.), MMPI-2: The practitioner’s handbook (pp. 445–476). Washington, DC: American Psychological Association.Google Scholar
  8. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. Scholar
  9. Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Interaction. 2013, Scholar
  10. Bioulac, S., Lallemand, S., Rizzo, A., Philip, P., Fabrigoule, C., & Bouvard, M. P. (2012). Impact of time on task on ADHD patient’s performances in a virtual classroom. European Journal of Pediatric Neurology, 16(5), 514–521. Scholar
  11. Butcher, J. N. (2006). Assessment in clinical psychology: A perspective on the past, present challenges, and future prospects. Clinical Psychology: Science and Practice, 13(3), 205–209. Scholar
  12. Centers for Disease Control and Prevention. (2010). Increasing prevalence of parent-reported attention-deficit/hyperactivity disorder among children - United States, 2003 and 2007. Morbidity and Mortality Weekly Report, 59(44), 1439–1443.Google Scholar
  13. Centers for Disease Control and Prevention. (2014). Prevalence of autism spectrum disorder among children aged 8 years - Autism and developmental disabilities monitoring network, United States, 2010. Morbidity and Mortality Weekly Report, 63(2), 1–21.Google Scholar
  14. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment., 6(4), 284–290. Scholar
  15. Corbett, B. A., Constantine, L. J., Hendren, R., Rocke, D., & Ozonoff, S. (2009). Examining executive functioning in children with autism spectrum disorder, attention deficit hyperactivity disorder and typical development. Psychiatry Research, 166(2), 210–222. Scholar
  16. Craig, F., Margari, F., Legrottaglie, A. R., Palumbi, R., De Giambattista, C., & Margari, L. (2016). A review of executive function deficits in autism spectrum disorder and attention-deficit/hyperactivity disorder. Neuropsychiatric Disease and Treatment, 12, 1191.PubMedCentralGoogle Scholar
  17. De Vries, M., Prins, P. J. M., Schmand, B. A., & Geurts, H. M. (2015). Working memory and cognitive flexibility-training for children with an autism spectrum disorder: A randomized controlled trial. Journal of Child Psychology and Psychiatry, 56, 566–576. Scholar
  18. Deveau, J., Jaeggi, S. M., Zordan, V., Phung, C., & Seitz, A. R. (2015). How to build better memory training games. Frontiers in Systems Neuroscience, 8, 1–6. Scholar
  19. Diamond, A., & Liang, D. S. (2016). Conclusions about interventions, programs, and approaches for improving executive functions that appear justified and those that, despite much hype, do not. Developmental Cognitive Neuroscience, 18, 34–48. Scholar
  20. Dolan, R., Hall, T. E., Banerjee, M., Chun, E., & Strangman, N. (2005). Applying principles of universal design to test delivery: The effect of computer-based read-aloud on test performance of high school students with learning disabilities. The Journal of Technology, Learning and Assessment, 3(7), 3–32.Google Scholar
  21. Dovis, S., Van der Oord, S., Wiers, R. W., & Prins, P. J. M. (2015). Improving executive functioning in children with ADHD: Training multiple executive functions within the context of a computer game. A randomized double-blind placebo controlled trial. PLoS ONE, 10(4), e0121651. Scholar
  22. Frazier, T. W., Youngstrom, E. A., Glutting, J. J., & Watkins, M. W. (2007). ADHD and achievement: Meta-analysis of the child, adolescent, and adult literatures and a concomitant study with college students. Journal of Learning Disabilities, 40(1), 49–65. Scholar
  23. Gamberini, L., Barresi, G., Maier, A., & Scarpetta, F. (2008). A game a day keeps the doctor away: A short review of computer games in mental healthcare. Journal of CyberTherapy and Rehabilitation, 1(2), 127–145.Google Scholar
  24. Hanson, E., Bernier, R., Porche, K., Jackson, F. I., Goin-Kochel, R. P., Snyder, L. G., & Chen, Q. (2015). The cognitive and behavioral phenotype of the 16p11. 2 deletion in a clinically ascertained population. Biological Psychiatry, 77(9), 785–793.CrossRefGoogle Scholar
  25. Happé, F., Booth, R., Charlton, R., & Hughes, C. (2006). Executive function deficits in autism spectrum disorders and attention-deficit/hyperactivity disorder: Examining profiles across domains and ages. Brain and Cognition, 61(1), 25–39.CrossRefGoogle Scholar
  26. Kazmi, S., Ugail, H., Lesk, V., & Palmer, I. (2014). Interactive digital serious games for the assessment, rehabilitation, and prediction of dementia. International Journal of Computer Games Technology, 2014, 12.Google Scholar
  27. Klingberg, T., Fernell, E., Olesen, P. J., Johnson, M., Gustafsson, P., Dahlstrçm, K., Gillberg, C. G., Forssberg, H., & Westerberg, H. (2005). Computerized training of working memory in children with ADHD – a randomized, controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 44(2), 177–186.CrossRefGoogle Scholar
  28. Lange, B., Koenig, S., McConnell, E., Chang, C. Y., Juang, R., Suma, E., Bolas, M., & Rizzo, A. (2012). Interactive game-based rehabilitation using the Microsoft Kinect. In Virtual Reality Short Papers and Posters (VRW), 2012 IEEE (pp. 171–172). IEEE.Google Scholar
  29. Lawson, R. A., Papadakis, A. A., Higginson, C. I., Barnett, J. E., Wills, M. C., Strang, J. F., Wallace, G. L., & Kenworthy, L. (2015). Everyday executive function impairments predict comorbid psychopathology in autism spectrum and attention deficit hyperactivity disorders. Neuropsychology, 29(3), 445. Scholar
  30. Leek, M. R. (2001). Adaptive procedures in psychophysical research. Perception & psychophysics, 63(8), 1279–1292.CrossRefGoogle Scholar
  31. Lumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., & Munafò, M. R. (2016a). Gamification of cognitive assessment and cognitive training: A systematic review of applications and efficacy. JMIR Serious Games. Scholar
  32. Matthews, T., King, M. L., Kupzyk, K., & Lake, C. (2014). Findings and implications of developmental screening for highrisk children referred to a tertiary developmental disability center. Journal of Pediatric Health Care, 28(6), 507–515. Scholar
  33. Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49–100. Scholar
  34. Piotrowski, C. (1999). Assessment practices in the era of managed care: Current status and future directions. Journal of Clinical Psychology, 55(7), 787–796.;2-U.CrossRefGoogle Scholar
  35. Rapport, M. D., Orban, S. A., Kofler, M. J., & Friedman, L. M. (2013). Do programs designed to train working memory, other executive functions, and attention benefit children with ADHD? A meta-analytic review of cognitive, academic, and behavioral outcomes. Clinical Psychology Review, 33, 1237–1252.CrossRefGoogle Scholar
  36. Rideout, V. (2016). Measuring time spent with media: The common sense census of media use by US 8- to 18-year-olds. Journal of Children and Media, 10(1), 138–144. Scholar
  37. Robinson, K. E., Kaizar, E., Catroppa, C., Godfrey, C., & Yeates, K. O. (2014). Systematic review and meta-analysis of cognitive interventions for children with central nervous system disorders and neurodevelopmental disorders. Journal of Pediatric Psychology, 39(8), 846–865.CrossRefGoogle Scholar
  38. Sansosti, F. J., & Powell-Smith, K. A. (2008). Using computer-presented social stories and video models to increase the social communication skills of children with high-functioning autism spectrum disorders. Journal of Positive Behavior Interventions, 10(3), 162–178. Scholar
  39. Tong, T., Chignell, M., Tierney, M. C., & Lee, J. (2016). A serious game for clinical assessment of cognitive status: Validation study. JMIR Serious Games, 4(1), e7. Scholar
  40. Tseng, H. M., Tiplady, B., Macleod, H. A., & Wright, P. (1998). Computer anxiety: A comparison of pen-based personal digital assistants, conventional computer and paper assessment of mood and performance. British Journal of Psychology, 89(4), 599–610. Scholar
  41. Watters, C., Oore, S., Shepherd, M., Abouzied, A., Cox, A., Kellar, M., & Otley, A. (2006). Extending the use of games in health care. In System Sciences, 2006. HICSS’06. Proceedings of the 39th Annual Hawaii International Conference on (Vol. 5, pp. 88b). IEEE.Google Scholar
  42. Yerys, B. E., Wallace, G. L., Sokoloff, J. L., Shook, D. A., James, J. D., & Kenworthy, L. (2009). Attention deficit/hyperactivity disorder symptoms moderate cognition and behavior in children with autism spectrum disorders. Autism Research, 2(6), 322–333. Scholar
  43. Zelazo, P. D., Qu, L., Müller, U., Schneider, W., Schumann-Hengsteler, R., & Sodian, B. (2005). Hot and cool aspects of executive function: Relations in early development. In Young children’s cognitive development: Interrelationships among executive functioning, working memory, verbal ability, and theory of mind (pp. 71–93). New York: Psychology Press.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Silver School of Social Work, McSilver Institute for Poverty Policy and ResearchNew York UniversityNew YorkUSA
  2. 2.Ramapo for ChildrenNew YorkUSA
  3. 3.Akili InteractiveBostonUSA
  4. 4.Feinberg School of MedicineNorthwestern UniversityChicagoUSA

Personalised recommendations