A Game-Based Repeated Assessment for Cognitive Monitoring: Initial Usability and Adherence Study in a Summer Camp Setting

Abstract

The current feasibility study examined the adherence, reliability, and assessment potential of an evidence-based game-like mobile Monitoring Tool (Akili Interactive Labs), to monitor 100 participants’ cognition for eight sessions at a summer camp for children with special needs. A validated measure of attention was administered at baseline. In the last session, participants completed an exit questionnaire. The Monitoring Tool was found to be enjoyable, and showed a high rate of adherence. No Monitor-related adverse events were reported. Monitor metrics showed good reliability across repeated measurements, indicating it is stable over long-term cognitive monitoring. There was evidence that the Monitoring Tool was able to detect differences in cognition between the children diagnosed with attention deficit hyperactivity disorder and autism spectrum disorders.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

References

  1. Aggarwal, A., & Lillystone, D. (2000). A follow up pilot study of objective measures in children with attention deficit hyperactivity disorder. Journal of Pediatrics and Child Health, 36(2), 134–138. https://doi.org/10.1046/j.1440-1754.2000.00464.x.

    Article  Google Scholar 

  2. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th edn.). Arlington: American Psychiatric Publishing.

    Book  Google Scholar 

  3. Anderson-Hanley, C., Tureck, K., & Schneiderman, R. (2011). Autism and exergaming: Effects on repetitive behaviors and cognition. Psychology Research and Behavior Management, 4, 129–137. https://doi.org/10.2147/PRBM.S24016.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Anguera, J. A., Boccanfuso, J., Rintoul, J. L., Al-Hashimi, O., Faraji, F., Janowich, J., Kong, E., Larraburo, Y., Johnston, E., & Gazzaley, A. (2013). Video game training enhances cognitive control in older adults. Nature, 501(7465), 97–101. https://doi.org/10.1038/nature12486.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Anguera, J. A., Brandes-Aitken, A. N., Rolle, C. E., Skinner, S. N., Desai, S. S., Bower, J. D., & Marco, E. J. (2016). Characterizing cognitive control abilities in children with 16p11. 2 deletion using adaptive ‘video game’ technology: a pilot study. Translational Psychiatry, 6(9), e893.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Anguera, J. A., Jordan, J. T., Castaneda, D., Gazzaley, A., & Areán, P. A. (2016). Conducting a fully mobile and randomized clinical trial for depression: Access, engagement and expense. BMJ Innovations, 2(1), 14–21. https://doi.org/10.1136/bmjinnov-2015-000098.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Atlis, M. M., Hahn, J., & Butcher, J. N. (2006). Computer-based assessment with the MMPI-2. In J. N. Butcher (Ed.), MMPI-2: The practitioner’s handbook (pp. 445–476). Washington, DC: American Psychological Association.

    Google Scholar 

  8. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01.

    Article  Google Scholar 

  9. Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Interaction. 2013, https://doi.org/10.1155/2013/136864.

    Article  Google Scholar 

  10. Bioulac, S., Lallemand, S., Rizzo, A., Philip, P., Fabrigoule, C., & Bouvard, M. P. (2012). Impact of time on task on ADHD patient’s performances in a virtual classroom. European Journal of Pediatric Neurology, 16(5), 514–521. https://doi.org/10.1016/j.ejpn.2012.01.006.

    Article  Google Scholar 

  11. Butcher, J. N. (2006). Assessment in clinical psychology: A perspective on the past, present challenges, and future prospects. Clinical Psychology: Science and Practice, 13(3), 205–209. https://doi.org/10.1111/j.1468-2850.2006.00025.x.

    Article  Google Scholar 

  12. Centers for Disease Control and Prevention. (2010). Increasing prevalence of parent-reported attention-deficit/hyperactivity disorder among children - United States, 2003 and 2007. Morbidity and Mortality Weekly Report, 59(44), 1439–1443.

    Google Scholar 

  13. Centers for Disease Control and Prevention. (2014). Prevalence of autism spectrum disorder among children aged 8 years - Autism and developmental disabilities monitoring network, United States, 2010. Morbidity and Mortality Weekly Report, 63(2), 1–21.

    Google Scholar 

  14. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment., 6(4), 284–290. https://doi.org/10.1037/1040-3590.6.4.284.

    Article  Google Scholar 

  15. Corbett, B. A., Constantine, L. J., Hendren, R., Rocke, D., & Ozonoff, S. (2009). Examining executive functioning in children with autism spectrum disorder, attention deficit hyperactivity disorder and typical development. Psychiatry Research, 166(2), 210–222. https://doi.org/10.1016/j.psychres.2008.02.005.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Craig, F., Margari, F., Legrottaglie, A. R., Palumbi, R., De Giambattista, C., & Margari, L. (2016). A review of executive function deficits in autism spectrum disorder and attention-deficit/hyperactivity disorder. Neuropsychiatric Disease and Treatment, 12, 1191.

    PubMed  PubMed Central  Google Scholar 

  17. De Vries, M., Prins, P. J. M., Schmand, B. A., & Geurts, H. M. (2015). Working memory and cognitive flexibility-training for children with an autism spectrum disorder: A randomized controlled trial. Journal of Child Psychology and Psychiatry, 56, 566–576. https://doi.org/10.1111/jcpp.12324.

    Article  PubMed  Google Scholar 

  18. Deveau, J., Jaeggi, S. M., Zordan, V., Phung, C., & Seitz, A. R. (2015). How to build better memory training games. Frontiers in Systems Neuroscience, 8, 1–6. https://doi.org/10.3389/fnsys.2014.00243.

    Article  Google Scholar 

  19. Diamond, A., & Liang, D. S. (2016). Conclusions about interventions, programs, and approaches for improving executive functions that appear justified and those that, despite much hype, do not. Developmental Cognitive Neuroscience, 18, 34–48. https://doi.org/10.1016/j.dcn.2015.11.005.

    Article  PubMed  Google Scholar 

  20. Dolan, R., Hall, T. E., Banerjee, M., Chun, E., & Strangman, N. (2005). Applying principles of universal design to test delivery: The effect of computer-based read-aloud on test performance of high school students with learning disabilities. The Journal of Technology, Learning and Assessment, 3(7), 3–32.

    Google Scholar 

  21. Dovis, S., Van der Oord, S., Wiers, R. W., & Prins, P. J. M. (2015). Improving executive functioning in children with ADHD: Training multiple executive functions within the context of a computer game. A randomized double-blind placebo controlled trial. PLoS ONE, 10(4), e0121651. https://doi.org/10.1371/journal.pone.0121651.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Frazier, T. W., Youngstrom, E. A., Glutting, J. J., & Watkins, M. W. (2007). ADHD and achievement: Meta-analysis of the child, adolescent, and adult literatures and a concomitant study with college students. Journal of Learning Disabilities, 40(1), 49–65. https://doi.org/10.1177/00222194070400010401.

    Article  PubMed  Google Scholar 

  23. Gamberini, L., Barresi, G., Maier, A., & Scarpetta, F. (2008). A game a day keeps the doctor away: A short review of computer games in mental healthcare. Journal of CyberTherapy and Rehabilitation, 1(2), 127–145.

    Google Scholar 

  24. Hanson, E., Bernier, R., Porche, K., Jackson, F. I., Goin-Kochel, R. P., Snyder, L. G., & Chen, Q. (2015). The cognitive and behavioral phenotype of the 16p11. 2 deletion in a clinically ascertained population. Biological Psychiatry, 77(9), 785–793.

    Article  PubMed  Google Scholar 

  25. Happé, F., Booth, R., Charlton, R., & Hughes, C. (2006). Executive function deficits in autism spectrum disorders and attention-deficit/hyperactivity disorder: Examining profiles across domains and ages. Brain and Cognition, 61(1), 25–39.

    Article  PubMed  Google Scholar 

  26. Kazmi, S., Ugail, H., Lesk, V., & Palmer, I. (2014). Interactive digital serious games for the assessment, rehabilitation, and prediction of dementia. International Journal of Computer Games Technology, 2014, 12.

  27. Klingberg, T., Fernell, E., Olesen, P. J., Johnson, M., Gustafsson, P., Dahlstrçm, K., Gillberg, C. G., Forssberg, H., & Westerberg, H. (2005). Computerized training of working memory in children with ADHD – a randomized, controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 44(2), 177–186.

    Article  PubMed  Google Scholar 

  28. Lange, B., Koenig, S., McConnell, E., Chang, C. Y., Juang, R., Suma, E., Bolas, M., & Rizzo, A. (2012). Interactive game-based rehabilitation using the Microsoft Kinect. In Virtual Reality Short Papers and Posters (VRW), 2012 IEEE (pp. 171–172). IEEE.

  29. Lawson, R. A., Papadakis, A. A., Higginson, C. I., Barnett, J. E., Wills, M. C., Strang, J. F., Wallace, G. L., & Kenworthy, L. (2015). Everyday executive function impairments predict comorbid psychopathology in autism spectrum and attention deficit hyperactivity disorders. Neuropsychology, 29(3), 445. https://doi.org/10.1037/neu0000145.

    Article  PubMed  Google Scholar 

  30. Leek, M. R. (2001). Adaptive procedures in psychophysical research. Perception & psychophysics, 63(8), 1279–1292.

    Article  Google Scholar 

  31. Lumsden, J., Edwards, E. A., Lawrence, N. S., Coyle, D., & Munafò, M. R. (2016a). Gamification of cognitive assessment and cognitive training: A systematic review of applications and efficacy. JMIR Serious Games. https://doi.org/10.2196/games.5888.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Matthews, T., King, M. L., Kupzyk, K., & Lake, C. (2014). Findings and implications of developmental screening for highrisk children referred to a tertiary developmental disability center. Journal of Pediatric Health Care, 28(6), 507–515. https://doi.org/10.1016/j.pedhc.2014.03.002.

    Article  Google Scholar 

  33. Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41(1), 49–100. https://doi.org/10.1006/cogp.1999.0734.

    Article  PubMed  Google Scholar 

  34. Piotrowski, C. (1999). Assessment practices in the era of managed care: Current status and future directions. Journal of Clinical Psychology, 55(7), 787–796. https://doi.org/10.1002/(SICI)1097-4679(199907)55:7%3C787::AID-JCLP2%3E3.0.CO;2-U.

    Article  PubMed  Google Scholar 

  35. Rapport, M. D., Orban, S. A., Kofler, M. J., & Friedman, L. M. (2013). Do programs designed to train working memory, other executive functions, and attention benefit children with ADHD? A meta-analytic review of cognitive, academic, and behavioral outcomes. Clinical Psychology Review, 33, 1237–1252.

    Article  PubMed  Google Scholar 

  36. Rideout, V. (2016). Measuring time spent with media: The common sense census of media use by US 8- to 18-year-olds. Journal of Children and Media, 10(1), 138–144. https://doi.org/10.1080/17482798.2016.1129808.

    Article  Google Scholar 

  37. Robinson, K. E., Kaizar, E., Catroppa, C., Godfrey, C., & Yeates, K. O. (2014). Systematic review and meta-analysis of cognitive interventions for children with central nervous system disorders and neurodevelopmental disorders. Journal of Pediatric Psychology, 39(8), 846–865.

    Article  PubMed  Google Scholar 

  38. Sansosti, F. J., & Powell-Smith, K. A. (2008). Using computer-presented social stories and video models to increase the social communication skills of children with high-functioning autism spectrum disorders. Journal of Positive Behavior Interventions, 10(3), 162–178. https://doi.org/10.1177/1098300708316259.

    Article  Google Scholar 

  39. Tong, T., Chignell, M., Tierney, M. C., & Lee, J. (2016). A serious game for clinical assessment of cognitive status: Validation study. JMIR Serious Games, 4(1), e7. https://doi.org/10.2196/games.5006.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Tseng, H. M., Tiplady, B., Macleod, H. A., & Wright, P. (1998). Computer anxiety: A comparison of pen-based personal digital assistants, conventional computer and paper assessment of mood and performance. British Journal of Psychology, 89(4), 599–610. https://doi.org/10.1177/1098300708316259.

    Article  PubMed  Google Scholar 

  41. Watters, C., Oore, S., Shepherd, M., Abouzied, A., Cox, A., Kellar, M., & Otley, A. (2006). Extending the use of games in health care. In System Sciences, 2006. HICSS’06. Proceedings of the 39th Annual Hawaii International Conference on (Vol. 5, pp. 88b). IEEE.

  42. Yerys, B. E., Wallace, G. L., Sokoloff, J. L., Shook, D. A., James, J. D., & Kenworthy, L. (2009). Attention deficit/hyperactivity disorder symptoms moderate cognition and behavior in children with autism spectrum disorders. Autism Research, 2(6), 322–333. https://doi.org/10.1002/aur.103.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Zelazo, P. D., Qu, L., Müller, U., Schneider, W., Schumann-Hengsteler, R., & Sodian, B. (2005). Hot and cool aspects of executive function: Relations in early development. In Young children’s cognitive development: Interrelationships among executive functioning, working memory, verbal ability, and theory of mind (pp. 71–93). New York: Psychology Press.

    Google Scholar 

Download references

Acknowledgments

We would like to recognize the children, staff and leadership at Ramapo for Children for their participation. We would especially like to recognize the research assistants who conducted the study Laurel Kiyabu Morgan, Juliana Demicco, Gabi Verdoni, Adriana Gonzalez, Beatrice Baker, Angela Law, Dottie Bradshaw, Reeti Pal, Brianna Morales, and Gabbi Gaddy. In addition, we would like to thank Dr. Mary McKay for her mentorship on this project. This work was presented as a poster in October 2017 at the annual meeting for the American Academy of Child & Adolescent Psychiatry, Washington, D.C.

Author information

Affiliations

Authors

Contributions

RF conceived of the study, planned its design, coordinated data collection, participated in data interpretation, and drafted the manuscript; NC participated in the study design, coordinated data collection and helped draft the manuscript; JB planned the study design, performed the statistical analysis, and participated in data interpretation; JZ performed the statistical analysis. All authors read, edited and approved the final manuscript.

Corresponding author

Correspondence to Rachel M. Flynn.

Ethics declarations

Conflict of interest

Rachel Flynn and Nirmaliz Colón-Acosta both were employees of Ramapo for Children, the data collection site, at various points of time. Jeffrey Bower is an employee of Akili Interactive whose product was used. In addition, Jimmy Zhou was an employee of Akili Interactive.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Flynn, R.M., Colón-Acosta, N., Zhou, J. et al. A Game-Based Repeated Assessment for Cognitive Monitoring: Initial Usability and Adherence Study in a Summer Camp Setting. J Autism Dev Disord 49, 2003–2014 (2019). https://doi.org/10.1007/s10803-019-03881-w

Download citation

Keywords

  • Autism spectrum disorders
  • Attention deficit hyperactivity disorder
  • Learning disorders
  • Cognition
  • Clinical assessment
  • Video game