Skip to main content

Advertisement

SpringerLink
Log in

Guess What?

Towards Understanding Autism from Structured Video Using Facial Affect

  • Research Article
  • Published:
Journal of Healthcare Informatics Research Aims and scope Submit manuscript

Abstract

Autism Spectrum Disorder (ASD) is a condition affecting an estimated 1 in 59 children in the United States. Due to delays in diagnosis and imbalances in coverage, it is necessary to develop new methods of care delivery that can appropriately empower children and caregivers by capitalizing on mobile tools and wearable devices for use outside of clinical settings. In this paper, we present a mobile charades-style game, Guess What?, used for the acquisition of structured video from children with ASD for behavioral disease research. We then apply face tracking and emotion recognition algorithms to videos acquired through Guess What? game play. By analyzing facial affect in response to various prompts, we demonstrate that engagement and facial affect can be quantified and measured using real-time image processing algorithms: an important first-step for future therapies, at-home screenings, and outcome measures based on home video. Our study of eight subjects demonstrates the efficacy of this system for deriving highly emotive structured video from children with ASD through an engaging gamified mobile platform, while revealing the most efficacious prompts and categories for producing diverse emotion in participants.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Battocchi A, Pianesi F, Tomasini D, Zancanaro M, Esposito G, Venuti P, Ben Sasson A, Gal E, Weiss PL (2009) Collaborative puzzle game: a tabletop interactive game for fostering collaboration in children with autism spectrum disorders (asd). In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, ACM, pp 197–204

  2. Bernardini S, Porayska-Pomsta K, Smith TJ (2014) Echoes: an intelligent serious game for fostering social communication in children with autism. Inf Sci 264:41–60

    Article  Google Scholar 

  3. Bernier R, Mao A, Yen J (2010) Psychopathology, families, and culture: autism. Child Adolesc Psychiatr Clin N Am 19(4):855–867

    Article  Google Scholar 

  4. Bros W Heads up - android apps on google play. https://play.google.com/store/apps/details?id=com.wb.headsup

  5. Christensen DL, Bilder DA, Zahorodny W, Pettygrove S, Durkin MS, Fitzgerald RT, Rice C, Kurzius-Spencer M, Baio J, Yeargin-Allsopp M (2016) Prevalence and characteristics of autism spectrum disorder among 4-year-old children in the autism and developmental disabilities monitoring network. J Dev Behav Pediatr 37(1):1–8

    Article  Google Scholar 

  6. Daniels J, Schwartz J, Haber N, Voss C, Kline A, Fazel A, Washington P, De T, Feinstein C, Winograd T et al (2017) 5.13 design and efficacy of a wearable device for social affective learning in children with autism. J Am Acad Child Adolesc Psychiatry 56(10):S257

    Article  Google Scholar 

  7. Dawson G (2008) Early behavioral intervention, brain plasticity, and the prevention of autism spectrum disorder. Dev Psychopathol 20(3):775–803

    Article  MathSciNet  Google Scholar 

  8. Dawson G, Bernier R (2013) A quarter century of progress on the early detection and treatment of autism spectrum disorder. Dev Psychopathol 25(4pt2):1455–1472

    Article  Google Scholar 

  9. Dawson G, Jones EJ, Merkle K, Venema K, Lowy R, Faja S, Kamara D, Murias M, Greenson J, Winter J et al (2012) Early behavioral intervention is associated with normalized brain activity in young children with autism. J Am Acad Child Adolesc Psychiatry 51(11):1150–1159

    Article  Google Scholar 

  10. Dawson G, Rogers S, Munson J, Smith M, Winter J, Greenson J, Donaldson A, Varley J (2010) Randomized, controlled trial of an intervention for toddlers with autism: the Early Start Denver Model. Pediatrics 125(1):e17–e23

    Article  Google Scholar 

  11. Dawson G, Webb SJ, McPartland J (2005) Understanding the nature of face processing impairment in autism: insights from behavioral and electrophysiological studies. Dev Neurobiol 27(3):403–424

    Google Scholar 

  12. Ekman P, Friesen WV, O’sullivan M, Chan A, Diacoyanni-Tarlatzis I, Heider K, Krause R, LeCompte WA, Pitcairn T, Ricci-Bitti PE et al (1987) Universals and cultural differences in the judgments of facial expressions of emotion. J Pers Soc Psychol 53(4):712

    Article  Google Scholar 

  13. Escobedo L, Nguyen DH, Boyd L, Hirano S, Rangel A, Garcia-Rosas D, Tentori M, Hayes G (2012) Mosoco: a mobile assistive tool to support children with autism practicing social skills in real-life situations. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp 2589–2598

  14. Escobedo L, Tentori M, Quintana E, Favela J, Garcia-Rosas D (2014) Using augmented reality to help children with autism stay focused. IEEE Pervasive Comput 13(1):38–46

    Article  Google Scholar 

  15. Feil-Seifer D, Matarić MJ (2009) Toward socially assistive robotics for augmenting interventions for children with autism spectrum disorders. In: Experimental robotics, Springer, pp 201–210

  16. Gal E, Bauminger N, Goren-Bar D, Pianesi F, Stock O, Zancanaro M, Weiss PLT (2009) Enhancing social communication of children with high-functioning autism through a co-located interface. AI Soc 24(1):75

    Article  Google Scholar 

  17. Giusti L, Zancanaro M, Gal E, Weiss PLT (2011) Dimensions of collaboration on a tabletop interface for children with autism spectrum disorder. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp 3295–3304

  18. Golan O, Ashwin E, Granader Y, McClintock S, Day K, Leggett V, Baron-Cohen S (2010) Enhancing emotion recognition in children with autism spectrum conditions: an intervention using animated vehicles with real emotional faces. J Autism Dev Disord 40(3):269–279

    Article  Google Scholar 

  19. Haber N, Voss C, Fazel A, Winograd T, Wall DP (2016) A practical approach to real-time neutral feature subtraction for facial expression recognition. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp 1–9. https://doi.org/10.1109/WACV.2016.7477675

  20. Hayes GR, Hirano S, Marcu G, Monibi M, Nguyen DH, Yeganyan M (2010) Interactive visual supports for children with autism. Pers Ubiquit Comput 14(7):663–680

    Article  Google Scholar 

  21. Hirano SH, Yeganyan MT, Marcu G, Nguyen DH, Boyd LA, Hayes GR (2010) vsked: evaluation of a system to support classroom activities for children with autism. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp 1633–1642

  22. International WB Heads up. https://play.google.com/store/apps/details?id=com.wb.headsup. Accessed: 2017-010-30

  23. Lab W Guess what? - android apps on google play. https://play.google.com/store/apps/details?id=walllab.guesswhat

  24. LoBue V, Thrasher C (2015) The child affective facial expression (cafe) set: validity and reliability from untrained adults. Front Psychol 5:1532

    Article  Google Scholar 

  25. Lord C, Rutter M, DiLavore PC, Risi S, Gotham K, Bishop S et al (2012) Autism diagnostic observation schedule: ADOS-2. Western Psychological Services, Los Angeles

    Google Scholar 

  26. Lord C, Rutter M, Le Couteur A (1994) Autism diagnostic interview-revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J Autism Dev Disord 24(5):659–685

  27. Newschaffer CJ, Croen LA, Daniels J, Giarelli E, Grether JK, Levy SE, Mandell DS, Miller LA, Pinto-Martin J, Reaven J et al (2007) The epidemiology of autism spectrum disorders. Annu Rev Public Health 28:235–258

    Article  Google Scholar 

  28. Park JH, Abirached B, Zhang Y (2012) A framework for designing assistive technologies for teaching children with ASDs emotions. In: CHI’12 Extended Abstracts on Human Factors in Computing Systems, ACM, pp 2423–2428

  29. Porayska-Pomsta K, Frauenberger C, Pain H, Rajendran G, Smith T, Menzies R, Foster ME, Alcorn A, Wass S, Bernadini S et al (2012) Developing technology for autism: an interdisciplinary approach. Pers Ubiquit Comput 16(2):117–127

    Article  Google Scholar 

  30. Sasson NJ, Pinkham AE, Weittenhiller LP, Faso DJ, Simpson C (2015) Context effects on facial affect recognition in schizophrenia and autism: behavioral and eye-tracking evidence. Schizophr Bull 42(3):675–683

    Article  Google Scholar 

  31. Scassellati B, Admoni H, Matarić M (2012) Robots for use in autism research. Annu Rev Biomed Eng 14:275–294

    Article  Google Scholar 

  32. Society A What is autism? http://www.autism-society.org/what-is/. Accessed: 2017-010-30

  33. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650

    Article  MathSciNet  MATH  Google Scholar 

  34. Teeters AC (2007) Use of a wearable camera system in conversation: toward a companion tool for social-emotional learning in autism. Ph.D. thesis Massachusetts Institute of Technology

  35. Voss C, Washington P, Haber N, Kline A, Daniels J, Fazel A, De T, McCarthy B, Feinstein C, Winograd T et al (2016) Superpower glass: delivering unobtrusive real-time social cues in wearable systems. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, ACM, pp 1218–1226

  36. Washington P, Voss C, Kline A, Haber N, Daniels J, Fazel A, De T, Feinstein C, Winograd T, Wall D (2017) Superpowerglass: a wearable aid for the at-home therapy of children with autism. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1(3):112

    Article  Google Scholar 

  37. Wiggins LD, Baio J, Rice C (2006) Examination of the time between first evaluation and first autism spectrum diagnosis in a population-based sample. J Dev Behav Pediatr 27(2):S79–S87

    Article  Google Scholar 

  38. Xavier J, Vignaud V, Ruggiero R, Bodeau N, Cohen D, Chaby L (2015) A multidimensional approach to the study of emotion recognition in autism spectrum disorders. Front Psychol 6(6):1954

    Google Scholar 

  39. Xiong X, De la Torre F (2013) Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 532–539

  40. Yeganyan MT, Cramer M, Boyd LA, Hayes GR (2010) vsked: an interactive visual schedule system for use in classrooms for children with autism. In: Proceedings of the 9th International Conference on Interaction Design and Children, ACM, pp 319–322

Download references

Funding

This study was supported by awards to D.P.W. by the National Institutes of Health (1R21HD091500-01 and 1R01EB025025-01). Additionally, we acknowledge support to D.P.W. from the Hartwell Foundation, the David and Lucile Packard Foundation Special Projects Grant, Beckman Center for Molecular and Genetic Medicine, Coulter Endowment Translational Research Grant, Berry Fellowship, Child Health Research Institute, Spectrum Pilot Program, and Thrasher Research Fund. The Dekeyser and Friends Foundation, the Mosbacher Family Fund for Autism Research, and Peter Sullivan provided additional funding.

Author information

Authors and Affiliations

  1. Department of Pediatrics, Stanford University, Stanford, CA, USA

    Haik Kalantarian, Peter Washington, Jessey Schwartz, Jena Daniels, Nick Haber & Dennis P. Wall

  2. Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

    Dennis P. Wall

Authors
  1. Haik Kalantarian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Washington
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jessey Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jena Daniels
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nick Haber
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dennis P. Wall
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haik Kalantarian.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kalantarian, H., Washington, P., Schwartz, J. et al. Guess What?. J Healthc Inform Res 3, 43–66 (2019). https://doi.org/10.1007/s41666-018-0034-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41666-018-0034-9

Keywords

Search

Navigation