Engaging Chinese Children with Autism to Interact with Portable Hand- and Finger-Gesture Based Applications: Experiment and Reflections

  • Tiffany Y. Tang
  • Mary Falzarano
  • Patricia A. MorrealeEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9753)


Recent work by researchers in China has demonstrated the potential for collaborative play as an educational tool for children on the Autism Spectrum Disorder (ASD). In this paper, researchers in the United States and China investigate the potential for accessible interface design and learning by students with ASD. Through the use of known tools and interventions, an educational protocol has been designed to evaluate two applications. The pilot studies, including experimental design and outcomes, are presented here, and provide a solid foundation for comparative assessment of mid-air finger-gesture interaction as well as hand-gesture interaction. Early results in China are promising, based on experiences in the United States.


Autism Hand-and-finger gesture Accessible interface design Collaborative play 



We acknowledge Kean University’s financial support to Tiffany Tang under Wenzhou Kean University’s Student Partnering with Faculty (SPF) Research Program. The authors are grateful to children and their families at Wenzhou XingLe Children’s Educational Development Center for participating in the experiments; thanks also go to Tina Xiaoting Fu, and Esther Mingyue Tang for developing the two games and their efforts during both the experiment and interviews as well as Leila Zeqiang Huang and Relic Yongfu Wang for their helps during the experiment.


  1. 1.
    American Psychiatric Association. DSM-V: Diagnostic and statistical manual of mental disorders, 5th Edition. American Psychiatric Publishing, Section 299.00 (2013)Google Scholar
  2. 2.
    Kasari, C., Freeman, S., Paparella, T.: Joint attention and symbolic play in young children with autism: A randomized controlled intervention study. J. Child Psychol. Psychiatry 47(6), 611–620 (2006)CrossRefGoogle Scholar
  3. 3.
    Machalicek, W., Shogren, K., Lang, R., Rispoli, M., O’Reilly, M.F., Franco, J.H., Sigafoos, J.: Increasing play and decreasing the challenging behavior of children with autism during recess with activity schedules and task correspondence training. Res. Autism Spectrum Disord. 3(2), 547–555 (2009)CrossRefGoogle Scholar
  4. 4.
    Case-Smith, J., Arbesman, M.: Evidence-based review of interventions for autism used in or of relevance to occupational therapy. Am. J. Occup. Ther. 62, 416–429 (2008)CrossRefGoogle Scholar
  5. 5.
    Sacrey, L.-A.R., Germani, T., Bryson, S.E., Zwaigenbaum, L.: Reaching and grasping in autism spectrum disorder: a review of recent literature. Front. Neurol. 5(6), 1–12 (2014)Google Scholar
  6. 6.
    Mostofsky, S.H., Dubey, P., Jerath, V.K., Jansiewicz, E.M., Goldberg, M.C., Denckla, M.B.: Developmental dyspraxia is not limited to imitation in children with autism spectrum disorders. J. Int. Neuropsychol. Soc. 12, 314–326 (2006)CrossRefGoogle Scholar
  7. 7.
    Leary, M.R., Hill, D.A.: Moving on: autism and movement disturbance. Ment. Retard. 34, 39–53 (1996)Google Scholar
  8. 8.
    Ming, X., Brimacombe, M., Wagner, G.C.: Prevalence of motor impairments in autism spectrum disorders. Brain Dev. 29, 565–570 (2007)CrossRefGoogle Scholar
  9. 9.
    Fournier, K.A., Hass, C.J., Naik, S.K., Lodha, N., Cauraugh, J.H.: Motor coordination in autism spectrum disorders: a synthesis and meta-analysis. J. Autism Dev. Disord. 10, 1227–1240 (2010)CrossRefGoogle Scholar
  10. 10.
    Jones, V., Prior, M.: Motor imitation abilities and neurological signs in autistic children. J. Autism Dev. Disord. 15, 37–46 (1985)CrossRefGoogle Scholar
  11. 11.
    Haswell, C.C., Izawa, J., Dowell, L.R., Mostofsky, S.H., Shadmehr, R.: Representations of internal models of action in the autistic brain. Nat. Neurosci. 12, 970–972 (2009)CrossRefGoogle Scholar
  12. 12.
    Battocchi, A., Pianesi, F., Tomasini, D., Zancanaro, M., Esposito, G., Venuti, P., Sasson, A.B., Gal, E., Weiss, P.L.: Collaborative puzzle game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS 2009), pp. 197–204. ACM Press (2009)Google Scholar
  13. 13.
    Grandin, T.: My experiences with visual thinking sensory problems and communication difficulties. Center for the Study of Autism (1996)Google Scholar
  14. 14.
    Grandin, T.: Visual abilities and sensory differences in a person with autism. Biol. Psychiatry 65(1), 15–16 (2009)CrossRefGoogle Scholar
  15. 15.
    Simmons, D.R., Robertson, A.E., McKay, L.S., Toal, E., McAleer, P., Pollick, F.E.: Vision in autism spectrum disorders. Vision. Res. 49(22), 2705–2739 (2009)CrossRefGoogle Scholar
  16. 16.
    Behrmann, M., Thomas, C., Humphreys, K.: Seeing it differently: visual processing in autism. Trends Cogn. Sci. 10(6), 258–264 (2006)CrossRefGoogle Scholar
  17. 17.
    Bartoli, L., Corradi, C., Garzotto, F., Valoriani, M.: Exploring motion-based touchless games for autistic children’s learning. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC 2013), pp. 102–111. ACM Press (2013)Google Scholar
  18. 18.
    Bartoli, L., Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M.: Designing and evaluating touchless playful interaction for ASD children. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC 2014), pp. 17–26. ACM Press (2014)Google Scholar
  19. 19.
    Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M: Motion-based touchless interaction for ASD children: a case study. In: Proceedings of AVI, pp. 117–120. ACM Press (2014)Google Scholar
  20. 20.
    Ringland, K.E., Zalapa, R. Neal, M, Escobedo, L. Tentori, M. Hayes, G.: SensoryPaint: a natural user interface supporting sensory integration in children with neurodevelopmental disorders. İn: CHI 2014 Extended Abstracts, pp. 1681–1686. ACM Press (2014)Google Scholar
  21. 21.
    Tung, J.Y., Lulic, T., Gonzalez, D.A., Tran, J., Dickerson, C.R., Roy, E.A.: Evaluation of a portable marker less finger position capture device: accuracy of the leap motion controller in healthy adults. Physiol. Meas. 36(5), 1025–1035 (2015)CrossRefGoogle Scholar
  22. 22.
    Tang, T., Winoto, P., Wang, R.: Having fun over a distance: supporting multiplayer online ball passing using multiple sets of kinect. In: CHI 2015 Extended Abstracts, pp. 1187–1192. ACM Press, New York (2015)Google Scholar
  23. 23.
    Zhu, G., Cai, S., Ma, Y., Liu, E.: A series of leap motion-based matching games for enhancing the fine motor skills of children with autism. In: 15th International IEEE Conference on Advanced Learning Technologies (ICALT 2015), pp. 430–431. IEEE (2105)Google Scholar
  24. 24.
    Greis, F., Silva, M., Raposo, A., Suplino, M.: Exploring collaboration patterns in a multitouch game to encourage social interaction and collaboration among users with autism spectrum disorder. Comput. Support. Coop. Work 24(2-3), 149–175 (2015)CrossRefGoogle Scholar
  25. 25.
    Goh, W., Shou, W., Tan, J., Lum, G.T.J.: Interaction design patterns for multi-touch tabletop collaborative games. In: CHI 2012 Extended Abstracts on Human Factors in Computing Systems (CHI EA 2012), pp. 141–150, New York, NY, USA (2012)Google Scholar
  26. 26.
    La Valle, C.: Chinese cultural factors impacting the educational schooling of children with autism in China. DePaul Discoveries, Vol. 2, Issue 1, Article 10 (2013).
  27. 27.
    Wang, S.: In China, the making of an app for autism. The Wall Street Journal, 19 May 2015.
  28. 28.
    Marco, E., Barett, L., Hinkley, N., Hill, S., Nagarajan, S.S.: Sensory processing in autism: a review of neurophysiologic findings. Pediatr. Res. 69(5 Pt 2), 48R–54R (2011)CrossRefGoogle Scholar
  29. 29.
    Goldsmith, T.R., LeBlanc, L.A.: Use of technology in interventions for children with autism. J. Early Intensive Behav. Interv. 1(2), 166 (2004)CrossRefGoogle Scholar
  30. 30.
    Hayes, G.R., Hirano, S., Marcu, G., Monibi, M., Nguyen, D.H., Yeganyan, M.: Interactive visual supports for children with autism. Pers. Ubiquit. Comput. 14(7), 663–680 (2010)CrossRefGoogle Scholar
  31. 31.
    Silver, M., Oakes, P.: Evaluation of a new computer intervention to teach people with autism or Asperger syndrome to recognize and predict emotions in others. Autism 5(3), 299–316 (2001)CrossRefGoogle Scholar
  32. 32.
    Clampton, N.: China moves to tackle autism with first study of prevalence. South China Morning Post (2013)Google Scholar
  33. 33.
    Grandin, T.: Thinking in Pictures, Expanded Edition: My Life with Autism. Vintage, New York (2008)Google Scholar
  34. 34.
    Frith, C., Law, J.: Cognitive and physiological processes underlying drawing skills. Leonardo 28(3), 203–205 (1995)CrossRefGoogle Scholar
  35. 35.
    Harper, R., Mentis, H.: The mocking gaze: the social organization of kinect use. In: Proceedings of ACM CSCW 2013, pp. 167–180. ACM Press, New York. (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Tiffany Y. Tang
    • 1
  • Mary Falzarano
    • 2
  • Patricia A. Morreale
    • 3
    Email author
  1. 1.Media Laboratory, Department of Computer ScienceKean UniversityWenzhouChina
  2. 2.Department of Occupational TherapyKean UniversityUnionUSA
  3. 3.Department of Computer ScienceKean UniversityUnionUSA

Personalised recommendations