Universal Access in the Information Society

, Volume 17, Issue 2, pp 275–290 | Cite as

Assessment of the utility of gesture-based applications for the engagement of Chinese children with autism

  • Tiffany Y. TangEmail author
  • Mary Falzarano
  • Patricia A. Morreale
Long Paper


Collaborative play, an educational tool for children on the Autism Spectrum Disorder (ASD) spectrum, has been demonstrated as having potential for increasing the engagement of children with ASD. Researchers in China and the USA have assessed three approaches for accessible interface design and learning by students with ASD. With the use of known tools and appropriate occupational therapy interventions, an educational protocol was designed to evaluate the two selected applications and a commercially available application. The pilot studies, including experimental design and outcomes, are presented in this paper in the context of prior ASD intervention research, correlated with child development studies, and provide a solid foundation for comparative usability assessment of mid-air finger gesture interaction as well as hand gesture interaction for the wider population of users. Early results in China are promising, based on experiences in the USA.


Autism Hand and finger gesture Accessible interface design Collaborative play 



The authors would like to acknowledge Kean University’s financial support to Tiffany Tang under Wenzhou-Kean University’s Student Partnering with Faculty (SpF) Research Program. The authors are also grateful to children and their families at Wenzhou XingLe Children’s Educational Development Center and Wenzhou Orange Wheat Children's Educational center for participating in the experiments; thanks also go to Tina Xiaoting Fu and Esther Mingyue Tang for developing the two games and their efforts during both the experiment and interviews as well as Leila Zeqiang Huang, Relic Yongfu Wang and Pinata Winoto for their assistance in the experiments.


  1. 1.
    American Psychiatric Association. DSM-V: Diagnostic and Statistical Manual of Mental Disorders, 5th Edn. American Psychiatric Publication, Section 299.00 (2013)Google Scholar
  2. 2.
    Kasari, C., Freeman, S., Paparella, T.: Joint attention and symbolic play in young children with autism: a randomized controlled intervention study. J. Child Psychol. Psychiatry 47(6), 611–620 (2006)CrossRefGoogle Scholar
  3. 3.
    Machalicek, W., Shogren, K., Lang, R., Rispoli, M., O’Reilly, M.F., Franco, J.H., Sigafoos, J.: Increasing play and decreasing the challenging behavior of children with autism during recess with activity schedules and task correspondence training. Res. Autism Spectr. Disord 3(2), 547–555 (2009)CrossRefGoogle Scholar
  4. 4.
    Case-Smith, J., Arbesman, M.: Evidence-based review of interventions for autism used in or of relevance to occupational therapy. Am. J. Occup. Ther. 62, 416–429 (2008)CrossRefGoogle Scholar
  5. 5.
    Azar, N.R., McKeen, P., Carr, K., Sutherland, C.A., Horton, S.: Impact of motor skills training in adults with autism spectrum disorder and an intellectual disability. J. Dev. Disabil. 22(1), 28–38 (2016)Google Scholar
  6. 6.
    Gal, E., Lamash, L., Bauminger-Zviely, N., Zancanaro, M., Weiss, P.L.: Using multitouch collaboration technology to enhance social interaction of children with high-functioning autism. Phys. Occup. Ther. Pediatr. 36(1), 46–58 (2016)CrossRefGoogle Scholar
  7. 7.
    Biscaldi, M., Rauh, R., Irion, L., Jung, N., Mall, V., Fleischhaker, C., Klein, C.: Deficits in motor abilities and developmental fractionation of imitation performance in high-functioning autism spectrum disorders. Eur. Child Adolesc. Psychiatry 23(7), 599–610 (2014). doi: 10.1007/s00787-013-0475-x CrossRefGoogle Scholar
  8. 8.
    Sumner, E., Leonard, H.C., Hill, E.L.: Overlapping phenotypes in autism spectrum disorder and developmental coordination disorder: a cross-syndrome comparison of motor and social skills. J. Autism Dev. Disord. 46(8), 2609–2620 (2016)CrossRefGoogle Scholar
  9. 9.
    Sacrey, L.-A.R., Germani, T., Bryson, S.E., Zwaigenbaum, L.: Reaching and grasping in autism spectrum disorder: a review of recent literature. Front. Neurol. 5(6), 1–12 (2014)Google Scholar
  10. 10.
    Mostofsky, S.H., Dubey, P., Jerath, V.K., Jansiewicz, E.M., Goldberg, M.C., Denckla, M.B.: Developmental dyspraxia is not limited to imitation in children with autism spectrum disorders. J. Int. Neuropsychol. Soc. 12, 314–326 (2006)CrossRefGoogle Scholar
  11. 11.
    Leary, M.R., Hill, D.A.: Moving on: autism and movement disturbance. Ment. Retard. 34, 39–53 (1996)Google Scholar
  12. 12.
    Ming, X., Brimacombe, M., Wagner, G.C.: Prevalence of motor impairments in autism spectrum disorders. Brain Dev. 29, 565–570 (2007)CrossRefGoogle Scholar
  13. 13.
    Fournier, K.A., Hass, C.J., Naik, S.K., Lodha, N., Cauraugh, J.H.: Motor coordination in autism spectrum disorders: a synthesis and meta-analysis. J. Autism Dev. Disord. 10, 1227–1240 (2010)CrossRefGoogle Scholar
  14. 14.
    Provost, B., Heimerl, S., Lopez, B.R.: Levels of gross and fine motor development in young children with autism spectrum disorder. Phys. Occup. Ther. Pediatr. 27(3), 21–36 (2007)CrossRefGoogle Scholar
  15. 15.
    Jones, V., Prior, M.: Motor imitation abilities and neurological signs in autistic children. J. Autism Dev. Disord. 15, 37–46 (1985)CrossRefGoogle Scholar
  16. 16.
    Haswell, C.C., Izawa, J., Dowell, L.R., Mostofsky, S.H., Shadmehr, R.: Representations of internal models of action in the autistic brain. Nat. Neurosci. 12, 970–972 (2009)CrossRefGoogle Scholar
  17. 17.
    Molloy, C.A., Dietrich, K.N., Bhattacharya, A.: Postural stability in children with autism spectrum disorder. J. Autism Dev. Disord. 33(6), 643–652 (2003)CrossRefGoogle Scholar
  18. 18.
    Vilensky, J.A., Damasio, A.R., Maurer, R.G.: Gait disturbances in patients with autistic behavior: a preliminary study. Arch. Neurol. 38(10), 646–649 (1981)CrossRefGoogle Scholar
  19. 19.
    Ghaziuddin, M., Butler, E.: Clumsiness in autism and Asperger syndrome: a further report. J. Intell. Disabil. Res. 42, 43–48 (1998)CrossRefGoogle Scholar
  20. 20.
    Morin, B., Reid, G.: A quantitative and qualitative assessment of autistic individuals on selected motor tasks. Adapt. Phys. Act. Q. 2, 43–55 (1985)CrossRefGoogle Scholar
  21. 21.
    Noterdaeme, M., Mildenberger, K., Minow, F., Amorosa, H.: Evaluation of neuromotor deficits in children with autism and children with a specific speech and language disorder. Eur. Child Adolesc. Psychiatry 11(5), 219–225 (2002)CrossRefGoogle Scholar
  22. 22.
    Grandin, T.: My Experiences with Visual Thinking Sensory Problems and Communication Difficulties. Center for the Study of Autism (1996).
  23. 23.
    Provost, B., Lopez, B.R., Heimerl, S.: A comparison of motor delays in young children: autism spectrum disorder, developmental delay, and developmental concerns. J. Autism Dev. Disord. 37(2), 321–328 (2007)CrossRefGoogle Scholar
  24. 24.
    Battocchi, A., Pianesi, F., Tomasini, D., Zancanaro, M., Esposito, G., Venuti, P., Ben Sasson, A., Gal, E., Weiss, P.L.: Collaborative puzzle game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ‘09), pp. 197–204. ACM Press (2009)Google Scholar
  25. 25.
    Grandin, T.: Visual abilities and sensory differences in a person with autism. Biol. Psychiatry 65(1), 15–16 (2009)CrossRefGoogle Scholar
  26. 26.
    Simmons, D.R., Robertson, A.E., McKay, L.S., Toal, E., McAleer, P., Pollick, F.E.: Vision in autism spectrum disorders. Vis. Res 49(22), 2705–2739 (2009)CrossRefGoogle Scholar
  27. 27.
    Behrmann, M., Thomas, C., Humphreys, K.: Seeing it differently: visual processing in autism. Trends Cogn. Sci. 10(6), 258–264 (2006)CrossRefGoogle Scholar
  28. 28.
    Virnes, M., Kärnä, E., Vellonen, V.: Review of research on children with autism spectrum disorder and the use of technology. J. Spec. Educ. Technol. 30(1), 13–27 (2015)CrossRefGoogle Scholar
  29. 29.
    Farr, W., Yuill, N., Hinske, S.: An augmented toy and social interaction in children with autism. Int. J. Arts Technol. 5(2–4), 104–125 (2012)CrossRefGoogle Scholar
  30. 30.
    Chang, Y.-J., Chen, S.-F., Huang, J.-D.: A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities. Res. Dev. Disabil. 32(6), 2566–2570 (2011)CrossRefGoogle Scholar
  31. 31.
    Wessels, R., Dijcks, B., Soede, M., Gelderblom, G.J., De Witte, L.: Non-use of provided assistive technology devices: a literature overview. Technol. Disabil. 15, 231–238 (2003)Google Scholar
  32. 32.
    Webster, D., Celik, O.: Systematic review of Kinect applications in elderly care and stroke rehabilitation. J. Neuro-Eng. Rehabil. 11, 108 (2014)CrossRefGoogle Scholar
  33. 33.
    Palacios-Navarro, G., García-Magariño, I., Ramos-Lorente, P.: A Kinect-based system for lower limb rehabilitation in Parkinson’s disease patients: a pilot study. J. Med. Syst. 39, 103 (2015)CrossRefGoogle Scholar
  34. 34.
    Freitas, D.Q., Da Gama, A.E.F., Figueiredo, L., Chaves, T.M., Marques-Oliveira, D., Teichrieb, V., Araujo, C.: Development and evaluation of a Kinect based motor rehabilitation game. In: Proceedings of SBGames, pp. 144–153. (2012)Google Scholar
  35. 35.
    Altanis, G., Boloudakis, M., Retalis, S., Nikou, N.: Children with motor impairments play a Kinect learning game: first findings from a pilot case in an authentic classroom environment. J. Interact Des. Archit. 19, 91–104 (2013)Google Scholar
  36. 36.
    Pompeu, J.E., Arduini, L.A., Botelho, A.R., Fonseca, M.B.F., Pompeu, S.M.A.A., Torriani-Pasin, C., Deutsch, J.E.: Feasibility, safety and outcomes of playing Kinect Adventures!™ for people with Parkinson’s disease: a pilot study. Physiotherapy 100(2), 162–168 (2014)CrossRefGoogle Scholar
  37. 37.
    Lee, J.D., Hsieh, C.H., Lin, T.Y.: A Kinect-based Tai Chi exercises evaluation system for physical rehabilitation. In: Proceedings of the IEEE International Conference on Consumer Electronics (ICCE), pp. 177–178. (2014)Google Scholar
  38. 38.
    Chang, C.Y., Lange, B., Zhang, M., Koenig, S., Requejo, P., Somboon, N., Sawchuk, A. A., Rizzo, A.A.: Towards pervasive physical rehabilitation using Microsoft Kinect. In: Proceedings of 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (Pervasive Health), pp. 159–162. (2012)Google Scholar
  39. 39.
    Mulroy, S.J., Thompson, L., Kemp, B., Hatchett, P.P., Newsam, C.J., Lupold, D.G., Haubert, L.L., Eberly, V., Ge, T.T., Azen, S.P., Winstein, C.J., Gordon, J.: Strengthening and optimal movements for painful shoulders (STOMPS) in chronic spinal cord injury: a randomized controlled trial. Phys. Ther. 91, 305–324 (2011)CrossRefGoogle Scholar
  40. 40.
    Clark, R., Kraemer, T.: Clinical use of Nintendo Wii bowling simulation to decrease fall risk in an elderly resident of a nursing home: a case report. J. Geriatr. Phys. Ther. 32(4), 174–180 (2009)CrossRefGoogle Scholar
  41. 41.
    Deutsch, J.E., Borbely, M., Filler, J., Huhn, K., Guarrera-Bowlby, P.: Use of a low-cost, commercially available gaming console (Wii) for rehabilitation of an adolescent with cerebral palsy. Phys. Ther. 88(10), 1196–1207 (2008)CrossRefGoogle Scholar
  42. 42.
    Deutsch, J.E., Brettler, A., Smith, C., Welsh, J., John, R., Guarrera-Bowlby, P., MichalKafri, M.: Nintendo Wii sports and Wii Fit game analysis, validation, and application to stroke rehabilitation. Clin. Appl. Technol. Pract. 18(6), 701–709 (2011)Google Scholar
  43. 43.
    Agarwal, R., Sampath, H.A., Indurkhya, B.: A usability study on natural interaction devices with ASD children. In: Stephanidis, C., Antona, M. (eds.) Universal Access in Human-Computer Interaction User and Context Diversity LNCS Volume 8010, pp. 447–453. Springer, Berlin, Heidelberg (2013)CrossRefGoogle Scholar
  44. 44.
    Saiano, M., Pellegrino, L., Casadio, M., Summa, S., Garbarino, E., Rossi, V., Dall’Agata, D., Sanguineti, V.: Natural interfaces and virtual environments for the acquisition of street crossing and path following skills in adults with Autism Spectrum Disorders: a feasibility study. J. NeuroEng. Rehabil. 12, 17 (2015)CrossRefGoogle Scholar
  45. 45.
    Bai, Z., Blackwell, A.F., Coulouris, G.: Using augmented reality to elicit pretend play for children with autism. IEEE Trans. Vis. Comput. Graph. 21(5), 598–610 (2015)CrossRefGoogle Scholar
  46. 46.
    Bartoli, L., Corradi, C., Garzotto, F., Valoriani, M.: Exploring motion-based touchless games for autistic children’s learning. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC’2013), pp. 102–111. ACM Press (2013)Google Scholar
  47. 47.
    Bartoli, L., Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M.: Designing and evaluating touchless playful interaction for ASD children. In: Proceedings of the ACM International Conference on Interaction Design and Children (IDC’14), pp. 17–26. ACM Press (2014)Google Scholar
  48. 48.
    Garzotto, F., Gelsomini, M., Oliveto, L., Valoriani, M: Motion-based touchless interaction for ASD children: a case study. In: Proceedings of AVI, pp. 117–120. ACM Press (2014)Google Scholar
  49. 49.
    Ringland, K.E., Zalapa, R. Neal, M, Escobedo, L. Tentori, M., Hayes, G.: SensoryPaint: a natural user interface supporting sensory integration in children with neurodevelopmental disorders. İn: CHI ‘14 Extended Abstracts, pp. 1681–1686. ACM Press (2014)Google Scholar
  50. 50.
    Casas, X., Herrera, G., Coma, I., Fernández, M.: A kinect-based augmented reality system for individuals with autism spectrum disorders. In: Proceedings of the International Conference on Computer Graphics Theory and Applications and International Conference on Information Visualization Theory and Applications (GRAPP/IVAPP ‘12). Pp. 440–446. SciTePress (2012)Google Scholar
  51. 51.
    Mora-Guiard, J., Crowell, C., Pares, N., Heaton, P.: Lands of fog: helping children with autism in social interaction through a full-body interactive experience. In: Proceedings of the 15th International Conference on Interaction Design and Children (ACM IDC’2016), pp. 262–274. (2016)Google Scholar
  52. 52.
    Van Zomeren, A.H., Brouwer, W.H.: Clinical Neuropsychology of Attention. Oxford University Press, USA (1994)Google Scholar
  53. 53.
    Brickenkamp, R., Zillmer, E.: The d2 Test of Attention. Hogrefe & Huber Publication, Göttingen (1998)Google Scholar
  54. 54.
    Bhattacharya, A., Gelsomini, M., Pérez-Fuster, P., Abowd, G.D., Rozga, A.: Designing motion-based activities to engage students with autism in classroom settings. In: Proceedings of the 14th International Conference on Interaction Design and Children (ACM IDC’2015), pp. 69–78. (2015)Google Scholar
  55. 55.
    Tung, J.Y., Lulic, T., Gonzalez, D.A., Tran, J., Dickerson, C.R., Roy, E.A.: Evaluation of a portable marker less finger position capture device: accuracy of the Leap Motion controller in healthy adults. Physiol. Meas. 36(5), 1025–1035 (2015)CrossRefGoogle Scholar
  56. 56.
    Hondori, H.M., Khademi, M.: A review on technical and clinical impact of microsoft kinect on physical therapy and rehabilitation. J. Med. Eng. 2014, 846514 (2014)Google Scholar
  57. 57.
    Ebner, M., Spot, M.: Game-based learning with the leap motion controller. In: Russell, D., Laffey, J.M. (eds.) Handbook of Research on Gaming Trends in P-12 Education. IGI Global, pp 555–565. (2015)Google Scholar
  58. 58.
    Iosa, M., Morone, G., Fusco, A., Castagnoli, M., Fusco, F.R., Pratesi, L., Paolucci, S.: Leap Motion controlled videogame-based therapy for rehabilitation of elderly patients with subacute stroke: a feasibility pilot study. Top. Stroke Rehabil. 22(4), 306–316 (2015)CrossRefGoogle Scholar
  59. 59.
    Potter, L.E., Araullo, J., Carter, L.: The Leap Motion controller: a view on sign language. In: Proceedings of the 25th Australian Computer-Human Interaction Conference on Augmentation, Application, Innovation, Collaboration—OzCHI’13, pp. 175–178. (2013)Google Scholar
  60. 60.
    Gieser, S.N., Boisselle, A., Makedon, F.: Real-time static gesture recognition for upper extremity rehabilitation using the leap motion. In: Lecture Notes in Computer Science, pp. 144–154. (2015)Google Scholar
  61. 61.
    Tang, T., Winoto, P., Wang, R.: Having fun over a distance: supporting multiplayer online ball passing using multiple sets of Kinect. In: CHI ‘15 Extended Abstracts, pp. 1187–1192. ACM Press, New York (2015)Google Scholar
  62. 62.
    Zhu, G., Cai, S., Ma, Y., Liu, E.: A series of leap motion-based matching games for enhancing the fine motor skills of children with autism. In: 15th International IEEE Conference on Advanced Learning Technologies (ICALT’2015), IEEE, pp. 430–431. (2015)Google Scholar
  63. 63.
    Zwaigenbaum, L., Bauman, M.L., Choueiri, R., Fein, D., Kasari, C., Pierce, K., Stone, W.L., Yirmiya, N., Estes, A., Hansen, R.L., McPartland, J.C., Natowicz, M.R., Buie, T., Carter, A., Davis, P.A., Granpeesheh, D., Mailloux, Z., Newschaffer, C., Robins, D., Roley, S.S., Wagner, S., Wetherby, A.: Early identification and interventions for autism spectrum disorder: executive summary. Pediatrics 136(Suppl 1), S1–S9 (2015)CrossRefGoogle Scholar
  64. 64.
    Greis, F., Silva, M., Raposo, A., Suplino, M.: Exploring collaboration patterns in a multitouch game to encourage social interaction and collaboration among users with autism spectrum disorder. Comput. Support. Coop. Work 24(2-3), 149–175 (2015)CrossRefGoogle Scholar
  65. 65.
    Goh, W., Shou, W., Tan, J., Lum, G.T.J. Interaction design patterns for multi-touch tabletop collaborative games. In: CHI ‘12 Extended Abstracts on Human Factors in Computing Systems (CHI EA´12), pp. 141–150. New York, NY, USA (2012)Google Scholar
  66. 66.
    Chelsea La Valle. Chinese Cultural Factors Impacting the Educational Schooling of Children with Autism in China. DePaul Discoveries: Vol. 2: Issue 1, Article 10. Available at: (2013)
  67. 67.
    Wang, S.: In China, the making of an App for autism. Wall Street J. May 19, 2015. (2015)
  68. 68.
    Marco, E., Barett, L., Hinkley, N., Hill, S., Nagarajan, S.S.: Sensory processing in autism: a review of neurophysiologic findings. Pediatr. Res. 69(5 Pt 2), 48R–54R (2011)CrossRefGoogle Scholar
  69. 69.
    Baron-Cohen, S., Golan, O., Ashwin, E.: Can emotion recognition be taught to children with autism spectrum conditions? Philos. Trans. R. Soc. Lond. Ser. B Biol. Sci 364(1535), 3567–3574 (2009). doi: 10.1098/rstb.2009.0191;10.1098/rstb.2009.0191 CrossRefGoogle Scholar
  70. 70.
    Gordon, I., Pierce, M., Bartlett, M., Tanaka, J.: Training facial expression production in children on the autism spectrum. J. Autism Dev. Disord. 44(10), 2486–2498 (2014). doi: 10.1007/s10803-014-2118-6 CrossRefGoogle Scholar
  71. 71.
    Ploog, B., Scharf, A., Nelson, D., Brooks, P.: Use of computer-assisted technologies (CAT) to enhance social, communicative, and language development in children with autism spectrum disorders. J. Autism Dev. Disord. 43(2), 301–322 (2013). doi: 10.1007/s10803-012-1571-3 CrossRefGoogle Scholar
  72. 72.
    Rice, L., Wall, C., Fogel, A., Shic, F.: Computer-assisted face processing instruction improves emotion recognition, mentalizing, and social skills in students with ASD. J. Autism Dev. Disord. 45(7), 2176–2186 (2015). doi: 10.1007/s10803-015-2380-2 CrossRefGoogle Scholar
  73. 73.
    Tang, T.Y., Flatla, D.: Autism awareness and technology-based intervention research in China: the good, the bad, and the challenging. In: Proceedings of Workshop on Autism and Technology—Beyond Assistance and Intervention, in Conjunction with the 34th ACM International Conference on Human Factors in Computing Systems (CHI’2016) (2016)Google Scholar
  74. 74.
    Goldsmith, T.R., LeBlanc, L.A.: Use of technology in interventions for children with autism. J. Early Intensive Behav. Interv. 1(2), 166 (2004)CrossRefGoogle Scholar
  75. 75.
    Hayes, G.R., Hirano, S., Marcu, G., Monibi, M., Nguyen, D.H., Yeganyan, M.: Interactive visual supports for children with autism. Pers. Ubiquitous Comput. 14(7), 663–680 (2010)CrossRefGoogle Scholar
  76. 76.
    Silver, M., Oakes, P.: Evaluation of a new computer intervention to teach people with autism or Asperger syndrome to recognize and predict emotions in others. Autism 5(3), 299–316 (2001)CrossRefGoogle Scholar
  77. 77.
    Clampton, N.: China moves to tackle autism with first study of prevalence. South China Morning Post. (2013)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  • Tiffany Y. Tang
    • 1
    Email author
  • Mary Falzarano
    • 2
  • Patricia A. Morreale
    • 3
  1. 1.Media Lab, Department of Computer ScienceWenzhou-Kean UniversityZhejiangChina
  2. 2.Department of Occupational TherapyKean UniversityUnionUSA
  3. 3.Department of Computer ScienceKean UniversityUnionUSA

Personalised recommendations