“Tell Your Day”: Developing Multimodal Interaction Applications for Children with ASD

  • Diogo Vieira
  • Ana Leal
  • Nuno Almeida
  • Samuel SilvaEmail author
  • António Teixeira
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10277)


The development of applications for children, and particularly for those diagnosed with autism spectrum disorders (ASD), is a challenging task. In this context, careful consideration of the characteristics of these users, along with those of different stakeholders, such as parents and teachers, is essential. Also, it is important to provide different ways of using applications through multimodal interaction, in order to adapt, as much as possible, to the users’ needs, capabilities and preferences. Providing multimodality does not mean that users will interact multimodally, but provides freedom of choice to the user. Additionally, enabling multiple forms of interaction might also help understanding what actually works better, for an audience that is not always able to express an opinion regarding what might work. In this article, we take on previous work regarding the definition of a Persona for a child diagnosed with ASD and, considering the goals above, propose and evaluate a first prototype of an application targeting the audience represented by this Persona. This application, aims to serve as a place for communication and information exchange among the child, her family, and teachers and supports multimodal interaction.


Multimodal interaction Design and development Children Autism spectrum disorder ASD 



Research partially funded by IEETA Research Unit funding (UID/CEC/00127/2013) and Marie Curie Actions IRIS (ref. 610986, FP7-PEOPLE-2013-IAPP). Samuel Silva acknowledges funding from FCT grant SFRH/BPD/108151/2015.

The authors thank the participants in the evaluation and the collaboration of Professor Ana Isabel Martins, particularly during the evaluation process.


  1. 1.
    Almeida, N., Silva, S., Teixeira, A.: Design and development of speech interaction: a methodology. In: Kurosu, M. (ed.) HCI 2014. LNCS, vol. 8511, pp. 370–381. Springer, Cham (2014). doi: 10.1007/978-3-319-07230-2_36 Google Scholar
  2. 2.
    AssistiveWare: Proloquo2go: symbol-based AAC for iOS – assistiveware.
  3. 3.
    Buzzi, M.C., Buzzi, M., Gazzé, D., Senette, C., Tesconi, M.: ABCD SW: autistic behavior & computer-based didactic software. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A 2012, pp. 28:1–28:2. ACM, New York (2012)Google Scholar
  4. 4.
    Caronna, E.B., Milunsky, J.M., Tager-Flusberg, H.: Autism spectrum disorders: clinical and research frontiers. Arch. Dis. Child. 93(6), 518–523 (2008)CrossRefGoogle Scholar
  5. 5.
    Chien, M.E., Jheng, C.M., Lin, N.M., Tang, H.H., Taele, P., Tseng, W.S., Chen, M.Y.: iCAN: a tablet-based pedagogical system for improving communication skills of children with autism. Int. J. Hum. Comput. Stud. 73, 79–90 (2015)CrossRefGoogle Scholar
  6. 6.
    Cooper, A., Reimann, R., Cronin, D.: About Face 3.0: The Essentials of Interaction Design. Wiley, Indianapolis (2007)Google Scholar
  7. 7.
    Dumas, B., Lalanne, D., Oviatt, S.: Multimodal interfaces: a survey of principles, models and frameworks. In: Lalanne, D., Kohlas, J. (eds.) Human Machine Interaction. LNCS, vol. 5440, pp. 3–26. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-00437-7_1 CrossRefGoogle Scholar
  8. 8.
    Freitas, J., Candeias, S., Dias, M.S., Lleida, E., Ortega, A., Teixeira, A., Silva, S., Acarturk, C., Orvalho, V.: The IRIS project: a liaison between industry and academia towards natural multimodal communication. In: Proceedings of IberSPEECH, Las Palmas de Gran Canaria, Spain, pp. 338–347 (2014)Google Scholar
  9. 9.
    Gay, V., Leijdekkers, P., Agcanas, J., Wong, F., Wu, Q.: CaptureMyEmotion: helping autistic children understand their emotions using facial expression recognition and mobile technologies. Stud. Health Technol. Inform. 189, 71–76 (2013)Google Scholar
  10. 10.
    Kagohara, D.M., van der Meer, L., Ramdoss, S., O’Reilly, M.F., Lancioni, G.E., Davis, T.N., Rispoli, M., Lang, R., Marschik, P.B., Sutherland, D., et al.: Using iPods\({\textregistered }\) and iPads\({\textregistered }\) in teaching programs for individuals with developmental disabilities: a systematic review. Res. Dev. Disabil. 34(1), 147–156 (2013)CrossRefGoogle Scholar
  11. 11.
    Leal, A., Teixeira, A., Silva, S.: On the creation of a persona to support the development of technologies for children with autism spectrum disorder. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2016. LNCS, vol. 9739, pp. 213–223. Springer, Cham (2016). doi: 10.1007/978-3-319-40238-3_21 CrossRefGoogle Scholar
  12. 12.
    Martins, A.I., Rosa, A.F., Queirós, A., Silva, A., Rocha, N.P.: Definition and validation of the ICF-usability scale. Procedia Comput. Sci. 67, 132–139 (2015)CrossRefGoogle Scholar
  13. 13.
    Muñoz, R., Barcelos, T., Noël, R., Kreisel, S.: Development of software that supports the improvement of the empathy in children with autism spectrum disorder. In: Proceedings of 31st International Conference of the Chilean Computer Science Society, pp. 223–228, November 2012Google Scholar
  14. 14.
    Oviatt, S.: Advances in robust multimodal interface design. IEEE Comput. Graph. Appl. 23(5), 62–68 (2003)CrossRefGoogle Scholar
  15. 15.
    Quintela, M.A., Mendes, M., Correia, S.: Augmentative and alternative communication: Vox4all\({\textregistered }\) presentation. In: 2013 8th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6. IEEE (2013)Google Scholar
  16. 16.
    Rehabilitation Engineering Research Center on Communication Enhancement: Mobile devices and communication apps.
  17. 17.
    Ribeiro, V.S., Martins, A.I., Queirós, A., Silva, A.G., Rocha, N.P.: Usability evaluation of a health care application based on IPTV. Procedia Comput. Sci. 64, 635–642 (2015)CrossRefGoogle Scholar
  18. 18.
    Rosa, A.F., Martins, A.I., Costa, V., Queirós, A., Silva, A., Rocha, N.P.: European Portuguese validation of the post-study system usability questionnaire (PSSUQ). In: Proceedings of the 10th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–5. IEEE (2015)Google Scholar
  19. 19.
    Sampath, H., Indurkhya, B., Sivaswamy, J.: A communication system on smart phones and tablets for non-verbal children with autism. In: Miesenberger, K., Karshmer, A., Penaz, P., Zagler, W. (eds.) ICCHP 2012. LNCS, vol. 7383, pp. 323–330. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-31534-3_49 CrossRefGoogle Scholar
  20. 20.
    Sennott, S., Bowker, A.: Autism, AAC, and proloquo2go. SIG 12 Perspect. Augment. Altern. Commun. 18(4), 137–145 (2009)CrossRefGoogle Scholar
  21. 21.
    Silva, S., Almeida, N., Pereira, C., Martins, A.I., Rosa, A.F., Oliveira e Silva, M., Teixeira, A.: Design and development of multimodal applications: a vision on key issues and methods. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2015. LNCS, vol. 9175, pp. 109–120. Springer, Cham (2015). doi: 10.1007/978-3-319-20678-3_11 CrossRefGoogle Scholar
  22. 22.
    Teixeira, A., Almeida, N., Pereira, C., Oliveira e Silva, M., Vieira, D., Silva, S.: Applications of the multimodal interaction architecture in ambient assisted living. In: Dahl, D.A. (ed.) Multimodal Interaction with W3C Standards, pp. 271–291. Springer, Cham (2017). doi: 10.1007/978-3-319-42816-1_12 CrossRefGoogle Scholar
  23. 23.
    Teixeira, A., Ferreira, F., Almeida, N., Silva, S., Rosa, A.F., Pereira, J.C., Vieira, D.: Design and development of medication assistant: older adults centred design to go beyond simple medication reminders. Univ. Access Inf. Soc. 1–16 (2016)Google Scholar
  24. 24.
    Vieira, D., Freitas, J.A.D., Acartürk, C., Teixeira, A., Sousa, L., Silva, S., Candeias, S., Dias, M.S.: “Read that article”: exploring synergies between gaze and speech interaction. In: Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS 2015, pp. 341–342. ACM, New York (2015)Google Scholar
  25. 25.
    Walker, D.R., Thompson, A., Zwaigenbaum, L., Goldberg, J., Bryson, S.E., Mahoney, W.J., Strawbridge, C.P., Szatmari, P.: Specifying PDD-NOS: a comparison of PDD-NOS, asperger syndrome, and autism. J. Am. Acad. Child Adolesc. Psychiatry 43(2), 172–180 (2004)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Diogo Vieira
    • 1
  • Ana Leal
    • 1
  • Nuno Almeida
    • 1
  • Samuel Silva
    • 1
    Email author
  • António Teixeira
    • 1
  1. 1.DETI – Department of Electronics, Telecommunications and Informatics, IEETA – Institute of Electronics and Informatics Engineering of AveiroUniversity of AveiroAveiroPortugal

Personalised recommendations