Skip to main content

Understanding the musical interaction of children with autism spectrum disorder using elastic display

Personal and Ubiquitous Computing (2023)Cite this article

Abstract

Musical interaction greatly benefits children’s development. However, case studies reporting musical interactions with children, especially those with neurodevelopmental disorders, like autism spectrum disorders (ASD), are scarce and urgently needed. In this work, we present the results of a 2-month deployment study describing how 11 children with ASD used an elastic display called BendableSound. Our results show that children with ASD maintained their musical motivation over the course of the study. Also, during this time, children with ASD were capable of manipulating and were exposed to a wide range of variations in sound and musical structures. The musical interactions triggered by the use of BendableSound influenced the type of gestures performed by our children participants. We close reflecting from our lessons learned and by discussing directions for future work.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

39,95 €

Price includes VAT (Finland)

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Notes

  1. We use both “Autistic children” and “children with ASD” in this paper to show respect for the lack of consensus and the different views and preferences communities and Autistic people have expressed regarding the use of person-first language.

  2. Sound structures are features of the vibration that propagates through the environment and are perceived by the auditory system, such as pitch, intensity, and duration, while musical structures are a composition made up of a set of sounds and its features are melody, harmony, and tempo.

  3. We defined a gesture as a movement made with the hands, and BendableSound promotes the practicing of both the gesture of pushing and sliding.

  4. http://www.tsps.cc/

  5. We used the Kinect camera, although any alternative 3D depth cameras can also be used.

  6. For simplicity of reading, we will refer to children with ASD as participants.

  7. The DSM-V classifies ASD in three levels according to the severity of the symptoms and the level of support needed by everyone.

References

  1. Ribeiro FS, Braun Janzen T, Passarini L, Vanzella P (2021) Exploring changes in musical behaviors of caregivers and children in social distancing during the COVID-19 outbreak. Front Psychol, 12. https://doi.org/10.3389/FPSYG.2021.633499/FULL

  2. Hallam S (2010) The power of music: its impact on the intellectual, social and personal development of children and young people. Int J Music Educ 28(3):269–289. https://doi.org/10.1177/0255761410370658

    Article  Google Scholar 

  3. Brandt A, Gebrian M, Slevc LR (2012) Music and early language acquisition. Front Psychol 3(SEP):327. https://doi.org/10.3389/FPSYG.2012.00327/BIBTEX

    Google Scholar 

  4. Kim J, Wigram T, Gold C (2008) The effects of improvisational music therapy on joint attention behaviors in autistic children: a randomized controlled study. J Autism Dev Disord 38:1758–1766. https://doi.org/10.1007/S10803-008-0566-6

    Article  Google Scholar 

  5. Quintin EM, Bhatara A, Poissant H, Fombonne E, Levitin DJ (2011) Emotion perception in music in high-functioning adolescents with autism spectrum disorders. J Autism Dev Disord 41:1240–1255. https://doi.org/10.1007/S10803-010-1146-0

    Article  Google Scholar 

  6. Magee W (2006) Electronic technologies in clinical music therapy: a survey of practice and attitudes. Technol Disabil 18(3):139–146

    Article  Google Scholar 

  7. Burland K, Magee W (2014) Developing identities using music technology in therapeutic settings. Psychol Music 42(2):177–189. https://doi.org/10.1177/0305735612463773

    Article  Google Scholar 

  8. Prisco RD, Malandrino D, Zaccagnino G, Zaccagnino R (2016) Natural user interfaces to support and enhance real-time music performance, pp. 204–211. ACM ???, https://doi.org/10.1145/2909132.2909249

  9. Hobbs D, Worthington-Eyre B (2008) The efficacy of combining augmented reality and music therapy with traditional teaching - preliminary results. In: i-CREATe 2008 - International Convention on Rehabilitation Engineering and Assistive Technology 2008, pp 241–244

  10. Gorman M, Lahav A, Saltzman E, Betke M (2007) A camera-based music-making tool for physical rehabilitation. Comput Music J 31:39–53. https://doi.org/10.1162/COMJ.2007.31.2.39

    Article  Google Scholar 

  11. Boulay M, Benveniste S, Boespflug S, Jouvelot P, Rigaud AS (2011) A pilot usability study of minwii, a music therapy game for demented patients. Technol Health Care 19:233–246. https://doi.org/10.3233/THC-2011-0628

    Article  Google Scholar 

  12. Troiano GM, Pedersen EW, Hornbæk K. (2015) Deformable interfaces for performing music. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp 377-386. ACM, New York, NY, USA. https://doi.org/10.1145/2702123.2702492

  13. Vazquez V, Cibrian F, Tentori M (2019) Stretchystars: a multitouch elastic display to support cooperative play among preschoolers. Pers Ubiquit Comput 23(1):99–115. https://doi.org/10.1007/s00779-018-1179-5

    Article  Google Scholar 

  14. Cibrian F, Weibel N, Tentori M (2016) Collective use of a fabric-based interactive surface to support early development in toddler classrooms. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. UbiComp ’16, pp 328-339, Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2971648.2971695

  15. Cibrian F, Ley-Flores J, Newbold JW, Singh A, Bianchi-Berthouze N, Tentori M (2021) Interactive sonification to assist children with autism during motor therapeutic interventions. Pers Ubiquit Comput 25(2):391–410. https://doi.org/10.1007/S00779-020-01479-Z/FIGURES/5

    Article  Google Scholar 

  16. Cibrian F, Peña O., Ortega D, Tentori M (2017) Bendablesound: an elastic multisensory surface using touch-based interactions to assist children with severe autism during music therapy. Int J Hum Comput Stud 107:22–37. https://doi.org/10.1016/j.ijhcs.2017.05.003

    Article  Google Scholar 

  17. Cibrian F, Pena O, Vazquez V, Cardenas C, Tentori M (2016) Designing a deformable musical surface for children with autism. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing Adjunct - UbiComp ’16, pp 977-982, ACM Press, New York, New York, USA. https://doi.org/10.1145/2968219.2968262

  18. Gallahue DL, Donnelly FC, Gallahue DL (2003) Developmental physical education for all children, 725

  19. Das S, Glickman S, Yen Hsiao F, Lee B (2017) Music everywhere-augmented reality piano improvisation learning system. In: Nime. https://goo.gl/D33VQt

  20. Hackl D, Anthes C (2017) Holokeys - an augmented reality application for learning the piano. In: CEUR Workshop Proceedings 2009, pp 140–144

  21. Löchtefeld M., Krüger A., Gehring S, Jung R (2011) GuitAR - Supporting guitar learning through mobile projection. In: Conference on Human Factors in Computing Systems - Proceedings, pp 1447–1452. https://doi.org/10.1145/1979742.1979789

  22. Marky K, Weift A, Matviienko A, Brandherm F (2021) Let’s frets! assisting guitar students during practice via capacitive sensing conference on human factors in computing systems - proceedings, https://doi.org/10.1145/3411764.3445595

  23. Ragone G (2020) Designing embodied musical interaction for children with autism ASSETS 2020 - 22nd International ACM SIGACCESS Conference on Computers and Accessibility. https://doi.org/10.1145/3373625.3417077

  24. Raymaekers L, Vermeulen J, Luyten K, Coninx K (2014) Game of tones. In: CHI ’14 Extended abstracts on human factors in computing systems. pp 411-414. ACM, New York, NY, USA. https://doi.org/10.1145/2559206.2574799

  25. Trujano F, Khan M, Maes P (2018) ARPIano efficient music learning using augmented reality. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) vol. 11003 LNCS, pp. 3–17. Springer, ???. https://doi.org/10.1007/978-3-319-99737-7_1

  26. Chow J, Feng H, Amor R, Wünsche B. C. (2013) Music education using augmented reality with a head mounted display. Conferences in Research and Practice in Information Technology Series 139:73–80

    Google Scholar 

  27. Jin W, Antle AN, Gromala D (2019) Ride N’ rhythm, bike as an embodied musical instrument to improve music perception for young children. arXiv, 1–7

  28. Lobo J, Matsuda S, Futamata I, Sakuta R, Suzuki K (2019) Chimelight: augmenting instruments in interactive music therapy for children with neurodevelopmental disorders. In: ASSETS 2019 - 21st International ACM SIGACCESS Conference on Computers and Accessibility, pp 124–135. https://doi.org/10.1145/3308561.3353784

  29. Oestermeier U, Mock P, Edelmann J, Gerjets P (2015) LEGO music: learning composition with bricks. In: Proceedings of IDC 2015: The 14th International Conference on Interaction Design and Children, pp 283–286. https://doi.org/10.1145/2771839.2771897

  30. Rogers K, Weber M, Röhlig A., Weing M, Gugenheimer J, Könings B., Klepsch M, Schaub F, Rukzio E, Seufert T (2014) P.I.A.N.O.. In: Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces - ITS ’14, pp. 149–158. ACM Press, New York, New York, USA. https://doi.org/10.1145/2669485.2669514

  31. Vandevelde C, Conradie P, De Ville J, Saldien J (2014) Playful interaction: designing and evaluating a tangible rhythmic musical interface. In: Proceedings of INTER-FACE: The Second International Conference on Live Interfaces (December 2015), pp 132–141

  32. Xiao X, Puentes P, Ackermann E, Ishii H (2016) Andantino. In: Proceedings of the The 15th International Conference on Interaction Design and Children, pp. 37–45. ACM, New York, NY, USA. https://doi.org/10.1145/2930674.2930689

  33. Sun CH, Chiang PY (2018) Mr. Piano: a portable piano tutoring system. In: Proceedings of the 2018 IEEE 25th International Conference on Electronics, Electrical Engineering and Computing, INTERCON 2018, 1?4. https://doi.org/10.1109/INTERCON.2018.8526423

  34. Bakker S, van den Hoven E, Antle AN (2010) Moso tangibles. In: Proceedings of the Fifth International Conference on Tangible, Embedded, and Embodied Interaction, pp. 85–92. ACM, New York, NY, USA. https://doi.org/10.1145/1935701.1935720

  35. Förster A, Komesker M, Sequencer S (2021) Loopblocks: design and preliminary evaluation of an accessible tangible musical step sequencer. https://doi.org/10.21428/92FBEB44.F45E1CAF

  36. Boem A, Troiano GM (2019) Non-rigid HCI: a review of deformable interfaces and input. https://doi.org/10.1145/3322276.3322347

  37. Boem A (2014) Sculpton: a malleable tangible interface for sound sculpting. In: Proceedings - 40th International Computer Music Conference, ICMC 2014 and 11th Sound and Music Computing Conference, SMC 2014 - Music Technology Meets Philosophy: From Digital Echos to Virtual Ethos (September), pp 737–743

  38. Bove VM, Wicaksono I, Rozendo C, Ye R, Trapp J, Ishii H, Dagdeviren C (2018) Perform: deformable interface for the exploration of sound through shapes. In: Conference on Human Factors in Computing Systems - Proceedings 2018-April, https://doi.org/10.1145/3170427.3188478

  39. Jensenius AR, Voldsund A (2012) The music ball project: concept, design, development, performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression (Figure 2), pp 300–303

  40. Tahirolu K, Svedström T., Wikström V., Overstall S, Kildal J, Ahmaniemi T (2014) SoundFLEX: designing audio to guide interactions with shape-retaining deformable interfaces. In: ICMI 2014 - Proceedings of the 2014 International Conference on Multimodal Interaction, pp 267–274. https://doi.org/10.1145/2663204.2663278

  41. Grierson M, Kiefer C (2013) Noisebear: a malleable wireless controller designed in participation with disabled children. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp 413–416

  42. Cibrian F, Beltran J, Tentori M (2018) Assessing the force and timing control of children with motor problems using elastic displays. In: Proceedings of the Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare – Demos, Posters, Doctoral Colloquium, pp. 1–4. EAI, ???. https://doi.org/10.4108/eai.20-4-2018.2276348

  43. Cibrian F, Madrigal M, Avelais M, Tentori M (2020) Supporting coordination of children with ASD using neurological music therapy: a pilot randomized control trial comparing an elastic touch-display with tambourines. Res Dev Disabil 106:103741. https://doi.org/10.1016/j.ridd.2020.103741

    Article  Google Scholar 

  44. Fournier KA, Hass CJ, Naik SK, Lodha N, Cauraugh JH (2010) Motor coordination in autism spectrum disorders: a synthesis and Meta-Analysis. J Autism Dev Disord 40(10):1227–1240. https://doi.org/10.1007/s10803-010-0981-3

    Article  Google Scholar 

  45. Piochon C, Kloth AD, Grasselli G, Titley HK, Nakayama H, Hashimoto K, Wan V, Simmons DH, Eissa T, Nakatani J, Cherskov A, Miyazaki T, Watanabe M, Takumi T, Kano M, Wang SS-H, Hansel C (2014) Cerebellar plasticity and motor learning deficits in a copy-number variation mouse model of autism. Nat Commun 5:5586. https://doi.org/10.1038/ncomms6586

    Article  Google Scholar 

  46. American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders. 5th edn., Washington, DC

  47. Hannant P, Tavassoli T, Cassidy S (2016) The role of sensorimotor difficulties in autism spectrum conditions. Frontiers in Neurology 7(AUG) 124. https://doi.org/10.3389/fneur.2016.00124

  48. Politi P, Emanuele E, Grassi M, Besozzi M, Cambianica M, Ciavarella G, Croci E, Damiani S, Mandrini A, Migliardi M, Pozzato E, Provenzani U, Rocchetti M (2012) Development of the “Playing-in-Touch” (PiT) questionnaire: a measure of musical intouchness in people with low-functioning autism. Neuro Endocrinol Lett 33(5):552–558

    Google Scholar 

  49. Robertson CE, Baron-cohen S (2017) Sensory perception in autism. Nat Rev Neurosci, 671–684. https://doi.org/10.1038/nrn.2017.112

  50. Gori M, Sandini G, Burr D (2012) Development of visuo-auditory integration in space and time. Frontiers in Integrative Neuroscience, https://doi.org/10.3389/fnint.2012.00077

Download references

Acknowledgements

We thank everyone involved in this project. To CONACYT, Jacobs Foundation and CERES Network. We also thank Concepción Valdez, Oscar Peña y Armando Beltran for her support in this work.

Author information

Author notes

    Authors and Affiliations

    1. Computer Science, CICESE, Ensenada, Baja California, Mexico

      Ivonne Monarca & Monica Tentori

    2. Dale E. and Sarah Ann Fowler School of Engineering, Chapman University, Orange, CA, USA

      Franceli L. Cibrian

    Authors
    1. Ivonne Monarca
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Monica Tentori
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Franceli L. Cibrian
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Franceli L. Cibrian.

    Ethics declarations

    Conflict of interest

    The authors declare no competing interests.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Monica Tentori and Franceli L. Cibrian contributed equally to this work.

    Rights and permissions

    Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

    Reprints and Permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Monarca, I., Tentori, M. & Cibrian, F.L. Understanding the musical interaction of children with autism spectrum disorder using elastic display. Pers Ubiquit Comput (2023). https://doi.org/10.1007/s00779-022-01703-y

    Download citation

    • Received:

    • Accepted:

    • Published:

    • DOI: https://doi.org/10.1007/s00779-022-01703-y

    Keywords

    • Musical interaction
    • Elastic displays
    • Autism spectrum disorder