Mobile Health pp 219-238 | Cite as

Paralinguistic Analysis of Children’s Speech in Natural Environments

  • Hrishikesh Rao
  • Mark A. Clements
  • Yin Li
  • Meghan R. Swanson
  • Joseph Piven
  • Daniel S. Messinger


Paralinguistic cues are the non-phonemic aspects of human speech that convey information about the affective state of the speaker. In children’s speech, these events are also important markers for the detection of early developmental disorders. Detecting these events in hours of audio data would be beneficial for clinicians to analyze the social behaviors of children. The chapter focuses on the use of spectral and prosodic baseline acoustic features to classify instances of children’s laughter and fussing/crying while interacting with their caregivers in naturalistic settings. In conjunction with baseline features, long-term intensity-based features, that capture the periodic structure of laughter, enable in detecting instances of laughter to a reasonably high degree of accuracy in a variety of classification tasks.



This work was supported by funds from NSF Award 1029035, “Computational Behavioral Science: Modeling, Analysis, and Visualization of Social and Communicative Behavior”. The work was also supported by an Autism Center of Excellence grant (NIMH and NICHD HD055741 and HD055741-S1(J. Piven); the LENA Research Foundation (JP); and the Participant Registry Core of the UNC IDDRC (NICHD U54 EB005149 to JP). Dr. Swanson was supported by a National Research Service Award (T32-HD40127) from NICHD (JP). Portions of this work were also supported by an NIGMS grant (1R01GM105004), “Modeling the Dynamics of Early Communication and Development”.


  1. 1.
    Ainsworth, M., Blehar, M., Waters, E., Wall, S.: Patterns of attachment. hills-dale. NJ Eribaum (1978)Google Scholar
  2. 2.
    Apple, W., Streeter, L.A., Krauss, R.M.: Effects of pitch and speech rate on personal attributions. Journal of Personality and Social Psychology 37(5), 715 (1979)CrossRefGoogle Scholar
  3. 3.
    Bachorowski, J.A., Smoski, M.J., Owren, M.J.: The acoustic features of human laughter. The Journal of the Acoustical Society of America 110(3), 1581–1597 (2001)CrossRefGoogle Scholar
  4. 4.
    Darwin, C.: The expression of the emotions in man and animals. Oxford University Press (2002)Google Scholar
  5. 5.
    Esposito, G., Venuti, P.: Comparative analysis of crying in children with autism, developmental delays, and typical development. Focus on Autism and Other Developmental Disabilities 24(4), 240–247 (2009)CrossRefGoogle Scholar
  6. 6.
    Estes, A., Zwaigenbaum, L., Gu, H., John, T.S., Paterson, S., Elison, J.T., Hazlett, H., Botteron, K., Dager, S.R., Schultz, R.T., et al.: Behavioral, cognitive, and adaptive development in infants with autism spectrum disorder in the first 2 years of life. Journal of neurodevelopmental disorders 7(1), 1 (2015)CrossRefGoogle Scholar
  7. 7.
    Eyben, F., Wöllmer, M., Schuller, B.: Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the International Conference on Multimedia, pp. 1459–1462. ACM (2010)Google Scholar
  8. 8.
    Hess, U., Bourgeois, P.: You smile–i smile: Emotion expression in social interaction. Biological psychology 84(3), 514–520 (2010)CrossRefGoogle Scholar
  9. 9.
    Hudenko, W.J., Stone, W., Bachorowski, J.A.: Laughter differs in children with autism: an acoustic analysis of laughs produced by children with and without the disorder. Journal of Autism and Developmental Disorders 39(10), 1392–1400 (2009)CrossRefGoogle Scholar
  10. 10.
    Kim, Y., Lee, H., Provost, E.M.: Deep learning for robust feature generation in audiovisual emotion recognition. In: Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 3687–3691. IEEE (2013)Google Scholar
  11. 11.
    Kraut, R.E., Johnston, R.E.: Social and emotional messages of smiling: An ethological approach. Journal of personality and social psychology 37(9), 1539 (1979)CrossRefGoogle Scholar
  12. 12.
    Lockard, J., Fahrenbruch, C., Smith, J., Morgan, C.: Smiling and laughter: Different phyletic origins? Bulletin of the Psychonomic Society 10(3), 183–186 (1977)CrossRefGoogle Scholar
  13. 13.
    Meadows, C.: Psychological Experiences of Joy and Emotional Fulfillment. Routledge (2013)Google Scholar
  14. 14.
    Mehu, M., Dunbar, R.I.: Relationship between smiling and laughter in humans (homo sapiens): Testing the power asymmetry hypothesis. Folia Primatologica 79(5), 269–280 (2008)CrossRefGoogle Scholar
  15. 15.
    Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: Proceedings of the 28th international conference on machine learning (ICML-11), pp. 689–696 (2011)Google Scholar
  16. 16.
    Oh, J., Cho, E., Slaney, M.: Characteristic contours of syllabic-level units in laughter. In: INTERSPEECH, pp. 158–162 (2013)Google Scholar
  17. 17.
    Oller, D.K., Niyogi, P., Gray, S., Richards, J.A., Gilkerson, J., Xu, D., Yapanel, U., Warren, S.F.: Automated vocal analysis of naturalistic recordings from children with autism, language delay, and typical development. Proceedings of the National Academy of Sciences 107(30), 13,354–13,359 (2010)CrossRefGoogle Scholar
  18. 18.
    Orozco, J., García, C.A.R.: Detecting pathologies from infant cry applying scaled conjugate gradient neural networks. In: European Symposium on Artificial Neural Networks, Bruges (Belgium), pp. 349–354 (2003)Google Scholar
  19. 19.
    Petridis, S., Martinez, B., Pantic, M.: The mahnob laughter database. Image and Vision Computing 31(2), 186–202 (2013)CrossRefGoogle Scholar
  20. 20.
    Poyatos, F.: Paralanguage: A Linguistic and Interdisciplinary Approach to Interactive Speech and Sounds, vol. 92. John Benjamins Publishing (1993)Google Scholar
  21. 21.
    Prince E.B., C.A.G.D.M.K.R.A.R.J.R.J., Messinger, D.: Automated measurement of dyadic interaction predicts expert ratings of attachment in the strange situation. Association for Psychological Science Annual Convention (2015)Google Scholar
  22. 22.
    Ranganath, R., Jurafsky, D., McFarland, D.A.: Detecting friendly, flirtatious, awkward, and assertive speech in speed-dates. Computer Speech & Language 27(1), 89–115 (2013)Google Scholar
  23. 23.
    Rehg, J., Abowd, G., Rozga, A., Romero, M., Clements, M., Scalaroff, S., Essa, I., Ousley, O., Li, Y., Kim, C.H., Rao, H., Kim, J., Presti, L., Zhang, J., Lantsman, D.,, Bidwell, J., Ye, Z.: Decoding children’s social behavior. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013. IEEE (2013)Google Scholar
  24. 24.
    Rothbart, M.K.: Laughter in young children. Psychological bulletin 80(3), 247 (1973)CrossRefGoogle Scholar
  25. 25.
    Tepperman, J., Traum, D., Narayanan, S.: “yeah right”: Sarcasm recognition for spoken dialogue systems. In: Ninth International Conference on Spoken Language Processing (2006)Google Scholar
  26. 26.
    Waters, E.: The reliability and stability of individual differences in infant-mother attachment. Child Development pp. 483–494 (1978)Google Scholar
  27. 27.
    Wolff, J.J., Gu, H., Gerig, G., Elison, J.T., Styner, M., Gouttard, S., Botteron, K.N., Dager, S.R., Dawson, G., Estes, A.M., et al.: Differences in white matter fiber tract development present from 6 to 24 months in infants with autism. American Journal of Psychiatry (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Hrishikesh Rao
    • 1
  • Mark A. Clements
    • 1
  • Yin Li
    • 1
  • Meghan R. Swanson
    • 2
  • Joseph Piven
    • 2
  • Daniel S. Messinger
    • 3
  1. 1.Georgia Institute of TechnologyAtlantaUSA
  2. 2.University of North Carolina at Chapel HillChapel HillUSA
  3. 3.University of MiamiCoral GablesUSA

Personalised recommendations