Journal of Nonverbal Behavior

, Volume 38, Issue 1, pp 53–66 | Cite as

Audiovisual Correlates of Interrogativity: A Comparative Analysis of Catalan and Dutch

  • Joan Borràs-ComesEmail author
  • Constantijn Kaland
  • Pilar Prieto
  • Marc Swerts
Original Paper


Languages employ different strategies to mark an utterance as a polar (yes–no) question, including syntax, intonation and gestures. This study analyzes the production and perception of information-seeking questions and broad focus statements in Dutch and Catalan. These languages use intonation for marking questionhood, but Dutch also exploits syntactic variation for this purpose. A production task revealed the expected language-specific auditory differences, but also showed that gaze and eyebrow-raising are used in this distinction. A follow-up perception experiment revealed that perceivers relied greatly on auditory information in determining whether an utterance is a question or a statement, but accuracy was further enhanced when visual information was added. Finally, the study demonstrates that the concentration of several response-mobilizing cues in a sentence is positively correlated with the perceivers’ ratings of these utterances as interrogatives.


Information-seeking questions Speech perception Gaze Dutch Catalan 



We thank Suleman Shahid for his help in setting up the recording sessions, and Igor Jauk for his help in labeling part of the Catalan corpus. We also thank the audiences of The 13th Conference on Laboratory Phonology and The 5th European Conference on Tone and Intonation. This research has been funded by two research grants awarded by the Spanish Ministerio de Ciencia e Innovación, namely HUM2006-13295-C02-01/FILO, FFI2009-07648/FILO and Consolider-Ingenio 2010 Program CSD2007-00012, by a grant awarded by the Generalitat de Catalunya to the Grup d’Estudis de Prosòdia (2009SGR-701), and by a “Study abroad scholarship for research outside of Catalunya” 2010 BE1 00207, awarded by the Generalitat de Catalunya.


  1. Ahmad, M. I., Tariq, H., Saeed, M., Shahid, S., & Krahmer, E. (2011). Guess who? An interactive and entertaining game-like platform for investigating human emotions. In Jacko, J. A. (Ed.), Human-computer interaction. Towards mobile and intelligent interaction environments (Vol. 3, pp. 543–551). Lecture Notes in Computer Science, 6763. Berlin: Springer.Google Scholar
  2. Argyle, M., & Cook, M. (1976). Gaze and mutual gaze. Cambridge: Cambridge University Press.Google Scholar
  3. Bolinger, D. L. (1989). Intonation and its uses: Melody in grammar and discourse. London: Edward Arnold.Google Scholar
  4. Borràs-Comes, J., & Prieto, P. (2011). ‘Seeing tunes’. The role of visual gestures in tune interpretation. Journal of Laboratory Phonology, 2(2), 355–380.Google Scholar
  5. Borràs-Comes, J., Puglesi, C., & Prieto, P. (2011). Audiovisual competition in the perception of counter-expectational questions. In Salvi, G., Beskow, J., Engwall, O., & Al Moubayed, S. (Eds.), Proceedings of the 11th international conference on auditory-visual speech processing (pp. 43–46). Volterra, Italy: KTH Royal Institute of Technology.Google Scholar
  6. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.CrossRefGoogle Scholar
  7. Cosnier, J. (1991). Les gestes de la question. In C. Kerbrat-Orecchioni (Ed.), la question (pp. 163–171). Lyon: Presses Universitaires de Lyon.Google Scholar
  8. Crespo-Sendra, V., Kaland, C., Swerts, M., & Prieto, P. (2013). Perceiving incredulity: The role of intonation and facial gestures. Journal of Pragmatics, 47, 1–13.CrossRefGoogle Scholar
  9. Cruttenden, A. (1981). Falls and rises: Meanings and universals. Journal of Linguistics, 17(1), 77–91.CrossRefGoogle Scholar
  10. Dryer, M. S. (2008). Polar questions. In Haspelmath, M., Dryer, M. S., Gil, D., & Comrie, B. (Eds.), The world atlas of language structures online (Chapter 116). Munich: Max Planck Digital Library. Retrieved from:
  11. Duncan, S., & Fiske, D. W. (1977). Face-to-face interaction: Research, methods, and theory. New York: Wiley.Google Scholar
  12. Englert, C. (2010). Questions and responses in Dutch conversations. Journal of Pragmatics, 42(10), 2666–2684.CrossRefGoogle Scholar
  13. Flecha-García, M. L. (2010). Eyebrow raises in dialogue and their relation to discourse structure, utterance function and pitch accents in English. Speech Communication, 52, 542–554.CrossRefGoogle Scholar
  14. Geluykens, R. (1988). On the myth of rising intonation in polar questions. Journal of Pragmatics, 12, 467–485.CrossRefGoogle Scholar
  15. Haan, J. (2002). Speaking of questions. An exploration of Dutch question intonation. LOT Dissertation Series, 52. Utrecht: LOT.Google Scholar
  16. House, D. (2002). Perception of question intonation and facial gestures. Fonetik, 44(1), 41–44.Google Scholar
  17. Kendon, A. (1967). Some functions of gaze direction in social interaction. Acta Psychologica, 26, 22–63.PubMedCrossRefGoogle Scholar
  18. Kendon, A. (1990). Conducting interaction: Patterns of behavior in focused encounters. New York: Cambridge University Press.Google Scholar
  19. Labov, W., & Fanshel, D. (1977). Therapeutic discourse. New York: Academic Press.Google Scholar
  20. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.PubMedCrossRefGoogle Scholar
  21. Levinson, S. C. (2010). Questions and responses in Yélî Dnye, the Papuan language of Rossel Island. Journal of Pragmatics, 42(10), 2741–2755.CrossRefGoogle Scholar
  22. Lysander, K., & Horton, W. S. (2012). Conversational grounding in younger and older adults: The effect of partner visibility and referent abstractness in task-oriented dialogue. Discourse Processes, 49(1), 29–60.CrossRefGoogle Scholar
  23. Peters, C., Pelachaud, C., Bevacqua, E., Mancini, M., & Poggi, I. (2005). A model of attention and interest using gaze behavior. In T. Panayiotopoulos, J. Gratch, R. Aylett, D. Ballin, P. Olivier, & T. Rist (Eds.), Intelligent virtual agents (Vol. 3661, pp. 229–240). Lecture Notes in Computer Science. London: Springer.Google Scholar
  24. Pomerantz, A. M. (1980). Telling my side: “Limited access” as a “fishing” device. Sociological Inquiry, 50, 186–198.CrossRefGoogle Scholar
  25. Prieto, P., Borràs-Comes, J., Crespo-Sendra, V., Roseano, P., Sichel-Bazin, R., & Vanrell, M. M. (2013). Intonational phonology of Catalan and its dialectal varieties. In S. Frota & P. Prieto (Eds.), Intonational variation in Romance. Oxford: Oxford University Press.Google Scholar
  26. Prieto, P., & Rigau, G. (2007). The syntax-prosody interface: Catalan interrogative sentences headed by que. Journal of Portuguese Linguistics, 6(2), 29–59.Google Scholar
  27. Psychology Software Tools Inc. (2009). E-Prime (version 2.0). Computer Program.Google Scholar
  28. Rialland, A. (2007). Question prosody: An African perspective. In C. Gussenhoven & T. Riad (Eds.), Tones and tunes (Vol. 2, pp. 35–62). Berlin: Mouton.Google Scholar
  29. Richardson, D. C., Dale, R., & Tomlinson, J. M. (2009). Conversation, gaze coordination, and beliefs about visual context. Cognitive Science, 33(8), 1468–1482.PubMedCrossRefGoogle Scholar
  30. Rossano, F. (2010). Questioning and responding in Italian. Journal of Pragmatics, 42(10), 2756–2771.CrossRefGoogle Scholar
  31. Snow, D., & Balog, H. L. (2002). Do children produce the melody before the words? A review of developmental intonation research. Lingua, 112, 1025–1058.CrossRefGoogle Scholar
  32. Srinivasan, R. J., & Massaro, D. W. (2003). Perceiving prosody from the face and voice. Distinguishing statements from echoic questions in English. Language and Speech, 46, 1–22.PubMedCrossRefGoogle Scholar
  33. Stivers, T. (2010). An overview of the question-response system in American English conversation. Journal of Pragmatics, 42(10), 2772–2781.CrossRefGoogle Scholar
  34. Stivers, T., & Rossano, F. (2010). Mobilizing response. Research on Language and Social Interaction, 43(1), 1–31.CrossRefGoogle Scholar
  35. Vilhjalmsson, H. H. (1997). Autonomous communicative behaviors in avatars. Unpublished Master’s thesis. Massachusetts Institute of Technology, Cambridge, MA.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Joan Borràs-Comes
    • 1
    Email author
  • Constantijn Kaland
    • 2
  • Pilar Prieto
    • 1
    • 3
  • Marc Swerts
    • 2
  1. 1.Departament de Traducció i Ciències del Llenguatge, Campus de la Comunicació, PoblenouUniversitat Pompeu FabraBarcelonaSpain
  2. 2.Tilburg UniversityTilburgThe Netherlands
  3. 3.Institució Catalana de Recerca i Estudis Avançats (ICREA)Universitat Pompeu FabraBarcelonaSpain

Personalised recommendations