Abstract
This paper introduces the HRI-CMU Corpus of Situated In-Car Interactions, a multimodal corpus of human-human interactions collected within highly sensored vehicles. The corpus consists of interactions between a driver and copilot performing tasks including navigation, scheduling and messaging. Data was captured synchronously across a wide range of sensors in the vehicle, including, near-field and far-field microphones, internal and external cameras, GPS, IMU, and OBD-II devices. The corpus is unique in that it not only contains transcribed speech, annotation of dialog acts and gestures, but also includes grounded object references and detailed discourse structure for the navigation task. We present the corpus and provide an early analysis of the data contained within. The initial analysis indicates that discourse behavior has strong variation across participants, and that general trends relate physical situation and multi-tasking to grounding behavior.
(The author is now at Lenovo Labs).
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Batliner A, Hacker C, Steidl S, Nöth E, D’Arcy S, Russell M, Wong M (2004) “you stupid tin box”-children interacting with the aibo robot: a cross-linguistic emotional speech corpus. In: LREC
Crocker M (2012) Grounding spoken interaction with real-time gaze in dynamic virtual environments. In: International conference on computational linguistics
Dan B, Horovitz E (2009) Dialog in the open-world: platform and applications. In: Proceeding of ICMI
Dey AK (2001) Situated interaction and context-aware computing, In Personal and ubiquitous computing
Eskenazi M, Lee S (2012) Pomdp-based let’s go system for spoken dialog challenge. In: Spoken language technology workshop
Gasic M (2011) On-line policy optimisation of spoken dialogue systems via live interaction with human subjects. In: Proceeding of automatic speech recognition and understanding
Giuliani M, Isard A Foster ME (2010) Situated reference in a hybrid human-robot interaction system. In: Proceeding of INLG
Joyce C, Prasov Z (2010) Fusing eye gaze with speech recognition hypotheses to resolve exophoric reference in situated dialogue. In: Proceeding of EMNLP
Kruijff GJM et al (2010) Situated dialogue processing for human-robot interaction. In: Cognitive systems
Lane I (2012) Cesar: The car environment sensor adjustable rig. Technical report, Carnegie Mellon University
Lane I, Ma Y, Raux A (2012) Immersive interaction within vehicles. In: Proceeding of spoken language technology workshop
Lim D, Soon WM, Ng HT (2001) A machine learning approach to coreference resolution of noun phrases. Comput. Linguist. 27(4):521–544
Maes P, Merrill D (2007) Augmenting looking, pointing and reaching gestures to enhance the searching and browsing of physical objects. In: Pervasive computing 5th international conference
Ma Y, Raux A, Ramachandran D, Gupta R (2012) Landmark-based location belief tracking in a spoken dialog system. In: Proceedings of the 13th annual meeting of the special interest group on discourse and dialogue, Association for Computational Linguistics, pp 169–178
Misu T et al (2013) Situated multi-modal dialog system in vehicles. In: Proceeding of ICMI
Möller S, Gödde F, Wolters M (2008) A corpus analysis of spoken smart-home interactions with older users. In: Proceeding of 6th international conference on language resources and evaluation
Muller C (2008) Fully automatic resolution of it, this and that in unrestricted multi-party dialog. Ph.D. thesis
Rekimoto J, Nagao K (1995) Ubiquitous talker: spoken language interaction with real world objects. arXiv:preprint cmp-lg/9505038
Seneff S et al (2007) Exploiting context information in spoken dialog interaction with mobile devices. In: Proceeding of international workshop on improved mobile user experience
Skantze G, Al Moubayed S (2011) Turn-taking control using gaze in multiparty human-computer dialogue: effects of 2d and 3d displays. In: Proceeding of international conference on auditory-visual speech processing
Tateishi M, Asami K, Akahori I, Judy S, Obuchi Y, Mitamura T, Nyberg E, Hataoka N (2005) A spoken dialog corpus for car telematics services. In: DSP for in-vehicle and mobile systems, Springer, pp 47–64
Tellex S, Kollar T, Dickerson S, Walter MR, Banerjee AG, Teller SJ, Roy N (2011) Understanding natural language commands for robotic navigation and mobile manipulation. In: AAAI
Tetreault J, Allen J (2006) Semantics, dialogue, and reference resolution. Rochester University Department of Computer Science,Technical Report
Thomson B (2007) Training a real-world pomdp-based dialogue system. In Proceeding of workshop on bridging the gap: academic and industrial research in dialog technologies
Traum DR (1994) A computational theory of grounding in natural language conversation. Ph.D. thesis, University of Rochester
Traum D, Rickel J (2002) Embodied agents for multi-party dialogue in immersive virtual worlds. In: Proceeding of AAMAS
Williams J (2012) A belief tracking challenge task for spoken dialog systems. In: NAACL-HLT Workshop on future directions and needs in the spoken dialog community: tools and data
Williams J, Raux A, Ramachandran D, Black A (2013) The dialog state tracking challenge. In: Proceedings of the SIGDIAL 2013 conference, Metz, France, Association for Computational Linguistics, pp 404–413
Zender H et al (2007) An integrated robotic system for spatial understanding and situated interaction in indoor environments. In: Proceeding of AAAI
Acknowledgments
This research was performed at CMU under the sponsored research agreements 26660 and 29831 with the Honda Research Institute, USA. We would like to thank Teruhisa Misu, Rakesh Gupta and Victor Ng-Thow-Hing from HRI-USA, for their useful feedback when designing, collecting and annotating this corpus.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Cohen, D., Chandrashekaran, A., Lane, I., Raux, A. (2016). The HRI-CMU Corpus of Situated In-Car Interactions. In: Rudnicky, A., Raux, A., Lane, I., Misu, T. (eds) Situated Dialog in Speech-Based Human-Computer Interaction. Signals and Communication Technology. Springer, Cham. https://doi.org/10.1007/978-3-319-21834-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-21834-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-21833-5
Online ISBN: 978-3-319-21834-2
eBook Packages: EngineeringEngineering (R0)