Assessing Agreement in Human-Robot Dialogue Strategies: A Tale of Two Wizards

  • Matthew Marge
  • Claire Bonial
  • Kimberly A. Pollard
  • Ron Artstein
  • Brendan Byrne
  • Susan G. Hill
  • Clare Voss
  • David Traum
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10011)

Abstract

The Wizard-of-Oz (WOz) method is a common experimental technique in virtual agent and human-robot dialogue research for eliciting natural communicative behavior from human partners when full autonomy is not yet possible. For the first phase of our research reported here, wizards play the role of dialogue manager, acting as a robot’s dialogue processing. We describe a novel step within WOz methodology that incorporates two wizards and control sessions: the wizards function much like corpus annotators, being asked to make independent judgments on how the robot should respond when receiving the same verbal commands in separate trials. We show that inter-wizard discussion after the control sessions and the resolution with a reconciled protocol for the follow-on pilot sessions successfully impacts wizard behaviors and significantly aligns their strategies. We conclude that, without control sessions, we would have been unlikely to achieve both the natural diversity of expression that comes with multiple wizards and a better protocol for modeling an automated system.

Keywords

Natural language dialogue Human-robot communication 

References

  1. 1.
    DeVault, D., Artstein, R., Benn, G., Dey, T., Fast, E., Gainer, A., Georgila, K., Gratch, J., Hartholt, A., Lhommet, M., et al.: SimSensei Kiosk: a virtual human interviewer for healthcare decision support. In: Proceedings of AAMAS (2014)Google Scholar
  2. 2.
    Marge, M., Bonial, C., Byrne, B., Cassidy, T., Evans, A.W., Hill, S.G., Voss, C.: Applying the Wizard-of-Oz technique to multimodal human-robot dialogue. In: Proceedings of RO-MAN (2016)Google Scholar
  3. 3.
    Passonneau, R.: Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In: Proceedings of LREC (2006)Google Scholar
  4. 4.
    Roque, A., Leuski, A., Rangarajan, V., Robinson, S., Vaswani, A., Narayanan, S., Traum, D.: Radiobot-CFF: a spoken dialogue system for military training. In: Proceedings of Interspeech (2006)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Matthew Marge
    • 1
  • Claire Bonial
    • 1
  • Kimberly A. Pollard
    • 1
  • Ron Artstein
    • 2
  • Brendan Byrne
    • 1
  • Susan G. Hill
    • 1
  • Clare Voss
    • 1
  • David Traum
    • 2
  1. 1.U.S. Army Research LaboratoryAdelphiUSA
  2. 2.USC Institute for Creative TechnologiesPlaya VistaUSA

Personalised recommendations