Skip to main content
SpringerLink
Log in

Voice Interaction for Training: Opportunities, Challenges, and Recommendations from HCI Perspective

  • Conference paper
  • First Online:
Learning and Collaboration Technologies. Human and Technology Ecosystems (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12206))

Included in the following conference series:

  • 3538 Accesses

Abstract

This paper examines voice interaction requirements in the context of learning and training. The authors reviewed relevant publications, focusing on usability and user experience with conversational interfaces, including speech and chat interfaces. We examined technology trends and limitations, HCI research on speech interfaces, usability evaluations of conversational interfaces in training, including formal evaluation methodologies, and our research projects on voice interaction for military and law enforcement training. Examples of voice technology applications in several domains are provided, along with a set of recommendations for successful design, evaluation and use of voice interaction in training applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ball, J., et al.: The synthetic teammate project. Comput. Math. Organ. Theory 16, 271–299 (2010). https://doi.org/10.1007/s10588-010-9065-3

    Article  Google Scholar 

  2. Demir, M., McNeese, N.J., Cooke, N.J., Ball, J.T., Myers, C., Friedman, M.: Synthetic teammate communication and coordination with humans. In: Proceedings of the Human Factors and Ergonomics Society, pp. 951–955 (2015). https://doi.org/10.1177/1541931215591275

  3. Jenkins, M., Wollocko, A., Negri, A., Ficthl, T.: Augmented reality and mixed reality prototypes for enhanced mission command/battle management command and control (BMC2) execution. In: Chen, J.Y.C., Fragomeni, G. (eds.) VAMR 2018. LNCS, vol. 10910, pp. 272–288. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91584-5_22

    Chapter  Google Scholar 

  4. Stedmon, A.W., Patel, H., Sharples, S.C., Wilson, J.R.: Developing speech input for virtual reality applications: a reality based interaction approach. Int. J. Hum. Comput. Stud. 69, 3–8 (2011). https://doi.org/10.1016/j.ijhcs.2010.09.002

    Article  Google Scholar 

  5. Weiss, B., Wechsung, I., Kühnel, C., Möller, S.: Evaluating embodied conversational agents in multimodal interfaces. Comput. Cogn. Sci. 1, 6 (2015). https://doi.org/10.1186/s40469-015-0006-9

    Article  Google Scholar 

  6. Cearley, D.W., Burke, B., Walker, M.J.: Top 10 strategic technology trends for 2018. Gart. Res. 10, 1–9 (2017)

    Google Scholar 

  7. Klopfenstein, L.C., Delpriori, S., Malatini, S., Bogliolo, A.: The rise of bots: a survey of conversational interfaces, patterns, and paradigms. In: Proceedings of the 2017 Conference on Designing Interactive Systems, pp. 555–565 (2017)

    Google Scholar 

  8. Goldberg, B., Cannon-Bowers, J.: Feedback source modality effects on training outcomes in a serious game: pedagogical agents make a difference. Comput. Hum. Behav. 52, 1–11 (2015). https://doi.org/10.1016/j.chb.2015.05.008

    Article  Google Scholar 

  9. Emond, B., et al.: Adaptive training simulation using speech interaction for training navy officers. In: Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), pp. 2924–2934. National Training and Simulation Association, Orlando (2016)

    Google Scholar 

  10. Emond, B., Kondratova, I., Durand, G., Valdés, J.J.: A multi-role reconfigurable trainer for naval combat information operators. In: Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), p. 14. National Training and Simulation Association, Orlando (2018)

    Google Scholar 

  11. Fournier, H., Lapointe, J.-F., Emond, B., Kondratova, I.: A multidisciplinary approach to enhancing infantry training through immersive technologies e-learning view project virtual reality interaction view project (2011)

    Google Scholar 

  12. Fournier, H., Lapointe, J.-F., Kondratova, I., Emond, B.: Crossing the barrier: a scalable simulator for course of fire training. In: Interservice/Industry Training, Simulation, and Education Conference, p. 10. National Training and Simulation Association, Orlando (2012)

    Google Scholar 

  13. Munteanu, C., Fournier, H., Lapointe, J.-F.J.-F., Kondratova, I., Emond, B.: We’ll take it from here: letting the users take charge of the evaluation and why that turned out well. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2013 Extended Abstracts on Human Factors in Computing Systems, Paris, France, pp. 2383–2384 (2013). https://doi.org/10.1145/2468356.2468778

  14. Yu, J., Wang, Z.F.: A video, text, and speech-driven realistic 3-D virtual head for human-machine interface. IEEE Trans. Cybern. 45, 977–988 (2015). https://doi.org/10.1109/TCYB.2014.2341737

    Article  Google Scholar 

  15. Hei, Z., Lvi, C., Pengi, D., Yu, D.: A speech recognition-based interaction approach applying to immersive virtual maintenance simulation (2017)

    Google Scholar 

  16. Barber, D., Wohleber, Ryan W., Parchment, A., Jentsch, F., Elliott, L.: Development of a squad level vocabulary for human-robot interaction. In: Shumaker, R., Lackey, S. (eds.) VAMR 2014. LNCS, vol. 8525, pp. 139–148. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07458-0_14

    Chapter  Google Scholar 

  17. Johnson, W.L., Lester, J.C.: Face-to-face interaction with pedagogical agents, twenty years later. Int. J. Artif. Intell. Educ. 26, 25–36 (2016). https://doi.org/10.1007/s40593-015-0065-9

    Article  Google Scholar 

  18. Traum, D., Rickel, J., Gratch, J., Marsella, S.: Negotiation over tasks in hybrid human-agent teams for simulation-based training. In: Proceedings of Second International Joint Conference Autonomous Agents and Multiagent Systems - AAMAS 2003, p. 441 (2003). https://doi.org/10.1145/860575.860646

  19. Knerr, B.W., Lampton, D.R., Thomas, M., Corner, B.D., Grosse, J.R.: Virtual environments for dismounted soldier simulation, training, and mission rehearsal: results of the FY 2002 culminating event. Army Research Inst Field Unit Orlando, FL (2003)

    Google Scholar 

  20. Candello, H., Pinhanez, C.: The role of dialogue user data in the information interaction design of conversational systems. In: Marcus, A., Wang, W. (eds.) DUXU 2018. LNCS, vol. 10919, pp. 414–426. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91803-7_31

    Chapter  Google Scholar 

  21. Filar, B., Seymour, R.J., Park, M.: Ask me anything: a conversational interface to augment information security workers. In: Symposium Usable Privacy Security (2017)

    Google Scholar 

  22. Barnes, M.J., Chen, J.Y., Hill, S.: Humans and autonomy: implications of shared decision making for military operations (2017)

    Google Scholar 

  23. Hamilton, P.L., Cooke, N.M., Brittain, R.D., Sepulveda, M., Cooke, N.M., Sepulveda, M.: simulated operational communications and coordination integration for aircrew learning (SOCIAL). In: AIAA Modeling and Simulation Technologies Conference, pp. 1–21 (2013). https://doi.org/10.2514/6.2013-5228

  24. Cooke, N.J., Demir, M., McNeese, N.: Synthetic teammates as team players: coordination of human and synthetic teammates (2016)

    Google Scholar 

  25. Clark, L.: Social boundaries of appropriate speech in HCI: a politeness perspective. In: HCI 2018, pp. 1–5. BCS Learning & Development Ltd. (2018). https://doi.org/10.14236/ewic/hci2018.76

  26. Weinschenk, S., Barker, D.T.: Designing Effective Speech Interfaces. Wiley, New York (2000)

    Google Scholar 

  27. Aylett, M.P., Vazquez-Alvarez, Y., Baillie, L.: Interactive radio: a new platform for calm computing. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 2085–2090. ACM (2015)

    Google Scholar 

  28. Munteanu, C., et al.: Designing speech, acoustic and multimodal interactions, May (2017). https://doi.org/10.1145/3027063.3027086

  29. Allison, F., Carter, M., Gibbs, M.: Word play. Games Cult. 155541201774630 (2017). https://doi.org/10.1177/1555412017746305

  30. Muller, T.J., Van Den Bosch, K., Kerbusch, P., Freulings, J.H.: LVC training in urban operation skills. In: 2011 Summer Simulation Multiconference, SummerSim 2011, Co-located with 2011 SISO European Simulation Interoperability Work, Euro SIW, pp. 115–120 (2011)

    Google Scholar 

  31. Hura, S.L.: Usability testing of spoken conversational systems. J. Usability Stud. 12, 155–163 (2017)

    Google Scholar 

  32. Latorre-Navarro, E.M., Harris, J.G.: An intelligent natural language conversational system for academic advising. Int. J. Adv. Comput. Sci. Appl. 6, 110–119 (2015)

    Google Scholar 

  33. Loddo, I., Martini, D.: The cocktail party effect. An inclusive vision of conversational interactions. Des. J. 20, S4076–S4086 (2017). https://doi.org/10.1080/14606925.2017.1352909

    Article  Google Scholar 

  34. Reeves, S.: Some conversational challenges of talking with machines. In: Talking with Conversational Agents in Collaborative Action, Workshop at the 20th ACM conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2017) (2017)

    Google Scholar 

  35. Ortiz, C.L.: The road to natural conversational speech interfaces. IEEE Internet Comput. 18, 74–78 (2014). https://doi.org/10.1109/MIC.2014.36

    Article  Google Scholar 

  36. Haggard, K.M.: Air support control officer individual position training simulation. Naval Postgraduate School Monterey United States (2017)

    Google Scholar 

  37. Khooshabeh, P., Choromanski, I., Neubauer, C., Krum, D.M., Spicer, R., Campbell, J.: Mixed reality training for tank platoon leader communication skills. In: 2017 IEEE Virtual Reality (VR), pp. 333–334. IEEE (2017). https://doi.org/10.1109/VR.2017.7892312

  38. Ilves, M.: Human responses to machine- generated speech with emotional content (2013)

    Google Scholar 

  39. Case, J.E., Twyman, N.W.: Embodied conversational agents: social or nonsocial? In: 2015 48th Hawaii International Conference on System Sciences, pp. 491–496 (2015). https://doi.org/10.1109/HICSS.2015.65

  40. Chou, C., Chan, T., Lin, C.: Redefining the learning companion: the past, present, and future of educational agents. 40, 255–269 (2003)

    Google Scholar 

  41. Krämer, N.C., Bente, G.: Personalizing e-learning. The social effects of pedagogical agents. Educ. Psychol. Rev. 22, 71–87 (2010). https://doi.org/10.1007/s10648-010-9123-x

    Article  Google Scholar 

  42. Graesser, A., McDaniel, B.: Conversational agents can provide formative assessment, constructive learning, and adaptive instruction. In: The Future of Assessment, pp. 85–112. Routledge (2017)

    Google Scholar 

  43. Mayer, R.E., Moreno, R.: A split-attention effect in multimedia learning: evidence for dual processing systems in working memory. 90, 312–320 (1998)

    Google Scholar 

  44. James-Reynolds, C., Currie, E.: EAI Endorsed Transactions Smart Feedback and the Challenges of Virtualisation. In: EAI Endorsed Transactions on Future Intelligent Educational Environments (2015). https://doi.org/10.4108/fiee.1.2.e6

  45. Harvey, P.H., Currie, E., Daryanani, P., Augusto, J.C.: Enhancing student support with a virtual assistant. In: Vincenti, G., Bucciero, A., Vaz de Carvalho, C. (eds.) eLEOT 2015. LNICST, vol. 160, pp. 101–109. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-28883-3_13

    Chapter  Google Scholar 

  46. Von der Pütten, A.M., Krämer, N.C., Gratch, J., Kang, S.-H.: It doesn’t matter what you are! Explaining social effects of agents and avatars. Comput. Hum. Behav. 26, 1641–1650 (2010)

    Article  Google Scholar 

  47. Hill, J., Randolph Ford, W., Farreras, I.G.: Real conversations with artificial intelligence: a comparison between human-human online conversations and human-chatbot conversations. Comput. Human Behav. 49, 245–250 (2015). https://doi.org/10.1016/j.chb.2015.02.026

    Article  Google Scholar 

  48. Cafaro, A., Vilhjálmsson, H.H., Bickmore, T., Heylen, D., Jóhannsdóttir, K.R., Valgarðsson, G.S.: First impressions: users’ judgments of virtual agents’ personality and interpersonal attitude in first encounters. In: Nakano, Y., Neff, M., Paiva, A., Walker, M. (eds.) IVA 2012. LNCS (LNAI), vol. 7502, pp. 67–80. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33197-8_7

    Chapter  Google Scholar 

  49. Clark, L., et al.: The state of speech in HCI: trends, themes and challenges (2018)

    Google Scholar 

  50. Pearl, C.: Designing Voice User Interfaces: Principles of Conversational Experiences. O’Reilly Media, Inc., Sebastopol (2016)

    Google Scholar 

  51. Murad, C., Munteanu, C., Clark, L., Cowan, B.R.: Design guidelines for hands-free speech interaction, August 2018. https://doi.org/10.1145/3236112.3236149

  52. Nye, B.D., Graesser, A.C., Hu, X.: AutoTutor and family: a review of 17 years of natural language tutoring. Int. J. Artif. Intell. Educ. 24, 427–469 (2014). https://doi.org/10.1007/s40593-014-0029-5

    Article  Google Scholar 

  53. Rus, V., Graesser, A.C., Hu, X., Cockroft, J.L.: Standardizing unstructured interaction data in adaptive instructional systems. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 217–226. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_18

    Chapter  Google Scholar 

  54. Freiman, M., Myers, C., Caisse, M., Halverson, T., Ball, J.: Assessing cognitive fidelity in a situation awareness process model. In: 2019 IEEE Conference on Cognitive and Computational Aspects of Situation Management, pp. 100–106 (2019). https://doi.org/10.1109/COGSIMA.2019.8724256

  55. Deriu, J., et al.: Survey on evaluation methods for dialogue systems (2019). http://arxiv.org/abs/1905.04071

  56. Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A.: PARADISE: a framework for evaluating spoken dialogue agents (1997)

    Google Scholar 

  57. Schmitt, A., Ultes, S.: Interaction quality: assessing the quality of ongoing spoken dialog interaction by experts—and how it relates to user satisfaction. Speech Commun. 74, 12–36 (2015). https://doi.org/10.1016/j.specom.2015.06.003

    Article  Google Scholar 

  58. Wolska, M., et al.: An annotated corpus of tutorial dialogs on mathematical theorem proving. In: The International Conference on Language Resources and Evaluation (LREC), pp. 1007–1010 (2004)

    Google Scholar 

  59. Serban, I.V., Lowe, R., Henderson, P., Charlin, L., Pineau, J.: A survey of available corpora for building data-driven dialogue systems: the journal version. Dialogue Discourse 9, 1–49 (2018). https://doi.org/10.5087/dad.2018.101

    Article  Google Scholar 

  60. Graesser, A.C., Chipman, P., Haynes, B.C., Olney, A.: AutoTutor: an intelligent tutoring system with mixed-initiative dialogue. IEEE Trans. Educ. 48, 612–618 (2005). https://doi.org/10.1109/TE.2005.856149

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Digital Technologies Research Centre, National Research Council Canada, Fredericton, Canada

    Irina Kondratova

  2. Digital Technologies Research Centre, National Research Council Canada, Ottawa, Canada

    Bruno Emond

Authors
  1. Irina Kondratova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bruno Emond
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Irina Kondratova .

Editor information

Editors and Affiliations

  1. Cyprus University of Technology, Limassol, Cyprus

    Panayiotis Zaphiris

  2. Cyprus University of Technology, Limassol, Cyprus

    Andri Ioannou

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kondratova, I., Emond, B. (2020). Voice Interaction for Training: Opportunities, Challenges, and Recommendations from HCI Perspective. In: Zaphiris, P., Ioannou, A. (eds) Learning and Collaboration Technologies. Human and Technology Ecosystems. HCII 2020. Lecture Notes in Computer Science(), vol 12206. Springer, Cham. https://doi.org/10.1007/978-3-030-50506-6_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-50506-6_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-50505-9

  • Online ISBN: 978-3-030-50506-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation