Assistive Robot Multi-modal Interaction with Augmented 3D Vision and Dialogue

  • Juan G. Victores
  • Félix R. Cañadillas
  • Santiago Morante
  • Alberto Jardón
  • Carlos Balaguer
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 252)


This paper presents a multi-modal interface for interaction between people with physical disabilities and an assistive robot. This interaction is performed through a dialogue mechanism and augmented 3D vision glasses to provide visual assistance to an end user commanding an assistive robot to perform Daily Life Activities (DLAs). The augmented 3D vision glasses may provide augmented reality vision of menus and information dialogues over the view of the real world, or in a simulator environment for laboratory tests and user evaluation. The actual dialogue is implemented as a finite state machine, and includes possibilities of Automatic Speech Recognition (ASR), and a Text-to-Speech (TTS) converter. The final study focuses on studying the effectiveness of these visual and auditory aids for enabling the end user to command the assistive robot ASIBOT to perform a given task.


assistive robotics end-user development human-robot interaction multi-modal interaction augmented reality speech recognition 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Huete, A.J., Victores, J.G., Martinez, S., Giménez, A., Balaguer, C.: Personal autonomy rehabilitation in home environments by a portable assistive robot. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 42(4), 561–570 (2012)CrossRefGoogle Scholar
  2. 2.
    Azuma, R.T.: A survey of augmented reality. Presence 6(4), 355–385 (1997)Google Scholar
  3. 3.
    Ronald, T.: Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon Julier, and Blair MacIntyre. Recent advances in augmented reality. IEEE Computer Graphics and Applications 21(6), 34–47 (2001)CrossRefGoogle Scholar
  4. 4.
    Van Krevelen, D.W.F., Poelman, R.: A survey of augmented reality technologies, applications and limitations. International Journal of Virtual Reality 9(2), 1 (2010)Google Scholar
  5. 5.
    Green, S.A., Billinghurst, M., Chen, X., Chase, G.: Human-robot collaboration: A literature review and augmented reality approach in design. International Journal of Advanced Robotic Systems 5(1), 1–18 (2008)Google Scholar
  6. 6.
    Green, S.A., Chase, G., Chen, X., Billinghurst, M.: Evaluating the augmented reality human-robot collaboration system. International Journal of Intelligent Systems Technologies and Applications 8(1), 130–143 (2010)CrossRefGoogle Scholar
  7. 7.
    Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., Ivkovic, M.: Augmented reality technologies, systems and applications. Multimedia Tools and Applications 51(1), 341–377 (2011)CrossRefGoogle Scholar
  8. 8.
    Milgram, P., Yin, S., Grodski, J.J.: An augmented reality based teleoperation interface for unstructured environments. In: ANS 7th Meeting on Robotics and Remote Systems, Augusta, pp. 101–123 (1997)Google Scholar
  9. 9.
    Fang, H.C., Ong, S.K., Nee, A.Y.C.: Interactive robot trajectory planning and simulation using augmented reality. Robotics and Computer-Integrated Manufacturing 28(2), 227–237 (2012)CrossRefGoogle Scholar
  10. 10.
    Tang, S.-L., Kwoh, C.-K., Teo, M.-Y., Sing, N.W., Ling, K.-V.: Augmented reality systems for medical applications. IEEE Engineering in Medicine and Biology Magazine 17(3), 49–58 (1998)CrossRefGoogle Scholar
  11. 11.
    Su, L.-M., Vagvolgyi, B.P., Agarwal, R., Reiley, C.E., Taylor, R.H., Hager, G.D.: Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3d-ct to stereoscopic video registration. Urology 73(4), 896–900 (2009)CrossRefGoogle Scholar
  12. 12.
    Luo, X., Kline, T., Fischer, H.C., Stubblefield, K.A., Kenyon, R.V., Kamper, D.G.: Integration of augmented reality and assistive devices for post-stroke hand opening rehabilitation. In: 27th Annual International Conference of the Engineering in Medicine and Biology Society, IEEE-EMBS 2005, pp. 6855–6858. IEEE (2005)Google Scholar
  13. 13.
    Braga, R.A., Petry, M., Moreira, A.P., Reis, L.P.: A development platform for intelligent wheelchairs for disabled people. In: International Conference on Informatics in Control, Automation and Robotics, pp. 115–121 (2008)Google Scholar
  14. 14.
    Alonso-Martin, F., Salichs, M.A.: Integration of a voice recognition system in a social robot. Cybernetics and Systems: An International Journal 42(4), 215–245 (2011)CrossRefGoogle Scholar
  15. 15.
    Foster, M.E., By, T., Rickert, M., Knoll, A.: Human-robot dialogue for joint construction tasks. In: Proceedings of the 8th International Conference on Multimodal Interfaces, pp. 68–71. ACM (2006)Google Scholar
  16. 16.
    Seabra Lopes, L., Teixeira, A.: Human-robot interaction through spoken language dialogue. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2000, pp. 528–534. IEEE (2000)Google Scholar
  17. 17.
    Becker, E., Le, Z., Park, K., Lin, Y., Makedon, F.: Event-based experiments in an assistive environment using wireless sensor networks and voice recognition. In: Proceedings of the 2nd International Conference on PErvasive Technologies Related to Assistive Environments, p. 17. ACM (2009)Google Scholar
  18. 18.
    Fitzpatrick, P., Metta, G., Natale, L.: Towards long-lived robot genes. Robotics and Autonomous Systems 56(1), 29–45 (2008)CrossRefGoogle Scholar
  19. 19.
    Morante, S.: Interfaz y librera para visión artificial, navegación y seguimiento en robótica. Undergraduate’s honors thesis, Universidad Carlos III de Madrid, Dpto. Ing. Sistemas y Automática (June 2012)Google Scholar
  20. 20.
    Victores, J.G., Morante, S., Jardón, A., Balaguer, C.: Give me the red can: Assistive robot task creation through multi-modal interaction. In: V Congreso Internacional de Diseño, Redes de Investigación y Tecnología para todos, DRT4ALL (September 2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Juan G. Victores
    • 1
  • Félix R. Cañadillas
    • 1
  • Santiago Morante
    • 1
  • Alberto Jardón
    • 1
  • Carlos Balaguer
    • 1
  1. 1.Robotics Lab Research Group within the Department of Systems Engineering and AutomationUniversidad Carlos III de Madrid (UC3M)MadridSpain

Personalised recommendations