Skip to main content

Advertisement

SpringerLink
Log in

Object interaction and task programming by demonstration in visuo-haptic augmented reality

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

A visuo-haptic augmented reality system is presented for object manipulation and task learning from human demonstration. The proposed system consists of a desktop augmented reality setup where users operate a haptic device for object interaction. Users of the haptic device are not co-located with the environment where real objects are present. A three degrees of freedom haptic device, providing force feedback, is adopted for object interaction by pushing, selection, translation and rotation. The system also supports physics-based animation of rigid bodies. Virtual objects are simulated in a physically plausible manner and seem to coexist with real objects in the augmented reality space. Algorithms for calibration, object recognition, registration and haptic rendering have been developed. Automatic model-based object recognition and registration are performed from 3D range data acquired by a moving laser scanner mounted on a robot arm. Several experiments have been performed to evaluate the augmented reality system in both single-user and collaborative tasks. Moreover, the potential of the system for programming robot manipulation tasks by demonstration is investigated. Experiments show that a precedence graph, encoding the sequential structure of the task, can be successfully extracted from multiple user demonstrations and that the learned task can be executed by a robot system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  1. Adcock, M., Hutchins, M., Gunn, C.: Augmented reality haptics: using ARToolKit for display of haptic applications. In: IEEE international augmented reality toolkit workshop (2003)

  2. Akinbiyi, T., Reiley, C.E., Saha, S., Burschka, D., Hasser, C.J., Yuh, D.D., Okamura, A.M.: Dynamic augmented reality for sensory substitution in robot-assisted surgical systems. In: Annual international conference of the IEEE engineering in medicine and biology society, (EMBS) (2006)

  3. Aleotti, J., Caselli, S.: Part-based robot grasp planning from human demonstration. In: IEEE international conference on robotics and automation (ICRA), pp. 4554–4560 (2011)

  4. Aleotti, J., Denaro, F., Caselli, S.: Object manipulation in visuo-haptic augmented reality with physics-based animation. In: IEEE international symposium on robot and human interactive communication (ROMAN), pp. 38–43 (2010)

  5. Aleotti, J., Rizzini, D.L., Caselli, S.: Object categorization and grasping by parts from range scan data. In: IEEE international conference on robotics and automation (ICRA) (2012)

  6. Aleotti, J., Micconi, G., Caselli, S.: Programming manipulation tasks by demonstration in visuo-haptic augmented reality. In: IEEE international symposium on haptic, audio and visual environments and games (HAVE) (2014)

  7. Argall, B.D., Chernova, S., Veloso, M., Browning, B.: A survey of robot learning from demonstration. Robot. Auton. Syst. 57(5), 469–483 (2009)

    Article  Google Scholar 

  8. Bayart, B., Didier, J.Y., Kheddar, A.: Force feedback virtual painting on real objects: a paradigm of augmented reality haptics. In: EuroHaptics, pp. 776–785 (2008)

  9. Bianchi, G., Knoerlein, B., Szekely, G., Harders, M.: High precision augmented reality haptics. In: Proceedings of EuroHaptics (2006)

  10. Billard, A., Calinon, S., Dillmann, R., Schaal, S.: Robot Programming by Demonstration. In: Siciliano, B., Oussama, K. (eds.) Springer Handbook of Robotics. Springer, Berlin, Heidelberg (2008)

    Google Scholar 

  11. Changhun, C., Kwanghee, K.: Introduction of physics simulation in augmented reality. In: International symposium on ubiquitous virtual reality, (ISUVR) (2008)

  12. Choi, S.W., Kim, H.C., Kang, H.S., Kim, S., Choi, J.: A haptic augmented reality surgeon console for a laparoscopic surgery robot system. In: 13th international conference on control, automation and systems (ICCAS), pp. 355–357 (2013)

  13. Chong, J.W.S., Ong, S.K., Nee, A.Y.C., Youcef-Youmi, K.: Robot programming using augmented reality: an interactive method for planning collision-free paths. Robot Comput Integr Manuf 25(3), 689–701 (2009)

    Article  Google Scholar 

  14. Coles, T.R., John, N., Gould, D.A., Caldwell, D.G.: Integrating haptics with augmented reality in a femoral palpation and needle insertion training simulation. IEEE Trans. Haptics 4(3), 199–209 (2011)

    Article  Google Scholar 

  15. Collett, T.H.J., MacDonald, B.A.: Augmented reality visualisation for player. In: IEEE international conference on robotics and automation, (ICRA) (2006)

  16. Comport, A.I., Marchand, E., Pressigout, M., Chaumette, F.: Real-time markerless tracking for augmented reality: the virtual visual servoing framework. IEEE Trans. Vis. Comput. Graph 12(4), 615–628 (2006)

    Article  Google Scholar 

  17. Cosco, F., Garre, C., Bruno, F., Muzzupappa, M., Otaduy, M.A.: Visuo-haptic mixed reality with unobstructed tool-hand integration. IEEE Trans. Visual Comput. Graph. 19(1), 159–172 (2013)

    Article  Google Scholar 

  18. Fang, H.C., Ong, S.K., Nee, A.Y.-C.: Robot programming using augmented reality. In: International conference on cyberWorlds, pp. 13–20 (2009)

  19. Fang, H.C., Ong, S.K., Nee, A.Y.C.: Interactive robot trajectory planning and simulation using augmented reality. Robot Comput Integr Manuf 28(2), 227–237 (2012)

    Article  Google Scholar 

  20. Giesler, B., Salb, T., Steinhaus, P., Dillmann, R.: Using augmented reality to interact with an autonomous mobile platform. In: IEEE international conference on robotics and automation, (ICRA), New Orleans, USA (2004)

  21. Harders, M., Bianchi, G., Knoerlein, B., Szekely, G.: Calibration, registration, and synchronization for high precision augmented reality haptics. IEEE Trans. Visual Comput. Graph. 15(1), 138–149 (2009)

    Article  Google Scholar 

  22. Hulin, T., Schmirgel, V., Yechiam, E., Zimmermann, U.E., Preusche, C., Pöhler, G.: Evaluating exemplary training accelerators for programming-by-demonstration. In: IEEE international symposium on robot and human interactive communication (ROMAN), pp. 440–445 (2010)

  23. Jeon, S., Choi, S.: Haptic augmented reality: taxonomy and an example of stiffness modulation. Presence. Teleoper. Virtual Environ. 18(5), 387–408 (2009)

    Article  Google Scholar 

  24. Jeon, S., Harders, M.: Extending haptic augmented reality: modulating stiffness during two-point squeezing. In: IEEE haptics symposium, pp. 141–146 (2012)

  25. Kato, H., Billinghurst, M.: Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In: Proceedings of the 2nd international workshop on augmented reality (IWAR) (1999)

  26. Kim, J., Janabi-Sharifi, F., Jung, K.: A physically-based haptic rendering for telemanipulation with visual information: macro and micro applications. In: IEEE/RSJ international conference on intelligent robots and systems, (IROS), Nice (2008)

  27. Kron, A., Schmidt, G., Petzold, B., Zah, M.I., Hinterseer, P., Steinbach, E.: Disposal of explosive ordnances by use of a bimanual haptic telepresence system. In: IEEE international conference on robotics and automation, (ICRA) (2004)

  28. Kuroda, Y., Oshiro, O.: Visual and haptic augmentation technologies for surgical navigation. In: International symposium on micro-nanomechatronics and human science (MHS), pp. 71–76 (2010)

  29. Lecuyer, A., Kheddar, A., Coquillart, S., Graux, L., Coiffet, P.: A haptic prototype for the simulations of aeronautics mounting/unmounting operations. In: IEEE international workshop on robot and human interactive communication (ROMAN), pp. 182–187 (2001)

  30. Liu, D.S.-M., Yung, C.-H., Chung, C.-H.: A physics-based augmented reality jenga stacking game. In: Workshop on digital media and digital content management (DMDCM), pp 1–8 (2011)

  31. Lok, B., Naik, S., Whitton, M., Brooks, F.P.: Incorporating dynamic real objects into immersive virtual environments. ACM Trans. Graph. 22(3), 701–701 (2003)

    Article  Google Scholar 

  32. Ma, D., Gausemeier, J., Fan, X., Grafe, M.: Virtual Reality and Augmented Reality in Industry. Springer, New York (2011)

    Book  Google Scholar 

  33. Martorella, L., Di Pietro, G., Bergamasco, M.: Hand: a haptic system for analysis and driving of hand movements in augmented reality environment. In: IEEE international workshop on robot and human interactive communication (ROMAN) (2003)

  34. Murakami, K., Kiyama, R., Narumi, T., Tanikawa, T., Hirose, M.: Poster: a wearable augmented reality system with haptic feedback and its performance in virtual assembly tasks. In: IEEE symposium on 3D user interfaces (3DUI), pp. 161–162 (2013)

  35. Nicolau, S.A., Goffin, L., Soler, L.: A low cost and accurate guidance system for laparoscopic surgery: validation on an abdominal phantom. In: ACM symposium on Virtual reality software and technology, Monterey, CA (2005)

  36. Nunez, R., Bandera, J.R., Perez-Lorenzo, J.M., Sandoval, F.: A human–robot interaction system for navigation supervision based on augmented reality. In: IEEE mediterranean electrotechnical conference, (MELECON), Malaga, Spain (2006)

  37. Pathomaree, N., Charoenseang, S.: Augmented reality for skill transfer in assembly task. In: IEEE international workshop on robot and human interactive communication (ROMAN) (2005)

  38. Pichler, A., Ankerl, M.: User centered framework for intuitive robot programming. In: IEEE international workshop on robotic and sensors environments (ROSE) pp. 1–6 (2010)

  39. Pilet, J., Lepetit, V., Fua, P.: Fast Non-Rigid Surface Detection, Registration and Realistic Augmentation. Int. J. Comput. Vision 76(2), 109–122 (2007)

    Article  Google Scholar 

  40. Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (FPFH) for 3D registration. In: IEEE international conference on robotics and automation (ICRA), pp. 3212–3217 (2009)

  41. Sandor, C., Uchiyama, S., Yamamoto, H.: Visuo-haptic systems: half-mirrors considered harmful. In: Second joint eurohaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems (WHC’07), pp. 292–297 (2007)

  42. Scheggi, S., Salvietti, G., Prattichizzo, D.: Shape and weight rendering for haptic augmented reality. In: IEEE international symposium on robot and human interactive communication (ROMAN), pp. 44–49 (2010)

  43. Sutherland, C., Hashtrudi-Zaad, K., Sellens, R., Abolmaesumi, P., Mousavi, P.: An augmented reality haptic training simulator for spinal needle procedures. IEEE Trans. Biomed. Eng. 60(11), 3009–3018 (2013)

    Article  Google Scholar 

  44. Vallino, J., Brown, C.: Haptics in augmented reality. In: IEEE international conference on multimedia computing and systems, Florence, Italy (1999)

  45. Wang, D., Zhang, Y., Zhou, W., Zhao, H., Chen, Z.: Collocation accuracy of visuo-haptic system: metrics and calibration. IEEE Trans. Haptics 4(4), 321–326 (2011)

    Article  Google Scholar 

  46. Ye, G. , Corso, J.J., Hager, G.D., Okamura, A.M.: VisHap: augmented reality combining haptics and vision. In: IEEE international conference on systems, man and cybernetics (2003)

  47. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)

    Article  Google Scholar 

  48. Zöllner, R., Pardowitz, M., Knoop, S., Dillmann, R.: Towards cognitive robots: building hierarchical task representations of manipulations from human demonstration. In: IEEE international conference on robotics and automation, pp 1535–1540 (2005)

Download references

Acknowledgments

This work was partially supported by project XDrone, flagship project “Factory of the Future”, coordinated by the Italian National Research Council (CNR).

Author information

Authors and Affiliations

  1. Dipartimento di Ingegneria dell’Informazione, University of Parma, Parma, Italy

    Jacopo Aleotti, Giorgio Micconi & Stefano Caselli

Authors
  1. Jacopo Aleotti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giorgio Micconi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefano Caselli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jacopo Aleotti.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aleotti, J., Micconi, G. & Caselli, S. Object interaction and task programming by demonstration in visuo-haptic augmented reality. Multimedia Systems 22, 675–691 (2016). https://doi.org/10.1007/s00530-015-0488-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-015-0488-z

Keywords

Search

Navigation