Skip to main content
SpringerLink
Log in

Visual and Tactile Fusion for Estimating the Pose of a Grasped Object

  • Conference paper
  • First Online:
Robot 2019: Fourth Iberian Robotics Conference (ROBOT 2019)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1093))

Included in the following conference series:

Abstract

This paper considers the problem of fusing vision and touch senses together to estimate the 6D pose of an object while it is grasped. Assuming that a textured 3D model of the object is available, first, Scale-Invariant Feature Transform (SIFT) keypoints of the object are extracted, and a Random sample consensus (RANSAC) method is used to match these features with the textured model. Then, optical flow is used to visually track the object while a grasp is performed. After the hand contacts the object, a tactile-based pose estimation is performed using a Particle Filter. During grasp stabilization and hand movement, the pose of the object is continuously tracked by fusing the visual and tactile estimations with an extended Kalman filter. The main contribution of this work is the continuous use of both sensing modalities to reduce the uncertainty of tactile sensing in those degrees of freedom in which there is no information available, as presented through the experimental validation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Lacey, S., Sathian, K.: Visuo-haptic multisensory object recognition, categorization, and representation. Front Psychol. 5, 730 (2014)

    Article  Google Scholar 

  2. Macura, Z., Cangelosi, A., Ellis, R., Bugmann, D., Fischer, M., Myachykov, A.: A cognitive robotic model of grasping. In: International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems, pp. 89–96 (2009)

    Google Scholar 

  3. Dogar, M., Hsiao, K., Ciocarlie, M., Srinivasa, S.: Physics-based grasp planning through clutter. In: Robotics: Science and Systems VIII (2012)

    Google Scholar 

  4. Vasconcelos, N., Pantoja, J., Belchior, H., Caixeta, F.V., Faber, J., Freire, M.A., Cota, V.R., de Macedo, E.A., Laplagne, D.A., Gomes, H.M., Ribeiro, S.: Cross-modal responses in the primary visual cortex encode complex objects and correlate with tactile discrimination. Proc. Natl. Acad. Sci. 108(37), 15408–15413 (2011)

    Article  Google Scholar 

  5. Petit, A., Marchand, E., Kanani, K.: A robust model-based tracker combining geometrical and color edge information. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3719–3724 (2013)

    Google Scholar 

  6. Choi, C., Christensen, H.I.: RGB-D object tracking: a particle filter approach on GPU. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1084–1091 (2013)

    Google Scholar 

  7. Vacchetti, L., Lepetit, V., Fua, P.: Stable real-time 3D tracking using online and offline information. IEEE Trans. Pattern Anal. Mach. Intell. 26, 1385–1391 (2004)

    Article  Google Scholar 

  8. Kyrki, V., Kragic, D.: Integration of model-based and model-free cues for visual object tracking in 3D. In: IEEE International Conference on Robotics and Automation, pp. 1566–1572 (2005)

    Google Scholar 

  9. Güler, P., Bekiroglu, Y., Pauwels, K., Kragic, D.: What’s in the container? Classifying object contents from vision and touch. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3961–3968 (2014)

    Google Scholar 

  10. Pauwels, K., Ivan, V., Ros, E., Vijayakumar, S.: Real-time object pose recognition and tracking with an imprecisely calibrated moving RGB-D camera. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2733–2740 (2014)

    Google Scholar 

  11. Jamali, N., Sammut, C.: Majority voting: material classification by tactile sensing using surface texture. IEEE Trans. Robot. 27(3), 508–521 (2011)

    Article  Google Scholar 

  12. Madry, M., Bo, L., Kragic, D., Fox, D.: ST-HMP: unsupervised spatio-temporal feature learning for tactile data. In: IEEE International Conference on Robotics and Automation, pp. 2262–2269 (2014)

    Google Scholar 

  13. Aggarwal, A., Kirchner, F.: Object recognition and localization: the role of tactile sensors. Sensors 14, 3227–3266 (2014)

    Article  Google Scholar 

  14. Haidacher, S., Hirzinger, G.: Estimating finger contact location and object pose from contact measurements in 3-D grasping. In: IEEE International Conference on Robotics and Automation, pp. 1805–1810 (2003)

    Google Scholar 

  15. Chalon, M., Reinecke, J., Pfanne, M.: Online in-hand object localization. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2977–2984 (2013)

    Google Scholar 

  16. Pfanne, M., Chalon, M.: EKF-based in-hand object localization from joint position and torque measurements. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2464–2470 (2017)

    Google Scholar 

  17. Álvarez, D., Roa, M.A., Moreno, L.: Tactile-based in-hand object pose estimation. In: Third Iberian Robotics Conference, ROBOT 2017, Advances in Intelligent Systems and Computing, vol. 694, pp. 716–728. Springer (2018)

    Google Scholar 

  18. Allen, P.K.: Integrating vision and touch for object recognition tasks. Int. J. Robot. Res. 7, 15–33 (1988)

    Article  Google Scholar 

  19. Ilonen, J., Bohg, J., Kyrki, V.: Three-dimensional object reconstruction of symmetric objects by fusing visual and tactile sensing. Int. J. Rob. Res. 33(2), 321–341 (2014)

    Article  Google Scholar 

  20. Alkkiomäki, O., Kyrki, V., Kälviäinen, H., Liu, Y., Handroos, H.: Complementing visual tracking of moving targets by fusion of tactile sensing. Rob. Auton. Syst. 57, 1129–1139 (2009)

    Article  Google Scholar 

  21. Kolycheva, E., Kyrki, V.: Task-specific grasping of similar objects by probabilistic fusion of vision and tactile measurements. In: IEEE-RAS International Conference on Humanoid Robots, pp. 704–710 (2015)

    Google Scholar 

  22. Zhang, M.M., Detry, R., Matthies, L., Daniilidis, K.: Tactile-vision integration for task-compatible fine-part manipulation. In: Robotics: Science and Systems. Workshop on Revisiting Contact—Turning a Problem into a Solution (2017)

    Google Scholar 

  23. Bimbo, J., Rodríguez-Jiménez, S., Liu, H., Song, X., Burrus, N., Senerivatne, L.D., Abderrahim, M., Althoefer, K.: Object pose estimation and tracking by fusing visual and tactile information. In: IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 65–70 (2012)

    Google Scholar 

  24. Bimbo, J., Seneviratne, L., Althoefer, K., Liu, H.: Combining touch and vision for the estimation of an object’s pose during manipulation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4021–4026 (2013)

    Google Scholar 

  25. Schmidt, T., Hertkorn, K., Newcombe, R., Marton, Z., Suppa, M., Fox, D.: Depth-based tracking with physical constraints for robot manipulation. In: IEEE International Conference on Robotics and Automation, pp. 119–126 (2015)

    Google Scholar 

  26. Wu, C.: SiftGPU: a GPU implementation of scale invariant feature transform (SIFT). http://github.com/pitzer/SiftGPU

  27. Lepetit, V., Fua, P.: Monocular model-based 3D tracking of rigid objects. Found. Trends Comput. Graph. Vis. 1, 1–89 (2005)

    Article  Google Scholar 

  28. Pauwels, K., Tomasi, M., Diaz Alonso, J., Ros, E., Van Hulle, M.: A comparison of FPGA and GPU for real-time phase-based optical flow, stereo, and local image features. IEEE Trans. Comput. 61(7), 999–1012 (2012)

    Article  MathSciNet  Google Scholar 

  29. Pan, J., Chitta, S., Manocha, D.: FCL: a general purpose library for collision proximity queries. In: IEEE International Conference on Robotics and Automation, pp. 3859–3866 (2012)

    Google Scholar 

  30. Olson, E.: AprilTag: a robust and flexible visual fiducial system. In: IEEE International Conference on Robotics and Automation, pp. 3400–3407 (2011)

    Google Scholar 

  31. Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., Dollar, A.: The YCB object and model set: towards common benchmarks for manipulation research. In: IEEE International Conference on Advanced Robotics, pp. 510–517 (2015)

    Google Scholar 

Download references

Acknowledgments

The authors want to thank Naiara Escudero for her assistance on the implementation of the Extended Kalman Filter, and Karl Pauwels for insights given on the use of Simtrack.

The research leading to these results has received funding from RoboCity2030-DIH-CM, Madrid Robotics Digital Innovation Hub, S2018/NMT-4331, funded by “Programas de Actividades I + D en la Comunidad de Madrid” and co-funded by Structural Funds of the EU. This work has also received funding from the Spanish Ministry of Economy, Industry and Competitiveness under the project DPI2016-80077-R.

Author information

Authors and Affiliations

  1. Systems Engineering and Automation Department, Carlos III University of Madrid, Getafe, Spain

    David Álvarez & Luis Moreno

  2. Institute of Robotics and Mechatronics, DLR - German Aerospace Center, Cologne, Germany

    Máximo A. Roa

Authors
  1. David Álvarez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Máximo A. Roa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luis Moreno
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Álvarez .

Editor information

Editors and Affiliations

  1. School of Engineering, Polytechnic Institute of Porto, Porto, Portugal

    Manuel F. Silva

  2. Department of Electrical Engineering, Polytechnic Institute of Bragança, Bragança, Portugal

    José Luís Lima

  3. Faculty of Engineering, University of Porto, Porto, Portugal

    Luís Paulo Reis

  4. UPC, Universitat Politècnica de Catalunya, Barcelona, Spain

    Alberto Sanfeliu

  5. Centro Universitario de la Defensa (CUD), Zaragoza, Spain

    Danilo Tardioli

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Álvarez, D., Roa, M.A., Moreno, L. (2020). Visual and Tactile Fusion for Estimating the Pose of a Grasped Object. In: Silva, M., Luís Lima, J., Reis, L., Sanfeliu, A., Tardioli, D. (eds) Robot 2019: Fourth Iberian Robotics Conference. ROBOT 2019. Advances in Intelligent Systems and Computing, vol 1093. Springer, Cham. https://doi.org/10.1007/978-3-030-36150-1_16

Download citation

Publish with us

Policies and ethics

Search

Navigation