Advertisement

Visual and Tactile Fusion for Estimating the Pose of a Grasped Object

  • David ÁlvarezEmail author
  • Máximo A. Roa
  • Luis Moreno
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1093)

Abstract

This paper considers the problem of fusing vision and touch senses together to estimate the 6D pose of an object while it is grasped. Assuming that a textured 3D model of the object is available, first, Scale-Invariant Feature Transform (SIFT) keypoints of the object are extracted, and a Random sample consensus (RANSAC) method is used to match these features with the textured model. Then, optical flow is used to visually track the object while a grasp is performed. After the hand contacts the object, a tactile-based pose estimation is performed using a Particle Filter. During grasp stabilization and hand movement, the pose of the object is continuously tracked by fusing the visual and tactile estimations with an extended Kalman filter. The main contribution of this work is the continuous use of both sensing modalities to reduce the uncertainty of tactile sensing in those degrees of freedom in which there is no information available, as presented through the experimental validation.

Keywords

Pose estimation Sensor fusion Tactile sensors Visual information 

Notes

Acknowledgments

The authors want to thank Naiara Escudero for her assistance on the implementation of the Extended Kalman Filter, and Karl Pauwels for insights given on the use of Simtrack.

The research leading to these results has received funding from RoboCity2030-DIH-CM, Madrid Robotics Digital Innovation Hub, S2018/NMT-4331, funded by “Programas de Actividades I + D en la Comunidad de Madrid” and co-funded by Structural Funds of the EU. This work has also received funding from the Spanish Ministry of Economy, Industry and Competitiveness under the project DPI2016-80077-R.

References

  1. 1.
    Lacey, S., Sathian, K.: Visuo-haptic multisensory object recognition, categorization, and representation. Front Psychol. 5, 730 (2014)CrossRefGoogle Scholar
  2. 2.
    Macura, Z., Cangelosi, A., Ellis, R., Bugmann, D., Fischer, M., Myachykov, A.: A cognitive robotic model of grasping. In: International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems, pp. 89–96 (2009)Google Scholar
  3. 3.
    Dogar, M., Hsiao, K., Ciocarlie, M., Srinivasa, S.: Physics-based grasp planning through clutter. In: Robotics: Science and Systems VIII (2012)Google Scholar
  4. 4.
    Vasconcelos, N., Pantoja, J., Belchior, H., Caixeta, F.V., Faber, J., Freire, M.A., Cota, V.R., de Macedo, E.A., Laplagne, D.A., Gomes, H.M., Ribeiro, S.: Cross-modal responses in the primary visual cortex encode complex objects and correlate with tactile discrimination. Proc. Natl. Acad. Sci. 108(37), 15408–15413 (2011)CrossRefGoogle Scholar
  5. 5.
    Petit, A., Marchand, E., Kanani, K.: A robust model-based tracker combining geometrical and color edge information. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3719–3724 (2013)Google Scholar
  6. 6.
    Choi, C., Christensen, H.I.: RGB-D object tracking: a particle filter approach on GPU. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1084–1091 (2013)Google Scholar
  7. 7.
    Vacchetti, L., Lepetit, V., Fua, P.: Stable real-time 3D tracking using online and offline information. IEEE Trans. Pattern Anal. Mach. Intell. 26, 1385–1391 (2004)CrossRefGoogle Scholar
  8. 8.
    Kyrki, V., Kragic, D.: Integration of model-based and model-free cues for visual object tracking in 3D. In: IEEE International Conference on Robotics and Automation, pp. 1566–1572 (2005)Google Scholar
  9. 9.
    Güler, P., Bekiroglu, Y., Pauwels, K., Kragic, D.: What’s in the container? Classifying object contents from vision and touch. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3961–3968 (2014)Google Scholar
  10. 10.
    Pauwels, K., Ivan, V., Ros, E., Vijayakumar, S.: Real-time object pose recognition and tracking with an imprecisely calibrated moving RGB-D camera. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2733–2740 (2014)Google Scholar
  11. 11.
    Jamali, N., Sammut, C.: Majority voting: material classification by tactile sensing using surface texture. IEEE Trans. Robot. 27(3), 508–521 (2011)CrossRefGoogle Scholar
  12. 12.
    Madry, M., Bo, L., Kragic, D., Fox, D.: ST-HMP: unsupervised spatio-temporal feature learning for tactile data. In: IEEE International Conference on Robotics and Automation, pp. 2262–2269 (2014)Google Scholar
  13. 13.
    Aggarwal, A., Kirchner, F.: Object recognition and localization: the role of tactile sensors. Sensors 14, 3227–3266 (2014)CrossRefGoogle Scholar
  14. 14.
    Haidacher, S., Hirzinger, G.: Estimating finger contact location and object pose from contact measurements in 3-D grasping. In: IEEE International Conference on Robotics and Automation, pp. 1805–1810 (2003)Google Scholar
  15. 15.
    Chalon, M., Reinecke, J., Pfanne, M.: Online in-hand object localization. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2977–2984 (2013)Google Scholar
  16. 16.
    Pfanne, M., Chalon, M.: EKF-based in-hand object localization from joint position and torque measurements. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2464–2470 (2017)Google Scholar
  17. 17.
    Álvarez, D., Roa, M.A., Moreno, L.: Tactile-based in-hand object pose estimation. In: Third Iberian Robotics Conference, ROBOT 2017, Advances in Intelligent Systems and Computing, vol. 694, pp. 716–728. Springer (2018)Google Scholar
  18. 18.
    Allen, P.K.: Integrating vision and touch for object recognition tasks. Int. J. Robot. Res. 7, 15–33 (1988)CrossRefGoogle Scholar
  19. 19.
    Ilonen, J., Bohg, J., Kyrki, V.: Three-dimensional object reconstruction of symmetric objects by fusing visual and tactile sensing. Int. J. Rob. Res. 33(2), 321–341 (2014)CrossRefGoogle Scholar
  20. 20.
    Alkkiomäki, O., Kyrki, V., Kälviäinen, H., Liu, Y., Handroos, H.: Complementing visual tracking of moving targets by fusion of tactile sensing. Rob. Auton. Syst. 57, 1129–1139 (2009)CrossRefGoogle Scholar
  21. 21.
    Kolycheva, E., Kyrki, V.: Task-specific grasping of similar objects by probabilistic fusion of vision and tactile measurements. In: IEEE-RAS International Conference on Humanoid Robots, pp. 704–710 (2015)Google Scholar
  22. 22.
    Zhang, M.M., Detry, R., Matthies, L., Daniilidis, K.: Tactile-vision integration for task-compatible fine-part manipulation. In: Robotics: Science and Systems. Workshop on Revisiting Contact—Turning a Problem into a Solution (2017)Google Scholar
  23. 23.
    Bimbo, J., Rodríguez-Jiménez, S., Liu, H., Song, X., Burrus, N., Senerivatne, L.D., Abderrahim, M., Althoefer, K.: Object pose estimation and tracking by fusing visual and tactile information. In: IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 65–70 (2012)Google Scholar
  24. 24.
    Bimbo, J., Seneviratne, L., Althoefer, K., Liu, H.: Combining touch and vision for the estimation of an object’s pose during manipulation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4021–4026 (2013)Google Scholar
  25. 25.
    Schmidt, T., Hertkorn, K., Newcombe, R., Marton, Z., Suppa, M., Fox, D.: Depth-based tracking with physical constraints for robot manipulation. In: IEEE International Conference on Robotics and Automation, pp. 119–126 (2015)Google Scholar
  26. 26.
    Wu, C.: SiftGPU: a GPU implementation of scale invariant feature transform (SIFT). http://github.com/pitzer/SiftGPU
  27. 27.
    Lepetit, V., Fua, P.: Monocular model-based 3D tracking of rigid objects. Found. Trends Comput. Graph. Vis. 1, 1–89 (2005)CrossRefGoogle Scholar
  28. 28.
    Pauwels, K., Tomasi, M., Diaz Alonso, J., Ros, E., Van Hulle, M.: A comparison of FPGA and GPU for real-time phase-based optical flow, stereo, and local image features. IEEE Trans. Comput. 61(7), 999–1012 (2012)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Pan, J., Chitta, S., Manocha, D.: FCL: a general purpose library for collision proximity queries. In: IEEE International Conference on Robotics and Automation, pp. 3859–3866 (2012)Google Scholar
  30. 30.
    Olson, E.: AprilTag: a robust and flexible visual fiducial system. In: IEEE International Conference on Robotics and Automation, pp. 3400–3407 (2011)Google Scholar
  31. 31.
    Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., Dollar, A.: The YCB object and model set: towards common benchmarks for manipulation research. In: IEEE International Conference on Advanced Robotics, pp. 510–517 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Systems Engineering and Automation DepartmentCarlos III University of MadridGetafeSpain
  2. 2.Institute of Robotics and MechatronicsDLR - German Aerospace CenterCologneGermany

Personalised recommendations