Integrating Vision and Touch for Grasping of an Object

  • Ruzena Bajcsy


The aim of this paper is to present considerations that go into the design of a system that will be able to grasp a variety of objects. Grasping depends on the assumption of two properties: The shape and size of the three-dimensional object and on the task that follows after grasping; hence, our interest in the integrated approach to 3-D shape recognition. We shall present an overview of currently available and being developed algorithms for representation and recognition of three-dimensional shape of an object as perceived by visual and/or tactile sensors. For the visual sensor we assume that we have available stereo cameras or their equivalent. As the tactile sensor we use an articulated multifingered hand equipped with tactile sensory arrays as the data acquisition device. We shall present available configurations of these devices. Then we shall investigate the sensory processing, in particular, what representation schemas should be considered. We shall argue that the object-centered representation as opposed to observer-centered representation is more important for grasping. Finally, a rule based schema for the control strategies will be outlined. As examples, first some artificial geometric objects then some real laboratory objects from the blocks world will be analyzed.


Vision Module Tactile Information Data Acquisition Device Tactile Module Rule Base Schema 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    S. Barnard and M. Fischler, “Computational stereo,” ACM Comp. Surveys, Vol. 14, No. 4, December 1982.Google Scholar
  2. [2]
    L. Blumenthal, “A surface follower and edge detector for the finger,” Computer and Information Science Dept., Univ. of Pennsylvania, Senior Project, May 1983.Google Scholar
  3. [3]
    V. Buono, “The lord sensor,” Computer and Information Science Dept., Univ. of Pennsylvania, Senior Project, May 1983.Google Scholar
  4. [4]
    R. Mohr and R. Bajcsy, “Packing volumes by spheres,” IEEE Trans. on PAMI, Vol. 6, No. 1, January 1983.Google Scholar
  5. [5]
    C. A. Dane, “An object-centered ‘three-dimensional model builder,” Ph.D. dissertation, Computer Science Dept., Univ. of Pennsylvania, Philadelphia, 1982.Google Scholar
  6. [6]
    R. Bajcsy, “What can one learn from one finger experiments?,” Proceedings of the International Symposium of Robotics Research, 1983, Bretton Woods, NH, August 1983.Google Scholar
  7. [7]
    G. I. Kinoshita, S. Aida, and M. Mori, “A pattern classification by dynamic tactile sense information processing,” Pattern Recognition, Vol. 7, pp. 243–251, 1975.MATHCrossRefGoogle Scholar
  8. [8]
    T. Okada and S. Tsuchiya, “Object recognition by grasping,” Pattern Recognition, Vol. 9, pp. 111–119, 1977.CrossRefGoogle Scholar
  9. [9]
    H. Ozaki, S. Waku, A. Mohri, and M. Takata, “Pattern recognition of a grasped object by unit-vector distribution,” IEEE Trans. on Systems, Man and Cybernetics, Vol. SMC-12, No. 3, pp. 315–324, May/June 1982.Google Scholar
  10. [10]
    R. Bajcsy, “Shape From Touch,” in Advances in Automation and Robotics, (G. N. Saridis, ed.) JAI Press, 1984.Google Scholar
  11. [11]
    P. Allen, “Visually driven tactile recognition and acquisition,” Image Processing and Robotics, Washington, DC, June 1982.Google Scholar
  12. [12]
    L. DeRisi, “3-D data acquisition system,” M.S. thesis, Univ. of Pennsylvania, Philadelphia, 1982.Google Scholar
  13. [13]
    L. Harmon, “Automated tactile sensing,” Int. J. Robotics Research, 1 (2), pp. 3–33, 1982.CrossRefGoogle Scholar
  14. [14]
    W. D. Hillis, “A high resolution image touch sensor,” Int. J. Robotics Research, 1 (2) pp. 33–44, 1982.CrossRefGoogle Scholar
  15. [15]
    R. Bajcsy and K. I. Goldberg, “A new approach to robotic tactile perception,” submitted to J. Cognition and Brain Theory, 1983.Google Scholar
  16. [16]
    R. Bajcsy, M. J. McCarthy, and J. C. Trinkle, “Feeling by grasping,” Proc. IEEE Int. Conf. on Robotics, Atlanta, 1984.Google Scholar
  17. [17]
    J. D. Abramowitz, J. W. Goodnow, and B. Paul, “Pennsylvania Articulated Mechanical Hand,” Proc. Int. Conf. on Computers and Engineering, ASME, Chicago, August 1983.Google Scholar

Copyright information

© Plenum Press, New York 1985

Authors and Affiliations

  • Ruzena Bajcsy
    • 1
  1. 1.Computer and Information Science DepartmentUniversity of PennsylvaniaPhiladelphiaUSA

Personalised recommendations