Active Manipulation for Perception

  • Anna PetrovskayaEmail author
  • Kaijen Hsiao
Part of the Springer Handbooks book series (SHB)


This chapter covers perceptual methods in which manipulation is an integral part of perception. These methods face special challenges due to data sparsity and high costs of sensing actions. However, they can also succeed where other perceptual methods fail, for example, in poor-visibility conditions or for learning the physical properties of a scene.

The chapter focuses on specialized methods that have been developed for object localization, inference, planning, recognition, and modeling in active manipulation approaches. We conclude with a discussion of real-life applications and directions for future research.


Particle Filter Polygonal Mesh Deformable Object Belief Space Bayesian Filter 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.









annealed particle filter


computer-aided design


coordinate measurement machine


center of mass


degree of freedom


guaranteed recursive adaptive bounding


histogram filter




importance sampling


k-nearest neighbor


Kullback–Leibler divergence


Markov decision process


manifold particle filter


particle filter


probabilistic latent semantic analysis


partially observable Markov decision process


color camera with depth


relational Markov decision processes


self-posture changeability


support vector machine


  1. 41.1
    P.C. Gaston, T. Lozano-Perez: Tactile recognition and localization using object models: The case of polyhedra on a plane, IEEE Trans. Pattern Anal. Machine Intell. 6(3), 257–266 (1984)CrossRefGoogle Scholar
  2. 41.2
    W.E.L. Grimson, T. Lozano-Perez: Model-based recognition and localization from sparse range or tactile data, Int. J. Robotics Res. 3, 3–35 (1984)CrossRefGoogle Scholar
  3. 41.3
    O.D. Faugeras, M. Hebert: A 3-D recognition and positioning algorithm using geometrical matching between primitive surfaces, Proc. 8th Intl. Jt. Conf. Artif. Intell., Los Altos (1983) pp. 996–1002Google Scholar
  4. 41.4
    S. Shekhar, O. Khatib, M. Shimojo: Sensor fusion and object localization, Proc. IEEE Int. Conf. Robotics Autom. (ICRA), Vol. 3 (1986) pp. 1623–1628Google Scholar
  5. 41.5
    Y.-B. Jia, M. Erdmann: Pose from pushing, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (1996) pp. 165–171CrossRefGoogle Scholar
  6. 41.6
    K.Y. Goldberg: Orienting polygonal parts without sensors, Algorithmica 10(2-4), 201–225 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 41.7
    M.A. Erdmann, M.T. Mason: An exploration of sensorless manipulation, IEEE J. Robotics Autom. 4(4), 369–379 (1988)CrossRefGoogle Scholar
  8. 41.8
    H.T. Yau, C.H. Menq: An automated dimensional inspection environment for manufactured parts using coordinate measuring machines, Int. J. Prod. Res. 30(7), 1517–1536 (1992)CrossRefGoogle Scholar
  9. 41.9
    K.T. Gunnarsson, F.B. Prinz: CAD model-based localization of parts in manufacturing, Computer 20(8), 66–74 (1987)CrossRefGoogle Scholar
  10. 41.10
    K.T. Gunnarsson: Optimal Part Localization by Data Base Matching with Sparse and Dense Data, Ph.D. Thesis (Dept. Mech. Eng., Carnegie Mellon Univ. Pittsburgh 1987)Google Scholar
  11. 41.11
    H.J. Pahk, W.J. Ahn: Precision inspection system for aircraft parts having very thin features based on CAD/CAI integration, Int. J. Adv. Manuf. Technol. 12(6), 442–449 (1996)CrossRefGoogle Scholar
  12. 41.12
    M.W. Cho, T.I. Seo: Inspection planning strategy for the on-machine measurement process based on CAD/CAM/CAI integration, Int. J. Adv. Manuf. Technol. 19(8), 607–617 (2002)CrossRefGoogle Scholar
  13. 41.13
    Z. Xiong: Workpiece Localization and Computer Aided Setup System, Ph.D. Thesis (Hong Kong Univ. Sci. Technol., Hong Kong 2002)CrossRefGoogle Scholar
  14. 41.14
    J. Hong, X. Tan: Method and apparatus for determining position and orientation of mechanical objects, US Patent US5208763 A (1993)Google Scholar
  15. 41.15
    B.K.P. Horn: Closed-form solution of absolute orientation using unit quaternions, J. Opt. Soc. Am. A 4(4), 629–642 (1987)CrossRefGoogle Scholar
  16. 41.16
    C.H. Menq, H.T. Yau, G.Y. Lai: Automated precision measurement of surface profile in CAD-directed inspection, IEEE Trans. Robotics Autom. 8(2), 268–278 (1992)CrossRefGoogle Scholar
  17. 41.17
    Y. Chu: Workpiece Localization: Theory, Algorithms and Implementation, Ph.D. Thesis (Hong Kong Univ. Sci. Technol., Hong Kong 1999)Google Scholar
  18. 41.18
    Z. Xiong, M.Y. Wang, Z. Li: A near-optimal probing strategy for workpiece localization, IEEE Trans. Robotics 20(4), 668–676 (2004)CrossRefGoogle Scholar
  19. 41.19
    Y. Huang, X. Qian: An efficient sensing localization algorithm for free-form surface digitization, J. Comput. Inform. Sci. Eng. 8, 021008 (2008)CrossRefGoogle Scholar
  20. 41.20
    L.M. Zhu, H.G. Luo, H. Ding: Optimal design of measurement point layout for workpiece localization, J. Manuf. Sci. Eng. 131, 011006 (2009)CrossRefGoogle Scholar
  21. 41.21
    L.M. Zhu, Z.H. Xiong, H. Ding, Y.L. Xiong: A distance function based approach for localization and profile error evaluation of complex surface, J. Manuf. Sci. Eng. 126, 542–554 (2004)CrossRefGoogle Scholar
  22. 41.22
    Y. Sun, J. Xu, D. Guo, Z. Jia: A unified localization approach for machining allowance optimization of complex curved surfaces, Precis. Eng. 33(4), 516–523 (2009)CrossRefGoogle Scholar
  23. 41.23
    K. Gadeyne, H. Bruyninckx: Markov techniques for object localization with force-controlled robots, 10th Int. Conf. Adv. Robotics (ICAR) (2001) pp. 91–96Google Scholar
  24. 41.24
    S.R. Chhatpar, M.S. Branicky: Particle filtering for localization in robotic assemblies with position uncertainty, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (2005)Google Scholar
  25. 41.25
    C. Corcoran, R. Platt: A measurement model for tracking hand-object state during dexterous manipulation, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2010)Google Scholar
  26. 41.26
    K. Hsiao, L. Kaelbling, T. Lozano-Perez: Task-driven tactile exploration, Robotics Sci. Syst. Conf. (2010)Google Scholar
  27. 41.27
    A. Petrovskaya, O. Khatib: Global localization of objects via touch, IEEE Trans. Robotics 27(3), 569–585 (2011)CrossRefGoogle Scholar
  28. 41.28
    A. Petrovskaya: Towards Dependable Robotic Perception, Ph.D. Thesis (Stanford Univ., Stanford 2011)Google Scholar
  29. 41.29
    A. Petrovskaya, O. Khatib, S. Thrun, A.Y. Ng: Bayesian estimation for autonomous object manipulation based on tactile sensors, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2006) pp. 707–714Google Scholar
  30. 41.30
    S. Arulampalam, S. Maskell, N. Gordon, T. Clapp: A tutorial on particle filters for on-line non-linear/non-Gaussian Bayesian tracking, IEEE Trans. Signal Process. 50(2), 174–188 (2002)CrossRefGoogle Scholar
  31. 41.31
    A. Petrovskaya, S. Thrun, D. Koller, O. Khatib: Guaranteed inference for global state estimation in human environments, Mob. Manip. Workshop Robotics Sci. Syst. (RSS) (2010)Google Scholar
  32. 41.32
    M.C. Koval, M.R. Dogar, N.S. Pollard, S.S. Srinivasa: Pose estimation for contact manipulation with manifold particle filters, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (2013) pp. 4541–4548Google Scholar
  33. 41.33
    J. Deutscher, A. Blake, I. Reid: Articulated body motion capture by annealed particle filtering, IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2000)Google Scholar
  34. 41.34
    J. Deutscher, I. Reid: Articulated body motion capture by stochastic search, Int. J. Comput. Vis. 61(2), 185–205 (2005)CrossRefGoogle Scholar
  35. 41.35
    A.O. Balan, L. Sigal, M.J. Black: A quantitative evaluation of video-based 3D person tracking, 2nd Jt. IEEE Int. Workshop Vis. Surveill. Perform. Eval. Track. Surveill. (2005) pp. 349–356Google Scholar
  36. 41.36
    K. Huebner: BADGr – A toolbox for box-based approximation, decomposition and grasping, Robotics Auton. Syst. 60(3), 367–376 (2012)CrossRefGoogle Scholar
  37. 41.37
    A.T. Miller, P.K. Allen: Graspit! a versatile simulator for robotic grasping, IEEE Robotics Autom, Magaz. 11(4), 110–122 (2004)Google Scholar
  38. 41.38
    K. Hsiao, T. Lozano-Pérez, L.P. Kaelbling: Robust belief-based execution of manipulation programs, 8 Int. Workshop Algorithm. Found. Robotics (WAFR) (2008)Google Scholar
  39. 41.39
    P. Hebert, T. Howard, N. Hudson, J. Ma, J.W. Burdick: The next best touch for model-based localization, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2013) pp. 99–106Google Scholar
  40. 41.40
    J.L. Schneiter, T.B. Sheridan: An automated tactile sensing strategy for planar object recognition and localization, IEEE Trans. Pattern Anal. Mach. Intell. 12(8), 775–786 (1990)CrossRefGoogle Scholar
  41. 41.41
    S. Javdani, M. Klingensmith, J.A. Bagnell, N.S. Pollard, S.S. Srinivasa: Efficient touch based localization through submodularity, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2013) pp. 1828–1835Google Scholar
  42. 41.42
    K. Hsiao, L.P. Kaelbling, T. Lozano-Pérez: Robust grasping under object pose uncertainty, Auton. Robots 31(2/3), 253–268 (2011)CrossRefGoogle Scholar
  43. 41.43
    M. Dogar, S. Srinivasa: A framework for push-grasping in clutter. In: Robotics: Science and Systems VII, ed. by H.F. Durrant-Whyte, N. Roy, P. Abbeel (MIT, Cambridge 2011)Google Scholar
  44. 41.44
    R. Platt, L. Kaelbling, T. Lozano-Perez, R. Tedrake: Efficient planning in non-Gaussian belief spaces and its application to robot grasping, Int. Symp. Robotics Res. (2011)Google Scholar
  45. 41.45
    L.P. Kaelbling, T. Lozano-Pérez: Integrated task and motion planning in belief space, Int. J. Robotics Res. 32(9/10), 1194–1227 (2013)CrossRefGoogle Scholar
  46. 41.46
    M. Erdmann: Shape recovery from passive locally dense tactile data, Proc. Workshop Algorithm. Found. Robotics (1998) pp. 119–132Google Scholar
  47. 41.47
    M. Moll, M.A. Erdmann: Reconstructing the shape and motion of unknown objects with active tactile sensors, Algorithm. Found. Robotics, Vol. V (2003) pp. 293–309Google Scholar
  48. 41.48
    F. Mazzini: Tactile Mapping of Harsh, Constrained Environments, with an Application to Oil Wells, Ph.D. Thesis (MIT, Cambridge 2011)Google Scholar
  49. 41.49
    P. Slaets, J. Rutgeerts, K. Gadeyne, T. Lefebvre, H. Bruyninckx, J. De Schutter: Construction of a geometric 3-D model from sensor measurements collected during compliant motion, 9th. Proc. Int. Symp. Exp. Robotics (2004)Google Scholar
  50. 41.50
    A. Bierbaum, K. Welke, D. Burger, T. Asfour, R. Dillmann: Haptic exploration for 3D shape reconstruction using five-finger hands, 7th IEEE-RAS Int. Conf. Hum. Robots (2007) pp. 616–621Google Scholar
  51. 41.51
    S.R. Chhatpar, M.S. Branicky: Localization for robotic assemblies using probing and particle filtering, Proc. IEEE/ASME Int. Conf. Adv. Intell. Mechatron. (2005) pp. 1379–1384Google Scholar
  52. 41.52
    M. Meier, M. Schopfer, R. Haschke, H. Ritter: A probabilistic approach to tactile shape reconstruction, IEEE Trans. Robotics 27(3), 630–635 (2011)CrossRefGoogle Scholar
  53. 41.53
    S. Walker, J.K. Salisbury: Pushing using learned manipulation maps, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2008) pp. 3808–3813Google Scholar
  54. 41.54
    Z. Pezzementi, C. Reyda, G.D. Hager: Object mapping, recognition, and localization from tactile geometry, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2011) pp. 5942–5948Google Scholar
  55. 41.55
    J. Sturm, A. Jain, C. Stachniss, C.C. Kemp, W. Burgard: Operating articulated objects based on experience, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (2010) pp. 2739–2744Google Scholar
  56. 41.56
    D. Katz, Y. Pyuro, O. Brock: Learning to manipulate articulated objects in unstructured environments using a grounded relational representation, Robotics Sci. Syst. (2008)Google Scholar
  57. 41.57
    C. Rhee, W. Chung, M. Kim, Y. Shim, H. Lee: Door opening control using the multi-fingered robotic hand for the indoor service robot, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2004)Google Scholar
  58. 41.58
    D. Henrich, H. Wörn: Robot Manipulation of Deformable Objects (Springer, Berlin, Heidelberg 2000)CrossRefGoogle Scholar
  59. 41.59
    R. Platt, F. Permenter, J. Pfeiffer: Using bayesian filtering to localize flexible materials during manipulation, IEEE Trans. Robotics 27(3), 586–598 (2011)CrossRefGoogle Scholar
  60. 41.60
    S. Burion, F. Conti, A. Petrovskaya, C. Baur, O. Khatib: Identifying physical properties of deformable objects by using particle filters, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2008) pp. 1112–1117Google Scholar
  61. 41.61
    A. Jain, M.D. Killpack, A. Edsinger, C. Kemp: Reaching in clutter with whole-arm tactile sensing, Int. J. Robotics Res. 32(4), 458–482 (2013)CrossRefGoogle Scholar
  62. 41.62
    L.P. Jentoft, R.D. Howe: Determining object geometry with compliance and simple sensors, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (2011) pp. 3468–3473Google Scholar
  63. 41.63
    M. Kaneko, K. Tanie: Contact point detection for grasping an unknown object using self-posture changeability, IEEE Trans. Robotics Autom. 10(3), 355–367 (1994)CrossRefGoogle Scholar
  64. 41.64
    A. Petrovskaya, J. Park, O. Khatib: Probabilistic estimation of whole body contacts for multi-contact robot control, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2007) pp. 568–573Google Scholar
  65. 41.65
    A.M. Okamura, M.A. Costa, M.L. Turner, C. Richard, M.R. Cutkosky: Haptic surface exploration, Lec. Notes Control Inform. Sci. 250, 423–432 (2000)CrossRefGoogle Scholar
  66. 41.66
    A. Bicchi, J.K. Salisbury, D.L. Brock: Experimental evaluation of friction characteristics with an articulated robotic hand, Lec. Notes Control Inform. Sci. 190, 153–167 (1993)CrossRefGoogle Scholar
  67. 41.67
    J.A. Fishel, G.E. Loeb: Bayesian exploration for intelligent identification of textures, Frontiers Neurorobot. 6, 4 (2012)CrossRefGoogle Scholar
  68. 41.68
    P. Dario, P. Ferrante, G. Giacalone, L. Livaldi, B. Allotta, G. Buttazzo, A.M. Sabatini: Planning and executing tactile exploratory procedures, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (1992) pp. 1896–1903Google Scholar
  69. 41.69
    J. Sinapov, V. Sukhoy: Vibrotactile recognition and categorization of surfaces by a humanoid robot, IEEE Trans. Robotics 27(3), 488–497 (2011)CrossRefGoogle Scholar
  70. 41.70
    M.R. Tremblay, M.R. Cutkosky: Estimating friction using incipient slip sensing during a manipulation task, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (1993) pp. 429–434CrossRefGoogle Scholar
  71. 41.71
    R. Bayrleithner, K. Komoriya: Static friction coefficient determination by force sensing and its application, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (1994) pp. 1639–1646Google Scholar
  72. 41.72
    K.M. Lynch: Estimating the friction parameters of pushed objects, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (1993) pp. 186–193Google Scholar
  73. 41.73
    C.G. Atkeson, C.H. An, J.M. Hollerbach: Estimation of inertial parameters of manipulator loads and links, Int. J. Robotics Res. 5(3), 101–119 (1986)CrossRefGoogle Scholar
  74. 41.74
    Y. Yu, T. Arima, S. Tsujio: Estimation of object inertia parameters on robot pushing operation, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2005) pp. 1657–1662Google Scholar
  75. 41.75
    H.T. Tanaka, K. Kushihama: Haptic vision-vision-based haptic exploration, Proc. 16th Int. Conf. Pattern Recognit. (2002) pp. 852–855Google Scholar
  76. 41.76
    H. Yussof, M. Ohka, J. Takata, Y. Nasu, M. Yamano: Low force control scheme for object hardness distinction in robot manipulation based on tactile sensing, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2008) pp. 3443–3448Google Scholar
  77. 41.77
    J. Romano, K. Hsiao, G. Niemeyer, S. Chitta, K.J. Kuchenbecker: Human-inspired robotic grasp control with tactile sensing, IEEE Trans. Robotics 27, 1067–1079 (2011)CrossRefGoogle Scholar
  78. 41.78
    S. Omata, Y. Murayama, C.E. Constantinou: Real time robotic tactile sensor system for the determination of the physical properties of biomaterials, Sens. Actuators A 112(2), 278–285 (2004)CrossRefGoogle Scholar
  79. 41.79
    C.H. Lin, T.W. Erickson, J.A. Fishel, N. Wettels, G.E. Loeb: Signal processing and fabrication of a biomimetic tactile sensor array with thermal, force and microvibration modalities, IEEE Int. Conf. Robotics Biomim. (ROBIO) (2009) pp. 129–134Google Scholar
  80. 41.80
    R.A. Russell: Object recognition by a 'smart' tactile sensor, Proc. Aust. Conf. Robotics Autom. (2000) pp. 93–98Google Scholar
  81. 41.81
    P.K. Allen, K.S. Roberts: Haptic object recognition using a multi-fingered dextrous hand, Proc. IEEE Conf. Robotics Autom. (ICRA) (1989) pp. 342–347Google Scholar
  82. 41.82
    S. Caselli, C. Magnanini, F. Zanichelli, E. Caraffi: Efficient exploration and recognition of convex objects based on haptic perception, Proc. IEEE Int. Conf. Robotics Autom. (ICRA), Vol. 4 (1996) pp. 3508–3513CrossRefGoogle Scholar
  83. 41.83
    A. Schneider, J. Sturm: Object identification with tactile sensors using bag-of-features, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (2009) pp. 243–248Google Scholar
  84. 41.84
    M. Schöpfer, M. Pardowitz, R. Haschke, H. Ritter: Identifying relevant tactile features for object identification, Springer Tracts Adv, Robotics 76, 417–430 (2012)Google Scholar
  85. 41.85
    T. Bhattacharjee, J.M. Rehg, C.C. Kemp: Haptic classification and recognition of objects using a tactile sensing forearm, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (2012) pp. 4090–4097Google Scholar
  86. 41.86
    Z. Pezzementi, E. Plaku, C. Reyda, G.D. Hager: Tactile-object recognition from appearance information, IEEE Trans. Robotics 27(3), 473–487 (2011)CrossRefGoogle Scholar
  87. 41.87
    N. Gorges, S.E. Navarro: Haptic object recognition using passive joints and haptic key features, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2010) pp. 2349–2355Google Scholar
  88. 41.88
    Z. Pezzementi, G.D. Hager: Tactile object recognition and localization using spatially-varying appearance, Int. Symp. Robotics Res. (ISRR) (2011)Google Scholar
  89. 41.89
    S. Chitta, J. Sturm, M. Piccoli, W. Burgard: Tactile sensing for mobile manipulation, IEEE Trans. Robotics 27(3), 558–568 (2011)CrossRefGoogle Scholar
  90. 41.90
    A. Drimus, G. Kootstra, A. Bilberg, D. Kragic: Design of a flexible tactile sensor for classification of rigid and deformable objects, Robotics Auton. Syst. 62(1), 3–15 (2012)CrossRefGoogle Scholar
  91. 41.91
    P. Allen: Integrating vision and touch for object recognition tasks, Int. J. Robotics Res. 7(6), 15–33 (1988)CrossRefGoogle Scholar
  92. 41.92
    M. Boshra, H. Zhang: A constraint-satisfaction approach for 3D vision/touch-based object recognition, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), Vol. 2 (1995) pp. 368–373Google Scholar
  93. 41.93
    T. Nakamura: Multimodal object categorization by a robot, IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS) (2007) pp. 2415–2420Google Scholar
  94. 41.94
    M. Gupta, G. Sukhatme: Using manipulation primitives for brick sorting in clutter, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2012) pp. 3883–3889Google Scholar
  95. 41.95
    K. Hausman, F. Balint-Benczedi, D. Pangercic, Z.-C. Marton, R. Ueda, K. Okada, M. Beetz: Tracking-based interactive segmentation of textureless objects, Proc. IEEE Int. Conf. Robotics Autom. (ICRA) (2013) pp. 1122–1129Google Scholar
  96. 41.96
    M. Sutton, L. Stark, K. Bowyer: Function from visual analysis and physical interaction: A methodology for recognition of generic classes of objects, Image Vis. Comput. 16(11), 745–763 (1998)CrossRefGoogle Scholar
  97. 41.97
    Y. Bekiroglu: Learning to Assess Grasp Stability from Vision, Touch and Proprioception Ph.D. Thesis (KTH Roy. Inst. Technol., Stockholm 2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Department of Computer ScienceStanford UniversityStanfordUSA
  2. 2.Research and Technology Center, Palo AltoRobert Bosch LLCPalo AltoUSA

Personalised recommendations