Visual Servoing

  • François Chaumette
  • Seth Hutchinson
  • Peter Corke

Abstract

This chapter introduces visual servo control, using computer vision data in the servo loop to control the motion of a robot. We first describe the basic techniques that are by now well established in the field. We give a general overview of the formulation of the visual servo control problem, and describe the two archetypal visual servo control schemes: image-based and pose-based visual servo control. We then discuss performance and stability issues that pertain to these two schemes, motivating advanced techniques. Of the many advanced techniques that have been developed, we discuss two-and-a-half-dimensional (2.5-D), hybrid, partitioned, and switched approaches. Having covered a variety of control schemes, we deal with target tracking and controlling motion directly in the joint space and extensions to under-actuated ground and aerial robots. We conclude by describing applications of visual servoing in robotics.

2-D

two-dimensional

2.5-D

two-and-a-half-dimensional

3-D

three-dimensional

IBVS

image-based visual servo control

IMU

inertial measurement unit

LQG

linear quadratic Gaussian

MEMS

microelectromechanical system

PBVS

pose-based visual servo control

VS

visual servo

References

  1. 34.1
    L. Weiss, A. Sanderson, C. Neuman: Dynamic sensor-based control of robots with visual feedback, IEEE J. Robot. Autom. 3, 404–417 (1987)CrossRefGoogle Scholar
  2. 34.2
    S. Hutchinson, G. Hager, P. Corke: A tutorial on visual servo control, IEEE Trans. Robot. Autom. 12, 651–670 (1996)CrossRefGoogle Scholar
  3. 34.3
    B. Espiau, F. Chaumette, P. Rives: A new approach to visual servoing in robotics, IEEE Trans. Robot. Autom. 8, 313–326 (1992)CrossRefGoogle Scholar
  4. 34.4
    J. Feddema, O. Mitchell: Vision-guided servoing with feature-based trajectory generation, IEEE Trans. Robot. Autom. 5, 691–700 (1989)CrossRefGoogle Scholar
  5. 34.5
    D. Forsyth, J. Ponce: Computer Vision: A Modern Approach (Prentice Hall, Upper Saddle River 2003)Google Scholar
  6. 34.6
    Y. Ma, S. Soatto, J. Kosecka, S. Sastry: An Invitation to 3-D Vision: From Images to Geometric Models (Springer, New York 2003)MATHGoogle Scholar
  7. 34.7
    P. Corke: Robotics, Vision and Control: Fundamental Algorithms in MATLAB (Springer, Berlin, Heidelberg 2011)CrossRefMATHGoogle Scholar
  8. 34.8
    H. Michel, P. Rives: Singularities in the Determination of the Situation of a Robot Effector from the Perspective View of Three Points. Res. Rep. RR-1850 (INRIA 1993)Google Scholar
  9. 34.9
    M. Fischler, R. Bolles: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications ACM 24, 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  10. 34.10
    E. Malis: Improving vision-based control using efficient second-order minimization techniques, IEEE Int. Conf. Robot. Autom., New Orleans (2004) pp. 1843–1848Google Scholar
  11. 34.11
    E. Marchand, F. Spindler, F. Chaumette: ViSP for visual servoing: A generic software platform with a wide class of robot control skills, IEEE Robot. Autom. Mag. 12(4), 40–52 (2005), https://team.inria.fr/lagadic/visp/visp.html CrossRefGoogle Scholar
  12. 34.12
    P. Corke, S. Hutchinson: A new partitioned approach to image-based visual servo control, IEEE Trans. Robot. Autom. 17, 507–515 (2001)CrossRefGoogle Scholar
  13. 34.13
    F. Chaumette: Potential problems of stability and convergence in image-based and position-based visual servoing, Lect. Note. Contr. Inform. Sci. 237, 66–78 (1998)CrossRefGoogle Scholar
  14. 34.14
    E. Malis: Visual servoing invariant to changes in camera intrinsic parameters, IEEE Trans. Robot. Autom. 20, 72–81 (2004)CrossRefGoogle Scholar
  15. 34.15
    A. Isidori: Nonlinear Control Systems, 3rd edn. (Springer, Berlin, Heidelberg 1995)CrossRefMATHGoogle Scholar
  16. 34.16
    G. Hager, W. Chang, A. Morse: Robot feedback control based on stereo vision: Towards calibration-free hand-eye coordination, IEEE Control Syst. Mag. 15, 30–39 (1995)CrossRefGoogle Scholar
  17. 34.17
    M. Iwatsuki, N. Okiyama: A new formulation of visual servoing based on cylindrical coordinate system, IEEE Trans. Robot. Autom. 21, 266–273 (2005)CrossRefGoogle Scholar
  18. 34.18
    F. Chaumette, P. Rives, B. Espiau: Classification and realization of the different vision-based tasks, Robot. Autom. Syst. 7, 199–228 (1993)Google Scholar
  19. 34.19
    A. Castano, S. Hutchinson: Visual compliance: Task directed visual servo control, IEEE Trans. Robot. Autom. 10, 334–342 (1994)CrossRefGoogle Scholar
  20. 34.20
    G. Hager: A modular system for robust positioning using feedback from stereo vision, IEEE Trans. Robot. Autom. 13, 582–595 (1997)CrossRefGoogle Scholar
  21. 34.21
    F. Chaumette: Image moments: A general and useful set of features for visual servoing, IEEE Trans. Robot. Autom. 20, 713–723 (2004)CrossRefGoogle Scholar
  22. 34.22
    O. Tahri, F. Chaumette: Point-based and region-based image moments for visual servoing of planar objects, IEEE Trans. Robot. 21, 1116–1127 (2005)CrossRefGoogle Scholar
  23. 34.23
    C. Geyer, K. Daniilidis: Catadioptric projective geometry, Int. J. Comput. Vis. 45(3), 223–243 (2001)CrossRefMATHGoogle Scholar
  24. 34.24
    T. Hamel, R. Mahony: Visual servoing of an under-actuated dynamic rigid-body system: An image-based approach, IEEE Trans Robot. 18(2), 187–198 (2002)CrossRefGoogle Scholar
  25. 34.25
    I. Suh: Visual servoing of robot manipulators by fuzzy membership function based neural networks. In: Visual Servoing, Robotics and Automated Systems, Vol. 7, ed. by K. Hashimoto (World Scientific, Singapore 1993) pp. 285–315CrossRefGoogle Scholar
  26. 34.26
    G. Wells, C. Venaille, C. Torras: Vision-based robot positioning using neural networks, Image Vis. Comput. 14, 75–732 (1996)CrossRefGoogle Scholar
  27. 34.27
    J.T. Lapresté, F. Jurie, F. Chaumette: An efficient method to compute the inverse jacobian matrix in visual servoing, IEEE Int. Conf. Robot. Autom., New Orleans (2004) pp. 727–732Google Scholar
  28. 34.28
    K. Hosada, M. Asada: Versatile visual servoing without knowledge of true jacobian, IEEE/RSJ Int. Conf. Intell. Robots Syst., München (1994) pp. 186–193Google Scholar
  29. 34.29
    M. Jägersand, O. Fuentes, R. Nelson: Experimental evaluation of uncalibrated visual servoing for precision manipulation, IEEE Int. Conf. Robot. Autom., Albuquerque (1997) pp. 2874–2880Google Scholar
  30. 34.30
    J. Piepmeier, G.M. Murray, H. Lipkin: Uncalibrated dynamic visual servoing, IEEE Trans. Robot. Autom. 20, 143–147 (2004)CrossRefGoogle Scholar
  31. 34.31
    K. Deguchi: Direct interpretation of dynamic images and camera motion for visual servoing without image feature correspondence, J. Robot. Mechatron. 9(2), 104–110 (1997)MathSciNetCrossRefGoogle Scholar
  32. 34.32
    W. Wilson, C. Hulls, G. Bell: Relative end-effector control using cartesian position based visual servoing, IEEE Trans. Robot. Autom. 12, 684–696 (1996)CrossRefGoogle Scholar
  33. 34.33
    B. Thuilot, P. Martinet, L. Cordesses, J. Gallice: Position based visual servoing: Keeping the object in the field of vision, IEEE Int. Conf. Robot. Autom., Washington (2002) pp. 1624–1629Google Scholar
  34. 34.34
    D. Dementhon, L. Davis: Model-based object pose in 25 lines of code, Int. J. Comput. Vis. 15, 123–141 (1995)CrossRefGoogle Scholar
  35. 34.35
    D. Lowe: Three-dimensional object recognition from single two-dimensional images, Artif. Intell. 31(3), 355–395 (1987)CrossRefGoogle Scholar
  36. 34.36
    E. Malis, F. Chaumette, S. Boudet: 2-1/2 D visual servoing, IEEE Trans. Robot. Autom. 15, 238–250 (1999)CrossRefGoogle Scholar
  37. 34.37
    E. Malis, F. Chaumette: Theoretical improvements in the stability analysis of a new class of model-free visual servoing methods, IEEE Trans. Robot. Autom. 18, 176–186 (2002)CrossRefGoogle Scholar
  38. 34.38
    J. Chen, D. Dawson, W. Dixon, A. Behal: Adaptive homography-based visual servo tracking for fixed camera-in-hand configurations, IEEE Trans. Control Syst. Technol. 13, 814–825 (2005)CrossRefGoogle Scholar
  39. 34.39
    G. Morel, T. Leibezeit, J. Szewczyk, S. Boudet, J. Pot: Explicit incorporation of 2-D constraints in vision-based control of robot manipulators, Lect. Note. Contr. Inform. Sci. 250, 99–108 (2000)CrossRefGoogle Scholar
  40. 34.40
    F. Chaumette, E. Malis: 2 1/2 D visual servoing: a possible solution to improve image-based and position-based visual servoings, IEEE Int. Conf. Robot. Autom., San Fransisco (2000) pp. 630–635Google Scholar
  41. 34.41
    E. Cervera, A.D. Pobil, F. Berry, P. Martinet: Improving image-based visual servoing with three-dimensional features, Int. J. Robot. Res. 22, 821–840 (2004)CrossRefGoogle Scholar
  42. 34.42
    F. Schramm, G. Morel, A. Micaelli, A. Lottin: Extended 2-D visual servoing, IEEE Int. Conf. Robot. Autom., New Orleans (2004) pp. 267–273Google Scholar
  43. 34.43
    N. Papanikolopoulos, P. Khosla, T. Kanade: Visual tracking of a moving target by a camera mounted on a robot: A combination of vision and control, IEEE Trans. Robot. Autom. 9, 14–35 (1993)CrossRefGoogle Scholar
  44. 34.44
    K. Hashimoto, H. Kimura: LQ optimal and nonlinear approaches to visual servoing, Robot. Autom. Syst. 7, 165–198 (1993)Google Scholar
  45. 34.45
    B. Nelson, P. Khosla: Strategies for increasing the tracking region of an eye-in-hand system by singularity and joint limit avoidance, Int. J. Robot. Res. 14, 225–269 (1995)CrossRefGoogle Scholar
  46. 34.46
    B. Nelson, P. Khosla: Force and vision resolvability for assimilating disparate sensory feedback, IEEE Trans. Robot. Autom. 12, 714–731 (1996)CrossRefGoogle Scholar
  47. 34.47
    R. Sharma, S. Hutchinson: Motion perceptibility and its application to active vision-based servo control, IEEE Trans. Robot. Autom. 13, 607–617 (1997)CrossRefGoogle Scholar
  48. 34.48
    E. Marchand, F. Chaumette, A. Rizzo: Using the task function approach to avoid robot joint limits and kinematic singularities in visual servoing, IEEE/RSJ Int. Conf. Intell. Robots Syst., Osaka (1996) pp. 1083–1090Google Scholar
  49. 34.49
    E. Marchand, G. Hager: Dynamic sensor planning in visual servoing, IEEE Int. Conf. Robot. Autom., Leuven (1998) pp. 1988–1993Google Scholar
  50. 34.50
    N. Cowan, J. Weingarten, D. Koditschek: Visual servoing via navigation functions, IEEE Trans. Robot. Autom. 18, 521–533 (2002)CrossRefGoogle Scholar
  51. 34.51
    N. Gans, S. Hutchinson: An asymptotically stable switched system visual controller for eye in hand robots, IEEE/RSJ Int. Conf. Intell. Robots Syst., Las Vegas (2003) pp. 735–742Google Scholar
  52. 34.52
    G. Chesi, K. Hashimoto, D. Prattichizio, A. Vicino: Keeping features in the field of view in eye-in-hand visual servoing: a switching approach, IEEE Trans. Robot. Autom. 20, 908–913 (2004)CrossRefGoogle Scholar
  53. 34.53
    K. Hosoda, K. Sakamato, M. Asada: Trajectory generation for obstacle avoidance of uncalibrated stereo visual servoing without 3-D reconstruction, IEEE/RSJ Int. Conf. Intell. Robots Syst. 3, Pittsburgh (1995) pp. 29–34Google Scholar
  54. 34.54
    Y. Mezouar, F. Chaumette: Path planning for robust image-based control, IEEE Trans. Robot. Autom. 18, 534–549 (2002)CrossRefGoogle Scholar
  55. 34.55
    G. Chesi: Visual servoing path-planning via homogeneous forms and LMI optimizations, IEEE Trans. Robot. 25(2), 281–291 (2009)CrossRefGoogle Scholar
  56. 34.56
    L. Matthies, T. Kanade, R. Szeliski: Kalman filter-based algorithms for estimating depth from image sequences, Int. J. Comput. Vis. 3(3), 209–238 (1989)CrossRefGoogle Scholar
  57. 34.57
    C.E. Smith, N. Papanikolopoulos: Computation of shape through controlled active exploration, IEEE Int. Conf. Robot. Autom., San Diego (1994) pp. 2516–2521Google Scholar
  58. 34.58
    A. De Luca, G. Oriolo, P. Robuffo Giordano: Feature depth observation for image-based visual servoing: Theory and experiments, Int. J. Robot. Res. 27(10), 1093–1116 (2008)CrossRefGoogle Scholar
  59. 34.59
    R. Basri, E. Rivlin, I. Shimshoni: Visual homing: Surfing on the epipoles, Int. J. Comput. Vis. 33, 117–137 (1999)CrossRefGoogle Scholar
  60. 34.60
    E. Malis, F. Chaumette, S. Boudet: 2 1/2 D visual servoing with respect to unknown objects through a new estimation scheme of camera displacement, Int. J. Comput. Vis. 37, 79–97 (2000)CrossRefMATHGoogle Scholar
  61. 34.61
    O. Faugeras: Three-Dimensional Computer Vision: A Geometric Viewpoint (MIT Press, Cambridge 1993)Google Scholar
  62. 34.62
    G. Silveira, E. Malis: Direct visual servoing: Vision-based estimation and control using only nonmetric information, IEEE Trans. Robot. 28(4), 974–980 (2012)CrossRefGoogle Scholar
  63. 34.63
    P. Corke, M. Goods: Controller design for high performance visual servoing, 12th World Congr. IFAC'93, Sydney (1993) pp. 395–398Google Scholar
  64. 34.64
    F. Bensalah, F. Chaumette: Compensation of abrupt motion changes in target tracking by visual servoing, IEEE/RSJ Int. Conf. Intell. Robots Syst., Pittsburgh (1995) pp. 181–187Google Scholar
  65. 34.65
    P. Allen, B. Yoshimi, A. Timcenko, P. Michelman: Automated tracking and grasping of a moving object with a robotic hand-eye system, IEEE Trans. Robot. Autom. 9, 152–165 (1993)CrossRefGoogle Scholar
  66. 34.66
    K. Hashimoto, H. Kimura: Visual servoing with non linear observer, IEEE Int. Conf. Robot. Autom., Nagoya (1995) pp. 484–489Google Scholar
  67. 34.67
    A. Rizzi, D. Koditschek: An active visual estimator for dexterous manipulation, IEEE Trans. Robot. Autom. 12, 697–713 (1996)CrossRefGoogle Scholar
  68. 34.68
    R. Ginhoux, J. Gangloff, M. de Mathelin, L. Soler, M.A. Sanchez, J. Marescaux: Active filtering of physiological motion in robotized surgery using predictive control, IEEE Trans. Robot. 21, 67–79 (2005)CrossRefGoogle Scholar
  69. 34.69
    J. Gangloff, M. de Mathelin: Visual servoing of a 6-DOF manipulator for unknown 3-D profile following, IEEE Trans. Robot. Autom. 18, 511–520 (2002)CrossRefGoogle Scholar
  70. 34.70
    R. Tsai, R. Lenz: A new technique for fully autonomous efficient 3-D robotics hand-eye calibration, IEEE Trans. Robot. Autom. 5, 345–358 (1989)CrossRefGoogle Scholar
  71. 34.71
    N. Guenard, T. Hamel, R. Mahony: A practical visual servo control for an unmanned aerial vehicle, IEEE Trans. Robot. 24(2), 331–340 (2008)CrossRefGoogle Scholar
  72. 34.72
    G.L. Mariottini, G. Oriolo, D. Prattichizo: Image-based visual servoing for nonholonomic mobile robots using epipolar geometry, IEEE Trans. Robot. 23(1), 87–100 (2007)CrossRefGoogle Scholar
  73. 34.73
    G. Lopez-Nicolas, J.J. Guerrero, C. Sagues: Visual control through the trifocal tensor for nonholonomic robots, Robot. Auton. Syst. 58(2), 216–226 (2010)CrossRefGoogle Scholar
  74. 34.74
    P. Corke: Robotics, Vision and Control: Fundamental Algorithms in MATLAB, Springer Tracts in Advanced Robotics, Vol. 73 (Springer, Berlin, Heidelberg 2011)CrossRefMATHGoogle Scholar
  75. 34.75
    A. Crétual, F. Chaumette: Visual servoing based on image motion, Int. J. Robot. Res. 20(11), 857–877 (2001)CrossRefGoogle Scholar
  76. 34.76
    C. Collewet, E. Marchand: Photometric visual servoing, IEEE Trans. Robot. 27(4), 828–834 (2011)CrossRefGoogle Scholar
  77. 34.77
    R. Mebarki, A. Krupa, F. Chaumette: 2D ultrasound probe complete guidance by visual servoing using image moments, IEEE Trans. Robot. 26(2), 296–306 (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • François Chaumette
    • 1
  • Seth Hutchinson
    • 2
  • Peter Corke
    • 3
  1. 1.Lagadic GroupInria/IrisaRennesFrance
  2. 2.Department of Electrical and Computer EngineeringUniversity of IllinoisUrbana-ChampaignUSA
  3. 3.Department of Electrical Engineering and Computer ScienceQueensland University of TechnologyBrisbaneAustralia

Personalised recommendations