Skip to main content
SpringerLink
Log in

Vision System and Calibration

  • Chapter
  • First Online:
Vision Based Identification and Force Control of Industrial Robots

Part of the book series: Studies in Systems, Decision and Control ((SSDC,volume 404))

  • 364 Accesses

Abstract

This monograph highlights how different technologies, namely, vision, robotic manipulation, sensitivity and uncertainty analysis, kinematic and dynamic identification, force control etc., can be combined to get an end-to-end fully functional system. This chapter acts as a window to this treatment to explain perception of the external environment using sensors. The environment to be sensed contains a large number of symmetrical and identical feature-less, texture-less objects (black cylindrical pellets) randomly piled up in the bin, with arbitrary orientations and heavy occlusion. The task at hand is to use a set of sensors with complementary properties (a camera and a range sensor) for pose estimation in heavy occlusion, accordingly orienting the manipulator gripper to pick up a suitable pellet, and this process is repeated to pick all the pellets one-by-one. Furthermore, the manipulator avoids collision with the bin walls by identifying and characterizing the cases when the object is present in a blind spot for the manipulator. Thus, this chapter lays the foundation for the subsequent chapter, i.e., uncertainty and sensitivity analysis of the vision-based system. These chapters also prove that building a fully functional robust pipeline in real-world scenarios is quite challenging. Particular aspects that are of paramount importance for the successful accomplishment of the task is also presented. Another critical aspect to emphasize here is that it deals with real-world industrial problems, i.e., bin-picking. Different approaches that work best due to variations in assumptions and experimental protocols, e.g., sensors, lighting, robot arms, grippers, and objects, are dealt with in detail. A layout of the literature survey is provided as visual chart to give the major technologies and components involved in this chapter. The contents of this chapter will be suitable for practitioners, researchers, or even novices in robotics to gain insight into real-world problems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Such an artifact can be easily made. The artifact shown in Fig. 2.5c was fabricated by machining a cylindrical piece made of steel in a lathe machine.

References

  1. Zisserman, A., Forsyth, D., Mundy, J., Rothwell, C., Liu, J., and Pillow, N.: 3d object recognition using invariance. Artif. Intell. 78(1), 239–288 (1995). Special Volume on Computer Vision

    Google Scholar 

  2. Ponce, J., Chelberg, D., Mann, W.B.: Invariant properties of straight homogeneous generalized cylinders and their contours. IEEE Trans. Pattern Anal. Mach. Intell. 11(9), 951–966 (1989)

    Article  Google Scholar 

  3. Zerroug, M., Nevatia, R.: Three-dimensional descriptions based on the analysis of the invariant and quasi-invariant properties of some curved-axis generalized cylinders. IEEE Trans. Pattern Anal. Mach. Intell. 18(3), 237–253 (1996)

    Article  Google Scholar 

  4. Faugeras, O., Hebert, M.: The representation, recognition, and locating of 3-d objects. 5, 27–52 (1986)

    Google Scholar 

  5. Ansar, A., Daniilidis, K.: Linear pose estimation from points or lines. IEEE Trans. Pattern Anal. Mach. Intell. 25(5), 578–589 (2003)

    Article  Google Scholar 

  6. Johnson, A.E., Hebert, M.: Using spin images for efficient object recognition in cluttered 3d scenes. IEEE Trans. Pattern Anal. Mach. Intell. 21(5), 433–449 (1999)

    Article  Google Scholar 

  7. Mittrapiyanumic, P., DeSouza, G.N., Kak, A.C.: Calculating the 3d-pose of rigid-objects using active appearance models. In: 2004 IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA ’04, vol. 5, pp. 5147–5152 (2004)

    Google Scholar 

  8. Ling, H., Jacobs, D.W.: Shape classification using the inner-distance. IEEE Trans. Pattern Anal. Mach. Intell. 29(2), 286–299 (2007)

    Article  Google Scholar 

  9. Berg, A.C., Berg, T.L., Malik, J.: Shape matching and object recognition using low distortion correspondences. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 26–33 (2005)

    Google Scholar 

  10. Ferrari, V., Tuytelaars, T., Van Gool, L.: Object detection by contour segment networks. In: Proceedings of the 9th European Conference on Computer Vision—Volume Part III, ECCV’06, pp. 14–28. Springer, Berlin

    Google Scholar 

  11. Ferrari, V., Fevrier, L., Jurie, F., Schmid, C.: Groups of adjacent contour segments for object detection. IEEE Trans. Pattern Anal. Mach. Intell. 30(1), 36–51 (2008)

    Article  Google Scholar 

  12. Ferrari, V., Jurie, F., Schmid, C.: From images to shape models for object detection. Int. J. Comput. Vision 87(3), 284–303 (2010)

    Article  Google Scholar 

  13. Qi, C.R., Liu, W., Wu, C., Su, H., Guibas, L.J.: Frustum pointnets for 3d object detection from rgb-d data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 918–927 (2018)

    Google Scholar 

  14. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017a)

    Google Scholar 

  15. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)

    Article  Google Scholar 

  16. Suga, A., Fukuda, K., Takiguchi, T., Ariki, Y.: Object recognition and segmentation using sift and graph cuts. In: 2008 19th International Conference on Pattern Recognition, pp. 1–4. IEEE (2008)

    Google Scholar 

  17. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision (2017)

    Google Scholar 

  18. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A.C.: Ssd: Single shot multibox detector. In: European Conference on Computer Vision, pp. 21–37. Springer, Berlin (2016)

    Google Scholar 

  19. Redmon, J., Farhadi, A.: Yolo9000: better, faster, stronger. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7263–7271 (2017)

    Google Scholar 

  20. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)

    Article  Google Scholar 

  21. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017b)

    Google Scholar 

  22. Li, Y., Bu, R., Sun, M., Wu, W., Di, X., Chen, B.: Pointcnn: Convolution on x transformed points (2018). ArXiv:180107791

  23. Dong, Z., Liu, S., Zhou, T., Cheng, H., Zeng, L., Yu, X., Liu, H.: Ppr-net: point-wise pose regression network for instance segmentation and 6d pose estimation in bin-picking scenarios. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1773–1780. IEEE (2019)

    Google Scholar 

  24. Drost, B., Ulrich, M., Navab, N., and Ilic, S.: Model globally, match locally: Efficient and robust 3d object recognition. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 998–1005. IEEE (2010)

    Google Scholar 

  25. Kruglyak, L., Lander, E.S.: Complete multipoint sib-pair analysis of qualitative and quantitative traits. Am. J. Human Gene. 57(2), 439 (1995)

    Google Scholar 

  26. Vidal, J., Lin, C.-Y., and Martí, R.: 6d pose estimation using an improved method based on point pair features. In: 2018 4th International Conference on Control, Automation and Robotics (iccar), pp. 405–409. IEEE (2018)

    Google Scholar 

  27. Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes (2017a). ArXiv:1711.00199

  28. Sock, J., Kim, K.I., Sahin, C., Kim, T.-K.: Multi-task deep networks for depth-based 6d object pose and joint registration in crowd scenarios (2018). arXiv:1806.03891

  29. Zeng, A., Yu, K.-T., Song, S., Suo, D., Walker, E., Rodriguez, A., Xiao, J.: Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1386–1383. IEEE (2017)

    Google Scholar 

  30. Bicchi, A., Kumar, V.: Robotic grasping and contact: A review. In: Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), vol. 1, pp. 348–353. IEEE (2020)

    Google Scholar 

  31. Caldera, S., Rassau, A., Chai, D.: Review of deep learning methods in robotic grasp detection. Multimodal Technol. Interact. 2(3), 57 (2018)

    Article  Google Scholar 

  32. Kuo, H.Y., Su, H.R., Lai, S.H., Wu, C.C.: 3d object detection and pose estimation from depth image for robotic bin picking. In: 2014 IEEE International Conference on Automation Science and Engineering (CASE), pp. 1264–1269 (2014)

    Google Scholar 

  33. Vosselman, G., Dijkman, S.: 3D building model reconstruction from point clouds and ground plans, pp. 37–43. Number XXXIV-3/W4 in (International Archives of Photogrammetry and Remote Sensing : IAPRS : ISPRS. International Society for Photogrammetry and Remote Sensing (ISPRS) (2001)

    Google Scholar 

  34. Papazov, C., Haddadin, S., Parusel, S., Krieger, K., Burschka, D.: Rigid 3d geometry matching for grasping of known objects in cluttered scenes. I. J. Robot. Res. 31(4), 538–553 (2012)

    Article  Google Scholar 

  35. Nieuwenhuisen, M., Droeschel, D., Holz, D., Stückler, J., Berner, A., Li, J., Klein, R., Behnke, S.: Mobile bin picking with an anthropomorphic service robot. In: 2013 IEEE International Conference on Robotics and Automation, pp. 2327–2334 (2013)

    Google Scholar 

  36. Rabbani, T., Heuvel, F.V.D.: Efficient hough transform for automatic detection of cylinders in point clouds (2005)

    Google Scholar 

  37. Bolles, R.C., Horaud, P.: 3DPO: A Three-Dimensional Part Orientation System, pp. 399–450. Springer US, Boston, MA (1987)

    Google Scholar 

  38. Lu, F., Milios, E.E.: Robot pose estimation in unknown environments by matching 2d range scans. In: 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 935–938 (1994)

    Google Scholar 

  39. Latecki, L.J., Lakaemper, R., Sun, X., Wolter, D.: Building polygonal maps from laser range data (2004)

    Google Scholar 

  40. Baillard, C., Schmid, C., Zisserman, A., Fitzgibbon, A., England, O.O.: Automatic line matching and 3d reconstruction of buildings from multiple views. In: ISPRS Conference on Automatic Extraction of GIS Objects from Digital Imagery, IAPRS vol. 32, Part 3-2W5, pp. 69–80 (1999)

    Google Scholar 

  41. Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’96, pp. 303–312, ACM, New York, NY, USA (1996)

    Google Scholar 

  42. Weingarten, J.W., Gruener, G., Siegwart, R.: A state-of-the-art 3d sensor for robot navigation. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), vol. 3, pp. 2155–2160 (2004)

    Google Scholar 

  43. Gummadi, V.M. Sarkodie-Gyan, T.: 2 d object recognition using the hough transform. In: Intelligent Systems Design and Applications, pp. 413–421. Springer, Berlin (2003)

    Google Scholar 

  44. Saxena, A., Driemeyer, J., Ng, A.Y.: Learning 3-d object orientation from images. In: 2009 IEEE International Conference on Robotics and Automation, pp. 794–800. IEEE (2009)

    Google Scholar 

  45. Sundermeyer, M., Marton, Z.-C., Durner, M., Triebel, R.: Augmented autoencoders: Implicit 3d orientation learning for 6d object detection. Int. J. Comput. Vision 128(3), 714–729 (2020)

    Article  Google Scholar 

  46. Matthias, N., Droeschel, D., Holz, D., Stückler, J., Berner, A., Li, J., Klein, R., Behnke, S.: Mobile bin picking with an anthropomorphic service robot. In: 2013 IEEE International Conference on Robotics and Automation, pp. 2327–2334 (2013c)

    Google Scholar 

  47. Yan, W., Xu, Z., Zhou, X., Su, Q., Li, S., Wu, H.: Fast object pose estimation using adaptive threshold for bin-picking. IEEE Access 8, 63055–63064 (2020)

    Article  Google Scholar 

  48. Kleeberger, K., Landgraf, C., Huber, M.F.: Large-scale 6d object pose estimation dataset for industrial bin-picking (2019). arXiv:1912.12125

  49. Correll, N., Bekris, K.E., Berenson, D., Brock, O., Causo, A., Hauser, K., Okada, K., Rodriguez, A., Romano, J.M., Wurman, P.R.: Analysis and observations from the first amazon picking challenge. IEEE Trans. Autom. Sci. Eng. 15(1), 172–188 (2016)

    Article  Google Scholar 

  50. Hackett, D., Pippine, J., Watson, A., Sullivan, C., Pratt, G.: An overview of the darpa autonomous robotic manipulation (arm) program. J. Robot. Soc. Jpn. 31(4), 326–329 (2013)

    Article  Google Scholar 

  51. Morgan, A.S., Hang, K., Bircher, W.G., Alladkani, F.M., Gandhi, A., Calli, B., Dollar, A.M.: Benchmarking cluttered robot pick-and-place manipulation with the box and blocks test. IEEE Robot. Autom. Lett. 5(2), 454–461 (2019)

    Article  Google Scholar 

  52. Xiang, Y., Schmidt, T., Narayanan, V., Fox, D.: Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes (2017b). arXiv:1711.00199

  53. Kumar, S., Majumder, A., Dutta, S., Raja, R., Jotawar, S., Kumar, A., Soni, M., Raju, V., Kundu, O., Behera, E.H.L. et al.: Design and development of an automated robotic pick and stow system for an e-commerce warehouse (2017). arXiv:1703.02340

  54. Sun, Y., Falco, J., Cheng, N., Choi, H.R., Engeberg, E.D., Pollard, N., Roa, M., Xia, Z.: Robotic grasping and manipulation competition: task pool. In: Robotic Grasping and Manipulation Challenge, pp. 1–18. Springer, Berlin (2016)

    Google Scholar 

  55. Murase, H., Nayar, S.K.: Visual learning and recognition of 3-D objects from appearance. Int. J. Comput. Vision 14(1), 5–24 (1995)

    Article  Google Scholar 

  56. Hema, C.R., Paulraj, M.P., Nagarajan, R., Sazali, Y.: Segmentation and location computation of bin objects. Int. J. Adv. Robot. Syst. 4(1), 57–62 (2007)

    Article  Google Scholar 

  57. Tombari, F., Stefano, L.D.: Object recognition in 3d scenes with occlusions and clutter by hough voting. In: Proceedings of the 2010 Fourth Pacific-Rim Symposium on Image and Video Technology, PSIVT ’10, pp. 349–355 (2010)

    Google Scholar 

  58. Liu, M.-Y., Tuzel, O., Veeraraghavan, A., Taguchi, Y., Marks, T.K., Chellappa, R.: Fast object localization and pose estimation in heavy clutter for robotic bin picking. Int. J. Robot. Res. 31(8), 951–973 (2012)

    Article  Google Scholar 

  59. Tiwan, P., Boby, R.A., Roy, S.D., Chaudhury, S., Saha, S.K.: Cylindrical pellet pose estimation in clutter using a single robot mounted camera. In: Proceedings of Conference on Advances In Robotics, pp. 1–6 (2013)

    Google Scholar 

  60. Raghuvanshi, T., Chaudhary, S., Jain, V., Agarwal, S., Chaudhury, S.: Automated monocular vision based system for picking textureless objects. In: Proceedings of Conference on Advances In Robotics, pp. 1–6 (2015)

    Google Scholar 

  61. Matthias, N., David, D., Dirk, H., Joerg, S., Alexander, B., Jun, L., Reinhard, B.: Mobile bin picking with an anthropomorphic service robot. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 2327–2334 (2013a)

    Google Scholar 

  62. Buchholz, D., Futterlieb, M., Winkelbach, S., Wahl, F.M.: Efficient bin-picking and grasp planning based on depth data. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 3245–3250 (2013)

    Google Scholar 

  63. Domae, Y., Okuda, H., Taguchi, Y., Sumi, K., Hirai, T.: Fast grasp ability evaluation on single depth maps for bin picking with general grippers. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1997–2004 (2014)

    Google Scholar 

  64. SICK: Gigabit 3D vision for tough environments (2017). https://www.sick.com/us/en/vision/3d-vision/ruler/c/g138562

  65. MicroEpsilon: scanCONTROL selection (2017b). http://www.micro-epsilon.in/2D_3D/laser-scanner/model-selection/

  66. MicroEpsilon: Laser line triangulation (2017a). http://www.micro-epsilon.in/service/glossar/Laser-Linien-Triangulation.html

  67. Zhang, Q., Pless, R.: Extrinsic calibration of a camera and laser range finder (improves camera calibration). In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2004. (IROS 2004). Proceedings, vol. 3, pp. 2301–2306 (2004)

    Google Scholar 

  68. Unnikrishnan, R., Hebert, M.: Fast extrinsic calibration of a laser rangefinder to a camera. CMU Technical Report (2005)

    Google Scholar 

  69. Horaud, R., Dornaika, F.: Hand-eye calibration. Int. J. Robot. Res. 14(3), 195–210 (1995)

    Article  Google Scholar 

  70. Roy, M., Boby, R.A., Chaudhary, S., Chaudhury, S., Roy, S.D., Saha, S.K.: Pose estimation of texture-less cylindrical objects in bin picking using sensor fusion. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2016, pp. 2279–2284 (2016)

    Google Scholar 

  71. Haralick, R.M., Shapiro, L.G.: Computer and Robot Vision, vol. 2. Addison-Wesley Publishing Company, Masachusets, USA (1993)

    Google Scholar 

  72. Tsai, R.Y.: A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J. Robot. Autom. 3(4), 323–344 (1987)

    Article  Google Scholar 

  73. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)

    Article  Google Scholar 

  74. Richardson, A., Strom, J., Olson, E.: Aprilcal: Assisted and repeatable camera calibration. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1814–1821 (2013)

    Google Scholar 

  75. Lepetit, V., Moreno-Noguer, F., Fua, P.: Epnp: An accurate o(n) solution to the pnp problem. Int. J. Comput. Vision 81, 155–166 (2009)

    Article  Google Scholar 

  76. Boby, R.A.: Hand-eye calibration using a single image and robotic picking up using images lacking in contrast. In: 2020 International Conference Nonlinearity, Information and Robotics (NIR), pp. 1–6. IEEE (2020)

    Google Scholar 

  77. Rusu, R.B.: Semantic 3d object maps for everyday manipulation in human living environments. PhD thesis, TU Munich, Munich, Germany (2009)

    Google Scholar 

  78. Pless, R.: Code for calibration of cameras to planar laser range finders (2004). http://research.engineering.wustl.edu/~pless/code/calibZip.zip

  79. Matthias, N., David, D., Dirk, H., Joerg, S., Alexander, B., Jun, L., Reinhard, B.: Mobile bin picking with an anthropomorphic service robot. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 2327–2334 (2013b)

    Google Scholar 

  80. Liu, M.-Y., Tuzel, O., Veeraraghavan, A., Taguchi, Y., Marks, T.K., Chellappa., R.: Fast object localization and pose estimation in heavy clutter for robotic bin picking. Int. J. Robot. Res. 31(8), 951–973 (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Singapore University of Technology and Design, Singapore, Singapore

    Abdullah Aamir Hayat

  2. Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, Delhi, India

    Shraddha Chaudhary

  3. Innopolis University, Innopolis, Russia

    Riby Abraham Boby

  4. Department of Mechanical Engineering, Indian Institute of Technology(ISM), Ranchi, Jharkhand, India

    Arun Dayal Udai

  5. Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India

    Sumantra Dutta Roy

  6. Department of Mechanical Engineering, Indian Institute of Technology Delhi, New Delhi, Delhi, India

    Subir Kumar Saha

  7. Indian Institute of Technology Jodhpur, Jodhpur, Rajasthan, India

    Santanu Chaudhury

Authors
  1. Abdullah Aamir Hayat
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shraddha Chaudhary
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Riby Abraham Boby
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arun Dayal Udai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sumantra Dutta Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Subir Kumar Saha
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Santanu Chaudhury
    View author publications

    You can also search for this author in PubMed Google Scholar

2.1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 14348 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Hayat, A.A. et al. (2022). Vision System and Calibration. In: Vision Based Identification and Force Control of Industrial Robots. Studies in Systems, Decision and Control, vol 404. Springer, Singapore. https://doi.org/10.1007/978-981-16-6990-3_2

Download citation

Publish with us

Policies and ethics

Search

Navigation