Automatic robot path integration using three-dimensional vision and offline programming

  • Amit Kumar Bedaka
  • Joel Vidal
  • Chyi-Yeu LinEmail author


In manufacturing industries, offline programming (OLP) platforms provide an independent methodology for robot integration using 3D model simulation away from the actual robot cell and production process, reducing integration time and costs. However, traditional OLP platforms still require prior knowledge of the workpiece position in a predefined environment, which requires complex human operations and specific-purpose designs, highly reducing the autonomy of the systems. The presented approach proposes to overcome these problems by defining a novel automated offline programming system (AOLP), which integrates a flexible and intuitive OLP platform with a state-of-the-art autonomous object pose estimation method, to achieve an environment and model independent platform for automatic robotic manufacturing. The autonomous recognition capabilities of the three-dimensional vision system provide the relative position of the workpiece model in the OLP platform, with robustness against clutter, illumination, and object material. After that, the user-friendly OLP platform allows an efficient and automatic path generation, simulation, robot code generation, and robot execution. The proposed system precision and robustness are analyzed and validated in a real-world environment on four different sets of experiment. Finally, the proposed system’s features are discussed and compared with other available solutions for practical industrial manufacturing, showing the advantages of the proposed approach. Overall, despite sensor resolution limitations, the proposed system shows a remarkable precision and promising direction towards highly efficient and productive manufacturing solutions.


Automated offline programming Path generation Industrial manipulator Machine vision 3D object recognition 6D pose estimation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


Funding information

This work is financially supported by both Taiwan Building Technology Center and Center for Cyber-Physical System Innovation from the Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan. Additionally, this work was financially supported by the Ministry of Science and Technology, Taiwan (R.O.C), under the grant—105-2221-E-011-088-MY2.


  1. 1.
    (2012). ROS-Industrial., accessed: 2018-09-10
  2. 2.
    (2017). OPEN CASCADE., [Accessed 18-June-2018]
  3. 3.
    Sett A, Vollmann K (2002) Computer based robot training in a virtual environment. In: IEEE International conference on industrial technology, 2002, vol 2. IEEE ICIT ’02, pp 1185–1189Google Scholar
  4. 4.
    Aldoma A, Vincze M, Blodow N, Gossow D, Gedikli S, Rusu RB, Bradski G (2011) CAD-model recognition and 6DOF pose estimation using 3D cues. In: 2011 IEEE international conference on computer vision workshops (ICCV Workshops), pp 585–592Google Scholar
  5. 5.
    Ames AL, Hinman-Sweeney EM, Sizemore JM (2005) Automated generation of weld path trajectories. In: (ISATP 2005). The 6th IEEE international symposium on assembly and task planning: from nano to macro assembly and manufacturing, 2005, pp 182–187Google Scholar
  6. 6.
    Andreopoulos A, Tsotsos JK (2013) 50 years of object recognition: directions forward. Comput Vis Image Underst 117(8):827–891CrossRefGoogle Scholar
  7. 7.
    Baizid K, Cukovic S, Iqbal J, Yousnadj A, Chellali R, Meddahi A, Devedzic G, Ghionea I (2016) IRoSim: industrial robotics simulation design planning and optimization platform based on CAD and knowledgeware technologies. Robot Comput Integr Manuf 42:121–134CrossRefGoogle Scholar
  8. 8.
    Bedaka AK, Lin CY (2017) Autonomous path generation platform for robot simulation. In: 2017 international conference on advanced robotics and intelligent systems (ARIS), pp 63–68Google Scholar
  9. 9.
    Bedaka AK, Lin CY (2018) CAD-based robot path planning and simulation using open cascade. Procedia Computer Science 133:779–785CrossRefGoogle Scholar
  10. 10.
    Bedaka AK, Mahmoud AM, Lee SC, Lin CY (2018) Autonomous robot-guided inspection system based on offline programming and RGB-D model. Sensors, vol 18 (11)Google Scholar
  11. 11.
    Beetz M, Klank U, Kresse I, Maldonado A, Mösenlechner L, Pangercic D, Rühr T, Tenorth M (2011) Robotic roommates making pancakes. In: 2011 11th IEEE-RAS international conference on humanoid robots (Humanoids), IEEE, pp 529–536Google Scholar
  12. 12.
    Besl PJ, Mckay ND (1992) Method for registration of 3-D shapes. In: Sensor fusion IV: control paradigms and data structures, International society for optics and photonics, vol 1611, pp 586–607Google Scholar
  13. 13.
    Blanco JL (2010) A tutorial on se(3) transformation parameterizations and on-manifold optimization. Tech. Rep. 012010, University of MalagaGoogle Scholar
  14. 14.
    Brachmann E, Krull A, Michel F, Gumhold S, Shotton J, Rother C (2014) Learning 6d object pose estimation using 3d object coordinates. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer vision – ECCV, vol 2014. Springer International Publishing, Cham, pp 536–551Google Scholar
  15. 15.
    Caccavale F, Uchiyama M (2008) Cooperative manipulators. Springer, Berlin, pp 701–718Google Scholar
  16. 16.
    Caruso L, Russo R, Savino S (2017) Microsoft kinect v2 vision system in a manufacturing application. Robot Comput Integr Manuf 48:174–181CrossRefGoogle Scholar
  17. 17.
    Chen Y, Medioni G (1992) Object modelling by registration of multiple range images. Image Vis Comput 10(3):145–155CrossRefGoogle Scholar
  18. 18.
    Correll N, Bekris KE, Berenson D, Brock O, Causo A, Hauser K, Okada K, Rodriguez A, Romano JM, Wurman PR (2018) Analysis and observations from the first amazon picking challenge. IEEE Trans Autom Sci Eng 15(1):172–188. CrossRefGoogle Scholar
  19. 19.
    Daniilidis K (1999) Hand-eye calibration using dual quaternions. Int J Robot Res 18(3):286–298MathSciNetCrossRefGoogle Scholar
  20. 20.
    Desouza GN, Kak AC (2002) Vision for mobile robot navigation: a survey. IEEE Trans Pattern Anal Mach Intell 24(2):237–267CrossRefGoogle Scholar
  21. 21.
    Drost B, Ulrich M, Navab N, Ilic S (2010) Model globally, match locally: efficient and robust 3D object recognition. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 998–1005Google Scholar
  22. 22.
    Eppner C, Höfer S, Jonschkowski R, Martín-Martín R, Sieverling A, Wall V, Brock O (2017) Lessons from the amazon picking challenge: four aspects of building robotic systems. In: Proceedings of the 26th international joint conference on artificial intelligence, IJCAI’17. AAAI Press, pp 4831–4835.
  23. 23.
    Everingham M, Eslami SMA, Van Gool L, Williams CKI, Winn J, Zisserman A (2015) The Pascal visual object classes challenge: a retrospective. Int J Comput Vis 111(1):98–136CrossRefGoogle Scholar
  24. 24.
    Geng J (2011) Structured-light 3D surface imaging: a tutorial. Adv Opt Photon 3(2):128–160CrossRefGoogle Scholar
  25. 25.
    Guo Y, Bennamoun M, Sohel F, Lu M, Wan J (2014) 3D object recognition in cluttered scenes with local surface features: a survey. IEEE Trans Pattern Anal Mach Intell 36(11):2270–2287CrossRefGoogle Scholar
  26. 26.
    Hinterstoisser S, Holzer S, Cagniart C, Ilic S, Konolige K, Navab N, Lepetit V (2011) Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In: 2011 international conference on computer vision, pp 858–865Google Scholar
  27. 27.
    Hodan T, Zabulis X, Lourakis M, Obdrzalek S, Matas J (2015) Detection and fine 3D pose estimation of texture-less objects in RGB-D images. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 4421–4428Google Scholar
  28. 28.
    Hodan T, Michel F, Brachmann E, Kehl W, GlentBuch A, Kraft D, Drost B, Vidal J, Ihrke S, Zabulis X, Sahin C, Manhardt F, Tombari F, Kim TK, Matas J, Rother C (2018) Bop: benchmark for 6d object pose estimation. In: The European conference on computer vision (ECCV)Google Scholar
  29. 29.
    Horaud R, Dornaika F (1995) Hand-eye calibration. Int J Robot Res 14(3):195–210CrossRefGoogle Scholar
  30. 30.
    Jing L, Fengshui J, En L (2016) RGB-D sensor-based auto path generation method for arc welding robot. In: 2016 Chinese control and decision conference (CCDC), pp 4390–4395,
  31. 31.
    Kehl W, Milletari F, Tombari F, Ilic S, Navab N Leibe B, Matas J, Sebe N, Welling M (eds) (2016) Deep learning of local RGB-D patches for 3D object detection and 6D pose estimation, vol 2016. Springer International Publishing, ChamGoogle Scholar
  32. 32.
    Kehl W, Manhardt F, Tombari F, Ilic S, Navab N (2017) SSD-6D: making RGB-based 3D detection and 6D pose estimation great again. In: The IEEE international conference on computer vision (ICCV)Google Scholar
  33. 33.
    Kržič P, Pušavec F, Kopač J (2013) Kinematic constraints and offline programming in robotic machining applications. Tehnički vjesnik/Technical Gazette 20(1):117–124Google Scholar
  34. 34.
    Larkin N, Milojevic A, Pan Z, Polden J, Norrish J (2011) Offline programming for short batch robotic welding. In: 16th joining of materials (JOM) conference 2012, pp 1–6Google Scholar
  35. 35.
    Larkin N, Pan Z, Duin SV, Norrish J (2013) 3D mapping using a ToF camera for self programming an industrial robot. In: 2013 IEEE/ASME international conference on advanced intelligent mechatronics, pp 494–499Google Scholar
  36. 36.
    Lippiello V, Siciliano B, Villani L (2007) Position-based visual servoing in industrial multirobot cells using a hybrid camera configuration. IEEE Trans Robot 23(1):73–86CrossRefGoogle Scholar
  37. 37.
    Mahmoud N, Ilhan EK (2016) Off-line nominal path generation of 6-DoF robotic manipulator for edge finishing and inspection processes. Int J Adv Manuf Technol 99(1-4):85–96Google Scholar
  38. 38.
    Maiolino P, Woolley RAJ, Popov A, Ratchev S (2016) Structural quality inspection based on a RGB-D sensor: supporting manual-to-automated assembly operations. SAE Int J Mater Manuf 9(1):12–15. Google Scholar
  39. 39.
    Maiolino P, Woolley R, Branson D, Benardos P, Popov A, Ratchev S (2017) Flexible robot sealant dispensing cell using RGB-D sensor and off-line programming. Robot Comput Integr Manuf 48:188–195CrossRefGoogle Scholar
  40. 40.
    Malamas EN, Petrakis EG, Zervakis M, Petit L, Legat JD (2003) A survey on industrial vision systems, applications and tools. Image Vis Comput 21(2):171–188CrossRefGoogle Scholar
  41. 41.
    Mitsi S, Bouzakis KD, Mansour G, Sagris D, Maliaris G (2004) Off-line programming of an industrial robot for manufacturing. Int J Adv Manuf Technol 26(3):262–267CrossRefGoogle Scholar
  42. 42.
    Neto P, Mendes N (2013) Direct off-line robot programming via a common CAD package. Robot Auton Syst 61(8):896–910CrossRefGoogle Scholar
  43. 43.
    Neto P, Mendes N, Araújo R, Pires JN, Moreira AP (2012) High-level robot programming based on CAD: dealing with unpredictable environments. Ind Robot Int J Robot Res Appl 39(3):294–303CrossRefGoogle Scholar
  44. 44.
    Newman TS, Jain AK (1995) A survey of automated visual inspection. Comput Vis Image Underst 61(2):231–262CrossRefGoogle Scholar
  45. 45.
    Pan Z, Polden J, Larkin N, Duin SV, Norrish J (2010) Recent progress on programming methods for industrial robots. In: ISR 2010 (41st international symposium on robotics) and ROBOTIK 2010 (6th German conference on robotics), pp 1–8Google Scholar
  46. 46.
    Polden J, Pan Z, Larkin N, Van Duin S, Norrish J (2011) Offline programming for a complex welding system using DELMIA automation. In: Chen SB, Fang G (eds) Robotic welding, intelligence and automation. Springer, Berlin, pp 341– 349Google Scholar
  47. 47.
    Pollard NS, Hodgins JK, Riley MJ, Atkeson CG (2002) Adapting human motion for the control of a humanoid robot. In: Proceedings 2002 IEEE international conference on robotics and automation (Cat. No.02CH37292), vol 2, pp 1390–1397Google Scholar
  48. 48.
    Radovan H, Daynier RDS, Peter K, Roman R (2014) Offline programming of an ABB robot using imported CAD models in the RobotStudio software environment. Appl Mech Mater 693:62–67CrossRefGoogle Scholar
  49. 49.
    Rocha LF, Ferreira M, Santos V, Moreira AP (2014) Object recognition and pose estimation for industrial applications: a cascade system. Robot Comput Integr Manuf 30(6):605–621CrossRefGoogle Scholar
  50. 50.
    Rusu RB, Cousins S (2011) 3d is here: point cloud library (pcl). In: 2011 IEEE international conference on robotics and automation, pp 1–4Google Scholar
  51. 51.
    Rusu RB, Bradski G, Thibaux R, Hsu J (2010) Fast 3d recognition and pose using the viewpoint feature histogram. In: 2010 IEEE/RSJ international conference on intelligent robots and systems, pp 2155–2162Google Scholar
  52. 52.
    Shah HNM, Sulaiman M, Shukor AZ, Kamis Z (2018) An experiment of detection and localization in tooth saw shape for butt joint using KUKA welding robot. Int J Adv Manuf Technol 97(5):3153–3162CrossRefGoogle Scholar
  53. 53.
    Shiu YC, Ahmad S (1989) Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB. IEEE Trans Robot Autom 5(1):16–29CrossRefGoogle Scholar
  54. 54.
    Tsai RY, Lenz RK (1989) A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Trans Robot Autom 5(3):345–358CrossRefGoogle Scholar
  55. 55.
    Tuytelaars T, Mikolajczyk K (2008) Local invariant feature detectors: a survey. Foundations and Trendsin Computer Graphics and Vision 3(3):177–280CrossRefGoogle Scholar
  56. 56.
    Ulrich M, Steger C (2002) Performance evaluation of 2D object recognition techniques. Int Arch Photogramm Remote Sens Spat Inf Sci 34(3/A):368–374Google Scholar
  57. 57.
    Vidal J, Lin C, Martí R (2018) 6D pose estimation using an improved method based on point pair features, pp 405–409.
  58. 58.
    Wohlkinger W, Vincze M (2011) Ensemble of shape functions for 3d object classification. In: 2011 IEEE international conference on robotics and biomimetics, pp 2987–2992Google Scholar
  59. 59.
    Zhu W, Qu W, Cao L, Yang D, Ke Y (2013) An off-line programming system for robotic drilling in aerospace manufacturing. Int J Adv Manuf Technol 68(9):2535–2545CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  • Amit Kumar Bedaka
    • 1
  • Joel Vidal
    • 1
  • Chyi-Yeu Lin
    • 1
    • 2
    • 3
    Email author
  1. 1.Department of Mechanical EngineeringNational Taiwan University of Science and TechnologyTaiwanRepublic of China
  2. 2.Taiwan Building Technology CenterNational Taiwan University of Science and TechnologyTaiwanRepublic of China
  3. 3.Center for Cyber-Physical System InnovationNational Taiwan University of Science and TechnologyTaiwanRepublic of China

Personalised recommendations