Skip to main content

Advertisement

SpringerLink
Log in

Strategy with machine learning models for precise assembly using programming by demonstration

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Programming by demonstration (PBD), which is applied in industry as a method of human–robot collaboration for assembly tasks, such as placing, inserting, and screwing, has seen limited application in commercial electronic product assembly due to a lack of trajectory planning optimization. In this research, we propose a framework with two custom algorithms to preprocess and classify contactless demonstration performance. This framework enables the generation of optimal motion paths based on that criteria of distance, smoothness, and trajectory variance rather than canonical methods. Machine learning methods, including convolutional neural network (CNN), artificial neural network (ANN), and support vector machine (SVM), are feasible for predicting the best motion path with an accuracy range of 80% to 85%. Among these methods, CNN, specifically DarkNet, achieves the highest accuracy. Future work involves development of hybrid CNN/ANN algorithms, which may yield higher accuracy in prediction. In addition, the proposed algorithms may be applied to robots equipped with dual assembly arms and other complex assemblies that emulate human arms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Algorithm 1
Fig. 11
Algorithm 2
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Code or data availability

The code and data are available on GitHub https://github.com/baiye225/PBD

References

  1. Zhu Z, Hu H (2018) Robot learning from demonstration in robotic assembly: a survey. Robotics 7(2):17. https://doi.org/10.3390/robotics7020017

    Article  Google Scholar 

  2. Takamatsu J, Ogawara K, Kimura H, Ikeuchi K (2007) Recognizing assembly tasks through human demonstration. Int J Robot Res 26(7):641–659

    Article  Google Scholar 

  3. Merlin Deib PdM (2020) 6-DOF lead through robot programming. YouTube, RealPars. https://youtube.com/watch?v=3hcqqPcMlow

  4. Peternel L, Petrič T, Babič J (2018) Robotic assembly solution by human-in-the-loop teaching method based on real-time stiffness modulation. Auton Robot 42(1):1–17

    Article  Google Scholar 

  5. Suomalainen M, Kyrki V (2017) A geometric approach for learning compliant motions from demonstration. In: 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids). IEEE, Birmingham, UK, pp 783–790. https://doi.org/10.1109/HUMANOIDS.2017.8246961

  6. Rozo L, Calinon S, Caldwell D, Jiménez P, Torras C (2013) Learning collaborative impedance-based robot behaviors. Proc AAAI Conf Artif Intell 27(1):1422–1428

    Google Scholar 

  7. Duque DA, Prieto FA, Hoyos JG (2019) Trajectory generation for robotic assembly operations using learning by demonstration. Robot Comput-Integr Manuf 57:292–302

    Article  Google Scholar 

  8. Eiband T, Saveriano M, Lee D (2019) Learning haptic exploration schemes for adaptive task execution. In: 2019 International Conference on Robotics and Automation (ICRA). ICRA, Montreal, QC, Canada, pp 7048–7054. https://doi.org/10.1109/ICRA.2019.8793934

  9. Jiang J, Huang Z, Bi Z, Ma X, Yu G (2020) State-of-the-art control strategies for robotic PiH assembly. Robot Comput-Integr Manuf 65:101894

    Article  Google Scholar 

  10. Wang B, Hu SJ, Sun L, Freiheit T (2020) Intelligent welding system technologies: state-of-the-art review and perspectives. J Manuf Syst 56:373–391

    Article  Google Scholar 

  11. Ortega-Aranda D, Jimenez-Vielma JF, Saha BN, Lopez-Juarez I (2021) Dual-arm peg-in-hole assembly using DNN with double force/torque sensor. Appl Sci 11(15):6970. https://doi.org/10.3390/app11156970

    Article  Google Scholar 

  12. Weng C-Y, Tan WC, and Chen IM (2019) "A survey of dual-arm robotic issues on assembly tasks," ROMANSY 22 – Robot Des, Dyn Control, (CISM Int Centre Mech Sci Chapter 59:474–480

  13. Ambhore S (2020) A comprehensive study on robot learning from demonstration. In: 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA). ICIMIA, Bangalore, India, pp 291-299. https://doi.org/10.1109/ICIMIA48430.2020.9074946

  14. Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res 37(4–5):421–436

    Article  Google Scholar 

  15. Ostanin M, Popov D, Klimchik A (2018) Programming by demonstration using two-step optimization for industrial robot. IFAC-PapersOnLine 51(11):72–77

    Article  Google Scholar 

  16. Wang Y, Jiao Y, Xiong R, Yu H, Zhang J, Liu Y (2018) MASD: a multimodal assembly skill decoding system for robot programming by demonstration. IEEE Trans Autom Sci Eng 15(4):1722–1734. https://doi.org/10.1109/tase.2017.2783342

    Article  Google Scholar 

  17. Janglová D (2004) Neural networks in mobile robot motion. Int J Adv Rob Syst 1(1):2

    Article  Google Scholar 

  18. Lembono TS, Paolillo A, Pignat E, Calinon S (2020) Memory of motion for warm-starting trajectory optimization. IEEE Robot Autom Lett 5(2):2594–2601. https://doi.org/10.1109/lra.2020.2972893

    Article  Google Scholar 

  19. Wang J (2020) "An intuitive tutorial to Gaussian processes regression," arXiv preprint arXiv:2009.10862

  20. Iklima Z, Muthahhar MI, Khan A, Zody A (2021) Self-learning of delta robot using inverse kinematics and artificial neural networks. Sinergi 25(3):237. https://doi.org/10.22441/sinergi.2021.3.001

    Article  Google Scholar 

  21. Panwar R, Sukavanam N (2018) Trajectory tracking using artificial neural network for stable human-like gait with upper body motion. Neural Comput Appl 32(7):2601–2619. https://doi.org/10.1007/s00521-018-3842-1

    Article  Google Scholar 

  22. Ang ZH, Ang CK, Lim WH, Yu LJ, and Solihin MI (2020) Development of an artificial intelligent approach in adapting the characteristic of polynomial trajectory planning for robot manipulator. Int J Mech Eng Robot Res 408–414. https://doi.org/10.18178/ijmerr.9.3.408-414

  23. Koenig A, y Baena FR, Secoli R (2021) Gesture-based teleoperated grasping for educational robotics. In: 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN). IEEE, Vancouver, BC, Canada, pp 222–228. https://doi.org/10.1109/RO-MAN50785.2021.9515390

  24. Marin G, Dominio F, Zanuttigh P (2014) Hand gesture recognition with leap motion and kinect devices. In: 2014 IEEE International Conference on Image Processing (ICIP). IEEE, Paris, France, pp 1565–1569. https://doi.org/10.1109/ICIP.2014.7025313

  25. Alchakov V, Kramar V, Larionenko A (2020) Basic approaches to programming by demonstration for an anthropomorphic robot. IOP Conf Ser: Mater Sci Eng 709(2):022092

    Article  Google Scholar 

  26. Mccrate MP (2010) Modern mechanical automata. University of Cincinnati

    Google Scholar 

  27. Guna J, Jakus G, Pogacnik M, Tomazic S, Sodnik J (2014) An analysis of the precision and reliability of the leap motion sensor and its suitability for static and dynamic tracking. Sensors (Basel) 14(2):3702–3720. https://doi.org/10.3390/s140203702

    Article  Google Scholar 

  28. Tung JY, Lulic T, Gonzalex DA, Tran J, Dickerson CR, Roy EA (2015) Evaluation of a portable markerless finger position capture device: accuracy of the Leap Motion controller in healthy adults. IOP Sci. https://doi.org/10.1088/0967-3334/36/5/1025

Download references

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, Texas A&M University, College Station, TX, USA

    Ye Bai

  2. Department of Engineering Technology (Joint Appointment With Department of Mechanical Engineering), Texas A&M University, College Station, TX, USA

    Sheng-Jen Hsieh

Authors
  1. Ye Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sheng-Jen Hsieh
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed conception, data analysis, and experiment design. The methodology, program, experiment setup, and 1st draft manuscript were written by Ye Bai, and all authors made comments and revisions.

Corresponding author

Correspondence to Ye Bai.

Ethics declarations

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The authors are with the Rockwell Automation System Integration Laboratory, Texas A&M University, College Station, TX 77840 USA (e-mail: yebai@tamu.edu; hsieh@tamu.edu).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bai, Y., Hsieh, SJ. Strategy with machine learning models for precise assembly using programming by demonstration. Int J Adv Manuf Technol 127, 3699–3714 (2023). https://doi.org/10.1007/s00170-023-11659-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-023-11659-9

Keywords

Search

Navigation