Skip to main content
SpringerLink
Log in

Automatic digitalization and performance assessment of manual assembly processes using a marker-less motion tracking approach

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Manual assembly operations are still essential in modern manufacturing industries because of their flexibility and adaptability for mass customized production. However, as the product variety increases, the complexity of the assembly operations and workstations increases. Thus, several investigations have focused on improving manual assembly operations. One of the main challenges is the capture, digitalization and transferring of the assembly knowledge and skills of qualified assembly operators. In this paper, a novel approach to digitalize and evaluate the performance of manual assembly operators based on a marker-less motion tracking strategy is proposed. The aim is twofold: firstly, to digitalize manual assembly operations in order to log the experience and knowledge of qualified operators; and secondly, to objectively evaluate the assembly performance of new operators or trainees in order to qualify them. In addition, several new assembly metrics are proposed to quantitatively evaluate and compare the operator’s assembly performance with the performance of a qualified operator. The proposed marker-less motion approach uses a leap motion sensor to track and capture the hand motions and execution times of an operator during the realization of manual assembly processes. A set of experimental tests was conducted to prove and validate the functionality of the proposed method. The results have demonstrated that the proposed strategy allows the digitalization of manual assembly operations and the immediate evaluation of the operator’s performance.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Relji´c VR, Milenkovi´c IM, Dudi´cdudi´c S, Šulc J, Bajči B (2021) Augmented reality applications in Industry 4.0 environment, Appl Sci 11(12):5592. https://doi.org/10.3390/APP11125592

  2. Santi GM, Ceruti A, Liverani A, Osti F (2021) Augmented reality in Industry 4.0 and future innovation programs. Technologies 9(2):33. https://doi.org/10.3390/technologies9020033

    Article  Google Scholar 

  3. Nee AYC, Ong SK, Chryssolouris G, Mourtzis D (2012) Augmented reality applications in design and manufacturing. CIRP Ann Manuf Technol 61:657–679. https://doi.org/10.1016/j.cirp.2012.05.010

    Article  Google Scholar 

  4. Rodriguez L, Quint F, Gorecky D, Romero D, Siller HR (2015) Developing a mixed reality assistance system based on projection mapping technology for manual operations at assembly workstations. Procedia Comput Sci 75:327–333. https://doi.org/10.1016/J.PROCS.2015.12.254

    Article  Google Scholar 

  5. Müller BC, Nguyen TD, Dang QV, Duc BM, Seliger G, Krüger J et al (2016) Motion tracking applied in assembly for worker training in different locations. Procedia CIRP 48:460–465. https://doi.org/10.1016/j.procir.2016.04.117

    Article  Google Scholar 

  6. Liu L, Liu Y, Zhang J (2019) Learning-based hand motion capture and understanding in assembly process. IEEE Trans Ind Electron 66(12):9703–9712. https://doi.org/10.1109/TIE.2018.2884206

    Article  Google Scholar 

  7. Pilati F, Faccio M, Gamberi M, Regattieri A (2020) Learning manual assembly through real-time motion capture for operator training with augmented reality. Procedia Manuf 45:189–195. https://doi.org/10.1016/j.promfg.2020.04.093

    Article  Google Scholar 

  8. Mitzner K, Doe B, Akulin A, Suponin A, Müller D (2019) Introduction to design for manufacturing. In: Mitzner IK, Doe B, Akulin A, Suponin A, Müller D (eds) Complete PCB Design Using OrCAD® Capture and PCB, 2nd edn. Academic Press, Amsterdam, pp 83–109. https://doi.org/10.1016/B978-0-12-817684-9.00005-9

    Chapter  Google Scholar 

  9. Miqueo A, Torralba M, Yagüe-Fabra JA (2020) Lean Manual Assembly 4.0: a systematic review. Appl Sci 10:8555. https://doi.org/10.3390/APP10238555.

    Article  Google Scholar 

  10. Johansson PEC, Malmsköld L, Fast-Berglund Å, Moestam L (2018) Enhancing future assembly information systems – putting theory into practice. Procedia Manuf 17:491–498. https://doi.org/10.1016/J.PROMFG.2018.10.088

    Article  Google Scholar 

  11. Turk M, Šimic M, Pipan M, Herakovič N (2022) Multi-criterial algorithm for the efficient and ergonomic manual assembly process. Int J Environ Res Public Health 19:3496. https://doi.org/10.3390/IJERPH19063496

    Article  Google Scholar 

  12. Cohen Y, Faccio M, Pilati F, Yao X (2019) Design and management of digital manufacturing and assembly systems in the Industry 4.0 era. Int J Adv Manuf Technol 105:3565–3577. https://doi.org/10.1007/S00170-019-04595-0/FIGURES/3

    Article  Google Scholar 

  13. Agethen P, Otto M, Mengel S, Rukzio E (2016) Using marker-less motion capture systems for walk path analysis in paced assembly flow lines. Procedia CIRP 54:152–157. https://doi.org/10.1016/j.procir.2016.04.125

    Article  Google Scholar 

  14. Ferrari E, Gamberi M, Pilati F, Regattieri A (2018) Motion analysis system for the digitalization and assessment of manual manufacturing and assembly processes. IFAC-PapersOnLine 51(11):411–416. https://doi.org/10.1016/j.ifacol.2018.08.329

    Article  Google Scholar 

  15. Bortolini M, Gamberi M, Pilati F, Regattieri A (2018) Automatic assessment of the ergonomic risk for manual manufacturing and assembly activities through optical motion capture technology. Procedia CIRP 72:81–86. https://doi.org/10.1016/j.procir.2018.03.198

    Article  Google Scholar 

  16. Wang P, Liu H, Wang L, Gao RX (2018) Deep learning-based human motion recognition for predictive context-aware human-robot collaboration. CIRP Annals 67:17–20. https://doi.org/10.1016/j.cirp.2018.04.066

    Article  Google Scholar 

  17. Bortolini M, Faccio M, Gamberi M, Pilati F (2020) Motion analysis system (MAS) for production and ergonomics assessment in the manufacturing processes. Comput Ind Eng 139:105485. https://doi.org/10.1016/j.cie.2018.10.046

    Article  Google Scholar 

  18. Hu H, Cao Z, Yang X, Xiong H, Lou Y (2021) Performance evaluation of optical motion capture sensors for assembly motion capturing. IEEE Access 9:61444–61454. https://doi.org/10.1109/ACCESS.2021.3074260

    Article  Google Scholar 

  19. Ja YC, Jong MK, Chang OK, Yoon SK, Seung JL (2007) Process start/end event detection and dynamic time warping algorithms for run-by-run process fault detection. In: IEEE International Symposium on Semiconductor Manufacturing Conference Proceedings, Santa Clara, CA, pp 1–4. https://doi.org/10.1109/ISSM.2007.4446846

  20. West N, Schlegl T, Deuse J (2021) Feature extraction for time series classification using univariate descriptive statistics and dynamic time warping in a manufacturing environment. In: IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE), Nanchang, China, pp 762–768. https://doi.org/10.1109/ICBAIE52039.2021.9389954

  21. Volke J, Heim HP (2023) Evaluation of the injection molding process behavior during start-up and after parameter changes using dynamic time warping correspondences. J Manuf Process 95:183–203. https://doi.org/10.1016/J.JMAPRO.2023.03.076

    Article  Google Scholar 

  22. Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13:6380–6393. https://doi.org/10.3390/s130506380

    Article  Google Scholar 

  23. Tao Y, Both A, Silveira RI, Buchin K, Sijben S, Purves RS et al (2021) A comparative analysis of trajectory similarity measures. GIsci Remote Sens 58:643–669. https://doi.org/10.1080/15481603.2021.1908927

    Article  Google Scholar 

  24. Su H, Liu S, Zheng B, Zhou X, Zheng K (2020) A survey of trajectory distance measures and performance evaluation. VLDB Journal 29:3–32. https://doi.org/10.1007/s00778-019-00574-9

    Article  Google Scholar 

  25. Gong S, Cartlidge J, Bai R, Yue Y, Li Q, Qiu G (2020) Extracting activity patterns from taxi trajectory data: a two-layer framework using spatio-temporal clustering, Bayesian probability and Monte Carlo simulation. Int J Geogr Inf Sci 34:1210–1234. https://doi.org/10.1080/13658816.2019.1641715

    Article  Google Scholar 

  26. Nathan R, Getz WM, Revilla E, Holyoak M, Kadmon R, Saltz D et al (2008) A movement ecology paradigm for unifying organismal movement research. Proc Natl Acad Sci USA 105:19052–19059. https://doi.org/10.1073/pnas.0800375105

    Article  Google Scholar 

  27. Andersson M, Gudmundsson J, Laube P, Wolle T, Andersson M, Gudmundsson J et al (2008) Reporting leaders and followers among trajectories of moving point objects. Geoinformatica 12:497–528. https://doi.org/10.1007/s10707-007-0037-9

    Article  Google Scholar 

  28. Wozniak P, Vauderwange O, Mandal A, Javahiraly N, Curticapean D (2016) Possible applications of the LEAP motion controller for more interactive simulated experiments in augmented or virtual reality. In: Proc. SPIE 9946, Optics Education and Outreach IV 9946:99460P. https://doi.org/10.1117/12.2237673

    Chapter  Google Scholar 

  29. Vaughan N, Gabrys B (2016) Comparing and combining time series trajectories using dynamic time warping. Procedia Comput Sci 96:465–474. https://doi.org/10.1016/j.procs.2016.08.106

    Article  Google Scholar 

Download references

Funding

The authors would like to thank the financial support from the National Science and Technology Council (CONACYT) of Mexico, grant number CB-2010-01-154430. The first author also acknowledges CONACYT for the scholarship provided for his PhD studies.

Author information

Authors and Affiliations

  1. Facultad de Ingeniería, Universidad Autónoma de San Luis Potosí, Av. Manuel Nava No. 8, Zona Universitaria, 78290, San Luis Potosí, S.L.P., México

    Rodrigo Delgadillo-Gaytan & Hugo I. Medellin-Castillo

Authors
  1. Rodrigo Delgadillo-Gaytan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hugo I. Medellin-Castillo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the design, development and evaluation of the proposed system, and in the analysis of the results and preparation of the manuscript. The first draft of the manuscript was written by Rodrigo Delgadillo-Gaytan and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Hugo I. Medellin-Castillo.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Delgadillo-Gaytan, R., Medellin-Castillo, H.I. Automatic digitalization and performance assessment of manual assembly processes using a marker-less motion tracking approach. Int J Adv Manuf Technol 129, 5101–5115 (2023). https://doi.org/10.1007/s00170-023-12496-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-023-12496-6

Keywords

Search

Navigation