Skip to main content

Advertisement

SpringerLink
Log in

SPECTRE: a deep learning network for posture recognition in manufacturing

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

Work-related musculoskeletal disorders are a very impactful problem, both socially and economically, in the manufacturing sector. To control their effect, standardised methods and technologies for ergonomic assessment have been developed. The main technologies used are inertial sensors and vision-based systems. The former are accurate and reliable, but invasive and not affordable for many companies. The latter use machine learning algorithms to detect human pose and assess ergonomic risks. In this paper, using data collecting by reproducing the working environment in LUBE, the major Italian kitchen manufacturer, we propose SPECTRE (Sensor-independent Parallel dEep ConvoluTional leaRning nEtwork): a fully sensor-independent learning model based on convolutional networks to classify postures in the workplace. This system assesses ergonomic risks in major body segments through Deep Learning with a minimal impact. SPECTRE’s performance is evaluated using established metrics for imbalanced data (precision, recall, F1-score and area under the precision-recall curve). Overall, SPECTRE shows good performance and, thanks to an agnostic explainable machine learning method, is able to extrapolate which patterns are significant in the input.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and code

The datasets and the code used and/or analysed during the current study are available from the corresponding author on reasonable request.

Notes

  1. https://github.com/google/mediapipe.

References

  • Abobakr, A., Nahavandi, D., Hossny, M., Iskander, J., Attia, M., Nahavandi, S., et al. (2019). RGB-D ergonomic assessment system of adopted working postures. Applied Ergonomics, 80, 75–88. https://doi.org/10.1016/j.apergo.2019.05.004.

    Article  Google Scholar 

  • Al-Amin, M., Qin, R., Moniruzzaman, M., Yin, Z., Tao, W., & Leu, M. (2021). An individualized system of skeletal data-based CNN classifiers for action recognition in manufacturing assembly. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-021-01815-x.

    Article  Google Scholar 

  • Andrade-Ambriz, Y. A., Ledesma, S., Ibarra-Manzano, M. A., Oros-Flores, M. I., & Almanza-Ojeda, D. L. (2022). Human activity recognition using temporal convolutional neural network architecture. Expert Systems with Applications, 191, 116287. https://doi.org/10.1016/j.eswa.2021.116287.

    Article  Google Scholar 

  • Battini, D., Berti, N., Finco, S., Guidolin, M., Reggiani, M., & Tagliapietra, L. (2022). WEM-Platform: A real-time platform for full-body ergonomic assessment and feedback in manufacturing and logistics systems. Computers & Industrial Engineering, 164, 107881. https://doi.org/10.1016/j.cie.2021.107881.

    Article  Google Scholar 

  • Battini, D., Persona, A., & Sgarbossa, F. (2014). Innovative real-time system to integrate ergonomic evaluations into warehouse design and management. Computers & Industrial Engineering, 11, 77. https://doi.org/10.1016/j.cie.2014.08.018.

    Article  Google Scholar 

  • Bibi, S., Anjum, N., & Sher, M. (2018). Automated multi-feature human interaction recognition in complex environment. Computers in Industry, 99, 282–293. https://doi.org/10.1016/j.compind.2018.03.015.

    Article  Google Scholar 

  • Biecek, P., & Burzykowski, T. (2021). Explanatory model analysis. Chapman and Hall/CRC.

    Book  Google Scholar 

  • Bordoni, L., Petracci, I., Pelikant-Malecka, I., Radulska, A., Piangerelli, M., Samulak, J. J., et al. (2021). Mitochondrial DNA copy number and trimethylamine levels in the blood: New insights on cardiovascular disease biomarkers. The FASEB Journal, 35(7), e21694. https://doi.org/10.1096/fj.202100056R.

    Article  Google Scholar 

  • Chan, V. C. H., Ross, G. B., Clouthier, A. L., Fischer, S. L., & Graham, R. B. (2022). The role of machine learning in the primary prevention of work-related musculoskeletal disorders: A scoping review. Applied Ergonomics, 98, 103574. https://doi.org/10.1016/j.apergo.2021.103574.

    Article  Google Scholar 

  • Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357. https://doi.org/10.1613/jair.953.

    Article  Google Scholar 

  • Chen, C., Wang, T., Li, D., & Hong, J. (2020). Repetitive assembly action recognition based on object detection and pose estimation. Journal of Manufacturing Systems, 55, 325–333. https://doi.org/10.1016/j.jmsy.2020.04.018.

    Article  Google Scholar 

  • Clark, R. A., Mentiplay, B. F., Hough, E., & Pua, Y. H. (2019). Three-dimensional cameras and skeleton pose tracking for physical function assessment: A review of uses, validity, current developments and Kinect alternatives. Gait & Posture, 68, 193–200. https://doi.org/10.1016/j.gaitpost.2018.11.029.

    Article  Google Scholar 

  • Damle, R., Gurjar, A., Joshi, A., & Nagre, K. (2015). Human body skeleton detection and tracking. Human Body Skeleton Detection and Tracking, 3, 222–225.

    Google Scholar 

  • Diego-Mas, J. A., Poveda-Bautista, R., & Garzon-Leal, D. (2017). Using RGB-D sensors and evolutionary algorithms for the optimization of workstation layouts. Applied Ergonomics, 65, 530–540. https://doi.org/10.1016/j.apergo.2017.01.012.

    Article  Google Scholar 

  • European Agency for Safety and Health at Work. (2019). Work-related musculoskeletal disorders: Prevalence, costs and demographics in the EU. Publications Office of the European Union.

    Google Scholar 

  • Fernández, M. M., Álvaro Fernández, J., Bajo, J. M., & Delrieux, C. A. (2020). Ergonomic risk assessment based on computer vision and machine learning. Computers & Industrial Engineering, 149, 106816. https://doi.org/10.1016/j.cie.2020.106816.

    Article  Google Scholar 

  • Gamra, M. B., & Akhloufi, M. A. (2021). A review of deep learning techniques for 2D and 3D human pose estimation. Image and Vision Computing, 114, 104282. https://doi.org/10.1016/j.imavis.2021.104282.

    Article  Google Scholar 

  • Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2018). A survey of methods for explaining black box models. ACM Computing Surveys (CSUR), 51(5), 1–42. https://doi.org/10.1145/3236009.

    Article  Google Scholar 

  • Kim, W., Sung, J., Saakes, D., Huang, C., & Xiong, S. (2021). Ergonomic postural assessment using a new open-source human pose estimation technology (OpenPose). International Journal of Industrial Ergonomics, 84, 103164. https://doi.org/10.1016/j.ergon.2021.103164.

    Article  Google Scholar 

  • Korhan, O., & Memon, A. A. (2019). Introductory chapter: work-related musculoskeletal disorders. In Work-related musculoskeletal disorders. IntechOpen.

  • Li, C., Zhong, Q., Xie, D., & Pu, S. (2017) Skeleton-based action recognition with convolutional neural networks. In 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) (pp. 597–600). IEEE.

  • Li, L., Martin, T., & Xu, X. (2020). A novel vision-based real-time method for evaluating postural risk factors associated with musculoskeletal disorders. Applied Ergonomics, 87, 103138. https://doi.org/10.1016/j.apergo.2020.103138.

    Article  Google Scholar 

  • Lim, S., & D’Souza, C. (2020). A narrative review on contemporary and emerging uses of inertial sensing in occupational ergonomics. International Journal of Industrial Ergonomics, 76, 102937. https://doi.org/10.1016/j.ergon.2020.102937.

    Article  Google Scholar 

  • Liu, B., Cai, H., Ju, Z., & Liu, H. (2020). Multi-stage adaptive regression for online activity recognition. Pattern Recognition, 98, 107053. https://doi.org/10.1016/j.patcog.2019.107053.

    Article  Google Scholar 

  • Liu, J., Wang, Y., Liu, Y., Xiang, S., & Pan, C. (2020). 3D PostureNet: A unified framework for skeleton-based posture recognition. Pattern Recognition Letters, 140, 143–149. https://doi.org/10.1016/j.patrec.2020.09.029.

    Article  Google Scholar 

  • Lopez, M., Beurton-Aimar, M., Diallo, G., & Maabout, S. (2022). A simple yet effective approach for log based critical errors prediction. Computers in Industry, 137, 103605. https://doi.org/10.1016/j.compind.2021.103605.

    Article  Google Scholar 

  • Malchaire, J., Gauthy, R., Piette, A., & Strambi, F. (2011). A classification of methods for assessing and/or preventing the risks of musculoskeletal disorders. European Trade Union Institute: ETUI.

    Google Scholar 

  • Mancini, A., Vito, L., Marcelli, E., Piangerelli, M., De Leone, R., Pucciarelli, S., et al. (2020). Machine learning models predicting multidrug resistant urinary tract infections using DsaaS. BMC Bioinformatics, 21(10), 1–12. https://doi.org/10.1186/s12859-020-03566-7.

    Article  Google Scholar 

  • McAtamney, L., & Nigel, Corlett E. (1993). RULA: A survey method for the investigation of work-related upper limb disorders. Applied Ergonomics, 24(2), 91–99. https://doi.org/10.1016/0003-6870(93)90080-S.

    Article  Google Scholar 

  • Merino, G., da Silva, L., Mattos, D., Guimarães, B., & Merino, E. (2019). Ergonomic evaluation of the musculoskeletal risks in a banana harvesting activity through qualitative and quantitative measures, with emphasis on motion capture (Xsens) and EMG. International Journal of Industrial Ergonomics, 69, 80–89. https://doi.org/10.1016/j.ergon.2018.10.004.

    Article  Google Scholar 

  • Nayak, G. K., & Kim, E. (2021). Development of a fully automated RULA assessment system based on computer vision. International Journal of Industrial Ergonomics, 86, 103218. https://doi.org/10.1016/j.ergon.2021.103218.

    Article  Google Scholar 

  • Peppoloni, L., Filippeschi, A., Ruffaldi, E., & Avizzano, C. A. (2016). A novel wearable system for the online assessment of risk for biomechanical load in repetitive efforts. International Journal of Industrial Ergonomics, 52, 1–11. https://doi.org/10.1016/j.ergon.2015.07.002 (New Approaches and Interventions to Prevent Work Related Musculoskeletal Disorders).

    Article  Google Scholar 

  • Piñero-Fuentes, E., Canas-Moreno, S., Rios-Navarro, A., Domínguez-Morales, M., Sevillano, J. L., & Linares-Barranco, A. (2021). A deep-learning based posture detection system for preventing telework-related musculoskeletal disorders. Sensors. https://doi.org/10.3390/s21155236.

    Article  Google Scholar 

  • Regazzoni, D., Vecchi, G. D., & Rizzi, C. (2014). RGB cams vs RGB-D sensors: Low cost motion capture technologies performances and limitations. Journal of Manufacturing Systems, 33(4), 719–728. https://doi.org/10.1016/j.jmsy.2014.07.011.

    Article  Google Scholar 

  • Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) “Why should i trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135–1144).

  • Seo, J., & Lee, S. (2021). Automated postural ergonomic risk assessment using vision-based posture classification. Automation in Construction, 128, 103725. https://doi.org/10.1016/j.autcon.2021.103725.

    Article  Google Scholar 

  • Slembrouck, M., Luong, H. Q., Gerlo, J., Schütte, K., Cauwelaert, DV., Clercq, D.D., et al. (2020). Multiview 3D markerless human pose estimation from OpenPose skeletons. In Advanced Concepts for Intelligent Vision Systems (pp. 166–178).

  • Tu, H., Wang, C., & Zeng, W. (2020) End-to-end estimation of multi-person 3D poses from multiple cameras. CoRR. abs/2004.06239. https://doi.org/10.1007/978-3-030-58604-1_29. arXiv:2004.06239.

  • Vignais, N., Miezal, M., Bleser, G., Mura, K., Gorecky, D., & Marin, F. (2013). Innovative system for real-time ergonomic feedback in industrial manufacturing. Applied Ergonomics, 44(4), 566–574. https://doi.org/10.1016/j.apergo.2012.11.008.

    Article  Google Scholar 

  • Vlasic, D., Adelsberger, R., Vannucci, G., Barnwell, J., Gross, M., Matusik, W., et al. (2007). Practical motion capture in everyday surroundings. ACM Transactions on Graphics, 10(1145/1276377), 1276421.

    Google Scholar 

  • Xiao, B., Xiao, H., Wang, J., & Chen, Y. (2022). Vision-based method for tracking workers by integrating deep learning instance segmentation in off-site construction. Automation in Construction, 136, 104148. https://doi.org/10.1016/j.autcon.2022.104148.

    Article  Google Scholar 

  • Xu, H., Bazavan, EG., Zanfir, A., Freeman, B., Sukthankar, R., & Sminchisescu, C. (2020) GHUM & GHUML: Generative 3D human shape and articulated pose models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6184–6193).

  • Yadav, S. K., Tiwari, K., Pandey, H. M., & Akbar, S. A. (2021). A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions. Knowledge-Based Systems, 223, 106970. https://doi.org/10.1016/j.knosys.2021.106970.

    Article  Google Scholar 

  • Yoshikawa, Y., Shishido, H., Suita, M., Kameda, Y., & Kitahara, I. (2021) Shot detection using skeleton position in badminton videos. In International Workshop on Advanced Imaging Technology (IWAIT) 2021 (Vol. 11766, p. 117661K). International Society for Optics and Photonics.

  • Zhu, S., Fang, Z., Wang, Y., Yu, J., & Du, J. (2019). Multimodal activity recognition with local block CNN and attention-based spatial weighted CNN. Journal of Visual Communication and Image Representation, 60, 38–43. https://doi.org/10.1016/j.jvcir.2018.12.026.

    Article  Google Scholar 

Download references

Acknowledgements

This work is partly funded by the URRÁ project “Usability of Robots and Reconfigurbility of processes: enabling technologies and use cases”, on the topics of User-Centered Manufacturing and Industry 4.0, which is part of the project EU ERDF, POR MARCHE Region FESR 2014/2020-AXIS 1-Specific Objective 2-ACTION 2.1, “HD3Flab-Human Digital Flexible Factory of the Future Laboratory”, coordinated by the Polytechnic University of Marche.

Author information

Authors and Affiliations

  1. Computer Science Division, University of Camerino, via Madonna delle Carceri, 62032, Camerino, Italy

    Flavio Corradini, Leonardo Mostarda & Marco Piangerelli

  2. Department of Industrial Engineering and Mathematical Sciences, Poytechnic University of Marche, via Brecce Bianche, 60131, Ancona, Italy

    Marianna Ciccarelli, Michele Germani, Giacomo Menchi & Alessandra Papetti

Authors
  1. Marianna Ciccarelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Flavio Corradini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michele Germani
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Giacomo Menchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Leonardo Mostarda
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alessandra Papetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Marco Piangerelli
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MC: conceptulization of this study, Methodology, Data curation, Writing Paper. FC: funding acquisition, writing—rerview & editing. MG: funding acquisition, writing—rerview & editing. GM: data curation, software. LM: supervision writing—review & editing. AP: conceptualization of this study, methodology, investigation, writing paper. MP: conceptualization of this study, methodology, formal analysis, software, writing paper.

Corresponding author

Correspondence to Marco Piangerelli.

Ethics declarations

Conflict of interest

All authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ciccarelli, M., Corradini, F., Germani, M. et al. SPECTRE: a deep learning network for posture recognition in manufacturing. J Intell Manuf 34, 3469–3481 (2023). https://doi.org/10.1007/s10845-022-02014-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-022-02014-y

Keywords

Search

Navigation