Skip to main content

Deep multi-sensorial data analysis for production monitoring in hard metal industry


The industry practice of machining hard metal parts using CNC lathe turning machines is through grinding and milling procedures. The typical practice for quality control is through manual inspection, as automated solutions are difficult to integrate in production and do not reach the same level of accuracy. In this scope, the proposed system aims to automate the manufacturing process for the machine condition monitoring and 3D inspection of defective hard metal parts, by utilizing deep neural networks (DNNs) and investigating the defects on real production samples. Concretely, data are collected with (a) shop floor sensors, (b) high-resolution laser microprofilometer and (c) ultrasound scanner. The proposed system analyzes the collected data through AI models for quality control. Moreover, a fusion scheme is proposed to further improve accuracy. The system is validated on the classification of defective and non-defective samples, using metrics including accuracy, F-score, precision and recall for the performance evaluation.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10


  1. 1.

    Ansari F, Erol S, Sihn W (2018) Rethinking human-machine learning in industry 4.0: how does the paradigm shift treat the role of human learning? Procedia Manuf 23:117–122

    Google Scholar 

  2. 2.

    Alom MZ, Taha TM, Yakopcic C, Westberg S, Sidike P, Nasrin MS, Hasan M, Van Essen BC, Awwal AA, Asari VK (2019) A state-of-the-art survey on deep learning theory and architectures. Electronics 8(3):292

    Google Scholar 

  3. 3.

    Wang J, Ma Y, Zhang L, Gao RX, Wu D (2018) Deep learning for smart manufacturing methods and applications. J Manuf Syst 48:144–156

    Google Scholar 

  4. 4.

    Panesar A (2019) Machine learning and AI for healthcare. Springer, Berlin

    Google Scholar 

  5. 5.

    Yang B (2019) Machine learning-based evolution model and the simulation of a profit model of agricultural products logistics financing. Neural Comput & Applic 31(9):4733–4759

    Google Scholar 

  6. 6.

    Bhavsar P, Safro I, Bouaynaya N, Polikar R, Dera D (2017) Machine learning in transportation data analytics. In: Data analytics for intelligent transportation systems. Elsevier, pp 283–307

  7. 7.

    Baudet N, Maire JL, Pillet M (2013) The visual inspection of product surfaces. Food Qual Prefer 27(2):153–160

    Google Scholar 

  8. 8.

    Xi J, Shentu L, Hu J, Li M (2017) Automated surface inspection for steel products using computer vision approach. Appl Opt 56(2):184

    Google Scholar 

  9. 9.

    Zhou F, Liu G, Xu F, Deng H (2019) A generic automated surface defect detection based on a bilinear model. Appl Sci 9(15): 3159

    Google Scholar 

  10. 10.

    Lee WJ, Wu H, Yun H, Kim H, Jun MBG, Sutherland JW (2019) Predictive maintenance of machine tool systems using artificial intelligence techniques applied to machine condition data. Procedia CIRP 80:506–511

    Google Scholar 

  11. 11.

    Arnanz R, Mara A, de Santiago G, Reones A, Rodrguez J, Saludes S (2011) Monitoring and fault diagnosis in manufacturing processes in the automotive industry. In: New trends and developments in automotive industry. InTech

  12. 12.

    Singh J, Singh H, Singh B (2020) Introduction. In: Prioritization of failure modes in manufacturing processes. Emerald Publishing Limited, pp 01–32

  13. 13.

    Georgios Stavropoulos, Panagiotis Moschonas, Konstantinos Moustakas, Dimitrios Tzovaras, Michael Gerassimos Strintzis (2010) 3-d model search and retrieval from range images using salient features. IEEE Trans Multimed 12(7):692–704

    Google Scholar 

  14. 14.

    Newman TS, Jain AK (1995) A system for 3d CAD-based inspection using range images. Pattern Recogn 28(10):1555–1574

    Google Scholar 

  15. 15.

    Gopinath D, Katz G, Pasareanu CS, Barrett C (2017) Deepsafe: A data-driven approach for checking adversarial robustness in neural networks

  16. 16.

    Cao K, Liu M, Su H, Wu J, Zhu J, Liu S (2020) Analyzing the noise robustness of deep neural networks

  17. 17.

    He K, Xiangyu Z, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 770–778

  18. 18.

    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  19. 19.

    Dimitriou N, Leontaris L, Vafeiadis T, Ioannidis D, Wotherspoon T, Tinker G, Tzovaras D (2020) A deep learning framework for simulation and defect prediction applied in microelectronics. Simul Model Pract Theory 102063:100

    Google Scholar 

  20. 20.

    Dimitriou N, Leontaris L, Vafeiadis T, Ioannidis D, Wotherspoon T, Tinker G, Tzovaras D (2019) Fault diagnosis in microelectronics attachment via deep learning analysis of 3d laser scans. IEEE Trans Ind Electron 67:1–1

    Google Scholar 

  21. 21.

    Vafeiadis T, Dimitriou N, Ioannidis D, Wotherspoon T, Tinker G, Tzovaras D (2018) A framework for inspection of dies attachment on pcb utilizing machine learning techniques. J Manag Analyt 5 (2):81–94

    Google Scholar 

  22. 22.

    Massaro A, Manfredonia I, Galiano A, Xhahysa B (2019) Advanced process defect monitoring model and prediction improvement by artificial neural network in kitchen manufacturing industry: a case of study. In: 2019 II Workshop on metrology for industry 4.0 and IoT (MetroInd4.0 IoT), pp 64–67

  23. 23.

    Kuo C, Chen C, Chen T, Tsai Z, Hung M, Lin Y, Chen Y, Wang D, Homg G, Su W (2019) A labor-efficient gan-based model generation scheme for deep-learning defect inspection among dense beans in coffee industry. In: 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE). pp 263–270

  24. 24.

    Li L, Ota K, Dong M (2018) Deep learning for smart industry efficient manufacture inspection system with fog computing. IEEE Trans Ind Inform 14(10):4665–4673

    Google Scholar 

  25. 25.

    Cao P, Zhang S, Tang J (2018) Preprocessing-free gear fault diagnosis using small datasets with deep convolutional neural network-based transfer learning. IEEE Access 6:26241–26253

    Google Scholar 

  26. 26.

    Huang Z, Angadi VC, Danishvar M, Mousavi A, Li M (2018) Zero defect manufacturing of microsemiconductors – an application of machine learning and artificial intelligence. In: 2018 5th International Conference on Systems and Informatics (ICSAI). pp 449–454

  27. 27.

    Salamanis A, Kehagias DD, Filelis-Papadopoulos CK, Tzovaras D, Gravvanis GA (2016) Managing spatial graph dependencies in large volumes of traffic data for travel-time prediction. IEEE Trans Intell Transp Syst 17(6):1678–1687

    Google Scholar 

  28. 28.

    Chen K, Hu J, Zhang Y, Yu Z, He J (2018) Fault location in power distribution systems via deep graph convolutional networks

  29. 29.

    Staar B, Lütjen M, Freitag M (2018) Anomaly detection with convolutional neural networks for industrial surface inspection. 79, 07

  30. 30.

    Giakoumis D, Drosou A, Cipresso P, Tzovaras D, Hassapis G, Gaggioli A, Riva Giuseppe (2012) Using activity-related behavioural features towards more effective automatic stress detection. PLoS ONE 7(9):e43571

    Google Scholar 

  31. 31.

    Eikema DJA, Hatzitaki V, Tzovaras D, Papaxanthis C (2014) Application of intermittent galvanic vestibular stimulation reveals age-related constraints in the multisensory reweighting of posture. Neurosci Lett 561:112–117

    Google Scholar 

  32. 32.

    Gelenbe E, Görbil G, Tzovaras D, Liebergeld S, Garcia D, Baltatu M, Lyberopoulos G (2013) NEMESYS: enhanced network security for seamless service provisioning in the smart mobile ecosystem. In: Information sciences and systems 2013. Springer International Publishing, Cham, pp 369–378

  33. 33.

    Iqbal R, Maniak T, Doctor F, Karyotis C (2019) Fault detection and isolation in industrial processes using deep learning approaches. IEEE Trans Ind Inform 15(5):3077–3084

    Google Scholar 

  34. 34.

    Namuduri S, Narayanan BN, Davuluru VSP, Burton L, Bhansali S (2020) Review—deep learning methods for sensor based predictive maintenance and future perspectives for electrochemical sensors. J Electrochem Soc 167(3):037552

    Google Scholar 

  35. 35.

    Mohamudally Nawaz, Peermamode-Mohaboob Mahejabeen (2018) Building an anomaly detection engine (ADE) for IoT smart applications. Procedia Comput Sci 134:10–17

    Google Scholar 

  36. 36.

    Han Y, Fan J, Yang X (2019) A structured light vision sensor for on-line weld bead measurement and weld quality inspection. Int J Adv Manuf Technol 106:1–14

    Google Scholar 

  37. 37.

    Satorres Martínez S, Ortega Vázquez C, Gámez García J, Gómez Ortega J (2017) Quality inspection of machined metal parts using an image fusion technique. Measurement 111:374– 383

    Google Scholar 

  38. 38.

    Moru D, Borro D (2020) A machine vision algorithm for quality control inspection of gears. Int J Adv Manuf Tech 106:01

    Google Scholar 

  39. 39.

    Zhao R, Wang D, Yan R, Mao K, Shen F, Wang J (2017) Machine health monitoring using local feature-based gated recurrent unit networks. IEEE Trans Ind Electron 65(2):1539– 1548

    Google Scholar 

  40. 40.

    Xie Q, Li D, Xu J, Yu Z, Wang J (2019) Automatic detection and classification of sewer defects via hierarchical deep learning. IEEE Trans Autom Sci Eng 16(4):1836–1847

    Google Scholar 

  41. 41.

    Opitz D, Maclin R (1999) Popular ensemble methods: an empirical study. J Artif Intell Res 11:169–198

    MATH  Google Scholar 

  42. 42.

    Jing L, Wang T, Zhao M, Wang P (2017) An adaptive multi-sensor data fusion method based on deep convolutional neural networks for fault diagnosis of planetary gearbox. Sensors 17(2): 414

    Google Scholar 

  43. 43.

    Stavropoulos P, Papacharalampopoulos A, Vasiliadis E, Chryssolouris G (2016) Tool wear predictability estimation in milling based on multi-sensorial data. Int J Adv Manuf Tech 82(1-4):509–521

    Google Scholar 

  44. 44.

    Mithil KM, Thejas GS, Ramani SK, Iyengar SS (2017) A low cost multi sensorial data fusion for high speed obstacle avoidance using 3-d point clouds and image processing in self balancing robots. In: 2017 2nd International Conference on Emerging Computation and Information Technologies (ICECIT). pp 1–9

  45. 45.

    Aijazi AK, Checchin P, Trassoudaine L (2017) Multi sensorial data fusion for efficient detection and tracking of road obstacles for inter-distance and anti-collision safety management. In: 2017 3rd International Conference on Control, Automation and Robotics (ICCAR). pp 617–621

  46. 46.

    Chen F, Jahanshahi MR (2018) Nb-cnn: deep learning-based crack detection using convolutional neural network and naïve bayes data fusion. IEEE Trans Ind Electron 65(5):4392–4400

    Google Scholar 

  47. 47.

    Razavi-Far R, Hallaji E, Farajzadeh-Zanjani M, Saif M, Kia SH, Henao H, Capolino G (2019) Information fusion and semi-supervised deep learning scheme for diagnosing gear faults in induction machine systems. IEEE Trans Ind Electron 66(8):6331–6342

    Google Scholar 

  48. 48.

    Bustillo A, Urbikain G, Perez JM, Pereira OM, de Lacalle LNL (2018) Smart optimization of a friction-drilling process based on boosting ensembles. J Manuf Syst 48:108–121. Special Issue on Smart Manufacturing

    Google Scholar 

  49. 49.

    Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6):381–395

    MathSciNet  Google Scholar 

  50. 50.

    Rusu R, Cousins S (2011) 3d is here: point cloud library (pcl) 05

  51. 51.

    Kurban R, Skuka F, Bozpolat H (2015) Plane segmentation of kinect point clouds using ransac. In: The 7th International Conference on Information Technology. pp 545–551

  52. 52.

    Yang MY, Förstner W (2010) Plane detection in point cloud data. In: Proceedings of the 2nd Int Conf on Machine Control Guidance. Bonn, vol. 1, pp 95–104

  53. 53.

    Osawa K, Tsuji Y, Ueno Y, Naruse A, Yokota R, Matsuoka S (2019) Large-scale distributed second-order optimization using kronecker-factored approximate curvature for deep convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 12359–12367

  54. 54.

    Hsu DK (2006) Nondestructive testing using air-borne ultrasound. Ultrasonics 44:e1019–e1024

    Google Scholar 

  55. 55.

    Iyer S, Sinha SK, Pedrick MK, Tittmann BR (2012) Evaluation of ultrasonic inspection and imaging systems for concrete pipes. Autom Constr 22:149–164

    Google Scholar 

  56. 56.

    Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22 (10):1345–1359

    Google Scholar 

  57. 57.

    Napoletano P, Piccoli F, Raimondo S (2018) Anomaly detection in nanofibrous materials by CNN-based self-similarity. Sensors 18(1):209

    Google Scholar 

  58. 58.

    Choi J, Wang Z, Venkataramani S, Chuang PIJ, Vijayalakshmi S, Gopalakrishnan K (2018) Pact: parameterized clipping activation for quantized neural networks

  59. 59.

    Zhang Z, Sabuncu M (2018) Generalized cross entropy loss for training deep neural networks with noisy labels. In: Advances in neural information processing systems. pp 8778–8788

  60. 60.

    Teahan WJ (2000) Text classification and segmentation using minimum cross-entropy. In: Content-Based Multimedia Information Access-Volume 2. pp 943–961

  61. 61.

    Guo B, Nixon MS, Damarla TR (2008) Acoustic information fusion for ground vehicle classification. In: 2008 11th International Conference on Information Fusion. IEEE, pp 1–7

  62. 62.

    Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: an imperative style, high-performance deep learning library. In: Wallach H, Larochelle H, Beygelzimer A, Alché-Buc FD, Fox E, Garnett R (eds) Advances in neural information processing systems 32. Curran Associates, Inc., pp 8024–8035

  63. 63.

    Kingma DP, Adam JB (2014) A method for stochastic optimization. arXiv:1412.6980

  64. 64.

    Duchi J, Hazan E, Singer Y (2011) Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res 12(Jul):2121–2159

    MathSciNet  MATH  Google Scholar 

  65. 65.

    Tieleman T, Hinton G (2012) Lecture 6.5-Rmsprop: divide the gradient by a running average of its recent magnitude. COURSERA: Neural Netw Mach Learn 4(2):26–31

    Google Scholar 

  66. 66.

    Wolpert DH (1992) Stacked generalization. Neural Netw 5(2):241–259

    Google Scholar 

  67. 67.

    Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, et al. (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12(Oct):2825–2830

    MathSciNet  MATH  Google Scholar 

Download references


This work has been co-funded by the European Union and the General Secretariat of Research and Technology, Ministry of Development & Investments, under the project INVIVO/T2DGE-0951 of the Bilateral S&T Cooperation Program Greece - Germany 2017. This work has been partially supported by the European Commission through project RECLAIM funded by the European Union H2020 programme under Grant Agreement No. 869884. This work has been also partially supported by the European Commission through application experiment CloudEcho of project CloudiFacturing funded by the European Union H2020 programme under Grant Agreement No. 768892.

Author information



Corresponding author

Correspondence to Thanasis Kotsiopoulos.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kotsiopoulos, T., Leontaris, L., Dimitriou, N. et al. Deep multi-sensorial data analysis for production monitoring in hard metal industry. Int J Adv Manuf Technol 115, 823–836 (2021).

Download citation


  • Production monitoring
  • Deep multi-sensorial data analysis
  • Smart manufacturing
  • Deep learning
  • Hard metal industry