Skip to main content
SpringerLink
Log in

Review of in situ and real-time monitoring of metal additive manufacturing based on image processing

  • Critical Review
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Metal additive manufacturing (AM) has the advantages of novel part morphology, new material property, and short production cycle, attracting significant attention in automobile, aerospace, and other fields. Nevertheless, the repeatability and stability of part quality are difficult to guarantee in the current metal AM process, hindering the development of metal AM technology. Vital process variables such as molten pool characteristics, spatter characteristics, surface morphology, and radiation temperature are directly related to the quality of parts. Monitoring the dynamic changes vital variables is critical. Image processing techniques such as image transformation, recognition, segmentation, and enhancement are introduced to obtain manufacturing process information through processing and analyzing rich image features. The monitoring system’s processing results are usually collected in real time to help solve the repeatability and stability problems of the metal AM process. Presently, in situ and real-time monitoring methods are the most popular, and unfortunately, the literature lacks a comprehensive report. Thus, this paper thoroughly describes in situ and real-time process monitoring on the basis of image processing for metal AM. This paper reviews the in situ and real-time monitoring on the basis of traditional image processing to analyze monitoring objects, process classifications, and image processing carriers. Then, the advantages and disadvantages of in situ and real-time monitoring of metal AM based on artificial intelligence are discussed and compared. Finally, image processing algorithm generalization, quality, small samples, and image labeling problems are analyzed and discussed. A technical route for metal AM real-time feedback control is proposed by combining image processing with other technologies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Availability of data and material

Not applicable.

Code availability

Not applicable.

References

  1. Lu B, Li D, Tian X (2015) Development trends in additive manufacturing and 3D printing. Engineering-prc 1:85–89

    Google Scholar 

  2. Herzog D, Seyda V, Wycisk E, Emmelmann C (2016) Additive manufacturing of metals. Acta Mater 117:371–392

    Article  Google Scholar 

  3. Wang Y, Zheng P, Peng T, Yang H, Zou J (2020) Smart additive manufacturing: current artificial intelligence-enabled methods and future perspectives. Sci China Technol Sc 63:1600–1611

    Article  Google Scholar 

  4. Gardan J (2016) Additive manufacturing technologies: state of the art and trends. Int J Prod Res 54:3118–3132

    Article  Google Scholar 

  5. Liu L, Ding Q, Zhong Y, Zou J, Wu J, Chiu Y, Li J, Zhang Z, Yu Q, Shen Z (2018) Dislocation network in additive manufactured steel breaks strength–ductility trade-off. Mater Today 21:354–361

    Article  Google Scholar 

  6. Kruth JP, Mercelis P, Van Vaerenbergh J, Froyen L, Rombouts M (2005) Binding mechanisms in selective laser sintering and selective laser melting. Rapid Prototyping J 11:26–36

    Article  Google Scholar 

  7. Rafi HK, Karthik NV, Gong H, Starr TL, Stucker BE (2013) Microstructures and mechanical properties of Ti6Al4V parts fabricated by selective laser melting and electron beam melting. J Mater Eng Perform 22:3872–3883

    Article  Google Scholar 

  8. Zhan Q, Liang Y, Ding J, Williams S (2017) A wire deflection detection method based on image processing in wire + arc additive manufacturing. Int J Adv Manuf Tech 89:755–763

    Article  Google Scholar 

  9. Lia F, Park JZ, Keist JS, Joshi S, Martukanitz RP (2018) Thermal and microstructural analysis of laser-based directed energy deposition for Ti-6Al-4V and Inconel 625 deposits. Mater Sci Eng A 717:1–10

    Article  Google Scholar 

  10. Tapia G, Elwany A (2014) A review on process monitoring and control in metal-based additive manufacturing. J Manuf Sci E-T Asme 136:060801

    Article  Google Scholar 

  11. Zhang Y, Mi J, Li H, Shen S, Yang Y, Song C, Zhou X (2022) In-situ monitoring plasma arc additive manufacturing process with fully convolutional network. Int J Adv Manuf Tech 120:2247–2257

    Article  Google Scholar 

  12. Angelo LD, Stefano PD, Guardiani E (2020) Search for the optimal build direction in additive manufacturing technologies: a review. J Manuf Mater Process 4:71

    Google Scholar 

  13. Tino R, Leary M, Yeo A, Kyriakou E, Kron T, Brandt M (2020) Additive manufacturing in radiation oncology: A review of clinical practice, emerging trends and research opportunities. Int J Extrem Manuf 2:012003

    Article  Google Scholar 

  14. Varela J, Merino J, Pickett C, Abu-Issa A, Arrieta E, Murr L, Wicker R, Ahlfors M, Godfrey D, Medina F (2020) Performance characterization of laser powder bed fusion fabricated inconel 718 treated with experimental hot isostatic processing cycles. J Manuf Mater Process 4:73

    Google Scholar 

  15. Leung CLA, Marussi S, Towrie M, Atwood RC, Withers PJ, Lee PD (2019) The effect of powder oxidation on defect formation in laser additive manufacturing. Acta Mater 166:294–305

    Article  Google Scholar 

  16. Cerniglia D, Montinaro N (2018) Defect detection in additively manufactured components: laser ultrasound and laser thermography comparison. Procedia Struct Integr 8:154–162

    Article  Google Scholar 

  17. Hossain MS, Taheri H (2020) In situ process monitoring for additive manufacturing through acoustic techniques. J Mater Eng Perform 29:6249–6262

    Article  Google Scholar 

  18. Leung CLA, Marussi S, Atwood RC, Towrie M, Withers PJ, Lee PD (2018) In situ X-ray imaging of defect and molten pool dynamics in laser additive manufacturing. Nat Commun 9:1–9

    Article  Google Scholar 

  19. Koester LW, Taheri H, Bigelow TA, Bond LJ, Faierson EJ (2018) In-situ acoustic signature monitoring in additive manufacturing processes. In AIP Conference Proceedings 1949:020006

    Article  Google Scholar 

  20. Donadello S, Motta M, Demir AG, Previtali B (2019) Monitoring of laser metal deposition height by means of coaxial laser triangulation. Opt Lasers Eng 112:136–144

    Article  Google Scholar 

  21. Li J, Duan Q, Hou J, Xie H, Liu S (2020) In-situ monitoring of substrate deformation in directed energy deposition process using the coherent gradient sensing method. Addit Manuf 36:101547

    Google Scholar 

  22. Min Y, Shen S, Li H, Sheng L, Mi J, Zhou J, Mai H, Chen J (2022) Online monitoring of an additive manufacturing environment using a time-of-flight mass spectrometer. Measurement 189:110473

    Article  Google Scholar 

  23. Huang TS, Schreiber WF, Tretiak OJ (1971) Image processing. In Proceedings of the IEEE 59:1586–1609

    Article  Google Scholar 

  24. Sarid O, Huss E (2011) Image formation and image transformation. Art Psychother 38:252–255

    Article  Google Scholar 

  25. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 770–778.

  26. Cheng HD, Jiang XH, Sun Y, Wang J (2001) Color image segmentation: advances and prospects. Pattern Recogn 34:2259–2281

    Article  MATH  Google Scholar 

  27. Wang DCC, Vagnucci AH, Li CC (1983) Digital image enhancement: a survey. Comput Vision, Graph Image Process 24:363–381

    Article  Google Scholar 

  28. Everton SK, Hirsch M, Stavroulakis PI, Leach RK, Clare AT (2016) Review of in-situ process monitoring and in-situ metrology for metal additive manufacturing. Mater Des 95:431–445

    Article  Google Scholar 

  29. Cunha FG, Santos TG, Xavier J (2021) In situ monitoring of additive manufacturing using digital image correlation: a review. Mater 14:1511

    Article  Google Scholar 

  30. Hofman JT, Pathiraj B, Van Dijk V, De Lange DF, Meijer J (2012) A camera based feedback control strategy for the laser cladding process. J Mater Process Technol 212:2455–2462

    Article  Google Scholar 

  31. Moralejo S, Penaranda X, Nieto S, Barrios A, Arrizubieta I, Tabernero I, Figueras J (2017) A feedforward controller for tuning laser cladding melt pool geometry in real time. Int J Adv Manuf Tech 89:821–831

    Article  Google Scholar 

  32. Song L, Wang F, Li S, Han X (2017) Phase congruency melt pool edge extraction for laser additive manufacturing. J Mater Process Technol 250:261–269

    Article  Google Scholar 

  33. Yang Q, Yuan Z, Zhi X, Yan Z, Tian H, Chen X (2020) Real-time width control of molten pool in laser engineered net shaping based on dual-color image. Opt Laser Technol 123:105925

    Article  Google Scholar 

  34. Seltzer D, Schiano J L, Nassar A R, Reutzel E W (2016) Illumination and image processing for real-time control of directed energy deposition additive manufacturing. In Proceedings of the 26th Annual International Solid Freeform Fabrication Symposium – An Additive Manufacturing Conference 1479–1486.

  35. Craeghs T, Clijsters S, Yasa E, Kruth J P (2011) Online quality control of selective laser melting. In Proceedings of the 20th Solid Freeform Fabrication (SFF) Symposium 212–226.

  36. Clijsters S, Craeghs T, Buls S, Kempen K, Kruth JP (2014) In situ quality control of the selective laser melting process using a high-speed, real-time melt pool monitoring system. Int J Adv Manuf Tech 75:1089–1101

    Article  Google Scholar 

  37. Zhang Y, Fuh JYH, Ye D, Hong GS (2019) In-situ monitoring of laser-based PBF via off-axis vision and image processing approaches. Addit Manuf 25:263–274

    Google Scholar 

  38. Asselin M, Toyserkani E, Iravani-Tabrizipour M, Khajepour A (2005) Development of trinocular CCD-based optical detector for real-time monitoring of laser cladding. In IEEE International Conference Mechatronics and Automation 3:1190–1196

    Google Scholar 

  39. Biegler M, Graf B, Rethmeier M (2018) In-situ distortions in LMD additive manufacturing walls can be measured with digital image correlation and predicted using numerical simulations. Addit Manuf 20:101–110

    Google Scholar 

  40. Yao B, Imani F, Sakpal AS, Reutzel EW, Yang H (2018) Multifractal analysis of image profiles for the characterization and detection of defects in additive manufacturing. J Manuf Sci E-T Asme 140:031014

    Article  Google Scholar 

  41. Barua S, Liou F, Newkirk J, Sparks T (2014) Vision-based defect detection in laser metal deposition process. Rapid Prototyp J 20:77–86

    Article  Google Scholar 

  42. Khanzadeh M, Tian W, Yadollahi A, Doude HR, Tschopp MA, Bian L (2018) Dual process monitoring of metal-based additive manufacturing using tensor decomposition of thermal image streams. Addit Manuf 23:443–456

    Google Scholar 

  43. Khanzadeh M, Chowdhury S, Marufuzzaman M, Tschopp MA, Bian L (2018) Porosity prediction: Supervised-learning of thermal history for direct laser deposition. J Manuf Syst 47:69–82

    Article  Google Scholar 

  44. Zhang B, Liu S, Shin YC (2019) In-process monitoring of porosity during laser additive manufacturing process. Addit Manuf 28:497–505

    Google Scholar 

  45. Mi J, Zhang Y, Li H, Shen S, Yang Y, Song C, Zhou X, Duan Y, Lu J, Mai H (2021) In-situ monitoring laser based directed energy deposition process with deep convolutional neural network. J Intell Manuf. https://doi.org/10.1007/s10845-021-01820-0

    Article  Google Scholar 

  46. Yang Z, Lu Y, Yeung H, Krishnamurty S (2019) Investigation of deep learning for real-time melt pool classification in additive manufacturing. In 2019 IEEE 15th International Conference on Automation Science and Engineering (case) 640–647.

  47. Fang Q, Tan Z, Li H, Song C, Zhou X, Yang Y, Shen S (2021) In-situ capture of melt pool signature in selective laser melting using U-Net-based convolutional neural network. J Manuf Process 648:347–355

    Article  Google Scholar 

  48. Tan Z, Fang Q, Li H, Liu S, Zhu W, Yang D (2020) Neural network based image segmentation for spatter extraction during laser-based powder bed fusion processing. Opt Laser Technol 130:106347

    Article  Google Scholar 

  49. Yang D, Li H, Liu S, Song C, Yang Y, Shen S, Lu J, Liu Z, Zhu Y (2020) In situ capture of spatter signature of SLM process using maximum entropy double threshold image processing method based on genetic algorithm. Opt Laser Technol 131:106371

    Article  Google Scholar 

  50. Wang Y, Zhang C, Lu J, Bai L, Zhao Z, Han J (2020) Weld reinforcement analysis based on long-term prediction of molten pool image in additive manufacturing. IEEE Access 8:69908–69918

    Article  Google Scholar 

  51. Lu J, He H, Shi Y, Bai L, Zhao Z, Han J (2021) Quantitative prediction for weld reinforcement in arc welding additive manufacturing based on molten pool image and deep residual network. Addit Manuf 41:101980

    Google Scholar 

  52. Caggiano A, Zhang J, Alfieri V, Caiazzo F, Gao R, Teti R (2019) Machine learning-based image processing for on-line defect recognition in additive manufacturing. Cirp Ann-Manuf Techn 68:451–454

    Article  Google Scholar 

  53. Scime L, Siddel D, Baird S, Paquit V (2020) Layer-wise anomaly detection and classification for powder bed additive manufacturing processes: a machine-agnostic algorithm for real-time pixel-wise semantic segmentation. Addit Manuf 36:101453

    Google Scholar 

  54. Colosimo BM, Grasso M (2018) Spatially weighted PCA for monitoring video image data with application to additive manufacturing. J Qual Technol 50:391–417

    Article  Google Scholar 

  55. Li X, Siahpour S, Lee J, Wang Y, Shi J (2020) Deep learning-based intelligent process monitoring of directed energy deposition in additive manufacturing with thermal images. Procedia Manuf 48:643–649

    Article  Google Scholar 

  56. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE T Pattern Anal 38:295–307

    Article  Google Scholar 

  57. Kitamura G, Chung CY, Moore BE (2019) Ankle fracture detection utilizing a convolutional neural network ensemble implemented with a small sample, de novo training, and multiview incorporation. J Digit Imaging 32:672–677

    Article  Google Scholar 

  58. Gao Y, Gao L, Li X (2019) A hierarchical feature fusion-based method for defect recognition with a small sample. In IEEE International Conference on Industrial Engineering and Engineering Management 1048–1052

  59. Li X, Yu L, Chen H, Fu C W, Heng P A (2018) Semi-supervised skin lesion segmentation via transformation consistent self-ensembling model. arXiv preprint arXiv:1412.7062

  60. Luo X, Chen J, Song T, Wang G (2020) Semi-supervised medical image segmentation through dual-task consistency. arXiv preprint arXiv:2009.04448

Download references

Funding

This work was supported by the Key Research and Development Program of Sichuan Province, China (2020YFSY0054).

Author information

Authors and Affiliations

  1. The Institute of Technological Sciences, Wuhan University, Wuhan, 430072, China

    Yikai Zhang, Hui Li & Yaowu Hu

  2. School of Power and Mechanical Engineering, Wuhan University, Wuhan, 430072, China

    Shengnan Shen, Hui Li & Yaowu Hu

  3. Key Laboratory of Hydraulic Machinery Transients, Ministry of Education, Wuhan University, Wuhan, 430072, China

    Shengnan Shen & Hui Li

Authors
  1. Yikai Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shengnan Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yaowu Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yikai Zhang: conceptualization, methodology, formal analysis, data curation, visualization. Shengnan Shen: writing—review and editing, supervision. Hui Li: writing—review and editing. Yaowu Hu: writing—review and editing.

Corresponding authors

Correspondence to Shengnan Shen or Hui Li.

Ethics declarations

Ethics approval

The authors confirm that the submitted work is original and has not been published elsewhere.

Consent to participate

All authors have agreed to participate in this research.

Consent for publication

All authors have informed and agreed to publish this research.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Shen, S., Li, H. et al. Review of in situ and real-time monitoring of metal additive manufacturing based on image processing. Int J Adv Manuf Technol 123, 1–20 (2022). https://doi.org/10.1007/s00170-022-10178-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-022-10178-3

Keywords

Search

Navigation