1 Introduction

Conventional machining processes such as turning, milling, drilling and grinding are among the most commonly used subtractive manufacturing processes in the industry [1]. The main task of subtractive manufacturing is to ensure the specified accuracy and roughness of machined surfaces are met using tools with sharp cutting edges [2]. However, the quality of the tool degrades with usage which makes its cutting edges dull and increases the contact area with the workpiece. This, in return, leads to higher friction, an increase in cutting forces and elastic deflections in the tool due to workpiece vibrations [3]. The rise in tool wear during the machining process is also a consequence of the loss of surface quality [4]. There was a decrease in dimensional accuracy, as well as sudden tool failure and breakage.

Before going through the comprehensive review on tool condition monitoring (TCM) systems, it is paramount to understand the various states of tool wear that significantly impact the machining processes. Tool wear is a complex phenomenon which occurs in several forms, each affecting the quality and efficiency of a machining process differently [5]. Flank wear is a common form of wear that occurs on the flank face of the tool, and it can lead to dimensional inaccuracies and poor surface finish due to the increased friction between the tool and the workpiece [6]. Crater wear occurs on the rake face of the tool and is often caused by the high temperature and pressure generated during the cutting process. It can lead to a decrease in the tool’s cutting efficiency and rise in cutting forces [7]. Edge chipping and notching are the results of mechanical and thermal loads, leading to the loss of material along the cutting edges, which directly affects the dimensional accuracy and surface integrity of the machined components [8]. Built-up edge (BUE) is characterized by the accumulation of workpiece material on the tool’s cutting edge. BUE is more common when machining metals at high cutting speeds when there is a chemical affinity between the elements that make up the cutting too material or its coating and the workpiece material. It impacts the quality of the machined surfaces and can also cause deviations in dimensional accuracy [9].

Tool wear has a significant impact on machining quality, as worn tools yield poor surface finish and induce vibrations, leading to unstable machining conditions. Efficient management of tool wear is crucial to maintain integrity in the machining processes and ensure the production of high-quality components [10]. The different forms of tool wear also influence energy consumption, production rate and overall operational costs requiring effective tool wear management to avoid increased power consumption and slowed production rates as a result of frequent tool changes or reduced cutting speeds [11, 12]. In relation to these phenomena, it is critical to assess the level of tool wear in order to replace the cutting tool before it reaches catastrophic values [13]. To manage changes in tool wear in conventional machining processes, techniques from TCM are used. Traditionally, the systematization of tool condition monitoring techniques into direct measurement methods and indirect measurement methods was developed in the past few decades. Measurement systems based on indirect methods cover a variety of sensors for measuring cutting forces [14], monitoring vibrations during cutting [15] or acoustic emission [16], or for detecting off-current and power [17], systems with thermal sensors [18] and others. A summary of the direct and indirect tool condition monitoring techniques was previously summarised in the open literature [15].

Along with these indirect methods, which are widely used and well described in review articles, the non-contact direct method based on sensorial systems using image processing in machining is acquiring importance on a constant basis [19]. For example, the overview article for TCM in machining operations deals with artificial intelligence (AI) techniques, where, among other things, the applications of image sensors in machining are also briefly considered [20]. However, up to date, there are no comprehensive review articles available in the open literature on the current trends and development prospects of imaging sensors for tool condition monitoring of regions of the cutting edge or the state of machined surface in the conventional machining domain. Some of the most common direct and indirect imaging techniques that are used in TCM are summarised in Fig. 1, some of which will be discussed in this review.

Fig. 1
figure 1

Direct and indirect tool condition monitoring using image processing

Full size image

In connection with the justifications above, in this review article, the goal is to provide a comprehensive analysis and research in conventional machining processes with the use of image processing techniques for tool condition monitoring to highlight the predominance, limitations and perspectives of their application. This review includes sections with the following content. The systematization of methods of direct control and indirect measurement in conventional machining processes in TCM with image processing is described in detail in the second section. Specific examples of the application of TCM on machining with image processing techniques as well as discussing the predominance, limitation and perspectives are comprehensively explored in the third section. The current problems and future trends the industry is expected to face in using typical TCM meters with image processing systems are shown in the fourth section. The perspectives of such sensor systems with image processing for Industry 4.0 are also shown, namely artificial intelligence, smart manufacturing, internet of things and big data.

2 Tool condition monitoring in conventional machining via image processing using direct and indirect measurement techniques

2.1 Tool condition monitoring via image processing: direct measurement techniques

Image processing based on a direct sensor system is a valuable technique that can be used in conventional machining to monitor tool wear using a microscopic imaging system. This system monitors the worn area of the cutting tool by temporarily stopping the machining process, as illustrated in Fig. 2. To obtain tool wear images, each processing parameter is applied, and the microscope is positioned on the milling machine from the same point as the pin to ensure short re-installation time and equal imaging distances. The tool wear value is then automatically determined by inputting the microscope images into the tool wear control system [21]. The use of the tool shape descriptor (TSD) helps to identify the critical area of tool wear, which allows for the definition of the basic features of the cutting edge and the region of interest (ROI). By following the steps in Fig. 2, the positions of the worn areas can be verified using TSD, and the maximum and minimum widths of the flank wear can be accurately computed. Overall, direct sensor system–based image processing offers a reliable and efficient method for monitoring tool wear in conventional machining [21].

Fig. 2
figure 2

Extraction of the tool wear regions of interest (ROIs) using TSD [21]

Full size image

Dai and Zhu [22] studied drill wear recognition and proposed a mechanism for real-time identification and ROI determination. The developed system consisted of three parts, namely motor driving, image grabbing and RMSD counting, which all operated simultaneously. By combining these three parts, the system was able to detect the wear regions of the drill in real time. The ROI was determined through a threshold process using 200 values, which helped extract the wear regions more accurately. This method proved to be highly effective in detecting drill wear and minimizing tool damage. The results of the study showed that the proposed mechanism was able to detect wear regions accurately, which can help prevent the early breakdown of the drill and prolong its lifespan. This technique has great potential for use in the manufacturing industry, where the tool wear can cause significant production losses and increased costs. By using real-time drill wear recognition systems, manufacturers can minimize tool damage and improve production efficiency. Figure 3a and b depicts the outcomes of the edge-preserving filter and mean filter, respectively. As shown in Fig. 3a, the worn areas that were inadequately extracted through binarization and mean filtering were obtained by aligning the actual and processed images [22]. A telecentric micro lens with × 200 magnification and an Olympus charge-coupled device (CCD) was utilized to capture the tool images. The raw images were processed using software written in C +  + . The processed images, working principle and actual system are presented in Fig. 3c.

Fig. 3
figure 3

System for drill wear recognition. a, b Outcome of the image conversion into binary format. c Experimental set-up [22]

Full size image

Figure 4 presents a series of images that were obtained during the area of interest delimitation process in the V2 view using morphological operations. In Fig. 4a, images of the worn area of the tool are displayed. The image processing steps are depicted in Fig. 4b–d. To begin with, the raw image is transformed into a binary image, where black is represented by pixel 0 and white by pixel 1 using the Otsu method. Following this, morphological processes are applied to fill the holes, resulting in the largest white region being determined in Fig. 4b. The contour points are then defined as 20% closer to the centre, moving from the bottom to the top, as shown in Fig. 4c. The worn area is then defined by the dominant values within the contour points. After all non-dominant pixels have been removed, the largest region is retained, and the worn part is determined as shown in Fig. 4d, thanks to a morphological hole-filling process [23]. The method provides a reliable means of determining the area of wear in a tool, making it an essential tool for monitoring tool wear in machining operations.

Fig. 4
figure 4

Morphological operations for delimiting the area of interest in V2 view. a–d Examples of obtained images [23]

Full size image

Section conclusions

Direct sensor system–based image processing is a powerful technique that provides real-time acquisition and processing of images by synchronizing the camera with the cutting tool. This allows for direct measurement of the tool wear from the processed screen images without the need for additional equipment. Nevertheless, the machining process can lead to image obstructions due to chip formations, which can interfere with image processing. While coolants are necessary to prevent rapid tool wear, they may also cause blurring in the imaging process. In contrast, scanning electron microscopy (SEM) images taken from the cutting tools after turning and milling operations can accurately determine the wear region by applying an algorithm. However, this method requires a temporary suspension of the machining process for SEM imaging, which is both time-consuming and costly. Moreover, it is not well suited for continuous monitoring of tool wear during the machining process. Despite these limitations, direct sensor system–based image processing is still a valuable technique for measuring tool wear in specific applications. It offers advantages such as real-time measurement, direct imaging and high accuracy, making it a valuable tool for quality control in manufacturing industries.

2.2 Tool condition monitoring via image processing: indirect measurement techniques

Indirect TCM refers to the process of monitoring changes in other machining process characteristics (during or after) and correlate the findings with tool wear. This includes monitoring and collecting the data of the cutting tool temperature, machining vibrations, cutting forces and post machining surface finish among others. A variety of sensorial systems are used for the monitoring and data collection, some of which use imaging techniques as shown previously in Fig. 1. Some TCM uses a combination of imaging and non-imaging techniques which collect force and vibration data during machining to correlate that with tool wear. Tool wear monitoring is a crucial aspect of machining operations, and deep learning techniques have been explored to monitor tool wear during complex part milling. This approach was demonstrated by Zhang et al. [24] where the parameters for the method were established using wavelet packet decomposition (WPD) and a cutting force model. To implement the method, the process was divided into several steps. In the first step, cutting forces, vibration and tool status parameters were fed into a deep autoencoder to perform dimension reduction. This helped to reduce the amount of data needed for the process and optimize the use of computational resources. Next, a deep multi-layer perceptron was installed to respond to sensors in the actual machining environment and predict the tool wear status. At this level, only the cutting force was used as the signal for tool wear monitoring, which was the third step of the process, as shown in Fig. 5. The deep autoencoder was then used to transmit the manually set parameters of the cutting force signal to the deep multi-layer perceptron for estimating the tool wear value. Overall, the framework of this tool wear monitoring method demonstrated the potential of deep learning techniques to improve tool wear monitoring during complex part milling [16].

Fig. 5
figure 5

Deep learning–based framework for monitoring tool wear in complex part milling [24]

Full size image

He et al. [25] conducted a study on tool wear measurement where a chassis was integrated into a personal computer (PC) system equipped with LabVIEW software for online data acquisition and recording of temperature parameters. The width of the flank wear was extracted using a CCD imaging device and a bi-telecentric lens during the machining process interruption. The captured image for a microscope with a CCD camera was used to calibrate the wear values. Figure 6a presents the configuration of the measurement system for the dry cutting process. The temperature time-domain signal was used as the actual input for the stacked sparse autoencoders (SSAEs), while the upmost hidden layer output was used as the input for the back propagation neural network (BPNN) to predict tool wear. The neural network model for tool wear prediction is demonstrated in Fig. 6b. This model is a full neural network, which comprehends the tool wear prediction.

Fig. 6
figure 6

A CCD camera–based system for tool wear analysis. a Experimental set-up and measurement system. b The architecture of the proposed model [25]

Full size image

In the field of micro-machining, Akkoyun et al. [26] conducted a study that aimed to automate the measurement process of slots and burrs by developing a software-based technique to calculate the borders and lengths of slots from images of machined workpieces. Written in C +  + , the software utilized the Open Source Computer Vision Library (OpenCV) to process SEM images. The algorithm analysed peak points and user-defined parameters to determine burr parameters and slot dimensions. By taking an SEM image as input, the software produced slot and burr parameters as outputs. To determine the pixel size, image processing techniques such as HSV conversion, thresholding and metric scale reference were used. Vertical and horizontal peaks were then calculated and charted to obtain the values. The software’s block diagram is presented in Fig. 7. The automation of slot and burr measurement can potentially reduce the time and cost of machining processes, as well as improve accuracy and consistency in the measurements.

Fig. 7
figure 7

An illustration of the block diagram for automated burr measurement and slot detection software, a study on using SEM images and image processing techniques for machined workpieces [26]

Full size image

In another study on chip properties analysis by Pagani et al. [27], the images were obtained from a digital camera (Nikon P7000 with a CCD sensor) (Fig. 8). Light bulbs were placed using the cross-lightning technique to minimize shadowing. The angle of light sources was 30° and 45° for the first and second levels, respectively. The exposure time is adjusted to obtain chip colours in detail. In the experiment, between 20 and 40 chips were laid on the imaging area (25–35 cm2) with white background. The chip shape and colour properties are affected by the deviations of optimal cutting conditions which are related to tool wear.

Fig. 8
figure 8

Camera-based system for chip analysis. a Image acquisition set-up. b Cause and effect diagram. Investigating the interplay between image-based classification and chip properties [27]

Full size image

Section conclusions

Indirect sensor system–based image processing has emerged as a highly effective method of estimating tool wear. By measuring changes in various parameters such as chip geometry, slot and burr widths, tool temperature, workpiece surface roughness and cutting forces, the image processing technique offers a comprehensive approach to identifying the extent of tool wear. Out of all the available methods, image processing applied over the cutting force graph is the most reliable and efficient way to measure tool wear accurately and quickly. This method not only helps in detecting abnormal changes in the cutting force graph but also protects the tool and workpiece while saving time by stopping the machining process. Despite being a slower process, acquiring data for image processing based on chip geometry and workpiece surface quality is more predictive, enabling a more reliable estimation of tool wear. Thus, by offering a reliable, efficient and predictive approach, image processing plays a vital role in preventing catastrophic failure of the cutting tool and ensuring the smooth functioning of the machining process.

2.3 Image recognition and classification systems in image processing

2.3.1 A neural network and BPNN in image processing

Neural networks are composed of interconnected processing units that communicate through one-way signal channels called connections. They can perform localized information processing due to their local memory. Additionally, they have a parallel and distributed computing structure. Each processing element can be subdivided into multiple side connections and branches as needed, with a single output connection. The output signal of each processing element is of a specified mathematical type. The processes within each processing element are localized and depend on the values stored in the local memory of the processing element and the current input signals [28, 29]. The BPNN is a design consisting of rows of processing units and fully interconnected layers. Each unit consists of several separate transaction items [28]. The BPNN method is straightforward, quick and simple to program, with no parameters to be set apart from the input numbers. It is a method in which the desired functions can be learned without specifically specifying the function features. However, the performance of the BPNN may be influenced by the input data. The algorithm of this approach is very susceptible to noisy data, and thus, a matrix-based approach should be employed instead of a mini-batch.

2.3.2 Applying Image Tool software in image processing

This software enables direct measurement of parameters such as distances, tilt angles, areas and depth on a screenshot. Measurements can be made in pixels or standard units of length using a conversion factor that is specified. A magnification of × 70 is sufficient to distinguish the desired features in a captured image [30]. The program’s parameters can be adjusted to the desired extent in the tool image software. The corresponding tool wear is then carried out, with 1 pixel equal to 0.001 mm [30]. A macro is written in the tool image script language to capture the parameters of the tool wear on an image automatically. This macro can also perform the necessary statistical calculations in the experiments that are measured [30]. This application facilitates image selection and interpretation, allowing for the real-time observation of the dimensions of each image and the pixel value of a selected area. However, it is a manual process and, therefore, slow when compared to other methods.

2.3.3 Image thresholding methods in image processing

The image thresholding technique is commonly used in image segmentation to separate objects from the background and objects with different grey levels. Local thresholding and global thresholding are two categories of this technique [31, 32]. Local thresholding determines a threshold value for every pixel based on specific details within the image. In contrast, global thresholding employs one single threshold value to differentiate the image’s foreground and background [32]. Some of the renowned thresholding methods, such as the Otsu method and K-means, are explored, along with other prevalent methods in the field of image recognition and classification.

Otsu method

The Otsu method is an effective image thresholding technique aiming to identify the optimal global threshold. This method minimizes the variance within the class and is efficient for applications requiring computation of a grey-level histogram before application [33, 34].

K-means method

The K-means method, on the other hand, is efficient for finding the local optimal threshold, particularly beneficial for large datasets and globular clusters due to its efficiency, speed and flexibility in computational cost. However, it does have limitations, such as expressing the uniform effect that results in clusters of the same size when input data are of different sizes, and the challenge of estimating the value of K [33, 34].

Random forest

Random forest is a versatile and widely adopted ensemble learning method. It constructs a multitude of decision trees at training time and outputs the mode of the classes (classification) of the individual trees for input data. It is eminent for its high accuracy, its ability to handle large datasets and its ability to handle thousands of input variables without variable deletion [20].

Decision tree

The decision tree method is a flowchart-like tree structure where an internal node represents a feature (or attribute), the branch represents a decision rule, and each leaf node represents the outcome. It is simple to understand and visualize, requires little data preparation and can handle both numerical and categorical data [35, 36].

Bayesian classifier

The Bayesian classifier is based on Bayes’ theorem, which provides a way of updating probability estimates. It is highly scalable and can manage high-dimensional data with efficacy. This method is mainly used when the dimensionality of the inputs is high, and it is particularly powerful for text classification problems [37]. While the Otsu and K-means methods are substantial for image thresholding, incorporating a diverse array of methods such as random forest, decision tree and Bayesian classifier enriches the comprehensiveness and adaptability of image recognition and classification processes. These methods, with their distinctive advantages and limitations, offer varied approaches to addressing challenges in image processing applications.

2.3.4 Tool shape descriptor

The shape identification technique is a morphometric tool used for the contour analysis of near-convex images [38]. This technique enables the prediction and classification of image shapes, appearance and components using shape descriptors. Two types of shape descriptors are available: local and global shape descriptors. The local shape descriptor requires connecting edges, which help distinguish one shape from another by resolving spatial ambiguity. This identifier can be employed to categorize shapes based on one or multiple structuring elements and to establish the characteristics of the shape, along with the delineation and partitioning of the image entity. Configuration items in the image object can be detected using certain filters, such as on–off filters. The spatial spread of one or more regional shapes, sometimes across the image object, allows the identification of multi-lateral and multi-regional shapes by the tool shape descriptor [38, 39]. When comparing region-based and contour-based shape descriptors, the region-based approach uses the area covered within the shape boundary, while the contour-based approach uses only the outer boundary of the shape.

3 Application for tool condition monitoring in machining using image processing techniques

3.1 Application of TCM of the tool cutting edge

A stable chip formation is achieved by a combination of adequate cutting parameters and tool geometry. Although the cutting parameters can be kept constant during the machining process, the tool geometry, especially at the cutting edge, is always transient since it is a tribological principle that surfaces in contact and in relative motion will always undergo wear [40]. Based on that, monitoring tool wear becomes one of the most critical steps in increasing the overall efficiency of a machining system. Despite not being a completely new idea [41], the use of image processing is regarded as one of the most promising methods of tool wear monitoring [42], as it allows direct information on the wear size, morphology and mode [43], and applies no external influence at the machining interface [44]. In general, the image acquisition of the tool wear is performed using a CCD camera [45] or complementary metal oxide semiconductor (CMOS) [46]. CCD cameras generally allow a better resolution, and CMOS cameras have a higher image capture rate [44]. The lighting system is one of the most critical aspects of the image capturing process since improper illumination can lead to insufficient or even misleading (due to shadows) information about the condition of the tool [47].

3.1.1 Application of TCM of cutting edge in conventional machining using image processing techniques

Using automatic image processing in turning process is already a well-established technique, mainly due to the stationary position of the tool in relation to the machining centre and the more stable wear patterns compared to processes such as milling and drilling. Mikołajczyk et al. [48] evaluated the use of the single category–based classifier neural network in visual images using a CCD camera in the C45 carbon steel turning process. Compared to manual wear measurements, the authors found that automatic image processing via neural networks leads to an average relative error of 6.7%, which significantly decreases as the tool wear increases. D’Addona et al. [49] found similar results using ANN and DNA-based computing techniques to predict tool wear in the milling process using images taken from the cutting tools. Mikołajczyk et al. [50] evaluated the use of machine vision to predict tool life in the turning process using artificial neural networks. The images were collected using a CCD camera, and the wear recognition was achieved using the Neural Wear software. The authors reported a good correlation between the measured and predicted tool flank wear, with mean errors lower than 2%. Qiu et al. [51] investigated the diamond tool wear in the lapping process, using in-process image measurements acquired through CCD cameras. The tool wear was measured using as a reference the ideal tool radius. The authors reported profile tolerances of the cutting edge within ± 0.5 µm in 40-min trials.

The use of automatic image acquisition and processing systems to identify tool wear is a great challenge in non-stationary processes such as milling. In this regard, Qin et al. [52] developed an automatic system to monitor the tool wear during face milling of the S45C steel. The system acquisition module was composed of an industrial CMOS camera, a light source, a camera holder and image acquisition software. The images were taken every 5 min, and the tool wear was measured based on dynamic image sequencing of a set of images that consisted of image processing, selection and wear measurement, as shown in Fig. 9. The results indicate that the average error of the measurement, compared to using an industrial microscope, was consistently below 10%, decreasing to negligible values as the tool wear increased.

Fig. 9
figure 9

Image processing steps in monitoring the tool wear during the face milling process [52]

Full size image

Elgargni et al. [53] evaluated tool tracking and recognition based on infrared and visual images, using and comparing neural networks coupled with the discrete wavelet transform or principal component analysis, aiming to identify tool wear in the milling process, as shown in Fig. 10. The authors reported that all combinations of images acquiring and processing methods presented a detecting rate success of over 95%, with the infrared image significantly outperforming the visual image, with a 100% detection rate when using the discrete wavelet transform algorithm.

Fig. 10
figure 10

Infrared and visual image acquisition to identify tool wear in the milling process [53]

Full size image

Bergs et al. [54] assessed the application of deep learning, convolutional and fully convolutional neural networks in the automatic detection of cutting tool wear. They used images of tool wear from drilling, ball milling and end milling operations for this evaluation. As shown in Fig. 11, deep learning outperformed the tuned traditional computer vision measurement, especially under lower light exposure. Similar results for using neural networks to improve the overall efficiency of tool wear detection using image detection were also reported by Jackson et al. [55] using SEM images.

Fig. 11
figure 11

Wear from an insert under different light exposures [54]

Full size image

Li and An [56] compared the use of an algorithm based on the sum-modified Laplacian (SML) and Markov random field (MRF) with the Nobuyuki Otsu algorithm (OTSU), in a micro-vision-based tool wear monitoring system using CCD cameras in the drilling process of the cold rolled steel using carbide tipped tools. The proposed algorithm increases between-class variation to confirm the achieved tool wear segmentation. The results showed that the proposed algorithm enabled a more precise and consistent tool wear measurement than the OTSU. Fong et al. [57] developed an offline on-machine universal monitoring system using a CMOS camera based on cross-correlational analysis. The authors tested the system under various machining tools such as a drill bit, end mill, taper tap and turning carbide insert. Despite the system only being capable of measuring tool flank wear greater than 100 µm, the authors reported that the technique provided a reliable measurement of tool wear for all machining tools evaluated. Similar results were observed by Castejón et al. [58] in the turning process of the AISI SAE 1045 steel using a CCD camera, describing the wear using the linear discriminant analysis (LDA) method. The method successfully identified the worn region of the tool, allowing reliable tool wear monitoring and evaluation. Yu et al. [59] developed a machine vision measurement method for monitoring the tool wear in the chisel edge region of the drill using a bidimensional local mean decomposition (BLMD) algorithm. The steps taken to measure the wear were composed of tool image capture using a CMOS camera and enhanced using the BLMD algorithm, and the contour is extracted using threshold segmentation as illustrated in Fig. 12.

Fig. 12
figure 12

Steps for measuring wear in the chisel edge region of a drill [59]

Full size image

The worn contour is compared with the tool’s initial condition, and the difference between the profiles measures the wear. The authors reported that the method outperformed the traditional camera measurement method, giving results closer to manual measurements. Similar results were found by Dai and Zhu [22] using a CCD camera to monitor the micro-milling of pure copper and T4 steel. García-Ordás et al. [60] proposed a low-cost solution combining machine vision and machine learning for tool wear monitoring during milling process. Parenti et al. [61] evaluated the use of high-resolution images from machined holes to quantify edge damage in the drilling process of sintered carbide using CVD diamond–coated tools. The author’s methodology was capable of characterizing the hole profile with an accuracy of up to 10–50 µm. Zhang et al. [62] studied the use of deep learning (Yolov3-tiny) to increase the accuracy in the tool wear detection based on the image processing method. The authors reported that the methodology successfully detected tool wear at the evaluated spiral of the milling cutter tool.

The greatest advantage of using image processing techniques for conventional machining processes is real-time monitoring at a low cost. For processes such as turning, where the tool does not rotate, this process is easier, having as main challenges the continuous monitoring during the cutting process and the accurate measurement of the worn/adhered volume. In processes where the tool rotates such as milling and drilling, the tool wear monitoring process can be challenging, as the camera must have a combination of high acquisition rate and resolution sufficient to enable wear detection. Among the remaining challenges for large-scale machine vision implementation in identifying the wear of cutting tools are light and tool rotation compensations. These two factors are aggravated by the high rotations used in current machining processes, which can be solved by developing cameras with higher resolution and image frame rates. However, the large amount of data generated by these cameras create another problem regarding the computational capacity for real-time processing, so it is recommended to continue investigating increasingly optimized algorithms for data processing.

3.1.2 Application of TCM of cutting edge in micro-machining and grinding using image processing techniques

Because of the smaller size, usually below 100 µm, tool wear identification in micro-machining processes is more challenging than in conventional tools. Micro-machining is more challenging due to the lower rigidity of the micro-cutting tool; therefore, accurate tool wear monitoring is essential to avoid premature tool failure. With increased image resolution and capture rate for both CMOS and CCD cameras [63], machine vision is increasingly considered a promising method to evaluate tool wear in micro-machining. Malhotra and Jha [64] developed an algorithm to evaluate tool wear in the micro-milling process using machining vision based on colour segmentation using the fuzzy c-means clustering technique. The algorithm was composed of three steps, as shown in Fig. 13, namely the ROI extraction, wear segmentation and wear measurement.

Fig. 13
figure 13

Algorithm to evaluate tool wear in micro-milling [64]

Full size image

The authors reported errors systematically below 5% compared to manual measurement, with a slight tendency of the error to decrease as the tool wear increases. Zhu and Yu [65] evaluated the use of an algorithm based on morphological component analysis (MCA) to monitor tool wear in the micro-milling process. The proposed algorithm decomposes the image in tool, background and noise then detects the tool wear from the tool image. The proposed algorithm also uses invariant rotation features to minimize the effects of inconsistent tool image orientation. The authors reported that the methodology was more accurate in the measurement of the width of the wear than its area. Despite mainly being investigated in the micro-milling process, tool wear analysis by machine vision is already gaining interest in micro-drilling and micro-turning processes. Chang et al. [66] evaluated the use of machine vision in the tool wear assessment of micro-drill probe heads. The tool wear was measured using SEM images of the drill tip and processed using grayscale treatment, histogram equalization and unsharp masking. The authors concluded that the proposed method was effective in tool wear measurement under the evaluated conditions. Similar results were also observed by Su et al. [67] using a similar methodology to investigate wear on micro-drills used to machine printed circuit boards (PCBs). Micro-turning is, by far, the least investigated micro-machining process regarding the use of machine vision to access tool wear.

Palani et al. [68] examined an online forecasting approach for assessing tool wear during the micro-turning process of pure tungsten, utilizing a PCD insert. The authors used a multi-response variable approach and an adaptive neuro-fuzzy inference system (ANFIS) to assist in the data evaluation. The tool wear images were taken using a CCD camera with a 3-megapixel resolution. The authors found that machine vision can be successfully employed for an in-process prediction of the tool wear and material removal rates. Oo et al. [69] investigated the use of machine vision in a methodology that combines a random forest classifier (RFC) and multiple linear regression (MLR) to evaluate abrasive fracture and detachment in robotic belt grinding. The image acquisition set-up is illustrated in Fig. 14a, and the flowchart for image processing is shown in Fig. 14b. The authors reported that the methodology achieved over 90% accuracy in the belt grinding condition monitoring compared to traditional image processing techniques and over 96% accuracy in modelling the remaining tool life.

Fig. 14
figure 14

Adapted from Oo et al. [69]

Application of image processing for a grinding process. a Image acquisition set-up for grinding belt monitoring; b flowchart of image processing.

Full size image

Kubo et al. [70] evaluated the use of bio-inspired DNA-based computing (DBC) to evaluate the surface topography in the dressing process of CBN grinding wheels. This approach correlates mechanical problems with DNA-RNA–protein-amino acid interactions, using decision rules to form different arrays of genetic codes that represent actual states in the grinding wheel image. The authors reported that the DBC methodology was suitable to monitor the grinding wheel state during the dressing process, mainly when the image process is processed as Boolean operations between the binary images as shown in Fig. 15.

Fig. 15
figure 15

DBC methodology proposed to monitor grinding wheel dressing [70]

Full size image

Kang et al. [71] evaluated the use of image processing to model the 3D topography of a grinding wheel. The authors modelled the alumina abrasives as pyramids with random vertices, quantifying them in relation to size, shape, volume and distribution. The authors reported that the methodology was successful in correlating the grain topography and wheel dressing state. Wang et al. [72] used machine vision for monitoring a grinding belt wear state using a random forest algorithm to process the data. The authors reported that the classifying methodology presented an accuracy of up to 99% in conditions at the accelerated wear stage. Liu et al. [73] evaluated the use of a profile reverse algorithm to evaluate the grinding wheel wear based on the image profile of the machined integral spiral grooved mills. The authors reported that the methodology resulted in successfully modelling the grinding wheel wear; however, the methodology is only applicable to flat or disc-shaped grinding wheels. Hatami et al. [74] investigated the effects of cleaning the grinding wheel surface with a compressed air jet in the grinding process of Inconel 718. The authors reported that using the air jet was possible to achieve a 53% better surface finish (Ra) and lower thermal damage.

Micro-machining and grinding are notoriously difficult processes to monitor. In the micro-machining process, the biggest challenge lies in the small dimensions of the cutting wedge coupled with the high cutting speeds, which require an even higher resolution and acquisition rate from the cameras, as well as greater accuracy in the methodology for characterizing the condition of the tool. In the grinding process, the high contact area between the tool and the workpiece as well as the random nature of the abrasives make image monitoring a challenge, requiring the use of more complex correlation algorithms than for conventional machining processes. As in both processes, the rigidity of the machine/tool system is essential, so machine vision monitoring techniques can therefore be considered ideal for monitoring.

3.2 Application of TCM of machined surface while machining with image processing techniques

In modern manufacturing, high product quality and operational efficiency are crucial. Machining processes shape and finish components to meet specifications, but tool wear and damage can degrade the surface quality and increase costs [75]. TCM is crucial in machining, allowing timely detection of wear, breakage and defects. Real-time monitoring enables informed decisions on tool replacement, maintenance and process optimization. Traditional TCM relies on indirect measurements like forces and vibrations but may miss subtle defects and requires extra equipment [76]. Image processing techniques revolutionize tool condition monitoring, harnessing visual information for analysis, interpretation and data extraction [77]. Capturing images of machined surfaces provides valuable insights into tool wear and surface integrity [78]. This part aims to explore image processing techniques for monitoring tool conditions in various machining processes (turning, milling, drilling, grinding and others). We analyse machined surface images to extract parameters like tool wear, surface roughness and defects, developing real-time monitoring algorithms.

Image processing techniques in TCM offer advantages over traditional methods. It provides direct visual information for detecting subtle defects. Additionally, it enables non-destructive, real-time monitoring, reducing downtime and enhancing productivity. Furthermore, it facilitates objective and quantitative assessment, improving decision-making for tool replacement, maintenance and process optimization [79]. This review explores image processing techniques in various machining processes to advance TCM methodologies. The findings provide insights into image processing effectiveness and lay the foundation for real-time automated systems detecting and classifying tool wear and surface defects. In the following sections, we explore image processing techniques for tool condition monitoring in various machining processes. Through analysis and evaluation, we demonstrate image processing’s potential for enhancing TCM and optimizing machining performance.

3.2.1 TCM of machined surfaces using image processing techniques in turning operations

Turning is a common and precise machining process for cylindrical component creation. Continuous tool-workpiece contact in turning leads to tool wear over time, and surface defects like scratches, chatter marks and built-up edge impact surface quality [80]. Monitoring starts with capturing images of the machined surfaces using integrated cameras or strategically positioned external imaging systems. These acquired images are processed with various algorithms to extract crucial information on tool condition and surface quality. In turning, image processing measures tool wear by analysing changes in the cutting tool’s geometry. Regularly capturing images of the tool during turning reveals wear patterns on the flank and rake faces. Image analysis algorithms detect and quantify wear, enabling timely tool replacement or reconditioning [81]. Image processing assesses surface roughness, a crucial factor in surface quality determination. Texture analysis algorithms extract features like Ra, Rq and Rz from acquired images, providing insights into surface quality [82]. This information guides adjustments in cutting parameters or tool selection for achieving the desired surface finish.

Image processing detects real-time surface defects in turning, complementing tool wear and surface roughness assessment. Defects like scratches, pits and cracks impact functionality and aesthetics. Through edge detection and morphological operations, image analysis algorithms identify and characterize defects, measuring dimensions, density and distribution. This aids in surface quality assessment and identifies process or tooling issues [56]. For efficient tool condition monitoring in turning, image processing algorithms can be integrated into a real-time feedback system. Continuously analysing images and comparing them to references or thresholds, the system alerts the operator or triggers actions like tool replacement, parameter adjustment or intervention for surface defects [83]. Applying image processing techniques for turning tool condition monitoring offers multiple benefits. Firstly, it provides non-destructive, real-time monitoring for proactive maintenance of machined surface quality. Secondly, it detects subtle changes in tool condition and surface quality that may be imperceptible to the human eye. Additionally, it enables objective, quantitative assessment of tool wear and surface defects, reducing reliance on subjective visual inspection [84]. Chethan et al. [85] discussed the value of non-contact surface texture assessment over more conventional techniques that employ a stylus. It claims that instead of focusing on a single line, machine vision can evaluate roughness throughout the complete surface area. The study focuses on observing the Nimonic75 material’s machined surface while varying the spindle speed and feed rate of a coated carbide tool. Histogram analysis, which determines the predominant grey level in an image, is used to examine the acquired data. Surface images are captured by machine vision cameras and preprocessed to enhance the image quality as shown in Fig. 16. However, difficulties including uneven lighting and image noise are noted. These problems are reduced by using image segmentation and enhancement algorithms. To calculate surface roughness, the histogram distributions of the surface pictures are examined. The findings show that histogram analysis is a viable method for measuring surface smoothness and that cutting conditions have little impact on surface roughness.

Fig. 16
figure 16

The images of the surface are displayed in grey scale. The surface was machined at a speed of 710 revolutions per minute (rpm), with a feed rate of 0.05 mm per revolution (mm/rev), and a depth of cut of 0.2 mm [85]

Full size image

Figure 17 illustrates the experimental design, comparisons of surfaces shaped by both a new and a degraded tool with their enhanced images and the development process of Voronoi divisions, including Canny edge detection and Voronoi tessellations for surfaces formed by a new tool. The monitoring of cutting tool conditions is crucial for maintaining product quality during machining. A recent study introduced an innovative on-machine tool condition monitoring method through analysing turned surface images. This method utilizes image texture analysis techniques such as the grey level co-occurrence matrix (GLCM), Voronoi tessellation (VT) and discrete wavelet transform (DWT). Eight key features were extracted to track tool flank wear. Voronoi tessellation clusters points based on local neighborhoods to gather precise topological data. In this study, it was applied to images of turned surfaces to identify specific texture primitives like feed marks. Turning experiments were conducted on a CNC lathe using C-50 steel. These experiments covered varied machining conditions and were replicated three times for accuracy. Post-machining, images of machined surfaces were captured consistently using a camera set-up. Flank wear was measured using an Olympus microscope. These surfaces reveal tool wear indicators such as waviness, feed mark details and surface roughness. Different texture analyses provided insights into these changes. The study compared individual and combined analysis techniques. The combined approach proved to be superior in predictive capability, with a maximum 4.9% prediction error, making it effective and trustworthy for practical use [50, 86].

Fig. 17
figure 17

a Experimental set-up, b images of surfaces turned by a fresh tool, c images of surfaces turned by a worn tool (VB_average = 362 µm), d preprocessed images of b, e preprocessed images of c, f construction of Voronoi tessellation, g Canny edge–detected image, h Voronoi-tessellated image of b, c, d and e [86]

Full size image

3.2.2 TCM of machined surfaces using image processing techniques in milling operations

Milling shapes and creates complex components but faces challenges of tool wear, chip formation and surface defects [87]. This section explores image processing techniques for milling tool condition monitoring, aiming to enhance tool performance and surface quality. Tool condition monitoring in milling begins with capturing images of the machined surfaces. Cameras can be integrated into the milling machine or positioned externally to capture comprehensive views of the cutting zone, ensuring accurate image acquisition through strategic placement and synchronization. In milling, analysing flank wear is crucial for tool condition monitoring [88]. Image processing techniques enable the quantification of flank wear by detecting changes in tool geometry over time. Algorithms like edge detection and feature extraction identify and measure worn areas on the tool’s flank faces. Comparing these measurements, such as flank wear land width, length and radius, to thresholds or historical data determines the remaining tool life. This facilitates timely replacement or reconditioning for optimal milling performance [46].

Image processing techniques also enable the monitoring and analysis of chip formation and disposal in milling. By capturing images of the milling process, chip size, shape and evacuation can be assessed [89]. Algorithms like edge detection and morphological operations identify and measure chip dimensions and morphology. Deviations from expected chip formation patterns, such as long or irregular chips, may indicate cutting tool or cutting condition issues. Real-time monitoring detects abnormalities, facilitating adjustments to cutting parameters or tool selection for optimized milling [90]. Image processing techniques assess surface roughness in milling for tool condition monitoring [91]. Algorithms for texture analysis scrutinize the texture and topology of a surface to derive statistical attributes such as mean surface roughness (Ra), root mean square roughness (Rq) and the difference between the highest peak and deepest valley (Rz) from the images gathered. Continuous monitoring of surface roughness allows adjustments to cutting parameters, tool selection or machining strategies for desired surface finish and consistent quality [92].

Image processing techniques detect and characterize surface defects in milling. Algorithms like edge detection, thresholding and pattern recognition identify and classify defects based on size, shape and intensity. By comparing acquired images with references or predefined models, defects are detected in real time, enabling prompt intervention and minimizing scrap or rework. Integrating image processing into real-time monitoring enables immediate feedback in milling. Continuous analysis and comparison with thresholds or reference data trigger alerts, and actions like tool replacement or parameter adjustment. Real-time monitoring enhances tool performance, improves surface quality and optimizes milling processes [93]. Ravimal et al. [94] proposed an innovative machine vision system developed to effectively examine and classify surface textures seen on medium and large mould products. These products are commonly used in items like cars, televisions and refrigerators. The current methods rely on the manual expertise of skilled workers, which leads to inconsistencies and consumes significant time. While numerous precise surface inspection and measurement approaches exist, their practical application in industrial settings or with finishing robots is challenging due to issues concerning speed, set-up limitations and reliability. This research suggests the use of two image processing algorithms as shown in Fig. 18 to facilitate automated surface examination during a lapping process, commonly used to remove milling tool marks when performed without human supervision. Initially, the state of the machined surface and the existence of tool marks are evaluated as the line light source moves in a counterclockwise direction. This evaluation involves analysing the form of the reflected light as well as the intensity of the near-field contrast image that is promptly seized following the specular light reflection. Second, by examining the normal map that reconstructs the surface, the photometric stereo approach is used to detect surface scratches. The suggested methods accurately classify data and successfully disclose localized machining patterns.

Fig. 18
figure 18

Procedure for detecting tool marks using image processing techniques [94]

Full size image

For the analysis and classification of surfaces using a deep learning algorithm, Carbone et al. [95] introduced a novel method in this study. The goal is to make it possible for machining centres to assess the acceptability of process parameters used during milling operations based on the impacts that can be seen on the machined surface. Operators can therefore use a reverse engineering technique to recreate the original process parameters accountable for the evaluated surface to change the process parameters to improve the quality of subsequent components. A shallow convolutional neural network was created to analyse surface image patches to accomplish this as shown in Fig. 19, utilizing a small training dataset made up of good and undesirable cutting circumstances. There are three layered convolutional blocks in the network design. The fivefold cross-validation method, which involved calculating the mean and standard deviation of the f1-score metric, was used to assess the effectiveness of the suggested solution. Surprisingly, the algorithm’s average classification accuracy outperformed the best state-of-the-art technique by 4.8% [95].

Fig. 19
figure 19

On the left is the original image of a milled surface sample, while on the right is the preprocessed version of the same sample image. During preprocessing, patches measuring 224 × 224 pixels are extracted from the image and subsequently converted to grayscale [95]

Full size image

3.2.3 TCM of machined surfaces using image processing techniques in drilling operations

Drilling creates holes in materials but causes tool wear, chip formation issues and surface defects [89]. We are investigating the use of image processing to monitor the condition of tools during drilling, to improve both performance and the quality of the finished surface. The initial step in using image processing methods for monitoring the condition of tools during drilling involves capturing images of the surfaces being machined. Cameras integrated into the drilling set-up or positioned externally capture clear and comprehensive views of the process, requiring proper placement and synchronization. One crucial aspect of drill condition monitoring in drilling is analysing drill wear using image processing techniques. Capturing images of the drill at regular intervals during the process allows for observing and measuring changes in tool geometry [96]. Image analysis algorithms, like edge detection and feature extraction, can detect and measure wear features, such as flank wear and chipping, on the drill’s cutting edges. These algorithms use techniques like thresholding and segmentation to accurately identify the edges and measure wear dimensions. Machine learning approaches can also develop predictive models to correlate wear features with tool life or remaining useful tool life (RUTL), enabling proactive tool replacement or reconditioning for optimal drilling performance [97].

Image processing techniques in drilling facilitate surface quality assessment, including surface roughness evaluation and defect detection. The analysis of acquired images allows texture analysis algorithms to extract statistical attributes like average surface roughness (Ra), peak-to-valley height (Rz) and root mean square roughness (Rq). These attributes quantify the level of surface roughness, offering valuable insights into the overall quality of the surface [98]. Image processing techniques aid in detecting and characterizing surface defects in drilled components by identifying, locating and classifying defects based on size, shape and intensity. Edge detection, morphological operations and machine learning techniques enhance real-time defect detection and classification in drilling [99]. This enables prompt actions like adjusting parameters, replacing the drill or implementing quality control measures for defect-free components. Integrating image processing into real-time monitoring enables prompt feedback and decision-making during drilling. Continuous analysis of images allows for timely actions based on deviations from desired tool conditions or surface quality, including tool replacement, parameter adjustments or intervention for surface defects [100].

Fibre-reinforced plastic (FRP) laminates are at risk of delamination, particularly during machining operations like drilling, when stress is concentrated. Davim et al.’s [101] study is to introduce a new method for measuring the adjusted delamination factor (Fda) through digital analysis. A drilling experiment was designed to evaluate FRP laminates under different cutting conditions. Digital analysis was employed to assess the extent of delamination caused by drilling with multi-image processing steps as shown in Fig. 20. The experimental findings demonstrated that digital analysis is a reliable approach for estimating the damage caused by drilling carbon fibre-reinforced plastics [101].

Fig. 20
figure 20

Image processing operations are employed to ascertain the final shape of the damage zone. This process involves three steps: a capturing a digital image, b conducting initial processing and c obtaining the resulting image, which represents the ultimate form of the damage zone [101]

Full size image

Kurek et al. [102] used an innovative application of transfer learning using a convolutional neural network (CNN) to discern the state of a drill based on images of holes drilled into laminated chipboard. The method categorizes the drill state into three classes: red, yellow and green. A red classification indicates a worn-out drill necessitating immediate replacement, while yellow signals a need for manual inspection by the operator. On the other hand, green represents a drill in good working condition, suitable for continued use. A significant benefit of this transfer learning approach is its ability to train a classification model with only a small amount of data, unlike traditional CNN methods which demand extensive databases for satisfactory recognition accuracy. In the study, the dataset comprised three subsets totaling 242-hole images: 102 images for the green class, 60 for yellow and 80 for red. Figure 21 provides visual examples of each class. Notably, the research employed the pretrained AlexNet CNN network, emphasizing its capability to achieve accurate drill state recognition with a limited sample size. Comparatively, results from this method outperformed those from traditionally learned CNNs, even when the latter used augmented data through rotation and scaling techniques. Furthermore, the pretrained network, which initially used over 1 million diverse images for 1000 different classes, required minimal adjustments in its final three layers, expediting the training process and enhancing the system’s generalization properties in class recognition.

Fig. 21
figure 21

Illustrations of void pictures representing three categories of data: a green, b yellow and c red [102]

Full size image

3.2.4 TCM of machined surfaces using image processing techniques in grinding operations

In grinding, tool condition monitoring is crucial for optimizing performance and surface quality. Image processing techniques are applied to monitor tool wear, thermal damage and surface defects, enhancing grinding operations [103]. In grinding, tool condition monitoring starts with capturing high-resolution images of the machined surfaces. Cameras are integrated into the grinding machine or positioned externally, considering optimal lighting and camera settings for clear and accurate image acquisition. Image processing techniques are crucial for monitoring tool wear in grinding. By capturing and comparing images of the cutting tool before and after grinding, changes in tool geometry can be observed and quantified.

Image analysis algorithms, including edge detection and feature extraction, detect and measure wear features like flank wear or crater wear on the tool’s surface. Techniques like thresholding and segmentation accurately identify the tool’s edges and quantify wear dimensions. Machine learning can develop predictive models linking wear features with tool life or RUTL, enabling proactive replacement or reconditioning for optimal grinding performance [46]. Image processing techniques assess surface quality in grinding by evaluating surface roughness and detecting defects. Algorithms for texture analysis scrutinize the images that have been gathered, deriving statistical characteristics such as mean surface roughness (Ra), root mean square roughness (Rq) and the disparity between the tallest peak and the deepest valley (Rz). These parameters quantify surface roughness, offering insights into the quality of ground surfaces [104].

Image processing techniques detect and characterize surface defects in ground components. By analysing acquired images, defects like grinding marks, burns or irregularities can be identified, located and classified based on size, shape and intensity. Edge detection, morphological operations and machine learning techniques enhance defect detection and classification in real time. This enables immediate actions like adjusting parameters, replacing the wheel or implementing quality control measures, ensuring high-quality ground components [98]. Integrating image processing techniques into a real-time monitoring system enables prompt feedback during grinding. Continuous analysis of captured images detects deviations from desired conditions or surface quality, allowing timely actions such as tool replacement, parameter adjustments or defect intervention [69].

For high-precision applications to perform as intended, a machined surface’s smoothness and flaws must be carefully inspected. Manish et al. [104] introduced uncomplicated image processing methods for inspecting the surface finish and defects of a ground surface. The study looks at how surface finish influences pixel intensity distribution and edge identification, using a machine vision system. This investigation is conducted on rectangular samples of mild steel, employing methods such as Canny edge detection and histogram analysis. Based on individual pixel values, the image’s grayscale intensity distribution is analysed as shown in Fig. 22. A visual comparison of edge recognition across various grinding surface conditions is possible thanks to the use of the Canny edge detection method. An experimental analysis that tries to categorize object surfaces under various surface grinding settings is offered as a starting point for further investigation [104].

Fig. 22
figure 22

The image on the left is a grayscale image, while the image on the right is the Canny edge image. These images were captured under conditions where the surface was smoothly ground [104]

Full size image

Grinding is a process of machining that involves abrasion and can result in extremely smooth surface finishes. However, the chips that are removed from the workpiece during grinding and become attached to the gaps between the abrasive grains can adversely affect the final surface finish of the machined part. To better understand the cutting conditions of the grinding wheel, it is important to quantitatively analyse the phenomenon of wheel loading by Gopan and Wins [105]. Their study aims to develop a system that utilizes image capturing and image processing techniques, taking advantage of advancements in machine vision and image processing as shown in Fig. 23. Microscope images of the grinding wheel were obtained at a magnification of × 20. The loaded sections of the wheel were separated from the rest of it using a technique called global thresholding for image segmentation. The experimental results demonstrate the practicality of the proposed system in quantitatively assessing the loading of the grinding wheel [105].

Fig. 23
figure 23

a The image of the grinding wheel as it appears after 90 min. b The grinding wheel image after 80 min it has been processed [105]

Full size image

3.2.5 TCM of machined surfaces using image processing techniques in other machining operations

Besides turning, milling, drilling and grinding, other machining processes such as broaching, honing, lapping and electrical discharge machining (EDM) are widely used in manufacturing. These processes involve cutting tools that can experience wear and degradation, affecting surface quality [106]. In this part, we examine how image processing techniques can monitor tool conditions in these operations, optimizing performance and ensuring quality surfaces. Image processing techniques for tool condition monitoring in other machining processes follow the same principle as discussed earlier. It begins with capturing images of the machined surfaces using integrated or external cameras. Good lighting conditions and camera settings are crucial for obtaining high-quality images for accurate analysis. After acquiring the images, image processing techniques are applied to monitor tool wear and evaluate surface quality. The algorithms and techniques used vary depending on the machining process. In broaching, image analysis algorithms detect and measure wear on the cutting edges [107]. In honing [108] or lapping [109], image processing assesses surface roughness and detects defects. The choice of image processing algorithms depends on surface characteristics and monitoring goals. Algorithms like edge detection, feature extraction, texture analysis and machine learning can identify wear patterns, quantify dimensions, assess surface roughness and detect defects. Continuous analysis of images allows for the detection of deviations, prompting actions such as tool replacement, parameter adjustments or defect interventions [110].

Real-time monitoring systems with image processing enable immediate feedback and informed decisions during machining. Continuous analysis of images reveals tool wear, surface roughness and defects, enabling proactive maintenance, parameter optimization and high-quality production. Monitoring the wear of tools is crucial to ensure a consistently high quality of machined products. In the past, tool wear in typical machining operations like turning or milling has been thoroughly understood and researched. However, when it comes to cutting complex profiles, such as linear broaching, the only reported method for quantifying tool wear has been manually characterizing flank wear. This approach results in a significant loss of information and a large variation in measurements. To overcome these drawbacks, Loizou et al. [107] proposed a new measurement system that measures broaching tool wear by taking into account the total wear area. The suggested solution does not require manual interaction because it uses automatic image cropping and digital imagery processing tools to identify the impacted area. A hexagonal linear broach has been used to perform a detailed examination of the measurement system to assess the variability produced by the measurement processes and image analysis as shown in Fig. 24. By implementing this measurement system, the characterization of tool wear in broaching tools becomes more accurate, enabling easier collaboration across industries, reducing the intensity of operator training and enhancing quality control practices.

Fig. 24
figure 24

The left image shows the result after cropping, while the right image displays the final binary image obtained from all the steps in digital imaging processing (DIP) [107]

Full size image

It is essential to have an accurate and automated examination of the completed surface during the site lapping process to successfully deploy robotic machining. Many sectors now routinely employ sophisticated inspection systems, which make use of smart sensor technology like image processing and machine vision. In addition to increasing inspection accuracy, these devices and the idea of a “smart factory” also greatly reduce the need for human labour. In a vision technique that uses specular light, Dar et al. [111] proposed a way for automatically levelling a machined surface based on its roughness values as shown in Fig. 25. The study’s major goal is to create a technique for levelling surface roughness by examining the topography and texture of the surface. This method makes use of the basic property of light reflection, which causes the reflection to switch from diffuse to specular depending on the texture of the surface. Grayscale values on a rough surface with tool marks show contrast, which decreases intensity value and vice versa. They were able to ascertain the underlying grayscale values of the investigated surface by using image processing techniques. As the roughness decreased, the data showed a nonlinear increase in grayscale values. They aligned the surface normal with the perspective centre of the camera to produce the maximum image resolution. This idea was expanded to include curved and sloped surfaces. They created a multiscale measuring approach that can be used for a wide range of roughness to acquire precise results with high accuracy. This approach depends on altering the camera distance to provide high-resolution readings rather than an isolated system. The proposed technique exhibited high accuracy and resolution, achieving a surface roughness levelling capability of up to 20 nm (Ra). These findings indicate that this technique can be utilized for levelling multiscale free-form metal surfaces.

Fig. 25
figure 25

The algorithm under consideration involves inserting an image, converting it into a binary image, reducing noise and then applying region labelling to determine the area of the target region [111]

Full size image

3.2.6 Conclusions on the subsection

This section discusses the applications of image processing techniques in monitoring tool condition during various machining processes. By analysing images obtained during machining, these techniques offer valuable information about tool wear, surface roughness and defects. This enables proactive maintenance and facilitates optimal machining for achieving high-quality surfaces. The section delves into different subsections focusing on specific machining processes. In the turning subsection, image processing is used to monitor tool wear, assess surface roughness and detect defects. Real-time integration of image processing enhances tool performance, productivity and component quality. Similarly, in the milling subsection, image processing is employed for monitoring tool wear, analysing chip formation, evaluating surface roughness and detecting surface defects. By analysing acquired images, milling processes can be optimized, ensuring the production of high-quality components. In the drilling subsection, image processing techniques are applied for tool wear analysis, surface roughness assessment and defect detection. Real-time monitoring systems integrated with image processing enable timely actions for optimal drilling performance and the production of defect-free components. In the grinding subsection, image processing is utilized to monitor tool wear, evaluate surface roughness and detect defects. Analysis of acquired images facilitates the optimization of grinding processes, leading to the production of high-quality ground components.

Furthermore, the section explores the application of image processing in other machining operations, including tool wear monitoring, surface roughness assessment and defect detection. The integration of image processing into real-time monitoring systems ensures proactive maintenance and high-quality results across different machining processes. Overall, the integration of image processing techniques for tool condition monitoring in machining processes offers significant benefits. It provides insights into tool wear, surface roughness and surface defects, allowing for timely actions to improve tool performance, productivity and surface quality.

In conclusion, the application of image processing techniques in tool condition monitoring during machining processes holds great promise. It offers advantages such as enhanced tool performance, optimized machining parameters and high-quality machined surfaces. However, further research is required to advance algorithms, real-time monitoring systems and decision-making models to enhance the effectiveness and efficiency of tool condition monitoring [112].

4 Challenges and future trends

4.1 Overall rating of image processing application in machining for recognizing and measuring tool wear

In Table 1, examples of image processing using tool wear recognized and measured in machining are presented. More applications are used for tool wear monitoring in different machining processes:

  • Turning and micro-turning

  • Milling, micro-milling, ball milling and end milling

  • Grinding

  • Belt grinding

  • Lapping

  • Tapping

  • Drilling

  • Dressing

Table 1 Image processing application in machining
Full size table

The second type of image processing application is indirect methods used to analyse the surface image of the machined workpiece for.

  • Turning and micro-Turning

  • Milling, micro-milling, end milling and face milling

  • Grinding

  • Drilling

The use of images for TCM requires the use of appropriate technology, which is achieved by using high-resolution cameras with both CMOS and CCD sensors. Obtaining a still image of excellent quality is not a problem. In some applications (milling, grinding, etc.), images of tools in motion are required. The blurring effect appears when saving the image moving workpiece and low speed of image capture. This effect can be reduced by using a high-speed camera, a better front lighting system and special software for the de-blurring of images. Also, it is possible to improve the image processing algorithm. However, to cope with the computational requirements, the image area for processing should be optimized with a simpler algorithm. The benefits of image processing vary according to the usage techniques. Each of these techniques used in image processing approaches the image from a different angle. At the core of the studies on image processing lies image analysis and therefore digitization, and nowadays, image processing, design, manufacturing, electronics, machinery, etc. It is a general field of study used in many areas, and considering the diversity of the areas used, the number of studies in this area is large [121]. For this reason, this review aims to make a general evaluation of the machining studies in the field of image processing.

4.2 Perspectives on using image processing in modern manufacturing systems

Image processing makes it possible to assess the condition of the tool edge in a non-contact way. The presented application is an important step toward the development of image processing for TCM. This technique provides data on the condition of the edge directly (tool image) or indirectly (assessment of the condition of the surface treated with the analysed tool). These methods of assessing the TCM in the production process should be used to support the manufacturing process:

  • Artificial intelligence

  • Smart manufacturing

  • Industry 4.0 (internet of things)

  • Digital twins

  • Big data

In the literature on the application of image analysis methods, examples of such implementations of TCM can be found.

An example of the application in support of the manufacturing process AI was presented in previous studies [48, 50, 122]. Mikołajczyk et al. [48] presented an AI automatic image analysis of cutting edge wear based on a neural network application. A study conducted by Jackson et al. [55] demonstrated the use of neural networks and image processing for forecasting the lifespan of tools in turning operations. The effect of process parameters on process performance was analysed using machine learning [122] which could prolong tool life. The smart manufacturing system was defined and discussed based on the current manufacturing system and was analysed by contribution to future implementation of the smart manufacturing system [123]. A smart milling tool with indexable inserts was presented as a system of cyber-physical configuration [124]. It was realized by capturing features on each edge (force, temperature, vibration). This information can be utilized to forecast tool degradation. The trained process can prepare messages about the critical state of the cutting insert (to be replaced or turned).

Industry 4.0 (or the internet of things) will change the way the machining process is approached [125]. Industry 4.0 streamlines the flow of data to the machine tool—it transmits details on what has been produced, the precision of the output and comprehensive data regarding the machine tool’s performance. With Industry 4.0, data from all sensors can be compiled and utilized for anticipatory maintenance. Ghionea et al. [126] presented an implementation for Industry 4.0 of measuring mechatronic systems for comparison of machined on the lathe with the theoretic CAD model of rail wheel. The vision system will be a critical part of automation systems in Industry 4.0 [127]. This system makes using data from digital images for process or quality control possible. This is better suited to repetitive tasks than human inspectors. Machine learning used with mathematical modelling were presented to tool life estimates for small lot production [128]. The cost of the application is reasonable. It was used for the development of digital twin for production. Authors analyse some models not connected with a system of tool wear control. Christiand and Kiswanto [129] show the digital twin (DT) application in micro tool wear monitoring (TWM) of micro-milling process based on spindle motor current. The proposed DT shows the possibility of TCM in a micro-milling application for Industry 4.0. Xie et al. [130] introduced a digital twin system–based tool condition prognosis model to increase prognostic accuracy and reaction speed. Vibration and current signals are employed as input data. The experimental findings suggest that the presented model is effective and generalizable.

As was presented in references, proposed advanced systems for TCM were based on AI, Internet 4.0, SM, DT, big data and cloud computing possibilities. In this review, using the presented application of image processing as input data is a good way to prepare a new application for an advanced manufacturing system based on the knowledge of previous events for building models useful in industrial practices. Based on vision methods, image recognition of tools’ edges or machined surfaces can make tool wear monitoring in real time possible. This will be included in the systems being built. It is expected that in the future, DT will save data by optimizing the performance and parameters of the process before starting the actual production process. It avoids the cost of trial and error. It should be expected that the rapid development of computer technology and mechatronics will also apply to solutions for recording images with appropriate resolution and speed (required especially for rotating tools). Of course, this requires the development of image analysis software as well as the improvement of edge wear (life) prediction models based on measurement results and prognostic models.

5 Conclusions

Vision systems equipped with image recording and analysis systems are also used in the industry for monitoring machining processes. This paper presents, for the first time, a comprehensive review of image processing applications in machining for tool condition monitoring (TCM), to recognize and measure tool wear. Both the direct and indirect image processing techniques of the cutting tool were analysed. Based on the literature analysis presented, conclusions are drawn concerning both the applications of image processing for TCM and the prospects for development:

  • In direct sensor system–based image processing, by synchronizing the camera with the tool, the instantaneous acquisition and processing of images can be performed. The tool wear parameter can be easily and quickly measured, directly on the processed PC screen image without using any other device. The direct method is more popular and is used for turning, milling, drilling, grinding, belt grinding and others.

  • For direct methods, the set-up was based on a camera (CCD or CMOS) equipped with an optical system and special system to the light of tool edge zone and testbeds specialized for machining methods. The captured image was analysed using image processing software. Elaborated on many applications that ensure the extraction of the image of the worn cutting edge, both stationary (turning, milling, drilling) and in motion (mainly milling). The assessment of the condition of the edge in motion creates greater requirements, but modern systems allow such an assessment thanks to the speed of the cameras used and the possibilities of computer technology.

  • In indirect sensor system–based image processing for the tool condition monitoring in machining, tool wear is estimated using changes in chip geometry, slot and burr widths, tool temperature, workpiece surface roughness and cutting force. The image processing method can be also useful to protect the tool and workpiece and save time by stopping the machining because of abnormal changes in the cutting force graph. In image processing measurements based on chip geometry and workpiece surface quality, obtaining data for image processing is slower and the process is more predictive.

  • Machine vision can track the gradual wear and tear of tools, tool breakage, non-compliant tolerance levels, chatter and vibrations. The shape and design of a tool influence the surface that has been machined. Evaluating images of the machined surfaces allows us to gain insights into the current state of the cutting tool.

  • The following image recognition and classification systems in TCM were mainly used:

    • A neural network and back propagation neural network (BPNN) are a flexible method in which the desired functions can be learned without specifically mentioning the function features. Moreover, good results are generally obtained with these standard methods.

    • Applying Image Tool software within this software, parameters such as distances, tilt angles, areas and depth can be measured directly on a screenshot. Measurement in pixels or mm is possible with the specified conversion factor. A magnification of × 70 is sufficient to distinguish the desired features in a captured image. The program parameters are adjusted to the desired extent in the tool image software. Tool wears are then carried out, corresponding to 1 pixel = 0.001 mm.

    • Image thresholding method (Otsu method and K-means). The image thresholding technique can separate objects from the background and objects with different grey levels.

    • The tool shape descriptor (TSD) is a method used for identifying shapes and is part of the dimension-based image classification techniques. It serves as a morphometric instrument in the analysis of the contours of images that are nearly convex [30]. The shape, appearance or component of the image can be predicted and classified with the tool shape descriptor. In this method, the shapes of edge-confined objects and basic images are compared basically.

  • The review article presented many set-ups for application of vision methods for TCM. This overview is very important for understanding the operation of such systems. Vision system components are commercially available. Current applications are not particularly expensive. Of course, the operation of vision systems requires the use of appropriate software.

The current state of advancement of TCM based on image processing both in terms of direct method of tool image analysis and indirect method for surface condition analysis will be used in the production process, and the following developed models should be used in the development of modern techniques to support the manufacturing process:

  • Artificial intelligence

  • Smart manufacturing

  • Industry 4.0 (internet of things)

  • Digital twins

  • Big data