1 Introduction

Blanking is an essential part of many process chains in the sheet metal working industry. By combining blanking with other forming operations, multi-stage tools can be constructed that enable the manufacturing of a wide range of products. These range from components for digitalization, such as electrical contacts or circuit boards, to consumer goods, medical components, automotive parts, and planar components such as car body panels or visible surfaces of buildings. As the blanking operation in these process chains defines the final geometric and functional product properties, reliable handling of this process step is crucial. In addition to the accuracy of the target cutting line, the characteristics of the blanked surface are a key performance parameter describing a fault-free blanking operation. According to Subramonian, the deterioration of the blanked surface is described in two ways [1]. On the one hand, the geometric properties can change, resulting in an uneven distribution of the shear zone or leading to excessive burrs. On the other hand, functional properties such as hardness distribution, sensitivity to cracking or magnetic properties can worsen. In addition, deterioration of the cutting edge can be a direct indication of a failed tool system (e.g., wear). Although the challenges posed by these demanding geometric and functional requirements have a major impact on process profitability, reliability, and safety, in practice, QA is limited to manual and random inspection by employees and data-based binary QA. There are four main reasons for this. (1) QA is highly employee dependent and therefore not very reproducible. (2) A detailed description of the relevant features of the blanked surface is limited. (3) When the description of blanked surface is based on data, it is binary and limited to a good/bad classification. (4) There is a long latency between the occurrence of blanked surface defects in the process and their detection in an offline QA system. Although the blanked surface is a critical factor in multi-stage processes, the authors are not aware of any industrial solution that allows 100% in-line QA of the blanked surface during blanking. With reference to industrial practice and literature, five categories are identified as barriers to an application: installation space, system dynamics, blanked surface, production rates, and cost.

Installation space: The installation space limits the integration of sensors and measuring systems in multi-stage tools. Due to the tight packing of the individual forming operations and the necessary components such as guides, blank holders, strippers, and ejectors, there is not enough space available, especially in existing tool systems. In addition, the integration of sensors must be considered in the tool design phase in order to create the necessary space and to control the properties of the tool system (e.g., rigidity of the tool system [2]). System dynamics: As a result of the press movement, impulse-like loads are created in the forming zone, causing the system to oscillate when the tool hits the metal strip and when the product is separated from the metal strip. On the one hand, this causes the sensor system to vibrate. On the other hand, blanked workpieces are ejected at a high speed in an uncontrolled manner due to the energy released to the system by the movement of the press. This effect is enhanced by the jamming and slugging of the blanked workpiece in the die [3]. This makes it difficult to reliably detect defined areas on the ejected product. Blanked surface: As the blanked surface sd is much smaller than the cutting line sl, so that sd ≪ sl applies, there are high demands on the sensor system to describe the features of the blanked surface. It is known from industrial practice that multi-stage tools with a blanking operation process material with a thickness of less than 2 mm, and when processing thin sheets (e.g., stator sheets, electrical connectors), thicknesses of less than 0.5 mm are encountered. Production rates: With processes involving a blanking operation running at production speeds of up to 2000 spm, the real-time capability of the measurement system is particularly important. This means that data acquisition, processing, and the return of blanked surface information must take place within a single stroke [4]. Cost: In addition to the technical challenges, the economic viability of the system must also be taken into account. For SMEs in particular, the investment required for such a sensor system, including software and hardware, must be kept as low as possible to achieve a direct return on investment in the shortest possible time.

Therefore, the aim of this work is to develop an optical system that allows the direct detection of the blanked surface as a critical parameter in a multi-stage process with a high degree of precision. Thereby, the quantification of the properties of the blanked surface is not limited to the base area (from small plug-in contacts to large automotive outer skin parts) of the workpieces, but mainly to the thickness of the components. The authors focus on an optical system as it allows a direct, qualitative, and quantitative inspection of the blanked surface. Indirect, data-based model estimation using time series as input (e.g., force or acceleration) should not be considered, as such supervised approaches require large amounts of labeled data. In particular, the generation of labeled data for fault conditions is costly and time-consuming for the company due to the low probability of occurrence and high technical effort. In this way, the system can generate information about the current state of the product and assign it as a label to the sensorial acquired process data for the further training step of a machine-learning model. The goal of the optical system in the context of modeling is therefore the inline preparation of labels for the training process.

In addition, a methodology will be developed to automatically evaluate images of the blanked surface and to enable segmentation and extraction of the relevant features. In the design of the optical system and the underlying methodology, particular attention will be paid to the use of commercially available components (software and hardware) to keep the investment requirements low. To demonstrate the functionality of the system, two use cases are investigated. In the first case, the quality features of the blanked surface, the so-called key performance parameters (KPP), are derived from the acquired images. Their ability to quantify the blanked surface is compared and evaluated with the time-consuming microscope inspection used in industry. In the second case, since there is a direct correlation between the wear condition of the blanking tool and the product properties, a classification model to describe the wear state of the tool is derived from the KPPs. It is expected that high performance of the model will be achieved by extracting high-quality features of the blanked surface.

In order to achieve these results, the paper is structured as follows: Section 2 gives an overview of existing methods in optical QA of forming processes and the application of existing systems and their limitations. The potential of image-based monitoring compared to indirect methods is also discussed. In Section 3, the experimental setup is shown and a correlation between tool wear and blanked surface degradation from preliminary investigations is demonstrated. The methodology for evaluating and extracting the KPPs is then presented. Section 4 describes the extracted KPPs and their suitability for quantifying the quality of blanked surface. In Section 5, an SVM for classifying wear states based on the extracted KPP is presented. Finally, Section 6 summarizes the results and provides a roadmap for future work.

2 Optically based quality assurance during forming

Monitoring of forming processes, especially sheet metal forming processes, has been part of the literature for several decades. Thereby, the monitoring is mainly based on the use of time series, as there is a known way to generate the necessary database. The most common monitoring systems are based on empirical rules such as threshold [5, 6], windows or envelopes [7], and linear discriminant functions [8] using force [9], temperature [10], or acceleration [11] as input signals. As the amount of data available in production increases, more detailed estimates of faulty process states using ML applications have become part of current research [12]. One of the first studies in this area was presented by Koh, who used a Haar transformation to extract features from a strain gauge–based force signal with threshold detection to describe discrete fault types [13]. In his work, Ge et al. presented a support vector machine (SVM) with different kernel functions to classify faulty process states during blanking, using strain gauge–based force signals as input to this classifying ML model. The authors’ result demonstrated the potential of ML by achieving a successful fault detection rate of up to 99% [14]. Beyond these simple fault detection use cases, increasingly complex problems are being solved through the application of ML. For example, Hambli et al. presented a backpropagation neural network for predicting burr height formation on blanked workpieces, taking into account tool clearance and wear condition [15]. In addition, Kubik et al. demonstrated that an SVM combined with features from a force signal extracted with a principal component analysis (PCA) was able to classify six wear states during blanking with 100% accuracy [16]. With the trend towards describing increasingly complex process relationships, deep learning (DL) models are being applied to the underlying problems. For example, Huang and Dzulfikri demonstrated that a convolutional neural network (CNN) was able to discriminate between seven wear states with over 99% accuracy [17]. The authors transformed acceleration signals into the frequency domain and used the power density spectrum as input to a one-dimensional CNN. Unterberg et al. captured magnetic Barkhausen noise from multiple coils and used it to generate recursion plots that served as input to a one-dimensional CNN [18]. Using this model, they were able to classify whether a section of material belonged to the beginning, middle, or end of a coil.

As the literature shows, these monitoring approaches in sheet metal forming processes are mainly based on time series, although images could directly represent critical quality features of the product. In production, a distinction is made between in-line and off-line optical quantification of these features. While offline systems often involve the use of handheld devices, mobile measuring stations, or collaborative robots, inline optical measurement is mainly carried out by sensors mounted on machines or robots [19]. The images are captured using lasers, projections (e.g., fringe projection), interferometry, or (high-speed) cameras.

Currently, optical systems are mainly used for geometric and dimensional inspection or surface detection. In this context, Tran and Ha proposed a high-resolution camera and multi-line laser to measure the gap and geometric dimensions of car body parts during the assembly process. The system allowed complex surfaces to be measured directly inline on the vehicle [20]. Mannan et al. extracted features from images, combined them with sound data, and fed the data into a neural network to describe different geometric states of a milling tool [21]. To validate the actual geometric shape of the tool, the results were validated with ground truth from an optical microscope. Chen et al. captured images of a milling tool and extracted the contour using a Sobel operator. By applying subpixel routine, the tool contour was determined with an accuracy of ± 2 μm [22]. Gupta and Ramen determined surface roughness during turning by analyzing images of a laser scattering pattern at rotation speeds ranging from 140 to 285 revolutions per minute (rpm) [23]. By extracting the gray levels in a histogram, the surface roughness heights were determined independent of the ambient lighting. Lee et al. proposed digital image processing to automatically detect cracks in aluminum alloys [24]. The images were captured with a microscope and converted into binary images. With this procedure, cracks with a width of 0.5 μm or larger were detected and segmented. Myant et al. demonstrated an optical interferometer to measure the lubricant film thickness during a sliding motion on an elastic surface [25]. The lubricant film thickness is measured continuously as KPP in the range of 50 to 400 μm.

Since there are a large number of scientific papers in the literature using optical systems for geometric and dimensional inspection or surface detection, the authors refer to surveys from Catalucci et al., Mohan and Poobal, and Qi et al. which provide a good overview of existing methods and approaches in this field [19, 26, 27].

Although the literature shows that optical systems offer a great potential for inline QA, especially in production environments, their transfer to blanking processes is limited. To the authors’ knowledge, there are only two papers in the literature that address optical image analysis for QA during blanking. In this context, Molitor et al. proposed a CNN to classify 16 wear states during blanking. The authors acquired offline 7440 images of the parts to train the network to classify the wear state with 99% accuracy [28]. Since there is no direct extraction of KPP to determine the product quality of the blanked workpiece, and the image acquisition is done offline, the presented work is not suitable for inline QA of the blanked surface. Another promising approach for the direct, inline quantification of the quality of blanked workpieces in the form of the blanked surface is presented by Lorenz et al. [29]. The authors developed a system consisting of a camera with a CMOS sensor and a laser as a light source to measure the blanked surface on a sheet metal strip. However, the work is limited to the detection of the blanked surface on the sheet metal strip, so that a detection of the quality of the blanked workpieces is not possible. In addition, the hardware of the optical system takes up a lot of space, and the combination of camera and laser results in high calibration effort and cost.

In summary, the detection of quality features of blanked workpieces, especially the blanked surface, is not considered in the literature. Although the KPPs of the blanked surface is crucial for the selection between a good product and a scrap product, there are no methods and applications that allow an inline determination even at high production rates. In Section 1, the authors show that the four challenges installation space, blanked surface, system dynamics, and cost are the main reasons for the lack of implementation in the industrial environment.

3 Materials and methods

In this section, the experimental setup of the blanking process and image data acquisition is described. In addition, the methodology considering the steps of preprocessing and transformation to extract the KPP is explained in detail. According to Kubik et al., the methodology proposed in this study is related to the Knowledge Discovery in Time series and Image data for Engineering Application (KDT-EA) process model [30]. This provides a systematic approach for knowledge discovery in process data (image data or time series) based on the phases of data acquisition, preparation, transformation, and modeling as well as a final validation step. Since the KPPs are used to classify wear in Section 5, the influence of a worn tool train on the blanked surface is investigated.

3.1 Blanking process

The blanking process is characterized by a vertical movement during which the tool penetrates the die and defines the final geometrical and functional characteristics of the workpiece [31]. During the operation, the workpiece is separated from the sheet metal strip without removing any material. According to Hohmann et al., the blanking process is divided into three phases [32]. During the punch phase (I), the tool hits the sheet and begins to deform elastically. As the tool continues to penetrate, the material is plastically deformed until the maximum shear stress is reached. At this point, the material breaks, as indicated by an abrupt drop in process force. In the following push phase (II), the tool pushes the workpiece out of the die and reaches its bottom dead center. In the withdraw phase (III), the backstroke movement of the ram pulls the tool out of the material, and the sign of the process force reverses due to clamping effects.

In addition to the three phases, the process is characterized by the properties of the blanked workpiece. According to Lange, these properties are quantified by form, dimensional, and positional errors. While dimensional and positional errors are caused by geometric deviations of the tool or an offset between the punch and die, form errors occur during the plastic deformation of the workpiece. In this context, form errors describe the quality features of the blanked surface as a function of tool (clearance, wear, cutting line, etc.), semi-finished product (sheet thickness, tensile strength, alloy composition, etc.), or process parameters (temperature, lubrication, stroke speed). According to the German Institute for Standardization (DIN EN 10140), these form errors are divided into the rollover zone he, the shear zone hl, the rupture zone hf, and the burr height hb [33]. During the last decades, many experimental and empirical studies have been carried out to investigate the main parameters of the blanking process that influence the evaluation of these form errors. In particular, it has been shown that a change in the tool contour has a significant effect on the formation of the blanked surface [34, 35]. For example, Hernández et al. demonstrated that increasing abrasive wear leads to a modification of the blanked surface of blanked workpieces [36, 37]. They found that a higher wear volume leads to burr formation and an increase in the rollover zone. A similar correlation was found by Hambli, who showed a strong dependence of the cutting edge radii on the burr formation in experimental investigations [15]. However, since the literature only examines the influence of wear on individual features of the blanked surface and the maximum cutting edge radii investigated are limited to 0.2 mm, the dependence of all blanked surface KPPs (he, hl, hf, and hb) on the cutting edge radii up to 0.50 mm was determined in preliminary tests [37]. Kubik et al. demonstrated that the morphology of the blanked surface changes as a function of several parameters. Clearance, cutting edge radii, and tensile strength of the semi-finished product were identified as the main influencing factors. Other parameters, such as the temperature profile, the tribological system, the dynamic behavior of the die and press, or the varying properties of the semi-finished product, always have an indirect but small influence on one or more of the main factors.

The preliminary work has shown that the burr on the part increases significantly with increasing cutting edge radii. On the other hand, if the rollover zone remains constant and the percentage of the shear zone increases slightly, the percentage of the fracture zone will increase. As the clearance is reduced, the shear zone increases significantly while the rollover and fracture zones decrease. A similar trend is seen in the reduction of tensile strength. Here, the rollover zone decreases, but the fracture zone increases due to the high material hardness and associated brittleness, resulting in a low shear zone fraction. The main influencing factors identified are always the result of a change in the system of tool, material, and press. For example, wear not only results in a rounding of the cutting edge radii, but also in a change in the clearance due to adhesions or breakouts on the lateral surface. Similarly, dynamic effects such as increased stroke rates or insufficient rigidity of the press or tooling system can cause misalignment between the punch and die, which affects the clearance. Other parameters, such as temperature profile, lubrication, or tool material properties, have little or no effect on cutting edge development and can therefore be neglected. Figure 1 shows the appearance of the blanked surface for one of the three main influence factors, the cutting edge radii, for three abrasive wear states (R1= 0.1 mm, R2= 0.3 mm and R3= 0.5 mm).

Fig. 1
figure 1

Dependency of blanked surface feature on abrasive wear state, characterized by rounded cutting edge of the blanking tool

Full size image

3.2 Experimental setup

All experiments were conducted on a Bruderer high speed mechanical press (BSTA 810). The press parameters were set to a stroke distance of 51 mm and a stroke speed of 200 spm. The tool consisted of a 6-mm-diameter cylindrical punch and a 6.3-mm internal diameter die, resulting in a clearance of 0.15 mm. A cold-rolled 1.0347 steel with a thickness of 2 ± 0.05 mm, a tensile strength of 299 ± 2.90 MPa, and an elongation of break of 33.6 ± 1.22% was used. The blanking tool consists of a lower and an upper part, which are connected to each other by four columns. The cylindrical punch is connected to the adapter plate in the upper tool via a plunger. In conjunction with the movement of the ram, the punch penetrates the sheet metal (punch phase), and the workpiece is pushed out of the die (push phase). The workpieces then fall onto a collection chute, which transports them to a motorized wheel with pockets. A conveyor picks up the workpieces stroke by stroke and rotates them forward. At the top of the conveyor, two camera systems (camera, Basler AG—acA1600-60gm and lens, K.K. Ricoh—FL-CC2514A-2M) capture the image data orthogonal and at an angle of 45° to the blanked surface of the workpiece. Since the conveyor is moved in discrete steps, the images are taken during the short standstill period. At a stroke rate of 300 spm, considering the time for acceleration and deceleration of the conveyor, a standstill time of 0.05 s is obtained. Figure 2 shows the blanking tool and the optical system, which consists of cameras, lenses, a collection chute, and a conveyor. In addition to the ability to retrofit the optical system to capture workpiece images without high integration effort or capital cost, the use of commercially available components reduces the total cost of the optical system to less than $800.

Fig. 2
figure 2

Tool assembly for performing the blanking operation and the downstream optical system for capturing workpieces images at 300 spm in orthogonal and angular views

Full size image

It should be noted at this point that it is not possible to capture the cutting line over the entire circumference with the camera system shown. In the case of a cylindrical workpiece, a range of approximately 160° can be analyzed after subtracting the distortion around the center axis of the workpiece. Inhomogeneities of the cutting line due to local defects of the punch or die, as well as variations in the material properties, can therefore only be detected to a limited extent. This applies in particular to the examination of rotationally symmetrical workpieces that have more complex cutting lines (e.g., concave or convex) or undercuts. In this case, before the optical system is designed, it must be determined based on process knowledge where critical areas of the cutting line lie that must be monitored from the perspective of quality assurance. To obtain an overall view of the distribution of the cutting lines over the circumference of the component, it may be necessary to add further cameras with different viewing angles.

3.3 Image acquisition

In order to prove the suitability of the method to be developed for the extraction of KPPs from the blanked surface, images are acquired with the optical system shown in Fig. 2. The images (orthogonal and angular view) with a size of 1602 × 1202 pixels are transmitted through the ethernet communication protocol GigE Vision to a microcontroller (National Instruments CompactRIO 9047). Since a realistic wear development is to be simulated and its influence on the blanked surface is to be represented via the KPPs, seven artificial wear states of the cutting edge radii up to 0.6 mm are generated. By generating seven different wear states, with an average of 3781 blanked parts per state, a realistic wear development that would require more than 300,000 strokes is mapped to a total of 26,473 strokes. This results in a data set of angular images Xang ∈ ℝm×n×k (45° angle to the blanked surface) and orthogonal images Xorth ∈ ℝm×n×k (orthogonal to blanked surface) and corresponding labels r ∈ ℝ1×k, where m and n are the image size and k is the number of observations (captured image) per wear state. Table 1 summarizes the underlying labeled data set. At this point, the authors note that a critical wear condition is not uniformly defined, and literature suggests a value between 0.20 mm and 0.40 mm, depending on the material, tooling system, and quality requirements [38, 39]. Since the evaluation of a critical wear state must be determined individually by the company and depends on the tolerable deviations of the workpiece from the target specifications, a greater value of 0.60 mm is determined in this work. This ensures that the developed method covers a wide range of possible deviations and is therefore applicable in practice for most blanking process operators.

Table 1 Overview of the labeled dataset consisting of angle and orthogonal images of blanked workpieces and the wear state represented by the cutting edge radii
Full size table

3.4 Image preprocessing

Based on the image acquisition, each wear state provides ki image pairs with a size of 1602 × 1202 pixels, depending on the wear state (see Table 1). To keep the amount of data to be processed as small as possible for the subsequent preprocessing and transformation steps, the region of interest is cropped in a first operation. This reduces the image area to be analyzed, resulting in \({X}_{i,\textrm{ang}}\in {\mathbb{R}}^{404\times 899\times {k}_i}\) and \({X}_{i,\textrm{orth}}\in {\mathbb{R}}^{600\times 900\times {k}_i}\). Filtering and contrast adjustments are neglected in this study because they distort the information value of the images and do not improve the performance of the ML model. While in the first step, a preprocessing is performed to remove images that are not suitable for further evaluation, in the second step, the KPPs are extracted by transforming and parameterizing the images. During the preprocessing, in order to quantify the insufficient images that are misplaced in the conveyor pocket, a normalized correlation coefficient between a reference image t and a workpiece image f is determined [40]. In Eq. (1), f(x, y) denotes the intensity value of the image with size Mx × My at the point (x, y) x ∈ {0, …Mx − 1}, y ∈ {0, …My − 1} and t the pattern of the reference image with size Nx × Ny. The normalized cross correlation γ is then calculated at each point (u, v) for f and the reference image t.

$$\gamma \left(u,v\right)=\frac{\sum_{x,y}\left(f\left(x,y\right)-{\overline{f}}_{u,v}\right)\left(t\left(x-u,y-v\right)-\overline{t}\right)}{\sqrt{\sum_{x,y}{\left(f\left(x,y\right)-{\overline{f}}_{u,v}\right)}^2{\sum}_{x,y}{\left(t\left(x-u,y-v\right)-\overline{t}\right)}^2}\ }$$
(1)
$${\overline{f}}_{u,v}=\frac{1}{N_{\textrm{x}}{N}_{\textrm{y}}}\sum_{x=u}^{u+{N}_{\textrm{x}}-1}\sum_{y=v}^{v+{N}_y-1}f\left(x,y\right)$$
(2)

In Eq. (2), \({\overline{f}}_{u,v}\) denotes the mean value of f(x, y) within the area of the reference image t shifted to (u, v), and \(\overline{t}\) is the mean value of the reference image. The maximum value of γ shifted to (u, v) in orthogonal direction (γmax, orth) and in angular direction (γmax, ang) indicates how accurately a current image of the workpiece matches the reference. A strong match between the pattern of the reference and the current image is quantified by a high normalized correlation coefficient. By setting a critical value, misplaced, excessively tilted or missing workpieces in the conveyor pockets can be identified and sorted out. Figure 3a shows the categories of misplaced or missing images in the angle and orthogonal views, as well as the image representation of a reference workpiece t. In the angular direction, an additional distinction is made between the interference of the maximal normalized correlation coefficients right way up (RWU) \({\gamma}_{\max, \kern0.5em {\textrm{ang}}_{\textrm{RWU}}}\) and the maximal normalized correlation coefficients upside down (USD) way up \({\gamma}_{\max, \kern0.5em {\textrm{ang}}_{\textrm{USD}}}\).

Fig. 3
figure 3

Rejection of misplaced workpieces depending on the maximum normalized correlation coefficient γmax

Full size image

Figure 3b shows the distribution of the maximum normalized correlation coefficient for each of the seven wear states, where each point represents a part image. To remove the erroneous images, an empirical limit is defined by the following inequalities (3) and (4), which must be satisfied for the angular and orthogonal views.

angular view

$${\gamma}_{\max, \kern0.5em {\textrm{ang}}_{\textrm{rwu}}}>0.65\ \textrm{and}\ {\upgamma}_{\max, \kern0.5em {\textrm{ang}}_{\textrm{rwu}}}<\frac{5}{16}+\frac{5}{8}\bullet {\gamma}_{\max, \kern0.5em {\textrm{ang}}_{\textrm{usd}}}$$
(3)

perpendicular view

$${\gamma}_{\max, \kern0.5em {\textrm{orth}}_{\textrm{rwu}}}>0.75$$
(4)

The empirical bounds given to the inequalities depend on the process configuration and must be individually defined for transfer to similar configurations. When selecting the bounds, care must be taken to set them conservatively (false negative classification) so that all detected false workpiece images are eliminated. Table 2 shows that, especially at high states of wear, almost 80% of the captured images are rejected, which is due to the strong irregularity in the burr area and the excessive rollover zone. On average, however, nearly 30% of the workpiece images are captured correctly, resulting in an analyzable image pair. In other words, every third workpiece is subjected to QA, which results in a QA interval of 0.6 s at a stroke speed of 300 spm.

Table 2 Identification and elimination of misplaced images
Full size table

3.5 Image transformation

According to the KDT-EA process model, data preprocessing is followed by a transformation of the image data, in which relevant quality features of the blanked surface (KPP) are extracted from the workpiece images. The transformation identifies distinctive lines on the blanked surface from which the KPP are extracted. The transformation consists of three steps: segmentation, contour finding, and parameterization.

Segmentation: During the segmentation step, the workpiece image is divided into different regions using the Random Walker algorithm. According to Fig. 1, the blanked surface for the orthogonal view is segmented in two regions (area I – he and hl; area II – hf and hb), while the angular view is segmented in three areas (area I – he and hl; area II – hf; area III – slug underside). To apply the Random Walker algorithm, the areas to be segmented must be approximately predefined by geometric masks as shown in Fig. 4b. According to Wechsler and Kidode, the algorithm assigns undefined pixels to the predefined masks according to the highest probability by concatenating random movements [41]. After the segmentation, each pixel is assigned to a region as shown in Fig. 4c. Contour finding: After the segmentation step, the contour between each segment is described by coordinates that quantify distinctive lines. Therefore, a contour-finding algorithm based on the two-dimensional Marching-Squares algorithm is applied [42]. Here, the image is divided into a uniform grid, where each pixel is located at the intersections of the resulting grid. The algorithm moves through this grid and decides how the contour line crosses each square of the grid based on four possible configurations. As a result, the segmented regions of the Random Walker are separated by distinctive lines (see Fig. 4d).

Fig. 4
figure 4

Random Walker algorithm combined with Marching-Squares algorithm for contour finding on a blanked surface (a) by defining a geometric mask (b) followed by segmentation of regions (c), determination of distinctive lines (d), and parameterization of the contour line (e)

Full size image

Parameterization: In the final step, the level-set method is used to parameterize the distinctive lines and to find a function that describes the shape of the segmented regions in the orthogonal and angular views. The level-set function describes the contour of a vertical section at the height z in a two-dimensional space, so that the final contour of the segmented region is described by the isoline z = 0. In this study, the level-set function f(px, py) describes the contour of the segmented workpiece image as a function of the pixel position px ∈ {0, …, Nx − 1} and py ∈ {0, …, Ny − 1}. Once the distinctive lines are interpolated and uniquely described by f(px, py), the KPPs can be extracted from the image. Since the level-set function is defined by the two variables px and py, a conversion from pixels to millimeters is required. Preliminary tests for this work have shown that the diameter of the blanked part remains constant at a value of 6288 ± 4.81 μm regardless of the prevailing wear state, so a constant conversion factor of 14.15 μm/pixel was used.

As a result of the transformation process, seven KPPs for the orthogonal view and nine KPPs for the angular view are available in absolute lengths in millimeters. Figure 5 summarizes the KPPs for both views. In the angular view, the KPPs are described by the height of the shear zone \({h}_{\textrm{s}}^{\textrm{ANG}}\), height of the shear fracture zone and burr, \({h}_{\textrm{f},\textrm{b}}^{\textrm{ANG}}\), diameter of the workpiece dANG, offset of the ellipse determines fracture zone and burr \({s}_{\textrm{f},\textrm{b}}^{\textrm{ANG}}\), offset of the ellipse determines shear zone \({s}_{\textrm{s}}^{\textrm{ANG}}\), small axis of the ellipse determines shear zone \({e}_{\textrm{s}}^{\textrm{ANG}}\), small axis of the ellipse determines fracture zone \({e}_{\textrm{f}}^{\textrm{ANG}}\), small axis of the ellipse determines the burr \({e}_{\textrm{b}}^{\textrm{ANG}}\), and big axis of the ellipse determines the burr \({E}_{\textrm{b}}^{\textrm{ANG}}\). In the orthogonal view, the KPPs are described by the height of the rollover zone \({h}_{\textrm{e}}^{\textrm{ORTH}}\), height of the shear zone \({h}_{\textrm{s}}^{\textrm{ORTH}}\), the sum of the height of fracture zone and burr \({h}_{\textrm{b},\textrm{f}}^{\textrm{ORTH}}\), total height of the workpiece hORTH, height of the width of the rollover zone \({w}_{\textrm{e}}^{\textrm{ORTH}}\), diameter of the workpiece dORTH, and the fracture angle α.

Fig. 5
figure 5

KPP extracted from the workpiece images in angular (a) and orthogonal (b) view

Full size image

To ensure real-time extraction of KPP at high stroke rates, the computational effort for preprocessing and transformation must be estimated. Table 3 summarizes the absolute computation time. With the used computer resources (type, Intel Core i7 4770; used cores, 4; internal clock, 3.9 GHz; physical RAM, 16 GB; and average load, 80%), it takes 0.536 s to analyze a single pair of workpiece images and extract the 16 KPPs from both views. Assuming a stroke rate of 300 spm and a measurement interval of 120°, in which images must be captured and stored, this results in an absolute time of 0.15 s per stroke for preprocessing and transformation, which enables QA of every 4th part. To further increase the real-time capability of the system, the absolute time required to extract the KPPs from the blanked surface images can be drastically reduced by increasing the computer resources or using FPGA-based frame grabber.

Table 3 Computational effort for extracting KPP from image acquired with the proposed optical system
Full size table

4 Suitability of KPP to quantify the quality of a blanked surface

To validate the accuracy of the optical system combined with the presented method, the extracted KPPs from the orthogonal view are compared with the same features obtained from microscope images. For microscopic measurement, microsections of the workpieces are made. The measurement is then performed by a confocal white microscope μsurf (nanofocus), which serves as the ground truth. According to the DIN EN 10140 (see Fig. 1), the rollover zone \({h}_{\textrm{e}}^{\textrm{ORTH}}\), the height of shear zone \({h}_{\textrm{s}}^{\textrm{ORTH}}\), the height of fracture zone \({h}_{\textrm{f}}^{\textrm{ORTH}}\), and height of the burr \({h}_{\textrm{b}}^{\textrm{ORTH}}\) are used to quantify the quality of a blanked surface. The burr height is calculated from the difference between the initial height of the workpiece (2 mm), which is assumed to be constant, and the total height hORTH.

Figure 6 shows the four directly extracted features from the orthogonal views determined by the confocal microscope, colored in black, and the four KPPs extracted from the blackened surfaces with the developed system, colored in red. For the heights of the shear zone and the fracture zone, the average deviation from the ground truth is \(\Delta {h}_{\textrm{s}}^{\textrm{ORTH}}\) = 8.59 μm for the shear zone and \(\Delta {h}_{\textrm{f}}^{\textrm{ORTH}}\) = 25.28 μm for the fracture zone. Due to the rough boundary between these two zones, the Random Walker algorithm detects a parting line, but it varies greatly around the perimeter of the blanked workpiece. In addition to this fluctuating transition between the shear and fracture zones, an asymmetric clearance influences the shape of the parting line. Such asymmetric clearance is related to poor alignment of the punch to the die or adhesions on the lateral surface of the tool. For comparison, Fig. 7a shows the parting line for the wear state r2 = 0.1 mm for a blanked workpiece with symmetrical (left) and asymmetrical (right) circumferences. In this case, the variation in the parting line is caused by temporary adhesions on the lateral surface of the blanking tool. As a result of the narrowed clearance at the point of adhesion, compressive and tensile stresses are superimposed, causing the material to flow longer, and creating a locally pronounced shear zone. For the rollover zone, an average deviation from the ground truth of \(\Delta {h}_{\textrm{e}}^{\textrm{ORTH}}\) = 13.63 μm is achieved. Due to the low contrast between the shear zone and the rollover zone, the boundary between these two zones is difficult for the algorithm to recognize, resulting in an increased variation. Figure 7b shows this phenomenon, which is mainly caused by the lighting scene and thus related to the illumination design of the optical system.

Fig. 6
figure 6

Comparison between the KPPs of the blanked surface extracted with the optical system combined with the presented method (red line) and directly extracted with the confocal microscope μSurf (black line)

Full size image
Fig. 7
figure 7

Effects that lead to the deviation between the KPP directly derived from the workpiece images and determined by the reflected light microscope (reference)

Full size image

In the literature, burr height is defined as a critical parameter for the customer. Excessive increase in burr height results in handling with risk of injury and inability to achieve functional and geometric target requirements. During the tests, an average deviation from the ground truth of \(\Delta {h}_b^{\textrm{ORTH}}\) = 26.74 μm is obtained. The deviation is caused by a jagged edge of the burr and hinders the detection of a sharp end contour. This effect is particularly pronounced as the wear on the tool cutting edge increases. Figure 7c shows the formation of the end contour of the burr for the wear states r2 = 0.1 mm and r6 = 0.5 mm. Up to a wear state of r6 = 0.50 mm, the average deviation from the ground truth is only 20.44 μm or 1% of the original sheet thickness. In practice, a value of 10% burr, relative to the initial sheet thickness, is often declared critical, so that with the results obtained, a burr is detected well before the rejection criterion occurs. As already explained in the objective of this work, the quantification of the KPPs of the cutting surface is not limited to the base area of the workpiece (dimension of the workpiece in the x-y plane), but only to the thickness of the initial semi-finished product. In the case of the optical system proposed in this study, a value of ± 20.44 μm is obtained with a probability of 96%, by which the measured burr height varies. This means that with a maximum tolerable burr height of 10% of the sheet thickness, the minimum sheet thickness of the blanked components to be monitored must be greater than 0.25 mm.

At this stage, it should already be pointed out that the definition of standard limits for each wear state at which a quantification of the blanked surface by the optical system is still considered trustworthy must be carried out for the individual KPP on a company-specific and product-dependent basis. Since the optical system is only designed to describe the characteristics of the blanked surface, fluctuations in the KPPs can be detected and quantified, but the cause of such a fluctuation cannot be distinguished.

5 Wear state detection using ML models

As shown in Section 4, the KPPs extracted from the blanked surface can be used to directly describe the quality of the workpiece and thus for QA. Since there is a relationship between the KPPs and the current wear state, as shown in Fig. 1 and Fig. 6, the KPPs are used to estimate the rounding at the cutting edge. As the wear state depends on many parameters and there is a non-linear function between the input variable cutting edge radii r and the KPPs of the blanked surface, ML models are used for the estimation. At this point, reference should be made to previous work where the development procedure for ML models for cutting radii estimation is discussed in detail [43]. Since the use of ML models in this paper is only used as a validation method for the quality of the extracted KPPs, the authors refer to these studies for further information. To find the most appropriate model for this use case, a two-step model optimization is performed. In the first step, nine models, including a random forest (RF), linear discriminant analysis (LDA), naive Bayes (NB), linear SVM (lSVM), quadratic SVM (qSVM), cubic SVM (cSVM), k-nearest neighbor (KNN), a one-layer neural network (1NN), and a three-layer neural network (3NN), are trained based on their default parameter configuration. The reference configuration of the selected ML models in the Python library Scikit-Learn can be found in Table 4.

Table 4 Reference configuration of the selected ML models based on the Scikit-Learn environment during the first step of model optimization
Full size table

In a second step, the best fitting model is optimized by hyperparameter optimization. All models are trained on a labeled data set, where the input is defined by the 16 KPPs (see Fig. 5) and the wear state is quantified by the cutting edge radii. Figure 8 demonstrates the accuracy achieved by a fivefold cross-validation, where the wear state is best estimated by a qSVM with 91.82 ± 0.11%. By further hyperparameter optimization, the classification accuracy for the optimized qSVM (qSVM-opt) is increased to 92.79% ± 0.14%.

Fig. 8
figure 8

Classification results obtained for each tested ML model

Full size image

If we take a closer look at the performance of qSVM-opt using the confusion matrix of the classification shown in Fig. 9, we can see that the error is mainly due to the misclassification of neighboring wear states, and an accumulation of misclassifications is found along the diagonal of the matrix. Thus, misclassified states are only one class away from the target class. Especially for small cutting radii, the classification task is challenging. For low wear states, the differences between the KPPs are small. In particular, the formation of a burr, which is clearly visible with increasing cutting edge radii, is an important way of describing the wear state. This trend can also be seen in the comparison between the KPPs and the ground truth in Fig. 6.

Fig. 9
figure 9

Cumulative confusion matrix for wear state classification of qSVM-opt quantifying the number of misclassifications

Full size image

6 Conclusions and outlook

Since the inline extraction of quality parameters blanking, represented by features of the blanked surface, is not yet used in practice due to four requirements: installation space, blanking surface, system dynamics, and cost, this work proposes an optical system in combination with a novel analysis methodology to fill this gap. The system allows real-time quality assurance of blanked parts up to 300 spm, although in this work the analysis is limited to every 4th workpiece due to the available computing resources. During the development of the optical system, the authors took care to use commercially available, low-cost components and to keep the total cost of the system below $800. For the analysis of the acquired image data, the applied methodology consists of the steps of preprocessing and transformation, where the transformation is divided into the steps of segmentation, contour finding and parameterization. The quality of the KPPs extracted by this method is validated by two use cases. In the first case, the extracted KPPs are compared with a ground truth obtained by a confocal microscope. Here, deviations from the ground truth of \(\Delta {h}_{\textrm{e}}^{\textrm{ORTH}}\) = 13.63 μm from the height of the rollover zone, \(\Delta {h}_{\textrm{s}}^{\textrm{ORTH}}\) = 8.59 μm from the height of the shear zone, \(\Delta {h}_{\textrm{f}}^{\textrm{ORTH}}\) = 25.28 μm from the height of the fracture zone, and \(\Delta {h}_{\textrm{b}}^{\textrm{ORTH}}\) = 26.74 μm from the height were achieved. In the second case, several ML models were built based on the extracted KPPs to classify the wear state of the blanking tool. Using an optimized SVM (qSVM-opt), the seven cutting edge radii were assigned to the correct wear state on the tool with an accuracy of 92.80%. In addition to this accurate estimation, the model accumulated almost all observations around the diagonal of the confusion matrix. Thus 92.80% are found on the main diagonal (correctly classified), 7.11% are directly above or below the main diagonal (one class next to the target class), and only 0.09% are further away from the diagonal.

To also obtain a conclusion about the cause of a fluctuation in the KPP in the future and to relate the characteristics of the blanked surface to the current process state, it is necessary to vary targeted parameters such as stroke speed, clearance, cutting line geometry, or semi-finished product properties. By using advanced segmentation methods, a detailed description of the cutting surface is possible, so that the system will not only be able to quantify the properties of the cutting surface, but also to describe its origin.