Optical inline monitoring of the burnish surface in the punching process

Punching is a wide-spread production process, applied when massive amounts of the ever-same cheap parts are needed. The punching process is sensitive to a multitude of parameters. Unfortunately, the precise dependencies are often unknown. A prerequisite for optimal, reproducible and transparent process alignment is the knowledge of how exactly parameters influence the quality of a punching part, which in turn requires a quantitative description of the quality of a part. We developed an optical inline monitoring system, which consists of a combined imaging and triangulation sensor as well as subsequent image processing. We show that it is possible to capture images of the cutting surface for every part within production. We automatically derive quality parameters using the example of the burnish height from 2D images. In addition, the 3D parameters are calculated and verified from the triangulation images. As an application, we show that the status of tool wear can be inferred by monitoring the burnish height, with immediate consequences for predictive maintenance. Although limited by slow images processing in our prototype, we conclude that connecting machine and process parameters with quality metrics in real time for every single part enables data-driven process modelling and ultimately the implementation of intelligent punching machines.

Attempts to optimize the quality of punching parts with respect to cutting surface parameters are currently based exclusively on expert knowledge and the trial-anderror method. This is due to the fact that there are many factors influencing the process in unknown ways; examples are shown in Fig. 2. Moreover, the wide variety of machine-, tool-, material-and process-parameters and their mutual dependencies render it very difficult to compare the results of different process studies [5,8,9]. These studies are often carried out in a controlled environment and therefore exclude fluctuation of these parameters within their tolerances paired with a degrading process over its lifetime. As summarized by [3,10,11], the process is still not fully captured by any comprehensive process model. Different experiments with different parameters show either an increase or a decrease in the cutting edge/surface parameters and thus come to contradictory conclusions; the high-dimensional parameter space renders comparison and interpretation of the results very difficult. Facing shorter product life cycles and customer-oriented serial production, it is important to improve the manufacturing process by an automated quality monitoring [3,[12][13][14].
In metal forming the monitoring of the product quality has been used to measure and improve the process. Purr  [6,7]. (a) Terming of a punching process. (b) Cutting surface parameters of the sheet metal et al. [15] showed the need for a quality analysis within a stamping plant of car body parts. Maier et al. [16] showed the benefits of the integration of a measuring system within the process and compared them to offline measurement system. Both studies also state that inline monitoring of quality metrics is important for a data mining approach or a controlled process. Other examples can be found for additive manufacturing [17].
Numerous studies have investigated the influence of parameters within the punching process (e.g. acceleration [18], force [19], acoustic emission [20]). In addition, different approaches for expert systems [21], feature engineering approaches [5,11,12,22] or by artificial neural networks [23,24] were made to predict the process quality. To our best knowledge, the current state-of-the-art of inline measurement systems for quality monitoring of the cutting surface parameters is only established via the correlation with indirect measured parameters or engineered features. However, these investigations are often verified by a small number of specimens. In terms of the large parameter space, a continuous quality measurement of the cutting surface is required.
Current methods for cutting surface analysis always require samples to be taken and analysed separately from the process. This is a crucial drawback for an industry which faces even shorter development processes with increased quality requirements as well as customized and applicationindividualized products. Therefore, an economic process design using empirical knowledge or the trial-and-error method is not possible [14].
Cutting surface measurement For the cutting surface analysis, different measurement methods are used. Metallography microsection and contour gauges are the oldest, which suffer from the inaccuracy of the subjective reading of the values and from the tactile procedure. Behrens et al. [4] developed a device with a higher degree of automation for a measurement based on laser triangulation and a curvematching-algorithm. Although the part must be prepared for the measurement the cutting surface parameters are detected automatically by the algorithm. For quality control of coils during manufacturing, a chromatic confocal sensor is provided by Burghardt+Schmidt GmbH (Germany). Only the rollover, burr height and fracture angle can be measured. All these measuring methods only deliver partial or not reproducible results, require significant amount of time and only measure one profile section. This is a drawback because of the fluctuating cutting surface parameters over the surface. The transition from one cutting surface parameter to another is not smooth and constant rather than depending on, e.g. material variations. If a larger area of the surface has to be observed often, a confocal microscope is used. While the high accuracy is an advantage, confocal microscopy is a time-consuming measurement because the surface has to be scanned several times; in addition, the microscope is a high price instrument. Furthermore, a survey with experts from different companies showed that there is no common standard for cutting surface analysis and documentation. Rather the metrics are defined with each customer separately. This leads to time consuming negotiations with every customer and to many different approaches for cutting surface analysis even within one company.
The goal of our contribution is twofold. First, we want to provide a monitoring system that is capable of measuring the shape and the texture of the cutting surface parameters of produced parts, thus providing clear and reproducible metrics-including the variability along the surface-to derive and document the quality of the parts. This enables manufacturers to provide a 100 % quality inspection and customers to standardize the quality requirements. Second, an inline monitoring system for cutting surface properties provides the chance to link machine parameters to quality metrics, as described by [3]. As an example, we show that the punch condition is reflected in the burnish height over time, which can be used to determine the ideal point in time for predictive maintenance. Selection of process parameters that are most likely to influence the cutting surface parameters [22,25]

Optical monitoring system
Monitoring by an optical system seems to be particularly suitable for this task. Optical methods ensure high processing speeds and through combination with image processing a high degree of automation and reproducibility of measurement results. Furthermore, all optical measuring methods combine the properties of high accuracy and non-contact inspection. The developed system captures data of cutting surface within the process for each part. It combines an image capturing and a triangulation method. Both share the same CMOS-sensor and a telecentric lens (see Fig. 4). This expands the analysis into a surface view rather than only a (traditional) profile section of the cutting surface. For the development, a rectangular punch with dimensions 5 × 5 mm 2 and a coil material with a minimum thickness of 0.5 mm was used (see Fig. 3). With respect to the accuracy of the measured cutting surface parameters, we aim for an accuracy of ±5μm for all linear measures, which corresponds to ±1% of the coil thickness, and ±1 • for the angular measure. For the production, a Bihler GRM-NC was used with a maximum production speed of 250 strokes/minute. For more process parameters, see Table 1.
The system was placed close to the tool outlet.

Image capturing
The image capturing provides 2D images of the cutting surface and is used to determine most of the linear measures such as the burnish height. The required accuracy of the linear measures of 5 μm translates to a resolution of 200 line pairs/mm at cutting surface level. To meet this requirement, we chose a telecentric lens with a magnification factor of M = 2 and a modulation transfer function (MTF) contrast of approximately 35 % at the desired image resolution. According to Nyquist sampling theorem, the image sampling resolution at sensor level needs to be 2.5 μm or less such that we decided for a sensor with a pixel pitch of p = 4.8 μm.
The closed contour of the punching geometry does not allow a head-on setup. The optical axis of the lens can not be set orthogonal to the cutting surface. The whole system has to be tilted downward with respect to the horizontal plane (see Fig. 4). This in turn leads to a perspective

Triangulation
The triangulation method is capable of measuring the 3D topography of the cutting surface and is used to determine, e.g. the fracture angle β, burr height. Available laser triangulation sensors can not be used due to space constraints. Hence, we utilize the image capturing sensor for our triangulation by adding a laser line illumination. In this triangulation configuration (see Fig. 4) the laser line illumination is orientated such that: -the triangulation angle ϕ between the laser axis and the optical axis is set to ϕ ≈ 30 • -the laser line is orientated along the y M -direction -the laser line is imaged close to the center of the CMOSsensor in x -direction The sensor (2D) data of projected laser line across the cutting surface is transformed into the x M , y M , z M -coordinate system with the triangulation angle ϕ. The 3D-Data is extracted by projection this data the y M , z M -plane.

Components of the monitoring system
The monitoring system (see Fig. 5a) consists of a telecentric lens with a magnitude M = 2 (f = 75 mm, NA = 0.164). An industrial camera (CMOS-sensor) with a pixel pitch of p = 4.8 μm and a resolution of 1280 x 1024 Pixels (Field of view: 3.072 mm × 2.457 mm) and a capturing rate of 170 frames per second (fps) and a laser module (Z-LASER Z30M18H3-F-450-lp45) with a wavelength of 470 nm which projects a line. The laser module comes with an adjustable focus (f = 100 mm, 100 kHz). All parts are mounted onto a baseplate. To adjust the laser line position in reference to the lens, an adjustable mounting is received. This allows to rotate the laser in his own axis, adjust the triangulation angle and shift the laser position along the image x-axis. The illumination is realized with two flashes (470 nm). It is crucial set up one flash above the part (see Fig. 5b). The angle between the normal on the burnish surface and incoming light should be nearly the same as the angle between the normal and optical axis. This ensures that the burnish surface is brightly illuminated (see Fig. 5c). Because of the different roughness of the fracture surface and the different angle, this enables a distinction to be made. The second flash is set up in such a way that the reflected light of the lower side of the part is reflected into the lens. The laser and the flashes are powered with 24 V and the output is controlled with a potentiometer. Both devices are connected to the camera for process control. The desired workflow is show in Fig. 6.

Calibration of the triangulation setup
In order to obtain precise 3D data, the triangulation angle ϕ must be calculated through calibration. For this end, a calibration target with four different height levels is used (see Fig. 7b). A confocal microscope (Keyence Corporation VK-X-100) was used to quantify the height levels z c of the target in high precision. For the calibration procedure the monitoring system is placed in front of a motorized stage. The calibration target is mounted onto the stage (see Fig. 7a). It is ensured that the optical axis of the imaging system and the surface normal of the calibration target coincide. The laser is aligned such, that the laser line projected on a flat surface illuminates a single pixel column on the CMOS-sensor, only. The laser was projected onto the target and captured with the CMOS-sensor. In this data the different height levels of the calibration target show up as four line segments. Each segment shifted to each other in x-direction (see Fig. 7c). To calculate ϕ, the differences in x-direction between the line segments have to be known. With a center of mass(COM) calculation the average x-positions for each line segment and the differences Δx between the line segments were determined. With the known physical height level from the microscope z c the triangulation angle ϕ is calculated.

Influence of the tilting angle γ
As stated above, the whole monitoring system has to be tilted around the x W -axis by γ = 20 • . As a side effect, the laser line illumination, which is set during the calibration process, is no longer orientated along the y W -direction. Instead, the laser line is titled as well. On a flat cutting surface orientated parallel to the x W , y W -plane, the laser line will be rotated around the z W -axis by γ = 20 • . This has to be kept in mind, in order to correctly calculate the fracture angle β of the measured triangulation data along the projected laser line.
After calibration, the monitoring system was mounted onto a Bihler GRM-NC (see Fig. 5b). Special care was taken to ensure that the optical monitoring system can be removed for the calibration process.

Image processing for image capturing
For image processing, the OpenCV package for Python was used. This is beneficial to create prototype applications rapidly. The captured images showed a bright stripe which is identified as the burnish surface (see Fig. 5c). Our desired task was a segmentation process which delivers the edges of the burnish surface. From that edges the burnish height was calculated. Edge detection operations such as Sobel filters were not applicable. Since a connected segmentation line along the transition between burnish and fracture surface could not be ensured due to the inhomogeneous gray values inside the burnish surface. Active Contour methods enable it to distinguish despite this fluctuation and create a connected transition line even with missing data points based on weighting factors [26,27].
Active Contours or Snakes denote an iterative energy minimization procedure, which deform a curve influenced by energy contributions such as conformity with the underlying image or local curvature of the curve [28]. The curve energy is divided into parts. The inner energy of the curve consists of a point spacing and a curvature term controlling the smoothness and elasticity, where less stretched and bent curves are favoured. The external energy depends on the image features itself. The curve is described by a finite amount n of points. The minimization problem is solved iteratively. In each iteration step curve energy is calculated. A minimization is made by a gradient descent method. All points are moved along the gradient of the energy and the energy is calculated and compared to its predecessor. The process is stopped after a certain amount of iteration steps or after the energy change is smaller than a convergence threshold. Details are given in [28].
To control the elasticity and smoothness of the curve, the weights ζ , η can be set, respectively. Also, the number of points have to be considered as a trade-off between a good approximation of the curve and speed of segmentation.
In order to attract the curve towards the interface between burnish surface and fracture surface, a feature image is derived from the original image, and the external energy of the curve is calculated based on the feature image.
Feature image Since working with gradient decent, the purpose of the feature image (see Fig. 8a) is to create descending values towards the interface between burnish and fracture surface, with minima's at the unknown transition. It is constructed by first adding a background image to the original image. The background image values are increasing with the distance to the bright burnish surface. This new image can be seen as a 'ramp' with decreasing structure towards the burnish surface (see Fig. 8b,c). In addition, the gradient magnitude of the original image is subtracted from the new image. This decreases the intensity at the locations of the edges and ensures the minima to be at the transition.
Technically, the background image that ramps up towards the boundaries is found by a distance transform. We first determine the center of the burnish part by projecting the intensity along the x-axis and finding two y-coordinates such that all values with a projected intensity larger than 90% of the maximum intensity included between them (see Fig. 9a). The center coordinate is defined as the mean of those two y-coordinates. A distance transform then creates a ramp that is falling linearly towards the center of the burnish part. This image is called E ramp .
To add dependency to brightness the blurred Gaussian of the original image E Image is added. The normalized gradient image E grad which is obtain with a Sobel-Operation of the original image is subtracted. This is beneficial over the distance transform of the gradient because this adds no offset. This results in the profile show in Fig. 8b-c. The feature image concludes to: The factors κ, χ are for weighting the terms. Active Contours are strongly dependent on the initial contour and the ratio of the weights ζ, η, κ, χ. To start with a contour which is close to the segmentation is beneficial in terms of processing time. Therefore, an iterative process is carried out. A threshold value y height has to be set by the operator. This serves as an estimate for the burnish height. At first the original image is divided along the xaxis into area segments of the same length (see Fig. 9b). For each area, a cumulative intensity projection within that length is performed (see Fig. 9a). Again the y-coordinates with intensity level above a certain threshold are selected for each region. The maximum distance between these ycoordinates is calculated. With pixel pitch, magnification and tilting angle p, M, γ a compare value is calculated and compared to y height . If the compare value is below y height , these boundary points are selected. If the compare value is greater than y height , the threshold is increased until condition one is satisfied. With the boundary points of all areas two curves are calculated. One for the upper and one for the lower contour. Figure 10 shows the results of the image processing. The initial contour shown as blue lines and result of the segmentation shown as the red lines. The burnish height can be calculated with respect to pixel pitch, magnification and tilting angle p, M, γ .

Image processing for triangulation
The triangulation image (Fig. 11a) shows the laser line projected onto the surface and viewed in the coordinate system of the CMOS-sensor. The triangulation image is mainly dark and only the laser line appears bright. The objective is to calculate the laser peak with sub pixel precision. Therefore, the laser image is blurred with a gaussiankernel, and a region of interest (ROI) which only contains the data is defined by means of threshold. Background pixels are set to zero value. Since the laser diode sends a line with a certain thickness a center of mass algorithm can be performed (see Fig. 11b). A fast-performing algorithm is desired. Therefore, only a 5-point neighbourhood around the maximum is used to compute the position of the peak xδ for every row yδ. Details given in [29].
The estimated peak positions xδ, yδ are projected into the distance space with: where x w , y w , z w are the world-coordinates of the scanned surface points and r the travel distance between images.
Multiple images combine to a scan of the cutting surface where the 3D data (e.g. fracture angle β) can be measured.

Verification of the monitoring system
In order to verify the monitoring system several tests have been carried out. For the test of the 2D image processing, images were recorded within the machine cycle and processed. The process parameters can be seen in Table 1.
The test for the triangulation data took place offline on the motorized stage. A confocal microscope was used for ground truth data acquisition.

Verification of image processing
Three parts of stainless steel and copper each where manufactured and captured with the system (see Fig. 10). These parts were taken from the production process and measured with a confocal microscope. Because of lack of definition of the cutting surface parameters in the surface view there is no clear feature for comparison. Although existing human heuristics, a manual point-by-point comparison-between the microscope-and image processing-data-of the burnish heights is considered as a good approximation for the ground truth. For each part, three positions were tested. Here, particularly prominent places such as increases or decreases in the transition between burnish/fracture surface were selected. The settings for the image processing can be seen in Table 2. The initial contours were calculated from 20 image segments. The maximum allowed iterations set to 80 and the convergence limit was 0.0001. The results of the validation can be seen in Table 3. The data shows good precision with a spread from + 4.28 to −2.70% and mean of −0.68%. This is within the desired

Verification of triangulation data
To verify the triangulation method three cutting surfaces of stainless steel were picked. For a clear assessment of the characteristics of the cutting surfaces and to scan the identical area notches were made. The surfaces were recorded on the motorized stage. One scan consisted of 200 images with a step width in the x-axis of r = 5 μm between them. The notched surfaces were also measured with the microscope and the measured surface was exported. The images of the triangulation were processed, and a pointcloud was calculated. The processing of all 200 images took about 20 s. For the comparison of the data the surface from the microscope and the point-cloud were imported in the software CloudCompare. At first a manual alignment by picking point pairs was performed. Afterwards an automatic registration with the iterative closest point algorithm followed and the distance between surface and cloud was calculated. The same task was calculated with the free version of GOM Inspect for a cross check. Since there were many converting steps involved, differences can result from this. This can therefore only be seen as a qualitative verification.
The results of the three parts can be seen in Table 4. The visual output of the software for the surface to cloud distance for part 1 can be seen in Fig. 15, where subfigure (a) shows the picture from the microscope, subfigure (b) the exported surface from the microscope, subfigure (c) the alignment of point cloud and surface, and subfigure (d) the distribution the surface to cloud distance. The results for the other two parts were qualitatively and quantitatively similar. Both software products provided comparable results. The mean difference of the data generated by the image processing was always beneath 2 μm and their standard deviation was 10 μm. The notches were clearly visible in all data. The largest deviations are found at the strongly deformed burrs and the sharp surface changes in the area of the notches. They cause changes in reflection behaviour and thus create strong singularities. The microscope automatically corrects these behaviours whereas the sensor lacks this option. Therefore, these deviations can be neglected. Additionally, some geometries could not be scanned because they were occulted due to the triangulation angle. Therefore, delivered false or no data is also neglected. The results show that the desired accuracy of ±5 μm was not achieved but the similar result show high confidence in the data.

Capabilities of the monitoring system
Depending on the field of view of the lens and the intended cutting line geometry the system is capable of measuring the burnish height with sheet thickness from 0.5 to 1 mm and a length of 3 mm with an accuracy of ±5 μm. The depth of field is 80 μm which allows for a maximum fracture angle of β = 30 • . In addition the system is capable of creating 3D-Data with an accuracy of ±10 μm. In theory, the determining factor regarding image acquisition is the capturing rate of our imaging sensor, which is 170 frames per second. However, with our components the determining factor is the exposure time for the triangulation image.
With the studied material the exposure time was set to 6.7 ms, resulting in 150 fps. Thus, we have shown that data acquisition is easily possible within the cycle time (240 ms) of the punching machine. Regarding data storage the image sensor comes with a buffer. If the storage unit cannot store the images immediately, the current images are placed in a buffer and all images can be stored later.
Since the image processing with active contours took 40 to 60 s it was only possible to derive the burnish height offline. This implementation was not optimized for performance but only serve as a proof of concept. Real time performance could be achieved with optimization and parallelization of the code. Since the images are all independent of each other an increase in performance can be achieved with a multitasking approach. For this paper, a sequential algorithm with only one CPU was used. In addition an increase in performance is expected by changing the implementation from pure python to C++.

Case study: capabilities in production
The capabilities of this monitoring system can be seen with an interesting example. We captured 17.000 images during a material test where punch failure occurred. For process parameters see Table 1. This is novel because it is the first continuous data of the cutting surface within a process. We analysed the cutting edge of the used punch with the confocal microscope for wear within given intervals (5.000, 10.000, 25.000 strokes). Figure 12 shows the condition of the punch recorded with the microscope. A fully functional After the test has been carried out, the captured images were processed for their burnish heights. The orientation of the punch in the cutting tool and the position of the failure is known. This position can be transferred to the image and appears on its left-end side. Figure 13 shows a selection of images from the monitoring system with the area of failure marked. Since there are images for every cutting surface, the point of failure can be localized more precisely by manually analysing them. The failure emerged around the 13.500 strokes.
To gather additional information about the evolution of the failure the burnish height of two regions of each image have been compared. These regions were the known failure region (left-end of the image) and a fully functional region (right-end of the image). To consider the feed tolerance of the production machine of ±0.1 mm the burnish heights were averaged over a range of 0.2 mm in the xdirection. The result for the analysed cutting surface in the interval from 7.000 to 25.000 strokes can be seen in Fig. 14. Even though the cutting edge of the punch was considered fully functional at 10.000 strokes with the confocal microscope one can see that burnish heights begin to decline from the start. The burnish height on the left-end shows a rapid decrease and higher fluctuation whereas the burnish Based on that data, the evolution of the failure starts around 7.000 strokes. In comparison, the new system enables earlier detection of failure. The point of failure is also met around 13.500 strokes. The following reapproach (grey) to the right-end graph is misleading. The produced surface quality of a worn-out punch still creates portions of burnish surface which are detected by the algorithm. These portions are often shifted to the normal burnish part or not continuous and thereby do not meet the quality requirements or the definition of the cutting surface parameters on which the developed algorithm is based. In the sense of predictive maintenance this result can be used for better planning reliability and following processes. A faster image processing would enable to view the graph during production.

Conclusion
Measuring meaningful metrics of the cutting surface of punching parts is key for process optimization and quality management but is still not implemented in commercial punching machines. We developed a combined 2D/3D optical monitoring system with corresponding image processing of punching parts for a thickness range of 0.5 to 1 mm and integrated it into a commercial punching machine. Unlike state-of-the-art methods that rely on the extraction of parts, this system captures data from the cutting surface within the process cycle at 250 strokes per minute. Although only evaluated offline at the moment, the image processing allows for an automatic and accurate measurement of the burnish height of the cutting surface for every stroke. Thereby, it is now possible to monitor the quality of produced parts over the tool lifetime. Within a case study, we provide first data that show how such a system can be used in the context of predictive maintenance. Our case study focuses on the evaluation of the burnish height with data that was collected within the process cycle. Although only evaluated offline at the moment, in theory, it is possible for the image processing to be accomplished close to process cycle time.
Future research will focus on the collection of production data and image processing within the process-cycle. Especially an automated processing of the 3D data to measure the burr height and fracture angle over the tool life has to be implemented. In terms of predictive maintenance, the increase and decrease of the cutting surface parameters over the lifetime of the punches will be investigated. With more data a correlation between process parameters and quality of the product and wear could be established. This could lead to a lifetime prediction of the punch/tool. Furthermore, this monitoring system is an important step towards data-driven modelling of the punching process. An accurate model relies heavily on the process output, which can now be captured for the first time with our system. In combination with a multisensor network, this could finally lead to an increased process understanding and a highly efficient, self-regulating punching process.
Funding Open Access funding enabled and organized by Projekt DEAL. The authors gratefully acknowledge financial support from the interdisciplinary technology network Efficient Production Technology (EffPro), co-funded by the European Union and the Free State of Bavaria from the European Fund for regional development (ERDF) and supported by the University of Applied Sciences Kempten and the Bavarian Ministry of Science and Art.

Availability of data and material
The data that support the findings of this study are available from the corresponding author on reasonable request.
Code availability For image processing, a custom code was implemented in Python. The Software GOM Inspect(Freeware) and Could-Compare(GNU) were used for 3D-Data comparison.

Conflict of interest The authors declare no competing interests.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.