An automated assessment method for integrated circuit chip detachment from printed circuit board by multistep binarization and template matching of X-ray transmission images

Given the increase in digital product waste, demand for recycling of printed circuit boards (PCBs) is increasing. Precious and minor metals are often well concentrated in integrated circuit (IC) chips, especially in PCBs; hence, IC chips are primary targets for recycling. The technology for the non-destructive detachment of IC chips from PCBs is increasing in sophistication; however, the effectiveness of IC chip detachment is currently assessed manually and visually. In the present study, an automated IC chip detachment assessment method was developed, which combines multistep binarization and template matching of X-ray transmission images of crushed PCBs. To validate the method, five types of mechanically crushed PCBs from mobile phones were examined, and the developed method successfully assessed the IC chip detachment rate, with an average error rate of only 2.2% compared to visual assessment.

On PCB, the precious metals and minor metals are specifically concentrated in electric parts. For their effective recycling, then, the electric parts should be efficiently detached from the PCB and separated according to their type. A physical separation technique specifically targeted at electric parts has been developed, which focuses both on the material features (e.g., density, magnetism, and conductivity) and the structural features (e.g., size and shape) [18]. The aim is to detach the electric parts from the PCB without damaging them (non-destructive detachment). Parts detachment technology is increasing in sophistication, with chemical [19], mechanical [20], and underwater explosion-based [21] approaches being investigated. A mechanical approach was adopted in the present study, because its economy and rapidity well suit the demands of recycling plants. In the abovecited studies, parts detachment was manually assessed, which is sufficient for laboratory testing, but suffers from problems of time-consumption and human error. Therefore, the authors attempted to develop an automated assessment method. Among the various potential types of relevant electric parts and PCBs, the detachment of integrated circuit (IC) chips from mobile phone PCBs was targeted in this study, because it represents a typical case where the effective recycling of the precious and minor metals from rapidly increasing e-waste is required.
In assessing IC chip detachment, the key is to distinguish the IC chip components from the other PCB fragments. Various methods are available for identification of IC chips using a combination of visual camera and image analysis (e.g., [22]). However, in general, double-sided PCBs, where IC chips are mounted on both sides of the board, are used in mobile phones; thus, both sides of the PCB must be observed in such assessments. Furthermore, in the case of mechanical (or underwater explosion-based) detachment, the PCBs are typically crushed into small and irregularly shaped fragments, making it difficult to turn them over and visually observe both sides of the fragmented PCB. Thus, in this case, X-ray photography is a more useful observational technique than those utilizing visual cameras.
The IC chips are composed of various types of resin and metal, and their internal structures vary with the type and manufacturer. The same may be said for the boards themselves (which are the remnants of the complete PCB after IC chip detachment). Figure 1a shows X-ray transmission images of various IC chips and boards, with their varying internal structures. Figure 1b compares the frequencies of the gray-scale values of chips A-C, and boards D-F in Fig. 1a. It is clear that the frequencies vary greatly, even among the IC chips (and boards) alone, and the frequencies of the IC chips and boards overlap. Therefore, binarization with single threshold values does not work well for IC chip identification. The authors could find no previous research on the identification of IC chips mounted on both sides of PCB fragments, regardless of the technique (including X-ray photography).
Artificial intelligence technology such as deep learning offers a possible solution for IC chip identification. However, the most important but difficult element of such deep learning is the preparation of a sufficient number (usually thousands) of training data items. As shown in Fig. 1a, the X-ray transmission images of IC chips vary with the type of chip, making it unrealistic to attempt to prepare a sufficient number of such items for each individual IC chip type.
For these reasons, an automated IC chip detachment assessment method was developed, in which the IC chips are identified by a combination of multistep binarization and template matching. The individual techniques (binarization and template matching) are well known. However, as shown in Fig. 1, the frequency of the gray-scale values of IC chips and boards varies widely, and the simple application of binarization with a single threshold and/or template matching is not effective for assessment. In the proposed method, IC chips with various gray-scale distributions were identified sequentially, along with their suitable thresholds, which is not possible with the simple application of binarization with a single threshold and/or template matching. In addition, the analysis parameters were investigated, and the proposed method was validated experimentally by assessment of mechanically crushed mobile phone PCBs.

Concept of the proposed method
As described in the Introduction, the internal structure of IC chips varies; however, the external structure is generally rectangular, and the authors exploited this external feature in developing the proposed IC chip identification technique. The basic concept of the technique lies in determining the shape of the IC chips through multistep binarization with gradual thresholds, and identifying target IC chips using template matching with rectangles. Figure 2 shows a trial of this concept. Figure 2a Figure 2b is a binarized image of (a) with a threshold ( T ) of 30,000. Neither Chip A or B is clearly identified at this threshold. In Fig. 2c, which shows the binarization at T = 35000 , Chip A can be identified by rectangle template matching, but Chip B cannot. Figure 2d shows the binarization at T = 50000 . This time, Chip B is identified, but Chip A has disappeared into the board and is unidentifiable.
Then, a gradual binarization sequence is performed, from a minimum threshold ( T min ) of 20,000 to a maximum threshold ( T max ) of 60,000, with a threshold interval ( T int ) of 5000. Figure 2e shows the result, with the identified IC chips (in black) superimposed on the image in (a). All the IC chips visually identified in (a) have been successfully identified automatically. Figure 2f shows the five templates used in the rectangle template matching above.
Equipment and algorithm The X-ray transmission images of crushed PCB were obtained by an X-ray camera system (MOS-X3010S, Hitachi Power Solutions Co., Ltd., Japan), with a time delay integration line sensor (sensor pitch of 0.048 mm and 4608 effective pixels) with rated output of 90 kV and 1.0 mA, and discrimination capability of 4.0 LP with a line-pair chart. The X-ray camera scanned the crushed PCBs as they moved along a conveyor belt of 200-mm width at 1 m/min. In this study, the average processing speed for the scan was approximately 91.1 g/min (corresponding to 5.4 PCBs/min).
The obtained images were analyzed on a normal-spec pc (Intel Core i7-8700 CPU@3.20 GHz, Windows 10, and 16.0 GB RAM). The IC chip identification program was written in HALCON.
The overall process of the developed IC chip detachment assessment method is as follows [and see Fig. 3 (a) Crushed PCB fragments are fed into the X-ray device, with physical overlap eliminated through the use of a vibrating feeder, and an X-ray transmission image (16-bit gray-scale in this study) is obtained. (b) The gray-scale image is binarized at a board threshold ( T b ; in this study T b = 60000 ), and adjacent pixels are recognized as constituting the same fragment. (c) Recognized board fragments' list is sequentially numbered, and all the pixels in the fragments are registered (initialized) as "board". (d) The IC chips in the fragments are identified. This is the main subject of the present study, and is detailed below. (e) The IC chip detachment rate is calculated as the ratio of the number of identified detached IC chips to the total number of identified (detached and undetached) IC chips. Determination of whether a given IC chip is detached or undetached is based on the ratio of the area of the IC chip to the overall area of the fragment that contains it: detached when the chip makes up 80% or more of the fragment, and undetached when it makes up less than 80%.
The detailed process for IC chip identification [i.e., (d) above] is as follows [and see Fig. 3(ii)].
(f) T min , T max , and T int are set. In this study, T min = 20, 000 , T max = 60, 000 , and T int = 5000 . Note that T int = 5000 is a default value whose influence will be examined in "Discussion".  ( R ncc ) between each binarized group of adjacent pixels (binarized fragment) and a given template is calculated, using the normalized cross-correlation method, by the following equation [23]: where P(i, j) and Q(i, j) denote the gray-scale (0 or 1 in the binarized images in this study) of the template and binarized fragment, respectively; and M and N denote the pixel number of the width and height of the template, respectively. R ncc takes a value between 0 and 1, with 1 indicating a complete match. In this study, an IC chip is identified when R ncc ≥ F c ∕100 , where F c is a designated percentage correlation factor. For each binarized fragment, template matching is attempted using all the templates, one by one, until the IC chip is identified. In this study, F c was set at 85% as a default (1) , value, but its general influence will be examined in "Discussion". (l) If any IC chips are identified, the process continues; when no chips are identified, the process ends. (m-n) Each pixel in an identified IC chip is reregistered as "chip", and then removed from the further template matching process to avoid duplicate registration. (o) T is renewed as T = T + T int .

Validation
To validate the developed automated method, mechanically crushed PCBs were assessed both manually and automatically, and the IC chip detachment rates obtained by each method were compared.

Samples
Generally, several PCBs are used in a mobile phone (smart phone or feature phone). In this study, the PCB on which Fig. 3 Flowchart of the automated detachment assessment method the largest number of IC chips were mounted was manually detached from the phone. Five such PCBs (from five randomly selected mobile phones) were treated for further batch processing. Then, 20 IC chips larger than 5 mm in long side-length were randomly selected from the five PCBs, and individually colored with lacquer composition, while the rest of the PCBs were colored white, to enable visual distinction of the colored IC chips from the boards and other parts.

Mechanical detachment
A drum-type chain agitation mill (cross-flow shredder, Sato Tekko Co., Ltd., Japan), specially improved for electric part detachment, was used for mechanical detachment of the PCBs. Two controlled parameters, which have great influence on parts' detachment [20,24], were selected: (1) crushing time: the time from input to output of the PCBs in the mill; and (2) rotational speed: the speed of rotation of the chains located at the bottom of the mill. Table 1 lists the five experimental conditions. Five colored PCBs were simultaneously processed in each experiment.

Manual and automated detachment assessment
For the manual assessment, mechanically crushed PCBs were visually observed; that is, the colored IC chips were manually identified, and their condition (detached or undetached, crushed or uncrushed) was visually determined. On average, 19.4 of the 20 IC chips (97.0%) were observed. Then, the manually assessed IC chip detachment rate ( R m ) was calculated as in "Methodology" (e).
The automatically assessed IC chip detachment rate ( R a ) was calculated as in "Methodology". Figure 4 compares R m and R a . The automated assessment showed good agreement with the manual assessment. The average error rate, calculated by 100 | | R m − R a | | ∕R m , was only 2.2%. These results validate the effectiveness of the proposed method.

Effect of template matching parameters
The effect of the template matching parameters [ F c (correlation factor), T int (interval of threshold), and number of templates ( N t )] on the template matching accuracy and analysis time was investigated. Additionally, in this examination, the template matching targets were expanded from 5-16 mm IC chips to 2-16 mm rectangular electric parts, to assess the influence of target size.
The examination involved the following procedures: an X-ray transmission image of the crushed PCBs was obtained; the electric parts were visually identified from the image; the template matching results, with various parameter patterns, were compared with the visual identification; three parameters were assessed: number of visually identified parts ( n v ), number of automatically identified parts which were also identified visually ( n + a ), and number of automatically identified parts which were not identified visually ( n − a ). The results were assessed in terms of three parameters: (1) success rate, calculated by 100n + a ∕n v ; (2) accuracy rate, calculated by 100n + a ∕ n + a + n − a ; and (3) analysis time.
Correlation factor, F c Figure 5 compares the success and accuracy rates with F c ranging from 80 to 90%, and with T int = 5000 and N t = 135 . With increasing F c , the accuracy rate increased, because greater coincidence between the template and parts was required; while the success rate decreased, because even small damage to the part's shape prevented identification. The success and accuracy rates were in a trade-off relationship, and an F c of 85% appeared to be a balanced value. Figure 6 compares the success and accuracy rates, and analysis time, with T int ranging from 500 to 5000, and with F c = 85 % and N t = 135 . The analysis time naturally decreased with increasing T int , because the number of repetitions of the template matching process (Fig. 3i-n) decreased, and almost converged when T int was around 5000. On the other hand, contrary to expectations, the success and accuracy rates showed constant values, irrespective of T int . This would appear to be because the small changes in the binarized fragment shapes, caused by the small T int , were absorbed by the moderate F c value of 85 %. Based on these results, a T int of 5000 is the optimal value.

Number of templates, N t
As a default and maximum value of N t , 135 different rectangles (as in Fig. 2f) were prepared [15 long-side lengths (2-16 mm at 1-mm intervals) times 9 long/short side-length ratios (1.0-1.8 at 0.1 intervals)]. Various N t values of less than 135 were prepared by subtracting from the default value of 135 templates. Figure 7 compares the success and accuracy rates, and analysis time, with N t ranging from 9 to 135, and with F c = 85 % and T int = 5000 . With increasing N t , the success rate increased, but the accuracy rate remained constant. This is because, with increasing N t , the range of identifiable parts increases, which results in an increased success rate, but no contribution is made to the identification accuracy, which results in a constant accuracy rate. The analysis time naturally increased with increasing N t , because the number of identification process repetitions [ Fig. 3(ii)] was increased.
These results show that N t has an influence on the statistical reliability, with larger N t values increasing the number of target parts, resulting in greater statistical reliability, and smaller N t values targeting a limited number of parts, as in a sampling test, resulting in less statistical reliability [25]. The effect of N t on the statistical reliability is in a trade-off relationship with the analysis time. Therefore, N t can be flexibly set according to the analytical requirements.

Limitations
The limitations on the proposed method are as follows, Crushed (i.e., disfigured) parts cannot be identified by the method. In the validation, the experimental conditions  were set to avoid overcrushing of the PCBs, to minimize the destructive detachment of parts, and thereby ensure reasonable results. To address this limitation, the assessment may be conducted both before and after the mechanical process, with the decrease in the number of uncrushed parts after the mechanical process representing the number of crushed parts.
Lesser resolution in the X-ray transmission images reduces the identification accuracy, as the accuracy rate in "Discussion", where smaller parts were targeted (2-16 mm), was less than in the "Methodology", where the target part size was 5-16 mm.
The overall throughput of the proposed method was calculated based on the processing speed of the X-ray camera scan (91.1 g/min) and the analysis time [65 s, corresponding to 62.7 g/min in the fastest case ( F c = 85 %, T int = 5000 , and N t = 12)]. These processes were conducted sequentially in the current system, but parallel processing is possible with the introduction of a dedicated workstation for image analysis, when the overall throughput then equals the lessor of the two (i.e., scanning and analysis) throughput values (62.7 g/ min). The calculated throughput is not close to being enough for complete analysis in a typical recycling plant. To address this limitation, sampling analysis, instead of complete analysis, may be performed.

Conclusion
An automated assessment method for IC chip detachment from printed circuit board (PCB), which combines multistep binarization and template matching of X-ray transmission images, was developed. As validation, the IC detachment rate for five types of mechanically crushed PCBs was examined manually and automatically. The developed automated method showed good agreement with the manual assessment, at an average error rate of 2.2%.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.