Introduction

With the dramatic increase in electronic equipment worldwide, the manufacturing of printed circuit board (PCB) is rapidly increasing [1]. Since precious metals (e.g., gold), base metals (e.g., copper), and minor metals (e.g., tantalum) are well concentrated in PCBs [2, 3], demand for PCB recycling is increasing [1, 4,5,6]. PCB consists not only of metals but of various other materials such as glass, resin, and paper; and various types of PCB recycling methodology, targeting different materials, have been proposed. Recent studies have, for example, employed hydrometallurgy [7, 8], leaching and digestion [9,10,11], flotation [12, 13], super-gravity separation [14], corona electrostatic separation [15], catalytic pyrolysis [16], and supercritical methanol [17].

On PCB, the precious metals and minor metals are specifically concentrated in electric parts. For their effective recycling, then, the electric parts should be efficiently detached from the PCB and separated according to their type. A physical separation technique specifically targeted at electric parts has been developed, which focuses both on the material features (e.g., density, magnetism, and conductivity) and the structural features (e.g., size and shape) [18]. The aim is to detach the electric parts from the PCB without damaging them (non-destructive detachment). Parts detachment technology is increasing in sophistication, with chemical [19], mechanical [20], and underwater explosion-based [21] approaches being investigated. A mechanical approach was adopted in the present study, because its economy and rapidity well suit the demands of recycling plants. In the above-cited studies, parts detachment was manually assessed, which is sufficient for laboratory testing, but suffers from problems of time-consumption and human error. Therefore, the authors attempted to develop an automated assessment method. Among the various potential types of relevant electric parts and PCBs, the detachment of integrated circuit (IC) chips from mobile phone PCBs was targeted in this study, because it represents a typical case where the effective recycling of the precious and minor metals from rapidly increasing e-waste is required.

In assessing IC chip detachment, the key is to distinguish the IC chip components from the other PCB fragments. Various methods are available for identification of IC chips using a combination of visual camera and image analysis (e.g., [22]). However, in general, double-sided PCBs, where IC chips are mounted on both sides of the board, are used in mobile phones; thus, both sides of the PCB must be observed in such assessments. Furthermore, in the case of mechanical (or underwater explosion-based) detachment, the PCBs are typically crushed into small and irregularly shaped fragments, making it difficult to turn them over and visually observe both sides of the fragmented PCB. Thus, in this case, X-ray photography is a more useful observational technique than those utilizing visual cameras.

The IC chips are composed of various types of resin and metal, and their internal structures vary with the type and manufacturer. The same may be said for the boards themselves (which are the remnants of the complete PCB after IC chip detachment). Figure 1a shows X-ray transmission images of various IC chips and boards, with their varying internal structures. Figure 1b compares the frequencies of the gray-scale values of chips A–C, and boards D–F in Fig. 1a. It is clear that the frequencies vary greatly, even among the IC chips (and boards) alone, and the frequencies of the IC chips and boards overlap. Therefore, binarization with single threshold values does not work well for IC chip identification. The authors could find no previous research on the identification of IC chips mounted on both sides of PCB fragments, regardless of the technique (including X-ray photography).

Fig. 1
figure 1

a X-ray transmission images of IC chips and boards. b Frequencies of gray-scale values of IC chips and boards

Full size image

Artificial intelligence technology such as deep learning offers a possible solution for IC chip identification. However, the most important but difficult element of such deep learning is the preparation of a sufficient number (usually thousands) of training data items. As shown in Fig. 1a, the X-ray transmission images of IC chips vary with the type of chip, making it unrealistic to attempt to prepare a sufficient number of such items for each individual IC chip type.

For these reasons, an automated IC chip detachment assessment method was developed, in which the IC chips are identified by a combination of multistep binarization and template matching. The individual techniques (binarization and template matching) are well known. However, as shown in Fig. 1, the frequency of the gray-scale values of IC chips and boards varies widely, and the simple application of binarization with a single threshold and/or template matching is not effective for assessment. In the proposed method, IC chips with various gray-scale distributions were identified sequentially, along with their suitable thresholds, which is not possible with the simple application of binarization with a single threshold and/or template matching. In addition, the analysis parameters were investigated, and the proposed method was validated experimentally by assessment of mechanically crushed mobile phone PCBs.

Methodology

Concept of the proposed method

As described in the Introduction, the internal structure of IC chips varies; however, the external structure is generally rectangular, and the authors exploited this external feature in developing the proposed IC chip identification technique. The basic concept of the technique lies in determining the shape of the IC chips through multistep binarization with gradual thresholds, and identifying target IC chips using template matching with rectangles.

Figure 2 shows a trial of this concept. Figure 2a is a 16-bit (65,536 Gy-scale) X-ray transmission image of mechanically crushed PCB, with the visually identified IC chips circled. Chips A and B in Fig. 2a, which are, respectively, detached and undetached from the PCB, are labeled.

Fig. 2
figure 2

X-ray transmission images, in 16-bit (65,536) gray-scale, of crushed PCB. a Visually identified IC chips circled; b–d binarized images of (a) at thresholds of 30,000, 35,000, and 50,000, respectively; e IC chips identified by the gradual binarization sequence and template matching (black rectangles), superimposed on (a); and f the templates used in template matching

Full size image

Figure 2b is a binarized image of (a) with a threshold (\(T\)) of 30,000. Neither Chip A or B is clearly identified at this threshold. In Fig. 2c, which shows the binarization at \(T=35000\), Chip A can be identified by rectangle template matching, but Chip B cannot. Figure 2d shows the binarization at \(T=50000\). This time, Chip B is identified, but Chip A has disappeared into the board and is unidentifiable.

Then, a gradual binarization sequence is performed, from a minimum threshold (\({T}_{\mathrm{min}}\)) of 20,000 to a maximum threshold (\({T}_{\mathrm{max}}\)) of 60,000, with a threshold interval (\({T}_{\mathrm{int}}\)) of 5000. Figure 2e shows the result, with the identified IC chips (in black) superimposed on the image in (a). All the IC chips visually identified in (a) have been successfully identified automatically. Figure 2f shows the five templates used in the rectangle template matching above.

Equipment and algorithm

The X-ray transmission images of crushed PCB were obtained by an X-ray camera system (MOS-X3010S, Hitachi Power Solutions Co., Ltd., Japan), with a time delay integration line sensor (sensor pitch of 0.048 mm and 4608 effective pixels) with rated output of 90 kV and 1.0 mA, and discrimination capability of 4.0 LP with a line-pair chart. The X-ray camera scanned the crushed PCBs as they moved along a conveyor belt of 200-mm width at 1 m/min. In this study, the average processing speed for the scan was approximately 91.1 g/min (corresponding to 5.4 PCBs/min).

The obtained images were analyzed on a normal-spec pc (Intel Core i7-8700 CPU@3.20 GHz, Windows 10, and 16.0 GB RAM). The IC chip identification program was written in HALCON.

The overall process of the developed IC chip detachment assessment method is as follows [and see Fig. 3(i)].

Fig. 3
figure 3

Flowchart of the automated detachment assessment method

Full size image
  1. (a)

    Crushed PCB fragments are fed into the X-ray device, with physical overlap eliminated through the use of a vibrating feeder, and an X-ray transmission image (16-bit gray-scale in this study) is obtained.

  2. (b)

    The gray-scale image is binarized at a board threshold (\(T_{{\text{b}}}\); in this study \(T_{{\text{b}}} = 60000\)), and adjacent pixels are recognized as constituting the same fragment.

  3. (c)

    Recognized board fragments’ list is sequentially numbered, and all the pixels in the fragments are registered (initialized) as “board”.

  4. (d)

    The IC chips in the fragments are identified. This is the main subject of the present study, and is detailed below.

  5. (e)

    The IC chip detachment rate is calculated as the ratio of the number of identified detached IC chips to the total number of identified (detached and undetached) IC chips. Determination of whether a given IC chip is detached or undetached is based on the ratio of the area of the IC chip to the overall area of the fragment that contains it: detached when the chip makes up 80% or more of the fragment, and undetached when it makes up less than 80%.

The detailed process for IC chip identification [i.e., (d) above] is as follows [and see Fig. 3(ii)].

  1. (f)

    \(T_{{{\text{min}}}}\), \(T_{{{\text{max}}}}\), and \(T_{{{\text{int}}}}\) are set. In this study, \(T_{{{\text{min}}}} = 20,000\), \(T_{{{\text{max}}}} = 60,000\), and \(T_{{{\text{int}}}} = 5000\). Note that \(T_{{{\text{int}}}}\) = 5000 is a default value whose influence will be examined in “Discussion”.

  2. (g)

    The rectangle templates are registered; in this study, totaling 108 rectangular shapes with black edges [as in Fig. 2(f), 12 long-side lengths (5–16 mm at 1 mm intervals) times 9 long/short side-length ratios (1.0–1.8 at 0.1 intervals)]. The templates are easily created using general software such as Microsoft Power Point. The influence of the number of templates will be examined in “Discussion”.

  3. (h)

    The \(T\) for binarization is initially set as \(T_{{{\text{min}}}}\).

  4. (i)

    If \(T \le T_{{{\text{max}}}}\) is true, the process continues to (j); if false, the process ends.

  5. (j)

    The X-ray transmission image is binarized at \(T\).

  6. (k)

    Template matching is conducted on the binarized image using the registered templates. The matching rate (\(R_{{{\text{ncc}}}}\)) between each binarized group of adjacent pixels (binarized fragment) and a given template is calculated, using the normalized cross-correlation method, by the following equation [23]:

    $$R_{{{\text{ncc}}}} = \frac{{\mathop \sum \nolimits_{j = 0}^{N - 1} \mathop \sum \nolimits_{i = 0}^{M - 1} P\left( {i, j} \right)Q\left( {i,j} \right)}}{{\sqrt {\mathop \sum \nolimits_{j = 0}^{N - 1} \mathop \sum \nolimits_{i = 0}^{M - 1} P\left( {i, j} \right)^{2} \times \mathop \sum \nolimits_{j = 0}^{N - 1} \mathop \sum \nolimits_{i = 0}^{M - 1} Q\left( {i,j} \right)} }},$$
    (1)

    where \(P\left( {i,j} \right)\) and \(Q\left( {i,j} \right)\) denote the gray-scale (0 or 1 in the binarized images in this study) of the template and binarized fragment, respectively; and \(M\) and \(N\) denote the pixel number of the width and height of the template, respectively. \(R_{{{\text{ncc}}}}\) takes a value between 0 and 1, with 1 indicating a complete match. In this study, an IC chip is identified when \(R_{{{\text{ncc}}}} \ge F_{{\text{c}}} /100\), where \(F_{{\text{c}}}\) is a designated percentage correlation factor. For each binarized fragment, template matching is attempted using all the templates, one by one, until the IC chip is identified. In this study, \(F_{{\text{c}}}\) was set at 85% as a default value, but its general influence will be examined in “Discussion”.

  7. (l)

    If any IC chips are identified, the process continues; when no chips are identified, the process ends.

  8. (m-n)

    Each pixel in an identified IC chip is reregistered as “chip”, and then removed from the further template matching process to avoid duplicate registration.

  9. (o)

    \(T\) is renewed as \(T = T + T_{{{\text{int}}}}\).

Validation

To validate the developed automated method, mechanically crushed PCBs were assessed both manually and automatically, and the IC chip detachment rates obtained by each method were compared.

Samples

Generally, several PCBs are used in a mobile phone (smart phone or feature phone). In this study, the PCB on which the largest number of IC chips were mounted was manually detached from the phone. Five such PCBs (from five randomly selected mobile phones) were treated for further batch processing. Then, 20 IC chips larger than 5 mm in long side-length were randomly selected from the five PCBs, and individually colored with lacquer composition, while the rest of the PCBs were colored white, to enable visual distinction of the colored IC chips from the boards and other parts.

Mechanical detachment

A drum-type chain agitation mill (cross-flow shredder, Sato Tekko Co., Ltd., Japan), specially improved for electric part detachment, was used for mechanical detachment of the PCBs. Two controlled parameters, which have great influence on parts’ detachment [20, 24], were selected: (1) crushing time: the time from input to output of the PCBs in the mill; and (2) rotational speed: the speed of rotation of the chains located at the bottom of the mill. Table 1 lists the five experimental conditions. Five colored PCBs were simultaneously processed in each experiment.

Table 1 Experimental conditions
Full size table

Manual and automated detachment assessment

For the manual assessment, mechanically crushed PCBs were visually observed; that is, the colored IC chips were manually identified, and their condition (detached or undetached, crushed or uncrushed) was visually determined. On average, 19.4 of the 20 IC chips (97.0%) were observed. Then, the manually assessed IC chip detachment rate (\(R_{{\text{m}}}\)) was calculated as in “Methodology” (e).

The automatically assessed IC chip detachment rate (\(R_{{\text{a}}}\)) was calculated as in “Methodology”.

Results

Figure 4 compares \(R_{{\text{m}}}\) and \(R_{{\text{a}}}\). The automated assessment showed good agreement with the manual assessment. The average error rate, calculated by \(100\left| {R_{{\text{m}}} - R_{{\text{a}}} } \right|/R_{{\text{m}}}\), was only 2.2%. These results validate the effectiveness of the proposed method.

Fig. 4
figure 4

Comparison of the IC chip detachment rates for manual assessment and the developed automated assessment

Full size image

Discussion

Effect of template matching parameters

The effect of the template matching parameters [\(F_{{\text{c}}}\) (correlation factor), \(T_{{{\text{int}}}}\) (interval of threshold), and number of templates (\(N_{{\text{t}}}\))] on the template matching accuracy and analysis time was investigated. Additionally, in this examination, the template matching targets were expanded from 5–16 mm IC chips to 2–16 mm rectangular electric parts, to assess the influence of target size.

The examination involved the following procedures: an X-ray transmission image of the crushed PCBs was obtained; the electric parts were visually identified from the image; the template matching results, with various parameter patterns, were compared with the visual identification; three parameters were assessed: number of visually identified parts (\(n_{{\text{v}}}\)), number of automatically identified parts which were also identified visually (\(n_{{\text{a}}}^{ + }\)), and number of automatically identified parts which were not identified visually (\(n_{{\text{a}}}^{ - }\)). The results were assessed in terms of three parameters: (1) success rate, calculated by \(100n_{{\text{a}}}^{ + } /n_{{\text{v}}}\); (2) accuracy rate, calculated by \(100n_{{\text{a}}}^{ + } /\left( {n_{{\text{a}}}^{ + } + n_{{\text{a}}}^{ - } } \right)\); and (3) analysis time.

Correlation factor, \(F_{{\text{c}}}\)

Figure 5 compares the success and accuracy rates with \(F_{{\text{c}}}\) ranging from 80 to 90%, and with \(T_{{{\text{int}}}} = 5000\) and \(N_{{\text{t}}} = 135\). With increasing \(F_{{\text{c}}}\), the accuracy rate increased, because greater coincidence between the template and parts was required; while the success rate decreased, because even small damage to the part’s shape prevented identification. The success and accuracy rates were in a trade-off relationship, and an \(F_{{\text{c}}}\) of 85% appeared to be a balanced value.

Fig. 5
figure 5

Effect of the correlation factor

Full size image

Threshold interval, \(T_{{{\text{int}}}}\)

Figure 6 compares the success and accuracy rates, and analysis time, with \(T_{{{\text{int}}}}\) ranging from 500 to 5000, and with \(F_{{\text{c}}} = 85\)% and \(N_{{\text{t}}} = 135\). The analysis time naturally decreased with increasing \(T_{{{\text{int}}}}\), because the number of repetitions of the template matching process (Fig. 3i–n) decreased, and almost converged when \(T_{{{\text{int}}}}\) was around 5000. On the other hand, contrary to expectations, the success and accuracy rates showed constant values, irrespective of \(T_{{{\text{int}}}}\). This would appear to be because the small changes in the binarized fragment shapes, caused by the small \(T_{{{\text{int}}}}\), were absorbed by the moderate \(F_{{\text{c}}}\) value of \(85\)%. Based on these results, a \(T_{{{\text{int}}}}\) of 5000 is the optimal value.

Fig. 6
figure 6

Effect of the threshold interval

Full size image

Number of templates, \(N_{{\text{t}}}\)

As a default and maximum value of \(N_{{\text{t}}}\), 135 different rectangles (as in Fig. 2f) were prepared [15 long-side lengths (2–16 mm at 1-mm intervals) times 9 long/short side-length ratios (1.0–1.8 at 0.1 intervals)]. Various \(N_{{\text{t}}}\) values of less than 135 were prepared by subtracting from the default value of 135 templates.

Figure 7 compares the success and accuracy rates, and analysis time, with \(N_{{\text{t}}}\) ranging from 9 to 135, and with \(F_{{\text{c}}} = 85\)% and \(T_{{{\text{int}}}} = 5000\). With increasing \(N_{{\text{t}}}\), the success rate increased, but the accuracy rate remained constant. This is because, with increasing \(N_{{\text{t}}}\), the range of identifiable parts increases, which results in an increased success rate, but no contribution is made to the identification accuracy, which results in a constant accuracy rate. The analysis time naturally increased with increasing \(N_{{\text{t}}}\), because the number of identification process repetitions [Fig. 3(ii)] was increased.

Fig. 7
figure 7

Effect of the number of templates

Full size image

These results show that \(N_{{\text{t}}}\) has an influence on the statistical reliability, with larger \(N_{{\text{t}}}\) values increasing the number of target parts, resulting in greater statistical reliability, and smaller \(N_{{\text{t}}}\) values targeting a limited number of parts, as in a sampling test, resulting in less statistical reliability [25]. The effect of \(N_{{\text{t}}}\) on the statistical reliability is in a trade-off relationship with the analysis time. Therefore, \(N_{{\text{t}}}\) can be flexibly set according to the analytical requirements.

Limitations

The limitations on the proposed method are as follows,

Crushed (i.e., disfigured) parts cannot be identified by the method. In the validation, the experimental conditions were set to avoid overcrushing of the PCBs, to minimize the destructive detachment of parts, and thereby ensure reasonable results. To address this limitation, the assessment may be conducted both before and after the mechanical process, with the decrease in the number of uncrushed parts after the mechanical process representing the number of crushed parts.

Lesser resolution in the X-ray transmission images reduces the identification accuracy, as the accuracy rate in “Discussion”, where smaller parts were targeted (2–16 mm), was less than in the “Methodology”, where the target part size was 5–16 mm.

The overall throughput of the proposed method was calculated based on the processing speed of the X-ray camera scan (91.1 g/min) and the analysis time [65 s, corresponding to 62.7 g/min in the fastest case (\(F_{{\text{c}}} = 85\)%, \(T_{{{\text{int}}}} = 5000\), and \(N_{{\text{t}}} = 12\))]. These processes were conducted sequentially in the current system, but parallel processing is possible with the introduction of a dedicated workstation for image analysis, when the overall throughput then equals the lessor of the two (i.e., scanning and analysis) throughput values (62.7 g/min). The calculated throughput is not close to being enough for complete analysis in a typical recycling plant. To address this limitation, sampling analysis, instead of complete analysis, may be performed.

Conclusion

An automated assessment method for IC chip detachment from printed circuit board (PCB), which combines multistep binarization and template matching of X-ray transmission images, was developed. As validation, the IC detachment rate for five types of mechanically crushed PCBs was examined manually and automatically. The developed automated method showed good agreement with the manual assessment, at an average error rate of 2.2%.