Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

In recent years environmental awareness and need for cost reduction has increasingly influenced the use of reusable industrial-textiles. Nowadays more than one billion cleaning textiles in Europe are being leased and reused per year. Besides the big volume of processed pieces, quality assurance of used industrial textiles has remained a mostly manually operated task. Compared to humans automated systems can have several advantages, such as lower costs, higher reliability and accuracy. Quality assurance after dry cleaning is one of the most cost intensive operating processes. Lowering its costs will lead to an overall cost reduction and therefore may encourage more customers to start using reusable industrial textiles. With increasing performance of artificial intelligence, automatic fabric defect detection has become one of the most relevant areas in this domain. So far, recent work in the field of textile inspection deals mostly with continuous 2D textures. This is because fabric inspection algorithms are mostly used during furling in the production process. Compared to that we present a solution intended to be used by the cleaning industry, that handles textiles individually in an assembly-line work flow. Due to high flow rates of textiles, an automatized mechanism to spread out textiles mechanically while in movement has not yet been invented. In this paper we focus on an inspection of defects in a pile-like arrangement, where every item is still dealt with separately, on an assembly-line. The uneven surface, varying colors of sewing pattern and weaving of different textile fibers are some of the challenges in this task. Textiles furthermore differ in the composition of fibers which include cotton, linen, polyester or compositions. Previous research on outspread fabrics achieve high recognition rates but are not resistant against some effects caused by voluminous shapes. Shape, folds, edges, borders, overlapping edges and ambient occlusion are some of these effects. They have a negative impact on correct detection of fiber-defects using known methods. The voluminous shape of textiles also results in loss of focus. The here presented method can therefore be seen as a baseline for fiber defects recognition on uniform textured textiles in voluminous shape. There are furthermore differences in fiber-defects as shown in Fig. 5a. Our database contains most of the defects as they are defined by the textile industry [1]. This includes: stains, bonding, silicon relics, holes, enclosures, dropped stitches, press-offs or others. After washing a textile multiple times, fibers may have changed color and appearance (see Fig. 5b). Several relevant steps like preprocessing and classification are shown in our inspection pipeline in Sect. 3. In view of addressing the mentioned problems of voluminous shape, we present a novel approach for fiber defect recognition in Sect. 4.1. In our experiments we implemented and evaluated this method in comparison with other known methods as described in Sect. 4.3. Our approach is invariant to different fiber-weavings and fulfills the requirements of compatibility with different uniform colored textiles. The effectiveness of our method is shown in experiments at Sect. 5.

2 Related Work

Web inspection is a common application of automatized textile defects inspection. It is mostly performed on spread fabrics, which are carried out during their manufacturing process. Most recent work focused on defect detection and classification. Mishra [2] distinguishes woven, knitted and dyeing/finishing defects which occur during spooning or weaving. Textiles can be categorized generally into uniform and different kinds of textured materials (uniform, random or patterned) [3]. For detection of defects on uniform textured fabrics, three defect-detection techniques exist: statistical, spectral and model-based [4]. Defect detection on (un-spread) textiles in voluminous shape is a relatively new field. Our work focuses on the inspection of textured material which has an almost homogeneous color and uses a combination of a statistical and model based approach.

Neural Networks (NN), AdaBoost [5] and Support Vector Machines (SVM) [6] are notable machine learning techniques that were used in a number of articles in this field. Some approaches on flat, and spread-out 2D surface achieve success rates in fabric defects detection higher than 90% [4, 7]. Compared to that, humans achieve detection rates of only 60–75% [8]. Supervised learning strategies achieved good performance using a counter-propagation NN, trained by a resilient back-propagation algorithm [9]. As several NN suffer from a high sensibility regarding changes in orientation and lightning, we decided to use other machine learning techniques. In case of un-spread (inhomogeneous) textile classification, we proposed a system for classification of textile fibers using LBP-features and local-interest points in our recent work [10]. The process was evaluated by the use of preselected image patches in order to reduce computational costs in the textile classification using SVM and AdaBoost. The most time consuming processes when using these supervised machine learning methods are the acquisition and labeling processes. Thousands of patches have to be acquired and labeled manually. In contrast our novel method reduces the required effort to be spent in labeling of the data and combines it with convolutional neural network classification. Two of the most challenging problems in fabric classification are ambient occlusion and folds. These effects are caused by the shape of the textile and the influence of illumination. In our recent work [11] a normalization method was introduced that reduces these effects, paid by a loss of information. This method, based on stereo-imaging is used in our work for preprocessing of the acquired images.

3 Methodology

We present an inspection system pipeline (see Fig. 1) that classifies dry-washed textiles in pile-like arrangement into the classes ‘fiber defect’ and ‘no defect’. The system is intended to be used in an assembly line like environment, where every item is served individually. It is built on a hierarchical decision tree model in which a first classifier determines stain defects and excludes them from further classification (see Fig. 1c. A second, in Sect. 4.1 presented classifier (see Fig. 1e) recognizes fiber defects and makes the final decision on whether an image shows a defect textile. The parts: Fig. (1a) Image Acquisition and Fig. (1b) Preprocessing follow the stereo-normalization approach presented in [11], in order to reduce the effect of shading in the captured images. A disparity map is used to exclude areas showing folds or shadows. Other areas classified as ‘stain defect’ are also excluded from the image as shown in step (see Fig. 1d).

Fig. 1.
figure 1

Pipeline showing a combination of the proposed preprocessing steps (b) and (d) and the classification decision tree steps (c) and (e).

Full size image

3.1 Capturing Environment and Database

Our approach is evaluated on an image database of dry cleaned woven cotton cleaning textiles as they are used in many different industrial applications. We use a soft-box with homogeneous illumination in the image acquisition, to guarantee a controlled capturing process with even lighting. Two synchronized CMOS color cameras with a CMOS 1/1.8” sensor and a resolution of \(1280\,\times \,1003\) pixels are used for image acquisition. The database contains 910 images of 258 different textiles with and without fiber defects (see Table 1). Fiber-defects are defects that can originate through the manufacturing/furling process (e.g. dropped stitches, press-offs or broken ends) or intensive stress. Most defects of that defect category were caused by intensive use of these textiles in industrial environments and show mostly holes and cut like defects (see Fig. 5). Because of diverse uses, the fibers also often show different levels of shading. Images are captured from a top-down perspective, therefore, some defects might be hidden (e.g. if they are in a fold or on the bottom side). Iterations in which the textile is physically reoriented and then classified could solve this problem.

Table 1. Quantities of textiles images.
Full size table

4 Fiber Defect Recognition

In this Chapter we introduce a novel method for classification of fiber-defects in images of textiles in voluminous shape. As discussed in the state of the art, other methods perform well on fiber-defect recognition of outspread fabric. On textiles in inhomogeneous shape, most of these methods fail, as shadow and stain elements influence the classification negatively [11]. We propose a novel method using a combination of SURF key points and convolution neural network classification (see Sect. 4.1). This method requires images whereas areas of shadow and stain defects are normalized, as described in [11]. In Sect. 4.3, adoptions of competitive methods are presented, that have shown their effectiveness in similar classification tasks [10]. Furthermore an experiment, using an illumination normalization technique is presented.

4.1 CNN Classification Based on Keypoint Preselection

In conventional supervised machine learning methods, the labeling of data is a costly process. In order to reduce the effort to be spent, we use a SURF detector with a minimum Hessian threshold of 500 to determine key points on distinctive areas of the textile. We generate partially overlapping patches of 32\(\,\times \,\)32 pixels in size, centered at each key point. These patches are then used to train a slightly modified LeNet-5 CNN [12] classifier. As a consequence patches of low dimensional data are generated, to be used in training of the CNN to recognize patterns. Instead of requiring an enormous amount of high dimensional data, we direct the network to key points of distinct areas of the image using SURF key points.

We labeled our database manually by defining a mask on regions with fiber-defects (see Fig. 2). If a feature is inside the masked defect-region, its corresponding patches are classified as a defect patch. Patches outside that masked region are classified as a non defect patch.

Fig. 2.
figure 2

Pipeline of micro-patch-based classification using CNN.

Full size image

We use two convolution layers which are combined with a max-pooling layer (see Fig. 3 for a visualization of the customized network). After the inner-product layer we receive a fully connected layer to score our data. The loss layer is built using the soft-max function. To implement the network we use the caffe libraries from BVLC [13].

Fig. 3.
figure 3

Our adjusted LeNet-5 CNN.

Full size image

The so created database consists of approx. 58000 image patches and is divided in a training (80%) and testing set (20%). The data is separated and shuffled taking into account the individual images in such a way that no image is simultaneously present in the test and training set. It is trained with 1500 iterations and a batch size of 1000 features, which results in approx. 30 epochs. The classifier predicts whether a patch belongs to a region showing some fiber-defect or not. For each textile we receive as many predictions as there are key points.

4.2 Fusion

To make a final decision for each textile, we use the weighted sum combination rule to calculate a fused unified decision. It is based on two features, the first one is the number of key points detected. The second feature the difference between positive and negative decisions. We represent both values as scores and normalized them to a comparable range using min-max normalization, which can be formulated as:

$$\begin{aligned} S' = \frac{S - min\{S_k\}}{max\{S_k\} - min\{S_k\}} \end{aligned}$$
(1)

where \( min\{S_k\}\) and \(max\{S_k\}\) are the minimum and maximum values of existing scores in the data of the corresponding sources and \(S'\) is the normalized score. We used the weighted sum score fusion, where for each score source a weight is defined that indicates its relevance on the fused decision. The weight is calculated by 1-EER and fused by the weighted sum rule F for N score sources.

$$\begin{aligned} F = \sum _{k=1}^{N}{w_k S_k} , k = \{1,\dots ,N\} \end{aligned}$$
(2)

4.3 Competitive Methods

Non-overlapping patches of 128\(\,\times \,\)128 Pixel are used to reduce the complexity of the analyzed pattern. Two different sets of features were selected from related work and examined as fiber defect classification step (see Fig. 4). As in the normalization step only ambient occlusion and stains were removed from the input image, another illumination normalization method is applied to the image to enhance the contrast between fibers and exclude differences in shading. We used a technique introduced by Tan and Triggs [14] which is based on a series of steps to counter the effects of illumination variation, local shadowing and highlights. The preprocessing chain consists of: gamma correction, which uses differences of Gaussian filtering, masking and contrast equalization (see Fig. 4a for an example of a normalized image).

Fig. 4.
figure 4

Pipeline of patch-based classification methods.

Full size image

The extracted patches contain all kind of different texture properties such as: seams, diverse edges and defects like cuts, open ends, holes, stains and others. During extraction, we used the Shannon-Entropy-Value to determine the amount of information in a patch. Our evaluation showed that patches with an entropy-value below 2000 do not contain enough information to be classified and are therefore rejected. Every patch is labeled manually by assigning them the class fiber-defect (see Fig. 5a) or none-defect (see Fig. 5b).

Fig. 5.
figure 5

Examples of the different extracted features.

Full size image

The local interest point descriptor SURF [15] has shown its effectiveness in many applications as local feature detectors and descriptors for non-rigid 3D objects [16]. They are scale-invariant and robust against rotation, translation and changing light conditions. A set of interest points is extracted into a 64-dimensional feature-vector, following a Bag of Words (BOW) approach [10].

Local binary patterns (LBP) features are a known technique, when dealing with textile fiber classification tasks [11]. The used LBP type [17] is invariant against rotation and pixel intensity variations and shows a relationship between a pixel and its neighborhood. It fulfills the requirements in regard to computational cost compared to other scale-invariant LBP approaches [18]. An evaluation on a subset of the database showed that a radius of 3 and a block size of 32 pixels is optimal for the given pictures. Histograms of rotation-invariant binary patterns are calculated and concatenated to a feature vector. To reduce the dimensionality of the feature vector, Principal Component Analysis (PCA), trained on a subset of the data set, is performed. Experiments showed that a reduction to 300 components gives best results.

An AdaBoost based machine-learning classifier is used for classification of the extracted feature vectors, using the REAL boosting method with confidence-rated predictions. Our evaluations showed that this classifier performed best among different evaluated classifiers (SVM, Random Forrest, AdaBoost and JRIP). All features are evaluated with- and without using the illumination normalization method.

5 Results and Discussion

We evaluated our proposed method shown in Sect. 4 on our novel database (see Sect. 3) with different preconditions. The given ‘Accuracy after Stain Filtering’ column describes the detection of fiber-defects on textiles of the defect categories: Fiber, Fiber and Stain and None (see Table 1). Patches showing other defects were excluded from the database using the first (stain-defect) classifier in the inspection process. The achieved accuracy (TP+TN/\(\sum \)Total population) was calculated on the full-image-level, which concludes to the final decision on whether a textile contains a fiber defect or not. In the calculation of that accuracy value the results of the first (stain-) detection classifier is not considered (see Sect. 3.4) (Table 2).

Table 2. Achieved Accuracy Rates of Fiber-Defect Detection Methods.
Full size table

The results using the SURF BOW approach in combination with the AdaBoost classifier showed an accuracy of 87.99%. Though using TanTriggs illumination normalization, appeared to us to be promising, the results showed to be less accurate than without using it over all tested methods. Rotation invariant unified Local Binary Patterns showed overall less accurate results than the bag of words approach. We suspect the reason for the bad results to be the rotational invariant variant which is less information conserving then the SURF BOW approach. Our approach using a combination of key point pre-selection and convolutional neural networks (CNN) classification achieved best results on the database. We found that the defined CNN is especially suitable to this recognition task. One reason might be the combination of dominant structural characteristics from a number of extracted patches that fit well to the NN characteristics, and the pre-selection of distinct regions in the image. Based on the huge amount of found SURF feature points, we were able to generate a huge data set used to train a deeper network. The best total rejection accuracy was achieved by that method gaining an accuracy of 90.15%. All approaches were evaluated using 4-fold cross validation with a regular distribution of defect-classes in each fold.

6 Conclusion

We presented a novel method for detection of fiber defects in textiles, that can be used when textiles are in an inhomogeneous, voluminous shape. Well performing aspects of the established methods: SURF key points, LBP, AdaBoost and CNNs were combined for an evaluation of this novel computer vision application. Our database showed textiles with different kind of fiber-defects such as holes and cuts in a pile-like arrangement, recorded with a stereo vision camera setup. Best results were achieved by the novel method using key points and CNN, which outperformed other recent methods used in the classification of voluminous textile fibers [10]. We described and proved in Sect. 4.1 how CNNs in combination with SURF-features were combined to effectively recognize and classify distinctive features by using low dimensional data as an input. A brief description of the performance of all proposed methods is given in Sect. 5. The future work may include a new normalization of image areas showing ambient occlusion, which will improve the performance of our system.