1 Introduction

With the rapid development of society and economy, the demand for high-quality garments is increasing. As a result, fabric defect detection has become a critical step in production. However, in industrial manufacturing, the detection of many different small and weak target defects remains a challenge.

Moreover, fabrics with too many defects can only be sold at \(45\%\sim 65\%\) of their normal price [1, 2]. If serious problems are not detected at the factory stage, the entire batch of products will be rendered unusable, which is detrimental to textile enterprises. Therefore, the implementation of an efficient, reliable, and integrated intelligent vision system for the textile industry can result in significant economic benefits. Today, the textile industry requires a more specific set of criteria for fabric defect detection, such as a more comprehensive scoring system beyond counting the number of defects or recognizing their type, and a more reasonable post-processing strategy based on different defect distribution patterns. Many researchers have made consistent efforts to address these issues and have proposed excellent detection approaches [39].

However, the implementation of automatic intelligent control in textile enterprises still faces several problems. First, training a detection model is difficult because there are many different fabric defects but few available samples, which can ultimately lead to higher detection errors. Second, the post-processing strategies for defect detection need to be improved, as enterprises are concerned with how to effectively handle and categorize the detected defective fabrics to maximize their residual value. Finally, the lack of mature and comprehensive examples of real-world deployments makes it difficult for enterprises to determine the feasibility and cost of intelligently transforming their production lines.

Our contributions are summarized as follows.

1) We propose a fabric defect detection system that integrates an optimized region with convolutional neural network (CNN) features (i.e., Faster R-CNN) to improve the detection accuracy.

2) We design a comprehensive scoring mechanism based on the type, area, and direction of fabric defects, and present a defective fabric post-processing scheme based on orbital conveyor transfer and stencil replenishment.

3) We develop an anomaly detection mechanism based on defect continuity evaluation. If the detection system detects a large area of continuous defects, or if the detection value exceeds the alarm threshold, a stop command is sent to the production line control system, and an audible visual alarm is triggered.

4) We propose an algorithm for automatically recommending a clipping region division scheme, which effectively addresses the current reliance on manual segmentation for the reuse of defective fabric.

5) We have built a low-cost, real-time vision system for fabric defect detection and post-processing, and developed the supporting industrial software.

2 Related work

2.1 Research progress in fabric defect detection

In 2016, Hanbay et al. [10] provided a detailed overview of fabric defects and proposed seven methods to detect them. Traditional machine vision methods [1113] and the emission spectra and statistical methods have been used to identify defective areas and count defects prior to cutting the fabric. However, these methods have difficulty detecting random changes in fabric surface texture and are easily affected by lighting conditions, resulting in low accuracy in industrial environments.

In recent years, the development of graphics processing units (GPUs) has led to significant advances in fabric defect detection based on deep learning techniques. For instance, YOLO [14], Faster R-CNN [15], and other models and algorithms have not only achieved better performance but also have the ability to identify defect types and their spatial locations compared to traditional machine vision methods. However, deep learning algorithms require powerful GPUs, which increases the difficulty and cost of their deployment in practical industrial applications.

To balance costs and benefits, researchers have to reconsider the use of low-cost and rapid computer vision technology in practical industrial applications. Khodier et al. [16] used a combination of CNNs to detect defects in jacquard fabrics with complex patterns and achieved an accuracy of approximately 99%. Wang et al. [17] proposed a double-branch parallel Faster R-CNN model for multi-category defect detection in practical applications, which could handle a variety of fabric defects and achieved a mean of average precision (mAP) of 0.574 (mAP@50).

2.2 Challenges and opportunities

Although there have been significant advances in fabric defect detection, traditional machine vision methods are still widely used in the textile industry [1820]. Manufacturers are increasingly demanding fabric defect detection systems that not only count the number of defects but also evaluate and process defective fabrics, which can be used to guide improvements in the manufacturing process. These requirements pose new challenges for vision technology, while offering new opportunities for the development of intelligent manufacturing.

To address these challenges, we design a comprehensive evaluation standard based on defect type, area, and direction, as shown in Fig. 1, and propose an algorithm that can automatically recommend region divisions for clipping and grading the defective fabric. This standard is also used for anomaly detection. All these functions are adjusted to meet the real needs of enterprises and have been successfully deployed in hardware modules.

Figure 1
figure 1

Illustration of the proposed fabric defect detection scheme

3 Detection model

We report three design improvements based on the classic Faster R-CNN [21].

1) Multi-source datasets. We primarily use open-source datasets, including the smart diagnosis of cloth flaw dataset from Tianchi [22, 23] (main dataset), the fabric defect dataset from Kaggle [24, 25] (fine-tuned dataset A) and the additional images of fabric with defects acquired in this work (fine-tuned dataset B). Based on the cluster analysis results of defect size and cause, the original 30 categories are merged into 14 categories to simplify the detection process, which is also a common practice in related work [15, 17].

2) Double-stream training (DST). To detect small targets more efficiently, ResNet50 [26] is used in the backbone neural network instead of VGG-16. Unlike the scheme proposed by Wang et al. [17], both the mainstream and substream network of the Faster R-CNN are trained on the main and fine-tuned datasets, respectively, to compute a mixed score to effectively balance the detection effects of large and small defects. CiouLoss [27] is used to replace L1Loss to improve the detection efficiency of defects with different aspect ratios. In addition, NMS [28] is employed to efficiently remove highly overlapping predicted boxes with relatively inaccurate calibrations, 16-bit floating point (FP16) [29] is utilized to reduce GPU memory consumption, and online hard example mining (OHEM) [30] is applied to improve the model’s ability to learn from difficult cases. Various loss functions are attempted to enhance the performance of the detection model.

3) Deep learning application of model output. The outputs of the detection model, such as the type, bounding box (bbox) position, and the confidence level, are utilized for further processing. Using the bbox position, the area and direction of each defect are first calculated and incorporated into the proposed scoring mechanism. Then, based on this scoring mechanism, anomaly detection and clipping region division are proposed and implemented. These algorithms and the hardware components are integrated into the proposed system.

The workflow of the proposed fabric defect detection is shown in Fig. 2. The main dataset contains many pairs of images. In each pair, there are fabrics with the same pattern design. In one image, the fabric has defects, while in the other image, the fabric has no defects. We first use the mainstream network of the Faster R-CNN to train our model, and then use the substream network of the Faster R-CNN to train the model with small defects in the fine-tuned dataset. Finally, the rating results of the two models are mixed, and the defect type with the highest confidence value is returned. In addition, Fig. 3(a) displays the number of defects contained in the dataset and Fig. 3(b) shows the comparison between the two datasets, the main dataset and the fine-tuned dataset.

Figure 2
figure 2

The workflow of the proposed fabric defect detection method and the rearranged fabric dataset in this work. For the main dataset, we integrate multiple fabric defect datasets [22, 23] into a new dataset. The fine-tuned dataset A consists of short lines and holes from the Kaggle’s fabric dataset [24, 25]. The fine-tuned dataset B consists of contaminations and short watermarks collected in this work. The CoCo stands for common objects in context. It is a dataset format that includes annotated images with object segmentation masks, bounding boxes, and labels. The VGG16 stands for visual geometry group 16-layer, the Faster R-CNN is the faster region-based convolutional neural network, and the ResNet is the residual network

Figure 3
figure 3

Details of our dataset. Conta: contaminations; Misp: misprints; Water: watermarks; Flora: floral_hairs; SHs: seam_heads; SHPs: seam_head_prints; Bug: bug_stick; Hos: holes; Fos: folds; WDs: weaving_defects; Omis: omissions; WaxS: wax_spots; Color: color_differences; Mesh: mesh_folding

4 Fabric defect scoring mechanism

From a technical perspective, training an artificial intelligent (AI) application model is only one step in the engineering process. Enterprises are not usually concerned with the type of detection models (Faster R-CNN, Cascade R-CNN or YOLO) used. In contrast, they are more concerned with having a feasible and cost-effective solution that includes hardware, software, and detection methods.

The training of a fabric defect detection model is considered a pre-task for system deployment. Applying the model with the industrial camera and the optimal use of the information returned by the detection models are areas that need to be explored. After all, the lack of accuracy of single shot detection can be compensated for by multiple shots. This shows that efficient and reasonable processing schemes are more important than high single shot detection accuracy in real industrial production. Therefore, this work proposes a scoring mechanism that provides criteria for post-processing, which is introduced in subsequent sections.

In actual production, enterprises do not always discard or recycle a defective piece of fabric. It is more economical to adopt different strategies depending on the specific situation of the defect distribution. Therefore, we have designed a comprehensive scoring criterion to evaluate the type, area, and direction (transverse or longitudinal) of fabric defects. This criterion is used as the basis for further processing of the defective fabric.

The total score of the fabric defects is denoted as \(P^{\mathrm{{Tot}}}\) and calculated as follows:

$$\begin{aligned} P^{\mathrm{{Tot}}} ={}& {\sum_{1}^{\mathit{n}}}P_{i} \\ ={}& {\sum_{1}^{\mathit{n}}}(\alpha P_{i, \text{typ}} + \beta P_{i,\text{are}} + \gamma P_{i,\text{tra}} \\ &{}+ \delta P_{i,\text{lon}}), \end{aligned}$$
(1)

where n represents the total number of detected defects, \(P_{i}\) denotes the score of the i-th detected defect, while \(P_{i,\text{typ}}\), \(P_{i,\text{are}}\), \(P_{i,\text{tra}}\), and \(P_{i,\text{lon}}\) are the scores in terms of type, area, transverse and longitudinal trend, respectively. α, β, γ and δ are the weight coefficients, whose values are negatively related to the acceptance of the corresponding defect characteristics and can be adjusted by the analytical hierarchy process or neural network according to different customer needs.

4.1 Single defect scoring

The coordinates returned by the detection model for positioning the bbox are \((x_{1},y_{1})\), \((x_{2},y_{2})\), as shown in Fig. 4. The output of the i-th detected defect is recorded as a vector (\(\boldsymbol{D}_{i}\)):

$$\begin{aligned} &\boldsymbol{D}_{i} = \begin{bmatrix} \mathrm{type}_{i}, x_{1,i}, y_{1,i}, x_{2,i}, y_{2,i} \end{bmatrix}, \end{aligned}$$
(2)

which is used to calculate the width \(\Delta x_{i}\) and length \(\Delta y_{i}\) of the defect as follows:

$$\begin{aligned} &\textstyle\begin{cases} \mathrm{\Delta} x_{i}= \vert x_{2,i}-x_{1,i} \vert , \\ \mathrm{\Delta} y_{i}= \vert y_{2,i}-y_{1,i} \vert . \end{cases}\displaystyle \end{aligned}$$
(3)
Figure 4
figure 4

An example of scoring fabric defects based on bounding box (bbox) location. The coordinates returned by the detection model for positioning the bbox are (\(x_{1}\), \(y_{1}\)), (\(x_{2}\), \(y_{2}\)). f and g indicate the resolution ratio of current image. \(N_{x}\) and \(N_{y}\) represent the transverse and longitudinal lengths, respectively. We use the bbox position to calculate the area and direction of each defect, and use the data to help us evaluate the defect. Three bounding boxes in the figure represent three defects in the fabric. For the first bounding box’s position, H = (251, 412), F = (273, 385); for the second bounding box’s position, L = (400, 400), J = (422, 367); for the third bounding box’s position, P = (246, 282), N = (392, 40). In addition, A = (0, 0), B = (640, 0), C = (640, 480), D = (0, 480)

The transverse trend score \(P_{i,\text{tra}}\) and longitudinal trend score \(P_{i,\text{lon}}\) of the defect can be obtained by normalizing its width and length. The industrial camera used in the proposed system has a direct output resolution ratio of 640 × 480, but the fabric does not occupy the entire industrial camera’s field of view. Thus, the direct resolution ratio is calculated using Eq. (4):

$$\begin{aligned} &\textstyle\begin{cases} P_{i,\text{tra}}= \mathrm{\Delta} x_{i}/N_{x}, \\ P_{i,\text{lon}}= \mathrm{\Delta} y_{i}/N_{y}, \end{cases}\displaystyle \end{aligned}$$
(4)

where \(N_{x}\) and \(N_{y}\) represent the transverse and longitudinal lengths, respectively. In addition, the score \(P_{i,\text{are}}\) of the defective area can be calculated using Eq. (5):

$$\begin{aligned} &P_{i,\text{are}} = \frac{\mathrm{\Delta} x_{i} \cdot \mathrm{\Delta} y_{i}}{N_{x} \cdot N_{y}} = P_{i,\text{tra}} \cdot P_{i,\text{lon}}. \end{aligned}$$
(5)

4.2 Video frame scoring

As shown in Fig. 4, the scores of all defects, including the three marked defects \((1, 2, 3)\) in the current frame, are calculated by Eq. (1).

However, this calculation is limited to individual frames of the video stream. In actual production, the area of the fabric exceeds the field of view of the industrial camera, requiring adaptation algorithms. Therefore, we propose a real-time scoring algorithm to calculate the total score \(P^{\text{Tot}}\) for the entire fabric.

4.3 Real-time fabric scoring mechanism

Our algorithm is based on frame-by-frame detection and scoring thresholds in ideal cases. However, practical industrial applications require a real-time detection system that can process successive frames of camera video, as the fabric on the conveyor belt moves in a direction relative to the industrial camera. In addition, the consistency of defect detection is limited to the detection model.

To solve these problems, we develop a mechanism using defect numbering and dictionary traversal. Defects are individually numbered and combined with other information to create a dictionary. Using this algorithm, the dictionary of defects between two adjacent frames is compared in real time to determine the duplication and loss of detected defects. Furthermore, the real-time scoring algorithm is combined with defect scoring and start/stop judgments to form a scoring mechanism. An example of this real-time scoring algorithm is shown in Fig. 5. By coupling this scoring mechanism with the hardware system described in the following sections, defective fabrics are evaluated more accurately.

Figure 5
figure 5

Example of a real-time defect detection scoring algorithm. The green border indicates that this defect was not detected in the previous frame. n is the index of the video frame being processed

4.4 Anomaly detection

On the production line, an abnormal condition of a knitting machine can cause major defects in the fabric. If left undetected, it can cause significant losses to the enterprise. To address this issue, we develop a more efficient scoring algorithm and an anomaly detection method based on the defect scoring density and the mechanisms introduced in the previous section. First, after defect scoring, the defective area is evaluated; in particular, the total score \(P^{\mathrm{{Tot}}}_{k}\) (k represents the k-th fabric) and the corresponding defect score density \(\rho _{k}\) (also shown in Fig. 6) are calculated for each frame. When the following conditions are satisfied, a shutdown command is sent to the control system.

Figure 6
figure 6

Recommended clipping range and quality rating. The D area indicates that a fabric defect has been found. The orange square represents the recommended clipping region, and the red square represents the clipping regions that are not recommended. A, B and C represent high quality segments, good segments, and defective fabric segments, respectively. We design an indicator called “Basis” to evaluate each clipping region, the \(S_{\text{cuttable area}}\) is the pixel square of the clippable area, and the \(S_{\text{this D area}}\) is the pixel square of the D area. \(\rho _{k}\) represents the defect score density of the k-th fabric, \(P^{\mathrm{{Tot}}}_{k}\) represents the total score of the k-th fabric. \(N_{x}\) and \(N_{y}\) represent the transverse and longitudinal lengths, respectively. k represents the k-th fabric

1) The defect score density exceeds the preset threshold

2) Any length, width, or area of a single defect exceeds its preset thresholds.

If the shutdown condition is not triggered after the timeout, the loop waits for the next detection area.

4.5 Clipping region division

The distribution of defects in different parts of a long piece of fabric on the production line can vary. To address this issue, inspired by previous work [3133], we design algorithms that recommend the division of clipping regions and evaluate the quality of each part, which can help enterprises maximize the economic value of the fabric. The experiment demonstrated the algorithms using only full-color red, yellow, and green fabrics, considering both point-like defects, such as contaminations, and line-like defects, such as watermarks, which are simpler and easier to detect with greater accuracy (Fig. 6).

When the industrial personal computer (IPC) provides the source instruction for defect processing, the system divides the fabric based on whether defects are present. Regions with defects are marked with yellow borders, while regions without defects are marked with blue borders. The previously mentioned scoring mechanism is used to provide ratings for the areas with defects. The irregular cutting of fabric is a non-deterministic polynomial (NP)-hard problem [31]; it is more practical and faster to cut and divide the fabric into rectangles in actual production. Therefore, the division of regions is simplified by solving the maximum rectangular area problem. We use orange borders to indicate recommended clipping regions and red borders to indicate non-recommended clipping regions.

Next, the system evaluates each clipping region of the fabric using different thresholds, where A, B, and C represent high quality segments, good segments, and defective fabric segments, respectively. Then, the partition information of the current fabric and the partition screenshot are logged for further classification processing.

4.6 Stencil identification and replenishment

To ensure efficient and continuous fabric production, defective fabrics must be quickly replaced with the defect-free fabrics in the same stencil to avoid disrupting the stacking sequence [33]. This section describes the stencil identification and replenishment device, which includes a stencil replenishment station, a steering device, deflectors of the stencil replenishment station, a replenishment conveyor, and controllers capable of identifying stencil of fabrics as they move. For fast fabric pattern recognition, current methods use simple classification tasks through neural network training [34], or pattern recognition through RGB histogram-based approaches [35]. The latter approach is used in this work, because providing additional models after fabric detection models will hinder the detection efficiency and speed of the system. An operation diagram of the stencil identification and replenishment device is shown in Fig. 7. Based on this approach, the corresponding execution logic in this paper is designed as follows.

Figure 7
figure 7

Schematic diagram of stencil identification and replenishment device. n indicates the number of stencils

1) When the defect score exceeds the set threshold, the IPC takes a screenshot of the camera video stream, compares its RGB channel histogram with the pre-stored pattern, calculates, and returns the ID of the pattern with the highest similarity.

2) The hardware system then releases the defect-free fabrics with the specified stencil in order as compensation, according to the identified pattern of the detected fabric.

5 Structure and hardware

Existing research on fabric defect detection has focused mainly on software implementation. Although some studies have proposed a semi-manual system [18], there has been limited exploration of hardware-based fabric defect detection and post-processing systems [36]. In industrial applications, models, software and hardware are equally important. Therefore, a fabric defect detection and post-processing system will be designed and applied in practice.

Fig. 8 shows the overall rendering effect of the system, a physical image of the system in the main view, a schematic diagram, and close-up photographs of the hardware module, and a close-up photograph of the fabric replenishment module and the user interface of the IPC program.

Figure 8
figure 8

Overall structure of our fabric defect detection and post-processing system. (a) Structural schematic diagram; (b) Real-time image of the system in the main view; (c) User interface of the industrial personal computer program; (d) Close-up photo of the fabric replenishment module; (e) Close-up photo of the hardware module; (f) Schematic diagram of the hardware module. The TFT is the thin-film transistor, the MCU is the micro-controller unit (the Node MCU is an ESP8266 based development board), the Arduino UNO is a product of Arduino. TB6600 is a commonly used stepper motor driver, which is used to control the movement of stepper motors

The electrical components and modules of the proposed system are shown in Fig. 9 and Table 1. It consists of eight major components: IPC, main processing, expansion and display, drive execution, power, alarm, fabric replenishment, and vision modules.

Figure 9
figure 9

Electrical devices and modules of the fabric defect detection and post-processing system. The IPC is the industrial personal computer

Table 1 Electrical devices and modules in the proposed system. “DIY” means that the device is self-developed

1) The IPC module, which is used to run the fabric defect detection model, the IPC program, and to communicate with the hardware system, consists of a Lenovo Legion R9000P personal laptop with an R7-5800H CPU and an NVIDIA GeForce RTX 3060 Laptop (6 GB) GPU.

2) The main processing module, which is responsible for organizing all the components of the hardware system, consists of two Arduino UNOs and a CNC Shield extended board.

3) The extension and display module, consisting of a Node MCU V3 WIFI board with ESP8266 and a 1.8-inch TFT screen of model ST7735, is used for human-machine interaction and communication extension.

4) The drive execution module contains two stepper motors of model 42BYGH47, two small conveyor belts we built ourselves, three motor drivers for the stepper motor of model TB6600, three motor drivers for the sliding platform of model A4988 and a light-sensitive sensor of model LM393. This module is used to receive control commands from the main processing module and drive the motors.

5) The power module provides different voltage requirements, using three power supply units: MW S-150-24 (150 W 24 V), MW S-100-12 (100 W 12 V), and MW S-50-5 (50 W 5 V).

6) The alarm module consists of an LTE-5061 alarm light and an FL-3FF-S-Z 5 V DC relay, providing hardware support for anomaly detection algorithms.

7) The fabric replenishment module consists of a PCA9685 servo controller board and three MG90 servo motors (which can be rotated from 90 degrees to 180 degrees). This module provides hardware support for stencil identification and replenishment for the post-processing session.

8) The vision module includes a HW500 industrial camera (5 million pixels, 30 frames per second, \(640\times 480\) direct output resolution ratio) and a self-developed 3D sliding platform. The industrial camera can capture moving images and centralize them via the controller and IPC.

6 Experiments and discussion

6.1 Experimental configuration

The rearranged fabric dataset used in this study is divided into main and fine-tuned datasets. Each part is further divided into training, verification, and test sets in the ratio of 6:2:2. The fine-tuned Faster R-CNN is used for training. The details of the training methods can be found in the introduction for detection models, and the training environment is as follows: Python 3.7.6, PyTorch 1.11.0+cu115, CuDNN 8.3.2, TorchVision 0.12.0+cu115, OpenCV 4.5.2, MMCV 1.4.8, and MMDetection 2.23.0.

The detection and post-processing system is deployed on a personal laptop with an R7-5800H CPU and an NVIDIA GeForce RTX 3060 Laptop (6 GB) GPU. With the CPU-assisted computation enabled, the system processes five frames per second of real-time detection video. Moreover, a monitoring software is developed using QT5 and connected to the hardware system via a serial port. The user interface of this software includes various functions, such as industrial camera video stream output, defect detection video stream output and industrial camera RGB adjustment, as shown in Fig. 8(c).

6.2 Model evaluation

6.2.1 Performance comparison

The improved model is validated on a rearranged dataset and evaluated using the mean average precision (mAP). The proposed model achieves promising performance, especially in reducing the GPU memory cost (during model training) and improving the small target detection accuracy, as shown in Table 2. Compared to those of the baseline, the values of mAP@0.5:0.95, mAP@0.5, and mAP_s@0.5:0.95 increase by 5.4%, 2%, and 9.1%, respectively.

Table 2 Ablation experiment of the proposed method. mAP@0.5:0.95, mAP@0.5, mAP@0.75, mAP_s@0.5:0.95, mAP_m@0.5:0.95, and mAP_l@0.5:0.95 represent the mAP values at different thresholds. For example, mAP@0.5:0.95 represents the average mAP value calculated at intersection over union (IoU) thresholds ranging from 0.5 to 0.95, with a step size of 0.05. mAP_s@0.5:0.95, mAP_m@0.5:0.95, and mAP_l@0.5:0.95 refer to small, medium, and large objects, respectively. The best results are marked in bold

As shown in Table 2, replacing L1Loss of the baseline with CiouLoss increases the mAP@0.5:0.95 by 2.4%. With NMS and FP16, the mAP@0.5:0.95 value improves by 2% compared to that of the baseline, and the GPU memory cost decreases by 1.37 GB, reducing the GPU workload. Compared to the optimized models with NMS and FP16, using OHEM and replacing L1Loss with CiouLoss increases the mAP@0.5:0.95 value by 1%. Using the DST as a training strategy and following the optimization methods above, the mAP@0.5:0.95 score improves by 2.4% compared to not using this method. Overall, the mAP@0.5:0.95 score increases by 5.4% compared to the baseline.

6.2.2 Ablation experiment

The performance of the proposed model is significantly enhanced when the FP16 + NMS + CiouLoss + OHEM + DST method is used simultaneously, as demonstrated in Table 3. In addition to evaluating the performance of the model on the dataset, we also include a live test session to determine the real-time frame rate for detection on the proposed system, including FPS_1, FPS_2 and FPS_3. FPS_1 refers to the frames per second (FPS) in the user interface of the IPC program when both the VSOIC and the VSODD modules are used. FPS_2 refers to the FPS when VSOIC is enabled but VSODD is disabled. FPS_2 refers to the FPS when both VSOIC and VSODD are disabled.

Table 3 Performance comparison of different algorithms and models. “-” indicates that the data are not provided or that this metric is not used. The best results are marked in bold

Compared to the studies of Wang et al. [17] and Zhou et al. [15], the network and the strategy proposed in this work achieve better mAP values on the dataset. However, it is not possible to compare their detection speeds on the system proposed in this work because the model of the above work is not open source. In addition, two-stage networks such as the Faster R-CNN cannot compete with networks such as YOLO and Retinanet in terms of detection speed, real-time frame rate, and GPU memory cost (during deployment). In particular, some decision algorithms of the detection system are executed directly at the output of the model. Although this slows down the detection speed of the models, we still consider the introduction of the metric “FPS” (used in the proposed system, including FPS_1, FPS_2, and FPS_3) in the model performance comparison phase to be significant and in line with the original intent of our work.

6.3 Function demonstration

According to the degree of defects, the fabric to be detected is divided into defect-free fabric, defective fabric, and fabric with large-area defects, and the corresponding post-processing process is designed for three situations, as shown in Fig. 10. In addition, the optimization of the scheduling and the time consumption of each process are implemented, so that the system is more stable, and the operation logic is self-consistent. The time consumption and scheduling of the functional demonstrations are shown in Fig. 11.

Figure 10
figure 10

Flowchart of the proposed system for processing defect-free fabric, defective fabric, and fabric with large area defects

Figure 11
figure 11

Time consumption and scheduling of each process in the functional demonstration of the hardware system

6.3.1 Anomaly detection

When a piece of fabric that exceeds the defect score alarm threshold is fed to the bottom conveyor, the IPC successfully sends an alarm command to the alarm light via the development board, causing all conveyors to shut down and display a warning tip in the user interface. When the operator clicks the “module disable” button on the large-area defect automatic shutdown module in the user interface, the module can be reset, and the entire system can resume its operation. Fig. 12(a) shows the operation and results of the system. As a result, this anomaly detection module can effectively solve the problem of large defects caused by abnormal conditions of a knitting machine by using vision technology-based automatic shutdown.

Figure 12
figure 12

Fabric image, defect identification, evaluation and processing on (a) shows the test of anomaly detection. There is no stencil identification, and the score is 265. (b) shows the test of the defective fabric for stencil 1. The result of the stencil identification is stencil 1 with a score of 150. (c) displays the test of the defective fabric for stencil 4. The result of the stencil identification is stencil 4 with a score of 125. (d) displays the test of defect-free fabric. There is no stencil identification, and the score is 10

6.3.2 Defect detection and post-processing

The experimental results for defective fabrics with different stencils are shown in Fig. 12(b) and Fig. 12(c). During the defective fabric detection experiment, the user interface returns the current fabric score, and the system enters the defective fabric processing stage. At the same time, a release command is sent to the fabric replenishment module, and the pattern number and information are displayed on the user interface. The replenishment module replenishes the defect-free fabric recycle station with defect-free fabric in same stencil. Meanwhile, the defective fabric recycle conveyor transfers the defective fabric itself to the defective fabric replenishment station.

The results of a defect-free fabric delivery experiment are shown in Fig. 12(d). The user interface displays the current fabric defect score without activating the defect processing module, and the fabric continues to pass through the conveyor as normal. The fabric replenishment module does not receive an “enable” command. As a result, this system is shown to be capable of detecting and processing defective full-color fabrics.

6.3.3 Multi-form information feedback

The system also provides a review function for fabrics with detected defects, which enables the output of detection results in the form of video streams, screenshots, GIFs, and logs. In addition, clipping region division experiments and defective fabric grading experiments are performed on the defective fabrics tested in the above experiments.

7 Conclusion and future work

Due to complex scenarios and high market demands that drive continuous research efforts, fabric defect detection and post-processing offer promising computer vision applications. We propose a fabric defect detection and post-processing system that integrates an optimized Faster R-CNN model for defect detection, defect localization and detection model evaluation. In addition, the system incorporates new approaches, such as a rearranged fabric dataset, anomaly detection, recommended clipping region division, and a replenishment device. Meeting the requirements of practical industrial production can be more valuable than incremental improvement, so computer vision and automation technology have been combined here, to bridge scientific research and industrial production.

However, resolving the conflict between detecting small defects and large receptive fields is still a challenging task. The efficiency of real-time detection needs to be improved, since \(5\sim 10\) frames per second is not enough for fabric manufacturing. In addition, the feedback of the results provided by the proposed system to the machine operators also needs to be improved. In future research, we will explore potential solutions for companies to address the above issues, improve the optimization of the system’s operational logic and scheduling scheme, and develop new features such as defect tagging (which uses additional hardware modules to provide feedback at the physical level by attaching tags to defects). To encourage further research in this area, our approach is available as open source on GitHub at https://github.com/linhuaizhou/yida_gedc_fabric4show.