Abstract
Defect inspection, also known as defect detection, is significant in mobile screen quality control. There are some challenging issues brought by the characteristics of screen defects, including the following: (1) the problem of interclass similarity and intraclass variation, (2) the difficulty in distinguishing low contrast, tiny-sized, or incomplete defects, and (3) the modeling of category dependencies for multi-label images. To solve these problems, a graph reasoning module, stacked on a classification module, is proposed to expand the feature dimension and improve low-quality image features by exploiting category-wise dependency, image-wise relations, and interactions between them. To further improve the classification performance, the classifier of the classification module is redesigned as a cosine similarity function. With the help of contrastive learning, the classification module can better initialize the category-wise graph of the reasoning module. Experiments on the mobile screen defect dataset show that our two-stage network achieves the following best performances: 97.7% accuracy and 97.3% F-measure. This proves that the proposed approach is effective in industrial applications.
摘要
缺陷检测是手机屏质量控制的重要环节。手机屏缺陷的特性带来了一些具有挑战性的问题, 包括: (1)类间相似性和类内差异性; (2)低对比度、微小尺寸或不完整缺陷的识别带来的困难; (3)针对多标签图像的类别相关性建模。为了解决这些问题, 本文提出一种图推理模块, 它可以堆放在常规的分类模块上。该推理模块利用类别间的依赖性、图像间的关系以及类别图像之间的相互作用来扩展特征维度, 并且达到改进低质量图像特征的目的。为了进一步提高分类性能, 分类模块的分类器被设计为一个余弦相似度函数。在对比学习的帮助下, 分类模块可以更好地初始化推理模块的类别图。在手机屏缺陷数据集上的实验表明, 所提出的两阶段网络取得了最佳性能: 准确率为97.7%, F-measure为97.3%。这证明了本文所提出的方法在工业应用中是有效的。
This is a preview of subscription content, log in via an institution to check access.
References
Bottou L, 2010. Large-scale machine learning with stochastic gradient descent. Proc 19th Int Conf on Computational Statistics, p.177–186. https://doi.org/10.1007/978-3-7908-2604-3_16
Chen T, Kornblith S, Norouzi M, et al., 2020. A simple framework for contrastive learning of visual representations. Proc 37th Int Conf on Machine Learning, p.1597–1607.
Chen ZM, Wei XS, Wang P, et al., 2019. Multi-label image recognition with graph convolutional networks. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.5177–5186. https://doi.org/10.1109/CVPR.2019.00532
Gidaris S, Komodakis N, 2018. Dynamic few-shot visual learning without forgetting. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.4367–4375. https://doi.org/10.1109/CVPR.2018.00459
Haurum JB, Moeslund TB, 2021. Sewer-ML: a multi-label sewer defect classification dataset and benchmark. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.13451–13462. https://doi.org/10.1109/CVPR46437.2021.01325
He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.770–778. https://doi.org/10.1109/CVPR.2016.90
He KM, Fan HQ, Wu YX, et al., 2020. Momentum contrast for unsupervised visual representation learning. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.9726–9735. https://doi.org/10.1109/CVPR42600.2020.00975
Hjelm RD, Fedorov A, Lavoie-Marchildon S, et al., 2019. Learning deep representations by mutual information estimation and maximization. Proc 7th Int Conf on Learning Representations.
Khosla P, Teterwak P, Wang C, et al., 2020. Supervised contrastive learning. Proc 34th Conf on Neural Information Processing Systems, p.18661–18673.
Kong YH, Liu X, Zhao ZB, et al., 2022. Bolt defect classification algorithm based on knowledge graph and feature fusion. Energy Rep, 8(Suppl 1):856–863. https://doi.org/10.1016/j.egyr.2021.11.127
Lei J, Gao X, Feng ZL, et al., 2018. Scale insensitive and focus driven mobile screen defect detection in industry. Neurocomputing, 294:72–81. https://doi.org/10.1016/j.neucom.2018.03.013
Li CS, Zhang XM, Huang YJ, et al., 2020. A novel algorithm for defect extraction and classification of mobile phone screen based on machine vision. Comput Ind Eng, 146: 106530. https://doi.org/10.1016/j.cie.2020.106530
Lu Y, Ma L, Jiang HQ, 2020. A light CNN model for defect detection of LCD. In: Hung JC, Yen NY, Chang JW (Eds.), Frontier Computing. Springer, Singapore, p.10–19. https://doi.org/10.1007/978-981-15-3250-4_2
Park JY, Hwang Y, Lee D, et al., 2020. MarsNet: multi-label classification network for images of various sizes. IEEE Access, 8:21832–21846. https://doi.org/10.1109/ACCESS.2020.2969217
Paszke A, Gross S, Chintala S, et al., 2017. Automatic differentiation in PyTorch. Proc 31st Conf on Neural Information Processing Systems.
Simonyan K, Zisserman A, 2015. Very deep convolutional networks for large-scale image recognition. Proc 3rd Int Conf on Learning Representations.
Szegedy C, Vanhoucke V, Ioffe S, et al., 2016. Rethinking the inception architecture for computer vision. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2818–2826. https://doi.org/10.1109/CVPR.2016.308
Wang T, Zhang C, Ding RW, et al., 2021. Mobile phone surface defect detection based on improved faster R-CNN. Proc 25th Int Conf on Pattern Recognition, p.9371–9377. https://doi.org/10.1109/ICPR48806.2021.9412119
Wang Y, He DL, Li F, et al., 2020. Multi-label classification with label graph superimposing. Proc AAAI Conf Artif Intell, 34(7):12265–12272. https://doi.org/10.1609/aaai.v34i07.6909
Wang YC, Gao L, Li XY, et al., 2021a. A new graph-based method for class imbalance in surface defect recognition. IEEE Trans Instrum Meas, 70:5007816. https://doi.org/10.1109/TIM.2021.3063755
Wang YC, Gao L, Gao YP, et al., 2021b. A new graph-based semi-supervised method for surface defect classification. Rob Comput Integr Manuf, 68:102083. https://doi.org/10.1016/j.rcim.2020.102083
Wei B, Hao KR, Gao L, et al., 2021. Bioinspired visual-integrated model for multilabel classification of textile defect images. IEEE Trans Cognit Dev Syst, 13(3):503–513. https://doi.org/10.1109/TCDS.2020.2977974
Wu ZR, Xiong YJ, Yu SX, et al., 2018. Unsupervised feature learning via non-parametric instance discrimination. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.3733–3742. https://doi.org/10.1109/CVPR.2018.00393
Xiao WW, Song KC, Liu J, et al., 2022. Graph embedding and optimal transport for few-shot classification of metal surface defect. IEEE Trans Instrum Meas, 71:5010310. https://doi.org/10.1109/TIM.2022.3169547
Xu H, Jiang CH, Liang XD, et al., 2019. Reasoning-RCNN: unifying adaptive global reasoning into large-scale object detection. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.6412–6421. https://doi.org/10.1109/CVPR.2019.00658
Yuan ZC, Zhang ZT, Su H, et al., 2018. Vision-based defect detection for mobile phone cover glass using deep neural networks. Int J Precis Eng Manuf, 19(6):801–810. https://doi.org/10.1007/s12541-018-0096-x
Zhang JB, Su H, Zou W, et al., 2021. CADN: a weakly supervised learning-based category-aware object detection network for surface defect detection. Patt Recogn, 109:107571. https://doi.org/10.1016/j.patcog.2020.107571
Zhao JW, Yan K, Zhao YF, et al., 2021. Transformer-based dual relation graph for multi-label image recognition. Proc IEEE/CVF Int Conf on Computer Vision, p.163–172. https://doi.org/10.1109/ICCV48922.2021.00023
Zhou BL, Khosla A, Lapedriza A, et al., 2016. Learning deep features for discriminative localization. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2921–2929. https://doi.org/10.1109/CVPR.2016.319
Zhu Y, Ding RW, Huang WB, et al., 2022. HMFCA-Net: hierarchical multi-frequency based channel attention net for mobile phone surface defect detection. Patt Recogn Lett, 153:118–125. https://doi.org/10.1016/j.patrec.2021.11.029
Author information
Authors and Affiliations
Corresponding author
Additional information
Project supported by the National Key Research and Development Program of China (No. 2020AAA0108302) and the Fundamental Research Funds for the Central Universities, China (No. xtr072022001)
Contributors
Chaofan ZHOU designed the research. Chaofan ZHOU and Meiqin LIU processed the data. Chaofan ZHOU drafted the paper. Senlin ZHANG helped organize the paper. Ping WEI and Badong CHEN revised and finalized the paper.
Compliance with ethics guidelines
Chaofan ZHOU, Meiqin LIU, Senlin ZHANG, Ping WEI, and Badong CHEN declare that they have no conflict of interest.
Data availability
Due to the nature of this research, participants of this study do not agree for mobile screen defect data to be shared publicly, so the supporting data are not available. The data used to validate the model generalization performance (i.e., the steel surface defect data) can be found on the web links given in this paper.
Rights and permissions
About this article
Cite this article
Zhou, C., Liu, M., Zhang, S. et al. A graph-based two-stage classification network for mobile screen defect inspection. Front Inform Technol Electron Eng 24, 203–216 (2023). https://doi.org/10.1631/FITEE.2200524
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2200524