Abstract
Identifying potential threats concealed within the baggage is of prime concern for the security staff. Many researchers have developed frameworks that can automatically detect baggage threats from security X-ray scans. However, to the best of our knowledge, all of these frameworks require extensive training efforts on large-scale and well-annotated datasets, which are hard to procure in the real world, especially for the rarely seen contraband items. This paper presents a novel unsupervised anomaly instance segmentation framework that recognizes baggage threats, in X-ray scans, as anomalies without requiring any ground truth labels. Furthermore, thanks to its stylization capacity, the framework is trained only once, and at the inference stage, it detects and extracts contraband items regardless of their scanner specifications. Our one-staged approach initially learns to reconstruct normal baggage content via an encoder–decoder network utilizing a proposed stylization loss function. The model subsequently identifies the abnormal regions by analyzing the disparities within the original and the reconstructed scans. The anomalous regions are then clustered and post-processed to fit a bounding box for their localization. In addition, an optional classifier can also be appended with the proposed framework to recognize the categories of these extracted anomalies. A thorough evaluation of the proposed system on four public baggage X-ray datasets, without any re-training, demonstrates that it achieves competitive performance as compared to the conventional fully supervised methods (i.e., the mean average precision score of 0.7941 on SIXray, 0.8591 on GDXray, 0.7483 on OPIXray, and 0.5439 on COMPASS-XP dataset) while outperforming state-of-the-art semi-supervised and unsupervised baggage threat detection frameworks by 67.37%, 32.32%, 47.19%, and 45.81% in terms of F1 score across SIXray, GDXray, OPIXray, and COMPASS-XP datasets, respectively.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability Statement
The proposed framework has been thoroughly evaluated on four baggage X-ray datasets, and all of these four datasets are publicly available.
Notes
The source code of the proposed framework, along with the complete documentation is available at https://github.com/taimurhassan/anomaly.
References
Akçay S, Breckon T (2020) Towards automatic threat detection: a survey of advances of deep learning within X-ray security imaging. arXiv:200101293
Akçay S et al (2016) Transfer learning using convolutional neural networks for object classification within X-ray baggage security imagery. In: IEEE ICIP, pp 1057–1061
Akçay S, Atapour-Abarghouei A, Breckon TP (2018a) GANomaly: semi-supervised anomaly detection via adversarial training. In: Asian conference on computer vision. Springer, pp 622–637
Akçay S, Kundegorski ME, Willcocks CG, Breckon TP (2018b) Using deep convolutional neural network architectures for object classification and detection within x-ray baggage security imagery. IEEE Trans Inf Forensics Secur 13(9):2203–2215
Akçay S, Atapour-Abarghouei A, Breckon TP (2019) Skip-GANomaly: skip connected and adversarially trained encoder-decoder anomaly detection. arXiv:190108954
An J et al (2019) Semantic segmentation for prohibited items in baggage inspection. In: International conference intelligence science and big data engineering. Visual data engineering, pp 495–505
Bastan M (2015) Multi-view object detection in dual-energy X-ray images. Mach Vis Appl 26:1045–1060
Bastan M et al (2011) Visual words on baggage X-ray images. In: International conference on computer analysis of images and patterns, pp 360–368
Cooley JW, Tukey JW (1965) An algorithm for the machine calculation of complex Fourier series. Math Comput 19(90):297–301
Council NR (1996) Airline passenger security screening: new technologies and implementation issues. The National Academics Press
Dumagpi JK, Jung WY, Jeong YJ (2020) A new GAN-based anomaly detection (GBAD) approach for multi-threat object classification on large-scale X-ray security images. IEICE Trans Inf Syst E103-D(2):454–458
Gaus YFA, Bhowmik N, Akçay S, Breckon T (2019a) Evaluating the transferability and adversarial discrimination of convolutional neural networks for threat object detection and classification within X-ray security imagery. In: 18th IEEE international conference on machine learning and applications (ICMLA)
Gaus YFA, Bhowmik N, Akçay S, Guillen-Garcia PM, Barker JW, Breckon TP (2019b) Evaluation of a dual convolutional neural network architecture for object-wise anomaly detection in cluttered X-ray security imagery. In: 2019 international joint conference on neural networks (IJCNN), pp 1–8
Griffin LD, Caldwell M, Andrews JTA (2019) COMPASS-XP dataset. Computational Security Science Group, UCL
Hassan T, Werghi N (2020) Trainable structure tensors for autonomous baggage threat detection under extreme occlusion. In: Asian conference on computer vision (ACCV)
Hassan T, Bettayeb M, Akçay S, Khan S, Bennamoun M, Werghi N (2020a) Detecting prohibited items in X-ray images: a contour proposal learning approach. 27th IEEE international conference on image processing (ICIP)
Hassan T, Shafay M, Akçay S, Khan S, Bennamoun M, Damiani E, Werghi N (2020b) Meta-transfer learning driven tensor-shot detector for the autonomous localization and recognition of concealed baggage threats. MDPI Sens 20(22):1–25
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
He K, Gkioxari G, Dollar P, Girshick R (2017) Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 2961–2969
Heitz G, Chechik G (2010) Object separation in X-ray image sets. In: International conference computer vision and pattern recognition, pp 2093–2100
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: efficient convolutional neural networks for mobile vision applications, pp 1–9. arXiv:1704.04861
Huang G, Liu Z, Laurens VDM, Weinberger KQ (2017) Densely connected convolutional networks. In: IEEE international conference on computer vision and pattern recognition (CVPR)
Jaccard N, Rogers TW, Griffin LD (2014) Automated detection of cars in transmission X-ray images of freight containers. In: AVSS, pp 387–392
Jaccard N et al (2017) Detection of concealed cars in complex Cargo X-ray imagery using deep learning. J X-ray Sci Technol 25(3): 323–339
Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision (ECCV)
Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Proceedings of the international conference on learning representations (ICLR)
Kundegorski M et al (2016) On using feature descriptors as visual words for object detection within x-ray baggage security screening. In: International conference on imaging for crime detection and prevention (ICDP)
Liu W et al (2016) SSD: single shot multibox detector. In: European conference on computer vision (ECCV)
Lin TY et al (2017) Focal loss for dense object detection. In: IEEE international conference on computer vision and pattern recognition (CVPR)
Liu Z, Li J, Shu Y, Zhang D (2018) Detection and recognition of security detection object based on Yolo9000. In: 2018 5th international conference on systems and informatics (ICSAI), IEEE, pp 278–282
Megherbi N, Breckon TP, Flitton GT, Mouton A (2012) Fully automatic 3D threat image projection: application to densely cluttered 3D computed tomography baggage images. In: International conference on image processing theory, tools and applications
Mery D et al (2015) GDXray: the database of X-ray images for nondestructive testing. J Nondestr Eval 34(4):42
Mery D et al (2016) Object recognition in baggage inspection using adaptive sparse representations of X-ray images. In: Pacific-Rim symposium on image and video technology, pp 709–720
Mery D, Svec E, Arias M, Riffo V, Saavedra JM, Banerjee S (2017) Modern computer vision techniques for X-ray testing in baggage inspection. IEEE Trans Syst Man Cybern: Syst 47(4):682–692
Mery D, Saavedra D, Prasad M (2020) X-ray baggage inspection with computer vision: a survey. IEEE Access 8(19974224): 145620–145633
Miao C et al (2019) SIXray: a large-scale security inspection X-ray benchmark for prohibited item discovery in overlapping images. In: IEEE CVPR, pp 2119–2128
Ren S et al (2016) Faster R-CNN: towards real-time object detection with region proposal networks. arXiv:150601497v3
Riffo V, Mery D (2015) Automated detection of threat objects using adapted implicit shape model. IEEE Trans Syst Man Cybern Syst 46(4):472–482
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition
Sun Q, Liu Y, Chua TS, Schiele B (2019) Meta-transfer learning for few-shot learning. In: IEEE international conference on computer vision and pattern recognition (CVPR)
Szegedy C et al (2014) Going deeper with convolutions. arXiv:14094842v1
Tao R, Wei Y, Li H, Liu A, Ding Y, Qin H, Liu X (2021) Over-sampling de-occlusion attention network for prohibited items detection in noisy X-ray images, pp 1–13. arXiv:2103.00809
Turcsany D, Mouton A, Breckon TP (2013) Improving feature-based object recognition for X-ray baggage security screening using primed visual words. In: 2013 IEEE international conference on industrial technology (ICIT), IEEE, pp 1140–1145
Wang Q, Breckon TP (2020) Contraband materials detection within volumetric 3D computed tomography baggage security screening imagery. arXiv:201211753
Wang Q, Megherb N, Breckon TP (2020a) Multi-class 3D object detection within volumetric 3D computed tomography baggage security screening imagery. arXiv:200801218
Wang Q, Megherbi N, Breckon TP (2020b) A reference architecture for plausible threat image projection (TIP) within 3D X-ray computed tomography volumes. J X-ray Sci Technol 28(3):507–526
Wei Y, Tao R, Wu Z, Ma Y, Zhang L, Liu X (2020) Occluded prohibited items detection: an X-ray security inspection benchmark and de-occlusion attention module. In: Proceedings of the 28th ACM International Conference on Multimedia, pp 138–146
Xu M et al (2018) Prohibited item detection in airport X-ray security images via attention mechanism based CNN. In: Chinese conference on pattern recognition and computer vision (PRCV), pp 429–439
Yang Y, Soatto S (2020) FDA: Fourier domain adaptation for semantic segmentation. In: IEEE international conference on computer vision and pattern recognition (CVPR)
Zhang J et al (2014) Joint shape and texture based X-ray Cargo image classification. In: International conference on computer vision and pattern recognition (CVPR) workshops
Zhao Z et al (2018) A GAN-based image generation method for X-ray security prohibited items. In: Chinese conference on pattern recognition and computer vision (PRCV)
Acknowledgements
This work is supported with a research fund from Khalifa University: Ref: CIRA-2019-047, and from ADEK Award for Research Excellence: AARE19-156.
Author information
Authors and Affiliations
Contributions
TH devised the idea, wrote the manuscript, and performed the experiments. SA also contributed to manuscript writing. MB co-supervised the research and reviewed the experiments. SK also reviewed the manuscript. NW supervised the complete research, contributed to manuscript writing, and reviewed the experiments.
Corresponding author
Ethics declarations
Conflict of interest
All the authors declare that there are no competing interests related to this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hassan, T., Akçay, S., Bennamoun, M. et al. Unsupervised anomaly instance segmentation for baggage threat recognition. J Ambient Intell Human Comput 14, 1607–1618 (2023). https://doi.org/10.1007/s12652-021-03383-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-021-03383-7