Skip to main content
SpringerLink
Log in

Tensor pooling-driven instance segmentation framework for baggage threat recognition

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Automated systems designed for screening contraband items from the X-ray imagery are still facing difficulties with high clutter, concealment, and extreme occlusion. In this paper, we addressed this challenge using a novel multi-scale contour instance segmentation framework that effectively identifies the cluttered contraband data within the baggage X-ray scans. Unlike standard models that employ region-based or keypoint-based techniques to generate multiple boxes around objects, we propose to derive proposals according to the hierarchy of the regions defined by the contours. The proposed framework is rigorously validated on three public datasets, dubbed GDXray, SIXray, and OPIXray, where it outperforms the state-of-the-art methods by achieving the mean average precision score of 0.9779, 0.9614, and 0.8396, respectively. Furthermore, to the best of our knowledge, this is the first contour instance segmentation framework that leverages multi-scale information to recognize cluttered and concealed contraband data from the colored and grayscale security X-ray imagery.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price includes VAT (Finland)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Availability of data and materials

All the datasets that have been used in this article are publicly available.

Notes

  1. The source code of the proposed framework along with its complete documentation is available at https://github.com/taimurhassan/tensorpooling.

References

  1. Tang Z, Tian E, Wang Y, Wang L, Yang T (2020) Nondestructive defect detection in castings by using spatial attention bilinear convolutional neural network. IEEE Trans Ind Inf 17:82–89

    Article  Google Scholar 

  2. Bastan M, Byeon W, Breuel T (2013) Object recognition in multi-view dual energy X-ray images. In: British machine vision conference

  3. Turcsany D, Mouton A, Breckon TP (2013) Improving feature-based object recognition for X-ray baggage security screening using primed visual words. In: 2013 IEEE international conference on industrial technology (ICIT). IEEE, pp 1140–1145

  4. Hu B, Zhang C, Wang L, Zhang Q, Liu Y (2020) Multi-label X-ray imagery classification via bottom-up attention and meta fusion. In: Asian conference on computer vision (ACCV)

  5. Akçay S et al (2016) Transfer learning using convolutional neural networks for object classification within X-ray baggage security imagery. In: IEEE ICIP, pp 1057–1061

  6. Akcay S et al (2018) Using deep convolutional neural network architectures for object classification and detection within X-ray baggage security imagery. IEEE Trans Inf Forensics Secur 13(9):2203–2215

    Article  Google Scholar 

  7. Gaus YFA et al (2019) Evaluation of a dual convolutional neural network architecture for object-wise anomaly detection in cluttered X-ray security imagery. In: 2019 international joint conference on neural networks (IJCNN), pp 1–8

  8. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: IEEE international conference on computer vision (ICCV), pp 2961–2969

  9. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell

  10. Lin TY et al (2017) Focal Loss for Dense Object Detection. In: IEEE international conference on computer vision and pattern recognition (CVPR)

  11. Redmon J, Farhadi A (2018) YOLOv3: an incremental improvement

  12. Wei Y et al (2020) Occluded prohibited items detection: an X-ray security inspection benchmark and de-occlusion attention module

  13. Miao C et al (2019) SIXray: a large-scale security inspection X-ray benchmark for prohibited item discovery in overlapping images. In: IEEE international conference on computer vision and pattern recognition (CVPR), pp 2119–2128

  14. Akçay S, Breckon T (2020) Towards automatic threat detection: a survey of advances of deep learning within X-ray security imaging. Preprint arXiv:2001.01293

  15. Gaus YFA et al (2019) Evaluating the transferability and adversarial discrimination of convolutional neural networks for threat object detection and classification within X-ray security imagery. arXiv preprint arXiv:1911.08966

  16. Hassan T et al (2020) Detecting prohibited items in X-ray images: a contour proposal learning approach. In: Accepted in 27th IEEE international conference on image processing (ICIP)

  17. Hassan T, Werghi N (2020) Trainable structure tensors for autonomous baggage threat detection under extreme occlusion. In: Asian conference on computer vision (ACCV), September

  18. Hassan T, Shafay M, Akçay S, Khan S, Bennamoun M, Damiani E, Werghi N (2020) Meta-transfer learning driven tensor-shot detector for the autonomous localization and recognition of concealed baggage threats. In: MDPI sensors, November

  19. Mery D, Svec E, Arias M (2016) Object recognition in baggage inspection using adaptive sparse representations of X-ray images. In: Pacific-Rim Symposium on image and video technology, pp 709–720

  20. Bastan M et al (2013) Object recognition in multi-view dual energy X-ray images. In: BMVC, vol 1, p 11

  21. Akçay S, Atapour-Abarghouei A, Breckon TP (2019) Skip-GANomaly: skip connected and adversarially trained encoder-decoder anomaly detection. In: International joint conference on neural networks (IJCNN)

  22. Mery D, Svec E, Arias M, Riffo V, Saavedra JM, Banerjee S (2017) Modern computer vision techniques for X-ray testing in baggage inspection. IEEE Trans Syst Man Cybern Syst 47(4):682–692

    Article  Google Scholar 

  23. Heitz G, Chechik G (2010) Object separation in X-ray image sets. In: IEEE international conference on computer vision and pattern recognition (CVPR), pp 2093–2100

  24. Bastan M (2015) Multi-view object detection in dual-energy X-ray images. Mach Vis Appl 25:1045–1060

    Article  Google Scholar 

  25. Kundegorski ME et al (2016) On using feature descriptors as visual words for object detection within X-ray baggage security screening. In: IEEE international conference on imaging for crime detection and prevention (ICDP)

  26. Bastan M, Yousefi MR, Breuel TM (2011) Visual words on baggage X-ray images. In: 14th international conference on computer analysis of images and patterns

  27. Riffo V, Mery D (2016) Automated detection of threat objects using adapted implicit shape model. IEEE Trans Syst Man Cybern Syst 46(4):472–482

    Article  Google Scholar 

  28. Liu Z, Li J, Shu Y, Zhang D (2018) Detection and recognition of security detection object based on YOLO9000. In: 2018 5th international conference on systems and informatics (ICSAI). IEEE, pp 278–282

  29. Xu M et al (2018) Prohibited item detection in airport X-ray security images via attention mechanism based CNN. In: Chinese conference on pattern recognition and computer vision, pp 429–439

  30. Jaccard N et al (2017) Detection of concealed cars in complex cargo X-ray imagery using deep learning. J X-ray Sci Technol 25:323–339

    Article  Google Scholar 

  31. Griffin LD, Caldwell M, Andrews JTA, Bohler H (2019) Unexpected item in the bagging area: anomaly detection in X-ray security images. IEEE Trans Inf Forensics Secur 14:1539–1553

    Article  Google Scholar 

  32. Zou L, Yusuke T, Hitoshi I (2018) Dangerous objects detection of X-ray images using convolution neural network. In: Security with intelligent computing and big-data services

  33. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: IEEE international conference on computer vision and pattern recognition (CVPR)

  34. Redmon J, Farhadi A (2017) YOLO9000: better, faster, stronger. In: IEEE international conference on computer vision and pattern recognition (CVPR)

  35. Szegedy C et al (2015) Going deeper with convolutions. In: IEEE international conference on computer vision and pattern recognition (CVPR)

  36. An J, Zhang H, Zhu Y, Yang J (2019) Semantic segmentation for prohibited items in baggage inspection. In: International conference on intelligence science and big data engineering. Visual data engineering, pp 495–505

  37. Xiao H et al (2018) R-PCNN method to rapidly detect objects on THz images in human body security checks In: IEEE martworld, ubiquitous intelligence & computing, advanced & trusted computing, scalable computing & communications. Cloud & big data computing, internet of people and smart city innovation

  38. Dhiraj KD (2019) An evaluation of deep learning based object detection strategies for threat object detection in aggage security imagery. Pattern Recognit Lett 120:112–119

    Article  Google Scholar 

  39. Mery D et al (2015) GDXray: the database of X-ray images for nondestructive testing. J Nondestruct Eval 34(4):42

    Article  Google Scholar 

  40. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE international conference on computer vision and pattern recognition (CVPR)

  41. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  42. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \(<\)0.5MB model size. arXiv preprint arXiv:1602.07360

  43. Dai J, Li Y, He K, Sun J (2016) R-FCN: object detection via region-based fully convolutional networks. In: Advances in neural information processing systems, pp 379–387

  44. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems

  45. Akçay S, Atapour-Abarghouei A, Breckon TP (2018) GANomaly: semi-supervised anomaly detection via adversarial training. In: Asian conference on computer vision. Springer, pp 622–637

  46. Zeiler MD (2012) ADADELTA: an adaptive learning rate method. arXiv:1212.5701

  47. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. arXiv:1505.04597

  48. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: IEEE international conference on computer vision and pattern recognition (CVPR), pp 3431–3440

  49. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: IEEE international conference on computer vision and pattern recognition (CVPR), pp 2881–2890

  50. Tian Z, Shen C, Chen H, He T (2019) FCOS: fully convolutional one-stage object detection. In: IEEE international conference on computer vision (CVPR)

  51. Liu W et al (2016) SSD: single shot multibox detector. In: European conference on computer vision

Download references

Funding

This work is supported by a research fund from ADEK (Grant Number: AARE19-156) and Khalifa University (Grant Number: CIRA-2019-047).

Author information

Authors and Affiliations

  1. Center for Cyber-Physical Systems (C2PS), Department of Electrical Engineering and Computer Sciences, Khalifa University, Abu Dhabi, United Arab Emirates

    Taimur Hassan & Naoufel Werghi

  2. Department of Computer Sciences, Durham University, Durham, UK

    Samet Akçay

  3. Department of Computer Science and Software Engineering, The University of Western Australia, Perth, Australia

    Mohammed Bennamoun

  4. Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

    Salman Khan

Authors
  1. Taimur Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Samet Akçay
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammed Bennamoun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Salman Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Naoufel Werghi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

TH formulated the idea, wrote the manuscript, and performed the experiments. SA improved the initial design of the framework and contributed to manuscript writing. MB co-supervised the whole research and reviewed the manuscript and experiments. SK reviewed the manuscript and experiments and improved the manuscript writing. NW supervised the whole research, contributed to manuscript writing, and reviewed the experimentation.

Corresponding author

Correspondence to Taimur Hassan.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to this article.

Financial and non-financial interests

All the authors declare that they have no financial or non-financial interests to disclose for this article.

Employment

The authors conducted this research during their employment in the following institutes: (1) T. Hassan (Khalifa University, UAE), (2) S. Akçay (Durham University, UK), (3) M. Bennamoun (The University of Western Australia, Australia), (4) S. Khan (Mohamed bin Zayed University of Artificial Intelligence, UAE), and (5) N. Werghi (Khalifa University, UAE).

Ethical approval

All the authors declare that no prior ethical approval was required from their institutes to conduct this research.

Consent for participate and publication

All the authors declare that no prior consent was needed to disseminate this article as there were no human (or animal) participants involved in this research.

Code availability

The source code of the proposed framework is released publicly on GitHub1.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hassan, T., Akçay, S., Bennamoun, M. et al. Tensor pooling-driven instance segmentation framework for baggage threat recognition. Neural Comput & Applic 34, 1239–1250 (2022). https://doi.org/10.1007/s00521-021-06411-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06411-x

Keywords

Search

Navigation