Skip to main content
SpringerLink
Log in

Cascaded structure tensor for robust baggage threat detection

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In the last two decades, baggage scanning has become one of the prime aviation security concerns worldwide. Manual screening of the baggage items is tedious and an error-prone process that also compromises privacy. Hence, many researchers have developed X-ray imagery-based autonomous systems to address these shortcomings. This paper presents a cascaded structure tensor framework that can automatically detect suspicious objects from the baggage X-ray scans under extreme class imbalance and irrespective of the baggage clutter. The proposed framework is unique as it intelligently extracts each object by iteratively picking its contour-based transitional information from different orientations and uses only a single feed-forward convolutional neural network for the recognition. The proposed framework has been rigorously evaluated on publicly available GDXray and SIXray datasets for detecting the highly cluttered and overlapping suspicious items, where it achieved the mean average precision score of 0.9343 and 0.9595, respectively, across both datasets, outperforming state-of-the-art works by 1.94% on the GDXray, and 8.21% on the SIXray. Furthermore, the proposed framework gives the best trade-off between detection performance and efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

All the datasets that have been used in this article are publicly available.

References

  1. Council NR (2022) Airline passenger security screening: new technologies and implementation issues. The National Academics Press, Washington, DC

    Google Scholar 

  2. Cargo Screening: technological options. Aviation Security International, Retrieved: 4 Dec 2019

  3. Miao C et al. (2019) SIXray: a large-scale security inspection X-ray benchmark for prohibited item discovery in overlapping images. In: IEEE international conference on computer vision and pattern recognition (CVPR), pp 2119–2128

  4. Mery D et al (2015) GDXray: the database of X-ray images for nondestructive testing. J Nondestr Eval 34(42):1–12

    Google Scholar 

  5. Bastan M, Byeon W, Breuel T (2013) Object recognition in multi-view dual energy X-ray images. In: British machine vision conference, pp 1–11

  6. Hassan T et al. (2020) Detecting prohibited items in x-ray images: a contour proposal learning approach. In: Accepted in 27th IEEE international conference on image processing (ICIP), pp 1–5

  7. Akçay S, Breckon T (2022) Towards automatic threat detection: a survey of advances of deep learning within X-ray security imaging. Pattern Recogn 122:1–21

    Article  Google Scholar 

  8. Bastan M (2015) Multi-view object detection in dual-energy X-ray images. Mach Vis Appl 26:1045–1060

    Article  Google Scholar 

  9. Bastan M, Yousefi M R, Breuel T M (2011) Visual words on baggage X-ray images. In: 14th international conference on computer analysis of images and patterns, pp 360–368 August 2011

  10. Turcsany D, Mouton A, Breckon TP (2013) Improving feature-based object recognition for X-ray baggage security screening using primed visual words. In: IEEE international conference on industrial technology, pp 1140–1145 Febr 25th–28th

  11. Riffo V, Mery D (2016) Automated detection of threat objects using adapted implicit shape model. IEEE Trans Syst Man Cybernet Syst 46(4):472–482

    Article  Google Scholar 

  12. Akçay S, Abarghouei AA, Breckon TP (2018) GANomaly: semi-supervised anomaly detection via adversarial training. In: Asian conference on computer vision, pp 622–637

  13. Akçay S, Abarghouei AA, Breckon TP (2019) Skip-GANomaly: skip connected and adversarially trained encoder-decoder anomaly detection. In: International joint conference on neural networks, pp 2161–2172, July 14th–19th

  14. Hassan T, Akçay S, Bennamoun M, Khan S, Werghi N (2021) Unsupervised anomaly instance segmentation for baggage threat recognition. J Ambient Intell Humaniz Comput 1:1–12

    Google Scholar 

  15. Akçay S, Kundegorski ME, Devereux M, Breckon TP (2016) Transfer learning using convolutional neural networks for object classification within X-ray baggage security imagery. In: International conference on image processing, pp 2381–2386

  16. Akçay S, Kundegorski ME, Willcocks CG, Breckon TP (2018) Using deep convolutional neural network architectures for object classification and detection within X-ray baggage security imagery. IEEE Trans Inf Forensics Secur 13(9):1556–1568

    Article  Google Scholar 

  17. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst pp 1–9

  18. Gaus YFA, Bhowmik N, Akçay S, Garcia PMG, Barker JW, Breckon TP (2019) Evaluation of a dual convolutional neural network architecture for object-wise anomaly detection in cluttered X-ray security imagery. In: The international joint conference on neural networks, pp 1–8, July 14th–19th

  19. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. Neural Inf Process Syst pp. 1–9

  20. He K, Gkioxari G, Dollar P, Girshick R (2017) Mask R-CNN. In: IEEE international conference on computer vision (ICCV), pp 1–12

  21. Lin TY, Goyal P, Girshick R, He K, Dollar P (2020) Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 42(2):1–10

    Article  Google Scholar 

  22. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition, pp 1–12

  23. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) SqueezeNet: alexNet-level accuracy with 50x fewer parameters and \(<\)0.5MB model size. arXiv preprint arXiv:1602.07360, pp 1–13

  24. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  25. Yona YFA, Bhowmik N, Akçay S, Breckon TP (2019) Evaluating the transferability and adversarial discrimination of convolutional neural networks for threat object detection and classification within X-ray security imagery. In: 18th IEEE international conference on machine learning and applications (ICMLA), pp 1–6 20 Novemb 2019

  26. Hassan T, Akcay S, Bennamoun M, Khan S, Werghi N (2022) A novel incremental learning driven instance segmentation framework to recognize highly cluttered instances of the contraband items. IEEE Trans Syst Man Cybernet Syst 52(11):6937–6951

    Article  Google Scholar 

  27. Wei Y, Tao R, Wu Z, Ma Y, Zhang L, Liu X (2020) Occluded prohibited items detection: an X-ray security inspection benchmark and de-occlusion attention module. In: ACM international conference on multimedia, pp 138–146

  28. Tao R, Wei Y, Li H, Liu A, Ding Y, Qin H, Liu X (2021) Over-sampling de-occlusion attention network for prohibited items detection in noisy X-ray images. arXiv:2103.00809, pp 1–13

  29. Hassan T, Werghi N (2020) Trainable structure tensors for autonomous baggage threat detection under extreme occlusion. In: Asian conference on computer vision (ACCV), pp 257–273

  30. Hassan T, Akçay S, Bennamoun M, Khan S, Werghi N (2022) Tensor pooling driven instance segmentation framework for baggage threat recognition. Neural Comput Appl 34:1239–1250

    Article  Google Scholar 

  31. Gaus YFA et al. (2019) Evaluation of a dual convolutional neural network architecture for object-wise anomaly detection in cluttered X-ray security imagery. In: 2019 international joint conference on neural networks (IJCNN), pp 1–8

  32. Akçay S, Kundegorski ME, Devereux M, Breckon TP (2016) Transfer learning using convolutional neural networks for object classification within X-ray baggage security imagery. In: IEEE international conference on image processing, pp 1057–1061

  33. Dhiraj K, Jain D (2019) An evaluation of deep learning based object detection strategies for threat object detection in baggage security imagery. Pattern Recognit Lett 120:112–119

    Article  Google Scholar 

  34. Morris T, Chien T, Goodman E (2018) Convolutional neural networks for automatic threat detection in security X-ray images. In: IEEE international conference on machine learning and applications, pp 1–2

  35. Zuiderveld K (1994) Contrast limited adaptive histograph equalization. Academic Press Professional, Graphic Gems IV, San Diego, pp 474–485

    Google Scholar 

  36. Bigun J, Granlund G (1987) Optimal orientation detection of linear symmetry. In: First international conference on computer vision (ICCV), pp 1–17

  37. Bigun J, Granlund G, Wiklund J (1991) Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Trans Pattern Anal Mach Intell 13(8):775–790

    Article  Google Scholar 

  38. Perona P, Malik J (1990) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639

    Article  Google Scholar 

  39. Hanan S, Tamminen M (1988) Efficient component labeling of images of arbitrary dimension represented by linear bintrees. IEEE Trans Pattern Anal Mach Intell 10(4):579–586

    Article  Google Scholar 

  40. Wu K, Otoo E, Suzuki K (2009) Optimizing two-pass connected-component labeling algorithms. Pattern Anal Appl 12(2):117–135

    Article  MathSciNet  Google Scholar 

  41. Kozdron M (2000) The discrete dirichlet problem, vol 1. Chicago University Press, Chicago, pp 1–11

    Google Scholar 

  42. Kingma DP, Ba J (2015) ADAM: a method for stochastic optimization. In: international conference for learning representations, pp 1–15

  43. Huang G et al. (2017) Densely connected convolutional networks. In: IEEE CVPR, pp 4700–4708

  44. Pleiss G, Chen D, Huang G, Li T, Maaten LVD, Weinberger KQ (2017) Memory-efficient implementation of DenseNets. arXiv:1707.06990, pp 1–8

  45. Khan S, Rahmani H, Shah SAA, Bennamoun M, Medioni G, Dickinson S (2018) A guide to convolutional neural networks for computer vision. Morgan Claypool Publishers 8(1):1–207

    Google Scholar 

  46. Hassan T, Shafay M, Akçay S, Khan S, Bennamoun M, Damiani E, Werghi N (2020) Meta-transfer learning driven tensor-shot detector for the autonomous localization and recognition of concealed baggage threats. Sensors 20(22):1–25

    Article  Google Scholar 

  47. Chui KT, Liu RW, Zhao M, Pablos POD (2020) Predicting students performance with school and family tutoring using generative adversarial network-based deep support vector machine. IEEE Access 8:86745–86752

    Article  Google Scholar 

  48. Redmon J, Farhadi A (2018) YOLOv3: an incremental improvement. arXiv:1804.02767, pp 1–6

  49. Redmon J, Farhadi A (2017) YOLO9000: better, faster, stronger. In: IEEE conference on computer vision and pattern recognition, pp 1063–1079

  50. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE conference on computer vision and pattern recognition, pp 580–587

Download references

Funding

This work is supported by a research fund from Advanced Technology Research Center Program (ASPIRE), (Grant Number: AARE20-279), and Khalifa University Center for Autonomous Robotic Systems (KUCARS) (Grant Number: CIRA-2021-052).

Author information

Authors and Affiliations

  1. C2PS and KUCARS, Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, UAE

    Taimur Hassan & Naoufel Werghi

  2. Department of Electrical and Computer Engineering, Abu Dhabi University, Abu Dhabi, UAE

    Taimur Hassan

  3. Intel R &D, Intel Corporation, Swindon, United Kingdom

    Samet Akcay

  4. Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, UAE

    Bilal Hassan

  5. Department of Computer Science and Software Engineering, The University of Western Australia, Crawley, Australia

    Mohammed Bennamoun

  6. Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

    Salman Khan

  7. KUCARS, Department of Electrical Engineering and Computer Science, Khalifa University, Abu Dhabi, UAE

    Jorge Dias

Authors
  1. Taimur Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Samet Akcay
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bilal Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohammed Bennamoun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Salman Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jorge Dias
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Naoufel Werghi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

TH formulated the idea, wrote the manuscript, and performed the experiments. SA performed the experiments and contributed to manuscript writing. BH improved the initial design of the framework and contributed to manuscript writing. MB co-supervised the whole research, reviewed the manuscript and experiments. SK reviewed the manuscript, experiments and improved the manuscript writing. JD refined the initial idea, co-supervised the experiments, and contributed to manuscript writing. NW supervised the whole research, contributed to manuscript writing, and reviewed the experimentation.

Corresponding author

Correspondence to Taimur Hassan.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to this article.

Financial and Non-Financial interests

All the authors declare that they have no financial or non-financial interests to disclose for this article.

Ethical approval

All the authors declare that no prior ethical approval was required from their institutes to conduct this research.

Consent for Participate and Publication

All the authors declare that no prior consent was needed to disseminate this article as there were no human (or animal) participants involved in this research.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hassan, T., Akcay, S., Hassan, B. et al. Cascaded structure tensor for robust baggage threat detection. Neural Comput & Applic 35, 11269–11285 (2023). https://doi.org/10.1007/s00521-023-08296-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08296-4

Keywords

Search

Navigation