Skip to main content

Advertisement

SpringerLink
Log in

Detection of threat objects in baggage inspection with X-ray images using deep learning

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In the field of security, baggage-screening with X-rays is used as nondestructive testing for threat object detection. This is a common protocol when inspecting passenger baggage particularly at airports. Unfortunately, the accuracy of such human inspection is around 80–90%, under optimal operator conditions. For this reason, it is quite necessary to assist human inspectors with the aid of computer vision algorithms. This work proposes a deep learning-based methodology designed to detect threat objects in (single spectrum) X-ray baggage scan images. For this purpose, our proposed framework simulates a large number of X-ray images, using a combination of PGGAN (Karras et al. in International conference on learning representations, 2018. https://openreview.net/forum?id=Hk99zCeAb) and superimposition (Mery and Katsaggelos in 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), 2017.https://doi.org/10.1109/CVPRW.2017.37) strategies, that are used to train state-of-the-art detection models such as YOLO (Redmon et al. in You only look once: unified, real-time object detection. CoRR abs/1506.02640, 2015. http://arxiv.org/ abs/1506.02640), SSD (Liu et al. in SSD: single shot multibox detector. CoRR abs/1512.02325, 2015. http://arxiv. org/abs/1512.02325) and RetinaNet (Lin et al. in Focal loss for dense object detection. CoRR abs/1708.02002, 2017. http://arxiv.org/abs/1708.02002). Our method has been tested on real X-ray images in the detection of four categories of threat objects: guns, knives, razor blades and shuriken (ninja stars). In our experiments, YOLOv3 (Redmon and Farhadi in Yolov3: An incremental improvement. CoRR abs/1804.02767, 2018. http://arxiv.org/abs/1804.02767) obtained the best mean average precision (mAP) with 96.3% for guns, 76.2% for knives, 86.9% for razor blades and 93.7% for shuriken, while the average mAP for all threat objects was 80.0%. We believe the effectiveness of our method in the detection of threat objects makes its use in checkpoints possible. Moreover, our methodology is scalable and can be easily extended to detect other categories automatically.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. Razor blades and shuriken are not present in Subset 2.

  2. Code and datasets can be downloaded from https://github.com/dlsaavedra/Detector_GDXray.

References

  1. Akcay S, Kundegorski ME, Willcocks CG, Breckon TP (2018) Using deep convolutional neural network architectures for object classification and detection within X-ray baggage security imagery. IEEE Trans Inf Forensics Secur 13(9):2203–2215. https://doi.org/10.1109/TIFS.2018.2812196

    Article  Google Scholar 

  2. Alcorn MA, Li Q, Gong Z, Wang C, Mai L, Ku WS, Nguyen A (2019) Strike (with) a pose: neural networks are easily fooled by strange poses of familiar objects. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4845–4854)

  3. Anh HN (2018) keras-yolo2. https://github.com/experiencor/keras-yolo2

  4. Anh HN (2018) keras-yolo3. https://github.com/experiencor/keras-yolo3

  5. Baştan M (2015) Multi-view object detection in dual-energy X-ray images. Mach Vis Appl 26(7):1045–1060. https://doi.org/10.1007/s00138-015-0706-x

    Article  Google Scholar 

  6. Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: Leonardis A, Bischof H, Pinz A (eds) Computer vision—ECCV 2006. Springer, Berlin, pp 404–417

    Chapter  Google Scholar 

  7. Bolfing A, Halbherr T, Schwaninger A (2008) How image based factors and human factors contribute to threat detection performance in X-ray aviation security screening. In: Holzinger A (ed) HCI and usability for education and work. Springer, Berlin, pp 419–438

    Chapter  Google Scholar 

  8. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: a large-scale hierarchical image database. In: CVPR09

  9. Dhiraj Jain DK (2019) An evaluation of deep learning based object detection strategies for threat object detection in baggage security imagery. Pattern Recognit Lett 120:112–119. https://doi.org/10.1016/j.patrec.2019.01.014

    Article  Google Scholar 

  10. Everingham M, Gool L, Williams CK, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. Int J Comput Vis 88(2):303–338. https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  11. Ferrari P (2018) ssd keras. https://github.com/pierluigiferrari/ssd_keras

  12. Fizyr: keras-retinanet. https://github.com/fizyr/keras-retinanet (2018)

  13. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. arXiv e-prints arXiv:1406.2661

  14. Goodfellow I, Bengio Y, Courville A (2016) Deep learning, vol 1. MIT Press, Cambridge, p 2

    MATH  Google Scholar 

  15. Harris C, Stephens M (1988) A combined corner and edge detector. In: In Proc. of fourth Alvey vision conference, pp. 147–151

  16. Hartley R, Zisserman A (2003) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  17. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR abs/1512.03385. http://arxiv.org/abs/1512.03385

  18. Kanazawa A (2014) Locally scale-invariant convolutional neural networks. In: NeurIPS workshops

  19. Karras T, Aila T, Laine S, Lehtinen J (2018) Progressive growing of GANs for improved quality, stability, and variation. In: international conference on learning representations. https://openreview.net/forum?id=Hk99zCeAb

  20. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems, vol 25. Curran Associates Inc., New York, pp 1097–1105

    Google Scholar 

  21. Lin CC, Shiou FJ (2018) Object recognition based on foreground detection using X-ray imaging. J Chin Inst Eng 41(5):395–402. https://doi.org/10.1080/02533839.2018.1482235.

    Article  Google Scholar 

  22. Lin T, Dollár P, Girshick RB, He K, Hariharan B, Belongie SJ (2016) Feature pyramid networks for object detection. CoRR abs/1612.03144. http://arxiv.org/abs/1612.03144

  23. Lin T, Goyal P, Girshick RB, He K, Dollár P (2017) Focal loss for dense object detection. CoRR abs/1708.02002. http://arxiv.org/abs/1708.02002

  24. Lin T, Maire M, Belongie SJ, Bourdev LD, Girshick RB, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: common objects in context. CoRR abs/1405.0312 (2014). http://arxiv.org/abs/1405.0312

  25. Liu W, Anguelov D, Erhan D, Szegedy C, Reed SE, Fu C, Berg AC (2015) SSD: single shot multibox detector. CoRR abs/1512.02325. http://arxiv.org/abs/1512.02325

  26. Mathanker S (2013) X-ray applications in food and agriculture: a review. Trans ASABE (American Society of Agricultural and Biological Engineers) 56:1227–1239. https://doi.org/10.13031/trans.56.9785

    Article  Google Scholar 

  27. McBee MP, Awan OA, Colucci AT, Ghobadi CW, Kadom N, Kansagra AP, Tridandapani S, Auffermann WF (2018) Deep learning in radiology. Acad. Radiol 25(11):1472–1480

    Article  Google Scholar 

  28. Mery D (2015) Computer vision for X-ray testing. Springer, Berlin

    Book  Google Scholar 

  29. Mery D (2015) Inspection of complex objects using multiple-X-ray views. IEEE/ASME Trans Mechatron 20(1):338–347. https://doi.org/10.1109/TMECH.2014.2311032

    Article  Google Scholar 

  30. Mery D, Katsaggelos AK (2017) A logarithmic X-ray imaging model for baggage inspection: Simulation and object detection. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp. 251–259 . 10.1109/CVPRW.2017.37

  31. Mery D, Riffo V, Zscherpel U, Mondragón G, Lillo I, Zuccar I, Lobel H, Carrasco M (2015) Gdxray: the database of X-ray images for nondestructive testing. J Nondestruct Eval 34:42. https://doi.org/10.1007/s10921-015-0315-7

    Article  Google Scholar 

  32. Mery D, Riffo V, Zuccar I, Pieringer C (2013) Automated X-ray object recognition using an efficient search algorithm in multiple views. In: 2013 IEEE conference on computer vision and pattern recognition workshops, pp. 368–374 . 10.1109/CVPRW.2013.62

  33. Mery D, Svec E, Arias M, Riffo V, Saavedra JM, Banerjee S (2017) Modern computer vision techniques for X-ray testing in baggage inspection. IEEE Trans Syst Man Cybern Syst 47(4):682–692. https://doi.org/10.1109/TSMC.2016.2628381

    Article  Google Scholar 

  34. Miao C, Xie L, Wan F, Su C, Liu H, Jiao J, Ye Q (2019) Sixray : A large-scale security inspection X-ray benchmark for prohibited item discovery in overlapping images. CoRR abs/1901.00303. http://arxiv.org/abs/1901.00303

  35. Michel S, Koller SM, de Ruiter JC, Moerland R, Hogervorst M, Schwaninger A (2007) Computer-based training increases efficiency in X-ray image interpretation by aviation security screeners. In: 2007 41st Annual IEEE international Carnahan conference on security technology, pp. 201–206. 10.1109/CCST.2007.4373490

  36. Mikhaylichenko A, Demyanenko Y, Grushko E (2016) Automatic detection of bone contours in X-ray images. In: AIST (Supplement), pp. 212–223

  37. Mikolajczyk A, Grochowski M (2018) Data augmentation for improving deep learning in image classification problem. In 2018 International interdisciplinary PhD workshop (IIPhDW) (pp. 117-122). IEEE

  38. Nercessian S, Panetta K, Agaian S (2008) Automatic detection of potential threat objects in X-ray luggage scan images. In: 2008 IEEE conference on technologies for Homeland Security, pp. 504–509 . 10.1109/THS.2008.4534504

  39. Perez L, Wang J (2017) The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621

  40. Radford A, Metz L, Chintala S (2016) Unsupervised representation learning with deep convolutional generative adversarial networks. In: International conference on learning representations (ICLR)

  41. Ramani R, Vanitha S, Valarmathy S (2013) The pre-processing techniques for breast cancer detection in mammography images. Int J Image Gr Signal Process 5:47–54. https://doi.org/10.5815/ijigsp.2013.05.06

    Article  Google Scholar 

  42. Redmon J, Divvala SK, Girshick RB, Farhadi A (2015) You only look once: unified, real-time object detection. CoRR abs/1506.02640. http://arxiv.org/abs/1506.02640

  43. Redmon J, Farhadi A (2016) YOLO9000: better, faster, stronger. CoRR abs/1612.08242. http://arxiv.org/abs/1612.08242

  44. Redmon J, Farhadi A (2018) Yolov3: An incremental improvement. CoRR abs/1804.02767. http://arxiv.org/abs/1804.02767

  45. Ren S, He K, Girshick RB, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. CoRR abs/1506.01497 . http://arxiv.org/abs/1506.01497

  46. RichardWebster B, Anthony SE, Scheirer WJ (2018) Psyphy: a psychophysics driven evaluation framework for visual recognition. IEEE Trans Pattern Anal Mach Intell 41(9):2280–2286

    Article  Google Scholar 

  47. Riffo V, Mery D (2016) Automated detection of threat objects using adapted implicit shape model. IEEE Trans Syst Man Cybern Syst 46(4):472–482. https://doi.org/10.1109/TSMC.2015.2439233

    Article  Google Scholar 

  48. Ruder S (2016) An overview of gradient descent optimization algorithms. CoRR abs/1609.04747 . http://arxiv.org/abs/1609.04747

  49. Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data 6(1):60

    Article  Google Scholar 

  50. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 . http://arxiv.org/abs/1409.1556

  51. Steitz JO, Saeedan F, Roth S (2018) Multi-view X-ray R-CNN. In: Proceedings of the German conference on pattern recognition (GCPR), LNCS 11269, pp. 153–158

  52. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9

  53. Szeliski R (2010) Computer vision: algorithms and applications. Springer, Berlin

    MATH  Google Scholar 

  54. Turcsany D, Mouton A, Breckon TP (2013) Improving feature-based object recognition for X-ray baggage security screening using primed visualwords. In: 2013 IEEE International conference on industrial technology (ICIT), pp. 1140–1145. IEEE

  55. Wu N, Phang J, Park J, Shen Y, Huang Z, Zorin M, Jastrzebski S, Fevry T, Katsnelson J, Kim E, Wolfson S, Parikh U, Gaddam S, Lin LLY, Ho K, Weinstein JD, Reig B, Gao Y, Toth H, Pysarenko K, Lewin A, Lee J, Airola, K, Mema E, Chung S, Hwang E, Samreen N, Kim SG, Heacock L, Moy L, Cho K, Geras KJ (2019) Deep neural networks improve radiologists’ performance in breast cancer screening. arXiv preprint arXiv:1903.08297

  56. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? CoRR abs/1411.1792. http://arxiv.org/abs/1411.1792

  57. Zentai G (2008) X-ray imaging for homeland security. In: 2008 IEEE international workshop on imaging systems and techniques, pp. 1–6 . 10.1109/IST.2008.4659929

  58. Zhao Z, Zheng P, Xu S, Wu X (2018) Object detection with deep learning: a review. CoRR abs/1807.05511. http://arxiv.org/abs/1807.05511

  59. Zou L, Yusuke T, Hitoshi I (2020) Dangerous objects detection of X-ray images using convolution neural network. In: Yang CN, Peng SL, Jain LC (eds) Security with intelligent computing and big-data services. Springer, Cham, pp 714–728

    Chapter  Google Scholar 

Download references

Funding

This work was supported in part by Fondecyt Grants 1161314 and 1191131 from CONICYT—Chile.

Author information

Authors and Affiliations

  1. Department of Computer Science, Pontificia Universidad Catolica de Chile, Santiago, Chile

    Daniel Saavedra & Domingo Mery

  2. Affectiva, Boston, MA, USA

    Sandipan Banerjee

Authors
  1. Daniel Saavedra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sandipan Banerjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Domingo Mery
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Domingo Mery.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saavedra, D., Banerjee, S. & Mery, D. Detection of threat objects in baggage inspection with X-ray images using deep learning. Neural Comput & Applic 33, 7803–7819 (2021). https://doi.org/10.1007/s00521-020-05521-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05521-2

Keywords

Search

Navigation