Skip to main content
SpringerLink
Log in

FishTwoMask R-CNN: Two-stage Mask R-CNN approach for detection of fishplates in high-altitude railroad track drone images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Maintenance of railroad track safety is of utmost importance as derailment accidents cause significant loss to life and property. Inspection of railroad tracks and their components is necessary in order to ensure security and well-being of goods as well as humans. Fishplate is an essential component in the railroad track environment hence, periodic maintenance of fishplates is an imperative goal. In this paper, we propose a method for detection and segmentation of fishplate instances in high-altitude drone images (DI) for a closer-view and consequent inspection of fishplate instances. For this purpose, a novel two-stage Mask R-CNN-based framework termed as FishTwoMask R-CNN is proposed. A new fine-tuning strategy has been developed for the purpose of improving the detections in the second stage (Stage 2) which includes a training trick of modifying the loss weights for Stage 2 training. In the first stage (Stage 1), we detect fishplate instances, which are then cropped and fed as input to Stage 2, along with Stage 1 dataset. The Stage 2 network is then trained through a modified weighted loss and produces final detections for segmentation and further inspection. The”layers” hyper-parameter is assigned as “heads” for Stage 1 and updated to “4 + ” for Stage 2. Also, the critical analysis of Mask R-CNN hyper-parameters has been carried out during both the stages which has lead to an improved detection precision rate of 97% in Stage 2 as opposed to 47% in Stage 1. We evaluate our proposed approach on five different test image scenarios in order to view fishplate instance detection results. There has been statistical evaluation on out-of-distribution test images also in order to compute the metrics values. The comparative results have been evaluated using metrics of precision, recall, and F1-score on Mask R-CNN Stage 1 and Stage 2 along with Faster R-CNN and YOLOv5 methods. It is inferred that the proposed approach achieves appreciable metrics values and thus can be gathered suitable for fishplate instance segmentation in drone images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are not publicly available due to privacy reason but are available from the corresponding author on reasonable request.

References

  1. Abdulla W (2017) Mask r-cnn for object detection and instance segmentation on keras and tensorflow

  2. Bharath B, Kanmani M (2017) Swarm intelligence based image fusion for thermal and visible images, in 2017 International Conference on Computation of Power, Energy Information and Commuincation (ICCPEIC), 043–048

  3. Bhat S, Karegowda D, Noushad I (2021) Smart railway track monitoring system

  4. Buggy SJ et al (2016) Railway track component condition monitoring using optical fibre Bragg grating sensors. Meas Sci Technol 27(5):055201

    Article  Google Scholar 

  5. Chen Z et al (2022) Foreign object detection for railway ballastless trackbeds: A Semisupervised Learning Method. Measurement, 110757

  6. Du C, Dutta S, Kurup P, Yu T, Wang X (2020) A review of railway infrastructure monitoring using fiber optic sensors. Sensors and Actuators A: Physical 303:111728. https://doi.org/10.1016/j.sna.2019.111728

    Article  Google Scholar 

  7. Gao M, Wu H, Shen Y, Wang X, Zeng Y (2019) A peak detection algorithm adopting magnetic sensor signal for rail spike location in tamping machine. Adv Mech Eng 11(11):1687814019891570

    Article  Google Scholar 

  8. Gavai G, Eldardiry H, Wu W, Xu B, Komatsu Y, Makino S (2019) Hybrid image-based defect detection for railroad maintenance. Electronic Imaging 2019(9):360–361

    Google Scholar 

  9. Güçlü E, Aydın İ, Akın E (2022) Measurement of railway sleepers spacing using mask R-CNN, in 2022 International Conference on Decision Aid Sciences and Applications (DASA), 1416–1420

  10. Guo F, Qian Y, Rizos D, Suo Z, Chen X (2021) Automatic rail surface defects inspection based on Mask R-CNN. Transp Res Rec 2675(11):655–668

    Article  Google Scholar 

  11. Guo F, Qian Y, Wu Y, Leng Z, Yu H (2021) Automatic railroad track components inspection using real-time instance segmentation. Computer-Aided Civil and Infrastructure Engineering 36(3):362–377

    Article  Google Scholar 

  12. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask r-cnn, in Proceedings of the IEEE international conference on computer vision, 2961–2969

  13. Hodge VJ, O’Keefe S, Weeks M, Moulds A (2015) Wireless sensor networks for condition monitoring in the railway industry: A Survey. IEEE Transactions on Intelligent Transportation Systems 16(3):1088–1106. https://doi.org/10.1109/TITS.2014.2366512

    Article  Google Scholar 

  14. Kafetzis D, Fourfouris I, Argyropoulos S, Koutsopoulos I (2020) UAV-assisted aerial survey of railways using deep learning, in 2020 International Conference on Unmanned Aircraft Systems (ICUAS), 1491–1500

  15. Kanmani M, Narasimhan V (2019) An optimal weighted averaging fusion strategy for remotely sensed images. Multidimension Syst Signal Process 30(4):1911–1935

    Article  Google Scholar 

  16. Kanmani M, Narasimhan V (2020) Optimal fusion aided face recognition from visible and thermal face images. Multimedia Tools and Applications 79(25):17859–17883

    Article  Google Scholar 

  17. Liu J, Wang Z, Wu Y, Qin Y, Cao X, Huang Y (2020) An Improved Faster R-CNN for UAV-Based Catenary Support Device Inspection. Int J Software Eng Knowl Eng 30(07):941–959

    Article  Google Scholar 

  18. Madheswari K, Venkateswaran N (2017) Swarm intelligence based optimisation in thermal image fusion using dual tree discrete wavelet transform. Quantitative Infrared Thermography Journal 14(1):24–43

    Article  Google Scholar 

  19. Madheswari K, Venkateswaran N, Sowmiya V (2016) Visible and thermal image fusion using curvelet transform and brain storm optimization, in 2016 IEEE Region 10 Conference (TENCON), 2826–2829

  20. Nayan MMR, Al Sufi S, Abedin AK, Ahamed R, Hossain MF (2020) An IoT based real-time railway fishplate monitoring system for early warning, in 2020 11th International Conference on Electrical and Computer Engineering (ICECE), 310–313

  21. Rampriya RS, Suganya R, Nathan S, Perumal PS (2022) A comparative assessment of deep neural network models for detecting obstacles in the real time aerial railway track images. Applied Artificial Intelligence, 1–33

  22. Ravichandran A, Raja A, Kanmani M (2017) Entropy optimized image fusion: Using particle swarm technology and discrete wavelet transform, in 2017 international conference on computation of power, energy information and commuincation (ICCPEIC), 068–074

  23. Saini A, Agarwal A, Singh D (2020) Feature-based template matching for joggled fishplate detection in railroad track with drone images, in IGARSS 2020–2020 IEEE International Geoscience and Remote Sensing Symposium, 2237–2240

  24. Saini A, Kishore KG, Sriram KSS, Singh D, Singh KP (2022) Machine learning approach for detection of track assets for railroad health monitoring with drone images, in IGARSS 2022–2022 IEEE International Geoscience and Remote Sensing Symposium, 4891–4894

  25. Saini A, Singh D (2018) Development of computer vision based robust approach for joggled fish plate detection in drone images, in 2018 9th International Symposium on Signal, Image, Video and Communications (ISIVC), 33–38

  26. Saini A, Singh D (2021) DroneRTEF: development of a novel adaptive framework for railroad track extraction in drone images. Pattern Anal Appl 24(4):1549–1568

    Article  Google Scholar 

  27. Singh P, Garg RD (2014) Classification of high resolution satellite images using spatial constraints-based fuzzy clustering. J Appl Remote Sens 8(1):083526

    Article  Google Scholar 

  28. Singh PP, Garg RD (2015) Fixed point ICA based approach for maximizing the non-Gaussianity in remote sensing image classification. Journal of the Indian Society of Remote Sensing 43(4):851–858

    Article  Google Scholar 

  29. Sumit SS, Watada J, Roy A, Rambli DRA (2020) In object detection deep learning methods, YOLO shows supremum to Mask R-CNN. J Phys: Conf Ser 1529(4):042086

    Google Scholar 

  30. Szeliski R (2010) Computer vision: algorithms and applications. Springer Science & Business Media

  31. Wu Y, Meng F, Qin Y, Qian Y, Xu F, Jia L (2023) UAV imagery based potential safety hazard evaluation for high-speed railroad using Real-time instance segmentation. Adv Eng Inform 55:101819

    Article  Google Scholar 

  32. Yilmazer M, Karakose M (2022) “Mask R-CNN architecture based railway fastener fault detection approach”. International Conference on Decision Aid Sciences and Applications (DASA) 2022:1363–1366

    Article  Google Scholar 

  33. Zheng D et al (2021) A defect detection method for rail surface and fasteners based on deep convolutional neural network. Computational Intelligence and Neuroscience

Download references

Acknowledgements

The authors would like to thank RailTel, India for supporting this work. The authors would also like to extend their thanks to British Council and IIT Roorkee for granting Newton Bhabha Fund under which a part of this research work has been carried out at The University of Sheffield, Sheffield, United Kingdom.

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, Indian Institute of Technology, Roorkee, Roorkee, Uttarakhand, India

    Aradhya Saini & Dharmendra Singh

  2. RailTel - IIT Roorkee Center of Excellence in Telecommunication, Roorkee, Uttarakhand, India

    Aradhya Saini & Dharmendra Singh

  3. Department of Computer Science, The University of Sheffield, Sheffield, S10 2TN, UK

    Mauricio Alvarez

Authors
  1. Aradhya Saini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dharmendra Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mauricio Alvarez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dharmendra Singh.

Ethics declarations

Conflict of interest

The authors would like to state that there is no conflict of interest amongst any of the authors.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix I

Appendix I

Tables 8 and 9

Table 8 Evaluation of metrics on 100 test images (50 test images each for fishplate and no fishplate instances)
Full size table
Table 9 Evaluation of metrics on 116 test images (58 test images each for fishplate and no fishplate instances)
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saini, A., Singh, D. & Alvarez, M. FishTwoMask R-CNN: Two-stage Mask R-CNN approach for detection of fishplates in high-altitude railroad track drone images. Multimed Tools Appl 83, 10367–10392 (2024). https://doi.org/10.1007/s11042-023-15924-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15924-7

Keywords

Search

Navigation