Skip to main content
SpringerLink
Log in

Class-aware edge-assisted lightweight semantic segmentation network for power transmission line inspection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The demand for real-time efficient scene comprehension has been increasing rapidly in the drone-based automatic inspection of power transmission lines (PTL). The extensive application of semantic segmentation in urban scenes proves that it can meet the requirements for scene understanding. However, existing methods have difficulty adapting to changes in the scene, which leads to problems of performance degradation and fuzzy contours of segmented objects. To overcome the existing problems, a class-aware edge-assisted lightweight semantic segmentation network is proposed in this paper. Class-aware edge detection is introduced as an auxiliary task, and a two-branch network is designed to locate instances and refine contours. Specifically, hybrid graph learning uses task-specific graph-based structures to reason attention information of region and edge features. Based on the complementary characteristic of region and edge features, cascaded shared decoders adopt specific interaction functions to enhance the ability of region features to locate targets and the ability of edge features to improve contour details. In addition, to verify the effectiveness of the proposed method, we construct two datasets named the transmission tower component recognition dataset (TTCRD) and the transmission line regional classification dataset (TLRCD). Comprehensive experiments on TTCRD and TLRCD prove that the proposed method can accurately refine the contour of objects and overcome the challenges in the two datasets. Comparison experiments and ablation experiments also demonstrate the superior performance of the proposed method and the effectiveness of each component in our architecture.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability Statement

The data are not publicly available due to the confidentiality of the research projects.

References

  1. Alhassan AB, Zhang X, Shen H, Xu H (2020) Power transmission line inspection robots: a review, trends and challenges for future research. Int J Electr Power Energy Sys 118:105862. https://doi.org/10.1016/j.ijepes.2020.105862

    Article  Google Scholar 

  2. Yu B, Yang L, Chen F (2018) Semantic segmentation for high spatial resolution remote sensing images based on convolution neural network and pyramid pooling module. IEEE J Sel Top Appl Earth Obs Remote Sens 11(9):3252–3261. https://doi.org/10.1109/JSTARS.2018.2860989

    Article  Google Scholar 

  3. Niu W, Ning B, Zhou H (2019) Design of data transmission system of human-autonomous devices for UAV inspection of transmission line status. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-019-01504-x

  4. Chen W, Li Y, Zhao Z (2021) InsulatorGAN: A transmission line insulator detection model using multi-granularity conditional generative adversarial nets for UAV inspection. Remote Sens 13(19):3971. https://doi.org/10.3390/rs13193971

    Article  Google Scholar 

  5. Wu Y, Zhao G, Hu J, Ouyang Y, Wang SX, He J, Gao F, Wang S (2019) Overhead transmission line parameter reconstruction for UAV inspection based on tunneling magnetoresistive sensors and inverse models. IEEE Trans Power Deliv 34(3):819–827. https://doi.org/10.1109/tpwrd.2019.2891119

    Article  Google Scholar 

  6. Alhassan AB, Zhang X, Shen H, Xu H (2020) Power transmission line inspection robots: a review, trends and challenges for future research. Int J Electr Power Energy Syst 118:105862. https://doi.org/10.1016/j.ijepes.2020.105862

    Article  Google Scholar 

  7. Lopez RL, Sanchez MJB, Jimenez MP, Arrue BC, Ollero A (2021) Autonomous UAV system for cleaning insulators in power line inspection and maintenance. Sensors 21(24):8488. https://doi.org/10.3390/s21248488

    Article  Google Scholar 

  8. Yao H, Qin R, Chen X (2019) Unmanned aerial vehicle for remote sensing applications—a review. Remote Sensing 11(12). https://doi.org/10.3390/rs11121443

  9. Xiao R, Wang Y, Tao C (2022) Fine-grained road scene understanding from aerial images based on semisupervised semantic segmentation networks. IEEE Geosci Remote Sens Lett 19:1–5. https://doi.org/10.1109/lgrs.2021.3059708

    Google Scholar 

  10. Lyu Y, Vosselman G, Xia G-S, Yang MY (2021) Bidirectional multi-scale attention networks for semantic segmentation of oblique uav imagery. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences V-2-2021:75–82. https://doi.org/10.5194/isprs-annals-v-2-2021-75-2021

    Article  Google Scholar 

  11. Liu S, Cheng J, Liang L, Bai H, Dang W (2021) Light-weight semantic segmentation network for uav remote sensing images. IEEE J Sel Top Appl Earth Obs Remote Sens 14:8287–8296. https://doi.org/10.1109/JSTARS.2021.3104382

    Article  Google Scholar 

  12. Li R, Zheng S, Zhang C, Duan C, Wang L, Atkinson PM (2021) Abcnet: Attentive bilateral contextual network for efficient semantic segmentation of fine-resolution remotely sensed imagery. ISPRS J Photogramm Remote Sens 181:84–98. https://doi.org/10.1016/j.isprsjprs.2021.09.005https://doi.org/10.1016/j.isprsjprs.2021.09.005

    Article  Google Scholar 

  13. Wu Q, Yang H, Wei M, Remil O, Wang B, Wang J (2018) Automatic 3d reconstruction of electrical substation scene from lidar point cloud. ISPRS J Photogramm Remote Sens 143:57–71. https://doi.org/10.1016/j.isprsjprs.2018.04.024

    Article  Google Scholar 

  14. Wang Y, Chen Q, Liu L, Li K (2019) A hierarchical unsupervised method for power line classification from airborne lidar data. Int J Digit Earth 12(12):1406–1422. https://doi.org/10.1080/17538947.2018.1503740https://doi.org/10.1080/17538947.2018.1503740

    Article  Google Scholar 

  15. Lo S-Y, Hang H-M, Chan S-W, Lin J-J (2019) Efficient dense modules of asymmetric convolution for real-time semantic segmentation. In: Proceedings of the ACM multimedia asia. https://doi.org/10.1145/3338533.3366558

  16. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2018) Unet++: A nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. https://doi.org/10.1007/978-3-030-00889-5_1, pp 3–11

  17. Oršic M, Krešo I, Bevandic P, Šegvic S (2019) In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR.2019.01289, pp 12599–12608

  18. Zhuang J, Yang J, Gu L, Dvornek N (2019) Shelfnet for fast semantic segmentation. In: 2019 IEEE/CVF International conference on computer vision workshop (ICCVW). https://doi.org/10.1109/ICCVW.2019.00113, pp 847–856

  19. Han H-Y, Chen Y-C, Hsiao P-Y, Fu L-C (2021) Using channel-wise attention for deep cnn based real-time semantic segmentation with class-aware edge information. IEEE Trans Intell Transp Syst 22 (2):1041–1051. https://doi.org/10.1109/TITS.2019.2962094

    Article  Google Scholar 

  20. Chen Y, Dapogny A, Cord M (2020) SEMEDA: Enhancing segmentation precision with semantic edge aware loss. Pattern Recogn 108:107557. https://doi.org/10.1016/j.patcog.2020.107557https://doi.org/10.1016/j.patcog.2020.107557

    Article  Google Scholar 

  21. Yu Z, Feng C, Liu M-Y, Ramalingam S (2017) Casenet: Deep category-aware semantic edge detection. In: Proceedings of the IEEE Conference on computer vision and pattern recognition. https://doi.org/10.1109/cvpr.2017.191, pp 5964–5973

  22. Zhao W, Dong Q, Zuo Z (2022) A method combining line detection and semantic segmentation for power line extraction from unmanned aerial vehicle images. 6 14:1367. https://doi.org/10.3390/rs14061367https://doi.org/10.3390/rs14061367

    Google Scholar 

  23. Meng L, Peng Z, Zhou J, Zhang J, Lu Z, Baumann A, Du Y (2020) Real-time detection of ground objects based on unmanned aerial vehicle remote sensing with deep learning: Application in excavator detection for pipeline safety. Remote Sensing 12(1). https://doi.org/10.3390/rs12010182

  24. Siddiqui ZA, Park U (2020) A drone based transmission line components inspection system with deep learning technique. Energies 13(13). https://doi.org/10.3390/en13133348

  25. Jiao R, Liu Y, He H, Xuehai M, Li Z (2021) A deep learning model for small-size defective components detection in power transmission tower. IEEE Transactions on Power Delivery, p 1–1. https://doi.org/10.1109/TPWRD.2021.3112285

  26. Liu J, Jia R, Li W, Ma F, Abdullah HM, Ma H, Mohamed MA (2020) High precision detection algorithm based on improved retinanet for defect recognition of transmission lines. Energy Reports 6:2430–2440. https://doi.org/10.1016/j.egyr.2020.09.002

    Article  Google Scholar 

  27. Li H, Yang Z, Han J, Lai S, Zhang Q, Zhang C, Fang Q, Hu G (2020) Tl-net: A novel network for transmission line scenes classification. Energies 13(15). https://doi.org/10.3390/en13153910

  28. Ma Y, Li Q, Chu L, Zhou Y, Xu C (2021) Real-time detection and spatial localization of insulators for uav inspection based on binocular stereo vision. Remote Sensing 13(2). https://doi.org/10.3390/rs13020230

  29. Tao X, Zhang D, Wang Z, Liu X, Zhang H, Xu D (2020) Detection of power line insulator defects using aerial images analyzed with convolutional neural networks. IEEE Trans Syst Man Cybern Syst 50(4):1486–1498. https://doi.org/10.1109/TSMC.2018.2871750

    Article  Google Scholar 

  30. Zhou B, Zhao H, Puig X, Xiao T, Fidler S, Barriuso A, Torralba A (2019) Semantic understanding of scenes through the ade20k dataset. Int J Comput Vis 127(3):302–321. https://doi.org/10.1007/s11263-018-1140-0

    Article  Google Scholar 

  31. Wang X, Ma H, You S (2020) Deep clustering for weakly-supervised semantic segmentation in autonomous driving scenes. Neurocomputing 381:20–28. https://doi.org/10.1016/j.neucom.2019.11.019https://doi.org/10.1016/j.neucom.2019.11.019

    Article  Google Scholar 

  32. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on computer vision and pattern recognition. https://doi.org/10.1109/cvpr.2015.7298965, pp 3431–3440

  33. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr.2017.660

  34. Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the european conference on computer vision (ECCV). https://doi.org/10.1007/978-3-030-01234-2_49, pp 833–851

  35. Zhao H, Zhang Y, Liu S, Shi J, Loy CC, Lin D, Jia J (2018) Psanet: Point-wise spatial attention network for scene parsing. In: Proceedings of the european conference on computer vision (ECCV). https://doi.org/10.1007/978-3-030-01240-3_17, pp 270–286

  36. Nekrasov V, Shen C, Reid I (2018) Light-weight refinenet for real-time semantic segmentation. In: 2018 British machine vision conference (BMVC)

  37. Yuan Y, Chen X, Wang J (2020) Object-contextual representations for semantic segmentation. In: Proceedings of the European conference on computer vision (ECCV). https://doi.org/10.1007/978-3-030-58539-6_11, pp 173–190

  38. Wang L, Li D, Zhu Y, Tian L, Shan Y (2020) Dual super- resolution learning for semantic segmentation. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR42600.2020.00383, pp 3773–3782

  39. Huang Z, Wang X, Wei Y, Huang L, Shi H, Liu W, Huang TS (2020) Ccnet: Criss-cross attention for semantic segmentation. IEEE Trans Pattern Anal Mach Intell, 1–1. https://doi.org/10.1109/TPAMI.2020.3007032

  40. Yu C, Wang J, Peng C, Gao C, Yu G, Sang N (2018) Bisenet: Bilateral segmentation network for real-time semantic segmentation. In: Proceedings of the european conference on computer vision (ECCV). https://doi.org/10.1007/978-3-030-01261-8_20, pp 325–341

  41. Li H, Xiong P, Fan H, Sun J (2019) Dfanet: Deep feature aggregation for real-time semantic segmentation. In: 2019 IEEE/CVF Conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR.2019.00975, pp 9514–9523

  42. Chao P, Kao C-Y, Ruan Y, Huang C-H, Lin Y-L (2019) Hardnet: A low memory traffic network. In: 2019 IEEE/CVF International conference on computer vision (ICCV). https://doi.org/10.1109/ICCV.2019.00365, pp 3551–3560

  43. Yu C, Gao C, Wang J, Yu G, Shen C, Sang N (2021) Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. Int J Comput Vis 129(11):3051–3068. https://doi.org/10.1007/s11263-021-01515-2

    Article  Google Scholar 

  44. Wu T, Tang S, Zhang R, Cao J, Zhang Y (2021) Cgnet: A light-weight context guided network for semantic segmentation. IEEE Trans Image Process 30:1169–1179. https://doi.org/10.1109/TIP.2020.3042065https://doi.org/10.1109/TIP.2020.3042065

    Article  Google Scholar 

  45. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE Conference on computer vision and pattern recognition. https://doi.org/10.1109/cvpr.2018.00813https://doi.org/10.1109/cvpr.2018.00813, pp 7794–7803

  46. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the ieee international conference on computer vision. https://doi.org/10.1109/iccv.2017.74, pp 618–626

  47. Chattopadhay A, Sarkar A, Howlader P, Balasubramanian VN (2018) Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). https://doi.org/10.1109/wacv.2018.00097

  48. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: International conference on learning representations (ICLR)

  49. Li K, Ye W (2022) Semi-supervised node classification via graph learning convolutional neural network. Applied Intelligence. https://doi.org/10.1007/s10489-022-03233-9

  50. Jamin A, Humeau-Heurtier A (2019) (Multiscale) cross-entropy methods: a review. Entropy 22(1):45. https://doi.org/10.3390/e22010045https://doi.org/10.3390/e22010045

    Article  MathSciNet  Google Scholar 

  51. Russell BC, Torralba A, Murphy KP, Freeman WT (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1–3):157–173. https://doi.org/10.1007/s11263-007-0090-8https://doi.org/10.1007/s11263-007-0090-8

    Article  Google Scholar 

  52. He J-Y, Liang S-H, Wu X, Zhao B, Zhang L (2021) Mgseg: Multiple granularity-based real-time semantic segmentation network. IEEE Trans Image Process 30:7200–7214. https://doi.org/10.1109/tip.2021.3102509https://doi.org/10.1109/tip.2021.3102509

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (grant number 62001156) and in part by the Key Research and Development Plan of Jiangsu Province (grant numbers BE2019036, BE2020092 and BE2020649)

Author information

Authors and Affiliations

  1. College of Internet of Things Engineering, Hohai University, Changzhou, Jiangsu, 213022, China

    Qingkai Zhou, Qingwu Li, Chang Xu, Qiuyu Lu & Yaqin Zhou

  2. Changzhou Key Laboratory of Sensor Networks and Environmental Sensing, Changzhou, Jiangsu, China

    Qingwu Li

Authors
  1. Qingkai Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingwu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qiuyu Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yaqin Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qingwu Li.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflicts of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Q., Li, Q., Xu, C. et al. Class-aware edge-assisted lightweight semantic segmentation network for power transmission line inspection. Appl Intell 53, 6826–6843 (2023). https://doi.org/10.1007/s10489-022-03932-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03932-3

Keywords

Search

Navigation