Skip to main content
SpringerLink
Log in

AI powered road network prediction with fused low-resolution satellite imagery and GPS trajectory

  • Research
  • Published:
Earth Science Informatics Aims and scope Submit manuscript

Abstract

This study presents an innovative approach for automatic road detection with deep learning, employing fusion strategies to utilize both lower-resolution satellite imagery and GPS trajectory data, a concept never explored before. We rigorously investigate both early and late fusion strategies and assess deep learning-based road detection performance using different fusion settings. Our extensive ablation studies evaluate the efficacy of our framework under diverse model architectures, loss functions, and geographic domains (Istanbul and Montreal). For an unbiased and complete evaluation of road detection results, we use both region-based and boundary-based evaluation metrics for road segmentation. The outcomes reveal that the ResUnet model outperforms U-Net and D-Linknet in road extraction tasks, achieving superior results over the benchmark study using low-resolution Sentinel-2 data. This research not only contributes to the field of automatic road detection but also offers novel insights into the utilization of data fusion methods in diverse applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Availability of data and material

The code of the experiments and the data used in the experiments of this study are made available online, and related website information is shared within the article.

Notes

  1. The pre-processed data can be downloaded from following URL: https://github.com/nagellette/sentinel_traj_nn/blob/master/Data.md

  2. The implementations of the methods and experiments can be downloaded from the following URL: https://github.com/nagellette/sentinel_traj_nn

References

  • Jiao C, Heitzler M, Hurni L (2021) A survey of road feature extraction methods from raster maps. Trans GIS 25(6):2734–2763

    Article  Google Scholar 

  • Zhou L, Zhang C, Wu M (2018) D-linknet: Linknet with pretrained encoder and dilated convolution for high resolution satellite imagery road extraction, vol. 2018-June, pp 192–196. IEEE Computer Society, Salt Lake City https://doi.org/10.1109/CVPRW.2018.00034

  • Demir I, Koperski K, Lindenbaum D, Pang G, Huang J, Basu S, Hughes F, Tuia D, Raskar R (2018) Deepglobe 2018: A challenge to parse the earth through satellite images. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 172–17209. https://doi.org/10.1109/CVPRW.2018.00031

  • Liu P, Wang Q, Yang G, Li L, Zhang H (2022) Survey of road extraction methods in remote sensing images based on deep learning. PFG - J Photogramm, Remote Sens Geoinf Sci 90(2):135–159. https://doi.org/10.1007/s41064-022-00194-z

    Article  Google Scholar 

  • Mnih V, Hinton GE (2010) Learning to detect roads in high-resolution aerial images. In: Daniilidis K, Maragos P, Paragios N (eds) Computer Vision - ECCV 2010. Springer, Berlin, Heidelberg, pp 210–223

    Chapter  Google Scholar 

  • Zhang Z, Liu Q, Wang Y (2018) Road extraction by deep residual U-Net. IEEE Geosci Remote Sens Lett 15(5):749–753. https://doi.org/10.1109/LGRS.2018.2802944

    Article  ADS  CAS  Google Scholar 

  • Sun T, Chen Z, Yang W, Wang Y (2018) Stacked U-Nets with multi-output for road extraction. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp 187–1874. https://doi.org/10.1109/CVPRW.2018.00033

  • Ayala C, Aranda C, Galar M (2021) Towards fine-grained road maps extraction using Sentinel-2 imagery. ISPRS Ann Photogramm, Remote Sens Spatial Inf Sci V-3-2021:9–14. https://doi.org/10.5194/isprs-annals-V-3-2021-9-2021

  • Johnson N, Treible W, Crispell D (2022) OpenSentinelMap: A large-scale land use dataset using OpenStreetMap and Sentinel-2 imagery. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp 1332–1340. https://doi.org/10.1109/CVPRW56347.2022.00139

  • Edelkamp S, Schrödl S (2003). In: Klein R, Six H-W, Wegner L (eds) Route Planning and Map Inference with Global Positioning Traces. Springer, Berlin, Heidelberg, pp 128–151

    Google Scholar 

  • Biagioni J, Eriksson J (2012) Map inference in the face of noise and disparity. In: Proceedings of the 20th international conference on advances in geographic information systems. SIGSPATIAL ’12. Association for Computing Machinery, New York, NY, USA, pp 79–88. https://doi.org/10.1145/2424321.2424333. https://doi.org/10.1145/2424321.2424333

  • Karagiorgou S, Pfoser D (2012) On vehicle tracking data-based road network generation. In: Proceedings of the 20th international conference on advances in geographic information systems. SIGSPATIAL ’12. Association for Computing Machinery, New York, NY, USA, pp 89–98. https://doi.org/10.1145/2424321.2424334. https://doi.org/10.1145/2424321.2424334

  • Tang J, Deng M, Huang J, Liu H, Chen X (2019) An automatic method for detection and update of additive changes in road network with gps trajectory data. ISPRS Int J Geo-Inf 8(9). https://doi.org/10.3390/ijgi8090411

  • Sun T, Di Z, Che P, Liu C, Wang Y (2019) Leveraging crowdsourced gps data for road extraction from aerial imagery. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  • Liu L, Yang Z, Li G, Wang K, Chen T, Lin L (2022) Aerial images meet crowdsourced trajectories: A new approach to robust road extraction. IEEE Trans Neural Netw Learn Syst 1–15. https://doi.org/10.1109/TNNLS.2022.3141821

  • Wu H, Zhang H, Zhang X, Sun W, Zheng B, Jiang Y (2020) DeepDualMapper: A gated fusion network for automatic map extraction using aerial images and trajectories. Proc AAAI Conf Artif Intell 34:1037–1045

    Google Scholar 

  • Yang J, Ye X, Wu B, Gu Y, Wang Z, Xia D, Huang J (2022) DuARE: Automatic road extraction with aerial images and trajectory data at baidu maps. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. KDD ’22. Association for Computing Machinery, New York, NY, USA , pp 4321–4331. https://doi.org/10.1145/3534678.3539029 . https://doi.org/10.1145/3534678.3539029

  • Ronneberger O, Fischer P, Brox T (2015) U-Net: Convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention - MICCAI 2015. Springer, Cham, pp 234–241

    Google Scholar 

  • Wu Z, Zhang J, Zhang L, Liu X, Qiao H (2022) Bi-HRNet: A road extraction framework from satellite imagery based on node heatmap and bidirectional connectivity. Remote Sens 14(7). https://doi.org/10.3390/rs14071732

  • Xie Y, Miao F, Zhou K, Peng J (2019) HsgNet: A road extraction network based on global perception of high-order spatial information. ISPRS Int J Geo-Inf 8(12). https://doi.org/10.3390/ijgi8120571

  • Dai L, Zhang G, Zhang R (2023) RADANet: Road augmented deformable attention network for road extraction from complex high-resolution remote-sensing images. IEEE Trans Geosci Remote Sens 61:1–13. https://doi.org/10.1109/TGRS.2023.3237561

    Article  Google Scholar 

  • Yang M, Yuan Y, Liu G (2022) SDUNet: Road extraction via spatial enhanced and densely connected unet. Pattern Recognit 126:108549. https://doi.org/10.1016/j.patcog.2022.108549

    Article  Google Scholar 

  • Li S, Liao C, Ding Y, Hu H, Jia Y, Chen M, Xu B, Ge X, Liu T, Wu D (2022) Cascaded residual attention enhanced road extraction from remote sensing images. ISPRS Int J Geo-Inf 11(1). https://doi.org/10.3390/ijgi11010009

  • Luo L, Wang J-X, Chen S-B, Tang J, Luo B (2022) BDTNet: Road extraction by bi-direction transformer from remote sensing images. IEEE Geosci Remote Sens Lett 19:1–5. https://doi.org/10.1109/LGRS.2022.3183828

    Article  Google Scholar 

  • Jiang X, Li Y, Jiang T, Xie J, Wu Y, Cai Q, Jiang J, Xu J, Zhang H (2022) RoadFormer: Pyramidal deformable vision transformers for road network extraction with remote sensing images. Int J Appl Earth Obs Geoinf 113:102987. https://doi.org/10.1016/j.jag.2022.102987

    Article  Google Scholar 

  • Tao J, Chen Z, Sun Z, Guo H, Leng B, Yu Z, Wang Y, He Z, Lei X, Yang J (2023) Seg-Road: A segmentation network for road extraction based on transformer and cnn with connectivity structures. Remote Sens 15(6). https://doi.org/10.3390/rs15061602

  • Ayala C, Sesma R, Aranda C, Galar M (2021) A deep learning approach to an enhanced building footprint and road detection in high-resolution satellite imagery. Remote Sens 13(16). https://doi.org/10.3390/rs13163135

  • Li P, He X, Qiao M, Miao D, Cheng X, Song D, Chen M, Li J, Zhou T, Guo X, Yan X, Tian Z (2021) Exploring multiple crowdsourced data to learn deep convolutional neural networks for road extraction. Int J Appl Earth Obs Geoinf 104:102544. https://doi.org/10.1016/j.jag.2021.102544

    Article  Google Scholar 

  • Zhou K, Xie Y, Gao Z, Miao F, Zhang L (2021) Funet: A novel road extraction network with fusion of location data and remote sensing imagery. ISPRS Int J Geo-Inf 10(1). https://doi.org/10.3390/ijgi10010039

  • Eftelioglu E, Garg R, Kango V, Gohil C, Chowdhury AR (2022) RING-Net: Road inference from gps trajectories using a deep segmentation network. In: Proceedings of the 10th ACM SIGSPATIAL international workshop on analytics for big geospatial data. BigSpatial ’22. Association for Computing Machinery, New York, NY, USA, pp 17–26. https://doi.org/10.1145/3557917.3567617 . https://doi.org/10.1145/3557917.3567617

  • Gao L, Wang J, Wang Q, Shi W, Zheng J, Gan H, Lv Z, Qiao H (2021) Road extraction using a dual attention dilated-linknet based on satellite images and floating vehicle trajectory data. IEEE J Sel Top Appl Earth Obs Remote Sens 14:10428–10438. https://doi.org/10.1109/JSTARS.2021.3116281

    Article  ADS  Google Scholar 

  • Zhang Y, Sidibé D, Morel O, Mériaudeau F (2021) Deep multimodal fusion for semantic image segmentation: A survey. Image Vision Comput 105:104042. https://doi.org/10.1016/j.imavis.2020.104042

    Article  Google Scholar 

  • Goodfellow I, Bengio Y, Courville A (2016) Deep Learning. MIT Press, Cambridge . http://www.deeplearningbook.org

  • Maxwell AE, Warner TA, Guillén LA (2021a) Accuracy assessment in convolutional neural network-based deep learning remote sensing studies–part 1: Literature review. Remote Sens 13(13). https://doi.org/10.3390/rs13132450

  • Maxwell AE, Warner TA, Guillén LA (2021b) Accuracy assessment in convolutional neural network-based deep learning remote sensing studies–part 2: Recommendations and best practices. Remote Sens 13(13). https://doi.org/10.3390/rs13132591

  • Etten AV (2020) City-scale road extraction from satellite imagery v2: Road speeds and travel times. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, Snowmass Village. https://doi.org/10.1109/wacv45572.2020.9093593 . https://doi.org/10.1109/wacv45572.2020.9093593

  • Etten AV, Shermeyer J, Hogan D, Weir N, Lewis R (2020) Road network and travel time extraction from multiple look angles with spacenet data. In: IGARSS 2020 - 2020 IEEE international geoscience and remote sensing symposium. IEEE, Virtual Symposium. https://doi.org/10.1109/igarss39084.2020.9324091

  • Maier-Hein L, Reinke A, Godau P, Tizabi MD, Büttner F, Christodoulou E, Glocker B, Isensee F, Kleesiek J, Kozubek M, Reyes M, Riegler MA, Wiesenfarth M, Kavur E, Sudre CH, Baumgartner M, Eisenmann M, Heckmann-Nötzel D, Rädsch AT, Acion L, Antonelli M, Arbel T, Bakas S, Benis A, Blaschko M, Cardoso MJ, Cheplygina V, Cimini BA, Collins GS, Farahani K, Ferrer L, Galdran A, Ginneken B, Haase R, Hashimoto DA, Hoffman MM, Huisman M, Jannin P, Kahn CE, Kainmueller D, Kainz B, Karargyris A, Karthikesalingam A, Kenngott H, Kofler F, Kopp-Schneider A, Kreshuk A, Kurc T, Landman BA, Litjens G, Madani A, Maier-Hein K, Martel AL, Mattson P, Meijering E, Menze B, Moons KGM, Müller H, Nichyporuk B, Nickel F, Petersen J, Rajpoot N, Rieke N, Saez-Rodriguez J, Sánchez CI, Shetty S, Smeden M, Summers RM, Taha AA, Tiulpin A, Tsaftaris SA, Calster BV, Varoquaux G, Jäger PF (2023) Metrics reloaded: Pitfalls and recommendations for image analysis validation. arXiv:2206.01653

  • Ozturk O, Isik MS, Sariturk B, Seker DZ (2022) Generation of istanbul road data set using google map api for deep learning-based segmentation. Int J Remote Sens 43:2793–2812. https://doi.org/10.1080/01431161.2022.2068989

    Article  Google Scholar 

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  • Ciampiconi L, Elwood A, Leonardi M, Mohamed A, Rozza A (2023) A survey and taxonomy of loss functions in machine learning

  • Jadon S (2020) A survey of loss functions for semantic segmentation. In: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB). pp 1–7. https://doi.org/10.1109/CIBCB48159.2020.9277638

  • Lin T-Y, Goyal P, Girshick R, He K, Dollár P (2020) Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 42(2):318–327. https://doi.org/10.1109/TPAMI.2018.2858826

    Article  PubMed  Google Scholar 

  • Cheng B, Girshick R, Dollar P, Berg AC, Kirillov A (2021) Boundary iou: Improving object-centric image segmentation evaluation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp 15334–15342

  • Istanbul Buyuksehir Belediyesi (2020) IBB ISTAÇ Araçlarinin Anlik Konum ve Hiz Bilgileri . https://data.ibb.gov.tr/dataset/ibb-istac-araclarinin-anlik-konum-ve-hiz-bilgileri. [Accessed 23-Jun-2020]

  • Ville de Montréal (2021) VMTL-MTL-Trajet. https://www.donneesquebec.ca/recherche/fr/dataset/vmtl-mtl-trajet. [Accessed 12-Apr-2023]

  • European Space Agency (2021) Sentinel-2 MSI - MultiSpectral Instrument User Guide. https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi. [Accessed 12-Apr-2023]

  • OpenStreetMap Contributors (2021) OpenStreetMap. https://www.openstreetmap.org/. [Accessed 12-Apr-2023]

  • Yuan M, Liu Z, Wang F, Jin F (2019) Rethinking labelling in road segmentation. Int J Remote Sens 40(22):8359–8378. https://doi.org/10.1080/01431161.2019.1608393

    Article  Google Scholar 

  • Gengec NE, Tari E (2021) Performance evaluation of gps trajectory rasterization methods. In: Computational Science and Its Applications – ICCSA 2021: 21st International Conference, Cagliari, Italy, September 13–16, 2021, Proceedings, Part I. Springer, Berlin, Heidelberg, pp 3–17. https://doi.org/10.1007/978-3-030-86653-2_1

  • Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Software available from tensorflow.org. https://www.tensorflow.org/

  • TensorFlow Addons Contributors (2021) TensorFlow Addons. https://github.com/tensorflow/addons

  • Perkiö J, Hyvärinen A (2009) Modelling image complexity by independent component analysis, with application to content-based image retrieval. In: Alippi C, Polycarpou M, Panayiotou C, Ellinas G (eds) Artificial Neural Networks - ICANN 2009. Springer, Berlin, Heidelberg, pp 704–714

    Chapter  Google Scholar 

  • Haralick RM, Shanmugam K, Dinstein I (1973) Textural features for image classification. IEEE Trans Syst, Man, Cybernet SMC-3(6):610–621. https://doi.org/10.1109/TSMC.1973.4309314

  • Rahane AA, Subramanian A (2020) Measures of complexity for large scale image datasets. In: 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC). pp 282–287. https://doi.org/10.1109/ICAIIC48513.2020.9065274

  • Bilgi S, Gulnerman AG, Arslanoglu B, Karaman H, Ozturk O (2019) Complexity measures of sports facilities allocation in urban area by metric entropy and public demand compatibility. Int J Eng Geosci 4(3):141–148. https://doi.org/10.26833/ijeg.540180

  • scikit-image (2021) Shannon entropy. https://scikit-image.org/docs/stable/api/skimage.measure.html#skimage.measure.shannon_entropy. [Online; accessed 12-Apr-2023]

  • scikit-image (2021b) Gray-level co-occurrence matrix properties. https://scikit-image.org/docs/stable/api/skimage.feature.html#skimage.feature.graycoprops. [Online; accessed 12-Apr-2023]

  • Mapillary (2023) Image API documentation. https://www.mapillary.com/developer/api-documentation#image. [Accessed 16-Nov-2023]

  • Open Street Map (2023) Public GPS Traces. https://www.openstreetmap.org/traces/. [Accessed 16-Nov-2023]

Download references

Acknowledgements

The numerical calculations reported in this paper were fully performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources). The authors would like to thank the Istanbul Metropolitan Municipality and City of Montreal for the GPS trajectory dataset, the European Space Agency (ESA) for Sentinel-2 data, and the OpenStreetMap Foundation and OpenStreetMap Contributors for OpenStreetMap data. This study is part of the Ph.D. thesis conducted at Istanbul Technical University by the first author. The authors would like to thank the Ph.D. thesis advancement monitoring committee members, Gulsen Kaya Taskin and Taskin Kavzoglu.

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Geomatics Engineering, Istanbul Technical University, Istanbul, Turkey

    Necip Enes Gengec & Ergin Tari

  2. Machine and Hybrid Intelligence Lab, Northwestern University, Chicago, Illinois, USA

    Necip Enes Gengec & Ulas Bagci

Authors
  1. Necip Enes Gengec
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ergin Tari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ulas Bagci
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Necip Enes Gengec contributed to the manuscript by conceptualizing the topic, running experiments, writing the main manuscript, and preparing figures and tables. Ergin Tari and Ulas Bagci provided valuable guidance and expertise in the development of the methodology and execution of the experiments. All authors actively participated in reviewing the manuscript and contributed to its improvement through discussions and suggestions.

Corresponding author

Correspondence to Necip Enes Gengec.

Ethics declarations

Conflicts of interest/Competing interests

The authors declare no conflict of interest or competing interest.

Additional information

Communicated by: H. Babaie.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gengec, N.E., Tari, E. & Bagci, U. AI powered road network prediction with fused low-resolution satellite imagery and GPS trajectory. Earth Sci Inform 17, 1013–1029 (2024). https://doi.org/10.1007/s12145-023-01201-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12145-023-01201-6

Keywords

Search

Navigation