Skip to main content
SpringerLink
Log in

A deep learning approach to satellite image time series coregistration through alignment of road networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The adverse effects of thawing permafrost on transportation infrastructure in northern regions are exacerbated by climate change. To address this issue, remote sensing techniques can be employed to track deformations in these structures over time. This will allow us to identify regions that are most vulnerable to permafrost degradation, and implement climate adaptation strategies accordingly. The Sentinel-2 mission provides highly suitable data for multitemporal analysis due to its high temporal resolution and multispectral coverage. However, the geometrical misalignment of Sentinel-2 imagery presents a significant challenge for such analysis. In this study, we propose an automatic sub-pixel coregistration algorithm for satellite image time series, specifically focusing on estimating the deformation of linear infrastructure in northern Canada. Our approach involves utilizing a deep learning model to generate binary masks of roads, which are then used to match and align the images. We demonstrate the feasibility of achieving sub-pixel coregistration through road alignment on a small dataset of high-resolution Sentinel-2 images from the town of Gillam in northern Canada. This represents an initial step toward training a road deformation prediction model, which can ultimately contribute to improved infrastructure resilience and adaptation to changing climatic conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5

Similar content being viewed by others

Data and code availability

The dataset and code are available at the following GitHub repository: https://github.com/afperezm/multi-temporal-coregistration.

Notes

  1. Mean SSIM between pairs of images for all the time series.

References

  1. Liu H, Maghoul P, Shalaby A (2022) Seismic physics-based characterization of permafrost sites using surface waves. The Cryosphere 16(4):1157–1180. https://doi.org/10.5194/tc-16-1157-2022

    Article  ADS  Google Scholar 

  2. Bishop MP, Björnsson H, Haeberli W, Oerlemans J, Shroder JF, Tranter M (2011) Encyclopedia of snow. Ice and Glaciers. Springer

  3. Bush E, Lemmen DS (2019) Canada’s changing climate report. Government of Canada, Ottawa, ON

  4. Palko K, Lemmen DS (2017) Climate risks and adaptation practices for the canadian transportation sector 2016:27–64

  5. Radoux J, Chomé G, Jacques DC, Waldner F, Bellemans N, Matton N, Lamarche C, D’Andrimont R, Defourny P (2016) Sentinel-2’s potential for sub-pixel landscape feature detection. Remote Sens. https://doi.org/10.3390/rs8060488

    Article  Google Scholar 

  6. Enache S, Clerc S (2023) Sentinel-2 L1C data quality report. https://sentinel.esa.int/documents/247904/4868341/OMPC.CS.DQR.001.12-2022+-+i83r0+-+MSI+L1C+DQR+January+2023.pdf. [Online; accessed 2-February-2023]

  7. European Space Agency: Data Access of Copernicus historical Sentinel-2 Collection-1 products starting in January 2023. https://sentinels.copernicus.eu/web/sentinel/-/data-access-of-copernicus-historical-sentinel-2-collection-1-products-starting-in-january-2023. [Online; accessed 6-June-2023] (2023)

  8. Scheffler D, Hollstein A, Diedrich H, Segl K, Hostert P (2017) Arosics: an automated and robust open-source image co-registration software for multi-sensor satellite data. Remote Sens. https://doi.org/10.3390/rs9070676

    Article  Google Scholar 

  9. Skakun S, Roger J-C, Vermote EF, Masek JG, Justice CO (2017) Automatic sub-pixel co-registration of landsat-8 operational land imager and sentinel-2a multi-spectral instrument images using phase correlation and machine learning based mapping. Int J Digit Earth 10(12):1253–1269. https://doi.org/10.1080/17538947.2017.1304586

    Article  Google Scholar 

  10. Stumpf A, Michéa D, Malet J-P (2018) Improved co-registration of sentinel-2 and landsat-8 imagery for earth surface motion measurements. Remote Sens. https://doi.org/10.3390/rs10020160

    Article  Google Scholar 

  11. Rufin P, Frantz D, Yan L, Hostert P (2021) Operational coregistration of the sentinel-2a/b image archive using multitemporal landsat spectral averages. IEEE Geosci Remote Sens Lett 18(4):712–716. https://doi.org/10.1109/LGRS.2020.2982245

    Article  ADS  Google Scholar 

  12. Evangelidis GD, Psarakis EZ (2008) Parametric image alignment using enhanced correlation coefficient maximization. IEEE Trans Pattern Anal Mach Intell 30(10):1858–1865. https://doi.org/10.1109/TPAMI.2008.113

    Article  PubMed  Google Scholar 

  13. Zhou L, Zhang C, Wu M (2018) D-linknet: Linknet with pretrained encoder and dilated convolution for high resolution satellite imagery road extraction. In: 2018 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), pp. 192–1924. https://doi.org/10.1109/CVPRW.2018.00034

  14. Brown LG (1992) A survey of image registration techniques. ACM Comput Surv 24(4):325–376. https://doi.org/10.1145/146370.146374

    Article  Google Scholar 

  15. Zitová B, Flusser J (2003) Image registration methods: a survey. Image Vis Comput 21(11):977–1000. https://doi.org/10.1016/S0262-8856(03)00137-9

    Article  Google Scholar 

  16. Fonseca LMG, Manjunath BS (1996) Registration techniques for multisensor remotely sensed imagery. Photogramm Eng Remote Sens 62:1049–1056

    Google Scholar 

  17. Dawn S, Saxena V, Sharma B (2010) Remote sensing image registration techniques: a survey. In: Elmoataz A, Lezoray O, Nouboud F, Mammass D, Meunier J (eds) Image and signal processing. Springer, Berlin, Heidelberg, pp 103–112

  18. Moigne JL, Netanyahu NS, Eastman RD (2018) Image registration for remote sensing. Cambridge University Press, USA

    Google Scholar 

  19. Tondewad MPS, Dale MMP (2020) Remote sensing image registration methodology: review and discussion. Procedia Computer Science 171, 2390–2399. https://doi.org/10.1016/j.procs.2020.04.259 . Third International Conference on Computing and Network Communications (CoCoNet’19)

  20. Yan L, Roy DP, Zhang H, Li J, Huang H (2016) An automated approach for sub-pixel registration of landsat-8 operational land imager (oli) and sentinel-2 multi spectral instrument (msi) imagery. Remote Sens. https://doi.org/10.3390/rs8060520

    Article  Google Scholar 

  21. Guizar-Sicairos M, Thurman ST, Fienup JR (2008) Efficient subpixel image registration algorithms. Opt Lett 33(2):156–158. https://doi.org/10.1364/OL.33.000156

    Article  ADS  PubMed  Google Scholar 

  22. Rosu A-M, Pierrot-Deseilligny M, Delorme A, Binet R, Klinger Y (2015) Measurement of ground displacement from optical satellite image correlation using the free open-source software micmac. ISPRS J Photogramm Remote Sens 100:48–59. https://doi.org/10.1016/j.isprsjprs.2014.03.002 . High-Resolution Earth Imaging for Geospatial Information

  23. Zhu Z, Wang S, Woodcock CE (2015) Improvement and expansion of the fmask algorithm: cloud, cloud shadow, and snow detection for landsats 4-7, 8, and sentinel 2 images. Remote Sens Environ 159:269–277. https://doi.org/10.1016/j.rse.2014.12.014

    Article  ADS  Google Scholar 

  24. Zupanc A (2017) Improving cloud detection with machine learning. https://medium.com/sentinel-hub/improving-cloud-detection-with-machine-learning-c09dc5d7cf13. [Online; accessed 10-May-2023]

  25. Foroosh H, Zerubia JB, Berthod M (2002) Extension of phase correlation to subpixel registration. IEEE Trans Image Process 11(3):188–200. https://doi.org/10.1109/83.988953

    Article  ADS  PubMed  Google Scholar 

  26. Lian R, Wang W, Mustafa N, Huang L (2020) Road extraction methods in high-resolution remote sensing images: a comprehensive review. IEEE J Sel Top Appl Earth Obs Remote Sens 13:5489–5507. https://doi.org/10.1109/JSTARS.2020.3023549

    Article  ADS  Google Scholar 

  27. Mnih V, Hinton GE (2010) Learning to detect roads in high-resolution aerial images. In: Daniilidis K, Maragos P, Paragios N (eds) Computer Vision—ECCV 2010. Springer, Berlin, Heidelberg, pp 210–223

    Chapter  Google Scholar 

  28. Mnih V (2013) Machine learning for aerial image labeling. PhD thesis. University of Toronto

    Google Scholar 

  29. Demir I, Koperski K, Lindenbaum D, Pang G, Huang J, Basu S, Hughes F, Tuia D, Raskar R (2018) Deepglobe 2018: a challenge to parse the earth through satellite images. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) workshops

  30. Oehmcke S, Thrysøe C, Borgstad A, Salles MAV, Brandt M, Gieseke F (2019) Detecting hardly visible roads in low-resolution satellite time series data. In: 2019 IEEE international conference on big data (big data), pp 2403–2412. https://doi.org/10.1109/BigData47090.2019.9006251

  31. Ayala C, Aranda C, Galar M (2021) Towards fine-grained road maps extraction using sentinel-2 imagery. Isprs Ann Photogramm Remote Sens Spatial Inf Sci 5(3):9–14

    Article  ADS  Google Scholar 

  32. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention - MICCAI 2015. Springer, Cham, pp 234–241

    Chapter  Google Scholar 

  33. Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  34. Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE international conference on computer vision (ICCV)

  35. He K, Zhang X., Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  36. Yu F, Koltun V, Funkhouser T (2017) Dilated residual networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  37. Zhang Z, Liu Q, Wang Y (2018) Road extraction by deep residual u-net. IEEE Geosci Remote Sens Lett 15(5):749–753. https://doi.org/10.1109/LGRS.2018.2802944

    Article  ADS  CAS  Google Scholar 

Download references

Acknowledgements

This project was supported by ESA Network of Resources Initiative.

Funding

The authors acknowledge the financial support of the New Frontiers in Research Fund—Exploration Grant [NFRF-2018-00966] as well as University of Manitoba Graduate Fellowship (UMGF).

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of Manitoba, 423 University Crescent, Winnipeg, MB, R3T 2N2, Canada

    Andres F. Pérez & Ahmed Ashraf

  2. Department of Civil, Geological and Mining Engineering, Polytechnique Montréal, 2500 Chem. de Polytechnique, Montréal, QC, H3T 1J4, Canada

    Pooneh Maghoul

  3. Department of Civil Engineering, University of Manitoba, 15 Gillson Street, Winnipeg, MB, R3T 5V6, Canada

    Pooneh Maghoul

Authors
  1. Andres F. Pérez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pooneh Maghoul
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed Ashraf
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, methodology, formal analysis, and review and editing were contributed by AP, PM, and AA. Software and algorithm implementation, original draft preparation, and writing were contributed by AP. All authors have read and agreed to the submitted version of the manuscript.

Corresponding authors

Correspondence to Andres F. Pérez, Pooneh Maghoul or Ahmed Ashraf.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pérez, A.F., Maghoul, P. & Ashraf, A. A deep learning approach to satellite image time series coregistration through alignment of road networks. Neural Comput & Applic 36, 3583–3593 (2024). https://doi.org/10.1007/s00521-023-09242-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09242-0

Keywords

Search

Navigation