Skip to main content
SpringerLink
Log in

Orthophoto improvement using urban-SnowflakeNet

  • Original Paper
  • Published:
Applied Geomatics Aims and scope Submit manuscript

Abstract

With the increasing use of drones for capturing images in urban areas, correcting for distortion and sawtooth effects on orthophotos generated with these images has become a critical issue. This is particularly challenging due to the larger displacements generated by high objects and lower flight altitude of drones compared to crewed aircraft. In addition, image-based point cloud generation methods often fail to produce complete point clouds due to occluded areas and radiometric changes between overlapping images, especially near the borders of high objects. To address these issues, a novel method is proposed in this article for improving the generated point clouds with image-based methods using a deep learning network, called urban-SnowflakeNet, which comprises the following steps: 1) preparing and normalizing the roof's point cloud; 2) completing the point clouds of the building using the proposed deep learning network; 3) restoring the completed point clouds of the buildings to the real coordinates and combining them with the background point cloud; and, 4) correcting the DSM and generating the final true orthophotos. On two different image datasets, our method reduced distortions at the building's edges by 40% on average when compared to the most recent orthophoto enhancement method. However, by maintaining this success on more datasets, the approach has the potential to improve the accuracy and completeness of point clouds in urban regions, as well as other applications such as 3D model improvement, which require further testing in future works.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Data availability

Please contact: m.ebrahimikia@email.kntu.ac.ir.

Notes

  1. center of projections.

References

  • Agisoft (2021) Agisoft PhotoScan User Manual: Professional Edition, Version 1.7. Agisoft LLC, St Petersburg (187 pages)

    Google Scholar 

  • Andrei O (2006) Master’s of Science Thesis in Geodesy No. 3091. School of Architecture and the Built Environment Royal Institute of Technology (KTH) 100 44 Stockholm, p 63

  • Barazzetti, L., Brumana, R., Oreni, D., Previtali, M., & Roncoroni, F. (2014). True-orthophoto generation from UAV images: implementation of a combined photogrammetric and computer vision approach. ISPRS Ann Photogramm Remote Sens Spatial Inf Sci 2(5). https://doi.org/10.5194/isprsannals-II-5-57-2014

  • Boccardo P, Dequal S, Lingua A, Rinaudo F (2001) True digital orthophoto for architectural and archaeological applications. Int Archi Photogramm Remote Sens Spat Inf Sci 34(5/W1):50–55

  • Canny, J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell 8(6):679–698

  • Chang Y, Jung C, Xu Y (2021) FinerPCN: High fidelity point cloud completion network using pointwise convolution. Neurocomputing 460:266–276. https://doi.org/10.1016/j.neucom.2021.06.080

    Article  Google Scholar 

  • Chen Y, Briese C, Karel W, Pfeifer N (2014) True orthophoto generation using multi-view aerial images. Int Arch Photogramm Remote Sens Spat Inf Sci 40(3):67. https://doi.org/10.5194/isprsarchives-XL-3-67-2014

    Article  CAS  Google Scholar 

  • Chen X, Chen B, Mitra NJ (2019) Unpaired point cloud completion on real scans using adversarial training. arXiv preprint arXiv:1904.00069. https://doi.org/10.48550/arXiv.1904.00069

  • Chibane J, Mir A, Pons-Moll G (2020) Neural unsigned distance fields for implicit function learning. arXiv preprint arXiv:2010.13938. https://doi.org/10.48550/arXiv.2010.13938

  • Czerniawski T, Ma JW, Leite F (2021) Automated building change detection with amodal completion of point clouds. Autom Constr 124:103568

    Article  Google Scholar 

  • Dai A, Ruizhongtai Qi C, Nießner M (2017) Shape completion using 3d-encoder-predictor cnns and shape synthesis. Paper presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  • Dai A, Diller C, Nießner M (2020) Sg-nn: Sparse generative neural networks for self-supervised scene completion of rgb-d scans. Paper presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

  • Dall'Asta E, Roncella R (2014) A comparison of semiglobal and local dense matching algorithms for surface reconstruction. Int Arch Photogramm Remote Sens Spat Inf Sci 45. https://doi.org/10.5194/isprsarchives-XL-5-187-2014

  • Dunbar D, Humphreys G (2006) A spatial data structure for fast Poisson-disk sample generation. ACM Trans Graph 25(3):503–508. https://doi.org/10.1145/1141911.1141915

  • Ebrahimikia M, Hosseininaveh A (2022) True orthophoto generation based on unmanned aerial vehicle images using reconstructed edge points. Photogram Rec. https://doi.org/10.1111/phor.12409

    Article  Google Scholar 

  • Furukawa Y, Ponce J (2009) Accurate, dense, and robust multiview stereopsis. IEEE Trans Pattern Anal Mach Intell 32(8):1362–1376. https://doi.org/10.1109/TPAMI.2009.161

    Article  Google Scholar 

  • Gkioxari G, Malik J, Johnson J (2019) Mesh r-cnn. Paper presented at the Proceedings of the IEEE/CVF International Conference on Computer Vision. https://doi.org/10.48550/arXiv.1906.02739

  • Groueix T, Fisher M, Kim V, Russell B, Aubry M (1802) AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. arXiv 2018. arXiv preprint arXiv:1802.05384. https://doi.org/10.48550/arXiv.1802.05384

  • Hirschmuller H (2008) Stereo processing by semiglobal matching and mutual information. IEEE Trans Pattern Anal Mach Intell 30(2):328–341. https://doi.org/10.1109/TPAMI.2007.1166

    Article  PubMed  Google Scholar 

  • Hou J, Dai A, Nießner M (2020) Revealnet: Seeing behind objects in rgb-d scans. Paper presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.1904.12012

  • Huang Z, Yu Y, Xu J, Ni F, Le X (2020) Pf-net: Point fractal network for 3d point cloud completion. Paper presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.2003.00410

  • Kato A, Moskal L, Schiess P, Calhoun D, Swanson ME (2010) True orthophoto creation through fusion of LiDAR derived digital surface model and aerial photos: ISPRS TC VII Symposium, Vienna, Austria, July 5–7, 2010, IAPRS, vol XXXVIII, Part 7A

  • Krauß T, Reinartz P (2010) Urban object detection using a fusion approach of dense urban digital surface models and VHR optical satellite stereo data. ISPRS Istanbul Workshop 2010, WG I/4

  • Li C-L, Zaheer M, Zhang Y, Poczos B, Salakhutdinov R (2018) Point cloud gan. arXiv preprint arXiv:1810.05795. https://doi.org/10.48550/arXiv.1810.05795

  • Liu M, Sheng L, Yang S, Shao J, Hu S-M (2020). Morphing and sampling network for dense point cloud completion. Paper presented at the Proceedings of the AAAI conference on artificial intelligence. https://doi.org/10.1609/aaai.v34i07.6827

  • Manhofer (2020) Line3Dpp, github. https://github.com/manhofer/Line3Dpp

  • Mohammadi H, Samadzadegan F, Reinartz P (2019) 2D/3D information fusion for building extraction from high-resolution satellite stereo images using kernel graph cuts. Int J Remote Sens 40(15):5835–5860. https://doi.org/10.1080/01431161.2019.1584417

    Article  Google Scholar 

  • Nielsen MØ (2004) True orthophoto generation: Informatics and mathematical modelling. IMM-Thesis. Dissertation No. 125. Lyngby,Technical University of Denmark, p 142

  • Noronha S, Nevatia R (2001) Detection and modeling of buildings from multiple aerial images. IEEE Trans Pattern Anal Mach Intell 23(5):501–518. https://doi.org/10.1109/34.922708

    Article  Google Scholar 

  • Qi CR, Su H, Mo K, Guibas LJ (2017) Pointnet: Deep learning on point sets for 3d classification and segmentation. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition. https://doi.org/10.48550/arXiv.1612.00593

  • Qin C, You H, Wang L, Kuo C-CJ, Fu Y (2019) Pointdan: A multi-scale 3d domain adaption network for point cloud representation. arXiv preprint arXiv:1911.02744. https://doi.org/10.48550/arXiv.1911.02744

  • Sarmad M, Lee HJ, Kim YM (2019) Rl-gan-net: A reinforcement learning agent controlled gan network for real-time point cloud shape completion. Paper presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.1904.12304

  • Siddiqui FU, Teng SW, Awrangjeb M, Lu G (2016) A robust gradient based method for building extraction from LiDAR and photogrammetric imagery. Sensors 16:1110. https://doi.org/10.3390/s16071110

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  • Tchapmi LP, Kosaraju V, Rezatofighi H, Reid I, Savarese S (2019) Topnet: Structural point cloud decoder. Paper presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

  • Venkatesh R, Karmali T, Sharma S, Ghosh A, Jeni LA, Babu RV, Singh M (2021) Deep Implicit Surface Point Prediction Networks. arXiv preprint arXiv:2106.05779. https://doi.org/10.48550/arXiv.2106.05779

  • Wang N, Zhang Y, Li Z, Fu Y, Liu W, Jiang Y-G (2018a) Pixel2mesh: Generating 3d mesh models from single rgb images. Paper presented at the Proceedings of the European Conference on Computer Vision (ECCV). https://doi.org/10.1007/978-3-030-01252-6_4

  • Wang Q, Yan L, Sun Y, Cui X, Mortimer H, Li Y (2018b) True orthophoto generation using line segment matches. Photogram Rec 33(161):113–130. https://doi.org/10.1111/phor.12229

    Article  Google Scholar 

  • Wang X, Ang Jr MH, Lee GH (2020) Cascaded refinement network for point cloud completion. Paper presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.2004.03327

  • Wang X, Jin Y, Cen Y, Wang T, Li Y (2021) Attention models for point clouds in deep learning: a survey. arXiv preprint arXiv:2102.10788. https://doi.org/10.48550/arXiv.2102.10788

  • Wei S (2008) Building boundary extraction based on lidar point clouds data. Proc Int Arch Photogramm Remote Sens Spat Inf Sci 37:157–161. https://doi.org/10.48550/arXiv.2102.10788

    Article  Google Scholar 

  • Widyaningrum E, Lindenbergh R, Gorte B, Zhou K (2018) Extraction of building roof edges from LiDAR data to optimize the digital surface model for true orthophoto generation. Int Arch Photogramm Remote Sens Spat Inf Sci 42:2. https://doi.org/10.5194/isprs-archives-XLII-2-1199-2018

    Article  Google Scholar 

  • Wu J, Jie S, Yao W, Stilla U (2011). Building boundary improvement for true orthophoto generation by fusing airborne LiDAR data. Paper presented at the Urban Remote Sensing Event (JURSE), 2011 Joint. https://doi.org/10.1109/JURSE.2011.5764735

  • Xiang P, Wen X, Liu Y-S, Cao Y-P, Wan P, Zheng W, Han Z (2021) SnowflakeNet: point cloud completion by snowflake point deconvolution with skip-transformer. ICCV 2021 (Oral), arXiv preprint arXiv:2108.04444. https://doi.org/10.48550/arXiv.2108.04444

  • Yang Y, Feng C, Shen Y, Tian D (2018) Foldingnet: Point cloud auto-encoder via deep grid deformation. Paper presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  • Yu L, Li X, Fu C-W, Cohen-Or D, Heng P-A (2018) Pu-net: Point cloud upsampling network. Paper presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.48550/arXiv.1801.06761

  • Yuan W, Khot T, Held D, Mertz C, Hebert M (2018) Pcn: Point completion network. Paper presented at the 2018 International Conference on 3D Vision (3DV). https://doi.org/10.48550/arXiv.1808.00671

  • Zhang Y, Liu Z, Li X, Zang Y (2019) Data-driven point cloud objects completion. Sensors 19(7):1514. https://doi.org/10.3390/s19071514

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  • Zhou Y, Wang L, Love PE, Ding L, Zhou C (2019) Three-dimensional (3D) reconstruction of structures and landscapes: a new point-and-line fusion method. Adv Eng Inform 42:100961. https://doi.org/10.1016/j.aei.2019.100961

    Article  Google Scholar 

Download references

Funding

No funding was obtained for this study.

Author information

Authors and Affiliations

  1. The Faculty of Surveying Engineering, K. N. Toosi University of Technology, Tehran, Iran

    Mojdeh Ebrahimikia & Ali Hosseininaveh

  2. The Faculty of Urban Planing, Malek Ashtar University of Technology, Tehran, Iran

    Mahdi Modiri

Authors
  1. Mojdeh Ebrahimikia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Hosseininaveh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mahdi Modiri
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Mojdeh Ebrahimikia and Ali Hosseininaveh guided Mojdeh to define the research question and supported her to provide the dataset and propose the method. The first draft of the manuscript was written by Mojdeh Ebrahimikia. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Mojdeh Ebrahimikia.

Ethics declarations

Competing interests

The authors have no competing interests in the submission of this manuscript.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ebrahimikia, M., Hosseininaveh, A. & Modiri, M. Orthophoto improvement using urban-SnowflakeNet. Appl Geomat (2024). https://doi.org/10.1007/s12518-024-00558-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s12518-024-00558-7

Keywords

Search

Navigation