Skip to main content
SpringerLink
Log in

Deep Learning on Image Stitching With Multi-viewpoint Images: A Survey

  • Review
  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Multi-viewpoint image stitching aims to stitch images taken from different viewpoints into pictures with a broader field of view. The stitched images are subject to artifacts, geometric distortion, and blur distortion due to the mismatch of feature points, inaccurate homography estimation, and improper fusion of the unstitched images. Deep learning has recently been increasingly applied to multi-viewpoint image stitching to overcome these problems. However, there has thus far been little related research to summarize the different deep learning techniques used for multi-viewpoint image stitching. Therefore, this review aims to explore the application of deep learning to multi-viewpoint image stitching. To better illustrate this topic, we first summarize the acquisition methods for multi-viewpoint images and the main challenges of image stitching. After which, deep learning techniques for multi-view image stitching with a single camera are sorted out. Subsequently, deep learning techniques for multi-view image stitching with camera arrays, including parallel-view multi-view image stitching and cross-view multi-view image stitching, are presented. Next, we summarize image stitching datasets, evaluation metrics, and experimental data of several leading stitching algorithms on public datasets. Finally, we discuss potential issues and future work on image stitching with multi-viewpoint images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://drive.google.com/drive/folders/1kC7KAULd5mZsqaWnY3-rSbQLaZ7LujTY?usp=sharing.

  2. https://drive.google.com/file/d/1KR5DtekPJin3bmQPlTGP4wbM1zFR80ak/view?usp=sharing.

References

  1. Laraqui A, Baataoui A, Saaidi A, Jarrar A, Masrar M, Satori K (2017) Image mosaicing using voronoi diagram. Multimed Tools Appl 76(6):8803–8829

    Article  Google Scholar 

  2. Ali I, Suominen OJ, Morales ER, Gotchev A (2020) Multi-view camera pose estimation for robotic arm manipulation. IEEE Access 8:174305–174316

    Article  Google Scholar 

  3. Ding Y, Li F, Ji Y, Yu J (2011) Dynamic fluid surface acquisition using a camera array. In: 2011 International conference on computer vision (pp 2478–2485). IEEE

  4. Sabater N, Boisson G, Vandame B, Kerbiriou P, Babon F, Hog M et al (2017) Dataset and pipeline for multi-view light-field video. In: Proceedings of the IEEE conference on computer vision and pattern recognition Workshops (pp 30–40)

  5. Nie L, Lin C, Liao K, Liu S, Zhao Y (2021) Unsupervised deep image stitching: reconstructing stitched features to images. IEEE Trans Image Process 30:6184–6197

    Article  MathSciNet  Google Scholar 

  6. DeTone D, Malisiewicz T, Rabinovich A (2016) Deep image homography estimation. arXiv preprint https://arxiv.org/abs/arXiv:1606.03798.

  7. Nguyen T, Chen SW, Shivakumar SS, Taylor CJ, Kumar V (2018) Unsupervised deep homography: a fast and robust homography estimation model. IEEE Robot Autom Lett 3(3):2346–2353

    Article  Google Scholar 

  8. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint https://arxiv.org/abs/arXiv:1409.1556

  9. Ye N, Wang C, Fan H, Liu S (2021) Motion basis learning for unsupervised deep homography estimation with subspace projection. In: Proceedings of the IEEE/CVF international conference on computer vision (pp 13117–13125)

  10. Niblick D, Kak A (2020) Homography estimation with convolutional neural networks under conditions of variance. arXiv preprint https://arxiv.org/abs/arXiv:2010.01041.

  11. Wu H, Zheng S, Zhang J, Huang K (2019) Gp-gan: towards realistic high-resolution image blending. In Proceedings of the 27th ACM international conference on multimedia (pp 2487–2495)

  12. Lai WS, Gallo O, Gu J, Sun D, Yang MH, Kautz J (2019) Video stitching for linear camera arrays. arXiv preprint https://arxiv.org/abs/arXiv:1907.13622

  13. Sheng M, Tang S, Cui Z, Wu W, Wan L (2020) A joint framework for underwater sequence images stitching based on deep neural network convolutional neural network. Int J Adv Rob Syst 17(2):1729881420915062

    Google Scholar 

  14. Jin, S., Liu, R., Ji, Y., Ye, J., Yu, J. (2018). Learning to dodge a bullet: Concyclic view morphing via deep learning. In: Proceedings of the European conference on computer vision (ECCV) (pp 218–233)

  15. Fotouhi J, Liu X, Armand M, Navab N, Unberath M (2021) Reconstruction of orthographic mosaics from perspective X-ray images. IEEE Trans Med Imaging 40(11):3165–3177

    Article  Google Scholar 

  16. Zhu A, Zhang L, Chen J, Zhou Y (2021) Pedestrian-aware panoramic video stitching based on a structured camera array. ACM Trans Multimed Comput Commun Appl TOMM 17(4):1–24

    Article  Google Scholar 

  17. Cheng H, Xu C, Wang J, Zhao L (2022) Quad-fisheye image stitching for monoscopic panorama reconstruction. Comput Graph Forum. https://doi.org/10.1111/cgf.14512

    Article  Google Scholar 

  18. Perazzi F, Sorkine-Hornung A, Zimmer H, Kaufmann P, Wang O, Watson S, Gross M (2015) Panoramic video from unstructured camera arrays. Comput Graph Forum 34(2):57–68

    Article  Google Scholar 

  19. Yuan X, Ji M, Wu J, Brady DJ, Dai Q, Fang L (2021) A modular hierarchical array camera. Light Sci Appl 10(1):1–9

    Article  Google Scholar 

  20. Zhao Q, Ma Y, Zhu C, Yao C, Feng B, Dai F (2021) Image stitching via deep homography estimation. Neurocomputing 450:219–229

    Article  Google Scholar 

  21. Yu F, Koltun V, Funkhouser T (2017) Dilated residual networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 472–480)

  22. Nie L, Lin C, Liao K, Liu M, Zhao Y (2020) A view-free image stitching network based on global homography. J Vis Commun Image Represent 73:102950

    Article  Google Scholar 

  23. Dai Q, Fang F, Li J, Zhang G, Zhou A (2021) Edge-guided composition network for image stitching. Pattern Recogn 118:108019

    Article  Google Scholar 

  24. Xie S, Tu Z (2015) Holistically-nested edge detection. In: Proceedings of the IEEE international conference on computer vision (pp 1395–1403)

  25. Nie L, Lin C, Liao K, Zhao Y (2020) Learning edge-preserved image stitching from large-baseline deep homography. arXiv preprint https://arxiv.org/abs/arXiv:2012.06194

  26. Kweon H, Kim H, Kang Y, Yoon Y, Jeong W, Yoon KJ (2021) Pixel-wise deep image stitching. arXiv preprint https://arxiv.org/abs/arXiv:2112.06171

  27. Ilg E, Mayer N, Saikia T, Keuper M, Dosovitskiy A, Brox T (2017). Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp 2462–2470)

  28. Teed Z, Deng J (2020) Raft: Recurrent all-pairs field transforms for optical flow. In: Vedaldi A, Bischof H, Brox T, Frahm JM (eds) European conference on computer vision. Springer, Cham, pp 402–419

    Google Scholar 

  29. Nie L, Lin C, Liao K, Liu S, Zhao Y (2022) Deep rectangling for image stitching: a learning baseline. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 5740–5748)

  30. Smith SM, Brady JM (1997) Susan: a new approach to low level image processing. Int J Comput Vision 23(1):45–78

    Article  Google Scholar 

  31. Trajkovic M, Hedley M (1998) Fast corner detection. Image Vis Comput 16(2):75–87

    Article  Google Scholar 

  32. Tian Y, Balntas V, Ng T, Barroso-Laguna A, Demiris Y, Mikolajczyk K (2020) D2d: Keypoint extraction with describe to detect approach. In: Proceedings of the Asian conference on computer vision

  33. Verdie Y, Yi K, Fua P, Lepetit V (2015) Tilde: a temporally invariant learned detector. Learning Covariant Feature Detectors

  34. Lenc K, Vedaldi A (2016) Learning covariant feature detectors. In: European conference on computer vision (pp 100–117). Springer, Cham

  35. Zhang X, Yu FX, Karaman S, Chang SF (2017) Learning discriminative and transformation covariant local feature detectors. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 6818–6826)

  36. Doiphode N, Mitra R, Ahmed S, Jain A (2018) An improved learning framework for covariant local feature detection. In: Asian conference on computer vision (pp 262–276). Springer, Cham

  37. Hoffer E, Ailon N (2015) Deep metric learning using triplet network. In: International workshop on similarity-based pattern recognition (pp 84–92). Springer, Cham

  38. Mishkin D, Radenovic F, Matas J (2018) Repeatability is not enough: learning affine regions via discriminability. In: Proceedings of the European conference on computer vision (ECCV) (pp 284–300)

  39. Barroso-Laguna A, Riba E, Ponsa D, Mikolajczyk K (2019) Key.net: keypoint detection by handcrafted and learned CNN filters. In: Proceedings of the IEEE international conference on computer vision, pp 5836–5844

  40. Altwaijry H, Veit A, Belongie SJ, Tech C (2016) Learning to detect and match keypoints with deep architectures. In BMVC

  41. Savinov N, Seki A, Ladicky L, Sattler T, Pollefeys M (2017) Quad-networks: unsupervised learning to rank for interest point detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 1822–1830)

  42. Zhang L, Rusinkiewicz S (2018) Learning to detect features in texture images. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 6325–6333)

  43. Zagoruyko S, Komodakis N (2015) Learning to compare image patches via convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 4353–4361)

  44. Han X, Leung T, Jia Y, Sukthankar R, Berg AC (2015) Matchnet: Unifying feature and metric learning for patch-based matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 3279–3286)

  45. Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: Proceedings of the European conference on computer vision, pp 404–417

  46. Simo-Serra E, Trulls E, Ferraz L, Kokkinos I, Fua P, Moreno-Noguer F (2015) Discriminative learning of deep convolutional feature point descriptors. In: Proceedings of the IEEE international conference on computer vision (pp 118–126)

  47. Dosovitskiy A, Springenberg JT, Riedmiller M, Brox T (2014) Discriminative unsupervised feature learning with convolutional neural networks. Adv Neural Inf Process Syst, 27

  48. Masci J, Migliore D, Bronstein MM, Schmidhuber J (2014) Descriptor learning for omnidirectional image matching. In: Registration and recognition in images and videos (pp 49–62). Springer, Berlin, Heidelberg

  49. Kumar BGV, Carneiro G, Reid I (2016) Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 5385–5394)

  50. Balntas V, Johns E, Tang L, Mikolajczyk K (2016) PN-Net: conjoined triple deep network for learning local image descriptors. arXiv preprint https://arxiv.org/abs/arXiv:1601.05030

  51. Balntas V, Riba E, Ponsa D, Mikolajczyk K (2016) Learning local feature descriptors with triplets and shallow convolutional neural networks. In Bmvc (Vol 1, No 2, p 3)

  52. Tian Y, Fan B, Wu F (2017) L2-net: Deep learning of discriminative patch descriptor in euclidean space. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 661–669)

  53. Mishchuk A, Mishkin D, Radenovic F, Matas J (2017) Working hard to know your neighbor's margins: Local descriptor learning loss. Adv Neural Inf Process Syst, 30

  54. Luo Z, Shen T, Zhou L et al (2018) Geodesc: learning local descriptors by integrating geometry constraints. In: Proceedings of the European conference on computer vision (ECCV) (pp 168–183)

  55. Tian Y, Yu X, Fan B, Wu F, Heijnen H, Balntas V (2019) Sosnet: Second order similarity regularization for local descriptor learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 11016–11025)

  56. Ebel P, Mishchuk A, Yi KM, Fua P, Trulls E (2019) Beyond cartesian representations for local descriptors. In: Proceedings of the IEEE/CVF international conference on computer vision (pp 253–262)

  57. Chen PH, Luo ZX, Huang ZK, Yang C, Chen KW (2020) IF-Net: an illumination-invariant feature network. In: 2020 IEEE international conference on robotics and automation (ICRA) (pp 8630–8636). IEEE

  58. Keller M, Chen Z, Maffra F, Schmuck P, Chli M (2018) Learning deep descriptors with scale-aware triplet networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp 2762–2770)

  59. He K, Lu Y, Sclaroff S (2018) Local descriptors optimized for average precision. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 596–605)

  60. Zhou Q, Sattler T, Leal-Taixe L (2021) Patch2pix: epipolar-guided pixel-level correspondences. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4669–4678)

  61. Rocco I, Cimpoi M, Arandjelovi ́c R, Torii A, Pajdla T, Sivic J (2018) Neighbourhood consensus networks. In: NeurIPS (pp 1651–1662)

  62. Yao G, Yilmaz A, Zhang L, Meng F, Ai H, Jin F (2021) Matching large baseline oblique stereo images using an end-to-end convolutional Neural network. Remote Sensing 13(2):274

    Article  Google Scholar 

  63. Yi KM, Trulls E, Lepetit V, Fua P (2016) Lift: learned invariant feature transform. In: European conference on computer vision (pp 467–483). Springer, Cham

  64. DeTone D, Malisiewicz T, Rabinovich A (2018) Superpoint: self-supervised interest point detection and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp 224–236)

  65. Li H, Li F (2013) Image encode method based on ifs with probabilities applying in image retrieval. In: 2013 Fourth global congress on intelligent systems (pp 291–295). IEEE

  66. Lie WN, Gao ZW (2006) Video error concealment by integrating greedy suboptimization and Kalman filtering techniques. IEEE Trans Circuits Syst Video Technol 16(8):982–992

    Article  Google Scholar 

  67. Christiansen PH, Kragh MF, Brodskiy Y, Karstoft H (2019) Unsuperpoint: end-to-end unsupervised interest point detector and descriptor. arXiv preprint https://arxiv.org/abs/arXiv:1907.04011

  68. Revaud J, Weinzaepfel P, De Souza C, Pion N, Csurka G, Cabon Y, Humenberger M (2019) R2D2: repeatable and reliable detector and descriptor. arXiv preprint https://arxiv.org/abs/arXiv:1906.06195

  69. Sarlin PE, DeTone D, Malisiewicz T, Rabinovich A (2020) Superglue: learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 4938–4947)

  70. Ono Y, Trulls E, Fua P, Yi KM (2018) LF-Net: Learning local features from images. Adv Neural Inf Process Syst, 31

  71. Sarlin PE, Cadena C, Siegwart R, Dymczyk M (2019). From coarse to fine: robust hierarchical localization at large scale. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 12716–12725)

  72. Wang Q, Zhang J, Yang K, Peng K, Stiefelhagen R (2022) MatchFormer: interleaving attention in transformers for feature matching. arXiv preprint https://arxiv.org/abs/arXiv:2203.09645

  73. Zhao X, Wu X, Miao J, Chen W, Chen PC, Li Z (2022) ALIKE: accurate and lightweight keypoint detection and descriptor extraction. IEEE Trans Multimed. https://doi.org/10.1109/TMM.2022.3155927

    Article  Google Scholar 

  74. Noh H, Araujo A, Sim J, Weyand T, Han B (2017) Large-scale image retrieval with attentive deep local features. In: Proceedings of the IEEE international conference on computer vision (pp 3456–3465).

  75. Dusmanu M, Rocco I, Pajdla T, Pollefeys M, Sivic J, Torii A, Sattler T (2019) D2-net: a trainable cnn for joint description and detection of local features. In Proceedings of the Ieee/cvf conference on computer vision and pattern recognition (pp 8092–8101)

  76. Luo Z, Zhou L, Bai X et al (2020) Aslfeat: learning local features of accurate shape and localization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 6589–6598)

  77. Yang TY, Nguyen DK, Heijnen H, Balntas V (2020) Ur2kid: unifying retrieval, keypoint detection, and keypoint description without local correspondence supervision. arXiv preprint https://arxiv.org/abs/arXiv:2001.07252.

  78. Tyszkiewicz M, Fua P, Trulls E (2020) DISK: learning local features with policy gradient. Adv Neural Inf Process Syst 33:14254–14265

    Google Scholar 

  79. Erlik Nowruzi F, Laganiere R, Japkowicz N (2017) Homography estimation from image pairs with hierarchical convolutional networks. In: Proceedings of the IEEE international conference on computer vision workshops (pp 913–920)

  80. Chang CH, Chou CN, Chang EY (2017) Clkn: Cascaded lucas-kanade networks for image alignment. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 2213–2221)

  81. Zhang J, Wang C, Liu S et al. (2020) Content-aware unsupervised deep homography estimation. In: European conference on computer vision (pp 653–669). Springer, Cham

  82. Zeng R, Denman S, Sridharan S, Fookes C (2018). Rethinking planar homography estimation using perspective fields. In: Asian conference on computer vision (pp 571–586). Springer, Cham

  83. Zhou Q, Li X (2019) Stn-homography: direct estimation of homography parameters for image pairs. Appl Sci 9(23):5187

    Article  Google Scholar 

  84. Wang C, Wang X, Bai X, Liu Y, Zhou J (2019) Self-supervised deep homography estimation with invertibility constraints. Pattern Recogn Lett 128:355–360

    Article  Google Scholar 

  85. Nie L, Lin C, Liao K, Liu S, Zhao Y (2021) Depth-aware multi-grid deep homography estimation with contextual correlation. arXiv preprint https://arxiv.org/abs/arXiv:2107.02524

  86. Koguciuk D, Arani E, Zonooz B (2021) Perceptual loss for robust unsupervised homography estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 4274–4283)

  87. Zhao Y, Huang X, Zhang Z (2021) Deep lucas-kanade homography for multimodal image alignment. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 15950–15959)

  88. Le H, Liu F, Zhang S, Agarwala A (2020) Deep homography estimation for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp 7652–7661).

  89. Shao R, Wu G, Zhou Y, Fu Y, Fang L, Liu Y (2021) Localtrans: a multiscale local transformer network for cross-resolution homography estimation. In Proceedings of the IEEE/CVF international conference on computer vision (pp 14890–14899)

  90. Zhang L, Wen T, Shi J (2020) Deep image blending. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp 231–240)

  91. Zheng C, Xia S, Robinson J, Lu C, Wu W, Qian C, Shao M (2020) Localin Reshuffle net: toward naturally and efficiently facial image blending. In: Proceedings of the Asian conference on computer vision

  92. Burt PJ, Adelson EH (1983) A multiresolution spline with application to image mosaics. ACM Trans Graph (TOG) 2(4):217–236

    Article  Google Scholar 

  93. Zhang H, Zhang J, Perazzi F, Lin Z, Patel VM (2021) Deep image compositing. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp 365–374).

  94. Lu CN, Chang YC, Chiu WC (2021) Bridging the visual gap: wide-range image blending. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 843–851). https://doi.org/10.1109/CVPR46437.2021.00090

  95. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 5505–5514)

  96. Goodfellow I, Pouget-Abadie J, Mirza M et al (2020) Generative adversarial networks. Commun ACM 63(11):139–144

    Article  MathSciNet  Google Scholar 

  97. Nazeri K, Ng E, Joseph T, Qureshi FZ, Ebrahimi M (2019) Edgeconnect: generative image inpainting with adversarial edge learning. arXiv preprint https://arxiv.org/abs/arXiv:1901.00212

  98. Xiong W, Yu J, Lin Z, Yang J, Lu X, Barnes C, Luo J (2019) Foreground-aware image inpainting. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 5840–5848)

  99. Ren Y, Yu X, Zhang R, Li TH, Liu S, Li G (2019) Structureflow: Image inpainting via structure-aware appearance flow. In: Proceedings of the IEEE/CVF international conference on computer vision (pp 181–190)

  100. Li J, He F, Zhang L, Du B, Tao D (2019) Progressive reconstruction of visual structure for image inpainting. In: Proceedings of the IEEE/CVF international conference on computer vision (pp 5962–5971)

  101. Song Y, Yang C, Shen Y, Wang P, Huang Q, Kuo CCJ (2018) Spg-net: Segmentation prediction and guidance network for image inpainting. arXiv preprint https://arxiv.org/abs/arXiv:1805.03356

  102. Liao L, Xiao J, Wang Z, Lin CW, Satoh SI (2020) Guidance and evaluation: semantic-aware image inpainting for mixed scenes. In: European conference on computer vision (pp 683–700). Springer, Cham

  103. Liao L, Xiao J, Wang Z, Lin CW, Satoh SI (2021) Image inpainting guided by coherence priors of semantics and textures. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 6539–6548)

  104. Zhang W, Wang Y, Zhu J, Tai Y, Ni B, Yang X (2021) Fully context-aware image inpainting with a learned semantic pyramid. arXiv preprint https://arxiv.org/abs/arXiv:2112.04107

  105. Wang L, Yu W, Li B (2020) Multi-scenes image stitching based on autonomous driving. In 2020 IEEE 4th information technology, networking, electronic and automation control conference (ITNEC) (Vol 1, pp 694–698). IEEE

  106. Sumantri JS, Park IK (2020) 360 Panorama synthesis from a sparse set of images with unknown field of view. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp 2386–2395)

  107. Li J, Zhao Y, Ye W, Yu K, Ge S (2019) Attentive deep stitching and quality assessment for 360° omnidirectional images. IEEE J Sel Top Signal Process 14(1):209–221

    Article  Google Scholar 

  108. Kang L, Wei Y, Jiang J, Xie Y (2019) Robust cylindrical panorama stitching for low-texture scenes based on image alignment using deep learning and iterative optimization. Sensors 19(23):5310

    Article  Google Scholar 

  109. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention (pp 234–241). Springer, Cham

  110. Gupta R, Hartley RI (1997) Linear pushbroom cameras. IEEE Trans Pattern Anal Mach Intell 19(9):963–975

    Article  Google Scholar 

  111. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D et al (2014) Microsoft coco: Common objects in context. In European conference on computer vision (pp 740–755). Springer, Cham

  112. Armeni I, Sax S, Zamir AR, Savarese S (2017) Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint https://arxiv.org/abs/arXiv:1702.01105

  113. Gao J, Li Y, Chin TJ, Brown MS (2013) Seam-driven image stitching. In: Eurographics (Short Papers) (pp 45–48)

  114. Lin K, Jiang N, Cheong LF, Do M, Lu J (2016) Seagull: Seam-guided local alignment for parallax-tolerant image stitching. In: Leibe B, Matas J, Sebe N, Welling M (eds) European conference on computer vision. Springer, Cham, pp 370–385

    Google Scholar 

  115. Li N, Liao T, Wang C (2018) Perception-based seam cutting for image stitching. SIViP 12(5):967–974

    Article  Google Scholar 

  116. Unberath M, Zaech JN, Lee SC, Bier B, Fotouhi J, Armand M, Navab N (2018) DeepDRR–a catalyst for machine learning in fluoroscopy-guided procedures. In: Frangi AF, Schnabel JA, Davatzikos C, Alberola-López C, Fichtinger G (eds) International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 98–106

    Google Scholar 

  117. Unberath M, Zaech JN, Gao C, Bier B, Goldmann F, Lee SC et al (2019) Enabling machine learning in X-ray-based procedures via realistic simulation of image formation. Int J Comput Assist Radiol Surg 14(9):1517–1528

    Article  Google Scholar 

  118. Dosovitskiy A, Ros G, Codevilla F, Lopez A, Koltun V (2017). CARLA: an open urban driving simulator. In Conference on robot learning (pp 1–16). PMLR

  119. Li J, Yu K, Zhao Y, Zhang Y, Xu L (2019) Cross-reference stitching quality assessment for 360 omnidirectional images. In Proceedings of the 27th ACM international conference on multimedia (pp 2360–2368).

  120. Varol G, Romero J, Martin X, Mahmood N, Black MJ, Laptev I, Schmid C (2017) Learning from synthetic humans. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp 109–117)

  121. Loper M, Mahmood N, Romero J, Pons-Moll G, Black MJ (2015) SMPL: a skinned multi-person linear model. ACM Trans Graph (TOG) 34(6):1–16

    Article  Google Scholar 

  122. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  123. Yang L, Tan Z, Huang Z, Cheung G (2017) A content-aware metric for stitched panoramic image quality assessment. In Proceedings of the IEEE international conference on computer vision workshops (pp 2487–2494)

  124. Li J, Yu K, Zhao Y, Zhang Y, Xu L (2019) Cross-reference stitching quality assessment for 360 omnidirectional images. In Proceedings of the 27th ACM international conference on multimedia (pp 2360–2368)

  125. Yu K, Li J, Zhang Y, Zhao Y, Xu L (2019) Image quality assessment for omnidirectional cross-reference stitching. arXiv preprint https://arxiv.org/abs/arXiv:1904.04960

  126. Yan W, Yue G, Fang Y, Chen H, Tang C, Jiang G (2020) Perceptual objective quality assessment of stereoscopic stitched images. Signal Process 172:107541

    Article  Google Scholar 

  127. Yu S, Li T, Xu X, Tao H, Yu L, Wang Y (2019) NRQQA: A no-reference quantitative quality assessment method for stitched images. In: Proceedings of the ACM multimedia Asia (pp. 1–6). 118

  128. Ullah H, Irfan M, Han K, Lee JW (2020) DLNR-SIQA: deep learning-based No-reference stitched image quality assessment. Sensors 20(22):6457

    Article  Google Scholar 

  129. Tian C, Chai X, Shao F (2021) Stitched image quality assessment based on local measurement errors and global statistical properties. J Vis Commun Image Represent 81:103324

    Article  Google Scholar 

  130. Dusmanu M, Rocco I, Pajdla T, Pollefeys M, Sivic J, Torii A, Sattler T (2019) D2-net: a trainable CNN for joint detection and description of local features. arXiv preprint https://arxiv.org/abs/arXiv:1905.03561

  131. Po LM, Liu M, Yuen WY, Li Y, Xu X, Zhou C et al (2019) A novel patch variance biased convolutional neural network for no-reference image quality assessment. IEEE Trans Circuits Syst Video Technol 29(4):1223–1229

    Article  Google Scholar 

  132. Bosse S, Maniry D, Müller KR, Wiegand T, Samek W (2017) Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans Image Process 27(1):206–219

    Article  MathSciNet  MATH  Google Scholar 

  133. Zhang Y, Mei X, Ma Y, Jiang X, Peng Z, Huang J (2022) Hyperspectral panoramic image stitching using robust matching and adaptive bundle adjustment. Remote Sensing 14(16):4038

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China “Analysis and feature recognition on feeding behavior of fish school in facility farming based on machine vision” (No. 62076244), in part by the Beijing Digital Agriculture Innovation Consortium Project (BAIC10-2022), and in part by the National Natural Science Foundation of China “Intelligent identification method of underwater fish morphological characteristics based on the binocular vision” (No. 62206021).

Author information

Authors and Affiliations

  1. National Innovation Center for Digital Fishery, China Agricultural University, 17 Tsinghua East Road, Beijing, 100083, People’s Republic of China

    Ni Yan, Yupeng Mei, Ling Xu, Boyang Sun, Zimao Wang & Yingyi Chen

  2. Key Laboratory of Smart Farming for Aquatic Animal and Livestock, Ministry of Agriculture and Rural Affairs, Beijing, 100083, People’s Republic of China

    Ni Yan, Yupeng Mei, Ling Xu, Boyang Sun, Zimao Wang & Yingyi Chen

  3. Beijing Engineering and Technology Research Centre for the Internet of Things in Agriculture, China Agricultural University, Beijing, 100083, People’s Republic of China

    Ni Yan, Yupeng Mei, Ling Xu, Boyang Sun, Zimao Wang & Yingyi Chen

  4. College of Information and Electrical Engineering, China Agricultural University, Beijing, 100083, People’s Republic of China

    Ni Yan, Yupeng Mei, Ling Xu, Boyang Sun, Zimao Wang & Yingyi Chen

  5. School of Information Science and Technology, Beijing Forestry University, Beijing, 100083, People’s Republic of China

    Huihui Yu

Authors
  1. Ni Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yupeng Mei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ling Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huihui Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Boyang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zimao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yingyi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Ni Yan: Conceptualization, Methodology, Investigation, Visualization, Writing - original draft, Writing - review & editing. Yupeng Mei: Conceptualization, Methodology, Visualization, Writing - review & editing. Ling Xu: Conceptualization, Methodology, Writing - review & editing. Huihui Yu: Methodology, Writing - review & editing. Boyang Sun: Methodology, Writing - review & editing. Zimao Wang: Methodology, Writing - review & editing. Yingyi Chen: Conceptualization, Methodology, Funding acquisition, Project administration, Writing - review & editing, Supervision.

Corresponding author

Correspondence to Yingyi Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, N., Mei, Y., Xu, L. et al. Deep Learning on Image Stitching With Multi-viewpoint Images: A Survey. Neural Process Lett 55, 3863–3898 (2023). https://doi.org/10.1007/s11063-023-11226-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11226-z

Keywords

Search

Navigation