Advertisement

The Visual Computer

, Volume 35, Issue 6–8, pp 823–835 | Cite as

Stereoscopic image stitching with rectangular boundaries

  • Yun ZhangEmail author
  • Yu-Kun Lai
  • Fang-Lue Zhang
Original Article

Abstract

This paper proposes a novel algorithm for stereoscopic image stitching, which aims to produce stereoscopic panoramas with rectangular boundaries. As a result, it provides wider field of view and better viewing experience for users. To achieve this, we formulate stereoscopic image stitching and boundary rectangling in a global optimization framework that simultaneously handles feature alignment, disparity consistency and boundary regularity. Given two (or more) stereoscopic images with overlapping content, each containing two views (for left and right eyes), we represent each view using a mesh and our algorithm contains three main steps: We first perform a global optimization to stitch all the left views and right views simultaneously, which ensures feature alignment and disparity consistency. Then, with the optimized vertices in each view, we extract the irregular boundary in the stereoscopic panorama, by performing polygon Boolean operations in left and right views, and construct the rectangular boundary constraints. Finally, through a global energy optimization, we warp left and right views according to feature alignment, disparity consistency and rectangular boundary constraints. To show the effectiveness of our method, we further extend our method to disparity adjustment and stereoscopic stitching with large horizon. Experimental results show that our method can produce visually pleasing stereoscopic panoramas without noticeable distortion or visual fatigue, thus resulting in satisfactory 3D viewing experience.

Keywords

Stereoscopic image stitching Rectangular boundaries Global optimization Rectangling Disparity consistency 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 61602402), Zhejiang Province Public Welfare Technology Application Research (Grant No. LGG19F020001), and the Royal Society (Grant No. \(\hbox {IES}{\setminus }\hbox {R}1{\setminus }180126\)).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

References

  1. 1.
    Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)Google Scholar
  2. 2.
    Chang, C., Liang, C., Chuang, Y.: Content-aware display adaptation and interactive editing for stereoscopic images. IEEE Trans. Multimed. 13(4), 589–601 (2011)Google Scholar
  3. 3.
    Chang, C., Sato, Y., Chuang, Y.: Shape-preserving half-projective warps for image stitching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3254–3261 (2014)Google Scholar
  4. 4.
    Chen, Y., Chuang, Y.: Natural image stitching with the global similarity prior. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 186–201 (2016)Google Scholar
  5. 5.
    Du, S., Hu, S., Martin, R.R.: Changing perspective in stereoscopic images. IEEE Trans. Vis. Comput. Graph. 19(8), 1288–1297 (2013)Google Scholar
  6. 6.
    Du, S., Masiá, B., Hu, S., Gutierrez, D.: A metric of visual comfort for stereoscopic motion. ACM Trans. Graph. 32(6), 222:1–222:9 (2013)Google Scholar
  7. 7.
    Gao, J., Kim, S.J., Brown, M.S.: Constructing image panoramas using dual-homography warping. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 49–56 (2011)Google Scholar
  8. 8.
    von Gioi, R.G., Jakubowicz, J., Morel, J., Randall, G.: LSD: a fast line segment detector with a false detection control. IEEE Trans. Pattern Anal. Mach. Intell. 32(4), 722–732 (2010)Google Scholar
  9. 9.
    Guo, H., Liu, S., He, T., Zhu, S., Zeng, B., Gabbouj, M.: Joint video stitching and stabilization from moving cameras. IEEE Trans. Image Process. 25(11), 5491–5503 (2016)MathSciNetzbMATHGoogle Scholar
  10. 10.
    He, K., Chang, H., Sun, J.: Rectangling panoramic images via warping. ACM Trans. Graph. 32(4), 79:1–79:10 (2013)zbMATHGoogle Scholar
  11. 11.
    Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., Gross, M.H.: Nonlinear disparity mapping for stereoscopic 3D. ACM Trans. Graph. 29(4), 75:1–75:10 (2010)Google Scholar
  12. 12.
    Lee, S., Kim, Y., Lee, J., Kim, K., Lee, K., Noh, J.: Depth manipulation using disparity histogram analysis for stereoscopic 3D. Vis. Comput. 30(4), 455–465 (2014)Google Scholar
  13. 13.
    Lin, C.C., Pankanti, S.U., Ramamurthy, K.N., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1155–1163 (2015)Google Scholar
  14. 14.
    Lin, K., Jiang, N., Cheong, L., Do, M.N., Lu, J.: SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 370–385 (2016)Google Scholar
  15. 15.
    Lin, K., Liu, S., Cheong, L., Zeng, B.: Seamless video stitching from hand-held camera inputs. Comput. Graph. Forum 35(2), 479–487 (2016)Google Scholar
  16. 16.
    Liu, S., Yuan, L., Tan, P., Sun, J.: Bundled camera paths for video stabilization. ACM Trans. Graph. 32(4), 78:1–78:10 (2013)Google Scholar
  17. 17.
    Liu, Y., Sun, L., Yang, S.: A retargeting method for stereoscopic 3D video. Comput. Vis. Med. 1(2), 119–127 (2015)Google Scholar
  18. 18.
    Martínez, F., Rueda, A.J., Feito, F.R.: A new algorithm for computing Boolean operations on polygons. Comput. Geosci. 35(6), 1177–1185 (2009)Google Scholar
  19. 19.
    Mu, T., Sun, J., Martin, R.R., Hu, S.: A response time model for abrupt changes in binocular disparity. Vis. Comput. 31(5), 675–687 (2015)Google Scholar
  20. 20.
    Mu, T., Wang, J., Du, S., Hu, S.: Stereoscopic image completion and depth recovery. Vis. Comput. 30(6–8), 833–843 (2014)Google Scholar
  21. 21.
    Szeliski, R.: Image alignment and stitching: a tutorial. Found. Trends Comput. Graph. Vis. 2(1), 1–104 (2006)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Tang, Y., Tong, R., Tang, M., Zhang, Y.: Depth incorporating with color improves salient object detection. Vis. Comput. 32(1), 111–121 (2016)Google Scholar
  23. 23.
    Tong, R., Zhang, Y., Cheng, K.: Stereopasting: interactive composition in stereoscopic images. IEEE Trans. Vis. Comput. Graph. 19(8), 1375–1385 (2013)Google Scholar
  24. 24.
    Wang, H., Zhou, Y., Wang, X., Fang, L.: A natural shape-preserving stereoscopic image stitching. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1812–1816 (2018)Google Scholar
  25. 25.
    Wang, M., Zhang, X., Liang, J., Zhang, S., Martin, R.R.: Comfort-driven disparity adjustment for stereoscopic video. Comput. Vis. Med. 2(1), 3–17 (2016)Google Scholar
  26. 26.
    Yan, W., Hou, C., Lei, J., Fang, Y., Gu, Z., Ling, N.: Stereoscopic image stitching based on a hybrid warping model. IEEE Trans. Circuits Syst. Video Technol. 27(9), 1934–1946 (2017)Google Scholar
  27. 27.
    Yan, W., Hou, C., Wang, B., Wang, L.: Content-aware disparity adjustment for different stereo displays. Multimed. Tools Appl. 76(8), 10465–10479 (2017)Google Scholar
  28. 28.
    Zaragoza, J., Chin, T., Tran, Q., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1285–1298 (2014)Google Scholar
  29. 29.
    Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3262–3269 (2014)Google Scholar
  30. 30.
    Zhang, F., Liu, F.: Casual stereoscopic panorama stitching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2002–2010 (2015)Google Scholar
  31. 31.
    Zhang, Y., Lai, Y., Zhang, F.: Content-preserving image stitching with regular boundary constraints. ArXiv e-prints (2018)Google Scholar
  32. 32.
    Zhu, Z., Lu, J., Wang, M., Zhang, S., Martin, R.R., Liu, H., Hu, S.: A comparative study of algorithms for realtime panoramic video blending. IEEE Trans. Image Process. 27(6), 2952–2965 (2018)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Institute of Zhejiang Radio and TV TechnologyCommunication University of ZhejiangHangzhouChina
  2. 2.School of Computer Science and InformaticsCardiff UniversityCardiffUK
  3. 3.School of Engineering and Computer ScienceVictoria University of WellingtonWellingtonNew Zealand

Personalised recommendations