Advertisement

Image Aligning and Stitching Based on Multilayer Mesh Deformation

  • Mingfu Xie
  • Jun ZhouEmail author
  • Xiao Gu
  • Hua Yang
Conference paper
  • 44 Downloads
Part of the Communications in Computer and Information Science book series (CCIS, volume 1181)

Abstract

This paper aims to solve the problem of strong disparity in image stitching. By studying current mesh deformation methods based on single grid, we propose an image stitching framework based on multilayer mesh deformation for aligning regions in different layer. With development of image depth perception and semantic segmentation technology, we can get layering maps of images or photos expediently. We introduce images representation with layers and get layer corresponding by using depth or disparity information for large parallax scenarios. Registration of each layer is carried out independently. To ensure the integrity of layer synthesis results, we apply deformation with translation and scaling compensation between different layers before blending. The experiment demonstrates that our method can adequately utilize the prior information in layering maps to decouple 2D transformation between different layers, finally achieve outstanding aligning performance in all layers and naturalness in complete stitching result.

Keywords

Image stitching Image layering Mesh deformation Seamless blending 

Notes

Acknowledgement

The paper was supported by Science and Technology Commission of Shanghai Municipality (STCSM) under Grant 18DZ1200102 and NSFC under Grant 61471234, 61771303.

References

  1. 1.
    Anderson, R., et al.: Jump: virtual reality video. ACM Trans. Graph. (TOG) 35(6), 198 (2016)CrossRefGoogle Scholar
  2. 2.
    Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)CrossRefGoogle Scholar
  3. 3.
    He, C., Zhou, J.: Mesh-based image stitching algorithm with linear structure protection (2018)Google Scholar
  4. 4.
    Chang, C.H., Sato, Y., Chuang, Y.Y.: Shape-preserving half-projective warps for image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3254–3261 (2014)Google Scholar
  5. 5.
    Chen, K., Tu, J., Yao, J., Li, J.: Generalized content-preserving warp: direct photometric alignment beyond color consistency. IEEE Access 6, 69835–69849 (2018)CrossRefGoogle Scholar
  6. 6.
    Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
  7. 7.
    Chen, Y.-S., Chuang, Y.-Y.: Natural image stitching with the global similarity prior. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 186–201. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46454-1_12CrossRefGoogle Scholar
  8. 8.
    Furukawa, R., Sagawa, R., Kawasaki, H.: Depth estimation using structured light flow-analysis of projected pattern flow on an object’s surface. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4640–4648 (2017)Google Scholar
  9. 9.
    Gao, J., Kim, S.J., Brown, M.S.: Constructing image panoramas using dual-homography warping. In: CVPR 2011, pp. 49–56. IEEE (2011)Google Scholar
  10. 10.
    Gao, J., Li, Y., Chin, T.J., Brown, M.S.: Seam-driven image stitching. In: Eurographics (Short Papers), pp. 45–48 (2013)Google Scholar
  11. 11.
    Guennebaud, G., Jacob, B., et al.: Eigen v3 (2010). http://eigen.tuxfamily.org
  12. 12.
    Johnson, J.W.: Adapting mask-RCNN for automatic nucleus segmentation. arXiv preprint arXiv:1805.00500 (2018)
  13. 13.
    Li, N., Liao, T., Wang, C.: Perception-based seam cutting for image stitching. SIViP 12(5), 967–974 (2018)CrossRefGoogle Scholar
  14. 14.
    Lin, C.C., Pankanti, S.U., Natesan Ramamurthy, K., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1155–1163 (2015)Google Scholar
  15. 15.
    Lin, K., Jiang, N., Liu, S., Cheong, L.F., Do, M., Lu, J.: Direct photometric alignment by mesh deformation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2405–2413 (2017)Google Scholar
  16. 16.
    Lin, W.Y., Liu, S., Matsushita, Y., Ng, T.T., Cheong, L.F.: Smoothly varying affine stitching. In: CVPR 2011, pp. 345–352. IEEE (2011)Google Scholar
  17. 17.
    Liu, F., Gleicher, M., Jin, H., Agarwala, A.: Content-preserving warps for 3D video stabilization. In: ACM Transactions on Graphics (TOG), vol. 28, p. 44. ACM (2009)Google Scholar
  18. 18.
    Shi, H., Guo, L., Tan, S., Li, G., Sun, J.: Improved parallax image stitching algorithm based on feature block. Symmetry 11(3), 348 (2019)CrossRefGoogle Scholar
  19. 19.
    Shum, H.Y., Szeliski, R.: Construction of panoramic image mosaics with global and local alignment. In: Benosman, R., Kang, S.B. (eds.) Panoramic Vision. MCS, pp. 227–268. Springer, New York (2001).  https://doi.org/10.1007/978-1-4757-3482-9_13CrossRefGoogle Scholar
  20. 20.
    Tu, W.C., et al.: Learning superpixels with segmentation-aware affinity loss. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 568–576 (2018)Google Scholar
  21. 21.
    Wang, C., Miguel Buenaposada, J., Zhu, R., Lucey, S.: Learning depth from monocular videos using direct methods. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2022–2030 (2018)Google Scholar
  22. 22.
    Xu, K., Zhou, J., Wang, Z.: A method of hole-filling for the depth map generated by Kinect with moving objects detection. In: IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, pp. 1–5. IEEE (2012)Google Scholar
  23. 23.
    Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2339–2346 (2013)Google Scholar
  24. 24.
    Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3262–3269 (2014)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  1. 1.Shanghai Key Laboratory of Multi Media Processing and TransmissionsShanghai Jiaotong UniversityShanghaiChina
  2. 2.Institute of Image Communication and Network EngineeringShanghai Jiaotong UniversityShanghaiChina

Personalised recommendations