Skip to main content
SpringerLink
Log in

Multiview Regenerative Morphing with Dual Flows

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13676))

Included in the following conference series:

  • 2089 Accesses

Abstract

This paper aims to address a new task of image morphing under a multiview setting, which takes two sets of multiview images as the input and generates intermediate renderings that not only exhibit smooth transitions between the two input sets but also ensure visual consistency across different views at any transition state. To achieve this goal, we propose a novel approach called Multiview Regenerative Morphing that formulates the morphing process as an optimization to solve for rigid transformation and optimal-transport interpolation. Given the multiview input images of the source and target scenes, we first learn a volumetric representation that models the geometry and appearance for each scene to enable the rendering of novel views. Then, the morphing between the two scenes is obtained by solving optimal transport between the two volumetric representations in Wasserstein metrics. Our approach does not rely on user-specified correspondences or 2D/3D input meshes, and we do not assume any predefined categories of the source and target scenes. The proposed view-consistent interpolation scheme directly works on multiview images to yield a novel and visually plausible effect of multiview free-form morphing. Code: https://github.com/jimtsai23/MorphFlow

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abdal, R., Qin, Y., Wonka, P.: Image2stylegan: how to embed images into the stylegan latent space? In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27–November 2, 2019, pp. 4431–4440. IEEE (2019)

    Google Scholar 

  2. Alexa, M., Cohen-Or, D., Levin, D.: As-rigid-as-possible shape interpolation. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 157–164 (2000)

    Google Scholar 

  3. Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: a multiscale representation for anti-aliasing neural radiance fields. In: ICCV (2021)

    Google Scholar 

  4. Benamou, J.D., Carlier, G., Cuturi, M., Nenna, L., Peyré, G.: Iterative bregman projections for regularized transportation problems. SIAM J. Sci. Comput. 37(2), A1111–A1138 (2015)

    Article  MathSciNet  Google Scholar 

  5. Bonneel, N., Peyré, G., Cuturi, M.: Wasserstein barycentric coordinates: histogram regression using optimal transport. ACM Trans. Graph. 35(4), 1–71 (2016)

    Article  Google Scholar 

  6. Bonneel, N., Van De Panne, M., Paris, S., Heidrich, W.: Displacement interpolation using lagrangian mass transport. In: Proceedings of the 2011 SIGGRAPH Asia Conference, pp. 1–12 (2011)

    Google Scholar 

  7. Chen, A., et al.: Mvsnerf: fast generalizable radiance field reconstruction from multi-view stereo. In: ICCV (2021)

    Google Scholar 

  8. Cheng, S., Bronstein, M.M., Zhou, Y., Kotsia, I., Pantic, M., Zafeiriou, S.: Meshgan: non-linear 3d morphable models of faces. arxiv CS.CV 1903.10384 (2019)

    Google Scholar 

  9. Cosmo, L., Norelli, A., Halimi, O., Kimmel, R., Rodolà, E.: LIMP: learning latent shape representations with metric preservation priors. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12348, pp. 19–35. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58580-8_2

    Chapter  Google Scholar 

  10. Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B., Sen, P.: Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. (TOG) 31(4), 1–10 (2012)

    Article  Google Scholar 

  11. Deng, K., Liu, A., Zhu, J., Ramanan, D.: Depth-supervised nerf: Fewer views and faster training for free. arxiv CS.CV 2107.02791 (2021)

    Google Scholar 

  12. Eisenberger, M., Cremers, D.: Hamiltonian dynamics for real-world shape interpolation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 179–196. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_11

    Chapter  Google Scholar 

  13. Eisenberger, M., Lähner, Z., Cremers, D.: Divergence-free shape correspondence by deformation. Comput. Graph. Forum 38(5), 1–12 (2019)

    Article  Google Scholar 

  14. Eisenberger, M., et al.: Neuromorph: unsupervised shape interpolation and correspondence in one go. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7473–7483 (2021)

    Google Scholar 

  15. De Bonet, J.S., Viola, P.: Poxels: probabilistic voxelized volume reconstruction. In: ICCV (1999)

    Google Scholar 

  16. Liao, J., Yao, Y., Yuan, L., Hua, G., Kang, S.B.: Visual attribute transfer through deep image analogy. arXiv:1705.01088 (2017)

  17. Szeliski, R., Golland, P.: Stereo Matching with Transparency and Matting. Int. J. Comput. Vis. 32, 45–61 (1999). https://doi.org/10.1023/A:1008192912624

  18. Feydy, J., Séjourné, T., Vialard, F.X., Amari, S.I., Trouvé, A., Peyré, G.: Interpolating between optimal transport and mmd using sinkhorn divergences. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2681–2690. PMLR (2019)

    Google Scholar 

  19. Fish, N., Zhang, R., Perry, L., Cohen-Or, D., Shechtman, E., Barnes, C.: Image morphing with perceptual constraints and STN alignment. Comput. Graph. Forum 39(6), 303–313 (2020)

    Article  Google Scholar 

  20. Gao, C., Saraf, A., Kopf, J., Huang, J.: Dynamic view synthesis from dynamic monocular video. In: ICCV (2021)

    Google Scholar 

  21. Garbin, S.J., Kowalski, M., Johnson, M., Shotton, J., Valentin, J.P.C.: Fastnerf: high-fidelity neural rendering at 200fps. arxiv CS.CV 2103.10380 (2021)

    Google Scholar 

  22. Genevay, A., Peyré, G., Cuturi, M.: Learning generative models with sinkhorn divergences. In: International Conference on Artificial Intelligence and Statistics, pp. 1608–1617. PMLR (2018)

    Google Scholar 

  23. Hedman, P., Srinivasan, P.P., Mildenhall, B., Barron, J.T., Debevec, P.E.: Baking neural radiance fields for real-time view synthesis. In: ICCV (2021)

    Google Scholar 

  24. Heeren, B., Rumpf, M., Schröder, P., Wardetzky, M., Wirth, B.: Splines in the space of shells. Comput. Graph. Forum 35(5), 111–120 (2016)

    Article  Google Scholar 

  25. Heeren, B., Rumpf, M., Wardetzky, M., Wirth, B.: Time-discrete geodesics in the space of shells. Comput. Graph. Forum 31(5), 1755–1764 (2012)

    Article  Google Scholar 

  26. Janati, H., Cuturi, M., Gramfort, A.: Debiased sinkhorn barycenters. In: International Conference on Machine Learning, pp. 4692–4701. PMLR (2020)

    Google Scholar 

  27. Jeong, Y., Ahn, S., Choy, C., Anandkumar, A., Cho, M., Park, J.: Self-calibrating neural radiance fields. In: ICCV (2021)

    Google Scholar 

  28. Jiang, C.M., Marcus, P.: Hierarchical detail enhancing mesh-based shape generation with 3d generative adversarial network. arxiv CS.CV 1709.07581 (2017)

    Google Scholar 

  29. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 4401–4410. Computer Vision Foundation/IEEE (2019)

    Google Scholar 

  30. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 8107–8116. Computer Vision Foundation/IEEE (2020)

    Google Scholar 

  31. Kilian, M., Mitra, N.J., Pottmann, H.: Geometric modeling in shape space. ACM Trans. Graph. 26(3), 64 (2007)

    Article  Google Scholar 

  32. Knapitsch, A., Park, J., Zhou, Q.Y., Koltun, V.: Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans. Graph. (ToG) 36(4), 1–13 (2017)

    Article  Google Scholar 

  33. Lerios, A., Garfinkle, C.D., Levoy, M.: Feature-based volume metamorphosis. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, pp. 449–456 (1995)

    Google Scholar 

  34. Li, C., Zaheer, M., Zhang, Y., Póczos, B., Salakhutdinov, R.: Point cloud GAN. In: Deep Generative Models for Highly Structured Data, ICLR 2019 Workshop, New Orleans, Louisiana, United States, 6 May 2019, OpenReview.net (2019)

    Google Scholar 

  35. Li, Z., Niklaus, S., Snavely, N., Wang, O.: Neural scene flow fields for space-time view synthesis of dynamic scenes. In: CVPR (2021)

    Google Scholar 

  36. Lin, C., Ma, W., Torralba, A., Lucey, S.: BARF: bundle-adjusting neural radiance fields. In: ICCV (2021)

    Google Scholar 

  37. Liu, L., Gu, J., Lin, K.Z., Chua, T., Theobalt, C.: Neural sparse voxel fields. In: NeurIPS (2020)

    Google Scholar 

  38. Liu, Y., et al.: Neural rays for occlusion-aware image-based rendering. arxiv CS.CV 2107.13421 (2021)

    Google Scholar 

  39. Martin-Brualla, R., Radwan, N., Sajjadi, M.S.M., Barron, J.T., Dosovitskiy, A., Duckworth, D.: Nerf in the wild: neural radiance fields for unconstrained photo collections. In: CVPR (2021)

    Google Scholar 

  40. Meng, Q., et al.: Gnerf: Gan-based neural radiance field without posed camera. In: ICCV (2021)

    Google Scholar 

  41. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)

    Google Scholar 

  42. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: The European Conference on Computer Vision (ECCV) (2020)

    Google Scholar 

  43. Oechsle, M., Peng, S., Geiger, A.: UNISURF: unifying neural implicit surfaces and radiance fields for multi-view reconstruction. arxiv CS.CV 2104.10078 (2021)

    Google Scholar 

  44. Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C.C., Luo, P.: Exploiting deep generative prior for versatile image restoration and manipulation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 262–277. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_16

    Chapter  Google Scholar 

  45. Park, J.J., Florence, P., Straub, J., Newcombe, R.A., Lovegrove, S.: Deepsdf: learning continuous signed distance functions for shape representation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 165–174. Computer Vision Foundation/IEEE (2019)

    Google Scholar 

  46. Schaefer, S., McPhail, T., Warren, J.: Image deformation using moving least squares. In: ACM SIGGRAPH 2006 Papers, pp. 533–540 (2006)

    Google Scholar 

  47. Seitz, S.M., Dyer, C.R.: View morphing. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 21–30 (1996)

    Google Scholar 

  48. Shechtman, E., Rav-Acha, A., Irani, M., Seitz, S.: Regenerative morphing. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 615–622. IEEE (2010)

    Google Scholar 

  49. Shu, D.W., Park, S.W., Kwon, J.: 3d point cloud generative adversarial network based on tree structured graph convolutions. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27–November 2, 2019, pp. 3858–3867. IEEE (2019)

    Google Scholar 

  50. Simon, D., Aberdam, A.: Barycenters of natural images - constrained wasserstein barycenters for image morphing. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 7907–7916. Computer Vision Foundation/IEEE (2020)

    Google Scholar 

  51. Solomon, J., et al.: Convolutional wasserstein distances: efficient optimal transportation on geometric domains. ACM Trans. Graph. (TOG) 34(4), 1–11 (2015)

    Article  Google Scholar 

  52. Sun, C., Sun, M., Chen, H.T.: Direct voxel grid optimization: super-fast convergence for radiance fields reconstruction. arXiv preprint arXiv:2111.11215 (2021)

  53. Wang, Q., et al.: Ibrnet: learning multi-view image-based rendering. In: CVPR (2021)

    Google Scholar 

  54. Wirth, B., Bar, L., Rumpf, M., Sapiro, G.: A continuum mechanical approach to geodesics in shape space. Int. J. Comput. Vis. 93(3), 293–318 (2011)

    Article  MathSciNet  Google Scholar 

  55. Wizadwongsa, S., Phongthawee, P., Yenphraphai, J., Suwajanakorn, S.: Nex: real-time view synthesis with neural basis expansion. In: CVPR (2021)

    Google Scholar 

  56. Wolberg, G.: Image morphing: a survey. Visual Comput. 14(8), 360–372 (1998)

    Article  Google Scholar 

  57. Wu, J., Zhang, C., Xue, T., Freeman, B., Tenenbaum, J.: Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, 5–10 December 2016, Barcelona, Spain, pp. 82–90 (2016)

    Google Scholar 

  58. Wu, Z., Nitzan, Y., Shechtman, E., Lischinski, D.: Stylealign: analysis and applications of aligned stylegan models. arxiv CS.CV 2110.11323 (2021)

    Google Scholar 

  59. Xian, W., Huang, J., Kopf, J., Kim, C.: Space-time neural irradiance fields for free-viewpoint video. In: CVPR (2021)

    Google Scholar 

  60. Yao, Y., et al.: Blendedmvs: a large-scale dataset for generalized multi-view stereo networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1790–1799 (2020)

    Google Scholar 

  61. Yu, A., Li, R., Tancik, M., Li, H., Ng, R., Kanazawa, A.: Plenoctrees for real-time rendering of neural radiance fields. In: ICCV (2021)

    Google Scholar 

  62. Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelNeRF: neural radiance fields from one or few images. https://arxiv.org/abs/2012.02190 (2020)

  63. Zhang, K., Riegler, G., Snavely, N., Koltun, V.: Nerf++: analyzing and improving neural radiance fields. arxiv CS.CV 2010.07492 (2020)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the MOST grants 110-2634-F-007-027 and 111-2221-E-001-011-MY2 of Taiwan. We are grateful to National Center for High-performance Computing for providing computational resources and facilities.

Author information

Authors and Affiliations

  1. National Tsing Hua University, Hsinchu, Taiwan

    Chih-Jung Tsai, Cheng Sun & Hwann-Tzong Chen

  2. ASUS AICS Department, Taipei, Taiwan

    Cheng Sun

  3. Aeolus Robotics, Taipei, Taiwan

    Hwann-Tzong Chen

Authors
  1. Chih-Jung Tsai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cheng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hwann-Tzong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hwann-Tzong Chen .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2406 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tsai, CJ., Sun, C., Chen, HT. (2022). Multiview Regenerative Morphing with Dual Flows. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13676. Springer, Cham. https://doi.org/10.1007/978-3-031-19787-1_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19787-1_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19786-4

  • Online ISBN: 978-3-031-19787-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation