Skip to main content
SpringerLink
Log in

Collaborative neural radiance fields for novel view synthesis

  • Research
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Neural radiance fields (NeRF) synthesize realistic novel views by estimating point attributes (density and color), followed by the volume rendering method. However, accurately predicting the arbitrary point attributes poses a challenge for the single NeRF-based model. Such limitation directly impacts the quality of novel view synthesis. To address this problem, a collaborative strategy with multiple NeRF-based models is proposed. This strategy is the first to introduce a multi-model cascaded architecture into NeRF for achieving high-quality novel view synthesis. Its purpose is to utilize a cascading architecture in space for the progressive enhancement of point attribute accuracy. The cascading architecture includes point adjustment and snapshots fusion. Specifically, point adjustment leverages a pretrained NeRF-based model to predict the initial density and color of each point in space. This step affords an initial rendering of target scene. Then, these initial density and color of points are directly transferred to the subsequent NeRF-based model. This process guides the subsequent NeRF-based model to focus on the refinement of initial point attributes and synthesize more realistic novel views. Finally, snapshots fusion fuses outputs (referred as snapshots) from multiple parallel subsequent NeRF-based models to synthesize the ultimate high-quality novel views. The proposed strategy is tested with a range of established NeRF-based methods, such as NeRF, Instant-NGP, and TensoRF. Experimental data for this research are sourced from the realistic 360 synthetic dataset and the LLFF dataset. Results indicate that the proposed collaborative strategy with established NeRF-based methods can improve the quality of novel view synthesis, surpassing the corresponding single model. Our project page is available at https://github.com/ZhenyangLiu/Collaborative-Neural-Radiance-Fields-for-Novel-View-Synthesis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

Datasets that support the findings of this study are available from publicly accessible websites (the realistic 360 synthetic dataset: https://drive.google.com/drive/folders/1JDdLGDruGNXWnM1eqY1FNL9PlStjaKWi, LLFF: https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7.)

References

  1. Sheng, B., Li, P., Fu, H., et al.: Efficient non-incremental constructive solid geometry evaluation for triangular meshes. Gr. Models 97, 1–16 (2018)

    Article  MathSciNet  Google Scholar 

  2. Ertugrul, E., Zhang, H., Zhu, F., et al.: Embedding 3D models in offline physical environments. Comput. Animat. Virtual Worlds 31(4–5), e1959 (2020)

    Article  Google Scholar 

  3. Qin, Y., Chi, X., Sheng, B., et al.: GuideRender: large-scale scene navigation based on multi-modal view frustum movement prediction. Vis. Comput. 1, 1–11 (2023)

    Google Scholar 

  4. Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.-G.: Pixel2mesh: Generating 3d mesh models from single rgb images. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 52–67 (2018)

  5. Ji, M., Gall, J., Zheng, H., Liu, Y., Fang, L.: Surfacenet: An end-to-end 3d neural network for multiview stereopsis. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2307–2315 (2017)

  6. Kutulakos, K.N., Seitz, S.M.: A theory of shape by space carving. Int. J. Comput. Vis. 38(3), 199–218 (2000)

    Article  Google Scholar 

  7. Miao, H., Lu, F., Liu, Z., Zhang, L., Manocha, D., Zhou, B.: Robust 2d/3d vehicle parsing in arbitrary camera views for CVIs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15631–15640 (2021)

  8. Ge, L., Liang, H., Yuan, J., et al.: Real-time 3D hand pose estimation with 3D convolutional neural networks. IEEE Trans. Pattern Anal. Mach. Intell. 41(4), 956–970 (2018)

    Article  Google Scholar 

  9. Sheng, B., Liu, B., Li, P., et al.: Accelerated robust Boolean operations based on hybrid representations. Comput. Aided Geomet. Des. 62, 133–153 (2018)

    Article  MathSciNet  Google Scholar 

  10. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99–106 (2021)

    Article  Google Scholar 

  11. Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: a multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855–5864 (2021)

  12. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470–5479 (2022)

  13. Martin-Brualla, R., Radwan, N., Sajjadi, M.S., Barron, J.T., Dosovitskiy, A., Duckworth, D.: Nerf in the wild: Neural radiance fields for unconstrained photo collections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7210–7219 (2021)

  14. Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields from one or few images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578–4587 (2021)

  15. Wang, Q., Wang, Z., Genova, K., Srinivasan, P.P., Zhou, H., Barron, J.T., Martin-Brualla, R., Snavely, N., Funkhouser, T.: Ibrnet: Learning multi-view image-based rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690–4699 (2021)

  16. Chen, A., Xu, Z., Zhao, F., Zhang, X., Xiang, F., Yu, J., Su, H.: Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14124–14133 (2021)

  17. Garbin, S.J., Kowalski, M., Johnson, M., Shotton, J., Valentin, J.: Fastnerf: High-fidelity neural rendering at 200fps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14346–14355 (2021)

  18. Xu, Q., Xu, Z., Philip, J., Bi, S., Shu, Z., Sunkavalli, K., Neumann, U.: Point-nerf: Point-based neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5438–5448 (2022)

  19. Müller, T., Evans, A., Schied, C., et al.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Gr. (ToG) 41(4), 1–15 (2022)

    Article  Google Scholar 

  20. Kazemy, A., Lam, J., Zhang, X.M.: Event-triggered output feedback synchronization of master-slave neural networks under deception attacks[J]. IEEE Trans. Neural Networks Learn. Syst. 33(3), 952–961 (2020)

    Article  MathSciNet  Google Scholar 

  21. Jiang, J., Sheng, B., Li, P., et al.: Real-time hair simulation with heptadiagonal decomposition on mass spring system. Gr. Models 111, 101077 (2020)

    Article  Google Scholar 

  22. Sheng, B., Li, P., Zhang, Y., et al.: GreenSea: visual soccer analysis using broad learning system. IEEE Trans. Cybern. 51(3), 1463–1477 (2020)

    Article  Google Scholar 

  23. Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin-Brualla, R.: Nerfies: Deformable neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865–5874 (2021)

  24. Park, K., Sinha, U., Hedman, P., Barron, J.T., Bouaziz, S., Goldman, D.B., Martin-Brualla, R., Seitz, S. M.: Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228 (2021)

  25. Pumarola, A., Corona, E., Pons-Moll, G., Moreno- Noguer, F.: D-nerf: Neural radiance fields for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318–10327 (2021)

  26. Reiser, C., Peng, S., Liao, Y., Geiger, A.: Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14335–14345 (2021)

  27. Schwarz, K., Liao, Y., Niemeyer, M., Geiger, A.: Graf: generative radiance fields for 3d-aware image synthesis. Adv. Neural. Inf. Process. Syst. 33, 20154–20166 (2020)

    Google Scholar 

  28. Zhang, K., Riegler, G., Snavely, N., Koltun, V.: Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 (2020)

  29. Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P.P., Barron, J.T., Kretzschmar, H.: Block-nerf: Scalable large scene neural view synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8248–8258 (2022)

  30. Wang, C., Wu, X., Guo, Y.-C., Zhang, S.-H., Tai, Y.-W., Hu, S.-M.: Nerf-sr: High-quality neural radiance fields using super-sampling. arXiv preprint arXiv:2112.01759 (2021)

  31. Ding, J., Bao, D., Wang, Q., He, X., Bai, H., Li, S.: A novel multi-dictionary method with global sensing matrix design for compressed sensing. Signal Process. 152, 69–78 (2018)

    Article  Google Scholar 

  32. Xiong, D., Gui, Q., Hou, W., Ding, M.: Gradient boosting for single image super-resolution. Inf. Sci. 454, 328–343 (2018)

  33. Morales, A., Alomar, A., Porras, A.R., et al.: Babynet: reconstructing 3d faces of babies from uncalibrated photographs. Pattern Recogn. 139, 109367 (2023)

    Article  Google Scholar 

  34. Luvizon, D.C., Tabia, H., Picard, D.: SSP-Net: scalable sequential pyramid networks for real-Time 3D human pose regression. Pattern Recogn. 142, 109714 (2023)

    Article  Google Scholar 

  35. Yang, G.W., Liu, Z.N., Li, D.Y., et al.: JNeRF: an efficient heterogeneous NeRF model zoo based on Jittor. Comput. Vis. Med. 9(2), 401–404 (2023)

    Article  Google Scholar 

  36. Yang, G.W., Zhou, W.Y., Peng, H.Y., et al.: Recursive-NeRF: An efficient and dynamically growing NeRF. IEEE Trans. Vis. Comput. Gr. (2022)

  37. Chen, A., Xu, Z., Geiger, A., et al. Tensorf: Tensorial radiance fields. In: European Conference on Computer Vision. Springer, Cham, pp. 333–350 (2022)

  38. Taherkhani, A., Belatreche, A., Li, Y., et al.: A supervised learning algorithm for learning precise timing of multiple spikes in multilayer spiking neural networks. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5394–5407 (2018)

    Article  MathSciNet  Google Scholar 

  39. Singha, T., Pham, D.S., Krishna, A.: A real-time semantic segmentation model using iteratively shared features in multiple sub-encoders. Pattern Recogn. 140, 109557 (2023)

    Article  Google Scholar 

  40. Xie, Z., Yang, X., Yang, Y., et al.: S3im: stochastic structural similarity and its unreasonable effectiveness for neural fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18024–18034 (2023)

  41. Avidan, S., Shashua, A.: Novel view synthesis in tensor space. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1034–1040. IEEE (1997)

Download references

Acknowledgements

The authors acknowledge the financial supported by the Natural Science Foundation of China (Grant No. 62206082), National Undergraduate Training Program for Innovation and Entrepreneurship (Grant No. 202310336014), Zhejiang Provincial Natural Science Foundation of China (Grant No. LY22F020028) and the National Natural Science Foundation of China (Grant No. U21B2040).

Author information

Authors and Affiliations

  1. College of Science, Zhejiang University of Technology, Hangzhou, 310023, China

    Junqing Yuan, Zhenyang Liu & Tongxuan Han

  2. School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, 310018, China

    Mengting Fan, Zhenzhong Kuang, Chihao Pan & Jiajun Ding

  3. School of Computer Science, Fudan University, Shanghai, 200433, China

    Zhenyang Liu

  4. Zhejiang Zhongcai Pipes Science And Technology Co., Ltd., Xinchang, 312500, China

    Jiajun Ding

Authors
  1. Junqing Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mengting Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenyang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tongxuan Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhenzhong Kuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chihao Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jiajun Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenyang Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yuan, J., Fan, M., Liu, Z. et al. Collaborative neural radiance fields for novel view synthesis. Vis Comput (2024). https://doi.org/10.1007/s00371-024-03379-2

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-024-03379-2

Keywords

Search

Navigation