Abstract
Neural fields have gained significant attention in the computer vision community due to their excellent performance in novel view synthesis, geometry reconstruction, and generative modeling. Some of their advantages are a sound theoretic foundation and an easy implementation in current deep learning frameworks. While neural fields have been applied to signals on manifolds, e.g., for texture reconstruction, their representation has been limited to extrinsically embedding the shape into Euclidean space. The extrinsic embedding ignores known intrinsic manifold properties and is inflexible wrt. Transfer of the learned function. To overcome these limitations, this work introduces intrinsic neural fields, a novel and versatile representation for neural fields on manifolds. Intrinsic neural fields combine the advantages of neural fields with the spectral properties of the Laplace-Beltrami operator. We show theoretically that intrinsic neural fields inherit many desirable properties of the extrinsic neural field framework but exhibit additional intrinsic qualities, like isometry invariance. In experiments, we show intrinsic neural fields can reconstruct high-fidelity textures from images with state-of-the-art quality and are robust to the discretization of the underlying manifold. We demonstrate the versatility of intrinsic neural fields by tackling various applications: texture transfer between deformed shapes & different shapes, texture reconstruction from real-world images with view dependence, and discretization-agnostic learning on meshes and point clouds.
L. Koestler and D. Grittner—Contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
For reference: a \(80\times 80\) 3-channel color texture image has over 17k pixel values.
References
Aubry, M., Schlickewei, U., Cremers, D.: The wave kernel signature: a quantum mechanical approach to shape analysis. In: IEEE International Conference on Computer Vision (ICCV) (2011)
Baatz, H., Granskog, J., Papas, M., Rousselle, F., Novák, J.: NeRF-Tex: neural reflectance field textures. In: Eurographics Symposium on Rendering (EGSR) (2021)
Basri, R., Galun, M., Geifman, A., Jacobs, D.W., Kasten, Y., Kritchman, S.: Frequency bias in neural networks for input of non-uniform density. In: International Conference on Machine Learning (ICML) (2020)
Benbarka, N., Höfer, T., ul Moqeet Riaz, H., Zell, A.: Seeing implicit neural representations as fourier series. In: IEEE Winter Conference of Applications on Computer Vision (WACV) (2022)
Boscaini, D., Masci, J., Rodolà, E., Bronstein, M.M., Cremers, D.: Anisotropic diffusion descriptors. In: Computer Graphics Forum (CGF), vol. 35 (2016)
Boss, M., Braun, R., Jampani, V., Barron, J.T., Liu, C., Lensch, H.: NeRD: neural reflectance decomposition from image collections. In: IEEE International Conference on Computer Vision (ICCV) (2021)
Boss, M., Jampani, V., Braun, R., Liu, C., Barron, J.T., Lensch, H.P.A.: Neural-PIL: neural pre-integrated lighting for reflectance decomposition. CoRR abs/2110.14373 (2021)
Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Numerical Geometry of Non-Rigid Shapes. Springer, Berlin (2009). https://doi.org/10.1007/978-0-387-73301-2
Burghard, O., Dieckmann, A., Klein, R.: Embedding shapes with green’s functions for global shape matching. Comput. Graph. 68, 1–10 (2017)
Chen, Z., Nobuhara, S., Nishino, K.: Invertible neural BRDF for object inverse rendering. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12350, pp. 767–783. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58558-7_45
Chibane, J., Pons-Moll, G.: Implicit feature networks for texture completion from partial 3D data. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12536, pp. 717–725. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-66096-3_48
Crane, K., Weischedel, C., Wardetzky, M.: The heat method for distance computation. Commun. ACM 60, 90–99 (2017)
Dai, F., Xu, Y.: Approximation Theory and Harmonic Analysis on Spheres and Balls. Springer, Berlin (2013)
Dwivedi, V.P., Bresson, X.: A generalization of transformer networks to graphs. In: AAAI Workshop on Deep Learning on Graphs: Methods and Applications (2021)
Eisenberger, M., Lähner, Z., Cremers, D.: Smooth shells: multi-scale shape registration with functional maps. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Gargan, D., Neelamkavil, F.: Approximating reflectance functions using neural networks. In: Drettakis, G., Max, N. (eds.) EGSR 1998. E, pp. 23–34. Springer, Vienna (1998). https://doi.org/10.1007/978-3-7091-6453-2_3
Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) (1997)
Hertz, A., Perel, O., Giryes, R., Sorkine-Hornung, O., Cohen-Or, D.: Mesh draping: parametrization-free neural mesh transfer. CoRR abs/2110.05433 (2021)
Hertz, A., Perel, O., Giryes, R., Sorkine-Hornung, O., Cohen-Or, D.: SAPE: spatially-adaptive progressive encoding for neural optimization. In: Conference on Neural Information Processing Systems (NeurIPS) (2021)
Jacot, A., Hongler, C., Gabriel, F.: Neural tangent kernel: convergence and generalization in neural networks. In: Conference on Neural Information Processing Systems (NeurIPS) (2018)
Kovnatsky, A., Bronstein, M.M., Bronstein, A.M., Glashoff, K., Kimmel, R.: Coupled quasi-harmonic bases. In: Computer Graphics Forum, vol. 32 (2013)
Lee, J., Jin, K.H.: Local texture estimator for implicit representation function. CoRR abs/2111.08918 (2021)
Liu, X., Donate, A., Jemison, M., Mio, W.: Kernel functions for robust 3D surface registration with spectral embeddings. In: International Conference on Pattern Recognition (ICPR) (2008)
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graph. (TOG) 34, 1–16 (2015)
Marin, R., Cosmo, L., Melzi, S., Rampini, A., Rodolá, E.: Spectral geometry in practice. 3DV Tutorial (2021)
Mehta, I., Gharbi, M., Barnes, C., Shechtman, E., Ramamoorthi, R., Chandraker, M.: Modulated periodic activations for generalizable local functional representations. CoRR abs/2104.03960 (2021)
Meronen, L., Trapp, M., Solin, A.: Periodic activation functions induce stationarity. CoRR abs/2110.13572 (2021)
Michel, O., Bar-On, R., Liu, R., Benaim, S., Hanocka, R.: Text2Mesh: text-driven neural stylization for meshes. CoRR abs/2112.03221 (2021)
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24
Morreale, L., Aigerman, N., Kim, V.G., Mitra, N.J.: Neural surface maps. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Novak, R., et al.: Neural tangents: fast and easy infinite neural networks in Python. In: International Conference on Learning Representations (ICLR) (2020)
Oechsle, M., Mescheder, L.M., Niemeyer, M., Strauss, T., Geiger, A.: Texture fields: learning texture representations in function space. In: IEEE International Conference on Computer Vision (ICCV) (2019)
Ovsjanikov, M., Ben-Chen, M., Solomon, J., Butscher, A., Guibas, L.: Functional maps: a flexible representation of maps between shapes. ACM Trans. Graph. (TOG) 31, 1–11 (2012)
Palafox, P., Bozic, A., Thies, J., Nießner, M., Dai, A.: Neural parametric models for 3D deformable shapes. In: IEEE International Conference on Computer Vision (ICCV) (2021)
Parlett, B.N.: The Symmetric Eigenvalue Problem. SIAM (1998)
Peng, L.W., Shamsuddin, S.M.H.: 3D object reconstruction and representation using neural networks. In: International Conference on Computer Graphics and Interactive Techniques in Australia and Southeast Asia (GRAPHITE) (2004)
Piperakis, E., Kumazawa, I.: Affine transformations of 3D objects represented with neural networks. In: IEEE International Conference on 3-D Digital Imaging and Modeling (2001)
Rahaman, N., et al.: On the spectral bias of neural networks. In: International Conference on Machine Learning (ICML) (2019)
Rahimi, A., Recht, B.: Random features for large-scale kernel machines. In: Conference on Neural Information Processing Systems (NeurIPS) (2007)
Ramasinghe, S., Lucey, S.: Beyond periodicity: towards a unifying framework for activations in coordinate-MLPs. CoRR abs/2111.15135 (2021)
Ramasinghe, S., Lucey, S.: Learning positional embeddings for coordinate-MLPs. CoRR abs/2112.11577 (2021)
Rustamov, R.M.: Laplace-Beltrami eigenfunctions for deformation invariant shape representation. In: Symposium on Geometry Processing (SGP) (2007)
Salti, S., Tombari, F., Di Stefano, L.: SHOT: unique signatures of histograms for surface and texture description. Comput. Vis. Image Underst. 125, 251–264 (2014)
Sharp, N., Attaiki, S., Crane, K., Ovsjanikov, M.: DiffusionNet: discretization agnostic learning on surfaces. ACM Trans. Graph. (TOG) 41(3), 1–16 (2022)
Sharp, N., Crane, K.: A Laplacian for nonmanifold triangle meshes. In: Computer Graphics Forum, vol. 39 (2020)
Sharp, N., Soliman, Y., Crane, K.: The vector heat method. ACM Trans. Graph. (TOG) 38, 1–19 (2019)
Singh, A., Sha, J., Narayan, K.S., Achim, T., Abbeel, P.: BigBIRD: a large-scale 3D database of object instances. In: IEEE International Conference on Robotics and Automation (ICRA) (2014)
Sitzmann, V., Martel, J.N.P., Bergman, A.W., Lindell, D.B., Wetzstein, G.: Implicit neural representations with periodic activation functions. In: Conference on Neural Information Processing Systems (NeurIPS) (2020)
Sorkine, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Symposium on Geometry Processing (SGP) (2007)
Sun, J., Ovsjanikov, M., Guibas, L.: A concise and provably informative multi-scale signature based on heat diffusion. In: Symposium on Geometry Processing (SGP) (2009)
Tancik, M., et al.: Fourier features let networks learn high frequency functions in low dimensional domains. In: Conference on Neural Information Processing Systems (NeurIPS) (2020)
Tewari, A., et al.: Advances in neural rendering. CoRR abs/2111.05849 (2021)
Vaxman, A., Ben-Chen, M., Gotsman, C.: A multi-resolution approach to heat kernels on discrete surfaces. ACM Trans. Graph. (TOG) 29, 1–10 (2010)
Vestner, M., et al.: Efficient deformable shape correspondence via kernel matching. In: International Conference on 3D Vision (3DV) (2017)
Wang, P., Liu, Y., Yang, Y., Tong, X.: Spline positional encoding for learning 3D implicit signed distance fields. In: International Joint Conference on Artificial Intelligence (IJCAI) (2021)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004)
Wimbauer, F., Wu, S., Rupprecht, C.: De-rendering 3D objects in the wild. CoRR abs/2201.02279 (2022)
Xiang, F., Xu, Z., Hasan, M., Hold-Geoffroy, Y., Sunkavalli, K., Su, H.: NeuTex: neural texture mapping for volumetric neural rendering. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Xie, Y., et al.: Neural fields in visual computing and beyond (2021)
Yifan, W., Rahmann, L., Sorkine-hornung, O.: Geometry-consistent neural shape representation with implicit displacement fields. In: International Conference on Learning Representations (ICLR) (2022)
Yüce, G., Ortiz-Jiménez, G., Besbinar, B., Frossard, P.: A structured dictionary perspective on implicit neural representations. CoRR abs/2112.01917 (2021)
Zhang, K., Luan, F., Wang, Q., Bala, K., Snavely, N.: PhySG: inverse rendering with spherical gaussians for physics-based material editing and relighting. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Zheng, J., Ramasinghe, S., Lucey, S.: Rethinking positional encoding. CoRR abs/2107.02561 (2021)
Acknowledgements
We thank Florian Hofherr, Simon Klenk, Dominik Muhle and Emanuele Rodolà for useful discussions and proofreading. ZL is funded by a KI-Starter grant from the Ministerium für Kultur und Wissenschaft NRW.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Koestler, L., Grittner, D., Moeller, M., Cremers, D., Lähner, Z. (2022). Intrinsic Neural Fields: Learning Functions on Manifolds. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13662. Springer, Cham. https://doi.org/10.1007/978-3-031-20086-1_36
Download citation
DOI: https://doi.org/10.1007/978-3-031-20086-1_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20085-4
Online ISBN: 978-3-031-20086-1
eBook Packages: Computer ScienceComputer Science (R0)