Abstract
Self-Supervised Learning (SSL) is a viable technique to unleash the scalability and generalization of the network. Nevertheless, the representations learned by existing 3D SSL are still insufficient for point cloud shape analysis. Compared to point clouds, images have better strength in providing fine-grained semantic information. In this paper, we propose a framework of cross-modal SSL, enabling knowledge transfer across modalities. Specifically, two self-supervised tasks are meticulously designed. The former constructs instance-level consistency, which computes the similarity between objects to discriminate instances of different modalities. The latter constructs cluster-level consistency, which mines category coherence by grouping point clouds and corresponding images into the same semantic regions. By jointly learning feature representations and cluster assignments, the model incorporates the latent category information to reduce the intra-cluster variance as well as increase the inter-cluster variance. Extensive experimental results demonstrate the effectiveness on three representative datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Afham, M., Dissanayake, I., Dissanayake, D., Dharmasiri, A., Thilakarathna, K., Rodrigo, R.: Crosspoint: self-supervised cross-modal contrastive learning for 3D point cloud understanding. In: CVPR, pp. 9902–9912 (2022)
Angelina, M., Pham, Q.H., Hua, B.S., Nguyen, T., Yeung, S.K.: Revisiting point cloud classification: a new benchmark dataset and classification model on real-world data. In: ICCV (2019)
Chang, A.X., et al.: Shapenet: an information-rich 3D model repository. arXiv preprint arXiv:1512.03012 (2015)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML, pp. 1597–1607 (2020)
Choy, C.B., Gwak, J., Savarese, S.: 4D spatio-temporal convnets: minkowski convolutional neural networks. In: CVPR (2019)
Du, B.a., Gao, X., Hu, W., Li, X.: Self-contrastive learning with hard negative sampling for self-supervised point cloud learning. In: ACMMM (2021)
Engel, N., Belagiannis, V., Dietmayer, K.: Point transformer. In: ICCV (2021)
Grill, J.B., et al.: Bootstrap your own latent: a new approach to self-supervised learning. In: NIPS, pp. 21271–21284 (2020)
Hassani, K., Haley, M.: Unsupervised multi-task feature learning on point clouds. In: ICCV (2019)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: CVPR, pp. 9729–9738 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
Hu, W., Miyato, T., Tokui, S., Matsumoto, E., Sugiyama, M.: Learning discrete representations via information maximizing self-augmented training. In: International Conference on Machine Learning, pp. 1558–1567. PMLR (2017)
Huang, S., Xie, Y., Zhu, S.C., Zhu, Y.: Spatio-temporal self-supervised representation learning for 3D point clouds. In: ICCV (2021)
Poursaeed, O., Jiang, T., Qiao, Q., Xu, N., Kim, V.G.: Self-supervised learning of point clouds via orientation estimation. In: 3DV (2020)
Qi, C., Su, H., Mo, K., Guibas, L.J.: Pointnet: deep learning on point sets for 3D classification and segmentation. In: CVPR (2017)
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. IJCV 115(3), 211–252 (2015)
Sauder, J., Sievers, B.: Self-supervised deep learning on point clouds by reconstructing space. In: NIPS (2019)
Sharma, C., Kaul, M.: Self-supervised few-shot learning on point clouds. In: NIPS (2020)
Wang, H., Liu, Q., Yue, X., Lasenby, J., Kusner, M.J.: Unsupervised point cloud pre-training via occlusion completion. In: ICCV (2021)
Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. TOG 38(5), 1–12 (2018)
Wu, Z., et al.: 3D shapenets: a deep representation for volumetric shapes. In: CVPR (2015)
Xie, S., Gu, J., Guo, D., Qi, C.R., Guibas, L., Litany, O.: PointContrast: unsupervised pre-training for 3D point cloud understanding. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12348, pp. 574–591. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58580-8_34
Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Deep implicit surface network for high-quality single-view 3D reconstruction. In: NIPS (2019)
Yi, L., et al.: A scalable active framework for region annotation in 3D shape collections. TOG 35(7), 1–12 (2016)
Zhao, Y., Birdal, T., Deng, H., Tombari, F.: 3D point capsule networks. In: CVPR (2019)
Acknowledgments
This work is supported by the National Key Research and Development Program of China No. 2020AAA0108301; National Natural Science Foundation of China under Grant No. 62176224; CCF-Lenovo Blue Ocean Research Fund.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, Z., Wu, Y., Qu, Y. (2024). Image Priors Assisted Pre-training for Point Cloud Shape Analysis. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14425. Springer, Singapore. https://doi.org/10.1007/978-981-99-8429-9_11
Download citation
DOI: https://doi.org/10.1007/978-981-99-8429-9_11
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8428-2
Online ISBN: 978-981-99-8429-9
eBook Packages: Computer ScienceComputer Science (R0)