Abstract
Landmarks on human body models are of great significance for applications such as digital anthropometry and clothing design. The diversity of pose and shape of human body models and the semantic gap make landmarking a challenging problem. In this paper, a learning-based method is proposed to locate landmarks on human body models by analyzing the relationship between geometric descriptors and semantic labels of landmarks. A shape alignment algorithm is proposed to align human body models to break symmetric ambiguity. A symmetry-aware descriptor is proposed based on the structure of the human body models, which is robust to both pose and shape variations in human body models. An AdaBoost regression algorithm is adopted to establish the correspondence between several descriptors and semantic labels of the landmarks. Quantitative and qualitative analyses and comparisons show that the proposed method can obtain more accurate landmarks and distinguish symmetrical landmarks semantically. Additionally, a dataset of landmarked human body models is also provided, containing 271 human body models collected from current human body datasets; each model has 17 landmarks labeled manually.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Creusot, C.; Pears, N.; Austin, J. A machine-learning approach to keypoint detection and landmarking on 3D meshes. International Journal of Computer Vision Vol. 102, Nos. 1–3, 146–179, 2013.
Wang, H.; Guo, J.; Yan, D. M.; Quan, W.; Zhang, X. Learning 3D keypoint descriptors for non-rigid shape matching. In: Computer Vision — ECCV 2018. Lecture Notes in Computer Science, Vol. 11212. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 3–20, 2018.
Guo, Y. L.; Bennamoun, M.; Sohel, F.; Lu, M.; Wan, J. W. 3D object recognition in cluttered scenes with local surface features: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 36, No. 11, 2270–2287, 2014.
Yang, Y.; Fu, X. M.; Chai, S. M.; Xiao, S. W.; Liu, L. G. Volume-enhanced compatible remeshing of 3D models. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 10, 2999–3010, 2019.
Jones, P. R. M.; Rioux, M. Three-dimensional surface anthropometry: Applications to the human body. Optics and Lasers in Engineering Vol. 28, No. 2, 89–117, 1997.
Treleaven, P.; Wells, J. 3D body scanning and healthcare applications. Computer Vol. 40, No. 7, 28–34, 2007.
You, Y.; Lou, Y. J.; Li, C. K.; Cheng, Z. J.; Li, L. W.; Ma, L. Z.; Lu, C.; Wang, W. KeypointNet: A large-scale 3D keypoint dataset aggregated from numerous human annotations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13644–13653, 2020.
Allen, B.; Curless, B.; Popović, Z. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Transactions on Graphics Vol. 22, No. 3, 587–594, 2003.
Giachetti, A.; Mazzi, E.; Piscitelli, F.; Aono, M.; Hamza, A. B.; Bonis, T.; Claes, P.; Godil, A.; Li, C.; Ovsjanikov, M.; et al. SHREC’14 track: Automatic location of landmarks used in manual anthropometry. In: Eurographics Workshop on 3D Object Retrieval (2014). Bustos, B.; Tabia, H.; Vandeborre, J. P.; Veltkamp, R. Eds. The Eurographics Association, 2014.
Sung, M.; Su, H.; Yu, R.; Guibas, L. Deep functional dictionaries: Learning consistent semantic structures on 3D models from functions. In: Proceedings of the 32nd Conference on Neural Information Processing Systems, 2018.
Chaouch, M.; Verroust-Blondet, A. Alignment of 3D models. Graphical Models Vol. 71, No. 2, 63–76, 2009.
Haim, N.; Segol, N.; Ben-Hamu, H.; Maron, H.; Lipman, Y. Surface networks via general covers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 632–641, 2019.
Hanocka, R.; Hertz, A.; Fish, N.; Giryes, R.; Fleishman, S.; Cohen-Or, D. MeshCNN: A network with an edge. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 90, 2019.
Wiersma, R.; Eisemann, E.; Hildebrandt, K. CNNs on surfaces using rotation-equivariant features. ACM Transactions on Graphics Vol. 39, No. 4, Article No. 92, 2020.
Johnson, A. E.; Hebert, M. Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 21, No. 5, 433–449, 1999.
Frome, A.; Huber, D.; Kolluri, R.; Bülow, T.; Malik, J. Recognizing objects in range data using regional point descriptors. In: Computer Vision — ECCV 2004. Lecture Notes in Computer Science, Vol. 3023. Pajdla, T.; Matas, J. Eds. Springer Berlin Heidelberg, 224–237, 2004.
Shapira, L.; Shamir, A.; Cohen-Or, D. Consistent mesh partitioning and skeletonisation using the shape diameter function. The Visual Computer Vol. 24, No. 4, 249–259, 2008.
Rustamov, R. M. Laplace—Beltrami eigenfunctions for deformation invariant shape representation. In: Proceedings of the 5th Eurographics Symposium on Geometry Processing, 225–233, 2007.
Sun, J.; Ovsjanikov, M.; Guibas, L. A concise and provably informative multi-scale signature based on heat diffusion. Computer Graphics Forum Vol. 28, No. 5, 1383–1392, 2009.
Bronstein, M. M.; Kokkinos, I. Scale-invariant heat kernel signatures for non-rigid shape recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1704–1711, 2010.
Aubry, M.; Schlickewei, U.; Cremers, D. The wave kernel signature: A quantum mechanical approach to shape analysis. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 1626–1633, 2011.
Meng, W.; Yi, F. Local diffusion map signature for symmetry-aware non-rigid shape correspondence. In: Proceedings of the 24th ACM International Conference on Multimedia, 526–530, 2016.
Ren, J.; Poulenard, A.; Wonka, P.; Ovsjanikov, M. Continuous and orientation-preserving correspondences via functional maps. ACM Transactions on Graphics Vol. 37, No. 6, Article No. 248, 2018.
Wang, Y. Q.; Guo, J. W.; Yan, D. M.; Wang, K.; Zhang, X. P. A robust local spectral descriptor for matching non-rigid shapes with incompatible shape structures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6224–6233, 2019.
Li, Y.; Zhong, Y. Q. Automatic detecting anthropometric landmarks based on spin image. Textile Research Journal Vol. 82, No. 6, 622–632, 2012.
Wuhrer, S.; Azouz, Z. B.; Shu, C. Semi-automatic prediction of landmarks on human models in varying poses. In: Proceedings of the Canadian Conference on Computer and Robot Vision, 136–142, 2010.
Azouz, Z. B.; Shu, C.; Mantel, A. Automatic locating of anthropometric landmarks on 3D human models. In: Proceedings of the International Symposium on 3D Data Processing, Visualization, and Transmission, 750–757, 2006.
Lovato, C.; Castellani, U.; Zancanaro, C.; Giachetti, A. Automatic labelling of anatomical landmarks on 3D body scans. Graphical Models Vol. 76, No. 6, 648–657, 2014.
Shu, Z. Y.; Xin, S. Q.; Xu, X.; Liu, L. G.; Kavan, L. Detecting 3D points of interest using multiple features and stacked auto-encoder. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 8, 2583–2596, 2019.
Xi, P. C.; Shu, C.; Goubran, R. Localizing 3-D anatomical landmarks using deep convolutional neural networks. In: Proceedings of the 14th Conference on Computer and Robot Vision, 197–204, 2017.
Yi, L.; Su, H.; Guo, X. W.; Guibas, L. SyncSpecCNN: Synchronized spectral CNN for 3D shape segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6584–6592, 2017.
Zhou, Z. K.; Hao, S. J. Anatomical landmark detection on 3D human shapes by hierarchically utilizing multiple shape features. Neurocomputing Vol. 253, 162–168, 2017.
Marin, R.; Melzi, S.; Rodolà, E.; Castellani, U. FARM: Functional automatic registration method for 3D human bodies. Computer Graphics Forum Vol. 39, No. 1, 160–173, 2020.
Guo, J. W.; Wang, H. Y.; Cheng, Z. L.; Zhang, X. P.; Yan, D. M. Learning local shape descriptors for computing non-rigid dense correspondence. Computational Visual Media Vol. 6, No. 1, 95–112, 2020.
Luo, S.; Feng, J. Q. Symmetry-aware kinematic skeleton generation of a 3D human body model. Multimedia Tools and Applications Vol. 79, Nos. 29–30, 20579–20602, 2020.
Baran, I.; Popović, J. Automatic rigging and animation of 3D characters. ACM Transactions on Graphics Vol. 26, No. 3, 72–es, 2007.
Anguelov, D.; Srinivasan, P.; Koller, D.; Thrun, S.; Davis, J. SCAPE: Shape completion and animation of people. ACM Transactions on Graphics Vol. 24, No. 3, 408–416, 2005.
Bogo, F.; Romero, J.; Loper, M.; Black, M. J. FAUST: Dataset and evaluation for 3D mesh registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3794–3801, 2014.
Yang, Y. P.; Yu, Y.; Zhou, Y.; Du, S. D.; Davis, J.; Yang, R. G. Semantic parametric reshaping of human body models. In: Proceedings of the 2nd International Conference on 3D Vision, 41–48, 2014.
Maron, H.; Galun, M.; Aigerman, N.; Trope, M.; Dym, N.; Yumer, E.; Kim, V. G.; Lipman, Y. Convolutional neural networks on surfaces via seamless toric covers. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 71, 2017.
Loper, M.; Mahmood, N.; Romero, J.; Pons-Moll, G.; Black, M. J. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics Vol. 34, No. 6, Article No. 248, 2015.
Chen, X. B.; Golovinskiy, A.; Funkhouser, T. A benchmark for 3D mesh segmentation. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 73, 2009.
Xu, Y. F.; Fan, T. Q.; Xu, M. Y.; Zeng, L.; Qiao, Y. SpiderCNN: Deep learning on point sets with parameterized convolutional filters. In: Computer Vision — ECCV 2018. Lecture Notes in Computer Science, Vol. 11212. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 90–105, 2018.
Wang, Y.; Sun, Y. B.; Liu, Z. W.; Sarma, S. E.; Bronstein, M. M.; Solomon, J. M. Dynamic graph CNN for learning on point clouds. ACM Transactions on Graphics Vol. 38, No. 5, Article No. 146, 2019.
Wu, W. X.; Qi, Z. A.; Li, F. X. PointConv: Deep convolutional networks on 3D point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9613–9622, 2019.
Acknowledgements
This work was jointly supported by the National Natural Science Foundation of China under Grants Nos. 61732015, 61932018, and 61472349.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Shan Luo received her B.S. degree in computer science from Jilin University, Changchun, China, in 2015. She is now a Ph.D. candidate in the State Key Lab of CAD&CG, Zhejiang University, Hangzhou, China. Her research interests include human body shape analysis and synthesis.
Qitong Zhang received her B.S. degree in digital media technology from Shandong University, Jinan, China, in 2017. She is now a Ph.D. candidate in the State Key Laboratory of CAD&CG, Zhejiang University. Her fields of interest are multi-view stereo and 3D reconstruction.
Jieqing Feng is a professor in the State Key Laboratory of CAD&CG, Zhejiang University. He received his B.Sc. degree in applied mathematics from the National University of Defense Technology in 1992 and his Ph.D. degree in computer graphics from Zhejiang University in 1997. His research interests include geometric modeling, real-time rendering, stereo vision, and modeling and simulation in solar thermal power.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Luo, S., Zhang, Q. & Feng, J. Automatic location and semantic labeling of landmarks on 3D human body models. Comp. Visual Media 8, 553–570 (2022). https://doi.org/10.1007/s41095-021-0254-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-021-0254-4