Abstract
3D hand pose estimation is a crucial subject in the domain of computer vision. Recently researchers transform a single depth image into multiple virtual view depth images. By projecting a single depth image through point cloud transformation and using the depth images of multiple virtual views together for hand pose estimation, these methods can effectively improve the estimation accuracy. However, current methods have issues with distorted generated depth images, insufficient usage of the depth image of each view, and high computational overhead. To overcome these problems, we introduce a multi-virtual view scoring network (MVSN). Our proposed MVSN consists of a single virtual view estimation module, virtual view feature encoding module, and virtual view scoring module. To generate an intermediate feature map suitable for virtual view scoring, the single virtual view estimation module uses a feature map offset loss function and enhance information interaction between channels in the backbone network. The virtual view feature encoding module adopts a two-branch structure to capture information about all joints and single joints from the intermediate feature map, respectively. This structure effectively improves model sensitivity to each view, better integrates information from each virtual view, and obtains a more appropriate scoring feature for each virtual view. The virtual view scoring module scores each view based on the scoring feature, and gives a higher score to the more accurately estimated virtual view. We also propose a dynamic virtual view removal strategy to remove poor quality views in the training process. Our model is tested on the NYU and ICVL datasets, and the mean joint error is 6.21 mm and 4.53 mm, respectively, exhibiting better estimation accuracy than existing methods.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Tompson, J., Stein, M., Lecun, Y., et al.: Re.-time continuous pose recovery of human hands using convolutional networks. ACM Trans. Graph. 33(5), 1–10 (2014)
Tang, D., Chang, H.J., Tejani, A., et al.: Latent regression forest: structured estimation of 3D articulated hand posture. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE (2014)
Sun, X., Wei, Y., Liang, S., et al.: Cascaded hand pose regression. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2015)
Yuan, S., Ye, Q., Garcia-Hernando, G., et al.: The 2017 Hands in the Million Challenge on 3D Hand Pose Estimation. https://doi.org/10.48550/arXiv.1707.02237 (2017)
Xiong, F., Zhang, B., Xiao, Y., et al.: A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Image (2019)
Cheng, J., Wan, Y., Zuo, D., et al.: Efficient virtual view selection for 3d hand pose estimation. Proc. AAAI Conf. Artif. Intell. 36(1), 419–426 (2022). https://doi.org/10.1609/aaai.v36i1.19919
Ren, P., Sun, H., Qi, Q., et al.: SRN: stacked regression network for real-time 3D hand pose estimation. In: The British Machine Vision Conference (2019)
Zhang, Z., Xie, S., Chen, M., et al.: HandAugmentation: a Simple Data Augmentation Method for Depth-Based 3D Hand Pose Estimation, https://doi.org/10.48550/arXiv.2001.00702 (2020)
Ren, P., Sun, H., Huang, W., et al.: Spatial-aware stacked regression network for real-time 3D hand pose estimation – ScienceDirect. Neurocomputing 437, 42–57 (2021)
Du, K., Lin, X., Sun, Y., et al.: Crossinfonet: multi-task information sharing based hand pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9896–9905 (2019)
Xu, L., Hu, C., Xue, J., et al.: Improved regression network on depth hand pose estimation with auxiliary variable. IEEE Trans. Circuits Syst. Video Technol. 31(3), 890–904 (2020)
Huang, W., Ren, P., Wang, J., et al.: AWR: adaptive weighting regression for 3D hand pose estimation. Proc. AAAI Conf. Artif. Intell. 34(7), 11061–11068 (2020)
Wan, C., Probst, T., Van Gool, L., et al.: Dense 3d regression for hand pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5147–5156 (2018)
Fang, L., Liu, X., Liu, L., et al.: Jgr-p2o: Joint graph reasoning based pixel-to-offset prediction network for 3d hand pose estimation from a single depth image. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 Aug 2020, Proceedings, Part VI 16, pp. 120–137. Springer International Publishing (2020)
Rezaei, M., Rastgoo, R., Athitsos, V.: TriHorn-Net: A Model for Accurate Depth-Based 3D Hand Pose Estimation. arXiv preprint arXiv:2206.07117 (2022)
Moon, G., Chang, J.Y., Lee, K.M.: V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map. https://doi.org/10.48550/arXiv.1711.07399
Cheng, W., Park, J.H., Ko, J.H.: HandFoldingNet: A 3D Hand Pose Estimation Network Using Multiscale-Feature Guided Folding of a 2D Hand Skeleton (2021)
Huang, L., Tan, J., Liu, J., et al.: Hand-transformer: non-autoregressive structured modeling for 3d hand pose estimation. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 Aug 2020, Proceedings, Part XXV 16, pp. 17–33. Springer International Publishing (2020)
Ge, L., Cai, Y., Weng, J., et al.: Hand pointnet: 3d hand pose estimation using point sets. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8417–8426 (2018)
Ge, L., Liang, H., Yuan, J., et al.: Robust 3D hand pose estimation in single depth images: from single-view CNN to multi-view CNNs. IEEE 27, 4422–4436 (2016)
Wang, Q., Wu, B., Zhu, P., et al.: ECA-Net: Efficient channel attention for deep convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 11534–11542 (2020)
Oberweger, M., Lepetit, V.: DeepPrior++: improving fast and accurate 3D hand pose estimation. IEEE Comput. Soc. (2017)
Zhou, X., Wan, Q., Zhang, W, et al.: Model-based Deep Hand Pose Estimation. AAAI Press (2016)
Ge, L., Hui, L., Yuan, J., et al.: 3D convolutional neural networks for efficient and robust hand pose estimation from single depth images. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2017)
Wang, G., Chen, X., Guo, H., et al.: Region ensemble network: towards good practices for deep 3D hand pose estimation. J. Vis. Commun. Image Represent. 55, 404–414 (2018)
Chen, X., Wang, G., Guo, H., et al.: Pose guided structured region ensemble network for cascaded hand pose estimation. Neurocomputing 395, 138–149 (2017)
Chen, X., Wang, G., Zhang, C., et al.: SHPR-Net: deep semantic hand pose regression from point clouds. IEEE Access 6, 43425–43439 (2018)
Poier, G., Opitz, M., Schinagl, D., et al.: MURAUER: Mapping Unlabeled Real Data for Label AUstERity (2018)
Ge, L., Ren, Z., Yuan, J.: Point-to-Point Regression PointNet for 3D Hand Pose Estimation. In: European Conference on Computer Vision (2018)
Zhao, F., Lu, H., Zhao, W., et al.: Image-scale-symmetric cooperative network for defocus blur detection. IEEE Trans. Circuits Syst. Video Technol. 32(5), 2719–2731 (2021)
Teng, Y., Lu, H., Li, Y., et al.: Multidimensional deformable object manipulation based on DN-transporter networks. IEEE Trans. Intell. Transp. Syst. 24(4), 4532–4540 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Tian, Y., Li, C., Tian, L. (2024). Multi-virtual View Scoring Network for 3D Hand Pose Estimation from a Single Depth Image. In: Lu, H., Cai, J. (eds) Artificial Intelligence and Robotics. ISAIR 2023. Communications in Computer and Information Science, vol 1998. Springer, Singapore. https://doi.org/10.1007/978-981-99-9109-9_15
Download citation
DOI: https://doi.org/10.1007/978-981-99-9109-9_15
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-9108-2
Online ISBN: 978-981-99-9109-9
eBook Packages: Computer ScienceComputer Science (R0)