Advertisement

BCNet: Learning Body and Cloth Shape from a Single Image

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12365)

Abstract

In this paper, we consider the problem to automatically reconstruct garment and body shapes from a single near-front view RGB image. To this end, we propose a layered garment representation on top of SMPL and novelly make the skinning weight of garment independent of the body mesh, which significantly improves the expression ability of our garment model. Compared with existing methods, our method can support more garment categories and recover more accurate geometry. To train our model, we construct two large scale datasets with ground truth body and garment geometries as well as paired color images. Compared with single mesh or non-parametric representation, our method can achieve more flexible control with separate meshes, makes applications like re-pose, garment transfer, and garment texture mapping possible.

Keywords

Clothed body reconstruction 3D garment shape 3D body shape Skinning weight 

Supplementary material

504476_1_En_2_MOESM1_ESM.pdf (23.2 mb)
Supplementary material 1 (pdf 23793 KB)

Supplementary material 2 (mp4 9115 KB)

References

  1. 1.
    Alldieck, T., Magnor, M., Bhatnagar, B.L., Theobalt, C., Pons-Moll, G.: Learning to reconstruct people in clothing from a single RGB camera. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1175–1186 (2019)Google Scholar
  2. 2.
    Alldieck, T., Magnor, M., Xu, W., Theobalt, C., Pons-Moll, G.: Detailed human avatars from monocular video. In: International Conference on 3D Vision (3DV), pp. 98–109. IEEE (2018)Google Scholar
  3. 3.
    Alldieck, T., Magnor, M., Xu, W., Theobalt, C., Pons-Moll, G.: Video based reconstruction of 3d people models. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8387–8397 (2018)Google Scholar
  4. 4.
    Alldieck, T., Pons-Moll, G., Theobalt, C., Magnor, M.: Tex2shape: detailed full human body geometry from a single image. In: IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  5. 5.
    Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., Davis, J.: Scape: shape completion and animation of people. In: ACM Transactions on Graphics (TOG), vol. 24, pp. 408–416. ACM (2005)Google Scholar
  6. 6.
  7. 7.
    Bhatnagar, B.L., Tiwari, G., Theobalt, C., Pons-Moll, G.: Multi-garment net: learning to dress 3D people from images. In: IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  8. 8.
    Blender (2019). https://www.blender.org/
  9. 9.
    Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M.J.: Keep it SMPL: automatic estimation of 3D human pose and shape from a single image. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 561–578. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46454-1_34CrossRefGoogle Scholar
  10. 10.
    Bouritsas, G., Bokhnyak, S., Ploumpis, S., Bronstein, M., Zafeiriou, S.: Neural 3D morphable models: spiral convolutional networks for 3D shape representation learning and generation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7213–7222 (2019)Google Scholar
  11. 11.
    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3606–3613 (2014)Google Scholar
  12. 12.
    Daněřek, R., Dibra, E., Öztireli, C., Ziegler, R., Gross, M.: Deepgarment: 3D garment shape estimation from a single image. In: Computer Graphics Forum, vol. 36, pp. 269–280. Wiley Online Library (2017)Google Scholar
  13. 13.
    De Aguiar, E., Sigal, L., Treuille, A., Hodgins, J.K.: Stable spaces for real-time clothing. ACM Trans. Graph. (TOG) 29(4), 1–9 (2010)CrossRefGoogle Scholar
  14. 14.
    Dibra, E., Jain, H., Oztireli, C., Ziegler, R., Gross, M.: Human shape from silhouettes using generative HKS descriptors and cross-modal neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4826–4836 (2017)Google Scholar
  15. 15.
  16. 16.
    Gong, K., Liang, X., Li, Y., Chen, Y., Yang, M., Lin, L.: Instance-level human parsing via part grouping network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 770–785 (2018)Google Scholar
  17. 17.
    Guan, P., Reiss, L., Hirshberg, D.A., Weiss, A., Black, M.J.: Drape: dressing any person. ACM Trans. Graph. (TOG) 31(4), 1–35 (2012)CrossRefGoogle Scholar
  18. 18.
    Gundogdu, E., Constantin, V., Seifoddini, A., Dang, M., Salzmann, M., Fua, P.: Garnet: A two-stream network for fast and accurate 3D cloth draping. In: IEEE International Conference on Computer Vision (ICCV), pp. 8739–8748 (2019)Google Scholar
  19. 19.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)Google Scholar
  20. 20.
    Jackson, A.S., Manafas, C., Tzimiropoulos, G.: 3D human body reconstruction from a single image via volumetric regression. In: European Conference on Computer Vision (ECCV), (2018)Google Scholar
  21. 21.
    Jiang, B., Zhang, J., Cai, J., Zheng, J.: Learning 3D human body embedding. arXiv preprint arXiv:1905.05622 (2019)
  22. 22.
    Kampouris, C., Zafeiriou, S., Ghosh, A., Malassiotis, S.: Fine-grained material classification using micro-geometry and reflectance. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 778–792. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46454-1_47CrossRefGoogle Scholar
  23. 23.
    Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7122–7131 (2018)Google Scholar
  24. 24.
    Kanazawa, A., Zhang, J.Y., Felsen, P., Malik, J.: Learning 3D human dynamics from video. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5614–5623 (2019)Google Scholar
  25. 25.
    Kato, H., Ushiku, Y., Harada, T.: Neural 3d mesh renderer. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3907–3916 (2018)Google Scholar
  26. 26.
    Lahner, Z., Cremers, D., Tung, T.: Deepwrinkles: accurate and realistic clothing modeling. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 667–684 (2018)Google Scholar
  27. 27.
    Lassner, C., Romero, J., Kiefel, M., Bogo, F., Black, M.J., Gehler, P.V.: Unite the people: Closing the loop between 3D and 2D human representations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6050–6059 (2017)Google Scholar
  28. 28.
    Liang, J., Lin, M.C.: Shape-aware human pose and shape reconstruction using multi-view images. In: The IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  29. 29.
    Liu, J., Akhtar, N., Mian, A.: Temporally coherent full 3D mesh human pose recovery from monocular video. arXiv preprint arXiv:1906.00161 (2019)
  30. 30.
    Liu, L., Zheng, Y., Tang, D., Yuan, Y., Fan, C., Zhou, K.: Neuroskinning: automatic skin binding for production characters with deep graph networks. ACM Trans. Graph. (TOG) 38(4), 114 (2019)Google Scholar
  31. 31.
    Loper, M., Mahmood, N., Black, M.J.: Mosh: motion and shape capture from sparse markers. ACM Trans. Graph. (TOG) 33(6), 220 (2014)CrossRefGoogle Scholar
  32. 32.
    Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: a skinned multi-person linear model. ACM Trans. Graph. (TOG) 34(6), 248 (2015)CrossRefGoogle Scholar
  33. 33.
  34. 34.
    Mocap, C.: 2019. http://mocap.cs.cmu.edu/
  35. 35.
    Natsume, R., Saito, S., Huang, Z., Chen, W., Ma, C., Li, H., Morishima, S.: Siclope: silhouette-based clothed people. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4480–4490 (2019)Google Scholar
  36. 36.
    Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., Schiele, B.: Neural body fitting: unifying deep learning and model based human pose and shape estimation. In: International Conference on 3D Vision (3DV), pp. 484–494. IEEE (2018)Google Scholar
  37. 37.
    Pavlakos, G., et al.: Expressive body capture: 3D hands, face, and body from a single image. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10975–10985 (2019)Google Scholar
  38. 38.
    Pavlakos, G., Zhu, L., Zhou, X., Daniilidis, K.: Learning to estimate 3D human pose and shape from a single color image. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 459–468 (2018)Google Scholar
  39. 39.
    Pons-Moll, G., Pujades, S., Hu, S., Black, M.J.: Clothcap: seamless 4D clothing capture and retargeting. ACM Trans. Graph. (TOG) 36(4), 1–15 (2017)CrossRefGoogle Scholar
  40. 40.
    Pumarola, A., Sanchez, J., Choi, G., Sanfeliu, A., Moreno-Noguer, F.: 3D people: modeling the geometry of dressed humans. In: IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  41. 41.
    Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: deep learning on point sets for 3D classification and segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 652–660 (2017)Google Scholar
  42. 42.
  43. 43.
    Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: Pifu: pixel-aligned implicit function for high-resolution clothed human digitization. In: IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  44. 44.
    Santesteban, I., Otaduy, M.A., Casas, D.: Learning-based animation of clothing for virtual try-on. In: Computer Graphics Forum, vol. 38, pp. 355–366. Wiley Online Library (2019)Google Scholar
  45. 45.
    Sorkine, O.: Laplacian mesh processing. In: Eurographics - State of the Art Reports, pp. 53–70 (2005)Google Scholar
  46. 46.
    Varol, G., et al.: Bodynet: volumetric inference of 3D human body shapes. In: European Conference on Computer Vision (ECCV), pp. 20–36 (2018)Google Scholar
  47. 47.
    Varol, G., et al.: Learning from synthetic humans. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 109–117 (2017)Google Scholar
  48. 48.
    Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations, ICLR (2018)Google Scholar
  49. 49.
    Wang, T.Y., Ceylan, D., Popovic, J., Mitra, N.J.: Learning a shared shape space for multimodal garment design. ACM Trans. Graph. (TOG) 37(6), 203:1–203:13 (2018)Google Scholar
  50. 50.
    Xu, W., Umentani, N., Chao, Q., Mao, J., Jin, X., Tong, X.: Sensitivity-optimized rigging for example-based real-time clothing synthesis. ACM Trans. Graph. (TOG) 33(4), 107 (2014)Google Scholar
  51. 51.
    Yang, J., Franco, J.S., Hétroy-Wheeler, F., Wuhrer, S.: Analyzing clothing layer deformation statistics of 3D human motions. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 237–253 (2018)Google Scholar
  52. 52.
    Yang, S., et al.: Physics-inspired garment recovery from a single-view image. ACM Trans. Graph. (TOG) 37(5), 170 (2018)CrossRefGoogle Scholar
  53. 53.
    Yang, Y., Yu, Y., Zhou, Y., Du, S., Davis, J., Yang, R.: Semantic parametric reshaping of human body models. In: International Conference on 3D Vision (3DV), vol. 2, pp. 41–48. IEEE (2014)Google Scholar
  54. 54.
    Yao, P., Fang, Z., Wu, F., Feng, Y., Li, J.: Densebody: directly regressing dense 3D human pose and shape from a single color image. arXiv preprint arXiv:1903.10153 (2019)
  55. 55.
    Yu, T., et al.: Simulcap: Single-view human performance capture with cloth simulation. arXiv preprint arXiv:1903.06323 (2019)
  56. 56.
    Zhang, C., Pujades, S., Black, M.J., Pons-Moll, G.: Detailed, accurate, human shape estimation from clothed 3D scan sequences. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4191–4200 (2017)Google Scholar
  57. 57.
    Zheng, Z., Yu, T., Wei, Y., Dai, Q., Liu, Y.: Deephuman: 3D human reconstruction from a single image. In: IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  58. 58.
    Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2017)CrossRefGoogle Scholar
  59. 59.
    Zhu, H., Zuo, X., Wang, S., Cao, X., Yang, R.: Detailed human shape estimation from a single image by hierarchical mesh deformation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4491–4500 (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of Science and Technology of ChinaHefeiChina
  2. 2.State Key Lab of CAD&CGZhejiang UniversityHangzhouChina

Personalised recommendations