Advertisement

DeepWrinkles: Accurate and Realistic Clothing Modeling

  • Zorah Lähner
  • Daniel Cremers
  • Tony TungEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11208)

Abstract

We present a novel method to generate accurate and realistic clothing deformation from real data capture. Previous methods for realistic cloth modeling mainly rely on intensive computation of physics-based simulation (with numerous heuristic parameters), while models reconstructed from visual observations typically suffer from lack of geometric details. Here, we propose an original framework consisting of two modules that work jointly to represent global shape deformation as well as surface details with high fidelity. Global shape deformations are recovered from a subspace model learned from 3D data of clothed people in motion, while high frequency details are added to normal maps created using a conditional Generative Adversarial Network whose architecture is designed to enforce realism and temporal consistency. This leads to unprecedented high-quality rendering of clothing deformation sequences, where fine wrinkles from (real) high resolution observations can be recovered. In addition, as the model is learned independently from body shape and pose, the framework is suitable for applications that require retargeting (e.g., body animation). Our experiments show original high quality results with a flexible model. We claim an entirely data-driven approach to realistic cloth wrinkle generation is possible.

Keywords

3D surface deformation modeling Cloth simulation Normal maps Deep neural networks 

Notes

Acknowledgements

We would like to thank the FRL teams for their support, and Vignesh Ganapathi-Subramanian for preliminary work on the subspace model.

Supplementary material

Supplementary material 1 (mp4 15141 KB)

474208_1_En_41_MOESM2_ESM.pdf (2.8 mb)
Supplementary material 2 (pdf 2917 KB)

References

  1. 1.
    Temporal-3dMD systems (4d) (2018). www.3dmd.com
  2. 2.
    de Aguiar, E., Sigal, L., Treuille, A., Hodgins, J.: Stable spaces for real-time clothing. In: Hart, J.C. (ed.) ACM Transactions for Graphics. Association for Computing Machinery (ACM) (2010)Google Scholar
  3. 3.
    de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H., Thrun, S.: Performance capture from sparse multi-view video. In: Hart, J.C. (ed.) ACM Transactions for Graphics, vol. 27, pp. 98:1–98:10. Association for Computing Machinery (ACM) (2008)Google Scholar
  4. 4.
    Baraff, D., Witkin, A., Kass, M.: Untangling cloth. In: Hart, J.C. (ed.) ACM Transactions on Graphics, vol. 22, pp. 862–870. Association for Computing Machinery (ACM), New York, NY, USA, July 2003CrossRefGoogle Scholar
  5. 5.
    Beeler, T., et al.: High-quality passive facial performance capture using anchor frames. In: Alexa, M. (ed.) ACM Transactions for Graphics, vol. 30, pp. 75:1–75:10. Association for Computing Machinery (ACM), New York, NY, USA, August 2011CrossRefGoogle Scholar
  6. 6.
    Bogo, F., Romero, J., Pons-Moll, G., Black, M.J.: Dynamic FAUST: registering human bodies in motion. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, July 2017Google Scholar
  7. 7.
    Bradley, D., Popa, T., Sheffer, A., Heidrich, W., Boubekeur, T.: Markerless garment capture. In: Hart, J.C. (ed.) ACM Transactions for Graphics, vol. 27, pp. 99:1–99:9. Association for Computing Machinery (ACM), New York, NY, USA, August 2008CrossRefGoogle Scholar
  8. 8.
    Bridson, R., Marino, S., Fedkiw, R.: Simulation of clothing with folds and wrinkles. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 28–36. Eurographics Association, San Diego, CA, USA, July 2003Google Scholar
  9. 9.
    Choi, K.J., Ko, H.S.: Stable but responsive cloth. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, pp. 604–611. ACM, New York, NY, USA (2002)Google Scholar
  10. 10.
    Chu, M., Thuerey, N.: Data-driven synthesis of smoke flows with CNN-based feature descriptors. In: Alexa, M. (ed.) ACM Transactions for Graphics, vol. 36, pp. 69:1–69:14. Association for Computing Machinery (ACM), New York, NY, USA, July 2017CrossRefGoogle Scholar
  11. 11.
    Cignoni, P., Montani, C., Scopigno, R., Rocchini, C.: A general method for preserving attribute values on simplified meshes. In: Proceedings of IEEE Conference on Visualization, pp. 59–66. IEEE (1998)Google Scholar
  12. 12.
    Cohen, J.D., Olano, M., Manocha, D.: Appearance-perserving simplification. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, pp. 115–122. ACM, Orlando, FL, USA, July 1998Google Scholar
  13. 13.
    Danerek, R., Dibra, E., Öztireli, C., Ziegler, R., Gross, M.: Deepgarment : 3d garment shape estimation from a single image. In: Chen, M., Zhang, R. (eds.) Computer Graphics Forum, vol. 36, pp. 269–280. Eurographics Association (2017)Google Scholar
  14. 14.
    Dou, M., et al.: Fusion4d: real-time performance capture of challenging scenes. In: Alexa, M. (ed.) ACM Transactions for Graphics, vol. 35, pp. 114:1–114:13. Association for Computing Machinery (ACM), New York, NY, USA, July 2016CrossRefGoogle Scholar
  15. 15.
    Fyffe, G., et al.: Multi-view stereo on consistent face topology. Comput. Graph. Forum 36(2), 295–309 (2017)CrossRefGoogle Scholar
  16. 16.
    Goldenthal, R., Harmon, D., Fattal, R., Bercovier, M., Grinspun, E.: Efficient simulation of inextensible cloth. In: Hart, J.C. (ed.) ACM Transactions for Graphics, vol. 26, p. 49. Association for Computing Machinery (ACM) (2007)Google Scholar
  17. 17.
    Goldlücke, B., Cremers, D.: Superresolution texture maps for multiview reconstruction. In: IEEE 12th International Conference on Computer Vision, ICCV, pp. 1677–1684. IEEE Computer Society, September 2009Google Scholar
  18. 18.
    Goodfellow, I.J., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger., K. (eds.) Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems (NIPS), pp. 2672–2680. Curran Associates Inc, Montreal, Quebec, Canada, December 2014Google Scholar
  19. 19.
    Guan, P., Reiss, L., Hirshberg, D.A., Weiss, A., Black, M.J.: DRAPE: dressing any person. ACM Trans. Graph. 31(4), 35:1–35:10 (2012)CrossRefGoogle Scholar
  20. 20.
    Guo, K., Xu, F., Wang, Y., Liu, Y., Dai, Q.: Robust non-rigid motion tracking and surface reconstruction using l0 regularization, pp. 3083–3091. IEEE Computer Society (2015)Google Scholar
  21. 21.
    Hahn, F., et al.: Subspace clothing simulation using adaptive bases. In: Alexa, M. (ed.) ACM Transactions for Graphics, vol. 33, pp. 105:1–105:9. Association for Computing Machinery (ACM) (2014)CrossRefGoogle Scholar
  22. 22.
    Hernández, C., Vogiatzis, G., Brostow, G.J., Stenger, B.,Cipolla, R.: Non-rigid photometric stereo with colored lights. In: IEEE 11th International Conference on Computer Vision, ICCV, pp. 1–8. IEEE Computer Society, Rio de Janeiro, Brazil, October 2007Google Scholar
  23. 23.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning, ICML, vol. 37, pp. 448–456. ACM (2015)Google Scholar
  24. 24.
    Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-imagetranslation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 5967–5976. IEEE Computer Society, Honolulu, HI, USA, July 2017Google Scholar
  25. 25.
    Kavan, L., Gerszewski, D., Bargteil, A.W., Sloan, P.P.: Physics-inspired upsampling for cloth simulation in games. In: Hart, J.C. (ed.) ACM Transactions for Graphics, vol. 30, pp. 93:1–93:10. Association for Computing Machinery (ACM), New York, NY, USA, July 2011CrossRefGoogle Scholar
  26. 26.
    Krishnamurthy, V., Levoy, M.: Fitting smooth surfaces to dense polygon meshes. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, New Orleans, LA, USA, August 1996Google Scholar
  27. 27.
    Lassner, C., Pons-Moll, G., Gehler, P.V.: A generative model of people in clothing. In: IEEE International Conference on Computer Vision, ICCV, pp. 853–862. IEEE Computer Society, Venice, Italy, October 2017Google Scholar
  28. 28.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 105–114. IEEE Computer Society, Honolulu, HI, USA, July 2017Google Scholar
  29. 29.
    Leroy, V., Franco, J., Boyer, E.: Multi-view dynamic shape refinement using local temporal integration. In: IEEE International Conference on Computer Vision, ICCV, pp. 3113–3122, Venice, Italy (2017)Google Scholar
  30. 30.
    Li, H., Adams, B., Guibas, L.J., Pauly, M.: Robust single-view geometry and motion reconstruction. In: Hart, J.C. (ed.) ACM Transactions for Graphics, vol. 28, pp. 175:1–175:10. Association for Computing Machinery (ACM), New York, NY, USA, December 2009CrossRefGoogle Scholar
  31. 31.
    Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. In: Alexa, M. (ed.) ACM Transactions for Graphics, vol. 34, pp. 248:1–248:16. Association for Computing Machinery (ACM) (2015)CrossRefGoogle Scholar
  32. 32.
    MarvelousDesigner (2018). www.marvelousdesigner.com
  33. 33.
    Matsuyama, T., Nobuhara, S., Takai, T., Tung, T.: 3D Video and Its Applications. Springer, London (2012).  https://doi.org/10.1007/978-1-4471-4120-4CrossRefGoogle Scholar
  34. 34.
    Miguel, E., et al.: Data-driven estimation of cloth simulation models. Comput. Graph. Forum 31(2), 519–528 (2012)CrossRefGoogle Scholar
  35. 35.
    Müller, M., Chentanez, N.: Wrinkle meshes. In: Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA 2010, pp. 85–92. Eurographics Association, Aire-la-Ville, Switzerland, Switzerland (2010)Google Scholar
  36. 36.
    Newcombe, R.A., Fox, D., Seitz, S.M.: Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 343–352. IEEE Computer Society, Boston, MA, USA, June 2015Google Scholar
  37. 37.
    Olszewski, K., et al.: Realistic dynamic facial textures from a single image using GANs. In: IEEE International Conference on Computer Vision, ICCV, pp. 5439–5448. IEEE Computer Society, Venice, Italy, October 2017Google Scholar
  38. 38.
    Pons-Moll, G., Pujades, S., Hu, S., Black, M.: ClothCap: seamless 4d clothing capture and retargeting. ACM Trans. Graph. (Proc. SIGGRAPH) 36(4), 73 (2017)CrossRefGoogle Scholar
  39. 39.
    Pons-Moll, G., Romero, J., Mahmood, N., Black, M.J.: Dyna: a model of dynamic human shape in motion. In: Alexa, M. (ed.) ACM Transactions for Graphics, vol. 34, pp. 120:1–120:14. Association for Computing Machinery (ACM), August 2015Google Scholar
  40. 40.
    Saito, S., Wei, L., Hu, L., Nagano, K., Li, H.: Photorealistic facial texture inference using deep neural networks. IEEE Conference on Computer Vision and Pattern Recognition, CVPR pp. 2326–2335 (Jul 2017)Google Scholar
  41. 41.
    Starck, J., Hilton, A.: Surface capture for performance-based animation. IEEE Comput. Graph. Appl. 27(3), 21–31 (2007)CrossRefGoogle Scholar
  42. 42.
    Sumner, R.W., Popović, J.: Deformation transfer for triangle meshes. In: Hart, J.C. (ed.) ACM Transactions for Graphics, vol. 23, pp. 399–405. Association for Computing Machinery (ACM), New York, NY, USA, August 2004CrossRefGoogle Scholar
  43. 43.
    Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Ghahramani, Z., Welling, M., Cortes, C.,Lawrence, N., Weinberger, K. (eds.) Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems (NIPS), pp. 3104–3112. Curran Associates, Inc., Montreal, Quebec, Canada, December 2014Google Scholar
  44. 44.
    Taylor, J., Shotton, J., Sharp, T., Fitzgibbon, A.: The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. In: Computer Vision and Pattern Recognition (CVPR), pp. 103–110. IEEE Computer Society, July 2012Google Scholar
  45. 45.
    Tsiminaki, V., Franco, J., Boyer, E.: High resolution 3d shape texture from multiple videos. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1502–1509. IEEE Computer Society, Columbus, OH, USA, June 2014Google Scholar
  46. 46.
    Tung, T., Matsuyama, T.: Intrinsic characterization of dynamic surfaces. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 233–240. IEEE Computer Society, Portland, OR, USA, June 2013Google Scholar
  47. 47.
    Tung, T., Nobuhara, S., Matsuyama, T.: Simultaneous super-resolution and 3d video using graph-cuts. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Anchorage, Alaska, USA, June 2008Google Scholar
  48. 48.
    Tung, T., Nobuhara, S., Matsuyama, T.: Complete multi-view reconstruction of dynamic scenes from probabilistic fusion of narrow and wide baseline stereo. In: IEEE 12th International Conference on Computer Vision, ICCV, pp. 1709–1716. IEEE Computer Society, Kyoto, Japan, September 2009Google Scholar
  49. 49.
    Vlasic, D., Baran, I., Matusik, W., Popović, J.: Articulated mesh animation from multi-view silhouettes. In: Hart, J.C. (ed.) ACM Transactions for Graphics, pp. 97:1–97:9. Association for Computing Machinery (ACM), New York, NY, USA (2008)Google Scholar
  50. 50.
    Vlasic, D., et al.: Dynamic shape capture using multi-view photometric stereo. In: Proceedings of ACM SIGGRAPH Asia, pp. 174:1–174:11. ACM, New York, NY, USA (2009)Google Scholar
  51. 51.
    Volino, P., Magnenat-Thalmann, N.: Virtual Clothing - Theory and Practice. Springer, Heidelberg (2000).  https://doi.org/10.1007/978-3-642-57278-4CrossRefzbMATHGoogle Scholar
  52. 52.
    Wang, H., Hecht, F., Ramamoorthi, R., O’Brien, J.F.: Example-based wrinkle synthesis for clothing animation. In: Alexa, M. (ed.) ACM Transactions for Graphics, vol. 29, Article no. 107:1–107:8. Association for Computing Machinery (ACM), Los Angles, CA, July 2010Google Scholar
  53. 53.
    Wang, H., Ramamoorthi, R., O’Brien, J.F.: Data-driven elastic models for cloth: Modeling and measurement. In: Alexa, M. (ed.) ACM Transactions for Graphics, vol. 30, pp. 71:1–71:11. Association for Computing Machinery (ACM), Vancouver, BC Canada, July 2011Google Scholar
  54. 54.
    White, R., Crane, K., Forsyth, D.A.: Capturing and animating occluded cloth. In: Hart, J.C. (ed.) ACM Transactions for Graphics, vol. 26, p. 34. Association for Computing Machinery (ACM) (2007)Google Scholar
  55. 55.
    Zhang, C., Pujades, S., Black, M.J., Pons-Moll, G.: Detailed, accurate, human shape estimation from clothed 3d scan sequences. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5484–5493. IEEE Computer Society, Honolulu, HI, USA, July 2017Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Facebook Reality LabsSausalitoUSA
  2. 2.Technical University MunichMunichGermany

Personalised recommendations