Skip to main content
SpringerLink
Log in

Multi-Feature Super-Resolution Network for Cloth Wrinkle Synthesis

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Existing physical cloth simulators suffer from expensive computation and difficulties in tuning mechanical parameters to get desired wrinkling behaviors. Data-driven methods provide an alternative solution. They typically synthesize cloth animation at a much lower computational cost, and also create wrinkling effects that are similar to the training data. In this paper we propose a deep learning based method for synthesizing cloth animation with high resolution meshes. To do this we first create a dataset for training: a pair of low and high resolution meshes are simulated and their motions are synchronized. As a result the two meshes exhibit similar large-scale deformation but different small wrinkles. Each simulated mesh pair is then converted into a pair of low- and high-resolution “images” (a 2D array of samples), with each image pixel being interpreted as any of three descriptors: the displacement, the normal and the velocity. With these image pairs, we design a multi-feature super-resolution (MFSR) network that jointly trains an upsampling synthesizer for the three descriptors. The MFSR architecture consists of shared and task-specific layers to learn multi-level features when super-resolving three descriptors simultaneously. Frame-to-frame consistency is well maintained thanks to the proposed kinematics-based loss function. Our method achieves realistic results at high frame rates: 12–14 times faster than traditional physical simulation. We demonstrate the performance of our method with various experimental scenes, including a dressed character with sophisticated collisions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Liang J, Lin M C. Machine learning for digital try-on: Challenges and progress. Computational Visual Media, 2021, 7(2): 159-167. https://doi.org/10.1007/s41095-020-0189-1.

    Article  Google Scholar 

  2. Wang M, Lyu X Q, Li Y J, Zhang F L. VR content creation and exploration with deep learning: A survey. Computational Visual Media, 2020, 6(1): 3-28. https://doi.org/10.1007/s41095-020-0162-z.

    Article  Google Scholar 

  3. Terzopoulos D, Platt J, Barr A, Fleischer K. Elastically deformable models. In Proc. the 14th Annual Conference on Computer Graphics and Interactive Techniques, August 1987, pp.205-214. https://doi.org/10.1145/37401.37427.

  4. Provot X. Collision and self-collision handling in cloth model dedicated to design garments. In Proc. the Eurographics Workshop on Computer Animation and Simulation, September 1997, pp.177-189. https://doi.org/10.1007/978-3-7091-6874-5_13.

  5. Baraff D, Witkin A. Large steps in cloth simulation. In Proc. the 25th Annual Conference on Computer Graphics and Interactive Techniques, July 1998, pp.43-54. https://doi.org/10.1145/280814.280821.

  6. Bridson R, Marino S, Fedkiw R. Simulation of clothing with folds and wrinkles. In Proc. the 2003 ACM SIG-GRAPH/Eurographics Symposium on Computer Animation, July 2003, pp.28-36. https://doi.org/10.1145/1198555.1198573.

  7. Wang H, Hecht F, Ramamoorthi R, O’Brien J. Example-based wrinkle synthesis for clothing animation. ACM Trans. Graph., 2010, 29(4): Article No. 107. https://doi.org/10.1145/1778765.1778844.

  8. Zurdo J S, Brito J P, Otaduy M A. Animating wrinkles by example on non-skinned cloth. IEEE Trans. Visual. Comput. Graph., 2013, 19(1): 149-158. https://doi.org/10.1109/TVCG.2012.79.

    Article  Google Scholar 

  9. Santesteban I, Otaduy M A, Casas D. Learning-based animation of clothing for virtual try-on. Computer Graphics Forum, 2019, 38(2): 355-366. https://doi.org/10.1111/cgf.13643.

    Article  Google Scholar 

  10. Feng W W, Yu Y, Kim B U. A deformation transformer for real-time cloth animation. ACM Trans. Graph., 2010, 29(4): Article No. 108. https://doi.org/10.1145/1778765.1778845.

  11. De Aguiar E, Sigal L, Treuille A, Hodgins J K. Stable spaces for real-time clothing. ACM Trans. Graph., 2010, 29(3): Article No. 106. https://doi.org/10.1145/1833351.1778843.

  12. Kavan L, Gerszewski D, Bargteil A W, Sloan P P. Physics-inspired upsampling for cloth simulation in games. ACM Trans. Graph., 2011, 30(4): Article No. 93. https://doi.org/10.1145/2010324.1964988.

  13. Chen L, Ye J, Jiang L, Ma C, Cheng Z, Zhang X. Synthesizing cloth wrinkles by CNN-based geometry image super-resolution. Computer Animation and Virtual Worlds, 2018, 29(3/4): Article No. e1810. https://doi.org/10.1002/cav.1810.

  14. Oh Y J, Lee T M, Lee I K. Hierarchical cloth simulation using deep neural networks. In Proc. the 2018 Computer Graphics International, June 2018, pp.139-146. https://doi.org/10.1145/3208159.3208162.

  15. Lähner Z, Cremers D, Tung T. DeepWrinkles: Accurate and realistic clothing modeling. In Proc. the 15th European Conference on Computer Vision, September 2018, pp.698-715. https://doi.org/10.1007/978-3-030-01225-0_41.

  16. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z. Photorealistic single image super-resolution using a generative adversarial network. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.105-114. https://doi.org/10.1109/CVPR.2017.19.

  17. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y. Residual dense network for image super-resolution. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.2472-2481. https://doi.org/10.1109/CVPR.2018.00262.

  18. Gu X, Gortler S J, Hoppe H. Geometry images. ACM Trans. Graph., 2002, 21(3): 355-361. https://doi.org/10.1145/566654.566589.

  19. Narain R, Samii A, O’Brien J F. Adaptive anisotropic remeshing for cloth simulation. ACM Trans. Graph., 2012, 31(6): Article No. 152. https://doi.org/10.1145/2366145.2366171.

  20. Liu T, Bargteil A W, O’Brien J F, Kavan L. Fast simulation of mass-spring systems. ACM Trans. Graph., 2013, 32(6): Article No. 124. https://doi.org/10.1145/2508363.2508406.

  21. Guan P, Reiss L, Hirshberg D A, Weiss A, Black M J. DRAPE: Dressing any person. ACM Trans. Graph., 2012, 31(4): Article No. 35. https://doi.org/10.1145/2185520.2185531.

  22. Kim D, Koh W, Narain R, Fatahalian K, Treuille A, O’Brien J F. Near-exhaustive precomputation of secondary cloth effects. ACM Trans. Graph., 2013, 32(4): Article No. 87. https://doi.org/10.1145/2461912.2462020.

  23. Gundogdu E, Constantin V, Seifoddini A, Dang M, Salzmann M, Fua P. GarNet: A two-stream network for fast and accurate 3D cloth draping. In Proc. the 2019 IEEE/CVF International Conference on Computer Vision, Oct. 27–Nov 2, 2019, pp.8738-8747. https://doi.org/10.1109/ICCV.2019.00883.

  24. Wang T Y, Ceylan D, Popovic J, Mitra N J. Learning a shared shape space for multimodal garment design. ACM Trans. Graph., 2018, 37(6): Article No. 203. https://doi.org/10.1145/3272127.3275074.

  25. Wang T Y, Shao T, Fu K, Mitra N J. Learning an intrinsic garment space for interactive authoring of garment animation. ACM Transactions on Graphics, 2019, 38(6): Article No. 220. https://doi.org/10.1145/3355089.3356512.

  26. Hahn F, Thomaszewski B, Coros S, Sumner R W, Cole F, Meyer M, DeRose T, Gross M. Subspace clothing simulation using adaptive bases. ACM Trans. Graph., 2014, 33(4): Article No. 105. https://doi.org/10.1145/2601097.2601160.

  27. Xiao Y P, Lai Y K, Zhang F L, Li C P, Gao L. A survey on deep geometry learning: From a representation perspective. Computational Visual Media, 2020, 6(2): 113-133. https://doi.org/10.1007/s41095-020-0174-8.

    Article  Google Scholar 

  28. Yuan Y J, Lai Y K, Wu T, Gao L, Liu L. A revisit of shape editing techniques: From the geometric to the neural viewpoint. arXiv: 2103.01694, 2021. https://arxiv.org/abs/2103.01694, Jan. 2021.

  29. Wang P S, Liu Y, Guo Y X, Sun C Y, Tong X. O-CNN: Octree-based convolutional neural networks for 3D shape analysis. ACM Transactions on Graphics, 2017, 36(4): Article No. 72. https://doi.org/10.1145/3072959.3073608.

  30. Su H, Maji S, Kalogerakis E, Learned-Miller E G. Multiview convolutional neural networks for 3D shape recognition. In Proc. the 2015 IEEE International Conference on Computer Vision, Dec. 2015, pp.945-953. https://doi.org/10.1109/ICCV.2015.114.

  31. Sinha A, Bai J, Ramani K. Deep learning 3D shape surfaces using geometry images. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.223-240. https://doi.org/10.1007/978-3-319-46466-4_14.

  32. Tan Q, Gao L, Lai Y, Yang J, Xia S. Mesh-based autoencoders for localized deformation component analysis. In Proc. the 32nd Conference on Artificial Intelligence, Feb. 2018, pp.2452-2459.

  33. Tan Q, Gao L, Lai Y, Yang J, Xia S. Variational autoencoders for deforming 3D mesh models. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.5841-5850. https://doi.org/10.1109/CVPR.2018.00612.

  34. Gao L, Lai Y K, Liang D, Chen S Y, Xia S. Efficient and flexible deformation representation for data-driven surface modeling. ACM Transactions on Graphics, 2016, 35(5): Article No. 158. https://doi.org/10.1145/2908736.

  35. Gao L, Lai Y K, Yang J, Zhang L X, Xia S, Kobbelt L. Sparse data driven mesh deformation. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(3): 2085-2100. https://doi.org/10.1109/TVCG.2019.2941200.

    Article  Google Scholar 

  36. Zhang M, Wang T, Ceylan D, Mitra N J. Deep detail enhancement for any garment. arXiv:2008.04367, 2020. https://arxiv.org/abs/2008.04367v1, Jan. 2021.

  37. Dong C, Loy C C, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. https://doi.org/10.1109/TPAMI.2015.2439281.

    Article  Google Scholar 

  38. Liu S, Gang R, Li C, Song R. Adaptive deep residual network for single image super-resolution. Computational Visual Media, 2019, 5(4): 391-401. https://doi.org/10.1007/s41095-019-0158-8.

    Article  Google Scholar 

  39. Yue H J, Shen S, Yang J Y, Hu H F, Chen Y F. Reference image guided super-resolution via progressive channel attention networks. Journal of Computer Science and Technology, 2020, 35(3): 551-563. https://doi.org/10.1007/s11390-020-0270-3.

    Article  Google Scholar 

  40. Dong C, Loy C C, Tang X. Accelerating the super-resolution convolutional neural network. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.391-407. https://doi.org/10.1007/978-3-319-46475-6_25.

  41. Shi W, Caballero J, Huszár F, Totz J, Aitken A P, Bishop R, Rueckert D, Wang Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.1874-1883. https://doi.org/10.1109/CVPR.2016.207.

  42. Haris M, Shakhnarovich G, Ukita N. Deep back projection networks for super-resolution. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018, pp.1664-1673. https://doi.org/10.1109/CVPR.2018.00179.

  43. Kappeler A, Yoo S, Dai Q, Katsaggelos A K. Video super-resolution with convolutional neural networks. IEEE Transactions on Computational Imaging, 2016, 2(2): 109-122. https://doi.org/10.1109/TCI.2016.2532323.

    Article  MathSciNet  Google Scholar 

  44. Chu M, Xie Y, Leal-Taixé L, Thuerey N. Learning temporal coherence via self-supervision for GAN-based video generation. ACM Trans. Graph., 2020, 39(4): Article No. 75. https://doi.org/10.1145/3386569.3392457.

  45. Bhattacharjee P, Das S. Directional attention based video frame prediction using graph convolutional networks. In Proc. the 2019 International Joint Conference on Neural Networks, July 2019, pp.4268-4277. https://doi.org/10.1109/IJCNN.2019.8852090.

  46. Xie Y, Franz E, Chu M, Thuerey N. TempoGAN: A temporally coherent, volumetric GAN for super-resolution fluid flow. ACM Transactions on Graphics, 2018, 37(4): Article No. 95. https://doi.org/10.1145/3197517.3201304.

  47. Kabsch W. A discussion of the solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Foundations and Advances, 1978, 34(5): 827-828. https://doi.org/10.1107/S0567739478001680.

    Article  Google Scholar 

  48. Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. the 13th International Conference on Artificial Intelligence and Statistics, May 2010, pp.249-256.

  49. Bergou M, Mathur S, Wardetzky M, Grinspun E. TRACKS: Toward directable thin shells. ACM Trans. Graph., 2007, 26(3): Article No. 50. https://doi.org/10.1145/1276377.1276439.

  50. Müller M, Gross M. Interactive virtual materials. In Proc. the 2004 Graphics Interface Conference, May 2004, pp.239-246.

  51. Caruana R. Multitask learning. Machine Learning, 1997, 28(1): 41-75. https://doi.org/10.1023/A:1007379606734.

    Article  MathSciNet  Google Scholar 

  52. Burden R, Faires J. Numerical Analysis (9th edition). Cengage Learning, 2010.

  53. Ye J, Ma G, Jiang L, Chen L, Li J, Xiong G, Zhang X, Tang M. A unified cloth untangling framework through discrete collision detection. Computer Graphics Forum, 2017, 36(7): 217-228. https://doi.org/10.1111/cgf.13287.

    Article  Google Scholar 

  54. Wang H, O’Brien J F, Ramamoorthi R. Data-driven elastic models for cloth: Modeling and measurement. ACM Trans. Graph., 2011, 30(4): Article No. 71. https://doi.org/10.1145/2010324.1964966.

  55. Kingma D P, Ba J. Adam: A method for stochastic optimization. In Proc. the 3rd International Conference on Learning Representations, May 2015.

Download references

Author information

Authors and Affiliations

  1. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Lan Chen, Juntao Ye & Xiaopeng Zhang

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China

    Lan Chen

  3. Zhejiang Lab, Hangzhou, 311121, China

    Xiaopeng Zhang

Authors
  1. Lan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juntao Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaopeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaopeng Zhang.

Supplementary Information

ESM 1

(PDF 210 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, L., Ye, J. & Zhang, X. Multi-Feature Super-Resolution Network for Cloth Wrinkle Synthesis. J. Comput. Sci. Technol. 36, 478–493 (2021). https://doi.org/10.1007/s11390-021-1331-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-021-1331-y

Keywords

Search

Navigation