Machine learning for digital try-on: Challenges and progress

Abstract

Digital try-on systems for e-commerce have the potential to change people's lives and provide notable economic benefits. However, their development is limited by practical constraints, such as accurate sizing of the body and realism of demonstrations. We enumerate three open challenges remaining for a complete and easy-to-use try-on system that recent advances in machine learning make increasingly tractable. For each, we describe the problem, introduce state-of-the-art approaches, and provide future directions.

References

  1. [1]

    Zheng, Z. H.; Zhang, H. T.; Zhang, F. L.; Mu, T. J. Image-based clothes changing system. Computational Visual Media Vol. 3, No. 4, 337–347, 2017.

    Article  Google Scholar 

  2. [2]

    Dibra, E.; Jain, H.; Öztireli, C.; Ziegler, R.; Gross, M. HS-Nets: Estimating human body shape from silhouettes with convolutional neural networks. In: Proceedings of the 4th International Conference on 3D Vision, 108–117, 2016.

    Google Scholar 

  3. [3]

    Bălan, A. O.; Black, M. J. The naked truth: Estimating body shape under clothing. In: Computer Vision -ECCV 2008. Lecture Notes in Computer Science, Vol. 5303. Forsyth, D.; Torr, P.; Zisserman, A. Eds. Springer Berlin, 15–29, 2008.

    Google Scholar 

  4. [4]

    Lassner, C.; Romero, J.; Kiefel, M.; Bogo, F.; Black, M. J.; Gehler, P. V. Unite the people: Closing the loop between 3D and 2D human representations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4704–4713, 2017.

    Google Scholar 

  5. [5]

    Loper, M.; Mahmood, N.; Romero, J.; Pons-Moll, G.; Black, M. J. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics Vol. 34, No. 6, Article No. 248, 2015.

    Google Scholar 

  6. [6]

    Wei, S.-E.; Ramakrishna, V.; Kanade, T.; Sheikh, Y. Convolutional pose machines. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 4724–4732, 2016.

    Google Scholar 

  7. [7]

    Cao, Z.; Simon, T.; Wei, S.; Sheikh, Y. Realtime multiperson 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1302–1310, 2017.

    Google Scholar 

  8. [8]

    Mehta, D.; Sridhar, S.; Sotnychenko, O.; Rhodin, H.; Shafiei, M.; Seidel, H.-P.; Xu, W.; Casas, D.; Theobalt, C. VNect: Realtime 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 44, 2017.

    Google Scholar 

  9. [9]

    Alldieck, T.; Magnor, M.; Xu, W.; Theobalt, C.; Pons-Moll, G. Video based reconstruction of 3D people models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8387–8397, 2018.

    Google Scholar 

  10. [10]

    Kanazawa, A.; Black, M. J.; Jacobs, D. W.; Malik, J. End-to-end recovery of human shape and pose. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7122–7131, 2018.

    Google Scholar 

  11. [11]

    Varol, G.; Ceylan, D.; Russell, B.; Yang, J.; Yumer, E.; Laptev, I. Bodynet: Volumetric inference of 3D human body shapes. In: Proceedings of the European Conference on Computer Vision, 20–36, 2018.

    Google Scholar 

  12. [12]

    Zheng, Z.; Yu, T.; Wei, Y.; Dai, Q.; Liu, Y. Deephuman: 3D human reconstruction from a single image. In: Proceedings of the IEEE International Conference on Computer Vision, 7739–7749, 2019.

    Google Scholar 

  13. [13]

    Saito, S.; Huang, Z.; Natsume, R.; Morishima, S.; Kanazawa, A.; Li, H. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: Proceedings of the IEEE International Conference on Computer Vision, 2304–2314, 2019.

    Google Scholar 

  14. [14]

    Xu, Y.; Zhu, S.-C.; Tung, T. Denserac: Joint 3D pose and shape estimation by dense render-and-compare. In: Proceedings of the IEEE International Conference on Computer Vision, 7760–7770, 2019.

    Google Scholar 

  15. [15]

    Smith, D.; Loper, M.; Hu, X.; Mavroidis, P.; Romero, J. FACSIMILE: Fast and accurate scans from an image in less than a second. In: Proceedings of the IEEE International Conference on Computer Vision, 5329–5338, 2019.

    Google Scholar 

  16. [16]

    Alldieck, T.; Magnor, M.; Bhatnagar, B. L.; Theobalt, C.; Pons-Moll, G. Learning to reconstruct people in clothing from a single RGB camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1175–1186, 2019.

    Google Scholar 

  17. [17]

    Kolotouros, N.; Pavlakos, G.; Black, M. J.; Daniilidis, K. Learning to reconstruct 3D human pose and shape via modelfitting in the loop. In: Proceedings of the IEEE International Conference on Computer Vision, 2252–2261, 2019.

    Google Scholar 

  18. [18]

    Liang, J.; Lin, M. C. Shape-aware human pose and shape reconstruction using multi-view images. In: Proceedings of the IEEE International Conference on Computer Vision, 4352–4362, 2019.

    Google Scholar 

  19. [19]

    Yang, S.; Pan, Z. R.; Amert, T.; Wang, K.; Yu, L. C.; Berg, T.; Lin, M. C. Physics-inspired garment recovery from a single-view image. ACM Transactions on Graphics Vol. 37, No. 5, Article No. 170, 2018.

    Google Scholar 

  20. [20]

    Yang, S.; Liang, J.; Lin, M. C.; Learning-based cloth material recovery from video. In: Proceedings of the IEEE International Conference on Computer Vision, 4383–4393, 2017.

    Google Scholar 

  21. [21]

    Qiao, Y. L.; Liang, J. B.; Koltun, V.; Lin, M. C. Scalable differentiable physics for learning and control. arXiv preprint arXiv:2007.02168, 2020.

    Google Scholar 

  22. [22]

    De Avila Belbute-Peres, F.; Smith, K. A.; Allen, K.; Tenenbaum, J.; Kolter, J. Z. End-to-end differentiable physics for learning and control. In: Proceedings of the Advances in Neural Information Processing Systems, 2018.

    Google Scholar 

  23. [23]

    Degrave, J.; Hermans, M.; Dambre, J.; Wyffels, F. A differentiable physics engine for deep learning in robotics. Frontiers in Neurorobotics Vol. 13, 6, 2019.

    Article  Google Scholar 

  24. [24]

    Liang, J.; Lin, M.; Koltun, V. Differentiable cloth simulation for inverse problems. In: Proceedings of the 33rd Conference on Neural Information Processing Systems, 2019.

    Google Scholar 

  25. [25]

    Hu, Y.; Liu, J.; Spielberg, A.; Tenenbaum, J. B.; Freeman, W. T.; Wu, J.; Rus, D.; Matusik, W. ChainQueen: A real-time differentiable physical simulator for soft robotics. In: Proceedings of the International Conference on Robotics and Automation, 6265–6271, 2019.

    Google Scholar 

  26. [26]

    Hu, Y. M.; Anderson, L.; Li, T. M.; Sun, Q.; Carr, N.; Ragan-Kelley, J.; Durand, F. DiffTaichi: Differentiable programming for physical simulation. arXiv preprint arXiv:1910.00935, 2019.

    Google Scholar 

  27. [27]

    Liu, K. X.; Zeng, X. Y.; Bruniaux, P.; Tao, X. Y.; Yao, X. F.; Li, V.; Wang, J. 3D interactive garment patternmaking technology. Computer-Aided Design Vol. 104, 113–124, 2018.

    Article  Google Scholar 

  28. [28]

    Huang, P.; Yao, J.; Zhao, H. Automatic realistic 3D garment generation based on two images. In: Proceedings of the International Conference on Virtual Reality and Visualization, 250–257, 2016.

    Google Scholar 

  29. [29]

    Wang, T. Y.; Ceylan, D.; Popović, J.; Mitra, N. J. Learning a shared shape space for multimodal garment design. ACM Transactions on Graphics Vol. 37, No. 6, Article No. 203, 2018.

    Google Scholar 

Download references

Acknowledgments

This research was supported in part by the Iribe Professorship and the National Science Foundation.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Junbang Liang.

Additional information

Junbang Liang is a 4th-year Ph.D. student in the University of Maryland, College Park. He received his B.E. degree from Tsinghua University in 2016, and his M.S. degree from the University of North Carolina in 2018. His research interests are physics-based cloth simulation, computer vision, and machine learning.

Ming C. Lin is a Distinguished University Professor and Elizabeth Stevinson Iribe Chair of Computer Science at the University of Maryland College Park and John R. and Louise S. Parker Distinguished Professor Emerita of Computer Science at the University of North Carolina, Chapel Hill. She obtained her B.S., M.S., and Ph.D. degrees in electrical engineering and computer science from the University of California, Berkeley. She is a Fellow of ACM, IEEE, and Eurographics, and a member of ACM SIGGRAPH Academy.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecomm-ons.org/licenses/by/4.0/ Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liang, J., Lin, M.C. Machine learning for digital try-on: Challenges and progress. Comp. Visual Media (2020). https://doi.org/10.1007/s41095-020-0189-1

Download citation

Keywords

  • machine learning
  • digital try-on
  • garment modeling
  • human body estimation
  • material modeling