Advertisement

Computational Visual Media

, Volume 3, Issue 4, pp 337–347 | Cite as

Image-based clothes changing system

  • Zhao-Heng Zheng
  • Hao-Tian Zhang
  • Fang-Lue ZhangEmail author
  • Tai-Jiang Mu
Open Access
Research Article

Abstract

Current image-editing tools do not match up to the demands of personalized image manipulation, one application of which is changing clothes in usercaptured images. Previous work can change single color clothes using parametric human warping methods. In this paper, we propose an image-based clothes changing system, exploiting body factor extraction and content-aware image warping. Image segmentation and mask generation are first applied to the user input. Afterwards, we determine joint positions via a neural network. Then, body shape matching is performed and the shape of the model is warped to the user’s shape. Finally, head swapping is performed to produce realistic virtual results. We also provide a supervision and labeling tool for refinement and further assistance when creating a dataset.

Keywords

clothing try-on image warping human segmentation 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Project No. 61521002), and Research Grant of Beijing Higher Institution Engineering Research Center. This work was finished during Zhao-Heng Zheng and Hao-Tian Zhang were undergraduate students in the Department of Computer Science and Technology at Tsinghua University.

References

  1. [1]
    Chen, T.; Tan, P.; Ma, L. Q.; Cheng, M. M.; Shamir, A.; Hu, S.-M. PoseShop: Human image database construction and personalized content synthesis. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 5, 824–837, 2013.CrossRefGoogle Scholar
  2. [2]
    Zhou, S.; Fu, H.; Liu, L.; Cohen-Or, D.; Han, X. Parametric reshaping of human bodies in images. ACM Transactions on Graphics Vol. 29, No. 4, Article No.126, 2010.Google Scholar
  3. [3]
    Ferrari, V.; Marin-Jimenez, M.; Zisserman, A. Progressive search space reduction for human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2008.Google Scholar
  4. [4]
    Andriluka, M.; Roth, S.; Schiele, B. Pictorial structures revisited: People detection and articulated pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1014–1021, 2009.Google Scholar
  5. [5]
    Johnson, S.; Everingham, M. Combining discriminative appearance and segmentation cues for articulated human pose estimation. In: Proceedings of the IEEE 12th International Conference on Computer Vision Workshops, 405–412, 2009.Google Scholar
  6. [6]
    Sun, M.; Savarese, S. Articulated part-based model for joint object detection and pose estimation. In: Proceedings of the International Conference on Computer Vision, 723–730, 2011.Google Scholar
  7. [7]
    Tian, Y.; Zitnick, C. L.; Narasimhan, S. G. Exploring the spatial hierarchy of mixture models for human pose estimation. In: Computer Vision–ECCV 2012. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer-Verlag Berlin Heidelberg, 256–269, 2012.CrossRefGoogle Scholar
  8. [8]
    Yang, Y.; Ramanan, D. Articulated pose estimation with flexible mixtures-of-parts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1385–1392, 2011.Google Scholar
  9. [9]
    Vineet, V.; Warrell, J.; Ladicky, L.; Torr, P. H. S. Human instance segmentation from video using detector-based conditional random fields. In: Proceedings of the 22nd British Machine Vision Conference, 80.1–80.11, 2011.Google Scholar
  10. [10]
    Tompson, J. J.; Jain, A.; LeCun, Y.; Bregler, C. Joint training of a convolutional network and a graphical model for human pose estimation. In: Proceedings of the Advances in Neural Information Processing Systems 27, 1799–1807, 2014.Google Scholar
  11. [11]
    Ramakrishna, V.; Munoz, D.; Hebert, M.; Bagnell, J. A.; Sheikh, Y. Pose machines: Articulated pose estimation via inference machines. In: Computer Vision–ECCV 2014. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer International Publishing Switzerland, 33–47, 2014.Google Scholar
  12. [12]
    Carreira, J.; Agrawal, P.; Fragkiadaki, K.; Malik, J. Human pose estimation with iterative error feedback. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4733–4742, 2016.Google Scholar
  13. [13]
    Wei, S.-E.; Ramakrishna, V.; Kanade, T.; Sheikh, Y. Convolutional pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4724–4732, 2016.Google Scholar
  14. [14]
    Blake, A.; Rother, C.; Brown, M.; Perez, P.; Torr, P. Interactive image segmentation using an adaptive GMMRF model. In: Computer Vision–ECCV 2004. Pajdla, T.; Matas, J. Eds. Springer-Verlag Berlin Heidelberg, 428–441, 2004.CrossRefGoogle Scholar
  15. [15]
    Chen, L.-C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A. L. Semantic image segmentation with deep convolutional nets and fully connected CRFS. In: Proceedings of the International Conference on Learning Representations, 2015.Google Scholar
  16. [16]
    Ladický, L.; Russell, C.; Kohli, P.; Torr, P. H. S. Associative hierarchical CRFs for object class image segmentation. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 739–746, 2009.Google Scholar
  17. [17]
    Arbeláez, P.; Hariharan, B.; Gu, C.; Gupta, S.; Bourdev, L.; Malik, J. Semantic segmentation using regions and parts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3378–3385, 2012.Google Scholar
  18. [18]
    Arbeláez, P.; Maire, M.; Fowlkes, C.; Malik, J. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 5, 898–916, 2011.CrossRefGoogle Scholar
  19. [19]
    Arbeláez, P.; Pont-Tuset, J.; Barron, J. T.; Marques, F.; Malik, J. Multiscale combinatorial grouping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 328–335, 2014.Google Scholar
  20. [20]
    Ko, B. C.; Nam, J.-Y. Object-of-interest image segmentation based on human attention and semantic region clustering. Journal of the Optical Society of America A Vol. 23, No. 10, 2462–2470, 2006.CrossRefGoogle Scholar
  21. [21]
    Cheng, M.-M.; Mitra, N. J.; Huang, X.; Torr, P. H. S; Hu, S.-M. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 3, 569–582, 2015.CrossRefGoogle Scholar
  22. [22]
    Cheng, M.-M.; Warrell, J.; Lin, W.-Y.; Zheng, S.; Vineet, V.; Crook, N. Efficient salient region detection with soft image abstraction. In: Proceedings of the IEEE International Conference on Computer Vision, 1529–1536, 2013.Google Scholar
  23. [23]
    Chen, L.-C.; Yang, Y.; Wang, J.; Xu, W.; Yuille, A. L. Attention to scale: Scale-aware semantic image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3640–3649, 2016.Google Scholar
  24. [24]
    Zheng, S.; Cheng, M.-M.; Warrell, J.; Sturgess, P.; Vineet, V.; Rother, C.; Torr, P. H. S. Dense semantic image segmentation with objects and attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3214–3221, 2014.Google Scholar
  25. [25]
    Igarashi, T.; Moscovich, T.; Hughes, J. F. As-rigidas-possible shape manipulation. ACM Transactions on Graphics Vol. 24, No. 3, 1134–1141, 2005.CrossRefGoogle Scholar
  26. [26]
    Arad, N.; Reisfeld, D. Image warping using few anchor points and radial functions. Computer Graphics Forum Vol. 14, No. 1, 35–46, 1995.CrossRefGoogle Scholar
  27. [27]
    Schaefer, S.; McPhail, T.; Warren, J. Image deformation using moving least squares. ACM Transactions on Graphics Vol. 25, No. 3, 533–540, 2006.CrossRefGoogle Scholar
  28. [28]
    Kaufmann, P.; Wang, O.; Sorkine-Hornung, A.; Sorkine-Hornung, O.; Smolic, A.; Gross, M. Finite element image warping. Computer Graphics Forum Vol. 32, No. 2, 31–39, 2013.CrossRefGoogle Scholar
  29. [29]
    Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.CrossRefGoogle Scholar
  30. [30]
    Chen, T.; Cheng, M.-M.; Tan, P.; Shamir, A.; Hu, S.-M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics Vol. 28, No. 5, Article No.124, 2009.Google Scholar
  31. [31]
    He, K.; Sun, J.; Tang, X. Guided image filtering. In: Computer Vision–ECCV 2010. Daniilidis, K.; Maragos, P.; Paragios, N. Eds. Springer-Verlag Berlin Heidelberg, 1–14, 2010.Google Scholar
  32. [32]
    Pérez, P.; Gangnet, M.; Blake, A. Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313–318, 2003.CrossRefGoogle Scholar
  33. [33]
    Shahrian, E.; Rajan, D. Weighted color and texture sample selection for image matting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 718–725, 2012.Google Scholar
  34. [34]
    Belongie, S.; Malik, J.; Puzicha, J. Shape context: A new descriptor for shape matching and object recognition. In: Proceedings of the 13th International Conference on Neural Information Processing Systems, 798–804, 2000.Google Scholar
  35. [35]
    Shilkrot, R.; Cohen-Or, D.; Shamir, A.; Liu, L. Garment personalization via identity transfer. IEEE Computer Graphics and Applications Vol. 33, No. 4, 62–72, 2013.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2017

Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Authors and Affiliations

  • Zhao-Heng Zheng
    • 1
  • Hao-Tian Zhang
    • 2
  • Fang-Lue Zhang
    • 3
    Email author
  • Tai-Jiang Mu
    • 4
  1. 1.Computer Science and EngineeringUniversity of MichiganAnn ArborUSA
  2. 2.Computer Science DepartmentStanford UniversityStanfordUSA
  3. 3.School of Engineering and Computer ScienceVictoria University of WellingtonWellingtonNew Zealand
  4. 4.TNListTsinghua UniversityBeijingChina

Personalised recommendations