Abstract
Current image-editing tools do not match up to the demands of personalized image manipulation, one application of which is changing clothes in usercaptured images. Previous work can change single color clothes using parametric human warping methods. In this paper, we propose an image-based clothes changing system, exploiting body factor extraction and content-aware image warping. Image segmentation and mask generation are first applied to the user input. Afterwards, we determine joint positions via a neural network. Then, body shape matching is performed and the shape of the model is warped to the user’s shape. Finally, head swapping is performed to produce realistic virtual results. We also provide a supervision and labeling tool for refinement and further assistance when creating a dataset.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Chen, T.; Tan, P.; Ma, L. Q.; Cheng, M. M.; Shamir, A.; Hu, S.-M. PoseShop: Human image database construction and personalized content synthesis. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 5, 824–837, 2013.
Zhou, S.; Fu, H.; Liu, L.; Cohen-Or, D.; Han, X. Parametric reshaping of human bodies in images. ACM Transactions on Graphics Vol. 29, No. 4, Article No.126, 2010.
Ferrari, V.; Marin-Jimenez, M.; Zisserman, A. Progressive search space reduction for human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2008.
Andriluka, M.; Roth, S.; Schiele, B. Pictorial structures revisited: People detection and articulated pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1014–1021, 2009.
Johnson, S.; Everingham, M. Combining discriminative appearance and segmentation cues for articulated human pose estimation. In: Proceedings of the IEEE 12th International Conference on Computer Vision Workshops, 405–412, 2009.
Sun, M.; Savarese, S. Articulated part-based model for joint object detection and pose estimation. In: Proceedings of the International Conference on Computer Vision, 723–730, 2011.
Tian, Y.; Zitnick, C. L.; Narasimhan, S. G. Exploring the spatial hierarchy of mixture models for human pose estimation. In: Computer Vision–ECCV 2012. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer-Verlag Berlin Heidelberg, 256–269, 2012.
Yang, Y.; Ramanan, D. Articulated pose estimation with flexible mixtures-of-parts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1385–1392, 2011.
Vineet, V.; Warrell, J.; Ladicky, L.; Torr, P. H. S. Human instance segmentation from video using detector-based conditional random fields. In: Proceedings of the 22nd British Machine Vision Conference, 80.1–80.11, 2011.
Tompson, J. J.; Jain, A.; LeCun, Y.; Bregler, C. Joint training of a convolutional network and a graphical model for human pose estimation. In: Proceedings of the Advances in Neural Information Processing Systems 27, 1799–1807, 2014.
Ramakrishna, V.; Munoz, D.; Hebert, M.; Bagnell, J. A.; Sheikh, Y. Pose machines: Articulated pose estimation via inference machines. In: Computer Vision–ECCV 2014. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer International Publishing Switzerland, 33–47, 2014.
Carreira, J.; Agrawal, P.; Fragkiadaki, K.; Malik, J. Human pose estimation with iterative error feedback. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4733–4742, 2016.
Wei, S.-E.; Ramakrishna, V.; Kanade, T.; Sheikh, Y. Convolutional pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4724–4732, 2016.
Blake, A.; Rother, C.; Brown, M.; Perez, P.; Torr, P. Interactive image segmentation using an adaptive GMMRF model. In: Computer Vision–ECCV 2004. Pajdla, T.; Matas, J. Eds. Springer-Verlag Berlin Heidelberg, 428–441, 2004.
Chen, L.-C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A. L. Semantic image segmentation with deep convolutional nets and fully connected CRFS. In: Proceedings of the International Conference on Learning Representations, 2015.
Ladický, L.; Russell, C.; Kohli, P.; Torr, P. H. S. Associative hierarchical CRFs for object class image segmentation. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 739–746, 2009.
Arbeláez, P.; Hariharan, B.; Gu, C.; Gupta, S.; Bourdev, L.; Malik, J. Semantic segmentation using regions and parts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3378–3385, 2012.
Arbeláez, P.; Maire, M.; Fowlkes, C.; Malik, J. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 33, No. 5, 898–916, 2011.
Arbeláez, P.; Pont-Tuset, J.; Barron, J. T.; Marques, F.; Malik, J. Multiscale combinatorial grouping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 328–335, 2014.
Ko, B. C.; Nam, J.-Y. Object-of-interest image segmentation based on human attention and semantic region clustering. Journal of the Optical Society of America A Vol. 23, No. 10, 2462–2470, 2006.
Cheng, M.-M.; Mitra, N. J.; Huang, X.; Torr, P. H. S; Hu, S.-M. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 37, No. 3, 569–582, 2015.
Cheng, M.-M.; Warrell, J.; Lin, W.-Y.; Zheng, S.; Vineet, V.; Crook, N. Efficient salient region detection with soft image abstraction. In: Proceedings of the IEEE International Conference on Computer Vision, 1529–1536, 2013.
Chen, L.-C.; Yang, Y.; Wang, J.; Xu, W.; Yuille, A. L. Attention to scale: Scale-aware semantic image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3640–3649, 2016.
Zheng, S.; Cheng, M.-M.; Warrell, J.; Sturgess, P.; Vineet, V.; Rother, C.; Torr, P. H. S. Dense semantic image segmentation with objects and attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3214–3221, 2014.
Igarashi, T.; Moscovich, T.; Hughes, J. F. As-rigidas-possible shape manipulation. ACM Transactions on Graphics Vol. 24, No. 3, 1134–1141, 2005.
Arad, N.; Reisfeld, D. Image warping using few anchor points and radial functions. Computer Graphics Forum Vol. 14, No. 1, 35–46, 1995.
Schaefer, S.; McPhail, T.; Warren, J. Image deformation using moving least squares. ACM Transactions on Graphics Vol. 25, No. 3, 533–540, 2006.
Kaufmann, P.; Wang, O.; Sorkine-Hornung, A.; Sorkine-Hornung, O.; Smolic, A.; Gross, M. Finite element image warping. Computer Graphics Forum Vol. 32, No. 2, 31–39, 2013.
Rother, C.; Kolmogorov, V.; Blake, A. “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics Vol. 23, No. 3, 309–314, 2004.
Chen, T.; Cheng, M.-M.; Tan, P.; Shamir, A.; Hu, S.-M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics Vol. 28, No. 5, Article No.124, 2009.
He, K.; Sun, J.; Tang, X. Guided image filtering. In: Computer Vision–ECCV 2010. Daniilidis, K.; Maragos, P.; Paragios, N. Eds. Springer-Verlag Berlin Heidelberg, 1–14, 2010.
Pérez, P.; Gangnet, M.; Blake, A. Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313–318, 2003.
Shahrian, E.; Rajan, D. Weighted color and texture sample selection for image matting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 718–725, 2012.
Belongie, S.; Malik, J.; Puzicha, J. Shape context: A new descriptor for shape matching and object recognition. In: Proceedings of the 13th International Conference on Neural Information Processing Systems, 798–804, 2000.
Shilkrot, R.; Cohen-Or, D.; Shamir, A.; Liu, L. Garment personalization via identity transfer. IEEE Computer Graphics and Applications Vol. 33, No. 4, 62–72, 2013.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Project No. 61521002), and Research Grant of Beijing Higher Institution Engineering Research Center. This work was finished during Zhao-Heng Zheng and Hao-Tian Zhang were undergraduate students in the Department of Computer Science and Technology at Tsinghua University.
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is published with open access at Springerlink.com
Zhao-Heng Zheng is currently a master student at University of Michigan, Ann Arbor, USA. He received his B.S. degree from Tsinghua University in 2017. His research interests include image and video processing, semantic video understanding, and computer vision.
Hao-Tian Zhang is currently a Ph.D. student at Stanford University, USA. He received his B.S. degree from Tsinghua University in 2017. His research interests include image and video editing, and physically-based simulation.
Fang-Lue Zhang is a lecturer at Victoria University of Wellington, New Zealand. He received his doctoral degree from Tsinghua University in 2015 and bachelor degree from Zhejiang University in 2009. His research interests include image and video editing, computer vision, and computer graphics. He is a member of ACM and IEEE.
Tai-Jiang Mu is currently a postdoctoral researcher in the Department of Computer Science and Technology, Tsinghua University, where he received his Ph.D. and B.S. degrees in 2016 and 2011, respectively. His research interests include computer graphics, stereoscopic image and video processing, and stereoscopic perception.
Rights and permissions
Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Zheng, ZH., Zhang, HT., Zhang, FL. et al. Image-based clothes changing system. Comp. Visual Media 3, 337–347 (2017). https://doi.org/10.1007/s41095-017-0084-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-017-0084-6