Skip to main content

From 2D to 3D geodesic-based garment matching

Multimedia Tools and Applications volume 78, pages 25829–25853 (2019)Cite this article

Abstract

A new approach for 2D to 3D garment retexturing is proposed based on Gaussian mixture models and thin plate splines (TPS). An automatically segmented garment of an individual is matched to a new source garment and rendered, resulting in augmented images in which the target garment has been retextured using the texture of the source garment. We divide the problem into garment boundary matching based on Gaussian mixture models and then interpolate inner points using surface topology extracted through geodesic paths, which leads to a more realistic result than standard approaches. We evaluated and compared our system quantitatively by root mean square error (RMS) and qualitatively using the mean opinion score (MOS), showing the benefits of the proposed methodology on our gathered dataset.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Notes

  1. The landmarks error calculations were extracted from intermediate results of VITON.

References

  1. Amberg B, Romdhani S, Vetter T (2007) Optimal step nonrigid icp algorithms for surface registration. In: 2007 IEEE conference on computer vision and pattern recognition. IEEE, pp 1–8

  2. Ament M, Dachsbacher C (2016) Anisotropic ambient volume shading. IEEE Trans Vis Comput Graph 22(1):1015–1024

    Article  Google Scholar 

  3. Avots E, Daneshmand M, Traumann A, Escalera S, Anbarjafari G (2016) Automatic garment retexturing based on infrared information. Comput Graph 59:28–38

    Article  Google Scholar 

  4. Besl NM, Paul J (1992) A method for registration of 3-d shapes. In: IEEE transactions on pattern analysis and machine intelligence. IEEE, pp 239–256

  5. Bier E, Sloan K (1986) Two-part texture mappings. IEEE Comput Graph Appl 6(9):40–53

    Article  Google Scholar 

  6. Bookstein FL (1989) Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans Pattern Anal Mach Intell 11(6):567–585

    Article  MATH  Google Scholar 

  7. Chang WC, Chang WC (2014) Real-time 3d rendering based on multiple cameras and point cloud. In: 2014 7th international conference on Ubi-Media computing and workshops. IEEE, pp 121–126

  8. Cushen G A, Nixon MS (2011) Markerless real-time garment retexturing from monocular 3d reconstruction. In: 2011 IEEE international conference on signal and image processing applications (ICSIPA). IEEE, pp 88–93

  9. Daneshmand M, Aabloo A, Ozcinar C, Anbarjafari G (2016) Real-time, automatic shape-changing robot adjustment and gender classification. SIViP 10(4):753–760

    Article  Google Scholar 

  10. Daneshmand M, Avots E, Anbarjafari G (2018) Proportional error back-propagation (peb): real-time automatic loop closure correction for maintaining global consistency in 3d reconstruction with minimal computational cost. Meas Sci Rev 18 (3):86–93

    Article  Google Scholar 

  11. Daneshmand M, Helmi A, Avots E, Noroozi F, Alisinanoglu F, Arslan H S, Gorbova J, Haamer RE, Ozcinar C, Anbarjafari G (2018) 3d scanning: a comprehensive survey. arXiv:1801.08863

  12. Deschamps T, Cohen LD (2001) Fast extraction of minimal paths in 3D images and applications to virtual endoscopy. Med Image Anal 5(4):281–299

    Article  Google Scholar 

  13. Eckstein I, Surazhsky V, Gotsman C (2001) Texture mapping with hard constrains. Comput Graph Forum 20(3):95–104

    Article  Google Scholar 

  14. Fan H, Cong Y, Tang Y (2015) Object detection based on scale-invariant partial shape matching. Mach Vis Appl 26(6):711–721

    Article  Google Scholar 

  15. Fezza SA, Larabi M-C (2015) Color calibration of multi-view video plus depth for advanced 3D video. SIViP 9(1):177–191

    Article  Google Scholar 

  16. Fu Z, Jeong W-K, Pan Y, Kirby RM, Whitaker RT (2011) A fast iterative method for solving the eikonal equation on triangulated surfaces. SIAM J Sci Comput 33(5):2468–2488

    Article  MathSciNet  MATH  Google Scholar 

  17. Gold S, Rangarajan A, Lu C-P, Pappu S, Mjolsness E (1998) New algorithms for 2D and 3D point matching: pose estimation and correspondence. Pattern Recogn 31(8):1019–1031

    Article  Google Scholar 

  18. Haili C, Anand R (2003) A new point matching algorithm for non-rigid registration. In: Computer vision and image understanding - special issue on nonrigid image registration, vol 89, issue 2–3. Elsevier Science Inc., pp 114–141

  19. Han X, Wu Z, Wu Z, Yu R, Davis L S (2018) Viton: an image-based virtual try-on network. In: CVPR

  20. Hanrahan P, Haeberli P (1990) Direct wysiwyg painting and texturing on 3d shapes. ACM SIGGRAPH Comput Graph 24(4):215–223

    Article  Google Scholar 

  21. Harvent J, Coudrin B, Brèthes L, Orteu J-J, Devy M (2013) Multi-view dense 3d modelling of untextured objects from a moving projector-cameras system. Mach Vis Appl 24(8):1645–1659

    Article  Google Scholar 

  22. Hauswiesner S, Straka M, Reitmayr G (2011) Free viewpoint virtual try-on with commodity depth cameras. In: Proceedings of the 10th international conference on virtual reality continuum and its applications in industry. ACM, pp 23–30

  23. Henry P, Krainin M, Herbst E, Ren X, Fox D (2012) Rgb-d mapping: using kinect-style depth cameras for dense 3d modeling of indoor environments. Int J Robot Res 31(5):647–663

    Article  Google Scholar 

  24. Hilsmann A, Eisert P (2009) Tracking and retexturing cloth for real-time virtual clothing applications. In: International conference on computer vision/computer graphics collaboration techniques and applications. Springer, pp 94–105

  25. Hong S, Jeong W-K (2016) A multi-gpu fast iterative method for eikonal equations using on-the-fly adaptive domain decomposition. Procedia Comput Sci 80:190–200

    Article  Google Scholar 

  26. Ichikari R, Onishi M, Kurata T (2018) Fitting simulation based on mobile body scanning for wheelchair users. In: Journal on Technology and Persons with Disabilities

  27. Jian B, Vemuri BC (2010) Robust point set registration using gaussian mixture models. In: IEEE transactions on pattern analysis and machine intelligence. IEEE, pp 1633–1645

  28. Kaewrat C, Boonbrahm P (2017) A survey for a virtual fitting room by a mixed reality technology. Walailak J Sci Technol (WJST) 14(10):759–767

    Google Scholar 

  29. Kanamori Y, Yamada H, Hirose M, Mitani J, Fukui Y (2016) Image-based virtual try-on system with garment reshaping and color correction. In: Transactions on computational science XXVI. Springer, pp 1–16

  30. Kraevoy V, Sheffer A, Gotsman C (2003) Matchmaker: constructing constrained texture maps, vol 22, no 3. ACM

  31. Liang X, Xu C, Shen X, Yang J, Liu S, Tang J, Lin L, Yan S (2015) Human parsing with contextualized convolutional neural network. In: Proceedings of the IEEE international conference on computer vision, pp 1386–1394

  32. Liu L, Zhang L, Xu Y, Gotsman C, Gortler SJ (2008) A local/global approach to mesh parameterization. Comput Graphics Forum 27(5):1495–1504. Wiley Online Library

    Article  Google Scholar 

  33. Lui LM, Lam KC, Wong TW, Gu X (2013) Texture map and video compression using beltrami representation. SIAM J Imag Sci 6(4):1880–1902

    Article  MathSciNet  MATH  Google Scholar 

  34. Ma Y, Zheng J, Xie J (2015) Foldover-free mesh warping for constrained texture mapping. IEEE Trans Vis Comput Graph 21(3):375–388

    Article  Google Scholar 

  35. Myronenko A, Song X (2010) Point set registration: coherent point drift. IEEE Trans Pattern Anal Mach Intell 32:2262–2275

    Article  Google Scholar 

  36. Nie L, Wang M, Zha Z-J, Chua T-S (2012) Oracle in image search: a content-based approach to performance prediction. ACM Trans Inf Syst (TOIS) 30(2):13

    Article  Google Scholar 

  37. Nie L, Yan S, Wang M, Hong R, Chua T-S (2012) Harvesting visual concepts for image search with complex queries. In: Proceedings of the 20th ACM international conference on multimedia. ACM, pp 59–68

  38. Ozcinar C, Ekmekcioglu E, Ćalić J, Kondoz A (2016) Adaptive delivery of immersive 3d multi-view video over the internet. Multimed Tools Appl 75(20):12431–12461

    Article  Google Scholar 

  39. Ozcinar C, Ekmekcioglu E, Anbarjafari G, Kondoz A (2019) Adaptive multi-view video streaming using side information over peer-to-peer networks. Multimed Tools Appl 76(6):7225–7242

    Article  Google Scholar 

  40. Raj A, Sangkloy P, Chang H, Hays J, Ceylan D, Lu J (2018) Swapnet: Image based garment transfer. In: European conference on computer vision. Springer, pp 679–695

  41. Rother C, Kolmogorov V, Blake A (2004) Grabcut: interactive foreground extraction using iterated graph cuts. ACM Trans Graph (TOG) 23(3):309–314

    Article  Google Scholar 

  42. Sekhavat YA (2017) Privacy preserving cloth try-on using mobile augmented reality. IEEE Trans Multimedia 19(5):1041–1049

    Article  Google Scholar 

  43. Sengupta S, Chaudhuri P (2013) Virtual Garment Simulation. In: Fourth national conference on computer vision, pattern recognition, image processing and graphics (NCVPRIPG). IEEE, pp 1–4

  44. Shirley P, Marschner S (2009) Fundamentals of computer graphics, 3rd edn. A. K. Peters, Ltd., Natick

    Book  MATH  Google Scholar 

  45. Tong J, Zhou J, Liu L, Pan Z, Yan H (2012) Scanning 3D full human bodies using kinects. IEEE Trans Vis Comput Graph 18(4):643–650

    Article  Google Scholar 

  46. Traumann A, Anbarjafari G, Escalera S (2015) A new retexturing method for virtual fitting room using kinect 2 camera. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 75–79

  47. Traumann A, Daneshmand M, Escalera S, Anbarjafari G (2015) Accurate 3d measurement using optical depth information. Electron Lett 51(18):1420–1422

    Article  Google Scholar 

  48. Turquin E, Cani MP, Hughes JF (2007) Sketching garments for virtual characters. In: ACM SIGGRAPH

  49. Wan Y, Lu D, Wang X (2018) Real-time virtual fitting technique based on kinect. In: Recent developments in intelligent computing, communication and devices

  50. Windheuser T, Schlickewei U, Schmidt F, Cremers D (2011) Geometrically consistent elastic matching of 3d shapes: a linear programming solution. In: 2011 IEEE international conference on computer vision (ICCV), pp 2134–2141

  51. Xu S, Keyser J (2014) Texture mapping for 3d painting using geodesic distance. In: 18th meeting of the ACM SIGGRAPH symposium on interactive 3D graphics and games

  52. Yang L, Zhang L, Dong H, Alelaiwi A, El Saddik A (2015) Evaluating and improving the depth accuracy of Kinect for Windows v2. IEEE Sensors J 15(8):4275–4285

    Article  Google Scholar 

  53. Yanwen G, Pan Y, Cui X, Peng Q (2005) Harmonic maps based constrained texture mapping method. J Comput Aided Des Comput Graph 7:1457–1462

    Google Scholar 

  54. Yao J, Lysandra L, Yang L, Yang B, Huang Z (2015) A virtual dressing room approach based on microsoft kinect. In: Informatics, networking and intelligent computing

  55. Yasseen Z, Nasri A, Boukaram W, Volino P, Magnenat-Thalmann N (2013) Sketch-based garment design with quad meshes. Comput Aided Des 45(2):562–567

    Article  Google Scholar 

  56. Yatziv L, Bartesaghi A, Sapiro G (2006) O (n) implementation of the fast marching algorithm. J Comput Phys 212(2):393–399

    Article  MATH  Google Scholar 

  57. Zeng W, Zeng Y, Wang Y, Yin X, Gu X, Samaras D (2008) 3d non-rigid surface matching and registration based on holomorphic differentials. In: ECCV, pp 1–14

  58. Zhang Y, Sun Z, Liu K, Zhang Y (2009) A method of 3D garment model generation using sketchy contours. In: Sixth international conference on computer graphics, imaging and visualization. IEEE, pp 205–210

  59. Zhang M, Lin L, Pan Z, Xiang N (2015) Topology-independent 3D garment fitting for virtual clothing. Multimed Tools Appl 74(9):3137–3153

    Article  Google Scholar 

  60. Zhao Y-L, Nie L, Wang X, Chua T-S (2014) Personalized recommendations of locally interesting venues to tourists via cross-region community matching. ACM Trans Intell Syst Technol (TIST) 5(3):50

    Google Scholar 

  61. Zhou F, De la Torre F (2012) Factorized graph matching. In: 2012 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 127–134

  62. Zhou Z, Shu B, Zhuo S, Deng X, Tan P, Lin S (2012) Image-based clothes animation for virtual fitting. In: SIGGRAPH Asia 2012 technical briefs. ACM, p 33

  63. Zhou B, Chen X, Fu Q, Guo K, Tan P (2013) Garment modeling from a single image. Comput Graph Forum 32(7):85–91

    Article  Google Scholar 

Download references

Acknowledgments

This work has been partially supported by Estonian Research Council Grant PUT638, Fits.Me (Rakuten) through the Research and Development Project LLTTI16056, the Scientific and Technological Research Council of Turkey (TUBITAK) Project (116E097), the Spanish Projects TIN2015-65464-R and TIN2016-74946-P (MINECO/FEDER, UE), CERCA Programme / Generalitat de Catalunya, ICREA under the ICREA Academia programme and the Estonian Centre of Excellence in IT (EXCITE) funded by the European Regional Development Fund. The authors also gratefully acknowledge the support of NVIDIA Corporation with the donation of a Titan X Pascal GPU.

Author information

Authors and Affiliations

  1. iCV Research Group, Institute of Technology, University of Tartu, Tartu, Estonia

    Egils Avots & Gholamreza Anbarjafari

  2. Computer Vision Center and Universitat Autònoma de Barcelona, Catalonia, Spain

    Meysam Madadi & Jordi Gonzàlez

  3. Computer Vision Center and University of Barcelona, Catalonia, Spain

    Sergio Escalera

  4. Computer Vision Center and Universitat Oberta de Catalunya, Catalonia, Spain

    Xavier Baro

  5. Find Fashion, Tartu, Estonia

    Paul Pällin

  6. Department of Electrical and Electronics Engineering, Hasan Kalyoncu University, Gaziantep, Turkey

    Gholamreza Anbarjafari

  7. Institute for Digital Technologies, Loughborough University, London, UK

    Gholamreza Anbarjafari

Authors
  1. Egils Avots
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meysam Madadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sergio Escalera
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jordi Gonzàlez
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xavier Baro
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Paul Pällin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Gholamreza Anbarjafari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Egils Avots.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Avots, E., Madadi, M., Escalera, S. et al. From 2D to 3D geodesic-based garment matching. Multimed Tools Appl 78, 25829–25853 (2019). https://doi.org/10.1007/s11042-019-7739-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7739-5

Keywords

  • Shape matching
  • Geodesic distance
  • Texture mapping
  • RGBD image processing
  • Gaussian mixture model