Advertisement

Multimedia Systems

, Volume 25, Issue 6, pp 593–602 | Cite as

Fashion clothes matching scheme based on Siamese Network and AutoEncoder

  • Guangyu GaoEmail author
  • Liling Liu
  • Li Wang
  • Yihang Zhang
Special Issue Paper
  • 115 Downloads

Abstract

Owing to the rise of living standard, people attach greater importance to personal appearance, especially clothes matching. With image processing and machine learning technology, we can analyze the pattern of clothes matching for recommendation on clothes images. However, we still face great challenges. To be more specific, there exist excessive complicated factors influencing relation among clothes items, such as color or material, and we also struggle against the problem about how to extract efficient and accurate features. Thus, with the purpose of dealing with such challenges, this paper proposes an efficient clothes matching scheme with Siamese Network and AutoEncoder based on both labeled data from dataset FashionVC and unlabeled data from MicroBlog. More specifically, at first, except for clothes suiting with text from FashionVC, the gallery data also include matching clothes outfits recommended by fashionista in MicroBlog (MbFashion). Meanwhile, a semi-supervised clustering based on assembling was also proposed to generate negative samples to form a comprehensive dataset. Secondly, with consideration of matching patterns from MbFashion, we promoted the Siamese Network properly to more efficiently extract vision features on the constructed training dataset. After that, the traditional features are also extracted, while the Triple AutoEncoder and Bayesian Personalized Ranking are used to map the three kinds of features into the same latent space to learn the compatibility between tops and bottoms. Finally, we conducted a series of experiments and evaluated our results to demonstrate the usefulness and effectiveness of the whole scheme on FashionVC and MbFashion.

Keywords

Clothes matching Fashion analysis Siamese Network AutoEncoder 

Notes

References

  1. 1.
    Liu, S., Liu, L., Yan, S.: Fashion analysis: current techniques and future directions. IEEE Multimed 21(2), 72–79 (2014)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8), 30–37 (2009)CrossRefGoogle Scholar
  3. 3.
    Salakhutdinov, R., Mnih, R.: Bayesian probabilistic matrix factorization using markov chain monte carlo. In: Proceedings of the 25th ACM international conference on machine learning, pp. 880–887 (2008)Google Scholar
  4. 4.
    Kalantidis, Y., Kennedy, L., Li, L.: Getting the look: clothing recognition and segmentation for automatic product suggestions in everyday photos. In: Proceedings of Int’l Conf. on Multimedia Retrieval, pp. 105–112 (2013)Google Scholar
  5. 5.
    Song, X., Feng, F., Liu, J., et al.: NeuroStylist: neural compatibility modeling for clothing matching. In: Proceedings of the ACM Int’l Conf. on Multimedia, pp. 753–761 (2017)Google Scholar
  6. 6.
    Rendle, S., Freudenthaler, C., Gantner, Z., et al.: BPR: Bayesian personalized ranking from implicit feedback. In: Proceedings of Conf. on Uncertainty in Artificial Intelligence, pp. 452–461 (2009)Google Scholar
  7. 7.
    Bromley, J., Guyon, I., Lecun, Y., Sckinger, E., Shah, R.: Signature verification using a “Siamese” time delay neural network. In: Proceedings of NIPS, pp. 737–744 (1994)Google Scholar
  8. 8.
    Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: Proceedings of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 4353–4361 (2015)Google Scholar
  9. 9.
    Yamaguchi, K., Hadi Kiapour, M., Berg, T. L.: Paper doll parsing: Retrieving similar styles to parse clothing items. In: Proceedings of IEEE Int’l Conf. on Computer Vision, pp. 3519–3526 (2013)Google Scholar
  10. 10.
    Yamaguchi, K., Kiapour, M. H., Ortiz, L. E., Berg, T. L.: Parsing clothing in fashion photographs. In: Proceedings of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 3570–3577 (2012)Google Scholar
  11. 11.
    Dong, J., Chen, Q., Shen, X., Yang, J., Yan, S.: Towards unified human parsing and pose estimation. In: Proceedings of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 843–850 (2014)Google Scholar
  12. 12.
    Kalantidis, Y., Kennedy, L., Li, L.: Getting the look: clothing recognition and segmentation for automatic product suggestions in everyday photos. In: Proceeding of Int’l Conf. on Multimedia Retrieval, pp. 105–112 (2013)Google Scholar
  13. 13.
    Liu, S., Song, Z., Liu, G., Xu, C., Lu, H., Yan, S.: Street-to-shop: cross-scenario clothing retrieval via parts alignment and auxiliary set. In: in IEEE Confer. on Computer Vision and Pattern Recognition, pp. 3330–3337 (2012)Google Scholar
  14. 14.
    Kiapour, M., Han, X., Lazebnik, S., Berg, A. C., Berg, T. L.: Where to buy it: matching street clothing photos in online shops. In: Proceedings of IEEE Int’l Conf. On Computer Vision, pp. 3343–3351 (2015)Google Scholar
  15. 15.
    Huang, C. M., Wei, C. P., Wang, Y. C. F.: Active learning based clothing image recommendation with implicit user preferences. In: Proceedings of IEEE Int’l Conf. on Multimedia and Expo Workshops, pp. 1–4 (2013)Google Scholar
  16. 16.
    Hu, Y., Yi, X., Davis, L. S.: Collaborative fashion recommendation: a functional tensor factorization approach. In: Proceedings of the 23rd ACM Int’l Conf. on Multimedia, pp. 129–138 (2015)Google Scholar
  17. 17.
    McAuley, J., Targett, C., Shi, Q., Van Den Hengel. A.: Image-based recommendations on styles and substitutes. In: Proceedings of the Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval, pp. 43–52 (2015)Google Scholar
  18. 18.
    Chen, J., Zhang, H., He, X., Nie, L., Liu, W., Chua, T.-S.: Attentive collaborative filtering: multimedia recommendation with item- and component-level attention. In: Proceeding of ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 335–344 (2017)Google Scholar
  19. 19.
    He, X., He, Z., Song, J., Liu, Z., Jiang, Y.-G., Chua, T.-S.: NAIS: neural attentive item similarity model for recommendation. IEEE Trans. Knowl. Data Eng. 30(12), 2354–2366 (2018)CrossRefGoogle Scholar
  20. 20.
    Liu, S., Feng, J., Song, Z., Zhang, T., Lu, H., Xu, C., Yan, S.: Hi, magic closet, tell me what to wear!. In: Proceedings of the ACM Int’l Conf. on Multimedia, pp. 1333–1334 (2012)Google Scholar
  21. 21.
    Vartak, M., Madden, S.: CHIC: a combination-based recommendation system. In: Proceedings of ACM SIGMOD Int’l Conf. on Management of Data, pp. 981–984 (2013)Google Scholar
  22. 22.
    Veit, A., Kovacs, B., Bell, S. et al.: Learning visual clothing style with heterogeneous dyadic co-occurrences. In: Proceedings of IEEE Int’l Conf. on Computer Vision, pp. 4642–4650 (2015)Google Scholar
  23. 23.
    Liu, S., Liu, L., Yan, S.: Fashion Analysis: current techniques and future directions. IEEE Multimed. 21(2), 72–79 (2014)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Yu, W., Zhang, H., He, X., Chen, X., Xiong, L., Qin, Z.: Aesthetic-based clothing recommendation. In: Proceedings of the 2018 World Wide Web Conference on World Wide Web, pp. 649–658 (2018)Google Scholar
  25. 25.
    Song, X., Feng, F., Han, X., Yang, X., Liu, W., Nie, L.: Neural Compatibility Modeling with Attentive Knowledge Distillation. In: Proceedings of the 41st Int’l ACM SIGIR Conf. on Research & Development in Information Retrieval, pp. 5–14 (2018)Google Scholar
  26. 26.
    He, X., He, Z., Du, X., Chua, T. S.: Adversarial personalized ranking for recommendation. In: Proceedings of The 41st Int’l ACM SIGIR Conf. on Research & Development in Information Retrieval, pp. 355–364 (2018)Google Scholar
  27. 27.
    Liu, Z., Cheng, L., Liu, A., Zhang, L., He, X., Zimmermann, R.: Multiview and multimodal pervasive indoor localization. In: Proceedings of the ACM on Multimedia Conference, pp. 109–117 (2017)Google Scholar
  28. 28.
    Chen, J., Song, X., Nie, L., Wang, X., Zhang, H., Chua, T.-S.: Micro Tells Macro: Predicting the Popularity of Micro-Videos via a Transductive Model. In: Proceedings of the ACM on Multimedia, pp. 898–907 (2016)Google Scholar
  29. 29.
    Demers, D., Cottrell, G. W.: Non-linear dimensionality reduction. In: Proceedings of Neural Information Processing Systems, pp. 580–587 (1992)Google Scholar
  30. 30.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  32. 32.
    Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7), 971–987 (2002)CrossRefGoogle Scholar
  33. 33.
    Smith, A. R.: Color gamut transformation pairs. In: Proceedings of the 5th annual Conf. on Computer Graphics and Interactive Techniques, pp. 12–19 (1978)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Guangyu Gao
    • 1
    Email author
  • Liling Liu
    • 1
  • Li Wang
    • 2
  • Yihang Zhang
    • 1
  1. 1.Beijing Institute of TechnologyBeijingChina
  2. 2.Earth Observation System and Data CenterBeijingChina

Personalised recommendations