Skip to main content
SpringerLink
Log in

Outfit compatibility model using fully connected self-adjusting graph neural network

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Outfit compatibility modeling is an increasingly important task that has garnered much attention from researchers, and graph neural network (GNN)-based methods have become the mainstream approach to address this task. Despite significant progress achieved by existing research, most of them have overlooked the importance of low-order connectivity in graph data and the relationship between individual fashion items and overall outfit style. To address this issue, we propose an outfit compatibility modeling scheme based on outfit-level relationships. This scheme consists of two key components: FCSA-GNN and GOR. The former consists of multiple layers of Fashion Items Relationship Propagation, where the weights of the graph edges are adapted based on the Category Co-occurrence Matrix. This approach avoids potential imbalances in the quantities of different fashion item categories within the dataset. Subsequently, by connecting the outputs of GNN at each layer, it explores the relationships between fashion item visual signals at different levels. The latter integrates the overall outfits style and further enhances the model’s ability to learn a comprehensive representation of the overall outfits through the proposed outfit-level relationships. Experimental results on two real datasets, Polyvore outfit-ND and Polyvore outfit-D, demonstrate that our method outperforms existing state-of-the-art methods when considering only visual information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The data can be found from https://github.com/thethati/FCSA-NET.

References

  1. Saranya, M., Geetha, P.: A deep learning-based feature extraction of cloth data using modified grab cut segmentation. Visual Comput. 39, 4195–4211 (2022)

    Article  Google Scholar 

  2. Liu, W., Liu, Q., Tang, R., Chen, J., He, X., Heng, P.A.: Personalized re-ranking with item relationships for e-commerce. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 925–934 (2020)

  3. Sarkar, R., Bodla, N., Vasileva, M., Lin, Y.-L., Beniwal, A., Lu, A., Medioni, G.: Outfittransformer: Outfit representations for fashion recommendation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2263–2267 (2022)

  4. Amin, M.S., Wang, C., Jabeen, S.: Fashion sub-categories and attributes prediction model using deep learning. Visual Comput., 1–14 (2022)

  5. Wang, X., He, X., Wang, M., Feng, F., Chua, T.-S.: Neural graph collaborative filtering. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 165–174 (2019)

  6. Li, Q., Han, Z., Wu, X.-M.: Deeper insights into graph convolutional networks for semi-supervised learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

  7. Wijesinghe, A., Wang, Q.: A new perspective on" how graph neural networks go beyond weisfeiler-lehman?". In: International Conference on Learning Representations (2022)

  8. Su, T., Song, X., Zheng, N., Guan, W., Li, Y., Nie, L.: Complementary factorization towards outfit compatibility modeling. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4073–4081 (2021)

  9. Shajini, M., Ramanan, A.: A knowledge-sharing semi-supervised approach for fashion clothes classification and attribute prediction. Visual Comput. 38(11), 3551–3561 (2022)

    Article  Google Scholar 

  10. Song, X., Feng, F., Liu, J., Li, Z., Nie, L., Ma, J.: Neurostylist: Neural compatibility modeling for clothing matching. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 753–761 (2017)

  11. McAuley, J., Targett, C., Shi, Q., Van Den Hengel, A.: Image-based recommendations on styles and substitutes. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 43–52 (2015)

  12. Song, X., Han, X., Li, Y., Chen, J., Xu, X.-S., Nie, L.: Gp-bpr: Personalized compatibility modeling for clothing matching. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 320–328 (2019)

  13. Han, X., Song, X., Yin, J., Wang, Y., Nie, L.: Prototype-guided attribute-wise interpretable scheme for clothing matching. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 785–794 (2019)

  14. Kaicheng, P., Xingxing, Z., Wong, W.K.: Modeling fashion compatibility with explanation by using bidirectional lstm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3894–3898 (2021)

  15. Dong, X., Song, X., Zheng, N., Wu, J., Dai, H., Nie, L.: Tryoncm2: try-on-enhanced fashion compatibility modeling framework. IEEE Transact. Neural Netw. Learn. Syst. (2022)

  16. Wang, H., Zeng, Y., Chen, J., Zhao, Z., Chen, H.: A spatiotemporal graph neural network for session-based recommendation. Expert Syst. Appl. 202, 117114 (2022)

    Article  Google Scholar 

  17. Lin, Y.-L., Tran, S., Davis, L.S.: Fashion outfit complementary item retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3311–3319 (2020)

  18. Yang, X., Xie, D., Wang, X., Yuan, J., Ding, W., Yan, P.: Learning tuple compatibility for conditional outfit recommendation. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2636–2644 (2020)

  19. Jing, P., Cui, K., Guan, W., Nie, L., Su, Y.: Category-aware multimodal attention network for fashion compatibility modeling. IEEE Transact. Multimed. (2023)

  20. Zhan, H., Lin, J., Ak, K.E., Shi, B., Duan, L.-Y., Kot, A.C.: \(a^{3}\)-fkg: attentive attribute-aware fashion knowledge graph for outfit preference prediction. IEEE Transact. Multimed. 24, 819–831 (2021)

    Article  Google Scholar 

  21. Song, X., Fang, S.-T., Chen, X., Wei, Y., Zhao, Z., Nie, L.: Modality-oriented graph learning toward outfit compatibility modeling. IEEE Transact. Multimed. (2021)

  22. Liu, X., Sun, Y., Liu, Z., Lin, D.: Learning diverse fashion collocation by neural graph filtering. IEEE Transact. Multimed. 23, 2894–2901 (2020)

    Article  Google Scholar 

  23. Li, X., Wang, X., He, X., Chen, L., Xiao, J., Chua, T.-S.: Hierarchical fashion graph network for personalized outfit recommendation. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 159–168 (2020)

  24. Li, Z., Li, J., Wang, T., Gong, X., Wei, Y., Luo, P.: Ocphn: outfit compatibility prediction with hypergraph networks. Mathematics 10(20), 3913 (2022)

    Article  Google Scholar 

  25. Deldjoo, Y., Nazary, F., Ramisa, A., Mcauley, J., Pellegrini, G., Bellogin, A., Di Noia, T.: A review of modern fashion recommender systems. arXiv preprint arXiv:2202.02757 (2022)

  26. Cucurull, G., Taslakian, P., Vazquez, D.: Context-aware visual compatibility prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12617–12626 (2019)

  27. Zhang, B., Sheng, B., Li, P., Lee, T.-Y.: Depth of field rendering using multilayer-neighborhood optimization. IEEE Transact. Vis. Comput. Graphics 26(8), 2546–2559 (2019)

    Article  Google Scholar 

  28. Guan, W., Wen, H., Song, X., Yeh, C.-H., Chang, X., Nie, L.: Multimodal compatibility modeling via exploring the consistent and complementary correlations. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 2299–2307 (2021)

  29. Zhang, J., Xiao, X., Huang, L.-K., Rong, Y., Bian, Y.: Fine-tuning graph neural networks via graph topology induced optimal transport. arXiv preprint arXiv:2203.10453 (2022)

  30. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  31. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Adv. Neural Inform. Process. Syst. 30 (2017)

  33. Han, X., Wu, Z., Jiang, Y.-G., Davis, L.S.: Learning fashion compatibility with bidirectional lstms. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 1078–1086 (2017)

  34. Tan, R., Vasileva, M.I., Saenko, K., Plummer, B.A.: Learning similarity conditions without explicit supervision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10373–10382 (2019)

  35. Vasileva, M.I., Plummer, B.A., Dusad, K., Rajpal, S., Kumar, R., Forsyth, D.: Learning type-aware embeddings for fashion compatibility. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 390–405 (2018)

  36. Lin, Y.-L., Tran, S., Davis, L.S.: Fashion outfit complementary item retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3311–3319 (2020)

  37. Xiao, L., Yamasaki, T.: Sat: Self-adaptive training for fashion compatibility prediction. In: 2022 IEEE International Conference on Image Processing (ICIP), pp. 2431–2435 (2022). IEEE

  38. Cui, Z., Li, Z., Wu, S., Zhang, X.-Y., Wang, L.: Dressing as a whole: Outfit compatibility learning based on node-wise graph neural networks. In: The World Wide Web Conference, pp. 307–317 (2019)

  39. Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. Adv. Neural Inform. Process. Syst. 29 (2016)

  40. Brody, S., Alon, U., Yahav, E.: How attentive are graph attention networks? arXiv preprint arXiv:2105.14491 (2021)

Download references

Funding

This paper is supported by the National Natural Science Foundation of China with No.61901308.

Author information

Author notes

  1. Neng Yu, Kai Ma, Tao Peng and Xinrong Hu have contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Science, Wuhan Textile University, No. 1 Sunshine Avenue, Miaoshan Development Zone, Jiangxia District, Wuhan, 430200, Hubei, China

    Hong Liu, Tao Peng & Xinrong Hu

  2. Department of Artificial Intelligence, Wuhan Textile University, No. 1 Sunshine Avenue, Miaoshan Development Zone, Jiangxia District, Wuhan, 430200, Hubei, China

    Li Li, Neng Yu, Kai Ma & Xinrong Hu

Authors
  1. Hong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Neng Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kai Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tao Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xinrong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Li.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, H., Li, L., Yu, N. et al. Outfit compatibility model using fully connected self-adjusting graph neural network. Vis Comput (2024). https://doi.org/10.1007/s00371-023-03238-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-023-03238-6

Keywords

Search

Navigation