Abstract
Pre-trained CNN models are frequently employed for a variety of machine learning tasks, including visual recognition and recommendation. We are interested in examining the application of attacks generated by adversarial machine learning techniques to the vertical domain of fashion and retail products. Specifically, the present work focuses on the robustness of cutting-edge CNN models against state-of-the-art adversarial machine learning attacks that have shown promising performance in general visual classification tasks. In order to achieve this objective, we conducted adversarial experiments on two prominent fashion-related tasks: visual clothing classification and outfit recommendation. Large-scale experimental validation of the fashion category classification task on a real dataset of PolyVore consisting of various outfits reveals that ResNet50 is one of the most resilient networks for the fashion categorization task, whereas DenseNet169 and MobileNetV2 are the most vulnerable. Performance-wise however, DenseNet169 is the most time-consuming network to attack. However, the results of the outfit recommendation task were somewhat unexpected. In both of the push or nuke attack scenarios and altogether, it was demonstrated that adversarial attacks were unable to degrade the quality of outfit recommenders. The only exception was the more complicated adversarial attack of DeepFool, which could only weaken the quality of visual recommenders at large attack budget (\(\epsilon \)) values. Numerous explanations could be provided for this phenomenon, which can be attributed to the fact that a collection of adversarially perturbed images can nonetheless appear pleasing to the human eye. This may possibly be a result of the greater image sizes in the selected dataset. Overall, the results of this study are intriguing and encourage more studies in the field of adversarial attacks and fashion recommendation system security.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anelli VW, Bellogin A, Deldjoo Y, Di Noia T, Merra FA (2021) Msap: multi-step adversarial perturbations on recommender systems embeddings. In: The 34th international FLAIRS conference. The Florida AI Research Society (FLAIRS), AAAI Press, pp 1–6
Anelli VW, Deldjoo Y, Noia TD, Malitesta D, Merra FA (2021) A study of defensive methods to protect visual recommendation against adversarial manipulation of images. In: Diaz F, Shah C, Suel T, Castells P, Jones R, Sakai T (eds) SIGIR ’21: the 44th international ACM SIGIR conference on research and development in information retrieval, virtual Event, Canada, 11–15 July 2021. ACM, pp 1094–1103. https://doi.org/10.1145/3404835.3462848
Anelli VW, Di Noia T, Di Sciascio E, Malitesta D, Merra FA (2021c) Adversarial attacks against visual recommendation: an investigation on the influence of items’ popularity. In: Proceedings of the 2nd workshop on online misinformation-and harm-aware recommender systems (OHARS 2021), Amsterdam, Netherlands
Anelli VW, Deldjoo Y, DiNoia T, Merra FA (2022) Adversarial recommender systems: attack, defense, and advances. In: Recommender systems handbook. Springer, pp 335–379
Biggio B, Corona I, Maiorca D, Nelson B, Srndic N, Laskov P, Giacinto G, Roli F (2017) Evasion attacks against machine learning at test time. CoRR abs/1708.06131, http://arxiv.org/abs/1708.06131, eprint1708.06131
Cheng W, Song S, Chen C, Hidayati SC, Liu J (2021) Fashion meets computer vision: a survey. ACM Comput Surv 54(4):72:1–72:41. https://doi.org/10.1145/3447239
Deldjoo Y, Di Noia T, Merra FA (2019) Assessing the impact of a user-item collaborative attack on class of users. In: ImpactRS@RecSys’19 workshop on the impact of recommender systems
Deldjoo Y, Schedl M, Cremonesi P, Pasi G (2020) Recommender systems leveraging multimedia content. ACM Comput Surv (CSUR) 53(5):1–38
Deldjoo Y, Noia TD, Malitesta D, Merra FA (2021) A study on the relative importance of convolutional neural networks in visually-aware recommender systems. In: IEEE conference on computer vision and pattern recognition workshops, CVPR Workshops 2021, virtual, 19–25 June 2021. Computer Vision Foundation/IEEE, pp 3961–3967. https://doi.org/10.1109/CVPRW53098.2021.00445. https://openaccess.thecvf.com/content/CVPR2021W/CVFAD/html/Deldjoo_A_Study_on_the_Relative_Importance_of_Convolutional_Neural_Networks_CVPRW_2021_paper.html
Deldjoo Y, Noia TD, Merra FA (2021) A survey on adversarial recommender systems: from attack/defense strategies to generative adversarial networks. ACM Comput Surv 54(2):35:1–35:38. https://doi.org/10.1145/3439729
Deldjoo Y, Nazary F, Ramisa A, McAuley J, Pellegrini G, BellogÃn A, Noia TD (2023) A review of modern fashion recommender systems. ACM Comput Surv
Deldjoo Y, Schedl M, Hidasi B, Wei Y, He X (2022) Multimedia recommender systems: algorithms and challenges. In: Recommender systems handbook. Springer, pp 973–1014
Deng J, Dong W, Socher R, Li L, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Computer Society conference on computer vision and pattern recognition (CVPR 2009), 20–25 June 2009, Miami, FL, USA. IEEE Computer Society, pp 248–255. https://doi.org/10.1109/CVPR.2009.5206848
Goodfellow IJ, Shlens J, Szegedy C (2015) Explaining and harnessing adversarial examples. In: Bengio Y, LeCun Y (eds) 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings. http://arxiv.org/abs/1412.6572
Han X, Wu Z, Jiang Y, Davis LS (2017) Learning fashion compatibility with bidirectional LSTMs. In: Liu Q, Lienhart R, Wang H, Chen SK, Boll S, Chen YP, Friedland G, Li J, Yan S (eds) Proceedings of the 2017 ACM on multimedia conference, MM 2017, Mountain View, CA, USA, 23–27 October 2017. ACM, pp 1078–1086. https://doi.org/10.1145/3123266.3123394
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016. IEEE Computer Society, pp 770–778. https://doi.org/10.1109/CVPR.2016.90
Huang G, Liu Z, Weinberger KQ (2016) Densely connected convolutional networks. CoRR abs/1608.06993, http://arxiv.org/abs/1608.06993, eprint1608.06993
Kurakin A, Goodfellow IJ, Bengio S (2017) Adversarial examples in the physical world. In: 5th international conference on learning representations, ICLR 2017, Toulon, France, 24–26 April 2017, Workshop track proceedings, OpenReview.net. https://openreview.net/forum?id=HJGU3Rodl
Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A (2018) Towards deep learning models resistant to adversarial attacks. In: 6th international conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, Conference Track Proceedings, OpenReview.net. https://openreview.net/forum?id=rJzIBfZAb
McAuley J, Targett C, Shi Q, Van Den Hengel A (2015) Image-based recommendations on styles and substitutes. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp 43–52
Moosavi-Dezfooli S, Fawzi A, Frossard P (2016) Deepfool: a simple and accurate method to fool deep neural networks. In: 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016. IEEE Computer Society, pp 2574–2582. https://doi.org/10.1109/CVPR.2016.282
Nicolae MI, Sinn M, Tran MN, Buesser B, Rawat A, Wistuba M, Zantedeschi V, Baracaldo N, Chen B, Ludwig H et al (2018) Adversarial robustness toolbox v1. 0.0. arXiv preprint arXiv:1807.01069
Noia TD, Malitesta D, Merra FA (2020) TAaMR: targeted adversarial attack against multimedia recommender systems. In: DSN workshops. IEEE, pp 1–8
Pan T, Dai Y, Tsai W, Hu M (2017) Deep model style: cross-class style compatibility for 3D furniture within a scene. In: Nie J, Obradovic Z, Suzumura T, Ghosh R, Nambiar R, Wang C, Zang H, Baeza-Yates R, Hu X, Kepner J, Cuzzocrea A, Tang J, Toyoda M (eds) 2017 IEEE international conference on big bata (IEEE BigData 2017), Boston, MA, USA, 11–14 Dec 2017. IEEE Computer Society, pp 4307–4313. https://doi.org/10.1109/BigData.2017.8258459
Pillai RS, Sreekumar K (2020) Classification of fashion images using transfer learning. In: Bhateja V, Peng S, Satapathy SC, Zhang Y (eds) Evolution in computational intelligence—frontiers in intelligent computing: theory and applications (FICTA 2020), vol 1, Karnataka, Surathkal, India, 4–5 Jan 2020. Advances in intelligent systems and computing, vol 1176. Springer, pp 325–332. https://doi.org/10.1007/978-981-15-5788-0_32
PolanÃa LF, Gupte S (2019) Learning fashion compatibility across apparel categories for outfit recommendation. In: 2019 IEEE international conference on image processing, ICIP 2019, Taipei, Taiwan, 22–25 Sept 2019. IEEE, pp 4489–4493. https://doi.org/10.1109/ICIP.2019.8803587
Sandler M, Howard AG, Zhu M, Zhmoginov A, Chen L (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: 2018 IEEE conference on computer vision and pattern recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018. Computer Vision Foundation/IEEE Computer Society, pp 4510–4520. https://doi.org/10.1109/CVPR.2018.00474. http://openaccess.thecvf.com/content_cvpr_2018/html/Sandler_MobileNetV2_Inverted_Residuals_CVPR_2018_paper.html
Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Bengio Y, LeCun Y (eds) 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015. Conference track proceedings. http://arxiv.org/abs/1409.1556
Sun G, Cheng Z, Wu X, Peng Q (2018) Personalized clothing recommendation combining user social circle and fashion style consistency. Multim Tools Appl 77(14):17731–17754
Veit A, Kovacs B, Bell S, McAuley J, Bala K, Belongie SJ (2015) Learning visual clothing style with heterogeneous dyadic co-occurrences. In: 2015 IEEE international conference on computer vision, ICCV 2015, Santiago, Chile, 7–13 Dec 2015. IEEE Computer Society, pp 4642–4650. https://doi.org/10.1109/ICCV.2015.527
Xu H, Ma Y, Liu H, Deb D, Liu H, Tang J, Jain AK (2020) Adversarial attacks and defenses in images, graphs and text: a review. Int J Autom Comput 17(2):151–178
Yin R, Li K, Lu J, Zhang G (2019) Enhancing fashion recommendation with visual compatibility relationship. In: Liu L, White RW, Mantrach A, Silvestri F, McAuley J, Baeza-Yates R, Zia L (eds) The world wide web conference, WWW 2019, San Francisco, CA, USA, 13–17 May 2019. ACM, pp 3434–3440. https://doi.org/10.1145/3308558.3313739
Zhao K, Hu X, Bu J, Wang C (2017) Deep style match for complementary recommendation WS-17. http://aaai.org/ocs/index.php/WS/AAAIW17/paper/view/15069
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Attimonelli, M., Amatulli, G., Gioia, L.D., Malitesta, D., Deldjoo, Y., Noia, T.D. (2023). Adversarial Attacks Against Visually Aware Fashion Outfit Recommender Systems. In: Corona PampÃn, H.J., Shirvany, R. (eds) Recommender Systems in Fashion and Retail. RECSYS 2022. Lecture Notes in Electrical Engineering, vol 981. Springer, Cham. https://doi.org/10.1007/978-3-031-22192-7_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-22192-7_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22191-0
Online ISBN: 978-3-031-22192-7
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)