Skip to main content

Advertisement

SpringerLink
Log in

The influence of dropout and residual connection against membership inference attacks on transformer model: a neuro generative disease case study

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Alzheimer’s patients necessitate consistent support from caregivers or family members, highlighting the urgency for advanced technologies to aid in their daily lives through early disease detection. Consequently, there has been substantial research and development of machine learning-based systems aimed at assisting Alzheimer’s patients. However, ensuring the protection of the sensitive and personal data utilized in these systems remains a critical concern. In this context, Membership Inference Attack poses a severe threat to the privacy of targeted models. This research focuses on enhancing the preservation of data privacy during the training phase. We conducted vulnerability testing on a Transformer deep-learning model against Membership Inference Attack and developed a defense strategy to mitigate its impact. To achieve this objective, we evaluated the studied attack on Transformer model using two datasets: DemCare and Oasis. These datasets contain sensitive and personal information, underscoring the need for their utmost protection. Subsequently, we proposed a defense strategy based on dropout and residual connections. Through comparative experiments, our proposed strategy demonstrated significant improvements (i.e. 20.97% and 18.43%) over the previous model, providing efficient results. Thus, we can confidently conclude that our defense approach enhances data privacy and effectively mitigates the impact of the analyzed attack.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

Notes

  1. https://colab.research.google.com/.

  2. https://www.python.org/.

  3. https://keras.io/.

  4. https://demcare.eu/datasets/.

  5. https://www.oasis-brains.org/.

References

  1. Bakkouri I, Afdel K (2020) Computer-aided diagnosis (cad) system based on multi-layer feature fusion network for skin lesion recognition in dermoscopy images. Multimed Tools Appl 79(29–30):20483–20518

    Article  Google Scholar 

  2. Bakkouri I, Afdel K (2022) Mlca2f: multi-level context attentional featurefusion for covid-19 lesion segmentation from ct scans. Signal, Image and Video Processing 1–8. https://doi.org/10.1007/s11760-022-02325-w

  3. Bentley JW, Gibney D, Hoppenworth G, Jha SK (2020) Quantifying membership inference vulnerability via generalization gap and other model metrics. arXiv preprint arXiv:2009.05669

  4. Ben amida S, Mrabet H, Belguith S, Alhomoud A, Jemai A et al (2022) Towards securing machine learning models against membership inference attacks. Comput Mater Continua 70(3):4897–4919

    Article  Google Scholar 

  5. Ben Hamida S, Mrabet H, Jemai A (2022) How differential privacy reinforces privacy of machine learning models? In Conference on Computational Collective Intelligence Technologies and Applications, pp 661–673. Springer

  6. Caruccio L, Desiato D, Polese G, Tortora G, Zannone N (2022) A decision-support framework for data anonymization with application to machine learning processes. Inf Sci 613:1–32

    Article  Google Scholar 

  7. De Gregorio G, Desiato D, Marcelli A, Polese G (2021) A multi classifier approach for supporting alzheimer’s diagnosis based on handwriting analysis. In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10–15, 2021, Proceedings, Part I, pages 559–574. Springer

  8. Diogo VS, Ferreira Hugo Alexandre, Prata Diana, and for the Alzheimer’s Disease Neuroimaging Initiative (2022) Early diagnosis of alzheimer’s disease using machine learning: a multi-diagnostic generalizable approach. Alzheimers Res Ther 14(1):1–21

    Article  Google Scholar 

  9. Dirgová Luptáková I, Kubovčík M, Pospíchal J (2022) Wearable sensor-based human activity recognition with transformer model. Sensors 22(5):1911

    Article  Google Scholar 

  10. Duca AL (2021) Adversarial Machine Learning: Attacks and Possible Defense Strategies. https://towardsdatascience.com/adversarial-machine-learning-attacks-and-possible-defense-strategies-c00eac0b395a. Accessed 2022-11-16

  11. Dwork C, Smith A, Steinke T, Ullman J (2017) Exposed! a survey of attacks on private data. Annu Rev Stat Appl 4(1):61–84

    Article  Google Scholar 

  12. Ganju K, Wang Q, Yang W, Gunter CA, Borisov N (2018) Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, pp 619–633

  13. Gong X, Wang Q, Chen Y, Yang W, Jiang X (2020) Model extraction attacks and defenses on cloud-based machine learning models. IEEE Commun Mag 58(12):83–89

    Article  Google Scholar 

  14. Guide to Auc-Roc Curve in Machine Learning : What Is Specificity? - Analytics Vidhya. https://www.analyticsvidhya.com/blog/2020/06/auc-roc-curve-machine-learning/ Accessed 2023-01-11

  15. Gupta U, Stripelis D, Lam PK, Thompson P, Ambite JL, Ver Steeg G (2021) Membership inference attacks on deep regression models for neuroimaging. In Medical Imaging with Deep Learning, pp 228–251. PMLR. https://proceedings.mlr.press/v143/gupta21a.html

  16. Hayes J, Melis L, Danezis G, De Cristofaro E (2017) Logan: Membership inference attacks against generative models. arXiv preprint arXiv:1705.07663

  17. Hendrycks D, Gimpel K (2016) Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415

  18. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  19. Homer N, Szelinger S, Redman M, Duggan D, Tembe W, Muehling J, Pearson John V, Stephan Dietrich A, Nelson Stanley F, Craig David W (2008) Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLoS Genet 4(8):e1000167

    Article  Google Scholar 

  20. Hongsheng H, Salcic Z, Sun L, Dobbie G, Yu PS, Zhang X (2022) Membership inference attacks on machine learning: A survey. ACM Comput Surv (CSUR) 54(11s):1–37

    Article  Google Scholar 

  21. Jain P, Kulkarni V, Thakurta A, Williams O (2015) To drop or not to drop: Robustness, consistency and differential privacy properties of dropout. arXiv preprint arXiv:1503.02031

  22. Jarray R, Snoun A, Bouchrika T, Jemai O (2021) Deep human action recognition system for assistance of alzheimer’s patients. In Hybrid Intelligent Systems: 20th International Conference on Hybrid Intelligent Systems (HIS 2020), December 14–16, 2020, pp 484–493. Springer

  23. Karakostas A, Briassouli A, Avgerinakis K, Kompatsiaris I, Tsolaki M (2016) The dem@ care experiments and datasets: a technical report. arXiv preprint arXiv:1701.01142

  24. Kaur P, Krishan K, Sharma SK, Kanchan T (2020) Facial-recognition algorithms: A literature review. Med Sci Law 60(2):131–139

    Article  Google Scholar 

  25. Liu Q, Li P, Zhao W, Cai W, Shui Y, Leung VCM (2018) A survey on security threats and defensive techniques of machine learning: A data driven view. IEEE Access 6:12103–12117

    Article  Google Scholar 

  26. Liu Y, Wen R, He X, Salem A, Zhang Z, Backes M, De Cristofaro E, Fritz M, Zhang Y (2022) ML-Doctor: Holistic risk assessment of inference attacks against machine learning models. In 31st USENIX Security Symposium (USENIX Security 22), pp 4525–4542

  27. Marcus DS, Fotenos AF, Csernansky JG, Morris JC, Buckner RL (2010) Open access series of imaging studies: longitudinal mri data in nondemented and demented older adults. J Cogn Neurosci 22(12):2677–2684

    Article  Google Scholar 

  28. Nasr M, Shokri R, Houmansad A (2019) Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In Proceedings of IEEE Symposium on Security and Privacy (SP), pp 739–753

  29. Praveen SP, Srinivasu PN, Shafi J, Wozniak M, Ijaz MF (2022) Resnet-32 and fastai for diagnoses of ductal carcinoma from 2d tissue slides. Sci Rep 12(1):20804

    Article  Google Scholar 

  30. Rajasekar V, Premalatha J, Sathya K, Saračević M (2021) Secure remote user authentication scheme on health care, iot and cloud applications: a multilayer systematic survey. Acta Polytech Hung 18(3):87–106

    Article  Google Scholar 

  31. Salem A, Zhang Y, Humbert M, Berrang P, Fritz M, Backes M (2018) Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246

  32. Sarker IH (2021) Deep learning: a comprehensive overview on techniques, taxonomy, applications and research directions. SN Comput Sci 2(6):420

    Article  Google Scholar 

  33. Shamshad F, Khan S, Zamir SW, Khan MH, Hayat M, Khan FS, Fu H (2023) Transformers in medical imaging: a survey. Med Image Anal 102802. https://doi.org/10.1016/j.media.2023.102802

  34. Shokri R, Stronati M, Song C, Shmatikov V (2017) Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp 3–18. IEEE

  35. Snoun A, Bouchrika T, Jemai O (2022) Deep-learning-based human activity recognition for alzheimer’s patients’ daily life activities assistance. Neural Comput Appl, pp 1–26. https://doi.org/10.1007/s00521-022-07883-1

  36. Snoun A, Bouchrika T, Jemai O (2022) A reinforcement learning and transformers based intelligent system for the support of alzheimer’s patients in daily life activities. In 14th International Conference on Computational Collective Intelligence, pp 534–547. Springer

  37. Snoun A, Bouchrika T, Jemai O (2022) View-invariant 3d skeleton-based human activity recognition based on transformer and spatio-temporal features. In ICPRAM, pp 706–715. https://doi.org/10.5220/0010895300003122

  38. Snoun A, Jlidi N, Bouchrika T, Jemai O, Zaied M (2021) Towards a deep human activity recognition approach based on video to image transformation with skeleton data. Multimed Tools Appl 80(19):29675–29698

    Article  Google Scholar 

  39. Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manage 45(4):427–437

    Article  Google Scholar 

  40. Song C, Raghunathan A (2020) Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pp 377–390

  41. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  Google Scholar 

  42. Truex S, Liu L, Gursoy ME, Lei Y, Wei W (2019) Demystifying membership inference attacks in machine learning as a service. IEEE Trans Serv Comput 14(6):2073–2089

    Article  Google Scholar 

  43. Ulhaq A, Akhtar N, Pogrebna G,Mian A (2022) Vision transformers for action recognition: A survey. arXiv preprint arXiv:2209.05700

  44. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30. https://doi.org/10.48550/arXiv.1706.03762

  45. Xiong R, Yang Y, He D, Zheng K, Zheng S, Xing C, Zhang H, Lan Y, Wang L, Liu T (2020) On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp 10524–10533. PMLR. https://proceedings.mlr.press/v119/ xiong20b.html

Download references

Author information

Author notes

  1. Sana Ben Hamida, Ahmed Snoun, Olfa Jemai and Abderrazek Jemai contributed equally to this work.

Authors and Affiliations

  1. STIC Department, Higher Institute of Technological Studies of Gabes, Route Medenine, 6011, Gabes, Tunisia

    Sameh Ben Hamida & Sana Ben Hamida

  2. Research Team in Intelligent Machines (RTIM), National Engineering School of Gabes (ENIG), University of Gabes, 6029, Gabes, Tunisia

    Sameh Ben Hamida, Sana Ben Hamida, Ahmed Snoun & Olfa Jemai

  3. UPHF, CNRS, UMR 8201 - LAMIH, F-59313, Valenciennes, France

    Ahmed Snoun

  4. SERCOM-Lab., Tunisia Polytechnic School, Carthage University, 1080, Tunis, Tunisia

    Abderrazek Jemai

  5. INSAT, Centre Urbain Nord BP 676, 1080, Tunis, Tunisia

    Abderrazek Jemai

Authors
  1. Sameh Ben Hamida
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sana Ben Hamida
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed Snoun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Olfa Jemai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Abderrazek Jemai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sameh Ben Hamida.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ben Hamida, S., Ben Hamida, S., Snoun, A. et al. The influence of dropout and residual connection against membership inference attacks on transformer model: a neuro generative disease case study. Multimed Tools Appl 83, 16231–16253 (2024). https://doi.org/10.1007/s11042-023-16126-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16126-x

Keywords

Search

Navigation