Skip to main content
SpringerLink
Log in

Do gradient-based explanations tell anything about adversarial robustness to android malware?

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

While machine-learning algorithms have demonstrated a strong ability in detecting Android malware, they can be evaded by sparse evasion attacks crafted by injecting a small set of fake components, e.g., permissions and system calls, without compromising intrusive functionality. Previous work has shown that, to improve robustness against such attacks, learning algorithms should avoid overemphasizing few discriminant features, providing instead decisions that rely upon a large subset of components. In this work, we investigate whether gradient-based attribution methods, used to explain classifiers’ decisions by identifying the most relevant features, can be used to help identify and select more robust algorithms. To this end, we propose to exploit two different metrics that represent the evenness of explanations, and a new compact security measure called Adversarial Robustness Metric. Our experiments conducted on two different datasets and five classification algorithms for Android malware detection show that a strong connection exists between the uniformity of explanations and adversarial robustness. In particular, we found that popular techniques like Gradient*Input and Integrated Gradients are strongly correlated to security when applied to both linear and nonlinear detectors, while more elementary explanation techniques like the simple Gradient do not provide reliable information about the robustness of such classifiers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability statement

The datasets generated during and/or analysed during the current study are available in the Androzoo repository, https://androzoo.uni.lu/, and upon request at https://www.sec.tu-bs.de/~danarp/drebin/.

Notes

  1. MD5: f8bcbd48f44ce973036fac0bce68a5d5.

  2. MD5: eb1f454ea622a8d2713918b590241a7e.

References

  1. Aafer Y, Du W, Yin H (2013) DroidAPIMiner: mining API-level features for robust malware detection in android. In: Proc. of international conference on security and privacy in communication networks (SecureComm). https://doi.org/10.1007/978-3-319-04283-1_6

  2. Adadi A, Berrada M (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE Access 6:52138–52160

    Article  Google Scholar 

  3. Allix K, Bissyandé TF, Klein J, Le Traon Y (2016) Androzoo: collecting millions of android apps for the research community. In: 2016 IEEE/ACM 13th working conference on mining software repositories (MSR), pp 468–471, IEEE

  4. Arp D, Spreitzenbarth M, Hübner M, Gascon H, Rieck K (2014) Drebin: efficient and explainable detection of android malware in your pocket. In: Proc. 21st annual network & distributed system security symposium (NDSS). The Internet Society

  5. Arzt S, Rasthofer S, Fritz C, Bodden E, Bartel A, Klein J, Le Traon Y, Octeau D, McDaniel P (2013) FlowDroid: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps. In: Proceedings of the 35th ACM SIGPLAN conference on programming language design and implementation—PLDI ’14, pp 259–269. ACM Press. https://doi.org/10.1145/2594291.2594299, http://dl.acm.org/citation.cfm?doid=2594291.2594299

  6. Baehrens D, Schroeter T, Harmeling S, Kawanabe M, Hansen K, Müller KR (2010) How to explain individual classification decisions. J Mach Learn Res 11:1803–1831

    MathSciNet  MATH  Google Scholar 

  7. Barreno M, Nelson B, Joseph A, Tygar J (2010) The security of machine learning. Mach Learn 81:121–148

    Article  MathSciNet  Google Scholar 

  8. Barreno M, Nelson B, Sears R, Joseph AD, Tygar JD (2006) Can machine learning be secure? In: Proc. ACM Symp. information, computer and comm. Sec., ASIACCS ’06, pp 16–25. ACM, New York

  9. Biggio B, Corona I, Maiorca D, Nelson B, Šrndić N, Laskov P, Giacinto G, Roli F (2013) Evasion attacks against machine learning at test time. In: Blockeel H, Kersting K, Nijssen S, Železný F (eds) Machine learning and knowledge discovery in databases (ECML PKDD), Part III, LNCS, vol 8190. Springer, Berlin, Heidelberg, pp 387–402

    Google Scholar 

  10. Biggio B, Fumera G, Roli F (2010) Multiple classifier systems for robust classifier design in adversarial environments. Int J Mach Learn Cybern 1(1):27–41

    Article  Google Scholar 

  11. Biggio B, Fumera G, Roli F (2014) Security evaluation of pattern classifiers under attack. IEEE Trans Knowl Data Eng 26(4):984–996

    Article  Google Scholar 

  12. Biggio B, Nelson B, Laskov P (2012) Poisoning attacks against support vector machines. In: Langford J, Pineau J (eds) 29th Int’l Conf. on Machine Learning, pp 1807–1814, Omnipress

  13. Biggio B, Roli F (2018) Wild patterns: ten years after the rise of adversarial machine learning. Pattern Recogn 84:317–331

    Article  Google Scholar 

  14. Cai H, Meng N, Ryder B, Yao D (2018) Droidcat: effective android malware detection and categorization via app-level profiling. IEEE Trans Inf Forensics Secur 14(6):1455–1470

    Article  Google Scholar 

  15. Calleja A, Martin A, Menendez HD, Tapiador J, Clark D (2018) Picking on the family: disrupting android malware triage by forcing misclassification. Expert Syst Appl 95:113–126

    Article  Google Scholar 

  16. Cara F, Scalas M, Giacinto G, Maiorca D (2020) On the feasibility of adversarial sample creation using the android system api. Information 11(9):433

    Article  Google Scholar 

  17. Chen J, Wang C, Zhao Z, Chen K, Du R, Ahn GJ (2018) Uncovering the Face of Android Ransomware: characterization and real-time detection. IEEE Trans Inf Forensics Secur 13(5):1286–1300. https://doi.org/10.1109/TIFS.2017.2787905, http://ieeexplore.ieee.org/document/8241433/

  18. Chen J, Wu X, Rastogi V, Liang Y, Jha S (2019) Robust attribution regularization. Adv Neural Inf Process Syst 2019:14300–14310

  19. Chen L, Hou S, Ye Y, Xu S (2018) Droideye: fortifying security of learning-based classifier against adversarial android malware attacks. In: Proceedings of the 2018 IEEE/ACM international conference on advances in social networks analysis and mining, ASONAM 2018, pp. 782–789. Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ASONAM.2018.8508284

  20. Chen S, Xue M, Tang Z, Xu L, Zhu H (2016) Stormdroid: a streaminglized machine learning-based system for detecting android malware. In: Proceedings of the 11th ACM on Asia conference on computer and communications security, pp 377–388

  21. Chen YM, Yang CH, Chen GC (2021) Using generative adversarial networks for data augmentation in android malware detection. In: 2021 IEEE conference on dependable and secure computing (DSC), pp 1–8, IEEE. https://doi.org/10.1109/DSC49826.2021.9346277, https://ieeexplore.ieee.org/document/9346277/

  22. Dalvi N, Domingos P, Mausam G, Sanghai S, Verma D (2004) Adversarial classification. In: Tenth ACM SIGKDD international conference on knowledge discovery and data mining (KDD), pp 99–108. Seattle

  23. Demontis A, Melis M, Biggio B, Maiorca D, Arp D, Rieck K, Corona I, Giacinto G, Roli F (2017) Yes, machine learning can be more secure! a case study on android malware detection. In: IEEE transactions on dependable and secure computing, pp 1–1. https://doi.org/10.1109/TDSC.2017.2700270

  24. Demontis A, Melis, M., Pintor M, Jagielski M, Biggio B, Oprea A, Nita-Rotaru C, Roli F (2019) Why do adversarial attacks transfer? Explaining transferability of evasion and poisoning attacks. In: 28th USENIX Security Symposium (USENIX Security 19), pp 321–338. USENIX Association, Santa Clara

  25. Demontis A, Russu P, Biggio B, Fumera G, Roli F (2016) On security and sparsity of linear classifiers for adversarial settings. In: Robles-Kelly A, Loog M, Biggio B, Escolano F, Wilson R (eds) Joint IAPR Int’l workshop on structural, syntactic, and statistical pattern recognition, LNCS, vol 10029. Springer International Publishing, Cham, pp 322–332

    Chapter  Google Scholar 

  26. Dombrowski AK, Alber M, Anders CJ, Ackermann M, Müller KR, Kessel P (2019) Explanations can be manipulated and geometry is to blame. arXiv:1906.07983

  27. Feng Y, Anand S, Dillig I, Aiken A (2014) Apposcopy: semantics-based detection of Android malware through static analysis. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering—FSE 2014, pp 576–587. ACM Press. https://doi.org/10.1145/2635868.2635869, http://dl.acm.org/citation.cfm?doid=2635868.2635869

  28. Fidel G, Bitton R, Shabtai A (2020) When explainability meets adversarial learning: Detecting adversarial examples using shap signatures. In: 2020 international joint conference on neural networks (IJCNN), pp 1–8, IEEE

  29. Goodfellow IJ, Shlens J, Szegedy C (2015) Explaining and harnessing adversarial examples. In: International conference on learning representations

  30. Goodman B, Flaxman S (2016) European Union regulations on algorithmic decision-making and a “right to explanation”. In: AI magazine, vol 38, pp 50–57

  31. Grosse K, Papernot N, Manoharan P, Backes M, McDaniel PD (2017) Adversarial examples for malware detection. In: ESORICS (2), LNCS, vol 10493, pp 62–79. Springer

  32. Guo W, Mu D, Xu J, Su P, Wang G, Xing X (2018) Lemna: explaining deep learning based security applications. In: Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, pp 364–379

  33. Hijawi W, Alqatawna J, Al-Zoubi AM, Hassonah MA, Faris H (2021) Android botnet detection using machine learning models based on a comprehensive static analysis approach. J Inf Secur Appl 58:102735. https://doi.org/10.1016/j.jisa.2020.102735, https://linkinghub.elsevier.com/retrieve/pii/S2214212620308711

  34. Kim B, Wattenberg M, Gilmer J, Cai C, Wexler J, Viegas F, Sayres R (2018) Interpretability beyond feature attribution: quantitative testing with concept activation vectors (TCAV). In: 35th international conference on machine learning (ICML 2018), vol 80, pp 2668–2677, Stockholm

  35. Koh PW, Liang P (2017) Understanding black-box predictions via influence functions. In: International conference on machine learning (ICML)

  36. Koh PW, Nguyen T, Tang YS, Mussmann S, Pierson E, Kim B, Liang P (2020) Concept bottleneck models. In: III HD, Singh A (eds) Proceedings of the 37th international conference on machine learning, Proceedings of Machine Learning Research, vol 119, pp 5338–5348, PMLR. http://proceedings.mlr.press/v119/koh20a.html

  37. Kolcz A, Teo CH (2009) Feature weighting for improved classifier robustness. In: Sixth conference on email and anti-spam (CEAS). Mountain View

  38. Li Q, Hu Q, Qi Y, Qi S, Liu X, Gao P. (2021)Semi-supervised two-phase familial analysis of Android malware with normalized graph embedding. Knowl Based Syst 218:106802. https://doi.org/10.1016/j.knosys.2021.106802, https://linkinghub.elsevier.com/retrieve/pii/S0950705121000654

  39. Lindorfer M, Neugschwandtner M, Platzer C (2015) Marvin: efficient and comprehensive mobile app classification through static and dynamic analysis. In: Proceedings of the 39th annual international computers, software & applications conference (COMPSAC)

  40. Lindorfer M, Neugschwandtner M, Platzer C (2015) MARVIN: efficient and comprehensive mobile app classification through static and dynamic analysis. In: 2015 IEEE 39th annual computer software and applications conference, vol 2, pp 422–433

  41. Lowd D, Meek C (2005) Adversarial learning. In: Proc. 11th ACM sigkdd international conference on knowledge discovery and data mining (KDD), pp 641–647. ACM Press, Chicago

  42. Lundberg SM, Erion G, Chen H, DeGrave A, Prutkin JM, Nair B, Katz R, Himmelfarb J, Bansal N, Lee SI (2020) From local explanations to global understanding with explainable AI for trees. Nature Mach Intell 2(1): 56–67. https://doi.org/10.1038/s42256-019-0138-9, http://www.nature.com/articles/s42256-019-0138-9

  43. Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. In: Advances in neural information processing systems, pp 4765–4774

  44. Mahindru A, Sangal AL (2021) MLDroid-framework for Android malware detection using machine learning techniques. Neural Comput Appl 33(10):5183–5240. https://doi.org/10.1007/s00521-020-05309-4

    Article  Google Scholar 

  45. Mahindru A, Sangal AL (2021) SemiDroid: a behavioral malware detector based on unsupervised machine learning techniques using feature selection approaches. Int J Mach Learn Cybern 12(5):1369–1411. https://doi.org/10.1007/s13042-020-01238-9

    Article  Google Scholar 

  46. Maiorca D, Biggio B, Giacinto G (2019) Towards adversarial malware detection: lessons learned from pdf-based attacks. ACM Comput Surv (CSUR) 52(4):1–36

    Article  Google Scholar 

  47. Maiorca D, Mercaldo F, Giacinto G, Visaggio CA, Martinelli F (2017) R-packdroid: Api package-based characterization and detection of mobile ransomware. In: Proceedings of the symposium on applied computing, SAC ’17, pp 1718–1723. ACM, New York. https://doi.org/10.1145/3019612.3019793

  48. Mariconti E, Onwuzurike L, Andriotis P, Cristofaro ED, Ross GJ, Stringhini G (2017) Mamadroid: Detecting android malware by building markov chains of behavioral models. In: NDSS. The Internet Society

  49. Melis M, Demontis A, Biggio B, Brown G, Fumera G, Roli F (2017) Is deep learning safe for robot vision? Adversarial examples against the icub humanoid. In: ICCV workshop on vision in practice on autonomous robots (ViPAR)

  50. Melis M, Demontis A, Pintor M, Sotgiu A, Biggio B (2019) secml: a python library for secure and explainable machine learning. arXiv:1912.10013

  51. Melis M, Maiorca D, Biggio B, Giacinto G, Roli F (2018) Explaining black-box android malware detection. In: 2018 26th european signal processing conference (EUSIPCO), pp 524–528, IEEE

  52. Pendlebury F, Pierazzi F, Jordaney R, Kinder J, Cavallaro L (2019) \(\{\)TESSERACT\(\}\): Eliminating experimental bias in malware classification across space and time. In: 28th \(\{\)USENIX\(\}\) Security Symposium (\(\{\)USENIX\(\}\) Security 19), pp 729–746

  53. Peng H, Gates C, Sarma B, Li N, Qi Y, Potharaju R, Nita-Rotaru C, Molloy I (2012) Using probabilistic generative models for ranking risks of android apps. In: Proceedings of the 2012 ACM conference on computer and communications security

  54. Pierazzi F, Pendlebury F, Cortellazzi J, Cavallaro L (2020) Intriguing properties of adversarial ml attacks in the problem space. In: 2020 IEEE symposium on security and privacy (SP), pp 1332–1349, IEEE

  55. Ribeiro MT, Singh S, Guestrin C (2016) “why should i trust you?”: explaining the predictions of any classifier. In: 22nd ACM SIGKDD Int’l Conf. Knowl. Disc. Data Mining, KDD ’16, pp 1135–1144. ACM, New York

  56. Rosenberg I, Meir S, Berrebi J, Gordon I, Sicard G, David EO (2020) Generating end-to-end adversarial examples for malware classifiers using explainability. In: 2020 international joint conference on neural networks (IJCNN), pp 1–10, IEEE

  57. Scalas M, Maiorca D, Mercaldo F, Visaggio CA, Martinelli F, Giacinto G (2019) On the effectiveness of system api-related information for android ransomware detection. Comput Secur 86:168–182

    Article  Google Scholar 

  58. Scalas M, Rieck K, Giacinto G (2021) Explanation-driven characterization of android ransomware. In: ICPR’2020 workshop on explainable deep learning—AI, pp 228–242. Springer, Cham. https://doi.org/10.1007/978-3-030-68796-0_17

  59. Shrikumar A, Greenside P, Shcherbina A, Kundaje A (2016) Not just a black box: learning important features through propagating activation differences

  60. Sundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. In: Proceedings of the 34th international conference on machine learning-vol 70, pp 3319–3328. JMLR. org

  61. Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfellow I, Fergus R (2014) Intriguing properties of neural networks. In: International conference on learning representations. arxiv:1312.6199

  62. Tam K, Khan SJ, Fattori A, Cavallaro L (2015) CopperDroid: automatic reconstruction of android malware behaviors. In: Proc. 22nd annual network & distributed system security symposium (NDSS). The Internet Society

  63. Tramer F, Carlini N, Brendel W, Madry A (2020) On adaptive attacks to adversarial example defenses. In: Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H (eds) Advances in neural information processing systems, vol 33, pp 1633–1645. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2020/file/11f38f8ecd71867b42433548d1078e38-Paper.pdf

  64. Šrndic N, Laskov P (2014) Practical evasion of a learning-based classifier: a case study. In: Proc. 2014 IEEE symp. security and privacy, SP ’14, pp 197–211. IEEE CS, Washington, DC

  65. Warnecke A, Arp D, Wressnegger C, Rieck K (2020) Evaluating explanation methods for deep learning in security. In: 2020 IEEE european symposium on security and privacy (EuroS&P), pp 158–174. IEEE, Genova. https://doi.org/10.1109/EuroSP48549.2020.00018

  66. Yang W, Kong D, Xie T, Gunter CA (2017) Malware detection in adversarial settings: exploiting feature evolutions and confusions in android apps. In: ACSAC, pp 288–302. ACM

  67. Zhang X, Zhang Y, Zhong M, Ding D, Cao Y, Zhang Y, Zhang M, Yang M (2020) Enhancing State-of-the-art Classifiers with API Semantics to Detect Evolved Android Malware. In: Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pp 757–770. ACM, New York. https://doi.org/10.1145/3372297.3417291

Download references

Acknowledgements

This work has been partly supported by the PRIN 2017 project RexLearn (grant no. 2017TWNMH2), and by the project PON AIM Research and Innovation 2014–2020 - Attraction and International Mobility, both funded by the Italian Ministry of Education, University and Research, and by BMK, BMDW, and the Province of Upper Austria in the frame of the COMET Programme managed by FFG in the COMET Module S3AI.

Author information

Authors and Affiliations

  1. Department of Electrical and Electronic Engineering, University of Cagliari, Piazza d’Armi, 09123, Cagliari, Italy

    Marco Melis, Ambra Demontis, Davide Maiorca, Battista Biggio, Giorgio Giacinto & Fabio Roli

  2. Pluribus One, Cagliari, Italy

    Michele Scalas, Battista Biggio & Fabio Roli

Authors
  1. Marco Melis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michele Scalas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ambra Demontis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Davide Maiorca
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Battista Biggio
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Giorgio Giacinto
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Fabio Roli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marco Melis.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Melis, M., Scalas, M., Demontis, A. et al. Do gradient-based explanations tell anything about adversarial robustness to android malware?. Int. J. Mach. Learn. & Cyber. 13, 217–232 (2022). https://doi.org/10.1007/s13042-021-01393-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-021-01393-7

Keywords

Search

Navigation