Are free Android app security analysis tools effective in detecting known vulnerabilities?

  • Venkatesh-Prasad RanganathEmail author
  • Joydeep Mitra


Increasing interest in securing the Android ecosystem has spawned numerous efforts to assist app developers in building secure apps. These efforts have resulted in tools and techniques capable of detecting vulnerabilities and malicious behaviors in apps. However, there has been no evaluation of the effectiveness of these tools and techniques in detecting known vulnerabilities. The absence of such evaluations puts app developers at a disadvantage when choosing security analysis tools to secure their apps. In this regard, we evaluated the effectiveness of vulnerability detection tools for Android apps. We reviewed 64 tools and empirically evaluated 14 vulnerability detection tools against 42 known unique vulnerabilities captured by Ghera benchmarks, which are composed of both vulnerable and secure apps. Of the 20 observations from the evaluation, the main observation is existing vulnerability detection tools for Android apps are very limited in their ability to detect known vulnerabilities — all of the evaluated tools together could only detect 30 of the 42 known unique vulnerabilities. More effort is required if security analysis tools are to help developers build secure apps. We hope the observations from this evaluation will help app developers choose appropriate security analysis tools and persuade tool developers and researchers to identify and address limitations in their tools and techniques. We also hope this evaluation will catalyze or spark a conversation in the software engineering and security communities to require a more rigorous and explicit evaluation of security analysis tools and techniques.


Empirical evaluation Effectiveness Security analysis tools Vulnerabilities Android 



We thank the readers and reviewers for their feedback to improve this manuscript.

We thank Aditya Narkar and Nasik Muhammad Nafi for their help in implementing 17 new benchmarks that are cataloged as Ghera benchmarks for the first time in this paper and were used in the evaluations described in this paper.

Supplementary material

10664_2019_9749_MOESM1_ESM.pdf (191 kb)
(PDF 190 KB)


  1. Abraham A, Schelecht D, Dobrushin M (2015) Mobile Security Framework., Accessed: 21-Nov-2017
  2. Agrawal A (2015) Mobile Security Wiki., Accessed: 01-May-2018
  3. Allen B (2018) AppCritique: Online Vulnerability Detection Tool., Accessed: 21-Nov-2017
  4. Allix K, Bissyande TF, Klein J, Traon YL (2016) Androzoo: Collecting millions of android apps for the research community. In: Proceedings of the 13th International Conference on Mining Software Repositories. ACM, pp 468–471Google Scholar
  5. Anand S, Naik M, Harrold MJ, Yang H (2012) Automated concolic testing of smartphone apps. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. ACM, pp 59:1–59:11Google Scholar
  6. Antunes N, Vieira M (2010) Benchmarking vulnerability detection tools for web services. In: 2010 IEEE International Conference on Web Services, pp 203–210Google Scholar
  7. App-Ray (2015) AppRay., Accessed: 04-Jun-2018
  8. Arzt S, Rasthofer S, Fritz C, Bodden E, Bartel A, Klein J, Le Traon Y, Octeau D, McDaniel P (2014) Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. In: Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, pp 259–269., Accessed: 21-Nov-2017
  9. Au KWY, Zhou YF, Huang Z, Lie D (2012) Pscout: Analyzing the android permission specification. In: Proceedings of the 2012 ACM Conference on Computer and Communications Security. ACM, pp 217–228,, Accessed: 21-Nov-2017
  10. Backes SRT GmbH (2014) SRT:AppGuard., Accessed: 04-Jun-2018
  11. Bagheri H, Sadeghi A, Garcia J, Malek S (2015) Covert: Compositional analysis of android inter-app permission leakage. IEEE Transactions on Software Engineering. pp, 866–886., Accessed: 21-May-2018
  12. Bhatia A (2014) A collection of android security related resources., Accessed: 01-May-2018
  13. Bosu A (2017a) DIALDroidBench., Accessed: 12-Sep-2018
  14. Bosu A, Liu F, Yao DD, Wang G (2017b) Collusive data leak and more: Large-scale threat analysis of inter-app communications. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security. ACM, New York, pp 71–85., Accessed: 05-May-2018
  15. Bugiel S, Heuser S, Sadeghi AR (2013) Flexible and fine-grained mandatory access control on android for diverse security and privacy policies. In: 22nd USENIX Security Symposium. USENIX.
  16. Calzavara S, Grishchenko I, Maffei M (2016) Horndroid: Practical and sound static analysis of android applications by SMT solving. In: 2016 IEEE European Symposium on Security and Privacy, pp 47–62., Accessed: 05-May-2018
  17. Chin E, Felt A P, Greenwood K, Wagner D (2011) Analyzing inter-application communication in android. In: Proceedings of the 9th International Conference on Mobile Systems, Applications, and Services. ACM, pp 239–252Google Scholar
  18. Chin E, Wagner DA (2013) Bifocals: Analyzing webview vulnerabilities in android applications. In: WISA. Springer, pp 138–159Google Scholar
  19. Cuckoo (2015) CuckooDroid: Automated Android Malware Analysis., Accessed: 01-May-2018
  20. Debize T (2012) AndroWarn : Yet Another Static Code Analyzer for malicious Android applications., Accessed: 21-Nov-2017
  21. Deshotels L, Notani V, Lakhotia A (2014) Droidlegacy: Automated familial classification of android malware. In: Proceedings of ACM SIGPLAN on Program Protection and Reverse Engineering Workshop 2014. ACM, pp 3:1–3:12Google Scholar
  22. DevKnox (2016) DevKnox - Security Plugin for Android Studio., Accessed: 21-Nov-2017
  23. DroidBench (2013) DroidBench: A micro-benchmark suite to assess the stability of taint-analysis tools for Android., Accessed: 01-June-2018
  24. Egele M, Brumley D, Fratantonio Y, Kruegel C (2013) An empirical study of cryptographic misuse in android applications. In: Proceedings of the 2013 ACM SIGSAC Conference on Computer & Communications Security. ACM, pp 73–84Google Scholar
  25. Enck W, Gilbert P, Chun BG, Cox LP, Jung J, McDaniel P, Sheth AN (2010) Taintdroid: An information-flow tracking system for realtime privacy monitoring on smartphones. In: Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, USENIX, pp 393–407Google Scholar
  26. Ernst MD, Just R, Millstein S, Dietl W, Pernsteiner S, Roesner F, Koscher K, Barros PB, Bhoraskar R, Han S, Vines P, Wu EX (2014) Collaborative verification of information flow for a high-assurance app store. In: Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. ACM, pp 1092–1104., Accessed : 01-June-2018
  27. Fahl S, Harbach M, Muders T, Baumgärtner L, Freisleben B, Smith M (2012) Why eve and mallory love android: An analysis of android ssl (in)security. In: Proceedings of the 2012 ACM Conference on Computer and Communications Security. ACM, pp 50–61., Accessed: 15-Apr-2018
  28. Fuchs A, Chaudhuri A, Foster JS (2009) Scandroid: Automated security certification of android applications. Tech. rep., University of Maryland., Accessed: 04-Jun-2018
  29. Garcia J, Hammad M, Ghorbani N, Malek S (2017) Automatic generation of inter-component communication exploits for android applications. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, pp, 661–671., Accessed: 24-Apr-2018
  30. Google Inc (2017) Android Intent with Chrome., Accessed: 29-May-2018
  31. Google Inc (2018a) Android developer documentation - Binder., Accessed: 01-Jun-2018
  32. Google Inc (2018b) Android developer documentation - Content Provider.,%20java.lang.String,%20android.os.Bundle), Accessed: 07-Mar-2018
  33. Google Inc (2018c) Android Security Tips., Accessed: 01-Jun-2017
  34. Gordon MI, Kim D, Perkins J, Gilham L, Nguyen N, Rinard M (2015) Information-flow analysis of Android applications in DroidSafe. In: Proceedings of the 22nd Annual Network and Distributed System Security Symposium (NDSS’15)., Accessed: 21-Apr-2018
  35. Gray J (1992) Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., San MateoGoogle Scholar
  36. Green M, Smith M (2016) Developers are not the enemy!: The need for usable security apis. IEEE Secur Priv 14(5):40–46CrossRefGoogle Scholar
  37. IBM (2018) IBM App Scan., Accessed: 01-June-2018
  38. Jia YJ, Chen QA, Lin Y, Kong C, Mao ZM (2017) Open doors for bob and mallory: Open port usage in android apps and security implications. In: EuroS & P. IEEE, pp 190–203Google Scholar
  39. Keen T (2016) Joint Advanced Application Defect Assessment for Android Application (JAADAS)., Accessed: 21-Nov-2017
  40. Klieber W, Flynn L, Bhosale A, Jia L, Bauer L (2014) Android taint flow analysis for app sets. In: Proceedings of the 3rd ACM SIGPLAN International Workshop on the State of the Art in Java Program Analysis. ACM., Accessed: 21-Apr-2018
  41. Li L, Bartel A, Bissyandé TF, Klein J, Le Traon Y, Arzt S, Rasthofer S, Bodden E, Octeau D, McDaniel P (2015) Iccta: Detecting inter-component privacy leaks in android apps. In: Proceedings of the 37th International Conference on Software Engineering - Volume 1. IEEE Press, pp 280–291., Accessed: 05-May-2018
  42. Lin YC (2015) AndroBugs Framework., Accessed: 21-Nov-2017
  43. LinkedIn (2015) Quick Android Review Kit., Accessed: 21-Nov-2017
  44. Mitra J, Ranganath V (2017a) Ghera: A repository of android app vulnerability benchmarks. In: Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. ACM, pp 43–52., Accessed: 21-Nov-2017
  45. Mitre Corporation (2017b) Common vulnerabilities and exposures., Accessed: 08-Jun-2017
  46. MWR Labs (2012) Drozer., Accessed: 20-Apr-2018
  47. Nadkarni A, Andow B, Enck W, Jha S (2016) Practical DIFC enforcement on android. In: 25th USENIX Security Symposium (USENIX Security 16). USENIX, pp 1119–1136., Accessed : 01-June-2018
  48. Nguyen DC, Wermke D, Acar Y, Backes M, Weir C, Fahl S (2017) A stitch in time: Supporting android developers in writing secure code. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, pp 1065–1077., Accessed: 21-Apr-2018
  49. Pauck F, Bodden E, Wehrheim H (2018) Do android taint analysis tools keep their promises? In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 331–341Google Scholar
  50. Powers DMW (2011) Evaluation: From precision, recall and f-measure to roc., informedness, markedness & correlation. J Mach Learn Technol 2(1):37–63MathSciNetGoogle Scholar
  51. Reaves B, Bowers J, Gorski III SA, Anise O, Bobhate R, Cho R, Das H, Hussain S, Karachiwala H, Scaife N, Wright B, Butler K, Enck W, Traynor P (2016) *droid: Assessment and evaluation of android application analysis tools. ACM Comput Surv 49(3):55:1–55:30CrossRefGoogle Scholar
  52. Ren C, Zhang Y, Xue H, Wei T, Liu P (2015) Towards discovering and understanding task hijacking in android. In: 24th USENIX Security Symposium (USENIX Security 15). USENIX, pp 945–959Google Scholar
  53. Rinaudo J, Heguiabehere J (2016) Marvin Static Analyzer., Accessed: 21-Nov-2017
  54. Rogue Wave Software (2017) Klocwork., Accessed: 01-June-2018
  55. Ryohei K (2014) CVE-2014-5319: Directory traversal vulnerability., Accessed: 29-May-2018
  56. Ryoji T (2014) CVE-2014-0806: Geolocation disclosure., Accessed: 29-May-2018
  57. Sadeghi A, Bagheri H, Garcia J, Malek S (2017) A taxonomy and qualitative comparison of program analysis techniques for security assessment of android software. IEEE Trans Softw Eng 43(6):492–530CrossRefGoogle Scholar
  58. Satoru T (2014) CVE-2014-1977: Weak Permissions., Accessed: 29-May-2018
  59. Shen F, Vishnubhotla N, Todarka C, Arora M, Dhandapani B, Lehner EJ, Ko SY, Ziarek L (2014) Information flows as a permission mechanism. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering. ACM, pp 515–526., Accessed : 01-June-2018
  60. Sounthiraraj D, Sahs J, Greenwood G, Lin Z, Khan L (2014) Smv-hunter: Large scale, automated detection of ssl/tls man-in-the-middle vulnerabilities in android apps. In: Inproceedings of the 21st Annual Network and Distributed System Security Symposium (NDSS’14)., Accessed: 10-Jun-2018
  61. Sufatrio, Tan DJJ, Chua TW, Thing VLL (2015) Securing android: a survey, taxonomy, and challenges. ACM Comput Surv 47(4):58:1–58:45CrossRefGoogle Scholar
  62. Tendulkar V, Enck W (2014) An application package configuration approach to mitigating android SSL vulnerabilities. CoRR arXiv:
  63. van der Veen V, Rossow C (2013) Tracedroid., Accessed: 01-May-2018
  64. Wang T (2014a) CVE-2014-8507: SQL Injection vulnerability., Accessed: 29-May-2018
  65. Wang T (2014b) CVE-2014-8609: Android Settings application privilege leakage vulnerability., Accessed: 29-May-2018
  66. Wei F, Roy S, Robby OX (2014) Amandroid: A precise and general inter-component data flow analysis framework for security vetting of android apps. In: Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. ACM, pp 1329–1341., Accessed: 05-May-2018
  67. Wei F (2017) ICC-Bench., Accessed: 12-Sep-2018
  68. Wei F, Roy S, Ou X (2018) Robby Amandroid: A precise and general inter-component data flow analysis framework for security vetting of android apps. ACM Trans Privacy Secur 21(3):14:1–14:32Google Scholar
  69. Xu R, Saïdi H, Anderson R (2012) Aurasium: Practical policy enforcement for android applications. In: Proceedings of the 21st USENIX Conference on Security Symposium. USENIX, pp 27–27,, Accessed : 01-June-2018
  70. Yu D (2014) CVE-2014-1566: Leak information to SD card., Accessed: 29-May-2018
  71. Zhauniarovich Y, Ahmad M, Gadyatskaya O, Crispo B, Massacci F (2015) Stadyna: Addressing the problem of dynamic code updates in the security analysis of android applications. In: Proceedings of the 5th ACM Conference on Data and Application Security and Privacy. ACM, pp 37–48., Accessed: 11-Jun-2018
  72. Zhou Y, Jiang X (2012) Dissecting android malware: Characterization and evolution. In: Proceedings of the 2012 IEEE Symposium on Security and Privacy. IEEE Computer Society, pp 95–109Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Kansas State UniversityManhattanUSA

Personalised recommendations