Idea: Benchmarking Android Data Leak Detection Tools

  • Claudio Corrodi
  • Timo Spring
  • Mohammad Ghafari
  • Oscar Nierstrasz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10953)


Virtual application stores for mobile platforms contain many malign and benign applications that exhibit security issues, such as the leaking of sensitive data. In recent years, researchers have proposed a myriad of techniques and tools to detect such issues automatically. However, it is unclear how these approaches perform compared to each other. The tools are often no longer available, thus comparing different approaches is almost infeasible.

In this work, we propose an approach to execute static analysis tools and collect their output to obtain unified reports in a common format. We review the current state-of-the-art in Android data leak detection tools, and from a list of 87 approaches, of which we were able to obtain and execute five. We compare these using a set of known vulnerabilities and discuss the overall performance of the tools. We further present an approach to compare security analysis tools by normalising their interfaces, which simplifies result reproduction and extension.


Data leak Android Benchmarking 



We gratefully acknowledge the financial support of the Swiss National Science Foundation for the project “Agile Software Analysis” (SNSF project No. 200020–162352, Jan 1, 2016 - Dec. 30, 2018). We also thank CHOOSE, the Swiss Group for Original and Outside-the-box Software Engineering of the Swiss Informatics Society, for its financial contribution to the presentation of this paper.


  1. 1.
    Egele, M., Scholte, T., Kirda, E., Kruegel, C.: A survey on automated dynamic malware-analysis techniques and tools. ACM Comput. Surv. (CSUR) 44(2), 6:1–6:42 (2008)Google Scholar
  2. 2.
    Reaves, B., Bowers, J., Gorski III, S.A., Anise, O., Bobhate, R., Cho, R., Das, H., Hussain, S., Karachiwala, H., Scaife, N., Wright, B., Butler, K., Enck, W., Traynor, P.: *Droid: assessment and evaluation of Android application analysis tools. ACM Comput. Surv. 49(3), 55:1–55:30 (2016)CrossRefGoogle Scholar
  3. 3.
    Sadeghi, A., Bagheri, H., Garcia, J., Malek, S.: A taxonomy and qualitative comparison of program analysis techniques for security assessment of Android software. IEEE Trans. Softw. Eng. 43(6), 492–530 (2017)CrossRefGoogle Scholar
  4. 4.
    Tam, K., Feizollah, A., Anuar, N.B., Salleh, R., Cavallaro, L.: The evolution of Android malware and Android analysis techniques. ACM Comput. Surv. 49(4), 76:1–76:41 (2017)CrossRefGoogle Scholar
  5. 5.
    Sufatrio, Tan, D.J.J., Chua, T.-W., Thing, V.L.L.: Securing Android: a survey, taxonomy, and challenges. ACM Comput. Surv. 47(4), 58:1–58:45 (2015). Article no. 58CrossRefGoogle Scholar
  6. 6.
    Gadient, P.: Security in Android applications. Masters thesis. University of Bern, August 2017Google Scholar
  7. 7.
    Ghafari, M., Gadient, P., Nierstrasz, O.: Security smells in Android. In: 17th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM), pp. 121–130, September 2017Google Scholar
  8. 8.
    Amann, S., Nadi, S., Nguyen, H.A., Nguyen, T.N., Mezini, M.: MUBench: a benchmark for API-misuse detectors. In: 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR), pp. 464–467 (2016)Google Scholar
  9. 9.
    Arzt, S., Rasthofer, S., Fritz, C., Bodden, E., Bartel, A., Klein, J., Le Traon, Y., Octeau, D., McDaniel, P.: Flowdroid: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps. SIGPLAN Notices, vol. 49, no. 6, pp. 259–269 (2014)Google Scholar
  10. 10.
    Calzavara, S., Grishchenko, I., Maffei, M.: Horndroid: practical and sound static analysis of Android applications by SMT solving. In: 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 47–62, March 2016Google Scholar
  11. 11.
    Li, L., Bartel, A., Bissyandé, T.F., Klein, J., Traon, Y.L., Arzt, S., Rasthofer, S., Bodden, E., Octeau, D., McDaniel, P.: IccTA: detecting inter-component privacy leaks in Android apps. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering - Volume 1, pp. 280–291 (2015)Google Scholar
  12. 12.
    Octeau, D., McDaniel, P., Jha, S., Bartel, A., Bodden, E., Klein, J., Le Traon, Y.: Effective inter-component communication mapping in Android with Epicc: an essential step towards holistic security analysis (2013)Google Scholar
  13. 13.
    Bagheri, H., Sadeghi, A., Garcia, J., Malek, S.: Covert: compositional analysis of Android inter-app permission leakage. IEEE Trans. Softw. Eng. 41(9), 866–886 (2015)CrossRefGoogle Scholar
  14. 14.
    Bu, W., Xue, M., Xu, L., Zhou, Y., Tang, Z., Xie, T.: When program analysis meets mobile security: an industrial study of misusing Android internet sockets. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pp. 842–847. ACM (2017)Google Scholar
  15. 15.
    McNemar, Q.: Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157 (1947)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Claudio Corrodi
    • 1
  • Timo Spring
    • 1
  • Mohammad Ghafari
    • 1
  • Oscar Nierstrasz
    • 1
  1. 1.Software Composition GroupUniversity of BernBernSwitzerland

Personalised recommendations