Skip to main content

AV-Meter: An Evaluation of Antivirus Scans and Labels

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNSC,volume 8550)


Antivirus scanners are designed to detect malware and, to a lesser extent, to label detections based on a family association. The labeling provided by AV vendors has many applications such as guiding efforts of disinfection and countermeasures, intelligence gathering, and attack attribution, among others. Furthermore, researchers rely on AV labels to establish a baseline of ground truth to compare their detection and classification algorithms. This is done despite many papers pointing out the subtle problem of relying on AV labels. However, the literature lacks any systematic study on validating the performance of antivirus scanners, and the reliability of those labels or detection.

In this paper, we set out to answer several questions concerning the detection rate, correctness of labels, and consistency of detection of AV scanners. Equipped with more than 12,000 malware samples of 11 malware families that are manually inspected and labeled, we pose the following questions. How do antivirus vendors perform relatively on them? How correct are the labels given by those vendors? How consistent are antivirus vendors among each other? We answer those questions unveiling many interesting results, and invite the community to challenge assumptions about relying on antivirus scans and labels as a ground truth for malware analysis and classification. Finally, we stress several research directions that may help addressing the problem.


This is a preview of subscription content, log in via an institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. ZeroAccess (July 2011),

  2. Sykipot is back (July 2012),

  3. Arbor Networks. Another family of DDoS bots: Avzhan (September 2010),

  4. Arbor Networks. JKDDOS: DDoS bot with an interest in the mining industry (March 2011),

  5. Arbor Networks. A ddos family affair: Dirt jumper bot family continues to evolve (July 2012),

  6. Bailey, M., Oberheide, J., Andersen, J., Mao, Z.M., Jahanian, F., Nazario, J.: Automated classification and analysis of internet malware. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 178–197. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  7. Bayer, U., Comparetti, P.M., Hlauschek, C., Krügel, C., Kirda, E.: Scalable, behavior-based malware clustering. In: NDSS (2009)

    Google Scholar 

  8. Canto, J., Dacier, M., Kirda, E., Leita, C.: Large scale malware collection: lessons learned. In: IEEE SRDS Workshop on Sharing Field Data and Experiment Measurements on Resilience of Distributed Computing Systems (2008)

    Google Scholar 

  9. Damballa. The IMDDOS Botnet: Discovery and Analysis (March 2010),

  10. DDoSpedia. Darkness (Optima) (December 2013),

  11. Gashi, I., Stankovic, V., Leita, C., Thonnard, O.: An experimental study of diversity with off-the-shelf antivirus engines. In: Eighth IEEE International Symposium on Network Computing and Applications, NCA 2009., pp. 4–11. IEEE (2009)

    Google Scholar 

  12. Jose Nazario. BlackEnergy DDoS Bot Analysis (October 2007),

  13. Kelly Jackson Higgins. Dropbox, WordPress Used As Cloud Cove. In: New APT Attacks (July 2013),

  14. Kerr, D.: Ubisoft hacked; users’ e-mails and passwords exposed (July 2013),

  15. Kinable, J., Kostakis, O.: Malware classification based on call graph clustering. Journal in Computer Virology 7(4), 233–245 (2011)

    Article  Google Scholar 

  16. Kong, D., Yan, G.: Discriminant malware distance learning on structural information for automated malware classification. In: Proceedings of the 19th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2013)

    Google Scholar 

  17. Kruss, P.: Complete zeus source code has been leaked to the masses (March 2011),

  18. Lanzi, A., Sharif, M.I., Lee, W.: K-tracer: A system for extracting kernel malware behavior. In: NDSS (2009)

    Google Scholar 

  19. Lévesque, F.L., Nsiempba, J., Fernandez, J.M., Chiasson, S., Somayaji, A.: A clinical study of risk factors related to malware infections. In: ACM Conference on Computer and Communications Security, pp. 97–108 (2013)

    Google Scholar 

  20. Maggi, F., Bellini, A., Salvaneschi, G., Zanero, S.: Finding non-trivial malware naming inconsistencies. In: Jajodia, S., Mazumdar, C. (eds.) ICISS 2011. LNCS, vol. 7093, pp. 144–159. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  21. Malware Intel. n0ise Bot. Crimeware particular purpose for DDoS attacks (June 2010),

  22. Revealed: Operation Shady RAT (March 2011),

  23. Microsoft - Malware Protection Center. Spyeye (December 2013),

  24. Mohaisen, A., Alrawi, O.: Amal: High-fidelity, behavior-based automated malware analysis and classification. Technical report, VeriSign Labs (2013)

    Google Scholar 

  25. Mohaisen, A., Alrawi, O.: Unveiling zeus: automated classification of malware samples. In: WWW (Companion Volume), pp. 829–832 (2013)

    Google Scholar 

  26. NYTimes. Nissan is latest company to get hacked (April 2013),

  27. Oberheide, J., Cooke, E., Jahanian, F.: Cloudav: N-version antivirus in the network cloud. In: USENIX Security Symposium, pp. 91–106 (2008)

    Google Scholar 

  28. OPSWAT. Antivirus market analysis (December 2012),

  29. Park, Y., Reeves, D., Mulukutla, V., Sundaravel, B.: Fast malware classification by automated behavioral graph matching. In: CSIIR Workshop. ACM (2010)

    Google Scholar 

  30. Perdisci, R., Lee, W., Feamster, N.: Behavioral clustering of http-based malware and signature generation using malicious network traces. In: USENIX NSDI (2010)

    Google Scholar 

  31. Perdisci, R.,, M.U.: Vamo: towards a fully automated malware clustering validity analysis. In: ACSAC, pp. 329–338. ACM (2012)

    Google Scholar 

  32. Rieck, K., Holz, T., Willems, C., Düssel, P., Laskov, P.: Learning and classification of malware behavior. In: Zamboni, D. (ed.) DIMVA 2008. LNCS, vol. 5137, pp. 108–125. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  33. Rieck, K., Trinius, P., Willems, C., Holz, T.: Automatic analysis of malware behavior using machine learning. Journal of Computer Security 19(4), 639–668 (2011)

    Google Scholar 

  34. Rossow, C., Dietrich, C.J., Grier, C., Kreibich, C., Paxson, V., Pohlmann, N., Bos, H., van Steen, M.: Prudent practices for designing malware experiments: Status quo and outlook. In: IEEE Sec. and Privacy (2012)

    Google Scholar 

  35. Sharif, M.I., Lanzi, A., Giffin, J.T., Lee, W.: Automatic reverse engineering of malware emulators. In: IEEE Sec. and Privacy (2009)

    Google Scholar 

  36. Silveira, V.: An update on linkedin member passwords compromised (July 2012),

  37. Strayer, W.T., Lapsley, D.E., Walsh, R., Livadas, C.: Botnet detection based on network behavior. In: Botnet Detection (2008)

    Google Scholar 

  38. Symantec. Advanced persistent threats (December 2013),

  39. Tian, R., Batten, L., Versteeg, S.: Function length as a tool for malware classification. In: IEEE MALWARE (2008)

    Google Scholar 

  40. Trend Micro. Trend Micro Exposes LURID APT (September 2011),

  41. West, A.G., Mohaisen, A.: Metadata-driven threat classification of network endpoints appearing in malware. In: Dietrich, S. (ed.) DIMVA 2014. LNCS, vol. 8550, pp. 152–171. Springer, Heidelberg (2014)

    Google Scholar 

  42. Zhao, H., Xu, M., Zheng, N., Yao, J., Ho, Q.: Malicious executables classification based on behavioral factor analysis. In: IC4E (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations


Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Mohaisen, A., Alrawi, O. (2014). AV-Meter: An Evaluation of Antivirus Scans and Labels. In: Dietrich, S. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2014. Lecture Notes in Computer Science, vol 8550. Springer, Cham.

Download citation

  • DOI:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-08508-1

  • Online ISBN: 978-3-319-08509-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics