Abstract
Antivirus scanners are designed to detect malware and, to a lesser extent, to label detections based on a family association. The labeling provided by AV vendors has many applications such as guiding efforts of disinfection and countermeasures, intelligence gathering, and attack attribution, among others. Furthermore, researchers rely on AV labels to establish a baseline of ground truth to compare their detection and classification algorithms. This is done despite many papers pointing out the subtle problem of relying on AV labels. However, the literature lacks any systematic study on validating the performance of antivirus scanners, and the reliability of those labels or detection.
In this paper, we set out to answer several questions concerning the detection rate, correctness of labels, and consistency of detection of AV scanners. Equipped with more than 12,000 malware samples of 11 malware families that are manually inspected and labeled, we pose the following questions. How do antivirus vendors perform relatively on them? How correct are the labels given by those vendors? How consistent are antivirus vendors among each other? We answer those questions unveiling many interesting results, and invite the community to challenge assumptions about relying on antivirus scans and labels as a ground truth for malware analysis and classification. Finally, we stress several research directions that may help addressing the problem.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
ZeroAccess (July 2011), http://bit.ly/IPxi0N
Sykipot is back (July 2012), http://www.alienvault.com/open-threat-exchange/blog/sykipot-is-back
Arbor Networks. Another family of DDoS bots: Avzhan (September 2010), http://bit.ly/IJ7yCz
Arbor Networks. JKDDOS: DDoS bot with an interest in the mining industry (March 2011), http://bit.ly/18juHoS
Arbor Networks. A ddos family affair: Dirt jumper bot family continues to evolve (July 2012), http://bit.ly/JgBI12
Bailey, M., Oberheide, J., Andersen, J., Mao, Z.M., Jahanian, F., Nazario, J.: Automated classification and analysis of internet malware. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 178–197. Springer, Heidelberg (2007)
Bayer, U., Comparetti, P.M., Hlauschek, C., Krügel, C., Kirda, E.: Scalable, behavior-based malware clustering. In: NDSS (2009)
Canto, J., Dacier, M., Kirda, E., Leita, C.: Large scale malware collection: lessons learned. In: IEEE SRDS Workshop on Sharing Field Data and Experiment Measurements on Resilience of Distributed Computing Systems (2008)
Damballa. The IMDDOS Botnet: Discovery and Analysis (March 2010), http://bit.ly/1dRi2yi
DDoSpedia. Darkness (Optima) (December 2013), http://bit.ly/1eR40Jc
Gashi, I., Stankovic, V., Leita, C., Thonnard, O.: An experimental study of diversity with off-the-shelf antivirus engines. In: Eighth IEEE International Symposium on Network Computing and Applications, NCA 2009., pp. 4–11. IEEE (2009)
Jose Nazario. BlackEnergy DDoS Bot Analysis (October 2007), http://bit.ly/1bidVYB
Kelly Jackson Higgins. Dropbox, WordPress Used As Cloud Cove. In: New APT Attacks (July 2013), http://ubm.io/1cYMOQS
Kerr, D.: Ubisoft hacked; users’ e-mails and passwords exposed (July 2013), http://cnet.co/14ONGDi
Kinable, J., Kostakis, O.: Malware classification based on call graph clustering. Journal in Computer Virology 7(4), 233–245 (2011)
Kong, D., Yan, G.: Discriminant malware distance learning on structural information for automated malware classification. In: Proceedings of the 19th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2013)
Kruss, P.: Complete zeus source code has been leaked to the masses (March 2011), http://www.csis.dk/en/csis/blog/3229
Lanzi, A., Sharif, M.I., Lee, W.: K-tracer: A system for extracting kernel malware behavior. In: NDSS (2009)
Lévesque, F.L., Nsiempba, J., Fernandez, J.M., Chiasson, S., Somayaji, A.: A clinical study of risk factors related to malware infections. In: ACM Conference on Computer and Communications Security, pp. 97–108 (2013)
Maggi, F., Bellini, A., Salvaneschi, G., Zanero, S.: Finding non-trivial malware naming inconsistencies. In: Jajodia, S., Mazumdar, C. (eds.) ICISS 2011. LNCS, vol. 7093, pp. 144–159. Springer, Heidelberg (2011)
Malware Intel. n0ise Bot. Crimeware particular purpose for DDoS attacks (June 2010), http://bit.ly/1kd24Mg
mcafee.com. Revealed: Operation Shady RAT (March 2011), http://bit.ly/IJ9fQG
Microsoft - Malware Protection Center. Spyeye (December 2013), http://bit.ly/1kBBnky
Mohaisen, A., Alrawi, O.: Amal: High-fidelity, behavior-based automated malware analysis and classification. Technical report, VeriSign Labs (2013)
Mohaisen, A., Alrawi, O.: Unveiling zeus: automated classification of malware samples. In: WWW (Companion Volume), pp. 829–832 (2013)
NYTimes. Nissan is latest company to get hacked (April 2013), http://nyti.ms/Jm52zb
Oberheide, J., Cooke, E., Jahanian, F.: Cloudav: N-version antivirus in the network cloud. In: USENIX Security Symposium, pp. 91–106 (2008)
OPSWAT. Antivirus market analysis (December 2012), http://bit.ly/1cCr9zE
Park, Y., Reeves, D., Mulukutla, V., Sundaravel, B.: Fast malware classification by automated behavioral graph matching. In: CSIIR Workshop. ACM (2010)
Perdisci, R., Lee, W., Feamster, N.: Behavioral clustering of http-based malware and signature generation using malicious network traces. In: USENIX NSDI (2010)
Perdisci, R.,, M.U.: Vamo: towards a fully automated malware clustering validity analysis. In: ACSAC, pp. 329–338. ACM (2012)
Rieck, K., Holz, T., Willems, C., Düssel, P., Laskov, P.: Learning and classification of malware behavior. In: Zamboni, D. (ed.) DIMVA 2008. LNCS, vol. 5137, pp. 108–125. Springer, Heidelberg (2008)
Rieck, K., Trinius, P., Willems, C., Holz, T.: Automatic analysis of malware behavior using machine learning. Journal of Computer Security 19(4), 639–668 (2011)
Rossow, C., Dietrich, C.J., Grier, C., Kreibich, C., Paxson, V., Pohlmann, N., Bos, H., van Steen, M.: Prudent practices for designing malware experiments: Status quo and outlook. In: IEEE Sec. and Privacy (2012)
Sharif, M.I., Lanzi, A., Giffin, J.T., Lee, W.: Automatic reverse engineering of malware emulators. In: IEEE Sec. and Privacy (2009)
Silveira, V.: An update on linkedin member passwords compromised (July 2012), http://linkd.in/Ni5aTg
Strayer, W.T., Lapsley, D.E., Walsh, R., Livadas, C.: Botnet detection based on network behavior. In: Botnet Detection (2008)
Symantec. Advanced persistent threats (December 2013), http://bit.ly/1bXXdj9
Tian, R., Batten, L., Versteeg, S.: Function length as a tool for malware classification. In: IEEE MALWARE (2008)
Trend Micro. Trend Micro Exposes LURID APT (September 2011), http://bit.ly/18mX82e
West, A.G., Mohaisen, A.: Metadata-driven threat classification of network endpoints appearing in malware. In: Dietrich, S. (ed.) DIMVA 2014. LNCS, vol. 8550, pp. 152–171. Springer, Heidelberg (2014)
Zhao, H., Xu, M., Zheng, N., Yao, J., Ho, Q.: Malicious executables classification based on behavioral factor analysis. In: IC4E (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Mohaisen, A., Alrawi, O. (2014). AV-Meter: An Evaluation of Antivirus Scans and Labels. In: Dietrich, S. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2014. Lecture Notes in Computer Science, vol 8550. Springer, Cham. https://doi.org/10.1007/978-3-319-08509-8_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-08509-8_7
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-08508-1
Online ISBN: 978-3-319-08509-8
eBook Packages: Computer ScienceComputer Science (R0)