Skip to main content
Log in

A defense framework against malware and vulnerability exploits

  • Regular Contribution
  • Published:
International Journal of Information Security Aims and scope Submit manuscript

Abstract

Current anti-malware tools have proved to be insufficient in combating ever-evolving malware attacks and vulnerability exploits due to inevitable vulnerabilities present in the complex software used today. In addition, the performance penalty incurred by anti-malware tools is magnified when security approaches designed for desktops are migrated to modern mobile devices, such as tablets and laptops, due to their relatively limited processing capabilities and battery capacities. In this paper, we propose a fine-grained anomaly detection defense framework that offers a cost-efficient way to detect malicious behavior and prevent vulnerability exploits in resource-constrained computing platforms. In this framework, a trusted third party (e.g., the publisher) first tests a new application by running it in a heavily monitored testing environment that emulates the target system and extracts a behavioral model from its execution paths. Extensive security policies are enforced during this process. In case of a violation, the program is denied release to the user. If the application passes the tests, the user can download the behavioral model along with the tested application binary. At run-time, the application is monitored against the behavioral model. In the unlikely event that a new execution path is encountered, conservative but lightweight security policies are applied. To reduce overhead at the user end, the behavioral model may be further reduced by the publisher through static analysis. We have implemented the defense framework using a netbook with the Intel Atom processor and evaluated it with a suite of 51 real-world Linux viruses and malware. Experiments demonstrate that our tool achieves a very high coverage (98 %) of considered malware and security threats. The four antivirus tools we compare our tool against were found to have poor virus coverage, especially of obfuscated viruses. By removing safe standard library blocks from the behavioral model, we reduce the model size by 8.4\(\times \) and the user’s run-time overhead by 23 %.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. CSI: CSI computer crime and security survey. http://www.gocsi.com (2010)

  2. Symantec: Security response. http://www.messagelabs.com/resources/mlireports.aspx (2011)

  3. McAfee: Threats predictions. http://www.mcafee.com/us/resources/reports/rp-threat-predictions-2011 (2011)

  4. CERT: CERT vulnerability notes database. http://www.kb.cert.org/vuls (2011)

  5. CVE: The standard for information security vulnerability names. http://cve.mitre.org/ (2011)

  6. Aaraj, N., Raghunathan, A., Jha, N.K.: Virtualization-based framework for malware defense. In: Proceedings of Conference Detection of Intrusions and Malware and Vulnerability, Assessment, pp. 64–87 (2008).

  7. Kaspersky Anti-Virus Mobile. http://usa.kaspersky.com/products-services/home-computer-security/mobile-security (2014)

  8. Christodorescu, M., Jha, S., Seshia, S., Song, D., Bryant, R.: Semantics-aware malware detection. In: Proceedings of IEEE Symposium Security and Privacy, pp. 32–46 (2005).

  9. Morales, J.A., Clarke, P.J., Deng, Y., Kibria, B.M.G.: Testing and evaluating virus detectors for handheld devices. Comput. Virol. 2, 135–147 (2006)

    Article  Google Scholar 

  10. Kaspersky: Using heuristic analysis in Kaspersky anti-virus. http://www.kaspersky.com (2010)

  11. Necula, G.C.: Sandbox technology inside AV scanners. In: Proceedings of Virus Bulletin Conference (2001).

  12. Goldberg, I., Wagner, D., Thomas, R., Brewer, E.A.: A secure environment for untrusted helper applications confining the wily hacker. In: Proceedings of Conference USENIX Security Symposium, pp. 1–1 (1996).

  13. Peterson, D.S., Bishop, M., Pandey, R.: A flexible containment mechanism for executing untrusted code. In: Proceedings of USENIX Security Symposium, pp. 207–225 (2002).

  14. Jacob, G., Debar, H., Filiol, E.: Behavioral detection of malware: from a survey towards an established taxonomy. Comput. Virol. 4, 251–266 (2008)

    Article  Google Scholar 

  15. Schmidt, A.-D., Peters, F., Lamour, F., Scheel, C., Çamtepe, S.A., Albayrak, S.: Monitoring smartphones for anomaly detection. Mobile Netw. Appl. 14, 92–106 (2009)

    Article  Google Scholar 

  16. Jacob, G., Debar, H., Filiol, E.: Malware behavioral detection by attribute-automata using abstraction from platform and language. In: Proceedings of International Symposium, Recent Advances in Intrusion Detection, pp. 81–100 (2009).

  17. Wang, H., Jha, S., Ganapathy, V.: Netspy: automatic generation of spyware signatures for NIDS. In: Proceedings of Computer Security Applications Conference, pp. 99–108 (2006).

  18. Bose, A., Hu, X., Shin, K.G., Park, T.: Behavioral detection of malware on mobile handsets. In: Proceedings of International Conference on Mobile Systems, Applications, and Services, pp. 225–238 (2008).

  19. Xie, L., Zhang, X., Seifert, J.-P., Zhu, S.: pBMDS: a behavior-based malware detection system for cellphone devices. In: Proceedings of ACM Conference on Wireless, Network Security, pp. 37–48 (2010).

  20. Forrest, S., Hofmeyr, S., Somayaji, T., Longstaff, T.: A sense of self for Unix processes. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 120–128 (1996).

  21. Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 144–155 (2001).

  22. Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: Alternative data models. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 133–145 (1999).

  23. Baratloo, A., Singh, N., Tsai, T.: Transparent run-time defense against stack smashing attacks. In: Proceedings of USENIX Annual Technical Conference, pp. 251–262 (2000).

  24. Kiriansky, V., Bruening, D., Amarasinghe, S.P.: Secure execution via program shepherding. In: Proceedings of USENIX Security Symposium, pp. 191–206 (2002).

  25. Nethercote, N: Bounds-checking entire programs without recompiling. In: Proceedings of Workshop, Semantics, Program Analysis, and Computing Environments for Memory Management (2004).

  26. Cheng, J., Wong, S.H.Y., Yang, H., Lu, S.: SmartSiren: virus detection and alert for smartphones. In: Proceedings of MobiSys, pp. 133–145 (2007).

  27. Bose, A., Shin, K.G.: Proactive security for mobile messaging networks. In: Proceedings of ACM Workshop on Wireless, Security, pp. 95–104 (2006).

  28. Necula, G.C.: Proof-carrying code. In: Proceedings of ACM Symposium on Principles of Programming Languages, pp. 106–119 (1997).

  29. Necula, G.C., Lee, P.: The design and implementation of a certifying compiler. In: Proceedings of ACM Conference on Programming Language Design and Implementation, pp. 333–344 (1998).

  30. Sekar, R., Venkatakrishnan, V., Basu, S., Bhatkar, S., DuVarney, D.C.: Model-carrying code: a practical approach for safe execution of untrusted applications. In: Proceedings of ACM Symposium on Operating Systems Principles, pp. 15–28 (2003).

  31. Liang Z., Sekar, R.: Automatic generation of buffer overflow attack signatures: an approach based on program behavior models. In: Proceedings of Computer Security Applications Conference, pp. 215–224 (2005).

  32. Luk, C., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Janapa, V., Hazelwood R.K.: Pin: building customized program analysis tools with dynamic instrumentation. In: Proceedings of ACM Conference on Programming Language Design and Implementation, pp. 190–200 (2005).

  33. Newsome, J., Song, D.: Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In: Proceedings of Symposium on Network and Distributed System Security (2005).

  34. Uppuluri, P., Sekar, R.: Experiences with specification-based intrusion detection. In: Proceedings of International Symposium, Recent Advances in Intrusion Detection, pp. 172–189 (2001).

  35. Shahriar, H., Zulkernine, M.: Classification of static analysis-based buffer overflow detectors. In: Proceedings of International Conference on Secure Software Integration and Reliability Improvement Companion, pp. 94–101 (2010).

  36. Dill, V.: A decision procedure for bitvectors and arrays. In: Proceedings of International Conference on Computer-Aided Verification, pp. 519–531 (2007).

  37. Cadar, C., Ganesh, V., Pawlowski, P.M., Dill, D.L., Engler, D.R.: Exe: automatically generating inputs of death. In: Proceedings of ACM Conference on Computer and Communications, Security, pp. 322–335 (2006).

  38. Flawfinder, v1. http://www.dwheeler.com/flawfinder (2011)

  39. VXHEAVENS. http://forum.vxheavens.com (2011)

  40. Windows, Linux Virus Collection & Creation Tools. http://virus-codes.blogspot.com (2011)

  41. The Ultimate Packer for eXecutables. http://upx.sourceforge.net/ (2013)

  42. F-PROT. http://www.f-prot.com (2011)

  43. ClamAV. http://www.clamav.net (2011)

  44. Avira. http://www.avira.com (2011)

  45. AVG. http://www.avg.com (2011)

  46. Bircher, W.L., Law, J., Valluri, M., John, L.K.: Effective use of performance monitoring counters for run-time prediction of power. University of Texas at Austin, Technical, Report, TR-041104-01 (2004).

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Meng Zhang.

Additional information

This work was supported by NSF under Grant No. CNS-0914787.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, M., Raghunathan, A. & Jha, N.K. A defense framework against malware and vulnerability exploits. Int. J. Inf. Secur. 13, 439–452 (2014). https://doi.org/10.1007/s10207-014-0233-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10207-014-0233-1

Keywords

Navigation