Journal in Computer Virology

, Volume 8, Issue 1–2, pp 1–13 | Cite as

Shadow attacks: automatically evading system-call-behavior based malware detection

  • Weiqin Ma
  • Pu Duan
  • Sanmin Liu
  • Guofei Gu
  • Jyh-Charn Liu
Original Paper


Contemporary malware makes extensive use of different techniques such as packing, code obfuscation, polymorphism, and metamorphism, to evade signature-based detection. Traditional signature-based detection technique is hard to catch up with latest malware or unknown malware. Behavior-based detection models are being investigated as a new methodology to defeat malware. This kind of approaches typically relies on system call sequences/graphs to model a malicious specification/pattern. In this paper, we present a new class of attacks, namely “shadow attacks”, to evade current behavior-based malware detectors by partitioning one piece of malware into multiple “shadow processes”. None of the shadow processes contains a recognizable malicious behavior specification known to single-process-based malware detectors, yet those shadow processes as an ensemble can still fulfill the original malicious functionality. To demonstrate the feasibility of this attack, we have developed a compiler-level prototype tool, AutoShadow, to automatically generate shadow-process version of malware given the source code of original malware. Our preliminary result has demonstrated the effectiveness of shadow attacks in evading several behavior-based malware analysis/detection solutions in real world. With the increasing adoption of multi-core computers and multi-process programs, malware writers may exploit more such shadow attacks in the future. We hope our preliminary study can foster more discussion and research to improve current generation of behavior-based malware detectors to address this great potential threat before it becomes a security problem of the epidemic proportions.


Shared Memory System Call Intermediate Representation Simple Object Access Protocol Covert Channel 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., Mitchell, J. C.: A Layered Architecture for Detecting Malicious Behaviors. In: Proceedings of the 11th international Symposium on Recent Advances in intrusion Detection (RAID’08) (2008)Google Scholar
  2. 2.
    Lattner, C., Adve, V.: LLVM: A compilation framework for lifelong program analysis & transformation. In: Proceedings of the 2004 International Symposium on Code Generation and Optimization (CGO’04) (2004)Google Scholar
  3. 3.
    Christodorescu, M., Jha, S., Seshia, S., Song, D., Bryant, R.: Semantics-Aware Malware Detection. In: Proceedings of IEEE Symposium on Security and Privacy (2005)Google Scholar
  4. 4.
    Barford, P., Yagneswaran, V.: An Inside Look at Botnets. In: Advances in Information Security. Springer, Berlin (2006)Google Scholar
  5. 5.
    Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th ACM conference on Computer and communications security (CCS’02) (2002)Google Scholar
  6. 6.
    Filiol E.: Formalisation and implementation aspects of k-ary (malicious) codes. J. Comput. Virol. 3(3), 75–86 (2007) (EICAR 2007 Best Academic Papers)CrossRefGoogle Scholar
  7. 7.
  8. 8.
    Kolbitsch, C., Comparetti, P.M., Kruegel, C., Kirda, E., Zhou, X., Wang, X.: Effective and Efficient Malware Detection at the End Host. In: Proceedings of 18th USENIX Security Symposium (2009)Google Scholar
  9. 9.
    Nomenumbra: Counter Behavior Based Malware Analysis, Hacking at Random. HAR (2009)Google Scholar
  10. 10.
    Aciiçmez, O., Koç, Ç.K., Seifert, J.: On the power of simple branch prediction analysis. In: Proceedings of the 2nd ACM Symposium on information, Computer and Communications Security (ASIACCS’07) (2007)Google Scholar
  11. 11.
    Kernighan B.W., Lin S.: An efficient heuristic procedure for partition graphs. Bell Syst. Tech. J. 49, 291–307 (1970)zbMATHGoogle Scholar
  12. 12.
    Christodorescu, M., Jha, S., Kruegel, C.: Mining specifications of malicious behavior. In: Proceedings of the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (2007)Google Scholar
  13. 13.
  14. 14.
    Lamport L.: Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21(7), 558–565 (1978)zbMATHCrossRefGoogle Scholar
  15. 15.
    Jiang, X., Walters, A., Buchholz, F., Xu, D., Wang, Y.M., Spafford, E.H.: Provenance-Aware Tracing of Worm Break-in and Contaminations: A Process Coloring Approach. In: Proceedings of 26th IEEE Int’l Conf. Distributed Computing Systems (ICDCS’06) (2006)Google Scholar
  16. 16.
    Fletcher, T.: Sharing a File Descriptor Between Processes.
  17. 17.
    Yin, H., Song, D., Manuel, E., Kruegel, C., Kirda, E.: Panorama: Capturing system-wide information flow for malware detection and analysis. In: Proceedings of the 14th ACM Conferences on Computer and Communication Security (2007)Google Scholar
  18. 18.
    King, S.T., Chen, P.M.: Backtracking Intrusions. In: Proceedings of the 2003 Symposium on Operating Systems Principles, pp. 223–236 (2003)Google Scholar
  19. 19.
    Kirda, E., Kruegel, C., Banks, G., Vigna, G., Kemmerer, R.: Behavior-based Spyware Detection. In: Proceedings of the USENIX Security Symposium (2006)Google Scholar
  20. 20.
    Cohen F.: Computer viruses: theory and experiments. Comput. Secur. 6(1), 22–35 (1987)CrossRefGoogle Scholar
  21. 21.
    Phoenix. Scholar
  22. 22.
    Cavallaro, L., Saxena, P., Sekar, R.: On the limits of information flow techniques for malware analysis and containment. In: Proceedings of 5th international conference on Detection of Intrusions and Malware, and Vulnerability Assessment (2008)Google Scholar
  23. 23.
    Szor P.: The Art of Computer Virus Research and Defense. Addison-Wesley Professional, Reading (2005)Google Scholar
  24. 24.
    Forrest, S., Perelson, A.S., Allen, L., Cherukuri, R., Self-Nonself Discrimination in a Computer. In: Proceedings of IEEE Symposium on Security & Privacy (1994)Google Scholar
  25. 25.
    Stinson, E., Mitchell, J.C.: Characterizing Bots’ Remote Control Behavior. In: Detection of Intrusions & Malware, and Vulnerability Assessment (2007)Google Scholar
  26. 26.
    Willems, C., Holz, T., Freiling, F.: Toward Automated Dynamic Malware Analysis Using CWSandbox. In: Proceedings of IEEE Security and Privacy (2007)Google Scholar
  27. 27.
    Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Automating mimicry attacks using static binary analysis. In: Proceedings of the 14th conference on USENIX Security Symposium (2005)Google Scholar
  28. 28.
    Norman Sandbox Whitepaper.
  29. 29.
    Srivastava, A., Lanzi, A., Giffin, J.: System Call API Obfuscation. In: Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection (2008)Google Scholar
  30. 30.
    Rieck, K., Holz, T., Willems, C., Düssel, P., Laskov, P.: Learning and Classification of Malware Behavior. In: Proceedings of Detection of Intrusions and Malware, and Vulnerability Assessment (2008)Google Scholar
  31. 31.
    Percival, C.: Cache missing for fun and profit. BSDCan (2005).
  32. 32.
    Stevens R.: UNIX Network Programming, 2nd edn. Interprocess Communications, vol. 2. Prentice Hall, Englewood Cliffs (1999)Google Scholar
  33. 33.
    Dyshlevoi, K.V., Kamensky, V.E., Solovskaya, L.B.: Marshalling In Distributed Systems: Two Approaches (1997).
  34. 34.
    Borello J., Mé L.: Code obfuscation techniques for metamorphic viruses. J. Comput. Virol. 4, 211–220 (2008). doi: 10.1007/s11416-008-0084-2 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag France 2011

Authors and Affiliations

  • Weiqin Ma
    • 1
  • Pu Duan
    • 1
  • Sanmin Liu
    • 1
  • Guofei Gu
    • 1
  • Jyh-Charn Liu
    • 1
  1. 1.Department of Computer Science and EngineeringTexas A&M UniversityCollege StationUSA

Personalised recommendations