Skip to main content

Environment–Reactive Malware Behavior: Detection and Categorization

  • Conference paper
  • First Online:
Data Privacy Management, Autonomous Spontaneous Security, and Security Assurance (DPM 2014, QASA 2014, SETOP 2014)

Abstract

Present malicious threats have been consolidated in past few years by incorporating diverse stealthy techniques. Detecting these malwares on the basis of their dynamic behavior has become a potential approach as it suppresses the shortcomings of static approaches raised due to the obfuscated malware binaries. Additionally, existing behavior based malware detection approaches are resilient to zero–day malware attacks. These approaches rely on isolated analysis environment to monitor and capture the run–time malware behavior. Malware bundled with environment–aware payload may degrade detection accuracy of such approaches. These malicious programs detect the presence of execution environment and thus inspite of having their malicious payload they mimic a benign behavior to avoid detection. In this paper, we have presented an approach using system–calls to identify a malware on the basis of their malignant and environment–reactive behavior. The proposed approach offers an automated screening mechanism to segregate malware samples on the basis of aforementioned behaviors. We have built a decision model which is based on multi–layer perceptron learning with back propagation algorithm. Our proposed model decides the candidacy of a sample to be put into one of the four classes (clean, malignant, guest–crashing and infinite–running). Clean behavior denotes benign sample and rest of the behaviors denote the presence of malware sample. The proposed technique has been evaluated with known and unknown instances of real malware and benign programs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Anderson, B., Quist, D., Neil, J., Storlie, C., Lane, T.: Graph-based malware detection using dynamic analysis. J. Comput. Virol. 7(4), 247–258 (2011)

    Article  Google Scholar 

  2. Balzarotti, D., Cova, M., Karlberger, C., Kirda, E., Kruegel, C., Vigna, G.: Efficient detection of split personalities in malware. In: Proceedings of the Network and Distributed System Security Symposium, NDSS, San Diego, California, USA, pp. 1–16 (2010)

    Google Scholar 

  3. Bethencourt, J., Song, D., Waters, B.: Analysis-resistant malware. In: Proceedings of the Network and Distributed System Security Symposium, NDSS, San Diego, California, USA, pp. 1–13 (2008)

    Google Scholar 

  4. Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D., Yin, H.: Automatically identifying trigger-based behavior in malware. In: Lee, W., Wang, C., Dagon, D. (eds.) Botnet Detection. Advances in Information Security, vol. 36, pp. 65–88. Springer, New York (2008)

    Chapter  Google Scholar 

  5. Chen, X., Andersen, J., Mao, Z., Bailey, M., Nazario, J.: Towards an understanding of anti-virtualization and anti-debugging behavior in modern malware. In: Dependable Systems and Networks With FTCS and DCC, DSN, pp. 177–186, June 2008

    Google Scholar 

  6. Chester, D.L.: Why two hidden layers are better than one. In: Proceedings of the International Joint Conference on Neural Networks, IJCNN 1990, Washington, DC (1990)

    Google Scholar 

  7. Dinaburg, A., Royal, P., Sharif, M., Lee, W.: Ether: Malware analysis via hardware virtualization extensions. In: Proceedings of the 15th ACM Conference on Computer and Communications Security, CCS 2008, pp. 51–62. ACM, New York (2008)

    Google Scholar 

  8. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989)

    Article  Google Scholar 

  9. J00ru: Windows win32k.sys system call table, April 2014

    Google Scholar 

  10. Jacob, G., Hund, R., Kruegel, C., Holz, T.: Jackstraws: picking command and control connections from bot traffic. In: Proceedings of the 20th USENIX Conference on Security, SEC 2011, pp. 29–48. USENIX Association, Berkeley (2011)

    Google Scholar 

  11. Kang, M.G., Yin, H., Hanna, S., McCamant, S., Song, D.: Emulating emulation-resistant malware. In: Proceedings of the 1st ACM Workshop on Virtual Machine Security, VMSec 2009. ACM, New York, pp. 11–22 (2009)

    Google Scholar 

  12. Kevin, L., Bryce, D., David, G., Volker, R., Christophe, B.: Bochs user manual (2010)

    Google Scholar 

  13. Kolbitsch, C., Comparetti, P.M., Kruegel, C., Kirda, E., Zhou, X., Wang, X.: Effective and efficient malware detection at the end host. In: Proceedings of the 18th Conference on USENIX Security Symposium, SSYM 2009, pp. 351–366. USENIX Association (2009)

    Google Scholar 

  14. Lindorfer, M., Kolbitsch, C., Milani Comparetti, P.: Detecting environment-sensitive malware. In: Sommer, R., Balzarotti, D., Maier, G. (eds.) RAID 2011. LNCS, vol. 6961, pp. 338–357. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  15. Mark, R., David A, s., Alex, L.: Windows internal part 2

    Google Scholar 

  16. Moser, A., Kruegel, C., Kirda, E.: Exploring multiple execution paths for malware analysis. In: Proceedings of the 2007 IEEE Symposium on Security and Privacy, SP 2007, Washington, DC, pp. 231–245 (2007)

    Google Scholar 

  17. Ou, G., Murphey, Y.L.: Multi-class pattern classification using neural networks. Pattern Recogn. 40(1), 4–18 (2007)

    Article  MATH  Google Scholar 

  18. Park, Y., Reeves, D.S., Stamp, M.: Deriving common malware behavior through graph clustering. Comput. Secur. 39, 419–430 (2013)

    Article  Google Scholar 

  19. Pék, G., Bencsáth, B., Buttyán, L.: nEther: in-guest detection of out-of-the-guest malware analyzers. In: Proceedings of the Fourth European Workshop on System Security, EUROSEC 2011, pp. 3:1–3:6. ACM, New York (2011)

    Google Scholar 

  20. Quist, D., Liebrock, L., Neil, J.: Improving antivirus accuracy with hypervisor assisted analysis. J. Comput. Virol. 7(2), 121–131 (2011)

    Article  Google Scholar 

  21. Rieck, K., Holz, T., Willems, C., Düssel, P., Laskov, P.: Learning and classification of malware behavior. In: Zamboni, D. (ed.) DIMVA 2008. LNCS, vol. 5137, pp. 108–125. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  22. Rutkowska, J.: Red pill... or how to detect vmm using (almost) one cpu instruction

    Google Scholar 

  23. Srivastava, A., Lanzi, A., Giffin, J.T.: System call API obfuscation (extended abstract). In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 421–422. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  24. Sun, M.K., Lin, M.J., Chang, M., Laih, C.S., Lin, H.T.: Malware virtualization-resistant behavior detection. In: Proceedings of the 17th International Conference on Parallel and Distributed Systems (IEEE), ICPADS 2011, Washington, DC, USA, pp. 912–917 (2011)

    Google Scholar 

  25. Vinod, P., Laxmi, V., Gaur, M.S.: REFORM: relevant feature for malware analysis. In: Proceedings of Sixth IEEE International Conference of Security and Multimodality in Pervasive Environment (SMPE 2012), pp. 26–29. Fukuoka Institute of technology (FIT), Fukuoka, Japan (2012)

    Google Scholar 

Download references

Acknowledgments

Mauro Conti is supported by a Marie Curie Fellowship funded by the European Commission under the agreement No. PCIG11-GA-2012-321980. This work is also partially supported by the TENACE PRIN Project 20103P34XC funded by the Italian MIUR, and by the Project “Tackling Mobile Malware with Innovative Machine Learning Techniques” funded by the University of Padua.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Smita Naval .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Naval, S., Laxmi, V., Gaur, M.S., Raja, S., Rajarajan, M., Conti, M. (2015). Environment–Reactive Malware Behavior: Detection and Categorization. In: Garcia-Alfaro, J., et al. Data Privacy Management, Autonomous Spontaneous Security, and Security Assurance. DPM QASA SETOP 2014 2014 2014. Lecture Notes in Computer Science(), vol 8872. Springer, Cham. https://doi.org/10.1007/978-3-319-17016-9_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-17016-9_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-17015-2

  • Online ISBN: 978-3-319-17016-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics