Skip to main content
SpringerLink
Log in

[m]allotROPism: a metamorphic engine for malicious software variation development

  • regular contribution
  • Published:
International Journal of Information Security Aims and scope Submit manuscript

Abstract

For decades, code transformations have been a vital open problem in the field of system security, especially for cases like malware mutation engines that generate semantically equivalent forms of given malicious payloads. While there are abundant works on malware and on malware phylogenies classification and detection in general, the fundamental principles about malicious transformations to evade detection have been neglected. In the present work, we introduce a mutation engine, named [m]allotROPism, to generate malicious code deviations with equivalent semantics from a static-analysis point of view. To achieve this, we reduce the problem of generating semantically equivalent solutions of given assembly code into a decision problem, and we solve it with the aid of satisfiability modulo theories. Moreover, we leverage return-oriented programming techniques to alter the traditional execution control flow from text to stack memory segment. We have implemented our proposed mutation engine and evaluated its detection evasion capabilities. Results show that so far, our approach is undetectable against popular free and commercial anti-malware products. We release the implementation of [m]allotROPism as open source. Our intention is to provide a method to generate malware families for experimental purposes and inspire further state-of-the-art research in the field of malware analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. In chemistry, the ability of an element to exist in more than one physical form without change of state is called allotropism.

  2. https://gitlab.ds.unipi.gr/systems-security-laboratory/mallotropism.

  3. https://github.com/RolfRolles/SynesthesiaYS.

  4. https://github.com/RUB-SysSec/syntia.

  5. Introduced initially to execute Unix Shell commands and it is usually written in machine code.

  6. https://gitlab.ds.unipi.gr/systems-security-laboratory/mallotropism.

  7. https://github.com/gpoulios/ROPInjector.

References

  1. Shacham, H.: The geometry of innocent flesh on the bone: return-into-libc without function calls (on the x86). In: Proceedings of the 14th ACM Conference on Computer and Communications Security, pp. 552–561. ACM, New York (2007)

  2. Bauer, J.M., Van Eeten, M.J., Chattopadhyay, T., Wu, Y.: Itu study on the financial aspects of network security: Malware and spam. ICT Applications and Cybersecurity Division, International Telecommunication Union, Final Report (July 2008)

  3. PandaLabs, 2017 in Figures: The Exponential Growth of Malware (Accessed August 2, 2018). https://www.pandasecurity.com/mediacenter/malware/2017-figures/

  4. Schwartz, E.J., Avgerinos, T., Brumley, D.: Q: exploit hardening made easy. In: USENIX Security Symposium, pp. 25–41 (2011)

  5. Dullien, T., Kornau, T., Weinmann, R.-P.: A framework for automated architecture-independent gadget search. In: 4th USENIX Workshop on Offensive Technologies (WOOT 10) (2010)

  6. Ma, H., Lu, K., Ma, X., Zhang, H., Jia C., Gao, D.: Software watermarking using return-oriented programming. In: Proceedings of the 10th ACM Symposium on Information, Computer and Communications Security, ASIA CCS’15. ACM, New York, NY, USA, pp. 369–380 (2015)

  7. Lu, K., Xiong, S., Gao, D.: Ropsteg: program steganography with return oriented programming. In: Proceedings of the 4th ACM Conference on Data and Application Security and Privacy, CODASPY’14. ACM, New York, NY, USA, pp. 265–272 (2014)

  8. Mu, D., Guo, J., Ding, W., Wang, Z., Mao, B., Shi, L.: Ropob: obfuscating binary code via return oriented programming. In: Security and Privacy in Communication Networks. Springer International Publishing, London (2018)

  9. Weidler, N.R., Brown, D., Mitchell, S.A., Anderson, J., Williams, J.R., Costley, A., Kunz, C., Wilkinson, C., Wehbe, R., Gerdes, R.: Return-oriented programming on a resource constrained device. Sustain. Comput. Inf. Syst. 22, 244–256 (2019)

    Google Scholar 

  10. Mohan, V., Hamlen, K.W.: Frankenstein: a tale of horror and logic programming. Book Reviews (02) (2017)

  11. Mohan, V., Hamlen, K.W.: Frankenstein: stitching malware from benign binaries. In: 21s USENIX Workshop on Offensive Technologies (WOOT 12), Austin, TX, pp. 77–84 (2012)

  12. Poulios, G., Ntantogian, C., Xenakis, C.: Ropinjector: using return oriented programming for polymorphism and antivirus evasion, Blackhat USA (2015)

  13. Ming, J., Xu, D., Jiang, Y., Wu, D.: Binsim: trace-based semantic binary diffing via system call sliced segment equivalence checking. In: 26th USENIX Security Symposium (2017)

  14. Jha, S., Gulwani, S., Seshia, S.A., Tiwari, A.: Oracle-guided component-based program synthesis. In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1. ACM, New York, pp. 215–224 (2010)

  15. Rolles, R.: Synesthesia: a modern approach to shellcode generation (2016). http://www.msreverseengineering.com/blog/2016/11/8/synesthesia-modern-shellcode-synthesis-ekoparty-2016-talk

  16. Dutertre, B., De Moura, L.: The yices smt solver, Tool paper at SRI. International 2(2), 1–5 (2006)

    Google Scholar 

  17. Blazytko, T., Contag, M., Aschermann, C., Holz, T.: Syntia: synthesizing the semantics of obfuscated code. In: USENIX Security Symposium (2017)

  18. Shacham, H., Page, M., Pfaff, B., Goh, E.-J., Modadugu, N., Boneh, D.: On the effectiveness of address-space randomization. In: Proceedings of the 11th ACM Conference on Computer and Communications Security, pp. 298–307. ACM, New York (2004)

  19. Abadi, M., Budiu, M., Erlingsson, U., Ligatti, J.: Control-flow integrity principles. ACM Trans. Inf. Syst. Secur. 13(1), 4:1-4:40 (2009)

    Article  Google Scholar 

  20. Carlini, N., Wagner, D.: \(\{\)ROP\(\}\) is still dangerous: breaking modern defenses. In: 23rd USENIX Security Symposium, pp. 385–399 (2014)

  21. Schaefer, T.J.: The complexity of satisfiability problems. In: Proceedings of the 10th Annual ACM Symposium on Theory of Computing, pp. 216–226. ACM, New York (1978)

  22. De Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pp. 337–340. Springer, Berlin (2008)

  23. Park, D., Zhang, Y., Saxena, M., Daian, P., Roşu, G.: A formal verification tool for ethereum vm bytecode. In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 912–915. ACM, New York (2018)

  24. Vanhoef, M., Piessens, F.: Symbolic execution of security protocol implementations: handling cryptographic primitives. In: 12th USENIX Workshop on Offensive Technologies (WOOT 18) (2018)

  25. Vanegue, J., Heelan, S., Rolles, R.: SMT solvers in software security. In: 6th USENIX Workshop on Offensive Technologies (WOOT 12) (2012)

  26. Bornholt, J.: Program synthesis, explained (Accessed February 2, 2018). https://homes.cs.washington.edu/bornholt/post/synthesis-for-architects.html (2018)

  27. Szor, P.: The Art of Computer Virus Research and Defense. Pearson Education, London (2005)

    Google Scholar 

  28. O’Kane, P., Sezer, S., McLaughlin, K.: Obfuscation: the hidden malware. IEEE Secur. Privacy 9(5), 41–47 (2011)

    Article  Google Scholar 

  29. Spafford, E.H.: The internet worm program: an analysis. ACM SIGCOMM Comput. Commun. Rev. 19(1), 17–57 (1989)

    Article  Google Scholar 

  30. Wartell, R., Mohan, V., Hamlen, K.W., Lin, Z.: Binary stirring: self-randomizing instruction addresses of legacy x86 binary code. In: Proceedings of the 2012 ACM Conference on Computer and Communications Security, pp. 157–168. ACM, New York (2012)

  31. Pappas, V., Polychronakis, M., Keromytis, A.D.: Smashing the gadgets: hindering return-oriented programming using in-place code randomization. In: 2012 IEEE Symposium on Security and Privacy (SP), pp. 601–615. IEEE, New York (2012)

  32. Ispoglou, K.K., Payer, M.: Malwash: washing malware to evade dynamic analysis. In: 10th USENIX Workshop on Offensive Technologies (WOOT 16) (2016)

  33. Ming, J., Xin, Z., Lan, P., Wu, D., Liu, P., Mao, B.: Replacement attacks: automatically impeding behavior-based malware specifications. In: Applied Cryptography and Network Security—13th International Conference, ACNS 2015, pp. 497–517. Springer, Berlin (2015)

  34. Cohen, F.: Computer viruses: theory and experiments. Comput. Secur. 6(1), 22–35 (1987)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by: CUREX project of Horizon H2020 Framework Programme of the European Union under Grant agreement No. 826404. SECONDO project of H2020 MSCA RISE 2018 under Grant Agreement No. 823997. SPIDER project of Horizon H2020 Framework Programme of the European Union under Grant Agreement No. 833685. Content reflects only the authors’ view, and EU is not responsible for any use of the information it contains.

Author information

Authors and Affiliations

  1. Department of Digital Systems, University of Piraeus, Piraeus, Greece

    Christos Lyvas & Christos Xenakis

  2. Department of Informatics, Ionian University, Corfu, Greece

    Christoforos Ntantogian

Authors
  1. Christos Lyvas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christoforos Ntantogian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christos Xenakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christos Lyvas.

Ethics declarations

Competing interest

The authors declare that they have no competing interests.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Thanks to the title.

Appendix

Appendix

figure e

x86-64 Instruction Set. Following the technological shift from 32-bit to 64-bit CPUs, in this paper we will consider only the modern x86-64 architecture. In x86-64 assembly, there are sixteen registers: rsp, rbp, rax, rbx, rcx, rdx, rdi, rsi, and r8-r15. Registers rbx, rbp, r12, r15 are caller-saved registers, meaning that they are saved across function calls. In contrast rax, rcx,rdx, rdi, rsi, rsp, r8, r11 are considered caller-saved registers, meaning that they may not be saved across function calls. By convention, register rax is used to store functions return values (if they exist and are no more than 64 bits long). For a typical function invocation, the program should place the first six integer or pointer parameters in the registers rdi, rsi, rdx, rcx, r8, r9 to the called functions. For more than six parameters, their values must be pushed onto the stack with the first argument topmost. rdi, rsi, rdx, rcx, r8, r9.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lyvas, C., Ntantogian, C. & Xenakis, C. [m]allotROPism: a metamorphic engine for malicious software variation development. Int. J. Inf. Secur. 21, 61–78 (2022). https://doi.org/10.1007/s10207-021-00541-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10207-021-00541-y

Keywords

Search

Navigation