Skip to main content

Diversification of System Calls in Linux Binaries

Part of the Lecture Notes in Computer Science book series (LNSC,volume 9473)

Abstract

This paper studies the idea of using large-scale diversification to protect operating systems and make malware ineffective. The idea is to first diversify the system call interface on a specific computer so that it becomes very challenging for a piece of malware to access resources, and to combine this with the recursive diversification of system library routines indirectly invoking system calls. Because of this unique diversification (i.e. a unique mapping of system call numbers), a large group of computers would have the same functionality but differently diversified software layers and user applications. A malicious program now becomes incompatible with its environment. The basic flaw of operating system monoculture – the vulnerability of all software to the same attacks – would be fixed this way.

Specifically, we analyze the presence of system calls in the ELF binaries. We study the locations of system calls in the software layers of Linux and examine how many binaries in the whole system use system calls. Additionally, we discuss the different ways system calls are coded in ELF binaries and the challenges this causes for the diversification process. Also, we present a diversification tool and suggest several solutions to overcome the difficulties faced in system call diversification. The amount of problematic system calls is small, and our diversification tool manages to diversify the clear majority of system calls present in standard-like Linux configurations. For diversifying all the remaining system calls, we consider several possible approaches.

Keywords

  • Cal Systems
  • System Call Number
  • Diversification Tool
  • Malware
  • Invoke System Calls

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This research has been funded by MATINE project 3301.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-27998-5_2
  • Chapter length: 21 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   59.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-27998-5
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   79.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.

References

  1. Apvrille, A., Strazzere, T.: Reducing the window of opportunity for android malware gotta catch ’em all. Int. J. Ambient Comput. Intell. 8(1–2), 61–71 (2012)

    Google Scholar 

  2. Bruschi, D., Cavallaro, L., Lanzi, A.: An efficient technique for preventing mimicry and impossible paths execution attacks. In: Performance, Computing, and Communications Conference, 2007, IPCCC 2007. IEEE Internationa, pp. 418–425, April 2007

    Google Scholar 

  3. Chew, M., Song, D.: Mitigating buffer overflows by operating system randomization (2002)

    Google Scholar 

  4. Cohen, F.B.: Operating system protection through program evolution. Comput. Secur. 12(6), 565–584 (1993)

    CrossRef  Google Scholar 

  5. Collberg, C., Thomborson, C., Low, D.: A taxonomy of obfuscation tranformations. Technical report 148, The University of Auckland (1997)

    Google Scholar 

  6. TIS Committee: Tool Interface Standard. Executable and Linking Format (ELF) Specification. Version 1.2. Submitted to Journal of Information Security and Applications (Elsevier), under evaluation (1995)

    Google Scholar 

  7. Cooper, K.D., Harvey, T.J., Waterman, T.: Building a control-flow graph from scheduled assembly code. Technical report 02–399, Rice University (2002)

    Google Scholar 

  8. Falcarin, P., Carlo, S.D., Cabutto, A., Garazzino, N., Barberis, D.: Exploiting code mobility for dynamic binary obfuscation. In 2011 World Congress on Internet Security (WorldCIS), pp. 114–120, February 2011

    Google Scholar 

  9. Jang, M.H., Jang, M.: Security Strategies in Linux Platforms and Applications. Jones & Bartlett Publishers, Burlington (2010)

    Google Scholar 

  10. Jiang, X., Wang, H.J., Xu, D., Wang, Y.-M.: Randsys: thwarting code injection attacks with system service interface randomization. In: IEEE International Symposium on Reliable Distributed Systems, SRDS 2007, pp. 209–218 (2007)

    Google Scholar 

  11. Kerrisk, M.: The Linux Programming Interface. No Starch Press, San Francisco (2010)

    Google Scholar 

  12. Kinder, J., Zuleger, F., Veith, H.: An abstract interpretation-based framework for control flow reconstruction from binaries. In: Jones, N.D., Müller-Olm, M. (eds.) VMCAI 2009. LNCS, vol. 5403, pp. 214–228. Springer, Heidelberg (2009)

    CrossRef  Google Scholar 

  13. Liang, Z., Liang, B., Li, L.: A system call randomization based method for countering code injection attacks. In: International Conference on Networks Security, Wireless Communications and Trusted Computing, NSWCTC 2009, pp. 584–587 (2009)

    Google Scholar 

  14. Linn, C., Debray, S.: Obfuscation of executable code to improve resistance to static disassembly. In: Proceedings of the 10th ACM Conference on Computer and Communications Security, CCS 2003, pp. 290–299. ACM, New York, USA (2003)

    Google Scholar 

  15. Madou, M., Anckaert, B., De Bus, B., De Bosschere, K., Cappaert, J., Preneel, B.: On the effectiveness of source code transformations for binary obfuscation. In: Proceedings of the International Conference on Software Engineering Research and Practice (SERP06), pp. 527–533. CSREA Press (2006)

    Google Scholar 

  16. Popov, I.V., Debray, S.K., Andrews, G.R.: Binary obfuscation using signals. In: USENIX Security (2007)

    Google Scholar 

  17. S. Rauti, J. Holvitie, and V. Leppänen. Towards a Diversification Framework for Operating System Protection. In: Proceedings of International Conference on Computer Systems and Technologies, CompSysTech 2014 (2014)

    Google Scholar 

  18. Rauti, S., Leppänen, V.: Browser extension-based man-in-the-browser attacks against Ajax applications with countermeasures. In: Proceedings of International Conference on Computer Systems and Technologies, CompSysTech 2012, pp. 251–258. ACM Press (2012)

    Google Scholar 

  19. Rauti, S., Leppänen, V.: A proxy-like obfuscator for web application protection. Int. J. Inf. Technol. Secur. 5(1) (2014)

    Google Scholar 

  20. Lee, J.W., Lee, Y.J., Kim, H.K., Hwang, B., Ryu, K.H.: Discovering temporal relation rules mining from interval data. In: Shafazand, H., Tjoa, A.M. (eds.) EurAsia-ICT 2002. LNCS, vol. 2510, pp. 57–66. Springer, Heidelberg (2002)

    CrossRef  Google Scholar 

  21. Rauti, S., Leppänen, V.: Resilient code protection by JavaScript and HTML obfuscation for Ajax applications against man-in-the-browser attacks. Submitted to Journal of Information Security and Applications (Elsevier), under evaluation (2014)

    Google Scholar 

  22. Schwarz, B., Debray, S., Andrews, G.: Disassembly of executable code revisited. In: Proceedings of Ninth Working Conference on Reverse Engineering, pp. 45–54 (2002)

    Google Scholar 

  23. Sobell, M.G.: A Practical Guide to Linux. Addison-Wesley, Boston (1999)

    Google Scholar 

  24. Srivastava, A., Lanzi, A., Giffin, J., Balzarotti, D.: Operating system interface obfuscation and the revealing of hidden operations. In: Holz, T., Bos, H. (eds.) DIMVA 2011. LNCS, vol. 6739, pp. 214–233. Springer, Heidelberg (2011)

    CrossRef  Google Scholar 

  25. Tanenbaum, A.S.: Modern Operating Systems, 3rd edn. Prentice Hall Press, Upper Saddle River (2007)

    Google Scholar 

  26. Theiling, H.: Extracting safe and precise control flow from binaries. In: Proceedings of Seventh International Conference on Real-Time Computing Systems and Applications, pp. 23–30. IEEE (2000)

    Google Scholar 

  27. Wang, S.P.: Mastering Linux. CRC Press, Boca Raton (2011)

    Google Scholar 

  28. Wu, Z., Gianvecchio, S., Xie, M., Wang, H.: Mimimorphism: a new approach to binary code obfuscation. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS 2010, pp. 536–546. ACM, New York, USA (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sampsa Rauti .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Rauti, S., Laurén, S., Hosseinzadeh, S., Mäkelä, JM., Hyrynsalmi, S., Leppänen, V. (2015). Diversification of System Calls in Linux Binaries. In: Yung, M., Zhu, L., Yang, Y. (eds) Trusted Systems. INTRUST 2014. Lecture Notes in Computer Science(), vol 9473. Springer, Cham. https://doi.org/10.1007/978-3-319-27998-5_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-27998-5_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-27997-8

  • Online ISBN: 978-3-319-27998-5

  • eBook Packages: Computer ScienceComputer Science (R0)