Abstract
We describe a novel approach to monitoring high level behaviors using concepts from AI planning. Our goal is to understand what a program is doing based on its system call trace. This ability is particularly important for detecting malware. We approach this problem by building an abstract model of the operating system using the STRIPS planning language, casting system calls as planning operators. Given a system call trace, we simulate the corresponding operators on our model and by observing the properties of the state reached, we learn about the nature of the original program and its behavior. Thus, unlike most statistical detection methods that focus on syntactic features, our approach is semantic in nature. Therefore, it is more robust against obfuscation techniques used by malware that change the outward appearance of the trace but not its effect. We demonstrate the efficacy of our approach by evaluating it on actual system call traces.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
A system call is a mechanism used by a program to request from the operating system services it cannot perform directly, such access to hardware, files, network or memory.
- 2.
A more faithful model will use conditional effects instead, and will also consider their return value.
References
Baker, C.L., Tenenbaum, J.B., Saxe, R.R.: Bayesian models of human action understanding. In: Proceedings of the 18th International Conference on Neural Information Processing Systems, NIPS 2005, pp. 99–106. MIT Press, Cambridge (2005). http://dl.acm.org/citation.cfm?id=2976248.2976261
Beaucamps, P., Gnaedig, I., Marion, J.-Y.: Abstraction-based malware analysis using rewriting and model checking. In: Foresti, S., Yung, M., Martinelli, F. (eds.) ESORICS 2012. LNCS, vol. 7459, pp. 806–823. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33167-1_46
Canali, D., Lanzi, A., Balzarotti, D., Kruegel, C., Christodorescu, M., Kirda, E.: A quantitative study of accuracy in system call-based malware detection. In: ISSTA 2012, New York, NY, USA, pp. 122–132 (2012). https://doi.org/10.1145/2338965.2336768, http://doi.acm.org/10.1145/2338965.2336768
Canzanese, R., Mancoridis, S., Kam, M.: System call-based detection of malicious processes. In: International Conference on Software Quality, Reliability and Security, QRS 2015, pp. 119–124 (2015)
Christodorescu, M., Jha, S., Seshia, S.A., Song, D., Bryant, R.E.: Semantics-aware malware detection. In: Proceedings of the 2005 IEEE Symposium on Security and Privacy, SP 2005, pp. 32–46. IEEE Computer Society, Washington (2005). https://doi.org/10.1109/SP.2005.20
Draios Inc: Sysdig (2012–2016). http://sysdig.com/
Ezzati-Jivan, N., Dagenais, M.R.: A stateful approach to generate synthetic events from kernel traces. Adv. Soft. Eng. 2012, 6:6–6:6 (2012). https://doi.org/10.1155/2012/140368
Firdausi, I., lim, C., Erwin, A., Nugroho, A.S.: Analysis of machine learning techniques used in behavior-based malware detection. In: ACT 2010, pp. 201–203 (2010). https://doi.org/10.1109/ACT.2010.33
Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: IEEE Symposium on Security and Privacy, pp. 120–128, May 1996. https://doi.org/10.1109/SECPRI.1996.502675
Gao, D., Reiter, M.K., Song, D.: Gray-box extraction of execution graphs for anomaly detection. In: Proceedings of the 11th ACM Conference on Computer and Communications Security, CCS 2004, pp. 318–329. ACM, New York (2004). http://doi.acm.org/10.1145/1030083.1030126
Hykes, S.: Docker (2013–2017). http://docker.com/
Jacob, G., Debar, H., Filiol, E.: Malware behavioral detection by attribute-automata using abstraction from platform and language. In: Kirda, E., Jha, S., Balzarotti, D. (eds.) RAID 2009. LNCS, vol. 5758, pp. 81–100. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04342-0_5
Kim, G., Yi, H., Lee, J., Paek, Y., Yoon, S.: Lstm-based system-call language modeling and robust ensemble method for designing host-based intrusion detection systems. arXiv preprint arXiv:1611.01726 (2016)
Liu, A., Martin, C., Hetherington, T., Matzner, S.: A comparison of system call feature representations for insider threat detection. In: Proceedings from the Sixth Annual IEEE SMC Information Assurance Workshop, pp. 340–347, June 2005. https://doi.org/10.1109/IAW.2005.1495972
Long, D.: VAL: The plan validation system (2014). https://github.com/KCL-Planning/VAL
Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., Mitchell, J.C.: A layered architecture for detecting malicious behaviors. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 78–97. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87403-4_5
Mutz, D., Valeur, F., Vigna, G., Kruegel, C.: Anomalous system call detection. ACM Trans. Inf. Syst. Secur. 9(1), 61–93 (2006). https://doi.org/10.1145/1127345.1127348
Poulose Jacob, K., Surekha, M.V.: Anomaly detection using system call sequence sets. J. Software 2(6) (2007)
Ramírez, M., Geffner, H.: Plan recognition as planning. In: IJCAI 2009, pp. 1778–1783 (2009). http://ijcai.org/Proceedings/09/Papers/296.pdf
Rosenberg, I., Gudes, E.: Bypassing system calls-based intrusion detection systems. Concurrency Comput. Pract. Experience 29(16) (2017). https://doi.org/10.1002/cpe.4023
Sukthankar, G., Geib, C., Bui, H., Pynadath, D., Goldman, R.P. (eds.): Plan, Activity, and Intent Recognition. Elsevier (2014)
Tandon, G., Chan, P.K.: On the learning of system call attributes for host-based anomaly detection. Int. J. AI Tools 15(06), 875–892 (2006). https://doi.org/10.1142/S0218213006003028
Tokhtabayev, A., Skormin, V., Dolgikh, A.: Dynamic, resilient detection of complex malicious functionalities in the system call domain. In: MILCOM 2010, pp. 1349–1356, October 2010. https://doi.org/10.1109/MILCOM.2010.5680136
Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: alternative data models. In: Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344), pp. 133–145 (1999). https://doi.org/10.1109/SECPRI.1999.766910
Wressnegger, C., Schwenk, G., Arp, D., Rieck, K.: A close look on n-grams in intrusion detection: Anomaly detection vs. classification. In: Proceedings of the 2013 ACM Workshop on Artificial Intelligence and Security, AISec 2013, pp. 67–76. ACM, New York (2013). https://doi.org/10.1145/2517312.2517316, http://doi.acm.org/10.1145/2517312.2517316
Xu, J., Shelton, C.R.: Intrusion detection using continuous time Bayesian networks. JAIR 39, 745–774 (2010)
You, I., Yim, K.: Malware obfuscation techniques: a brief survey. In: BWCCA 2010, pp. 297–300 (2010). https://doi.org/10.1109/BWCCA.2010.85
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Cukier, A., Brafman, R.I., Perkal, Y., Tolpin, D. (2018). A Planning Approach to Monitoring Computer Programs’ Behavior. In: Dinur, I., Dolev, S., Lodha, S. (eds) Cyber Security Cryptography and Machine Learning. CSCML 2018. Lecture Notes in Computer Science(), vol 10879. Springer, Cham. https://doi.org/10.1007/978-3-319-94147-9_19
Download citation
DOI: https://doi.org/10.1007/978-3-319-94147-9_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-94146-2
Online ISBN: 978-3-319-94147-9
eBook Packages: Computer ScienceComputer Science (R0)