A Three-Way Decision Making Approach to Malware Analysis

  • Mohammad Nauman
  • Nouman Azam
  • JingTao Yao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9436)


Malware analysis techniques generally classify software behaviors as malicious (i.e., harmful) or benign (i.e., not harmful). Due to ambiguous nature of application behavior, there are cases where it may not be possible to confidently reach two-way conclusions. This may result in higher classification errors which in turn affect users trust on malware analysis outcomes. In this paper, we investigate a three-way decision making approach based on probabilistic rough set models, such as, information-theoretic rough sets and game-theoretic rough sets, for malware analysis. The essential idea is to add a third option of deferment or delaying a decision whenever the available information is not sufficient to reach certain conclusions. We demonstrate the applicability of the proposed approach with an example from system call sequences of a vulnerable Linux application.


Nash Equilibrium System Call Strategy Profile Game Solution Threshold Pair 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was partially supported by a discovery grant from NSERC canada.


  1. 1.
    Azam, N., Yao, J.T.: Analyzing uncertainties of probabilistic rough set regions with game-theoretic rough sets. Int. J. Approx. Reasoning 55(1), 142–155 (2014)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Biddle, R., van Oorschot, P.C., Patrick, A.S., Sobey, J., Whalen, T.: Browser interfaces and extended validation ssl certificates: an empirical study. In: Proceedings of the 2009 ACM Workshop on Cloud Computing Security, pp. 19–30 (2009)Google Scholar
  3. 3.
    Demchenko, Y., Ngo, C., de Laat, C., Membrey, P., Gordijenko, D.: Big security for big data: addressing security challenges for the big data infrastructure. In: Jonker, W., Petković, M. (eds.) SDM 2013. LNCS, vol. 8425, pp. 76–94. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  4. 4.
    Deng, X., Yao, Y.: An information-theoretic interpretation of thresholds in probabilistic rough sets. In: Li, T., Nguyen, H.S., Wang, G., Grzymala-Busse, J., Janicki, R., Hassanien, A.E., Yu, H. (eds.) RSKT 2012. LNCS, vol. 7414, pp. 369–378. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  5. 5.
    Egele, M., Scholte, T., Kirda, E., Kruegel, C.: A survey on automated dynamic malware-analysis techniques and tools. ACM Comput. Surv. 44(2), 6 (2012)CrossRefGoogle Scholar
  6. 6.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: UNM dataset. Accessed 6 April 2015
  7. 7.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: IEEE Symposium on Security and Privacy, pp. 120–128 (1996)Google Scholar
  8. 8.
    Gandotra, E., Bansal, D., Sofat, S.: Malware analysis and classification: a survey. J. Inf. Secur. 5, 56–64 (2014)CrossRefGoogle Scholar
  9. 9.
    Herbert, J.P., Yao, J.T.: Game-theoretic rough sets. Fundamenta Informaticae 108(3–4), 267–286 (2011)zbMATHMathSciNetGoogle Scholar
  10. 10.
    Leyton-Brown, K., Shoham, Y.: Essentials of Game Theory: A Concise Multidisciplinary Introduction. Morgan & Claypool Publishers, San Rafael (2008) Google Scholar
  11. 11.
    Liu, D., Yao, Y.Y., Li, T.R.: Three-way investment decisions with decision-theoretic rough sets. Int. J. Comput. Intel. Syst. 4(1), 66–74 (2011)CrossRefGoogle Scholar
  12. 12.
    Mehdi, B., Ahmed, F., Khayyam, S.A., Farooq, M.: Towards a theory of generalizing system call representation for in-execution malware detection. In: IEEE International Conference on Communications (ICC), pp. 1–5 (2010)Google Scholar
  13. 13.
    Pawlak, Z.: Rough sets. Int. J. Comput. Inf. Sci. 11, 241–256 (1982)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Symantec: Internet security threat report, vol. 19. Accessed 12 April 2015
  15. 15.
    Tanenbaum, A.S., Woodhull, A.S.: Operating systems: design and implementation, vol. 2. Prentice-Hall, Englewood Cliffs (1987) Google Scholar
  16. 16.
    Wright, C., Cowan, C., Morris, J., Smalley, S., Kroah-Hartman, G.: Linux security module framework. In: Ottawa Linux Symposium, vol. 8032, pp. 6–16 (2002)Google Scholar
  17. 17.
    Yang, H., Li, T., Hu, X., Wang, F., Zou, Y.: A survey of artificial immune system based intrusion detection. The Scientific World Journal (2014)Google Scholar
  18. 18.
    Yao, Y.Y.: Probabilistic rough set approximations. Int. J. Approx. Reasoning 49(2), 255–271 (2008)zbMATHCrossRefGoogle Scholar
  19. 19.
    Yao, Y.Y.: Three-way decisions with probabilistic rough sets. Inf. Sci. 180(3), 341–353 (2010)CrossRefGoogle Scholar
  20. 20.
    Yao, Y.Y.: Two semantic issues in a probabilistic rough set model. Fundamenta Informaticae 108(3–4), 249–265 (2011)zbMATHMathSciNetGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.National University of Computer and Emerging SciencesPeshawarPakistan
  2. 2.Department of Computer ScienceUniversity of ReginaReginaCanada

Personalised recommendations