Advertisement

Control Flow Graph Matching for Detecting Obfuscated Programs

  • Chandan Kumar Behera
  • Genius Sanjog
  • D. Lalitha Bhaskari
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 731)

Abstract

Malicious programs like the viruses, worms, Trojan horses, and backdoors infect host computers by taking advantage of flaws of the software and thereby introducing some kind of secret functionalities. The authors of these malicious programs attempt to find new methods to get avoided from detection engines. They use different obfuscation techniques such as dead code insertion, instruction substitution to make the malicious programs more complex. Initially, obfuscation techniques those are used by software developers to protect their software from piracy are now misused by these malware authors. This paper intends to detect such obfuscated programs or malware using control flow graph (CFG) matching technique, using VF2 algorithm. If the original CFG of the executable is found to be isomorphic to subgraph of obfuscated CFG (under examination), then it can be classified as an obfuscated one.

Keywords

Obfuscation Decompilation Optimization Graph isomorphism Control flow graph 

References

  1. 1.
    You, I., Yim, K.: Malware obfuscation techniques: a brief survey. In: International Conference on Broadband, Wireless Computing, Communication and Applications, IEEE Computer Society, pp. 297–300 (2010)Google Scholar
  2. 2.
    Sharif, M., et al.: Impeding malware analysis using conditional code obfuscation. In: Network and Distributed System Security Symposium (2008)Google Scholar
  3. 3.
    Walenstein, A., Lakhotia, A.: A transformation-based model of malware derivation. In: 7th IEEE International Conference on Malicious and Unwanted Software, pp. 17–25 (2012)Google Scholar
  4. 4.
    Durfina, L., Kroustek, J., Zemek, P.: Psyb0t malware: a step-by-step decompilation—case study. In: Working Conference on Reverse Engineering (WCRE), pp. 449–456. IEEE Computer Society (2013)Google Scholar
  5. 5.
    Ernst, M., et al.: Quickly detecting relevant program invariants. In: 22nd International Conference on Software Engineering, pp. 449–458 (2000) Google Scholar
  6. 6.
    Cordella, L.P., Foggia, P., Sansone, C., Vento, M.: Evaluating performance of the VF graph matching algorithm. In: Proceedings of the 10th International Conference on Image Analysis and Processing, pp. 1172–1177. IEEE Computer Society Press (1999)Google Scholar
  7. 7.
    Cordella, L.P., Foggia, P., Sansone, C., Vento, M.: An improved algorithm for matching large graphs. In: 3rd International Workshop on Graph-based Representations, Italy (2001)Google Scholar
  8. 8.
    McKay, B.D.: Practical graph isomorphism. Congressus Numerantium 30, 45–87 (1981)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Messmer, B.T., Bunke, H.: A decision tree approach to graph and subgraph isomorphism detection. J. Pattern Recog. 32, 1979–1998 (1999)CrossRefGoogle Scholar
  10. 10.
    Gold, R.: Reductions of control flow graphs. Int. J. Comput., Electr. Autom. Control Inf. Eng. 8(3), (2014)Google Scholar
  11. 11.
    Sadiq, W., Orlowska, M.E.: Analyzing process models using graph reduction techniques. Inf. Syst. 25(2), 117–134 (2000)CrossRefGoogle Scholar
  12. 12.
    Bondy, J.A., Murty. U.S.R.: Graph Theory. Springer, Berlin (2008)CrossRefGoogle Scholar
  13. 13.
    Abadi, M., Budiu, M., Erlingsson, U’., Ligatti, J.: Control flow integrity principles, implementations, and applications. ACM Trans. Inf. Syst. Secur. 13(1), 4:1–4:40 (2009)CrossRefGoogle Scholar
  14. 14.
    Brunel, J., Doligez, D., Hansen, R.R., Lawall, J.L., Muller, G.: A foundation for flow-based program matching, using temporal logic and model checking POPL. ACM (2009)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • Chandan Kumar Behera
    • 1
  • Genius Sanjog
    • 1
  • D. Lalitha Bhaskari
    • 1
  1. 1.Department of Computer Science & Systems EngineeringAndhra UniversityVisakhapatnamIndia

Personalised recommendations