Assisted Identification of Mode of Operation in Binary Code with Dynamic Data Flow Slicing

  • Pierre LestringantEmail author
  • Frédéric Guihéry
  • Pierre-Alain Fouque
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9696)


Verification of software security properties, when conducted at the binary code level, is a difficult and cumbersome task. This paper is focused on the reverse engineering task that needs to be performed prior to any thorough analysis. A previous line of work has been dedicated to the identification of cryptographic primitives. Relying on the techniques that have been proposed, we devise a semi-automated solution to identify modes of operation. Our solution produces a concise representation of the data transfers occurring within a cryptographic scheme. Inspired by program slicing techniques, we extract from a dynamic data flow a slice defined as the smallest subgraph that is distance preserving for the set of cryptographic parameters. We apply our solution to several modes of operation including CBC, CTR, HMAC and OCB. For each of them, we successfully obtain a complete and readable representation. Moreover, we show with an example that our solution can be applied on non standard schemes to quickly discover security flaw.


Binary analysis Reverse engineering Cryptography 


  1. 1.
    Albrecht, M.R., Paterson, K.G., Watson, G.J.: Plaintext recovery attacks against SSH. In: 30th IEEE Symposium on Security and Privacy (S&P 2009), pp. 16–26 (2009)Google Scholar
  2. 2.
    Balakrishnan, G., Reps, T.: Analyzing memory accesses in x86 executables. In: Duesterwald, E. (ed.) CC 2004. LNCS, vol. 2985, pp. 5–23. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  3. 3.
    Bayer, U., Comparetti, P.M., Hlauschek, C., Krügel, C., Kirda, E.: Scalable, behavior-based malware clustering. In: Proceedings of the Network and Distributed System Security Symposium, NDSS 2009 (2009)Google Scholar
  4. 4.
    Bellare, M., Jaeger, J., Kane, D.: Mass-surveillance without the state: strongly undetectable algorithm-substitution attacks. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1431–1440 (2015)Google Scholar
  5. 5.
    Bellare, M., Paterson, K.G., Rogaway, P.: Security of symmetric encryption against mass surveillance. IACR Cryptology ePrint Archive 2014, 438 (2014)Google Scholar
  6. 6.
    Bonfante, G., Kaczmarek, M., Marion, J.: Morphological detection of malware. In: 3rd International Conference on Malicious and Unwanted Software, MALWARE 2008, pp. 1–8 (2008)Google Scholar
  7. 7.
    Calvet, J., Fernandez, J.M., Marion, J.: Aligot: cryptographic function identification in obfuscated binary programs. In: The ACM Conference on Computer and Communications Security, CCS 2012, pp. 169–182 (2012)Google Scholar
  8. 8.
    Canvel, B., Hiltgen, A.P., Vaudenay, S., Vuagnoux, M.: Password interception in a SSL/TLS channel. In: Boneh, D. (ed.) CRYPTO 2003. LNCS, vol. 2729, pp. 583–599. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  9. 9.
    Conte, D., Foggia, P., Sansone, C., Vento, M.: Thirty years of graph matching in pattern recognition. IJPRAI 18(3), 265–298 (2004)Google Scholar
  10. 10.
    Coppersmith, D., Elkin, M.: Sparse sourcewise and pairwise distance preservers. SIAM J. Discrete Math. 20(2), 463–501 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Degabriele, J.P., Paterson, K.G.: On the (in)security of IPsec in MAC-then-encrypt configurations. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS 2010, pp. 493–504 (2010)Google Scholar
  12. 12.
    Gröbert, F., Willems, C., Holz, T.: Automated identification of cryptographic primitives in binary programs. In: Sommer, R., Balzarotti, D., Maier, G. (eds.) RAID 2011. LNCS, vol. 6961, pp. 41–60. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  13. 13.
    Hemel, A., Kalleberg, K.T., Vermaas, R., Dolstra, E.: Finding software license violations through binary code clone detection. In: Proceedings of the 8th International Working Conference on Mining Software Repositories, MSR 2011, pp. 63–72 (2011)Google Scholar
  14. 14.
    Katz, J., Schneier, B.: A chosen ciphertext attack against several e-mail encryption protocols. In: 9th USENIX Security Symposium (2000)Google Scholar
  15. 15.
    Khoo, W.M., Mycroft, A., Anderson, R.: Rendezvous: a search engine for binary code. In: Proceedings of the 10th Working Conference on Mining Software Repositories, MSR 2013, pp. 329–338 (2013)Google Scholar
  16. 16.
    Lestringant, P., Guihéry, F., Fouque, P.: Automated identification of cryptographic primitives in binary code with data flow graph isomorphism. In: Proceedings of the 10th ACM Symposium on Information, Computer and Communications Security, ASIA CCS 2015, pp. 203–214 (2015)Google Scholar
  17. 17.
    Li, X., Wang, X., Chang, W.: Cipherxray: exposing cryptographic operations and transient secrets from monitored binary execution. IEEE Trans. Dependable Sec. Comput. 11(2), 101–114 (2014)CrossRefGoogle Scholar
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
    Liskov, M., Rivest, R.L., Wagner, D.: Tweakable block ciphers. In: Yung, M. (ed.) CRYPTO 2002. LNCS, vol. 2442, pp. 31–46. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  23. 23.
    Liu, C., Chen, C., Han, J., Yu, P.S.: GPLAG: detection of software plagiarism by program dependence graph analysis. In: Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 872–881 (2006)Google Scholar
  24. 24.
    Luk, C., Cohn, R.S., Muth, R., Patil, H., Klauser, A., Lowney, P.G., Wallace, S., Reddi, V.J., Hazelwood, K.M.: Pin: building customized program analysis tools with dynamic instrumentation. In: Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation, pp. 190–200 (2005)Google Scholar
  25. 25.
    Paterson, K.G., AlFardan, N.J.: Plaintext-recovery attacks against datagram TLS. In: 19th Annual Network and Distributed System Security Symposium, NDSS 2012 (2012)Google Scholar
  26. 26.
    Rogaway, P., Bellare, M., Black, J., Krovetz, T.: OCB: a block-cipher mode of operation for efficient authenticated encryption. In: Proceedings of the 8th ACM Conference on Computer and Communications Security, CCS 2001, pp. 196–205 (2001)Google Scholar
  27. 27.
    Sæbjørnsen, A., Willcock, J., Panas, T., Quinlan, D.J., Su, Z.: Detecting code clones in binary executables. In: Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, ISSTA 2009, pp. 117–128 (2009)Google Scholar
  28. 28.
  29. 29.
    Tip, F.: A survey of program slicing techniques. J. Prog. Lang. 3(3) (1995).
  30. 30.
    Vaudenay, S.: Security flaws induced by CBC padding - applications to SSL, IPSEC, WTLS. In: Knudsen, L.R. (ed.) EUROCRYPT 2002. LNCS, vol. 2332, pp. 534–545. Springer, Heidelberg (2002)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Pierre Lestringant
    • 1
    • 2
    Email author
  • Frédéric Guihéry
    • 1
  • Pierre-Alain Fouque
    • 2
    • 3
  1. 1.AMOSSYS, R&D Security LabRennesFrance
  2. 2.Université de Rennes 1RennesFrance
  3. 3.Institut Universitaire de FranceParisFrance

Personalised recommendations