Vigenère scores for malware detection

  • Suchita Deshmukh
  • Fabio Di Troia
  • Mark Stamp
Original Paper


Previous research has applied classic cryptanalytic techniques to the malware detection problem. Specifically, scores that are based on simple substitution cipher cryptanalysis have been considered. In this research, we analyze two malware scoring techniques based on the classic Vigenère cipher. Our first approach relies only on the index of coincidence (IC), which is used for example, to determine the length of the keyword in a Vigenère ciphertext. To compute the IC, we consider both the Kasisky Test and Friedman’s Test. We also consider a score based on a more complete cryptanalysis of a Vigenère cipher, where the IC calculation is the first step. We find that both of these scores outperform comparable malware scores in selected cases.


  1. 1.
    Annachatre, C., Austin, T.H., Stamp, M.: Hidden Markov models for malware classification. J. Comput. Virol. Hacking Tech. 11(2), 59–73 (2015)CrossRefGoogle Scholar
  2. 2.
    Bradley, A.P.: The use of the area under the ROC curve in the evolution of machine learning algorithms. Pattern Recogn. 30, 1145–1159 (1997)CrossRefGoogle Scholar
  3. 3.
    Cryptanalysis of Vigenère Cipher and Substitution Cipher.
  4. 4.
    Deshmukh, S.: Vigenère Cipher for Malware Detection, Master’s Report, Department of Computer Science, San Jose State University (2016)Google Scholar
  5. 5.
  6. 6.
    Fawcett, T.: An introduction to ROC analysis. Pattern Recognit. Lett. 27(8), 861–874 (2006)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Friedman, W.F.: The Index of Coincidence and Its Applications in Cryptography. Aegean Park Press, Walnut Creek (1987)Google Scholar
  8. 8.
  9. 9.
  10. 10.
    Jakobsen, T.: A fast method for the cryptanalysis of substitution ciphers. Cryptologia 19, 265–274 (1995)CrossRefzbMATHGoogle Scholar
  11. 11.
  12. 12.
    Lin, D., Stamp, M.: Hunting for undetectable metamorphic viruses. J. Comput. Virol. 7(3), 201–214 (2011)CrossRefGoogle Scholar
  13. 13.
    Malicia Project, 2015.
  14. 14.
    Nappa, A., Zubair Rafique, M., Caballero, J.: Driving in the cloud: an analysis of drive-by download operations and abuse reporting. In: Proceedings of the 10th Conference on Detection of Intrusions and Malware and Vulnerability Assessment. Berlin, Germany, (July 2013)Google Scholar
  15. 15.
    Next Generation Virus Construction Kit (NGVCK).
  16. 16.
    Runwal, N., Low, R., Stamp, M.: Opcode graph similarity and metamorphic detection. J. Comput. Virol. Hacking Tech. 8(2), 37–52 (2012)CrossRefGoogle Scholar
  17. 17.
  18. 18.
    Shanmugam, G., Low, R.M., Stamp, M.: Simple substitution distance and metamorphic detection. J. Comput. Virol. Hacking Tech. 9(3), 159–170 (2013)CrossRefGoogle Scholar
  19. 19.
    S, Sing: The Code Book: The Science of Secrecy from Ancient Egypt to Quantum Cryptography. Anchor, New York (2011)Google Scholar
  20. 20.
  21. 21.
    Srinivasan, S.: SSCT Score for Malware Detection, Master’s report, Department of Computer Science, San Jose State University (2015).
  22. 22.
    Stamp, M., Low, R .M.: Applied Cryptanalysis: Breaking Ciphers in the Real World. Wiley, Hoboken (2006)Google Scholar
  23. 23.
    Stamp, M.: Information Security: Principles and Practice, 2nd edn. Wiley, Hoboken (2011)CrossRefGoogle Scholar
  24. 24.
    Stamp, M.: Machine Learning with Applications in Information Security. Chapman and Hall/CRC, Boca Raton (2017)Google Scholar
  25. 25.
  26. 26.
    Szor, P.: The Art of Computer Virus Research and Defense. Pearson Eduction, Upper Saddle River (2005)Google Scholar
  27. 27.
  28. 28.
  29. 29.
  30. 30.
    Vigenère and Gronsfeld Cipher, Practical Cryptography.
  31. 31.
    Vigenère Cipher, Crypto Museum.
  32. 32.
    Wong, W., Stamp, M.: Hunting for metamorphic engines. J. Comput. Virol. 2(3), 211–229 (2006)CrossRefGoogle Scholar
  33. 33.
  34. 34.
    Yi, J.: Cryptanalysis of Homophonic Substitution-Transposition Cipher, Department of Computer Science, San Jose State University (2014)

Copyright information

© Springer-Verlag France 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceSan Jose State UniversitySan JoseUSA
  2. 2.Department of EngineeringUniversità degli Studi del SannioBeneventoItaly

Personalised recommendations