On Improving the Accuracy and Performance of Content-Based File Type Identification

  • Irfan Ahmed
  • Kyung-suk Lhee
  • Hyunjung Shin
  • ManPyo Hong
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5594)


Types of files (text, executables, Jpeg images, etc.) can be identified through file extension, magic number, or other header information in the file. However, they are easy to be tampered or corrupted so cannot be trusted as secure ways to identify file types.In the presence of adversaries, analyzing the file content may be a more reliable way to identify file types, but existing approaches of file type analysis still need to be improved in terms of accuracy and speed. Most of them use byte-frequency distribution as a feature in building a representative model of a file type, and apply a distance metric to compare the model with byte-frequency distribution of the file in question. Mahalanobis distance is the most popular distance metric. In this paper, we propose 1) the cosine similarity as a better metric than Mahalanobis distance in terms of classification accuracy, smaller model size, and faster detection rate, and 2) a new type-identification scheme that applies recursive steps to identify types of files. We compare the cosine similarity to Mahalanobis distance using Wei-Hen Li et al.’s single and multi-centroid modeling techniques, which showed 4.8% and 13.10% improvement in classification accuracy (single and multi-centroid respectively). The cosine similarity showed reduction of the model size by about 90% and improvement in the detection speed by 11%. Our proposed type identification scheme showed 37.78% and 31.47% improvement over Wei-Hen Li’s single and multi-centroid modeling techniques respectively.


file type identification byte frequency distribution cosine similarity Mahalanobis distance linear discriminant cluster analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Exclusion option to skip the files for the scanning in Norton antivirus,
  2. 2.
  3. 3.
  4. 4.
    Wang, K., Stolfo, S.J.: Anomalous Payload-based Network Intrusion Detection. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 203–222. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  5. 5.
    Ahmed, I., Lhee, K.-s.: Detection of malcodes by packet classification. In: Workshop on Privacy and Security by means of Artificial Intelligence, ARES 2008, pp. 1028–1035 (2008)Google Scholar
  6. 6.
    Li, W.J., Wang, K., Stolfo, S., Herzog, B.: Fileprints: Identifying File Types by n-gram Analysis. In: Workshop on Information Assurance and security (IAW 2005), United States Military Academy, West Point, NY, pp. 64–71 (2005)Google Scholar
  7. 7.
    Srinivasan, N., Vaidehil, V.: Reduction of False Alarm Rate in Detecting Network Anomaly using Mahalanobis Distance and Similarity Measure. In: Proceedings of ICSCN, pp. 366–371 (2007)Google Scholar
  8. 8.
    Tan, P.-N., Steinbach, M., Kumar, V.: Introduction to data mining. Addison-Wesley, Reading (2005)Google Scholar
  9. 9.
    Martin, K., Nahid, S.: Oscar - file type identification of binary data in disk clusters and RAM pages. In: IFIP security and privacy in dynamic environments, pp. 413–424 (2006)Google Scholar
  10. 10.
    Martin, K., Nahid, S.: File type identification of data fragments by their binary structure. In: Proceedings of the IEEE workshop on information assurance, pp. 140–147 (2006)Google Scholar
  11. 11.
    Veenman, C.J.: Statistical disk cluster classification for file carving. In: IEEE third international symposium on information assurance and security, pp. 393–398 (2007)Google Scholar
  12. 12.
    Rencher, A.C.: Methods of Multivariate Analysis. Wiley Interscience, Hoboken (2002)CrossRefzbMATHGoogle Scholar
  13. 13.
  14. 14.
  15. 15.
    Nachenberg, C.: Polymorphic virus detection module, United States Patent # 5,826,013 (1998)Google Scholar
  16. 16.
    Szor, P., Ferrie, P.: Hunting for metamorphic. In: Proceedings of Virus Bulletin Conference, pp. 123–144 (2001)Google Scholar
  17. 17.
    RIX, Writing IA32 Alphanumeric Shell codes,
  18. 18.
    Eller, R.: Bypassing MSB Data Filters for Buffer Overflow Exploits on Intel platforms (2003),
  19. 19.
    McDaniel, M., Hossain Heydari, M.: Content Based File Type Detection Algorithms. In: Proceedings of the 36th Annual Hawaii International Conference on System Sciences (2003)Google Scholar
  20. 20.
    Kolmogorov, A.N.: Three approaches to the quantitative definition of information. Problems of Information Transmission, 1–11 (1965)Google Scholar
  21. 21.
    Calhoun, W.C., Coles, D.: Predicting the types of file fragments. Digital Investigation 5(1), 14–20 (2008)CrossRefGoogle Scholar
  22. 22.
    Wang, K., Parekh, J.J., Stolfo, S.J.: Anagram: A Content Anomaly Detector Resistant to Mimicry Attack. In: Zamboni, D., Krügel, C. (eds.) RAID 2006. LNCS, vol. 4219, pp. 226–248. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  23. 23.
    Gu, G., Porras, P., Yegneswaran, V., Fong, M., Lee, W.: BotHunter: Detecting Malware Infection Through IDS-Driven Dialog Correlation: in 16th USENIX Security Symposium (2007)Google Scholar
  24. 24.
    Ward, J.H.: Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 235–244 (1963)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Irfan Ahmed
    • 1
  • Kyung-suk Lhee
    • 1
  • Hyunjung Shin
    • 2
  • ManPyo Hong
    • 1
  1. 1.Digital Vaccine and Internet Immune System Lab Graduate School of Information and CommunicationAjou UniversitySouth Korea
  2. 2.Department of Industrial and Information Systems EngineeringAjou UniversitySouth Korea

Personalised recommendations