Similarity Preserving Hashing: Eligible Properties and a New Algorithm MRSH-v2
Hash functions are a widespread class of functions in computer science and used in several applications, e.g. in computer forensics to identify known files. One basic property of cryptographic Hash Functions is the avalanche effect that causes a significantly different output if an input is changed slightly. As some applications also need to identify similar files (e.g. spam/virus detection) this raised the need for Similarity Preserving Hashing. In recent years, several approaches came up, all with different namings, properties, strengths and weaknesses which is due to a missing definition.
Based on the properties and use cases of traditional Hash Functions this paper discusses a uniform naming and properties which is a first step towards a suitable definition of Similarity Preserving Hashing. Additionally, we extend the algorithm MRSH for Similarity Preserving Hashing to its successor MRSH-v2, which has three specialties. First, it fulfills all our proposed defining properties, second, it outperforms existing approaches especially with respect to run time performance and third it has two detections modes. The regular mode of MRSH-v2 is used to identify similar files whereas the f-mode is optimal for fragment detection, i.e. to identify similar parts of a file.
KeywordsDigital forensics Similarity Preserving Hashing fuzzy hashing MRSH-v2 properties of Similarity Preserving Hashing
Unable to display preview. Download preview PDF.
- 1.NIST, “National Software Reference Library” (May 2012), http://www.nsrl.nist.gov
- 2.Kornblum, J.: Identifying almost identical files using context triggered piecewise hashing. In: Digital Forensic Research Workshop (DFRWS), vol. 3S, pp. 91–97 (2006)Google Scholar
- 3.Roussev, V.: Data fingerprinting with similarity digests. In: Chow, K.-P., Shenoi, S. (eds.) Advances in Digital Forensics VI. IFIP AICT, vol. 337, pp. 207–226. Springer, Heidelberg (2010)Google Scholar
- 4.Breitinger, F., Baier, H.: A Fuzzy Hashing Approach based on Random Sequences and Hamming Distance. In: ADFSL Conference on Digital Forensics, Security and Law, pp. 89–101 (May 2012)Google Scholar
- 5.Roussev, V., Richard, G.G., Marziale, L.: Multi-resolution similarity hashing. In: Digital Forensic Research Workshop (DFRWS), pp. 105–113 (2007)Google Scholar
- 6.Roussev, V.: Scalable data correlation. International Conference on Digital Forensics (IFIP WG 11.9) (January 2012)Google Scholar
- 7.Tridgell, A.: Spamsum. Readme (2002), http://samba.org/ftp/unpacked/junkcode/spamsum/README
- 8.Chen, L., Wang, G.: An Efficient Piecewise Hashing Method for Computer Forensics. In: Workshop on Knowledge Discovery and Data Mining, pp. 635–638 (2008)Google Scholar
- 9.Seo, K., Lim, K., Choi, J., Chang, K., Lee, S.: Detecting Similar Files Based on Hash and Statistical Analysis for Digital Forensic Investigation. In: Computer Science and its Applications (CSA 2009), pp. 1–6 (December 2009)Google Scholar
- 11.Baier, H., Breitinger, F.: Security Aspects of Piecewise Hashing in Computer Forensics. In: IT Security Incident Management & IT Forensics (IMF), 21–36 (May 2011)Google Scholar
- 12.Breitinger, F.: Security Aspects of Fuzzy Hashing. Master’s thesis, Hochschule Darmstadt (February 2011), https://www.dasec.h-da.de/offerings/theses/
- 13.Roussev, V.: Building a Better Similarity Trap with Statistically Improbable Features. In: 42nd Hawaii International Conference on System Sciences, pp. 1–10 (2009)Google Scholar
- 14.SHS, “Secure Hash Standard” (1995)Google Scholar
- 16.Roussev, V.: An evaluation of forensic similarity hashes. In: Digital Forensic Research Workshop, vol. 8, pp. 34–41 (2011)Google Scholar
- 17.Breitinger, F., Baier, H., Beckingham, J.: Security and Implementation Analysis of the Similarity Digest sdhash. In: First International Baltic Conference on Network Security & Forensics (NeSeFo) (August 2012)Google Scholar
- 18.Noll, L.C.: Fowler / Noll / Vo (FNV) Hash (2001), http://www.isthe.com/chongo/tech/comp/fnv/index.html