Memory-Efficient Algorithms for Finding Needles in Haystacks

  • Itai Dinur
  • Orr Dunkelman
  • Nathan Keller
  • Adi Shamir
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9815)

Abstract

One of the most common tasks in cryptography and cryptanalysis is to find some interesting event (a needle) in an exponentially large collection (haystack) of \(N=2^n\) possible events, or to demonstrate that no such event is likely to exist. In particular, we are interested in finding needles which are defined as events that happen with an unusually high probability of \(p \gg 1/N\) in a haystack which is an almost uniform distribution on N possible events. When the search algorithm can only sample values from this distribution, the best known time/memory tradeoff for finding such an event requires \(O(1/Mp^2)\) time given O(M) memory.

In this paper we develop much faster needle searching algorithms in the common cryptographic setting in which the distribution is defined by applying some deterministic function f to random inputs. Such a distribution can be modelled by a random directed graph with N vertices in which almost all the vertices have O(1) predecessors while the vertex we are looking for has an unusually large number of O(pN) predecessors. When we are given only a constant amount of memory, we propose a new search methodology which we call NestedRho. As p increases, such random graphs undergo several subtle phase transitions, and thus the log-log dependence of the time complexity T on p becomes a piecewise linear curve which bends four times. Our new algorithm is faster than the \(O(1/p^2)\) time complexity of the best previous algorithm in the full range of \(1/N<p<1\), and in particular it improves the previous time complexity by a significant factor of \(\sqrt{N}\) for any p in the range \(N^{-0.75}<p< N^{-0.5}\). When we are given more memory, we show how to combine the NestedRho technique with the parallel collision search technique in order to further reduce its time complexity. Finally, we show how to apply our new search technique to more complicated distributions with multiple peaks when we want to find all the peaks whose probabilities are higher than p.

Keywords

Cryptanalysis Needles in haystacks Mode detection Rho algorithms Parallel collision search 

References

  1. 1.
    Bellare, M., Kohno, T.: Hash function balance and its impact on birthday attacks. In: Cachin, C., Camenisch, J.L. (eds.) EUROCRYPT 2004. LNCS, vol. 3027, pp. 401–418. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  2. 2.
    Richard, R.P.: An improved monte carlo factorization algorithm. BIT Numer. Math. 20(2), 176–184 (1980). doi:10.1007/BF01933190 MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Hellman, M.E.: A cryptanalytic time-memory trade-off. IEEE Trans. Inf. Theory 26(4), 401–406 (1980)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Joux, A., Lucks, S.: Improved generic algorithms for 3-collisions. In: Matsui, M. (ed.) ASIACRYPT 2009. LNCS, vol. 5912, pp. 347–363. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  5. 5.
    Kelsey, J., Kohno, T.: Herding hash functions and the nostradamus attack. In: Vaudenay, S. (ed.) EUROCRYPT 2006. LNCS, vol. 4004, pp. 183–200. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  6. 6.
    Kelsey, J., Schneier, B.: Second preimages on n-bit hash functions for much less than 2\(^{n}\) work. In: Cramer, R. (ed.) EUROCRYPT 2005. LNCS, vol. 3494, pp. 474–490. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  7. 7.
    Knuth, D.E.: The Art of Computer Programming: Seminumerical Algorithms, vol. II. Addison-Wesley, Reading (1969)MATHGoogle Scholar
  8. 8.
    Nivasch, G.: Cycle detection using a stack. Inf. Process. Lett. 90(3), 135–140 (2004)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Stevens, M., Sotirov, A., Appelbaum, J., Lenstra, A., Molnar, D., Osvik, D.A., de Weger, B.: Short chosen-prefix collisions for MD5 and the creation of a rogue CA certificate. In: Halevi, S. (ed.) CRYPTO 2009. LNCS, vol. 5677, pp. 55–69. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  10. 10.
    van Oorschot, P.C., Wiener, M.J.: Parallel collision search with cryptanalytic applications. J. Cryptol. 12(1), 1–28 (1999)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© International Association for Cryptologic Research 2016

Authors and Affiliations

  • Itai Dinur
    • 1
  • Orr Dunkelman
    • 2
  • Nathan Keller
    • 3
  • Adi Shamir
    • 4
  1. 1.Computer Science DepartmentBen-Gurion UniversityBeershebaIsrael
  2. 2.Computer Science DepartmentUniversity of HaifaHaifaIsrael
  3. 3.Department of MathematicsBar-Ilan UniversityRamat GanIsrael
  4. 4.Computer Science DepartmentThe Weizmann InstituteRehovotIsrael

Personalised recommendations