Finding Frequent Items in Data Streams

  • Moses Charikar
  • Kevin Chen
  • Martin Farach-Colton
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2380)

Abstract

We present a 1-pass algorithm for estimating the most frequent items in a data stream using very limited storage space. Our method relies on a novel data structure called a count sketch, which allows us to estimate the frequencies of all the items in the stream. Our algorithm achieves better space bounds than the previous best known algorithms for this problem for many natural distributions on the item frequencies. In addition, our algorithm leads directly to a 2-pass algorithm for the problem of estimating the items with the largest (absolute) change in frequency between two data streams. To our knowledge, this problem has not been previously studied in the literature.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [Ach01]
    Dimitris Achlioptas. Database-friendly random projections. In Proc. 20th ACM Symposium on Principles of Database Systems, pages 274–281, 2001.Google Scholar
  2. [AMS99]
    Noga Alon, Yossi Matias, and Mario Szegedy. The space complexity of approximating the frequency moments. Journal of Computer and System Sciences, 58(1):137–147, 1999.MATHCrossRefMathSciNetGoogle Scholar
  3. [FKSV99]
    Joan Feigenbaum, Sampath Kannan, Martin Strauss, and Mahesh Viswanathan. An approximate l 1-difference algotihm for massive data streams. In Proc. 40th IEEE Symposium on Foundations of Computer Science, pages 501–511, 1999.Google Scholar
  4. [FKSV00]
    Joan Feigenbaum, Sampath Kannan, Martin Strauss, and Mahesh Viswanathan. Testing and spot-checking of data streams. In Proc. 11th ACM-SIAM Symposium on Discrete Algorithms, pages 165–174, 2000.Google Scholar
  5. [FSGM+96]
    Min Fang, Narayanan Shivakumar, Hector Garcia-Molina, Rajeev Motwani, and Jeffrey Ullman. Computing iceberg queries efficiently. In Proc. 22nd International Conference on Very Large Data Bases, pages 307–317, 1996.Google Scholar
  6. [GG+02]
    Anna Gilbert, Sudipto Guha, Piotr Indyk, Yannis Kotidis, S. Muthukrishnan, and Martin Strauss. Fast, small-space algorithms for approximate histogram maintenance. In to appear in Proc. 34th ACM Symposium on Theory of Computing, 2002.Google Scholar
  7. [GM98]
    Phillip Gibbons and Yossi Matias. New sampling-based summary statistics for improving approximate query answers. In Proc. ACM SIGMOD International Conference on Management of Data, pages 331–342, 1998.Google Scholar
  8. [GM99]
    Phillip Gibbons and Yossi Matias. Synopsis data structures for massive data sets. In Proc. 10th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 909–910, 1999.Google Scholar
  9. [GMMO00]
    Sudipto Guha, Nina Mishra, Rajeev Motwani, and Liadan O’Callaghan. Clustering data streams. In Proc. 41st IEEE Symposium on Foundations of Computer Science, pages 359–366, 2000.Google Scholar
  10. [Goo]
    Google. Google zeitgeist-search patterns, trends, and surprises according to google. http://www.google.com/press/zeitgeist.html.
  11. [HRR98]
    Monika Henzinger, Prabhakar Raghavan, and Sridhar Rajagopalan. Computing on data streams. Technical Report SRC TR 1998-011, DEC, 1998.Google Scholar
  12. [Ind00]
    Piotr Indyk. Stable distributions, pseudorandom generators, embeddings and data stream computation. In Proc. 41st IEEE Symposium on Foundations of Computer Science, pages 148–155, 2000.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Moses Charikar
    • 1
  • Kevin Chen
    • 2
  • Martin Farach-Colton
    • 3
  1. 1.Princeton UniversityUSA
  2. 2.UC BerkeleyUSA
  3. 3.Rutgers University and Google Inc.USA

Personalised recommendations