Finding Frequent Items in Data Streams
We present a 1-pass algorithm for estimating the most frequent items in a data stream using very limited storage space. Our method relies on a novel data structure called a count sketch, which allows us to estimate the frequencies of all the items in the stream. Our algorithm achieves better space bounds than the previous best known algorithms for this problem for many natural distributions on the item frequencies. In addition, our algorithm leads directly to a 2-pass algorithm for the problem of estimating the items with the largest (absolute) change in frequency between two data streams. To our knowledge, this problem has not been previously studied in the literature.
KeywordsData Stream Hash Function Distinct Element Hash Table Sampling Algorithm
Unable to display preview. Download preview PDF.
- [Ach01]Dimitris Achlioptas. Database-friendly random projections. In Proc. 20th ACM Symposium on Principles of Database Systems, pages 274–281, 2001.Google Scholar
- [FKSV99]Joan Feigenbaum, Sampath Kannan, Martin Strauss, and Mahesh Viswanathan. An approximate l 1-difference algotihm for massive data streams. In Proc. 40th IEEE Symposium on Foundations of Computer Science, pages 501–511, 1999.Google Scholar
- [FKSV00]Joan Feigenbaum, Sampath Kannan, Martin Strauss, and Mahesh Viswanathan. Testing and spot-checking of data streams. In Proc. 11th ACM-SIAM Symposium on Discrete Algorithms, pages 165–174, 2000.Google Scholar
- [FSGM+96]Min Fang, Narayanan Shivakumar, Hector Garcia-Molina, Rajeev Motwani, and Jeffrey Ullman. Computing iceberg queries efficiently. In Proc. 22nd International Conference on Very Large Data Bases, pages 307–317, 1996.Google Scholar
- [GG+02]Anna Gilbert, Sudipto Guha, Piotr Indyk, Yannis Kotidis, S. Muthukrishnan, and Martin Strauss. Fast, small-space algorithms for approximate histogram maintenance. In to appear in Proc. 34th ACM Symposium on Theory of Computing, 2002.Google Scholar
- [GM98]Phillip Gibbons and Yossi Matias. New sampling-based summary statistics for improving approximate query answers. In Proc. ACM SIGMOD International Conference on Management of Data, pages 331–342, 1998.Google Scholar
- [GM99]Phillip Gibbons and Yossi Matias. Synopsis data structures for massive data sets. In Proc. 10th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 909–910, 1999.Google Scholar
- [GMMO00]Sudipto Guha, Nina Mishra, Rajeev Motwani, and Liadan O’Callaghan. Clustering data streams. In Proc. 41st IEEE Symposium on Foundations of Computer Science, pages 359–366, 2000.Google Scholar
- [Goo]Google. Google zeitgeist-search patterns, trends, and surprises according to google. http://www.google.com/press/zeitgeist.html.
- [HRR98]Monika Henzinger, Prabhakar Raghavan, and Sridhar Rajagopalan. Computing on data streams. Technical Report SRC TR 1998-011, DEC, 1998.Google Scholar
- [Ind00]Piotr Indyk. Stable distributions, pseudorandom generators, embeddings and data stream computation. In Proc. 41st IEEE Symposium on Foundations of Computer Science, pages 148–155, 2000.Google Scholar