Skip to main content

Novel structures for counting frequent items in time decayed streams

World Wide Web volume 20, pages 1111–1133 (2017)Cite this article

Abstract

Identifying frequently occurring items is a fundamental building block in many data stream applications. A great deal of work for efficiently identifying frequent items has been studied on the landmark and sliding window models. In this work, we revisit this problem on a new streaming model based on the time decay, where the importance of every arrival item is decreased over the time. To address the importance changes over time, we propose an innovative heap structure, named Quasi-heap, which maintains the item order using a lazy update mechanism. Two approximation algorithm, Space Saving with Quasi-heap (SSQ) and Filtered Space Saving with Quasi-heap (FSSQ), are proposed to find the frequently occurring items based on the Quasi-heap structure. To achieve better accuracy of frequency estimation for all the items in the stream, we introduce a new count-min-min (CMM) sketch structure, which can estimate the count of an item with almost error free. Extensive experiments conducted on both real-world and synthetic data demonstrate the superiority of proposed methods in terms of both efficiency (i.e., response time) and effectiveness (i.e., accuracy).

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11

Notes

  1. Frequent Itemset Mining Dataset Repository, available at http://fimi.cs.helsinki.fi/data/ (last accessed on 17 November, 2016)

References

  1. Aouad, L. M., Le-Khac, N. A., Kechadi, T. M.: Performance study of distributed apriori-like frequent itemsets mining. Knowl. Inf. Syst. 23(1), 55–72 (2010)

    Article  Google Scholar 

  2. Boley, M., Grosskreutz, H.: Approximating the number of frequent sets in dense data. Knowl. Inf. Syst. 21(1), 65–89 (2009)

    Article  Google Scholar 

  3. Brijs, T., Swinnen, G., Vanhoof, K., Wets, G.: Using association rules for product assortment decisions: a case study. In: SIGKDD, pp. 254–260. ACM (1999)

  4. Chakrabarti, A., Cormode, G., McGregor, A.: A near-optimal algorithm for computing the entropy of a stream. In: ACM-SIAM Symposium on Discrete Algorithms, pp. 328–335. Society for Industrial and Applied Mathematics (2007)

  5. Chang, J. H., Lee, W. S.: Finding recent frequent itemsets adaptively over online data streams. In: SIGKDD, pp. 487–492. ACM (2003)

  6. Charikar, M., Chen, K., Farach-Colton, M.: Finding frequent items in data streams. In: Automata, Languages and Programming, pp. 693–703. Springer (2002)

  7. Chen, L., Mei, Q.: Mining frequent items in data stream using time fading model. Inform. Sci. 257, 54–69 (2014)

    MathSciNet  Article  MATH  Google Scholar 

  8. Chen, L., Zhang, S., Tu, L.: An algorithm for mining frequent items on data stream using fading factor. In: COMPSAC, vol. 2, pp. 172–177. IEEE (2009)

  9. Chen, L., Zou, L. J., Tu, L.: A clustering algorithm for multiple data streams based on spectral component similarity. Inform. Sci. 183(1), 35–47 (2012)

    Article  Google Scholar 

  10. Cormode, G., Hadjieleftheriou, M.: Finding the frequent items in streams of data. Commun. ACM 52(10), 97–105 (2009)

    Article  Google Scholar 

  11. Cormode, G., Muthukrishnan, S.: An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms 55(1), 58–75 (2005)

    MathSciNet  Article  MATH  Google Scholar 

  12. Cormode, G., Muthukrishnan, S.: What’s hot and what’s not: tracking most frequent items dynamically. ACM Trans. Database Syst. 30(1), 249–278 (2005)

    Article  Google Scholar 

  13. Cormode, G., Shkapenyuk, V., Srivastava, D., Xu, B.: Forward decay: a practical time decay model for streaming systems. In: ICDE, pp. 138–149. IEEE (2009)

  14. Floyd, R. W.: Algorithm 245: Treesort. Commun. ACM 7(12), 701 (1964)

    Article  Google Scholar 

  15. Golab, L., DeHaan, D., Demaine, E. D., Lopez-Ortiz, A., Munro, J. I.: Identifying frequent items in sliding windows over on-line packet streams. In: SIGCOMM, pp. 173–178. ACM (2003)

  16. Homem, N., Carvalho, J. P.: Finding top-k elements in data streams. Inform. Sci. 180(24), 4958–4974 (2010)

    Article  Google Scholar 

  17. Jin, C., Qian, W., Sha, C., Yu, J. X., Zhou, A.: Dynamically maintaining frequent items over a data stream. In: CIKM, pp. 287–294. ACM (2003)

  18. Karp, R. M., Shenker, S., Papadimitriou, C. H.: A simple algorithm for finding frequent elements in streams and bags. ACM Trans. Database Syst. 28(1), 51–55 (2003)

    Article  Google Scholar 

  19. Li, H. F., Huang, H. Y., Lee, S. Y.: Fast and memory efficient mining of high-utility itemsets from data streams: with and without negative item profits. Knowl. Inf. Syst. 28(3), 495–522 (2011)

    Article  Google Scholar 

  20. Lim, Y., Choi, J., Kang, U.: Fast, accurate, and space-efficient tracking of time-weighted frequent items from data streams. In: CIKM, pp. 1109–1118. ACM (2014)

  21. Lin, Z., Jiang, B., Pei, J., Jiang, D.: Mining discriminative items in multiple data streams. World Wide Web Journal 13(4), 497–522 (2010)

    Article  Google Scholar 

  22. Manerikar, N., Palpanas, T.: Frequent items in streaming data: an experimental evaluation of the state-of-the-art. Data Knowl. Eng. 68(4), 415–430 (2009)

    Article  Google Scholar 

  23. Manku, G. S., Motwani, R.: Approximate Frequency Counts over Data Streams. In: VLDB, pp. 346–357. VLDB Endowment (2002)

  24. Mei, Q. L., Chen, L.: An algorithm for mining frequent stream data items using hash function and fading factor. In: Applied Mechanics and Materials, vol. 130, pp. 2661–2665. Trans Tech Publ (2012)

  25. Metwally, A., Agrawal, D., Abbadi, A. E.: An integrated efficient solution for computing frequent and top-k elements in data streams. ACM Trans. Database Syst. 31(3), 1095–1133 (2006)

    Article  Google Scholar 

  26. Shaker, A., Senge, R., Hüllermeier, E.: Evolving fuzzy pattern trees for binary classification on data streams. Inform. Sci. 220, 34–45 (2013)

    Article  Google Scholar 

  27. Tantono, F. I., Manerikar, N., Palpanas, T.: Efficiently discovering recent frequent items in data streams. In: Scientific and Statistical Database Management, pp. 222–239. Springer (2008)

  28. Tong, Y., Zhang, X., Chen, L.: Tracking frequent items over distributed probabilistic data. World Wide Web Journal, 1–26 (2015)

  29. Wei, Z., Liu, X., Li, F., Shang, S., Du, X., Wen, J.: Matrix sketching over sliding windows. In: SIGMOD, pp. 1465–1480 (2016)

  30. Woo, H. J., Lee, W. S.: Estmax: Tracing maximal frequent item sets instantly over online transactional data streams. IEEE Trans. Knowl. Data Eng. 21(10), 1418–1431 (2009)

    Article  Google Scholar 

  31. Wu, S., Lin, H., U, L.H., Gao, Y., Lu, D.: Finding frequent items in time decayed data streams. In: Apweb, pp. 17–29 (2016)

  32. Zhang, S., Chen, L., Tu, L.: Frequent items mining on data stream based on time fading factor. In: AICI, vol. 4, pp. 336–340. IEEE (2009)

  33. Zhang, S., Chen, L., Tu, L.: Frequent items mining on data stream using hash-table and heap. In: ICIS, vol. 1, pp. 141–145. IEEE (2009)

Download references

Acknowledgments

This work was supported by National Science and Technology Supporting plan (2014BAK16B02, 2015BAH45F01), the cultural relic protection science and technology project of Zhejiang Province, NSFC 61502548 from NSF of China, grant MYRG2014-00106-FST and MYRG2016-00182-FST from UMAC RC.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Zhejiang University, Hangzhou, China

    Shanshan Wu, Huaizhong Lin, Yunjun Gao & Dongming Lu

  2. Faculty of Science and Technology, University of Macau, Macau, China

    Leong Hou U

Authors
  1. Shanshan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huaizhong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leong Hou U
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunjun Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dongming Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huaizhong Lin.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wu, S., Lin, H., U, L.H. et al. Novel structures for counting frequent items in time decayed streams. World Wide Web 20, 1111–1133 (2017). https://doi.org/10.1007/s11280-017-0433-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-017-0433-5

Keywords

  • Frequent item
  • Data stream
  • Time decay model
  • Data structure