Fast Estimation of the Pattern Frequency Spectrum

  • Matthijs van Leeuwen
  • Antti Ukkonen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8725)

Abstract

Both exact and approximate counting of the number of frequent patterns for a given frequency threshold are hard problems. Still, having even coarse prior estimates of the number of patterns is useful, as these can be used to appropriately set the threshold and avoid waiting endlessly for an unmanageable number of patterns. Moreover, we argue that the number of patterns for different thresholds is an interesting summary statistic of the data: the pattern frequency spectrum.

To enable fast estimation of the number of frequent patterns, we adapt the classical algorithm by Knuth for estimating the size of a search tree. Although the method is known to be theoretically suboptimal, we demonstrate that in practice it not only produces very accurate estimates, but is also very efficient. Moreover, we introduce a small variation that can be used to estimate the number of patterns under constraints for which the Apriori property does not hold. The empirical evaluation shows that this approach obtains good estimates for closed itemsets.

Finally, we show how the method, together with isotonic regression, can be used to quickly and accurately estimate the frequency pattern spectrum: the curve that shows the number of patterns for every possible value of the frequency threshold. Comparing such a spectrum to one that was constructed using a random data model immediately reveals whether the dataset contains any structure of interest.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Agrawal, R., Srikant, R.: Fast algorithms for mining association rules in large databases. In: Proceedings of VLDB 1994, pp. 487–499 (1994)Google Scholar
  2. 2.
    Agrawal, R., Srikant, R.: Mining sequential patterns. In: Proceedings of ICDE 1995, pp. 3–14 (1995)Google Scholar
  3. 3.
    Barlow, R.E., Brunk, H.D.: The isotonic regression problem and its dual. Journal of the American Statistical Association 67(337), 140–147Google Scholar
  4. 4.
    De Bie, T.: Maximum entropy models and subjective interestingness: An application to tiles in binary databases. Data Min. Knowl. Discov. 23(3), 407–446 (2011)CrossRefMATHMathSciNetGoogle Scholar
  5. 5.
    Boley, M., Gärtner, T., Grosskreutz, H.: Formal concept sampling for counting and threshold-free local pattern mining. In: Proc. of SDM 2010, pp. 177–188 (2010)Google Scholar
  6. 6.
    Boley, M., Grosskreutz, H.: A randomized approach for approximating the number of frequent sets. In: Proceedings of ICDM 2008, pp. 43–52 (2008)Google Scholar
  7. 7.
    Bringmann, B., Nijssen, S., Tatti, N., Vreeken, J., Zimmermann, A.: Mining sets of patterns: Next generation pattern mining. In: Tutorial at ICDM 2011 (2011)Google Scholar
  8. 8.
    Chen, P.C.: Heuristic sampling: A method for predicting the performance of tree searching programs. SIAM Journal on Computing 21(2), 295–315 (1992)CrossRefMATHGoogle Scholar
  9. 9.
    Cheng, H., Yan, X., Han, J., Hsu, C.-W.: Discriminative frequent pattern analysis for effective classification. In: Proceedings of the ICDE, pp. 716–725 (2007)Google Scholar
  10. 10.
    Goethals, B., Moens, S., Vreeken, J.: MIME: A framework for interactive visual pattern mining. In: Proceedings of KDD 2011, pp. 757–760 (2011)Google Scholar
  11. 11.
    Gunopulos, D., Khardon, R., Mannila, H., Saluja, S., Toivonen, H., Sharm, R.S.: Discovering all most specific sentences. ACM Trans. Database Syst. 28(2), 140–174 (2003)CrossRefGoogle Scholar
  12. 12.
    Jerrum, M., Sinclair, A.: The markov chain monte carlo method: An approach to approximate counting and integration. Approximation Algorithms for NP-hard Problems, 482–520 (1996)Google Scholar
  13. 13.
    Kilby, P., Slaney, J.K., Thiébaux, S., Walsh, T.: Estimating search tree size. In: Proceedings of AAAI 2006, pp. 1014–1019 (2006)Google Scholar
  14. 14.
    Knuth, D.E.: Estimating the efficiency of backtrack programs. Mathematics of Computation 29(129), 122–136 (1975)CrossRefMathSciNetGoogle Scholar
  15. 15.
    Liu, G., Lu, H., Yu, J.X., Wang, W., Xiao, X.: AFOPT: An efficient implementation of pattern growth approach. In: Proc. of FIMI at ICDM 2003 (2003)Google Scholar
  16. 16.
    Purdom, P.W.: Tree size by partial backtracking. SIAM Journal on Computing 7(4), 481–491 (1978)CrossRefMATHMathSciNetGoogle Scholar
  17. 17.
    Vreeken, J., van Leeuwen, M., Siebes, A.: Krimp: mining itemsets that compress. Data Min. Knowl. Discov. 23(1), 169–214 (2011)CrossRefMATHMathSciNetGoogle Scholar
  18. 18.
    Yan, X., Han, J.: gSpan: Graph-based substructure pattern mining. In: Proceedings of ICDM 2002, pp. 721–724 (2002)Google Scholar
  19. 19.
    Yang, G.: The complexity of mining maximal frequent itemsets and maximal frequent patterns. In: Proceedings of KDD 2004, pp. 344–353 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Matthijs van Leeuwen
    • 1
  • Antti Ukkonen
    • 2
  1. 1.Department of Computer ScienceKU LeuvenBelgium
  2. 2.Helsinki Institute for Information Technology HIITAalto UniversityFinland

Personalised recommendations