Fast Estimation of the Pattern Frequency Spectrum
Both exact and approximate counting of the number of frequent patterns for a given frequency threshold are hard problems. Still, having even coarse prior estimates of the number of patterns is useful, as these can be used to appropriately set the threshold and avoid waiting endlessly for an unmanageable number of patterns. Moreover, we argue that the number of patterns for different thresholds is an interesting summary statistic of the data: the pattern frequency spectrum.
To enable fast estimation of the number of frequent patterns, we adapt the classical algorithm by Knuth for estimating the size of a search tree. Although the method is known to be theoretically suboptimal, we demonstrate that in practice it not only produces very accurate estimates, but is also very efficient. Moreover, we introduce a small variation that can be used to estimate the number of patterns under constraints for which the Apriori property does not hold. The empirical evaluation shows that this approach obtains good estimates for closed itemsets.
Finally, we show how the method, together with isotonic regression, can be used to quickly and accurately estimate the frequency pattern spectrum: the curve that shows the number of patterns for every possible value of the frequency threshold. Comparing such a spectrum to one that was constructed using a random data model immediately reveals whether the dataset contains any structure of interest.
Unable to display preview. Download preview PDF.
- 1.Agrawal, R., Srikant, R.: Fast algorithms for mining association rules in large databases. In: Proceedings of VLDB 1994, pp. 487–499 (1994)Google Scholar
- 2.Agrawal, R., Srikant, R.: Mining sequential patterns. In: Proceedings of ICDE 1995, pp. 3–14 (1995)Google Scholar
- 3.Barlow, R.E., Brunk, H.D.: The isotonic regression problem and its dual. Journal of the American Statistical Association 67(337), 140–147Google Scholar
- 5.Boley, M., Gärtner, T., Grosskreutz, H.: Formal concept sampling for counting and threshold-free local pattern mining. In: Proc. of SDM 2010, pp. 177–188 (2010)Google Scholar
- 6.Boley, M., Grosskreutz, H.: A randomized approach for approximating the number of frequent sets. In: Proceedings of ICDM 2008, pp. 43–52 (2008)Google Scholar
- 7.Bringmann, B., Nijssen, S., Tatti, N., Vreeken, J., Zimmermann, A.: Mining sets of patterns: Next generation pattern mining. In: Tutorial at ICDM 2011 (2011)Google Scholar
- 9.Cheng, H., Yan, X., Han, J., Hsu, C.-W.: Discriminative frequent pattern analysis for effective classification. In: Proceedings of the ICDE, pp. 716–725 (2007)Google Scholar
- 10.Goethals, B., Moens, S., Vreeken, J.: MIME: A framework for interactive visual pattern mining. In: Proceedings of KDD 2011, pp. 757–760 (2011)Google Scholar
- 12.Jerrum, M., Sinclair, A.: The markov chain monte carlo method: An approach to approximate counting and integration. Approximation Algorithms for NP-hard Problems, 482–520 (1996)Google Scholar
- 13.Kilby, P., Slaney, J.K., Thiébaux, S., Walsh, T.: Estimating search tree size. In: Proceedings of AAAI 2006, pp. 1014–1019 (2006)Google Scholar
- 15.Liu, G., Lu, H., Yu, J.X., Wang, W., Xiao, X.: AFOPT: An efficient implementation of pattern growth approach. In: Proc. of FIMI at ICDM 2003 (2003)Google Scholar
- 18.Yan, X., Han, J.: gSpan: Graph-based substructure pattern mining. In: Proceedings of ICDM 2002, pp. 721–724 (2002)Google Scholar
- 19.Yang, G.: The complexity of mining maximal frequent itemsets and maximal frequent patterns. In: Proceedings of KDD 2004, pp. 344–353 (2004)Google Scholar