Fast Estimation of the Pattern Frequency Spectrum

  • Matthijs van Leeuwen
  • Antti Ukkonen
Conference paper

DOI: 10.1007/978-3-662-44851-9_8

Part of the Lecture Notes in Computer Science book series (LNCS, volume 8725)
Cite this paper as:
van Leeuwen M., Ukkonen A. (2014) Fast Estimation of the Pattern Frequency Spectrum. In: Calders T., Esposito F., Hüllermeier E., Meo R. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2014. Lecture Notes in Computer Science, vol 8725. Springer, Berlin, Heidelberg

Abstract

Both exact and approximate counting of the number of frequent patterns for a given frequency threshold are hard problems. Still, having even coarse prior estimates of the number of patterns is useful, as these can be used to appropriately set the threshold and avoid waiting endlessly for an unmanageable number of patterns. Moreover, we argue that the number of patterns for different thresholds is an interesting summary statistic of the data: the pattern frequency spectrum.

To enable fast estimation of the number of frequent patterns, we adapt the classical algorithm by Knuth for estimating the size of a search tree. Although the method is known to be theoretically suboptimal, we demonstrate that in practice it not only produces very accurate estimates, but is also very efficient. Moreover, we introduce a small variation that can be used to estimate the number of patterns under constraints for which the Apriori property does not hold. The empirical evaluation shows that this approach obtains good estimates for closed itemsets.

Finally, we show how the method, together with isotonic regression, can be used to quickly and accurately estimate the frequency pattern spectrum: the curve that shows the number of patterns for every possible value of the frequency threshold. Comparing such a spectrum to one that was constructed using a random data model immediately reveals whether the dataset contains any structure of interest.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Matthijs van Leeuwen
    • 1
  • Antti Ukkonen
    • 2
  1. 1.Department of Computer ScienceKU LeuvenBelgium
  2. 2.Helsinki Institute for Information Technology HIITAalto UniversityFinland

Personalised recommendations