A Fast Method of Statistical Assessment for Combinatorial Hypotheses Based on Frequent Itemset Enumeration

  • Shin-ichi Minato
  • Takeaki Uno
  • Koji Tsuda
  • Aika Terada
  • Jun Sese
Conference paper

DOI: 10.1007/978-3-662-44851-9_27

Part of the Lecture Notes in Computer Science book series (LNCS, volume 8725)
Cite this paper as:
Minato S., Uno T., Tsuda K., Terada A., Sese J. (2014) A Fast Method of Statistical Assessment for Combinatorial Hypotheses Based on Frequent Itemset Enumeration. In: Calders T., Esposito F., Hüllermeier E., Meo R. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2014. Lecture Notes in Computer Science, vol 8725. Springer, Berlin, Heidelberg

Abstract

In many scientific communities using experiment databases, one of the crucial problems is how to assess the statistical significance (p-value) of a discovered hypothesis. Especially, combinatorial hypothesis assessment is a hard problem because it requires a multiple-testing procedure with a very large factor of the p-value correction. Recently, Terada et al. proposed a novel method of the p-value correction, called “Limitless Arity Multiple-testing Procedure” (LAMP), which is based on frequent itemset enumeration to exclude meaninglessly infrequent itemsets which will never be significant. The LAMP makes much more accurate p-value correction than previous method, and it empowers the scientific discovery. However, the original LAMP implementation is sometimes too time-consuming for practical databases. We propose a new LAMP algorithm that essentially executes itemset mining algorithm once, while the previous one executes many times. Our experimental results show that the proposed method is much (10 to 100 times) faster than the original LAMP. This algorithm enables us to discover significant p-value patterns in quite short time even for very large-scale databases.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Shin-ichi Minato
    • 1
    • 2
  • Takeaki Uno
    • 3
  • Koji Tsuda
    • 4
    • 5
    • 2
  • Aika Terada
    • 6
  • Jun Sese
    • 6
  1. 1.Graduate School of Information Science and TechnologyHokkaido UniversitySapporoJapan
  2. 2.JST ERATO Minato Discrete Structure Manipulation System ProjectSapporoJapan
  3. 3.National Institute of InformaticsTokyoJapan
  4. 4.Graduate School of Frontier SciencesThe University of TokyoKashiwaJapan
  5. 5.Computational Biology Research Center, National Institute of Advanced Industrial Science and TechnologyTokyoJapan
  6. 6.Department of Computer ScienceOchanomizu UniversityTokyoJapan

Personalised recommendations