Skip to main content
SpringerLink
Log in

Effect of data skewness in parallel mining of association rules

  • Papers
  • Conference paper
  • First Online:
Research and Development in Knowledge Discovery and Data Mining (PAKDD 1998)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 1394))

Included in the following conference series:

Abstract

An efficient parallel algorithm FPM(Fast Parallel Mining) for mining association rules on a shared-nothing parallel system has been proposed. It adopts the count distribution approach and has incorporated two powerful candidate pruning techniques, i.e., distributed pruning and global pruning. It has a simple communication scheme which performs only one round of message exchange in each iteration. We found that the two pruning techniques are very sensitive to data skewness, which describes the degree of non-uniformity of the itemset distribution among the database partitions. Distributed pruning is very effective when data skewness is high. Global pruning is more effective than distributed pruning even for the mild data skewness case. We have implemented the algorithm on an IBM SP2 parallel machine. The performance studies confirm our observation on the relationship between the effectiveness of the two pruning techniques and data skewness. It has also shown that FPM outperforms CD (Count Distribution) consistently, which is a parallel version of the popular Apriori algorithm [2, 3]. Furthermore, FPM has nice parallelism of speedup, scaleup and sizeup.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. R. Agrawal, T. Imielinski, and A. Swami. Mining association rules between sets of items in large databases. In Proc. of 1993 ACM-SIGMOD Int. Conf. On Management of Data, Washington, D.C., 1993.

    Google Scholar 

  2. R. Agrawal and R. Srikant. Fast algorithms for mining association rules. In Proc. of the 20th VLDB Conference, Santiago, Chile, 1994.

    Google Scholar 

  3. R. Agrawal and J.C. Shafer. Parallel mining of association rules: Design, implementation and experience. Special Issue in Data Mining, IEEE Trans. on Knowledge and Data Engineering, IEEE Computer Society, V8, N6, December 1996, pp. 962–969.

    Article  Google Scholar 

  4. S. Brin, R. Motwani, J. Ullman, S. Tsur. Dynamic itemset counting and implication rules for market basket data. In Proc. of 1997 ACM-SIGMOD Int. Conf. On Management of Data, pages 255–264, Tucson, Arizona, 1997.

    Google Scholar 

  5. T. M. Cover, T. A. Thomas. Elements of information theory. John Wiley & Sons, Inc., 1991.

    Google Scholar 

  6. D. W. Cheung, J. Han, V. Ng and C. Y. Wong. Maintenance of discovered association rules in large databases: An incremental updating technique. In Proc. of the 12th Int. Conf. on Data Eng., New Orleans, Louisiana, 1996.

    Google Scholar 

  7. D. W. Cheung, J. Han, V. T. Ng, A. W. Fu, Y. Fu. A fast distributed algorithm for mining association rules. In Proc. of 4th Int. Conf. on Parallel and Distributed Information Systems, pages 31–43, Miami Beach, Florida, December, 1996.

    Google Scholar 

  8. E. Han, G. Karypis and V. Kumar. Scalable parallel data mining for association rules. In Proc. of 1997 ACM-SIGMOD Int. Conf. On Management of Data, pages 277–288, Tucson, Arizona, 1997.

    Google Scholar 

  9. J Han and Y Fu. Discovery of multiple-level association rules from large databases. In Proc. of the 21th VLDB Conference, Zurich, Switzerland, 1995.

    Google Scholar 

  10. Message Passing Interface Forum. MPI: A Message-Passing Interface Standard, May 1994.

    Google Scholar 

  11. J. S. Park, M. S. Chen, and P. S. Yu, An effective hash-based algorithm for mining association rules. In Proc. of 1995 ACM-SIGMOD Int. Conf. on Management of Data, pages 175–186, San Jose, CA, May 1995.

    Google Scholar 

  12. J. S. Park, M. S. Chen, and P. S. Yu, Efficient parallel mining for association rules. In Proc. of the 4th Int. Conf. on Information and Knowledge Management, Baltimore, Maryland, 1995.

    Google Scholar 

  13. A. Savasere, E. Omiecinski, and S. Navathe. An efficient algorithm for mining association rules in large databases. In Proc. of the 21th VLDB Conference, Zurich, Switzerland, 1995.

    Google Scholar 

  14. R. Srikant and R. Agrawal. Mining quantitative association rules in large relational tables. In Proc. 1996 ACM-SIGMOD Int. Conf. on Management of Data, Montreal, Canada, 1996.

    Google Scholar 

  15. T. Shintani, M. Kitsuregawa. Hash based parallel algorithms for mining association rules. In Proc. of 4th Int. Conf. on Parallel and Distributed Information Systems, Miami Beach, Florida, 1996.

    Google Scholar 

  16. M. J. Zaki, M. Ogihara, S. Parthasarathy, and W. Li, Parallel data mining for association rules on shared-memory multi-processors. Supercomputing'96, Pittsburg, PA, Nov 17–22, 1996.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, The University of Hong Kong, Hong Kong

    David W. Cheung & Yongqiao Xiao

Authors
  1. David W. Cheung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongqiao Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Computer Science and Software Engineering, Monash university, 900 Dandenong Road, Caulfield East, Victoria, 3145, Australia

    Xindong Wu

  2. Department of Computer Science, The University of Melbourne, Parkville, Victoria, 3052, Australia

    Ramamohanarao Kotagiri

  3. School of Computer Science and Engineering, Monash university, Clayton, Victoria, 3168, Australia

    Kevin B. Korb

Rights and permissions

Reprints and permissions

Copyright information

© 1998 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Cheung, D.W., Xiao, Y. (1998). Effect of data skewness in parallel mining of association rules. In: Wu, X., Kotagiri, R., Korb, K.B. (eds) Research and Development in Knowledge Discovery and Data Mining. PAKDD 1998. Lecture Notes in Computer Science, vol 1394. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-64383-4_5

Download citation

  • DOI: https://doi.org/10.1007/3-540-64383-4_5

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-64383-8

  • Online ISBN: 978-3-540-69768-8

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation