Data Mining Using Relational Database Management Systems
Software packages providing a whole set of data mining and machine learning algorithms are attractive because they allow experimentation with many kinds of algorithms in an easy setup. However, these packages are often based on main-memory data structures, limiting the amount of data they can handle. In this paper we use a relational database as secondary storage in order to eliminate this limitation. Unlike existing approaches, which often focus on optimizing a single algorithm to work with a database backend, we propose a general approach, which provides a database interface for several algorithms at once. We have taken a popular machine learning software package, Weka, and added a relational storage manager as back-tier to the system. The extension is transparent to the algorithms implemented in Weka, since it is hidden behind Weka’s standard main-memory data structure interface. Furthermore, some general mining tasks are transfered into the database system to speed up execution. We tested the extended system, refered to as WekaDB, and our results show that it achieves a much higher scalability than Weka, while providing the same output and maintaining good computation time.
KeywordsMain Memory Machine Learning Algorithm Association Rule Mining Data Mining Algorithm Abstract Class
Unable to display preview. Download preview PDF.
- 1.Agrawal, R., Imielinski, T., Swami, A.: Database mining: A performance perspective. IEEE Transactions on Knowledge and Data Engieering 5(6) (1993)Google Scholar
- 2.Gehrke, J., Ramakrishnan, R., Ganti, V.: Rainforest: A framework for fast decision tree construction of large datasets. In: Int. Conf. on Very Large Data Bases (1998)Google Scholar
- 3.Moore, A.W., Lee, M.: Cached sufficient statistics for efficient machine learning with large data sets. Journal of Artificial Intelligence Research 8 (1998)Google Scholar
- 4.Du Mouchel, W., Volinsky, C., Johson, T., Cortes, C., Pregibon, D.: Squashing flat files flatter. In: ACM Int. Conf. on Knowledge Discovery and Data Mining (1999)Google Scholar
- 5.Pyle, D.: Data Preparation for Data Mining. Morgan Kaufmann Publishers, San Francisco (1999)Google Scholar
- 6.Ross, B.J., Gualtieri, A.G., Fueten, F., Budkewitsch, P.: Hyperspectral image analysis using genetic programmming. In: The Genetic and Evolutionary Computation Conf. (2002)Google Scholar
- 7.Sarawagi, S., Thomas, S., Agrawal, R.: Integrating association rule mining with relational database systems: alternatives and implications. In: ACM SIGMOD Int. Conf. on Management of Data (1998)Google Scholar
- 8.Shafer, J., Agrawal, R., Mehta, M.: SPRINT: A scalable parallel classifier for data mining. In: Int. Conf. on Very Large Data Bases (1996)Google Scholar
- 9.Witten, I.H., Frank, E.: Data mining software in Java, http://www.cs.waikato.ac.nz/ml/weka/
- 10.Ordonez, C.: Programming the K-means Clustering Alogrithm in SQL. In: ACM Int. Conf. on Knowledge Discovery and Data Mining (2004)Google Scholar