Parallel Implementation of Classification Algorithms Based on MapReduce
Data mining has attracted extensive research for several decades. As an important task of data mining, classification plays an important role in information retrieval, web searching, CRM, etc. Most of the present classification techniques are serial, which become impractical for large dataset. The computing resource is under-utilized and the executing time is not waitable. Provided the program mode of MapReduce, we propose the parallel implementation methods of several classification algorithms, such as k-nearest neighbors, naive bayesian model and decision tree, etc. Preparatory experiments show that the proposed parallel methods can not only process large dataset, but also can be extended to execute on a cluster, which can significantly improve the efficiency.
KeywordsData Mining Classification Parallel Implementation Large Dataset MapReduce
Unable to display preview. Download preview PDF.
- 1.Quinlan, J.R.: Induction of decision trees. Machine Learning 1(1), 81–106 (1986)Google Scholar
- 2.Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo (1997)Google Scholar
- 3.Weiss, S.M., Kulikowski, C.A.: Computer Systems that Learn: Classification and Prediction Methods from Statistics. In: Neural Nets, Machine Learning, and Expert Systems. Morgan Kaufmann, San Francisco (1991)Google Scholar
- 8.Dasarathy, B.V.: Nearest Neighbor (NN) Norms: NN Pattern Classification Techiniques. IEEE Computer Society Press, Los Alamitos (1991)Google Scholar
- 11.Buyya, R., Yeo, C.S., Venugopal, S.: Market-Oriented Cloud Computing: Vision, Hype, and Reality for Delivering IT Services as Computing Utilities. In: Proc. of the 10th IEEE International Conference on High Performance Computing and Communications, China (2008)Google Scholar
- 12.Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. In: Proc. of the 6th Symposium on Operating System Design and Implementation, USA (2004)Google Scholar
- 15.Elsayed, T., Jimmy, L., Douglas, W.O.: Pairwise Document Similarity in Large Collections with MapReduce. In: Proc. of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies, pp. 262–268 (2008)Google Scholar