Charismatic Document Clustering Through Novel K-Means Non-negative Matrix Factorization (KNMF) Algorithm Using Key Phrase Extraction

Abstract

The tedious challenging of Big Data is to store and retrieve of required data from the search engines. Problem Defined There is an obligation for the quick and efficient retrieval of useful information for the many organizations. The elementary idea is to arrange these computing files of organization into individual folders in an hierarchical order of folders. Manually, to order these files into folders, there is an ardent need to know about the file contents and name of the files to give impression of files, so that it provides an alignment of certain set of files as a bunch. Problem Statement Manual grouping of files has its own complications, for example when these files are in numerous amounts and also their contents cannot be illustrious by their labels. Therefore, it’s an intense requirement for Document clustering with data processing machines for enthusiastic results. Existing System A couple of analyzers are impending with dynamic algorithms and comprehensive analogy of extant algorithms, but, yet, these have been restricted to organizations and colleges. After recent updated rules of NMF their raised a self interest in document clustering. These rules gave trust in its performances with better results when compared to Latent Semantic Indexing with Singular Value Decomposition. Proposed System A new working miniature called Novel K-means Non-Negative Matrix Factorization (KNMF) is implemented using renovated guidelines of NMF which has been diagnosed for clustering documents consequently. A new data set called Newsgroup20 is considered for the exploratory purpose. Removal of common clutter/stop words using keywords from Key Phrase Extraction Algorithm and a new proposed Iterated Lovin stemming will be utilized in preprocessing step inassisting to KNMF. Compared to the Porter stemmer and Lovins stemmer algorithms, Iterative Lovins algorithm is providing 5% more reduction. 60% of the document terms are been minimized to root as remaining terms are already root words. Eventually, an appeal to these processes named “Progressive Text mining radical” is developed inlateral exertion of K-Means algorithm from the defined Apache Mahout Project which is used to analyze the performance of the MapReduce framework in Hadoop.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

References

  1. 1.

    Jain, A.K.: Data clustering: 50 years beyond K-means. Pattern Recognit. Lett. 31(8), 651–666 (2010)

    Article  Google Scholar 

  2. 2.

    Zhang, H., Fritts, J.E., Goldman, S.A.: Image segmentation evaluation: a survey of unsupervised methods. Comput. Vis. Image Underst. 110(2), 260–280 (2008)

    Article  Google Scholar 

  3. 3.

    Baeza Yates, R., Ribeiro Neto, B., et al.: Modern Information Retrieval. ACMPress, New York (1999)

    Google Scholar 

  4. 4.

    Miller, D.J., Wang, Y., Kesidis, G.: Emergent unsupervised clustering paradigms with potential application to bioinformatics. Front. Biosci. 13(1), 677–690 (2008)

    Article  Google Scholar 

  5. 5.

    Guduru, N.: Text Mining with Support Vector Machines (SVM) and Non-Negative Matrix Factorization (NMF) Algorithm. Master’s Thesis, University of Rhode Island, CS Department (2006)

  6. 6.

    Berry, M.W., Dumais, S.T., O’Brien, G.W.: Using linear algebra for intelligent information retrieval. SIAM Rev. 37(4), 573–595 (1994)

    MathSciNet  Article  Google Scholar 

  7. 7.

    Landauer, T.K., Foltz, P.W., Laham, D.: An introduction to latent semantic analysis. Discourse Process. 25(2–3), 259–284 (1998)

    Article  Google Scholar 

  8. 8.

    Lee, D.D., Seung, H.: Learning the parts of objects by non-negative matrix factorization (NMF). Nature 401, 788–791 (1999)

    Article  Google Scholar 

  9. 9.

    Lee, D.D., Seung, H.: Algorithm for non-negative matrix factorization. In: Dietterich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing Systems, Volume 13, Proceedings of the Conference: 556562. The MIT Press

  10. 10.

    Ding, C., He, X., Simon, H.D.: On the equivalence of nonnegative matrix factorization (NMF) and spectral clustering. In: Proceedings of the 2005 SIAM International Conference on Data Mining, pp. 606–610. Society for Industrial and Applied Mathematics (2005)

  11. 11.

    Xu, W., Liu, X., Gong, Y.: Document clustering based on NON-negative matrix factorization. In: Proceedings in ACM SIGIR, pp. 267–273 (2003)

  12. 12.

    Yang, C.F., Ye, M., Zhao, J.: Document clustering based on non-negative sparse matrix factorization. In: International Conference on Advances in Natural Computation, pp. 557–563 (2005)

  13. 13.

    Kanjani, K.: Parallel Non Negative Matrix Factorization for Document Clustering. CPSC-659 (Parallel and Distributed Numerical Algorithms) Course. Texas A&M University, Tech. Rep. (2007)

  14. 14.

    Porter, M.F.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980)

    Article  Google Scholar 

  15. 15.

    Lovins, J.B.: Development of a stemming algorithm. Mech. Translat. Comp. Linguistics 11(1–2), 22–31 (1968)

    Google Scholar 

  16. 16.

    Laxmi, H.V.T.E.V., Somasundaram, K.: 2HARS: heterogeneity-aware resource scheduling in grid environment using K-centroids clustering and PSO techniques. Int. J. Appl. Eng. Res. 10(7), 18047–18062 (2015)

    Google Scholar 

  17. 17.

    Laxmi Lydia, E., Ben Swarup, M., Narsimham, C.: A disparateness–aware scheduling using K-centroids clustering and PSO techniques in hadoop cluster. Int. J. Adv. Found. Res. Comput. 2(12) (2015)

  18. 18.

    Laxmi Lydia, E.: Text mining with lucene and hadoop: document clustering with updated rules of NMF non-negative matrix factorization. Int. J. Pure Appl. Math. 118, 191–198 (2018)

    Google Scholar 

Download references

Acknowledgement

This work is financially supported by the Department of Science and Technology (DST), Science and Engineering Research Board (SERB) under the scheme of ECR. We thank DST_SERB for the financial support to carry the research work.

Author information

Affiliations

Authors

Corresponding author

Correspondence to E. Laxmi Lydia.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Laxmi Lydia, E., Krishna Kumar, P., Shankar, K. et al. Charismatic Document Clustering Through Novel K-Means Non-negative Matrix Factorization (KNMF) Algorithm Using Key Phrase Extraction. Int J Parallel Prog 48, 496–514 (2020). https://doi.org/10.1007/s10766-018-0591-9

Download citation

Keywords

  • Document clustering
  • K-means non-negative matrix factorization (KNMF)
  • Iterated lovin stemmers
  • Key phrase extraction
  • Stopwords
  • MapReduce
  • Hadoop
  • Term frequency-inverse term frequency (Tf-IDF)