A Semi-clustering Scheme for Large-Scale Graph Analysis on Hadoop

  • Seungtae HongEmail author
  • Youngsung Shin
  • Dong Hoon Choi
  • Heeseung Jo
  • Jae-woo Chang
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 274)


With the evolution of IT technologies, large-scale graph data have lately become a growing interest. As a result, there are a lot of research results in large-scale graph analysis on Hadoop. The graph analysis based on Hadoop provides parallel programming models with data partitioning and contains iterative phases of MapReduce jobs. Therefore, the effectiveness of data partitioning depends on how the data partitioning maintains data locality in each node of cluster. In this paper, we propose a semi-clustering scheme for large-scale graph analysis such as PageRank algorithm on Hadoop and show that the proposed scheme is effective. With experiment results, PageRank computation with the semi-clustering improves the performance.


large-scale graph analysis semi-clustering Hadoop PageRank 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Malewicz, G., Austern, M., Bik, A., Dehnert, J., Horn, I.: Pregel: a system for large-scale graph processing. In: SIGMOD 2010 (2010)Google Scholar
  3. 3.
    Shinnar, A., Cunningham, D., Herta, B., Saraswat, V.: M3R: Increased performance for in-memory Hadoop jobs. In: VLDB 2012 (2012)Google Scholar
  4. 4.
    Bu, Y., Howe, B., Balazinska, M., Ernst, M.D.: HaLoop: Efficient iterative data processing on large clusters. In: VLDB 2010 (2010)Google Scholar
  5. 5.
    Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. In: WWW 1998 (1998)Google Scholar
  6. 6.
    Avrachenkov, K., Dobrynin, V., Nemirovsky, D., Pham, S., Smirnova, E.: PageRank based clustering of hypertext document collections. In: SIGIR 2008 (2008)Google Scholar
  7. 7.
    White, S., Smyth, P.: Algorithms for estimating relative importance in networks. In: KDD 2003 (2003)Google Scholar
  8. 8.
    Ivn, G., Grolmusz, V.: When the web meets the cell: Using personalized PageRank for analyzing protein interaction networks. Bioinformatics Advance Access (December 2010)Google Scholar
  9. 9.
    Kleinberg, J.: Authoritative sources in a hyperlinked environment. JACM 46(5), 604–632 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Lee, H.C., Borodin, A.: Perturbation of the hyperlinked environment. In: Warnow, T.J., Zhu, B. (eds.) COCOON 2003. LNCS, vol. 2697, pp. 272–283. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  11. 11.
    Lin, J., Schatz, M.: Design pattern for efficient graph algorithms in MapReduce. In: MLG 2010 (2010)Google Scholar
  12. 12.
  13. 13.
    Leskovec, J., Lang, K., Dasgupta, A., Mahoney, M.: Community Structure in Large Networks: Natural Cluster Sizes and the Absence of Large Well-Defined Clusters. Internet Mathematics (2009)Google Scholar
  14. 14.
    Yang, J., Leskovec, J.: Defining and Evaluating Network Communities based on Ground-truth. In: ICDM (2012)Google Scholar
  15. 15.
    Leskovec, J., Kleinberg, J., Faloutsos, C.: Graphs over Time: Densification Laws, Shrinking Diameters and Possible Explanations. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Seungtae Hong
    • 1
    Email author
  • Youngsung Shin
    • 1
  • Dong Hoon Choi
    • 2
  • Heeseung Jo
    • 1
  • Jae-woo Chang
    • 1
  1. 1.Dept. of Computer EngineeringChonbuk National UniversityJeonjuSouth Korea
  2. 2.Korea Institute of Science and Technology Information (KISTI)DaejeonSouth Korea

Personalised recommendations