Skip to main content

Effective partitioning mechanisms for time-evolving graphs in the Flink system


Graphs are suitable data structures for expressing the relationship between different types of data. With a continuous increase in the graph size, using suitable methods to divide graphs and parallelize the processing load becomes crucial. Balanced graph partitioning has been extensively studied for static and streaming graphs. However, for a time-evolving graph (TEG), whose size and structure are periodically updated, related partitioning methods are lacking. A straightforward approach is to capture snapshots of a TEG and adopt the partitioning methods designed for static or streaming graphs. Although feasible partitioning quality can be expected, the time overhead is high due to frequent repartitioning. This paper proposes two TEG partitioning methods, namely seed and similarity, to decrease the partitioning time. According to the experimental results, on average, seed and similarity require 29–39% of the partitioning time required by snapshot. Moreover, the proposed methods maintain reasonable partitioning quality.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10


  1. Oussous A, Benjelloun FZ, AitLahcen A, Belfkih S (2018) Big data technologies: a survey. J King Saud UnivComputInfScinces 30(4):431–448

    Google Scholar 

  2. Khan N, Yaqoob I, Hashem IA, Inayat Z, Ali WK, Alam M, Shiraz M, Gani A (2014) Big data: Survey, technologies, opportunities, and challenges. Sci World J.

    Article  Google Scholar 

  3. Zheng Y, Capra L, Wolfson O, Yang H (2014) Urban computing: concepts, methodologies, and applications. ACM Trans IntellSystTechnol.

    Article  Google Scholar 

  4. Elshawi R, Batarfi O, Fayoumi A, Bamawi A, Sakr S (2015) Big graph processing systems: state-of-the-art and open challenges. In: Proceedings of the 1st International Conference on Big Data Computing Service and Applications. 2015.11

  5. Lim SH, Lee S, Ganesh G, Brown TC, Sukumar SR (2015) Graph processing platforms at scale: practices and experiences”. Proc ISPASS.

    Article  Google Scholar 

  6. Sharma S, Chou J (2020) A survey of computation techniques on time evolving graphs. Int J Big Data Intell 7(1):1–14.

    Article  Google Scholar 

  7. Stanton I, Kliot G (2012) Streaming graph partitioning for large distributed graphs. In: Proceedings of KDD’12.

  8. Iyer AP, Li LE, Das T, Stoica I (2016) Time-evolving graph processing at scale. In: Proceedings of GRADES’16.

  9. Buluc A, Meyerhenke H, Safro I, Sanders, P, Schulz C (2015) Recent advances in graph partitioning. arXiv:1311.3144v3[cs.DS].

  10. Karypis G, Kumar V (1999) A fast and high quality multilevel scheme for partitioning irregular graphs. J Sci Comput 20(1):359–392

    MathSciNet  MATH  Google Scholar 

  11. Tsourakakis C, Gkantsidis C, Radunovic B, Vojnovic M (2014) Fennel: streaming graph partitioning for massive scale graphs. In: Proceedings of WSDM’14.

  12. Gonzalez JE, Low Y, Gu H, Bickson D, Guestrin C (2012) PowerGraph: distributed graph-parallel computation on natural graphs. In: Proceedings of OSDI’12, pp 17–30

  13. Petroni F, Querzoni L, Daudjee K, Kamali S, Iacoboni G (2015) HDRF: stream-based partitioning for power-law graphs. In: Proceedings of CIKM’15.

  14. Jiang JY, Lee YH, Lai KC (2018) A streaming graph partitioning approach on distributed systems. In: Proceedings of 8th International Conference on Engineering and Applied Science, pp 229–237.

  15. Abdolrashidi A, Ramaswamy L (2015) Incremental partitioning of large time-evolving graphs. Proc CIC.

    Article  Google Scholar 

  16. Filippidou I, Kotidis Y (2015) Online and on-demand partitioning of streaming graphs. ProcIntConf Big Data.

    Article  Google Scholar 

  17. Mayer C, Mayer R, Tariq MA, Geppert H, Laich L, Rieger L, Rothermel K (2018) ADWISE: adaptive window-based streaming edge partitioning for high-speed graph processing. In: Proceedings of ICDCS.

  18. Abbas A, Kalavri V, Carbone P, Vlassov V (2018) Streaming graph partitioning: an experimental study. In: Proceedings of the VLDB endowment.

  19. Leskovec J, Krevl A (2014) SNAP datasets: Stanford large network dataset collection, Jun. 2014, Accessed 18 Mar 2021

Download references


This study was sponsored by the Ministry of Science and Technology, Taiwan, R.O.C., under contract numbers MOST 106-2221-E-142-005 and MOST 107-2221-E-142-006.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Yi-Hsuan Lee.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, YH., Jian, SJ. Effective partitioning mechanisms for time-evolving graphs in the Flink system. J Supercomput 77, 12336–12354 (2021).

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


  • Graph partitioning
  • Time-evolving graph
  • Vertex-cut partitioning
  • Flink