Skip to main content
SpringerLink
Log in

FOG: A Fast Out-of-Core Graph Processing Framework

  • Published:
International Journal of Parallel Programming Aims and scope Submit manuscript

Abstract

In this paper we present FOG, an open source graph processing framework designed for out-of-core (external memory) graph processing (https://github.com/mrshawcode/fog). FOG provides a set of programming interfaces that break down update functions of vertices to their incident edges so as to process the functions with edge-centric manner. By these, FOG gives intuitive and productive programming interfaces, and achieves high main memory utilization rate and processing efficiency at the same time. Moreover, FOG proposes an in-place update shuffling mechanism to improve the performance by dramatically reducing disk I/Os during computing. By extensive evaluations on typical graph algorithms and large real-world graphs, we show that FOG outperforms existing out-of-core graph processing systems, including GraphChi, X-Stream and TurboGraph. By comparing the performances of FOG and those of state-of-art distributed graph processing frameworks, we show that only by using just a commodity PC, FOG achieves comparable or even better performance than the best distributed graph processing framework that uses an Amazon EC2 cluster with 128 nodes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Apache: Apache Giraph. http://giraph.apache.org/ (2012)

  2. Backstrom, L., Huttenlocher, D., Kleinberg, J., Lan, X.: Group formation in large social networks: membership, growth, and evolution. In: Proceedings of KDD, pp. 44–54 (2006)

  3. Beamer, S., Asanović, K., Patterson, D.: Direction-optimizing breadth-first search. In: Proceedings of SC, pp. 12:1–12:10 (2012)

  4. Bender, M.A., Brodal, G.S., Fagerberg, R., Jacob, R., Vicari, E.: Optimal sparse matrix dense vector multiplication in the I/O-model. In: Proceedings of SPAA, pp. 61–70 (2007)

  5. Boldi, P., Rosa, M., Santini, M., Vigna, S.: Layered label propagation: a multiresolution coordinate-free ordering for compressing social networks. In: Proceedings of WWW, pp. 587–596 (2011)

  6. Cheng, J., Liu, Q., Li, Z., Fan, W., Lui, J., He, C.: VENUS: vertex-centric streamlined graph computation on a single PC. In: ICDE, pp. 1131–1142 (2015)

  7. Gonzalez, J.E., Low, Y., Gu, H., Bickson, D., Guestrin, C.: PowerGraph: distributed graph-parallel computation on natural graphs. In: Proceedings of OSDI, pp. 17–30 (2012)

  8. Han, M., Daudjee, K., Ammar, K., Özsu, M.T., Wang, X., Jin, T.: An experimental comparison of pregel-like graph processing systems. Proc VLDB Endow 7(12), 1047–1058 (2014)

    Article  Google Scholar 

  9. Han, W.S., Lee, S., Park, K., Lee, J.H., Kim, M.S., Kim, J., Yu, H.: TurboGraph: a fast parallel graph engine handling billion-scale graphs in a single PC. In: Proceedings of KDD, pp. 77–85 (2013)

  10. Kwak, H., Lee, C., Park, H., Moon, S.: What is Twitter, a social network or a news media? In: Proceedings of WWW, pp. 591–600 (2010)

  11. Kyrola, A., Blelloch, G., Guestrin, C.: GraphChi: large-scale graph computation on just a PC. In: Proceedings of OSDI, pp. 31–46 (2012)

  12. Lawrence, P., Sergey, B., Motwani, R., Winograd, T.: The PageRank citation ranking: bringing order to the web. Technical report, Stanford University (1998)

  13. Lin, Z., Chau, D.H., U K: Leveraging memory mapping for fast and scalable graph computation on a PC. In: Proceedings of Big Data, pp. 95–98 (2013)

  14. Low, Y., Bickson, D., Gonzalez, J., Guestrin, C., Kyrola, A., Hellerstein, J.M.: Distributed GraphLab: a framework for machine learning and data mining in the cloud. Proc VLDB Endow 5(8), 716–727 (2012)

    Article  Google Scholar 

  15. Prabhakaran, V., Wu, M., Weng, X., McSherry, F., Zhou, L., Haridasan, M.: Managing large graphs on multi-cores with graph awareness. In: Proceedings of USENIX ATC, pp. 4–4 (2012)

  16. Roy A, Mihailovic I, Zwaenepoel W: X-Stream: edge-centric graph processing using streaming partitions. In: Proceedings of SOSP, pp. 472–488 (2013)

  17. Yahoo: Yahoo WebScope. Yahoo! altavista web page hyperlink connectivity graph. http://webscope.sandbox.yahoo.com/ (2002)

  18. Zheng, D., Mhembere, D., Burns, R., Vogelstein, J., Priebe, C.E., Szalay, A.S.: FlashGraph: processing billion-node graphs on an array of commodity SSDs. In: Proceedings of FAST, pp. 45–58 (2015)

  19. Zhu, X., Han, W., Chen, W.: GridGraph: large-scale graph processing on a single machine using 2-level hierarchical partitioning. In: Proceedings of USENIX ATC, pp. 375–386 (2015)

Download references

Acknowledgments

This work is supported by Natural Science Foundation of China under Grant No. 61433019.

Author information

Authors and Affiliations

  1. Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, 430074, China

    Zhiyuan Shao, Jian He, Huiming Lv & Hai Jin

Authors
  1. Zhiyuan Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huiming Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hai Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiyuan Shao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shao, Z., He, J., Lv, H. et al. FOG: A Fast Out-of-Core Graph Processing Framework. Int J Parallel Prog 45, 1259–1272 (2017). https://doi.org/10.1007/s10766-016-0468-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10766-016-0468-8

Keywords

Search

Navigation