Skip to main content
SpringerLink
Log in

CSMqGraph: Coarse-Grained and Multi-external-storage Multi-queue I/O Management for Graph Computing

  • Published:
International Journal of Parallel Programming Aims and scope Submit manuscript

Abstract

As graphs continue growing, external storage graph processing systems serve as a promising alternative to distributed in-memory solutions for low cost and high scalability. To obtain high I/O throughput, these systems usually use multiple external storage devices. They adopt the operating system I/O management method based on striped volume, resulting in unsatisfactory performance, such as low sequential bandwidth utilization of each external storage device, limited I/O parallelism and expensive management overhead. In this paper, we analyzed the problems of the operating system I/O management method based on striped volume. Then we designed CSMqGraph, a graph processing system adopts coarse-grained striping method matching sequential large I/O to fully utilize the maximum sequential bandwidth of each external storage device and an I/O management strategy based on multi-external-storage multi-queue making I/O threads dedicated to each external storage device to further improve I/O throughput and fully exploit the parallelism of multiple external storage devices. For different graph algorithms and datasets, our evaluation shows that CSMqGraph consistently outperforms state-of-the-art engines GridGraph by up to 40%, and has better I/O scalability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Chi, Y., Dai, G., Wang, Y., Sun, G., Li, G., Yang, H.: Nxgraph: an efficient graph processing system on a single machine. In: 2016 IEEE 32nd International Conference on Data Engineering (ICDE), pp. 409–420. IEEE (2016)

  2. Coffman, T., Greenblatt, S., Marcus, S.: Graph-based technologies for intelligence analysis. Commun. ACM 47(3), 45–47 (2004)

    Article  Google Scholar 

  3. Del Sol, A., Fujihashi, H., O’Meara, P.: Topology of small-world networks of protein-protein complex structures. Bioinformatics 21(8), 1311–1315 (2005)

    Article  Google Scholar 

  4. Doerr, C., Blenn, N.: Metric convergence in social network sampling. In: Proceedings of the 5th ACM Workshop on HotPlanet, pp. 45–50. ACM (2013)

  5. Gonzalez, J.E., Low, Y., Gu, H., Bickson, D., Guestrin, C.: Powergraph: distributed graph-parallel computation on natural graphs. In: Presented as part of the 10th \(\{\)USENIX\(\}\) Symposium on Operating Systems Design and Implementation (\(\{\)OSDI\(\}\) 12), pp. 17–30 (2012)

  6. Gonzalez, J.E., Xin, R.S., Dave, A., Crankshaw, D., Franklin, M.J., Stoica, I.: Graphx: graph processing in a distributed dataflow framework. In: 11th \(\{\)USENIX\(\}\) Symposium on Operating Systems Design and Implementation (\(\{\)OSDI\(\}\) 14), pp. 599–613 (2014)

  7. Huberman, B.A., Adamic, L.A.: Internet: growth dynamics of the world-wide web. Nature 401(6749), 131 (1999)

    Article  Google Scholar 

  8. Jeong, H., Mason, S.P., Barabási, A.L., Oltvai, Z.N.: Lethality and centrality in protein networks. Nature 411(6833), 41 (2001)

    Article  Google Scholar 

  9. Jeong, H., Tombor, B., Albert, R., Oltvai, Z.N., Barabási, A.L.: The large-scale organization of metabolic networks. Nature 407(6804), 651 (2000)

    Article  Google Scholar 

  10. Kang, U., Tsourakakis, C.E., Faloutsos, C.: Pegasus: a peta-scale graph mining system implementation and observations. In: Proceedings of the 2009 Ninth IEEE International Conference on Data Mining, pp. 229–238. Washington, DC, USA (2009)

  11. Khayyat, Z., Awara, K., Alonazi, A., Jamjoom, H., Williams, D., Kalnis, P.: Mizan: a system for dynamic load balancing in large-scale graph processing. In: Proceedings of the 8th ACM European Conference on Computer Systems, EuroSys ’13, pp. 169–182. ACM, New York, NY, USA (2013). https://doi.org/10.1145/2465351.2465369

  12. Kumar, P., Huang, H.H.: G-store: high-performance graph store for trillion-edge processing. In: SC’16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 830–841. IEEE (2016)

  13. Kwak, H., Lee, C., Park, H., Moon, S.: What is twitter, a social network or a news media? In: Proceedings of the 19th International Conference on World Wide Web, pp. 591–600. ACM (2010)

  14. Kyrola, A., Blelloch, G., Guestrin, C.: Graphchi: large-scale graph computation on just a \(\{\)PC\(\}\). In: Presented as part of the 10th \(\{\)USENIX\(\}\) Symposium on Operating Systems Design and Implementation (\(\{\)OSDI\(\}\) 12), pp. 31–46 (2012)

  15. Lee, E.K., Katz, R.H.: An analytic performance model of disk arrays. In: ACM SIGMETRICS Performance Evaluation Review, vol. 21, pp. 98–109. ACM (1993)

  16. Liu, H., Huang, H.H.: Graphene: fine-grained IO management for graph computing. In: 15th USENIX Conference on File and Storage Technologies (FAST 17), pp. 285–300. USENIX Association, Santa Clara, CA (2017). https://www.usenix.org/conference/fast17/technical-sessions/presentation/liu

  17. Malewicz, G., Austern, M.H., Bik, A.J., Dehnert, J.C., Horn, I., Leiser, N., Czajkowski, G.: Pregel: a system for large-scale graph processing. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pp. 135–146. ACM (2010)

  18. Nguyen, D., Lenharth, A., Pingali, K.: A lightweight infrastructure for graph analytics. In: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, SOSP ’13, pp. 456–471. ACM, New York, NY, USA (2013). https://doi.org/10.1145/2517349.2522739

  19. Randles, M., Lamb, D., Taleb-Bendiab, A.: A comparative study into distributed load balancing algorithms for cloud computing. In: 2010 IEEE 24th International Conference on Advanced Information Networking and Applications Workshops, pp. 551–556. IEEE (2010)

  20. Roy, A., Bindschaedler, L., Malicevic, J., Zwaenepoel, W.: Chaos: scale-out graph processing from secondary storage. In: Proceedings of the 25th Symposium on Operating Systems Principles, pp. 410–424. ACM (2015)

  21. Roy, A., Mihailovic, I., Zwaenepoel, W.: X-stream: edge-centric graph processing using streaming partitions. In: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pp. 472–488. ACM (2013)

  22. Shiloach, Y., Vishkin, U.: An o(log n) parallel connectivity algorithm. J. Algorithms 3, 57–67 (1982)

    Article  MathSciNet  Google Scholar 

  23. Shun, J., Blelloch, G.E.: Ligra: a lightweight graph processing framework for shared memory. In: ACM Sigplan Notices, vol. 48, pp. 135–146. ACM (2013)

  24. Vora, K., Xu, G., Gupta, R.: Load the edges you need: a generic i/o optimization for disk-based graph processing. In: 2016 \(\{\)USENIX\(\}\) Annual Technical Conference (\(\{\)USENIX\(\}\) \(\{\)ATC\(\}\) 16), pp. 507–522 (2016)

  25. Wang, P., Zhang, K., Chen, R., Chen, H., Guan, H.: Replication-based fault-tolerance for large-scale graph processing. In: 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, pp. 562–573. IEEE (2014)

  26. Wang, Z., Gu, Y., Bao, Y., Yu, G., Yu, J.X.: Hybrid pulling/pushing for i/o-efficient distributed and iterative graph computing. In: Proceedings of the 2016 International Conference on Management of Data, pp. 479–494. ACM (2016)

  27. Zhao, Y., Yoshigoe, K., Xie, M., Zhou, S., Seker, R., Bian, J.: Lightgraph: lighten communication in distributed graph-parallel processing. In: 2014 IEEE International Congress on Big Data, pp. 717–724. IEEE (2014)

  28. Zheng, D., Burns, R., Szalay, A.S.: Toward millions of file system iops on low-cost, commodity hardware. In: SC’13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, pp. 1–12. IEEE (2013)

  29. Zheng, D., Mhembere, D., Burns, R., Vogelstein, J., Priebe, C.E., Szalay, A.S.: Flashgraph: processing billion-node graphs on an array of commodity ssds. In: 13th \(\{\)USENIX\(\}\) Conference on File and Storage Technologies (\(\{\)FAST\(\}\) 15), pp. 45–58 (2015)

  30. Zhu, X., Han, W., Chen, W.: Gridgraph: large-scale graph processing on a single machine using 2-level hierarchical partitioning. In: 2015 \(\{\)USENIX\(\}\) Annual Technical Conference (\(\{\)USENIX\(\}\) \(\{\)ATC\(\}\) 15), pp. 375–386 (2015)

Download references

Acknowledgements

This work is supported by NSFC No. 61772216, 61821003, U1705261, Wuhan Application Basic Research Project No. 2017010201010103, Fund from Science, Technology and Innovation Commission of Shenzhen Municipality No. JCYJ20170307172248636, and Fundamental Research Funds for the Central Universities.

Author information

Authors and Affiliations

  1. Wuhan National Laboratory for Optoelectronics, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China

    Shuo Chen, Zhan Shi, Dan Feng, Shang Liu, Fang Wang, Lei Yang & Ruili Yu

Authors
  1. Shuo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhan Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dan Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ruili Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhan Shi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, S., Shi, Z., Feng, D. et al. CSMqGraph: Coarse-Grained and Multi-external-storage Multi-queue I/O Management for Graph Computing. Int J Parallel Prog 48, 98–118 (2020). https://doi.org/10.1007/s10766-019-00651-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10766-019-00651-0

Keywords

Search

Navigation