Abstract
Distributed processing of real-world graphs is challenging due to their size and the inherent irregular structure of graph computations. We present hipg, a distributed framework that facilitates high-level programming of parallel graph algorithms by expressing them as a hierarchy of distributed computations executed independently and managed by the user. hipg programs are in general short and elegant; they achieve good portability, memory utilization and performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Karypis, G., Kumar, V.: A parallel algorithm for multilevel graph partitioning and sparse matrix ordering. J. of Par. and Distr. Computing 48(1), 71–95 (1998)
Feige, U., Krauthgamer, R.: A polylog approximation of the minimum bisection. SIAM Review 48(1), 99–130 (2006)
Lumsdaine, A., Gregor, D., Hendrickson, B., Berry, J.: Challenges in parallel graph processing. PPL 17(1), 5–20 (2007)
Siek, J., Lee, L.-Q., Lumsdaine, A.: The Boost Graph Library. Addison-Wesley, Reading (2002)
Gregor, D., Lumsdaine, A.: The parallel BGL: A generic library for distributed graph computations. In: Parallel Object-Oriented Scientific Computing (2005)
Gregor, D., Lumsdaine, A.: Lifting sequential graph algorithms for distributed-memory parallel computation. OOPSLA 40(10), 423–437 (2005)
Cormen, T., Leiserson, C., Rivest, R.: Introduction to algorithms. MIT Press, Cambridge (1990)
Fleischer, L., Hendrickson, B., Pinar, A.: On identifying strongly connected components in parallel. In: Rolim, J.D.P. (ed.) IPDPS-WS 2000. LNCS, vol. 1800, pp. 505–511. Springer, Heidelberg (2000)
Bal, H.E., Maassen, J., van Nieuwpoort, R., Drost, N., Kemp, R., van Kessel, T., Palmer, N., Wrzesińska, G., Kielmann, T., van Reeuwijk, K., Seinstra, F., Jacobs, C., Verstoep, K.: Real-world distributed computing with Ibis. IEEE 43(8), 54–62 (2010)
The Java SE HotSpot virtual machine. java.sun.com/products/hotspot
Dijkstra, E.: Shmuel Safra’s version of termination detection. Circulated privately (January 1987)
Tel, G.: Introduction to distributed algorithms. Cambridge University Press, Cambridge (2000)
Distributed ASCI Supercomputer DAS-3, www.cs.vu.nl/das3
Pennock, D.M., Flake, G.W., Lawrence, S., Glover, E.J., Giles, C.L.: Winners don’t take all: Characterizing the competition for links on the web. PNAS 99(8), 5207–5211 (2002)
Malewicz, G., Austern, M.H., Bik, A.J., Dehnert, J.C., Horn, I., Leiser, N., Czajkowski, G.: Pregel: A system for large-scale graph processing. In: SIGMOD, pp. 135–146 (2010)
Barnat, J., Chaloupka, J., van de Pol, J.: Improved distributed algorithms for SCC decomposition. In: PDMC 2007. ENTCS, vol. 198(1), pp. 63–77 (2008)
Berry, J., Hendrickson, B., Kahan, S., Konecny, P.: Graph software development and performance on the MTA-2 and Eldorado. At 48-th Cray Users Group meeting (2006)
Coarfa, C.: et al. An evaluation of global address space languages: Co-array Fortran and Unified Parallel C. In: PPoPP 2005, pp. 36–47. ACM, New York (2005)
Charles, P.: et al. X10: An object-oriented approach to non-uniform cluster computing. In: OOPSLA, pp. 519–538. ACM, New York (2005)
Chamberlain, B.L., Choi, S.-E., Lewis, E.C., Snyder, L., Weathersby, W.D., Lin, C.: The case for high-level parallel programming in ZPL. IEEE Comput. Sci. Eng. 5(3), 76–86 (1998)
Cong, G., Kodali, S., Krishnamoorthy, S., Lea, D., Saraswat, V., Wen, T.: Solving large, irregular graph problems using adaptive work-stealing. In: ICPP, pp. 536–545. IEEE, Los Alamitos (2008)
Valiant, L.: A bridging model for parallel computation. Comm. ACM 33(8), 103–111 (1990)
Chan, A., Dehne, F., Taylor, R.: CGMgraph/CGMlib: Implementing and testing CGMgraph alg. on PC clusters and shared memory machines. J. of HPC App., 19(1):81–97 (2005)
MPI Forum: MPI: A message passing interface. J of Supercomp Appl. 8(3/4), 169–416 (1994)
Hielscher, F., Gottschling, P.: ParGraph library. pargraph.sourceforge.net (2004)
Blom, S., van Langevelde, I., Lisser, B.: Compressed and distributed file formats for labeled transition systems. In: PDMC 2003. ENTCS, vol. 89, pp. 68–83 (2003)
Denoyer, L., Gallinari, P.: The Wikipedia XML corpus. SIGIR Forum 40(1), 64–69 (2006)
Resource description framework, http://www.w3.org/RDF
Joyce, A.R., Palsson, B.O.: The model organism as a system: Integrating ’omics’ data sets. Nat. Rev. Mol. Cell. Biol. 7(3), 198–210 (2006)
Hammer, M., Weber, M.: To store or not to store reloaded: Reclaiming memory on demand. In: Brim, L., Haverkort, B.R., Leucker, M., van de Pol, J. (eds.) FMICS 2006 and PDMC 2006. LNCS, vol. 4346, pp. 51–66. Springer, Heidelberg (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Krepska, E., Kielmann, T., Fokkink, W., Bal, H. (2011). A High-Level Framework for Distributed Processing of Large-Scale Graphs. In: Aguilera, M.K., Yu, H., Vaidya, N.H., Srinivasan, V., Choudhury, R.R. (eds) Distributed Computing and Networking. ICDCN 2011. Lecture Notes in Computer Science, vol 6522. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17679-1_14
Download citation
DOI: https://doi.org/10.1007/978-3-642-17679-1_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-17678-4
Online ISBN: 978-3-642-17679-1
eBook Packages: Computer ScienceComputer Science (R0)