Abstract
Information aggregation is the process of summarizing information across the nodes of a distributed system. We present a hierarchical information aggregation system tailored for Peer-to-Peer Grids which typically exhibit a high degree of volatility and heterogeneity of resources. Aggregation is performed in a scalable yet efficient way by merging data along the edges of a logical self-healing tree with each inner node providing a summary view of the information delivered by the nodes of the corresponding subtree. We describe different tree management methods suitable for high-efficiency and high-scalability scenarios that take host capability and stability diversity into account to attenuate the impact of slow and/or unstable hosts. We propose an architecture covering all three phases of the aggregation process: Data gathering through a highly extensible sensing framework, data aggregation using reusable, fully isolated reduction networks, and application-sensitive data delivery using a broad range of propagation strategies. Our solution combines the advantages of approaches based on Distributed Hash Tables (DHTs) (i.e., load balancing and self-maintenance) and hierarchical approaches (i.e., respecting administrative boundaries and resource limitations). Our approach is integrated into our Peer-to-Peer Grid platform Cohesion. We substantiate its effectiveness through performance measurements and demonstrate its applicability through a graphical monitoring solution leveraging our aggregation system.
Similar content being viewed by others
References
Internet Systems Consortium: Internet domain survey—host count. http://www.isc.org/index.pl?/ops/ds/ (2008), January
van Renesse, R.: The importance of aggregation. In: Future Directions in Distributed Computing. Lecture Notes in Computer Science, vol. 2584, pp. 87–92. Springer, Heidelberg (2003), April
Risson, J., Moorsa, T.: Survey of research towards robust peer-to-peer networks: Search methods. Comput. Netw. 50(17), 3485–3521 (2006)
Schulz, S., Blochinger, W., Held, M., Dangelmayr, C.: COHESION—a microkernel based desktop grid platform for irregular task-parallel applications. Future Gener. Comput. Syst. 24(5), 354–370 (2008)
Montresor, A., Jelasity, M., Babaoglu, O.: Robust aggregation protocols for large-scale overlay networks. In: Proceedings of the 2004 International Conference on Dependable Systems and Networks (DSN’04), pp. 19–28 (2004)
Massie, M.L., Chun, B.N., Culler, D.E.: The Ganglia distributed monitoring system: design, implementation, and experience. Parallel Comput. 30(7), 817–840 (2004)
Van Renesse, R., Birman, K.P., Vogels, W.: Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining. ACM Trans. Comput. Syst. 21(2), 164–206 (2003)
Yalagandula, P., Dahlin, M.: A scalable distributed information management system. In: Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications, pp. 379–390 (2004)
Madden, S., Franklin, M.J., Hellerstein, J.M., Hong, W.: Tag: a tiny aggregation service for ad-hoc sensor networks. In: OSDI ’02: Proceedings of the 5th symposium on Operating systems design and implementation, pp. 131–146. ACM, New York, NY, USA (2002)
Kondo, D., Taufer, M., Brooks, C.L., Casanova, H., Chien, A.A.: Characterizing and evaluating desktop grids: An empirical study. In: Proc. of 18th International Parallel and Distributed Processing Symposium (IPDPS’04), p. 26. Sante Fe, New Mexico (2004)
Anderson, D.P.: BOINC: a system for public-resource computing and storage. In: Proc. of the 5th IEEE/ACM International Workshop on Grid Computing, pp. 365–372. Pittsburgh, USA (2004), November
Chien, A., Calder, B., Elbert, S., Bhatia, K.: Entropia: architecture and performance of an enterprise desktop grid system. J. Parallel Distrib. Comput. 63, 597–610 (2003)
Shudo, Y. T. K., Sekiguchi, S.: P3: P2P-based middleware enabling transfer and aggregation of computational resources. In: Proc. Cluster Computing and Grid 2005 (Fifth Int’l Workshop on Global and Peer-to-Peer Computing), pp. 259–265. Cardiff, UK (2005)
Verbeke, J., Nadgir, N., Ruetsch, G., Sharapov, I.: Framework for peer-to-peer distributed computing in a heterogeneous, decentralized environment. In: Proc. of the Third International Workshop on Grid Computing (GRID ’02), pp. 1–12. Springer, London, UK (2002)
Anderson, D.P., Fedak, G.: The computational and storage potential of volunteer computing. In: Proc. of the Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2006), pp. 73–80. Singapore (2006)
Foster, I., Iamnitchi, A.: On death, taxes, and the convergence of peer-to-peer and grid computing. In: 2nd International Workshop on Peer-to-Peer Systems (IPTPS 03, pp. 118–128 (2003)
Blochinger, W., Dangelmayr, C., Schulz, S.: Aspect-oriented parallel discrete optimization on the cohesion desktop grid platform. In: Proc. of the Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2006), pp. 49–56. Singapore (2006), May
Schulz, S., Blochinger, W.: An integrated approach for managing peer-to-peer desktop grid systems. In: Proc. of the Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2007), Rio de Janeiro, Brazil, (2007), May
OSGi Alliance: OSGiTM—The Dynamic Module System for JavaTM. http://www.osgi.org. Accessed February 2008
Ganesh, A. J., Kermarrec, A.-M., Massoulié, L.: Peer-to-peer membership management for gossip-based protocols. IEEE Trans. Comput., 52(2), 139–149 (2003)
Das, A., Gupta, I., Motivala, A.: SWIM: scalable weakly-consistent infection-style process group membership protocol. In: Proc. of the International Conference on Dependable Systems and Networks (DSN 2002), 23-26 June 2002, Bethesda, MD, USA, pp. 303–312. IEEE Computer Society, (2002), February
Kurzyniec, D., Wrzosek, T., Drzewiecki, D., Sunderam, V.: Towards self-organizing distributed computing frameworks: the H2O approach. Parallel Process. Lett., 13(2), 273–290 (2003)
Bhagwan, R., Savage, S., Voelker, G.: Understanding availability. In: Proc. of the 2nd International Workshop on Peer-to-Peer Systems (IPTPS ’03), pp. 256–267, (2003), February
Wolski, R., Spring, N., Hayes, J.: Predicting the cpu availability of time-shared unix systems on the computational grid. In: Proc. of the 8th IEEE International Symposium on High Performance Distributed Computing, pp. 105–112. Washington, DC, USA (1999)
Standard Performance Evaluation Corporation: Spec cpu2006 results, August (2008)
Demers, A., Greene, D., Houser, C., Irish, W., Larson, J., Shenker, S., Sturgis, H., Swinehart, D., Terry, D.: Epidemic algorithms for replicated database maintenance. SIGOPS Oper. Syst. Rev., 22(1):8–32, January (1988)
Birman, K.P., Hayden, M., Ozkasap, O., Xiao, Z., Budiu, M., Minsky, Y.: Bimodal multicast. ACM Trans. Comput. Syst., 17(2), 41–88 (1999)
Hyperic Inc.: SIGAR (System Information Gatherer and Reporter). http://www.hyperic.com/products/sigar.html. Accessed February 2008
Sun Microsystems Inc.: Java Management Extensions (JMX) Remote API (JSR-160). http://jcp.org/en/jsr/detail?id=160. Accessed February 2008
Distributed Management Task Force Inc.: Common Information Model (CIM) Standards. http://www.dmtf.org/standards/cim/. Accessed February 2008
Wolski, R., Spring, N., Hayes, J.: The network weather service: a distributed resource performance forecasting service for metacomputing. Future Gener. Comput. Syst., 15(5–6), 757–768 (1999)
Waldspurger, C. A., Weihl, W. E.: Lottery scheduling: flexible proportional-share resource management. In: Operating Systems Design and Implementation, pp. 1–11 (1994)
Case, J.D., Fedor, M., Schoffstall, M.L., Davin, J.: Simple Network Management Protocol (SNMP). RFC 1157 (Historic) (1990), May
Markovic, S.: JRobin, a Java port of RRDTool. https://rrd4j.dev.java.net. Accessed February 2008
Cohesion Project Website. http://www.cohesion.de. Accessed February 2008
yWorks GmbH: yFiles. http://www.yworks.com. Accessed February 2008
Kempe, D., Dobra, A., Gehrke, J.: Gossip-based computation of aggregate information. In: FOCS ’03: Proceedings of the 44th Annual IEEE Symposium on Foundations of Computer Science, pp. 482–491, Washington, DC, USA. IEEE Computer Society (2003)
Sun Microsystems Inc.: Java Management Extensions (JMX) Specification (JSR-3). http://jcp.org/en/jsr/detail?id=3. Accessed February 2008
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Schulz, S., Blochinger, W. & Hannak, H. Capability-Aware Information Aggregation in Peer-to-Peer Grids. J Grid Computing 7, 135–167 (2009). https://doi.org/10.1007/s10723-008-9114-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10723-008-9114-z