Democratizing Resource-Intensive e-Science Through Peer-to-Peer Grid Computing

  • Francisco BrasileiroEmail author
  • Nazareno Andrade
  • Raquel Lopes
  • Lívia Sampaio
Part of the Computer Communications and Networks book series (CCN)


The new ways of doing science rooted on the unprecedented processing, communication, and storage infrastructure that became available to scientists are collectively called e-Science. Due to their nature, most e-Science activities can only be successfully performed if researchers have access to high-performance computing facilities. Grid and voluntary computing are well-established solutions that cater to this need, but are not accessible to all labs and institutions. Peer-to-peer (P2P) grid computing has been proposed to address this very problem. In this chapter, we share our experience in developing a P2P grid middleware called OurGrid and deploying it to build the OurGrid Community. We describe the mechanisms that effectively promote collaboration and allow the assemblage of large P2P grids from the contributions of thousands of small sites. This includes a thorough review of the main mechanisms required to support the execution of bag-of-tasks applications on top of P2P grids: accounting, scheduling, security, and data caching. Besides, we discuss ways to allow P2P grids to interoperate with service grids. We also report a success case in the utilization of the OurGrid middleware in the context of e-Science. Finally, we summarize our experience in this area indicating the lessons we have learned, the present challenges, and future directions of research.


Discovery Service Grid Resource Resource Provider Resource Owner Cache Service 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Thain, D., Tannenbaum, T. and Livny, M. (2005). Distributed Computing in Practice: The Condor Experience. In: Concurrency and Computation: Practice and Experience, 17(2–4), pp. 23–356.Google Scholar
  2. 2.
    Litzkow M., Livny M. and Mutka M. (1988) Condor - A Hunter of Idle Workstations. In Proc. of the 8th Intl. Conference of Distributed Computing Systems.Google Scholar
  3. 3.
    Anderson, D. P., Cobb, J., Korpela, E., Lebofsky, M. and Werthimer D. (2002) SETI@home: An experiment in Public-resource Computing. In: Communications of the ACM, 45(11), pp. 56–61.CrossRefGoogle Scholar
  4. 4.
    Foster, I and Kesselman, C. (1998) The Grid: Blueprint for a New computing Infrastructure. Morgan Kaufmann.Google Scholar
  5. 5.
    Cirne, W., Brasileiro, F., Andrade, N., Costa, L., Andrade, A., Novaes, R. and Mowbray, M. (2006). Labs of the world, unite!!! In: Journal of Grid Computing, 4(3), pp. 225–246.zbMATHCrossRefGoogle Scholar
  6. 6.
    Butt, A., Zhang, R., Hu, Y.C. (2006) A self-organizing flock of condors, J. Parallel Distrib. Comput. 66 (1), pp. 145–161.zbMATHGoogle Scholar
  7. 7.
    Fedak, G., Germain, C., Neri, V., Cappello, F. (2001). XtremWeb: A generic global computing system. In: Proc of the IEEE Int’l Symp. Cluster Computing and the Grid, pp. 582–587.Google Scholar
  8. 8.
    Taylor, I., Shields, M., Wang, I., Philp, R. (2003) Distributed P2P computing within triana: A galaxy visualization test case. In: IPDPS ’03: Proceedings of the 17th International Symposium on Parallel and Distributed Processing.Google Scholar
  9. 9.
    Lo, V. M., Zappala, D., Zhou, D., Liu, Y., Zhao, S. (2004) Cluster computing on the fly: P2P scheduling of idle cycles in the internet. In: IPTPS’ 04: Proc. of the 3rd International Workshop on Peer-to-Peer Systems, 2004, pp. 227–236.Google Scholar
  10. 10.
    Oliveira, L., Lopes, L., Silva, F., P3 (Parallel Peer to Peer): An Internet Parallel Programming Environment (2002). In: Web Engineering and Peer-to-Peer Computing Workshop, pp. 274–288.Google Scholar
  11. 11.
    Kollock, P. (1998). Social Dilemmas: The Anatomy of Cooperation. In: Annual Review of Sociology, 24, pp. 183–214.CrossRefGoogle Scholar
  12. 12.
    Andrade, N., Brasileiro, F., Cirne, W. and Mowbray, M. (2007). Automatic grid assembly by promoting collaboration in peer-to-peer grids. In: Journal of Parallel and Distributed Computing, 67(8), pp. 957–966.zbMATHCrossRefGoogle Scholar
  13. 13.
    Nowark, M. A. (2006). Five Rules for the Evolution of Cooperation. In: Science, 314(5805), pp. 1560–1563.CrossRefGoogle Scholar
  14. 14.
    Feldman M., Lai K., Stoica I. and Chuang J. (2004). Robust incentive techniques for peer-to-peer networks. In: Proc of the 5th ACM Conference on Electronic Commerce, pp. 102–111.Google Scholar
  15. 15.
    Paranhos, D., Cirne, W. and Brasileiro, F.V. (2003). Trading cycles for information: using replication to schedule bag-of-tasks applications on computational grids. In: Proc. of the 9th Intl. Conference on Parallel and Distributed Computing, pp. 169–180.Google Scholar
  16. 16.
    Santos-Neto, E., Cirne, W., Brasileiro, F.V. and Lima, A. (2004). Exploiting replication and data reuse to efficiently schedule data-intensive applications on grids. In: Proc. of the 10th Workshop on Job Scheduling Strategies for Parallel Processing.Google Scholar
  17. 17.
    Kamvar, S.D., Schlosser M.T. and Garcia-Molina, H. (2003). The EigenTrust algorithm for reputation management in P2P networks. In: Proc. of the 12th Intl. World Wide Web Conference, pp. 640–651.Google Scholar
  18. 18.
    Vishnumurthy, V., Chandrakumar, S., Sirer, E. (2003). KARMA: a secure economic framework for peer-to-peer resource sharing. In: Proc. of the 1st Workshop on Economics of Peer-to-Peer Systems.Google Scholar
  19. 19.
    Buyya, R. and Vazhkudai, S. (2001). Compute Power Market: towards a market-oriented grid. In: Proc. of the 1st IEEE/ACM International Symposium on Cluster Computing and the Grid, pp. 574.Google Scholar
  20. 20.
    Lai, K., Huberman, B.A. and Fine, L. (2004). Tycoon: A distributed market-based resource allocation system. HP Labs Technical Report cs.DC/0404013.Google Scholar
  21. 21.
    Schneidman, J., Chaki, Ng., Parkes, D., AuYoung, A., Snoeren, A.C., Vahdat, A. and Chun, B. (2005). Why markets could (but don’t currently) solve resource allocation problems in systems. In: Proc. of the 10th USENIX Workshop on Hot Topics in Operating Systems, pp. 7.Google Scholar
  22. 22.
    Yamagishi, T. and Matsuda, M. (2003). The role of reputation in open and closed societies: an experimental study of online trading. Working paper 8, Center for the Study of Cultural and Ecological Foundations of the Mind,
  23. 23.
    Mowbray, M., Brasileiro, F.V., Andrade, N., Santana, J. and Cirne, W. (2006). A reciprocation-based economy for multiple services in peer-to-peer grids. In: Proc. of the 6th IEEE Intl. Conference on Peer-to-Peer Computing, pp. 193–202.Google Scholar
  24. 24.
    Lee, C.B., Schwartzman, Y., Hardy, J. and Snavely, A. (2004). Are User Runtime Estimates Inherently Inaccurate? In: Proc. of the Workshop on Job Scheduling Strategies for Parallel Processing, pp. 253–263.Google Scholar
  25. 25.
    Santos, R., Andrade, A., Cirne, W., Brasileiro, F.V and Andrade, N. (2007). Relative Autonomous Accounting for Peer-to-Peer Grids. In: Concurrency and Computation: Practice and Experience, 19(14), pp. 1937–1954.CrossRefGoogle Scholar
  26. 26.
    Casanova, H. et al. (2000). Heuristics for scheduling parameter sweep applications in grid environments. In: Proc. of 9th Heterogeneous Computing Workshop, pp. 349–363.Google Scholar
  27. 27.
    Nobrega, N., Assis, L. and Brasileiro, F. (2008). Scheduling CPU-Intensive Grid Applications Using Partial Information. In: 37th International Conference on Parallel Processing, pp. 262–269.Google Scholar
  28. 28.
    Massie, M., Chun, B. and Culler, D. (2004). The Ganglia Distributed Monitoring System: Design, Implementation, and Experience. In: Journal of Parallel Computing, 30(7), pp. 817–840.CrossRefGoogle Scholar
  29. 29.
    Yalagandula, P., Sharma, P., Banerjee, S., Basu, S. and Lee, S.-J. (2006). S3: A scalable sensing service for monitoring large networked systems. In: Proc. of SIGCOMM’06 Workshops, pp. 71–76.Google Scholar
  30. 30.
    Brasileiro, F., Costa, L., Andrade, A., Cirne, W., Basu, S. and Banerjee, S. (2006). A large scale fault-tolerant grid information service. In: Proc. of 4th Intl. Workshop on Middleware for Grid Computing, pp. 14.Google Scholar
  31. 31.
    Podlipnig, S. and Böszörmenyi, L. (2003). A survey of web cache replacement strategies. ACM Computing Surveys, 35(4), pp. 374–398.CrossRefGoogle Scholar
  32. 32.
    Soares, P., Oliveira, M., Guerrero, D. and Brasileiro, F. (2008). Solomon: Incentivando o Compartilhamento e Maior Disponibilidade em Sistemas de Armazenamento Entre-Pares. In: Anais do IV Workshop on Peer-to-Peer, pp. 1–12.Google Scholar
  33. 33.
    Silva, R., Brasileiro, F., and Lopes, R. Technical Report: Providing Efficiency and Fairness in a Disk Cache System for Computational Peer-to-Peer Grids. TR-1 (2010), available at
  34. 34.
    Santhanam, S., Elango, P., Arpaci-Dusseau, A. and Livny, M. (2005). Deploying virtual machines as sandboxes for the grid. In: Proc. of the 2nd conference on Real, Large Distributed Systems, pp. 7–12.Google Scholar
  35. 35.
    Sarmenta, L. Sabotage-Tolerance Mechanisms for Volunteer Computing Systems (2002). In: Future Generation Computer Systems, 18 (4), pp. 561–572.zbMATHCrossRefGoogle Scholar
  36. 36.
    Collberg, C. S. and Thomborson, C. (2002). Watermarking, tamper-proofing, and obfuscation - tools for software protection. In: Software Engineering, IEEE Transactions on, 28(8), pp. 735–746.CrossRefGoogle Scholar
  37. 37.
    Ellison, C., Frantz, B., Lampson, B., Rivest, R., Thomas, B., and Ylonen, T. (1999). RFC 2693: Spki certificate theory. Request for Comments (RFC) 2693, Network Working Group.Google Scholar
  38. 38.
    Kaliski, B. (1998). RFC 2315: Pkcs #7: Cryptographic message syntax. Request for Comments (RFC) 2315, Network Working Group.Google Scholar
  39. 39.
    Adams, C. and Farrell, S. (1999). RFC 2510: Internet X.509 public key infrastructure certificate management protocols. Request for Comments (RFC) 2510, Network Working Group.Google Scholar
  40. 40.
    Brasileiro, F., Duarte, A., Silva, R. and Gaudêncio, M. (2009). On the co-existence of service and opportunistic grids. In: Proc. of the First EELA-2 Conference, pp. 51–62.Google Scholar
  41. 41.
    Voorsluys, W., Araújo, E., Cirne, W., Galvão, C., Souza, E. and Cavalcanti, E. (2007). Fostering collaboration to better manage water resources. In: Concurrency and Computation. Practice and Experience, vol. 19, pp. 1609–1620.CrossRefGoogle Scholar
  42. 42.
    Cornillon, P., Gallagher, J. and Sgouros, T. (2003). OPeNDAP: accessing data in a distributed, heterogeneous environment. In: Data Science Journal, vol. 2, pp. 164–174.CrossRefGoogle Scholar
  43. 43.
    Basu, S., Banerjee, S., Sharma, P. and Lee, S. J. (2005). NodeWiz: peer-to-peer resource discovery for grids. In: Proc. 5th Intl. Workshop on Global and Peer-to-Peer Computing, pp. 213–220.Google Scholar
  44. 44.
    Galvão, C., Nobre, P., Braga, A. C. F. M. et al. (2005). Climatic predictability, hydrology and water resources over Nordeste Brazil. In: Regional Hydrological Impacts of Climatic Change – Impact Assessment and Decision Making (ed. By T. Wagener et al.), pp. 211–220. IAHS Publ. 295. IAHS Press, Wallingford, UK.Google Scholar
  45. 45.
    de Rose, C., Ferreto, T., Farias, M., Dias, V., Cirne, W., Oliveira, M., Saikoski, K., and Danieleski, M. (2006). Gerpavgrid: using the grid to maintain the city road system. In: Proceedings of the 18th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD’06), pp. 73–80.Google Scholar
  46. 46.
    Benkler, Y. (Forthcoming) Law, Policy, and Cooperation. In Balleisen, E. and Moss D., eds. Government and Markets: Toward a New Theory of Regulation. Cambridge University Press.Google Scholar
  47. 47.
    Galvão, C., Nóbrega, R., Brasileiro, F. and Araújo, E. (2009). An e-Science platform for collaborative generation of knowledge and technology in hydrology, hydrogeology and water resources. In: IAHS-AISH publication, vol. 331, pp. 500–504.zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Francisco Brasileiro
    • 1
    Email author
  • Nazareno Andrade
    • 1
  • Raquel Lopes
    • 1
  • Lívia Sampaio
    • 1
  1. 1.Departamento de Sistemas e Computação, Laboratório de Sistemas DistribuídosUniversidade Federal de Campina GrandeCampina GrandeBrazil

Personalised recommendations