Journal of Grid Computing

, 7:501 | Cite as

Computing Low Latency Batches with Unreliable Workers in Volunteer Computing Environments

  • Eric Martin Heien
  • David P. Anderson
  • Kenichi Hagihara
Open Access
Article

Abstract

Internet based volunteer computing projects such as SETI@home are currently restricted to performing coarse grained, embarrassingly parallel master-worker style tasks. This is partly due to the “pull” nature of task distribution in volunteer computing environments, where workers request tasks from the master rather than the master assigning tasks to arbitrary workers. In this paper we propose algorithms for computing batches of medium grained tasks with deadlines in pull-style volunteer computing environments. We develop models of unreliable workers based on analysis of trace data from an actual volunteer computing project. These models are used to develop algorithms for task distribution in volunteer computing systems with a high probability of meeting batch deadlines. We develop algorithms for perfectly reliable workers, computation-reliable workers and unreliable workers. Finally, we demonstrate the effectiveness of the algorithms through simulations using traces from actual volunteer computing environments.

Keywords

Volunteer computing Pull-style task distribution Stochastic scheduling Grid computing Resource scheduling 

References

  1. 1.
    Anderson, D.P., Cobb, J., Korpela, E., Lebofsky, M., Werthimer, D.: Seti@home: an experiment in public-resource computing. Commun. ACM 45(11), 56–61 (2002)CrossRefGoogle Scholar
  2. 2.
    Larson, S.M., Snow, C.D., Shirts, M., Pande, V.S.: Folding@home and genome@home: using distributed computing to tackle previously intractable problems in computational biology. In: Modern Methods in Computational Biology. Horizon, Marseille (2003)Google Scholar
  3. 3.
    Valiant, L.: A bridging model for parallel computation. Commun. ACM 33(8) (1990)Google Scholar
  4. 4.
    Schopf, J.M., Berman, F.: Stochastic scheduling. In: Supercomputing ’99: Proceedings of the 1999 ACM/IEEE Conference on Supercomputing (CDROM), p. 48. ACM, New York (1999)CrossRefGoogle Scholar
  5. 5.
    Budati, K., Sonnek, J., Chandra, A., Weissman, J.: Ridge: combining reliability and performance in open grid platforms. In: HPDC ’07: Proceedings of the 16th International Symposium on High Performance Distributed Computing, pp. 55–64. ACM, New York (2007)CrossRefGoogle Scholar
  6. 6.
    Kondo, D., Chien, A.A., Casanova, H.: Resource management for rapid application turnaround on enterprise desktop grids. In: SC ’04: Proceedings of the 2004 ACM/IEEE Conference on Supercomputing, p. 17. IEEE Computer Society, Washington, DC (2004)Google Scholar
  7. 7.
    Rood, B., Lewis, M.J.: Scheduling on the grid via multi-state resource availability prediction. In: 9th IEEE/ACM International Conference on Grid Computing, 2008, pp. 126–135 (2008)Google Scholar
  8. 8.
    Kondo, D., Taufer, M., Brooks, C., Casanova, H., Chien, A.: Characterizing and evaluating desktop grids: an empirical study. In: 2004 Proceedings of the 18th International Parallel and Distributed Processing Symposium, p. 26 (2004)Google Scholar
  9. 9.
    Malecot, P., Kondo, D., Fedak, G.: Xtremlab: a system for characterizing internet desktop grids. In: 2006 15th IEEE International Symposium on High Performance Distributed Computing, pp. 357–358 (2006)Google Scholar
  10. 10.
    Golle, P., Mironov, I.: Uncheatable distributed computations. In: Proceedings of the 2001 Conference on Topics in Cryptology: The Cryptographer’s Track at RSA, vol. 2020, pp. 425–440 (2001)Google Scholar
  11. 11.
    Sonnek, J., Chandra, A., Weissman, J.B.: Adaptive reputation-based scheduling on unreliable distributed infrastructures. IEEE Trans. Parallel Distrib. Syst. 18(11), 1551–1564 (2007)CrossRefGoogle Scholar
  12. 12.
    Anderson, D.P.: Boinc: a system for public-resource computing and storage. In: GRID ’04: Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, pp. 4–10. IEEE Computer Society, Washington, DC (2004)CrossRefGoogle Scholar
  13. 13.
    Kondo, D., Andrzejak, A., Anderson, D.P.: On correlated availability in internet-distributed systems. In: 9th IEEE/ACM International Conference on Grid Computing, 2008, pp. 276–283 (2008)Google Scholar
  14. 14.
    Anderson, D.P., Fedak, G.: The computational and storage potential of volunteer computing. In: CCGRID ’06: Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid, pp. 73–80. IEEE Computer Society, Washington, DC (2006)CrossRefGoogle Scholar
  15. 15.
    Kondo, D., Fedak, G., Cappello, F., Chien, A.A., Casanova, H.: Resource availability in enterprise desktop grids. Future Gener. Comput. Syst. 23(7), 888–903 (2007)CrossRefGoogle Scholar
  16. 16.
    Nurmi, D., Brevik, J., Wolski, R.: Modeling machine availability in enterprise and wide-area distributed computing environments. In: Euro-Par05, pp. 432–441 (2005)Google Scholar
  17. 17.
    Andrzejak, A., Kondo, D., Anderson, D.P.: Ensuring collective availability in volatile resource pools via forecasting. In: DSOM ’08: Proceedings of the 19th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management, pp. 149–161. Springer, Berlin (2008)Google Scholar
  18. 18.
    Heien, E., Fujimoto, N., Hagihara, K.: Computing low latency batches with unreliable workers in volunteer computing environments. In: IEEE International Symposium on Parallel and Distributed Processing. IPDPS 2008, pp. 1–8 (2008)Google Scholar
  19. 19.
    Stephens, M.A.: Edf statistics for goodness of fit and some comparisons. J. Am. Stat. Assoc. 69(347), 730–737 (1974)CrossRefGoogle Scholar
  20. 20.
    Parzen, E.: Stochastic Processes. Society for Industrial and Applied Mathematics, Philadelphia (1999)MATHGoogle Scholar
  21. 21.
    Byun, E., Choi, S., Baik, M., Hwang, C., Park, C., Jung, S.Y.: Scheduling scheme based on dedication rate in volunteer computing environment. In: The 4th International Symposium on Parallel and Distributed Computing. ISPDC 2005, pp. 234–241 (2005)Google Scholar
  22. 22.
    Kondo, D., Kindarji, B., Fedak, G., Cappello, F.: Towards soft real-time applications on enterprise desktop grids. In: CCGRID ’06: Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid, pp. 65–72. IEEE Computer Society, Washington, DC (2006)CrossRefGoogle Scholar
  23. 23.
    Kondo, D., Araujo, F., Domingues, P., Silva, L.: Validating desktop grid results by comparing intermediate checkpoints. Technical Report TR-0059 (2006)Google Scholar
  24. 24.
    Christensen, C., Aina, T., Stainforth, D.: The challenge of volunteer computing with lengthy climate model simulations. In: First International Conference on e-Science and Grid Computing, pp. 8–15 (2005)Google Scholar
  25. 25.
    Estrada, T., Fuentes, O., Taufer, M.: A distributed evolutionary method to design scheduling policies for volunteer computing. In: CF ’08: Proceedings of the 2008 Conference on Computing Frontiers, pp. 313–322. ACM, New York (2008)CrossRefGoogle Scholar
  26. 26.
    Murata, Y., Inaba, T., Takizawa, H., Kobayashi, H.: Implementation and evaluation of a distributed and cooperative load-balancing mechanism for dependable volunteer computing. In: IEEE International Conference on Dependable Systems and Networks With FTCS and DCC. DSN 2008, pp. 316–325 (2008)Google Scholar

Copyright information

© The Author(s) 2009

Authors and Affiliations

  • Eric Martin Heien
    • 1
  • David P. Anderson
    • 2
  • Kenichi Hagihara
    • 1
  1. 1.Graduate School of Information Science and TechnologyOsaka UniversitySuitaJapan
  2. 2.University of California, BerkeleyBerkeleyUSA

Personalised recommendations