ComputErl – Erlang-Based Framework for Many Task Computing
This paper shows how Erlang programming language can be used for creating a framework for distributing and coordinating the execution of many task computing problems. The goals of the proposed solution are (1) to disperse the computation into many tasks, (2) to support multiple well-known computation models (such as master-worker, map-reduce, pipeline), (3) to exploit the advantages of Erlang for developing an efficient and scalable framework and (4) to build a system that can scale from small to large number of tasks with minimum effort. We present the results of work on designing, implementing and testing ComputErl framework. The preliminary experiments with benchmarks as well as real scientific applications show promising scalability on a computing cluster.
Keywordsmany task computing Erlang grid distributed computing parallelism
Unable to display preview. Download preview PDF.
- 2.DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: Amazon’s highly available key-value store. In: SOSP 2007: Proceedings of twenty-first ACM SIGOPS Symposium on Operating Systems Principles, vol. 41, pp. 205–220. ACM, New York (2007)CrossRefGoogle Scholar
- 3.Foster, I.: Many Tasks Computing: What’s in a Name? (July 2008)Google Scholar
- 5.AB Ericsson: OTP Design Principles User’s Guide (February 2010)Google Scholar
- 7.Mościcki, J.T.: Diane - distributed analysis environment for grid-enabled simulation and analysis of physics data. In: Nuclear Science Symposium Conference Record, vol. 3, pp. 1617–1620. IEEE, Los Alamitos (2003)Google Scholar
- 8.Mościcki, J.T., Brochu, F., Ebke, J., Egede, U., Elmsheuser, J., Harrison, K., Jones, R.W.L., Lee, H.C., Liko, D., Maier, A.: Ganga: a tool for computational-task management and easy access to grid resources. Computer Physics Communications (June 2009)Google Scholar
- 10.Shao, G., Berman, F., Wolski, R.: Master/slave computing on the grid. In: Heterogeneous Computing Workshop, pp. 3–16 (2000)Google Scholar
- 13.Massie, M.L., Chun, B.N., Culler, D.E.: The Ganglia Distributed Monitoring System: Design, Implementation, and Experience. Parallel Computing 30(7) (July 2004)Google Scholar