Skip to main content
SpringerLink
Log in

High Performance Parallel Computing with Clouds and Cloud Technologies

  • Conference paper
Cloud Computing (CloudComp 2009)

Included in the following conference series:

Abstract

Infrastructure services (Infrastructure-as-a-service), provided by cloud vendors, allow any user to provision a large number of compute instances fairly easily. Whether leased from public clouds or allocated from private clouds, utilizing these virtual resources to perform data/compute intensive analyses requires employing different parallel runtimes to implement such applications. Among many parallelizable problems, most “pleasingly parallel” applications can be performed using MapReduce technologies such as Hadoop, CGL-MapReduce, and Dryad, in a fairly easy manner. However, many scientific applications, which have complex communication patterns, still require low latency communication mechanisms and rich set of communication constructs offered by runtimes such as MPI. In this paper, we first discuss large scale data analysis using different MapReduce implementations and then, we present a performance analysis of high performance parallel applications on virtualized resources.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Amazon Elastic Compute Cloud (EC2), http://aws.amazon.com/ec2/

  2. Amazon Simple Storage Service (S3), http://aws.amazon.com/s3/

  3. GoGrid Cloud Hosting, http://www.gogrid.com/

  4. Keahey, K., Foster, L, Freeman, T., Zhang, X.: Virtual Workspaces: Achieving Quality of Service and Quality of Life in the Grid. Scientific Programming Journal 13(4), 265–276 (2005); Special Issue: Dynamic Grids and Worldwide Computing

    Google Scholar 

  5. Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The Eucalyptus Open-source Cloud-computing System. In: CCGrid 2009: the 9th IEEE International Symposium on Cluster Computing and the Grid, Shanghai, China (2009)

    Google Scholar 

  6. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the art of virtualization. In: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP 2003, pp. 164–177. ACM, New York (2003), http://doi.acm.org/10.1145/945445.945462

    Chapter  Google Scholar 

  7. Apache Hadoop, http://hadoop.apache.org/core/

  8. Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: Distributed data-parallel programs from sequential building blocks. In: European Conference on Computer Systems (2007)

    Google Scholar 

  9. Yu, Y., Isard, M., Fetterly, D., Budiu, M., Erlingsson, U., Gunda, P., Currey, J.: Dryad-LINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In: Symposium on Operating System Design and Implementation (OS-DI), San Diego, CA (2008)

    Google Scholar 

  10. Ekanayake, J., Pallickara, S., Fox, G.: MapReduce for Data Intensive Scientific Analysis. In: Fourth IEEE International Conference on eScience, Indianapolis, pp. 277–284 (2008)

    Google Scholar 

  11. Huang, X., Madan, A.: CAP3: A DNA Sequence Assembly Program. Genome Research 9(9), 868–877 (1999)

    Article  Google Scholar 

  12. Hartigan, J.: Clustering Algorithms. Wiley, Chichester (1975)

    MATH  Google Scholar 

  13. Dean, J., Ghemawat, S.: Mapreduce: Simplified data processing on large clusters. ACM Commun. 51, 107–113 (2008)

    Article  Google Scholar 

  14. MPI (Message Passing Interface), http://www-unix.mcs.anl.gov/mpi/

  15. Dongarra, J., Geist, A., Manchek, R., Sunderam, V.: Integrated PVM framework supports heterogeneous network computing. Computers in Physics 7(2), 166–175 (1993)

    Google Scholar 

  16. Ludäscher, B., Altintas, I., Berkley, C., Higgins, D., Jaeger-Frank, E., Jones, M., Lee, E., Tao, J., Zhao, Y.: Scientific Workflow Management and the Kepler System. Concurrency and Computation: Practice & Experience (2005)

    Google Scholar 

  17. Hull, D., Wolstencroft, K., Stevens, R., Goble, C., Pocock, M., Li, P., Oinn, T.: Taverna: a tool for building and running workflows of services. Nucleic Acids Research (Web Server issue), W729 (2006)

    Google Scholar 

  18. Raicu, I., Zhao, Y., Dumitrescu, C., Foster, L, Wilde, M.: Falkon: a Fast and Light-weight tasK executiON framework. In: Proceedings of the ACM/IEEE Conference on Supercom-puting, SC 2007, Nevada, ACM, New York (2007), http://doi.acm.org/10.1145/1362622.1362680

    Google Scholar 

  19. Pallickara, S., Pierce, M.: SWARM: Scheduling Large-Scale Jobs over the Loosely-Coupled HPC Clusters. In: Fourth IEEE International Conference on eScience, pp. 285–292 (2008)

    Google Scholar 

  20. Frey, J.: Condor DAGMan: Handling Inter-Job Dependencies, http://www.bo.infn.it/calcolo/condor/dagman/

  21. Foster, I.: The Anatomy of the Grid: Enabling Scalable Virtual Organizations. In: Proceedings of the 7th international Euro-Par Conference Manchester on Parallel Processing (2001)

    Google Scholar 

  22. Ghemawat, S., Gobioff, H., Leung, S.: The Google file system. SIGOPS Oper. Syst. Rev. 37(5), 29–43 (2003), http://doi.acm.org/10.1145/1165389.945450

    Article  Google Scholar 

  23. Pallickara, S., Fox, G.: NaradaBrokering: A Distributed Middleware Framework and Architecture for Enabling Durable Peer-to-Peer Grids. In: Endler, M., Schmidt, D.C. (eds.) Middleware 2003. LNCS, vol. 2672, pp. 41–61. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  24. Gu, Y., Grossman, R.: Sector and Sphere: The Design and Implementation of a High Performance Data Cloud. Philosophical Transactions A Special Issue associated with the UK e-Science All Hands Meeting (2008)

    Google Scholar 

  25. Moretti, C., Bui, H., Hollingsworth, K., Rich, B., Flynn, P., Thain, D.: All-Pairs: An Abstraction for Data Intensive Computing on Campus Grids. IEEE Transactions on Parallel and Distributed Systems (2009)

    Google Scholar 

  26. Youseff, L., Wolski, R., Gorda, B., Krintz, C: Evaluating the Performance Impact of Xen on MPI and Process Execution For HPC Systems. In: Proceedings of the 2nd international Workshop on Virtualization Technology in Distributed Computing. IEEE Computer Society, Washington (2006), http://dx.doi.org/10.1109/VTDC.2006.4

    Google Scholar 

  27. Constantinos, E., Hill, N.: Cloud Computing for parallel Scientific HPC Applications: Feasibility of Running Coupled Atmosphere-Ocean Climate Models on Amazon’s EC2. In: Cloud Computing and Its Applications, Chicago, IL (2008)

    Google Scholar 

  28. Walker, E.: benchmarking Amazon EC2 for high-performance scientific computing, http://www.usenix.org/publications/login/2008-10/openpdfs/walker.pdf

  29. Gavrilovska, A., Kumar, S., Raj, K., Gupta, V., Nathuji, R., Niranjan, A., Saraiya, P.: High-Performance Hypervisor Architectures: Virtualization in HPC Systems. In: 1st Workshop on System-level Virtualization for High Performance Computing (2007)

    Google Scholar 

  30. Fox, G., Bae, S., Ekanayake, J., Qiu, X., Yuan, H.: Parallel Data Mining from Multicore to Cloudy Grids. In: High Performance Computing and Grids workshop (2008)

    Google Scholar 

  31. Johnsson, S., Harris, T., Mathur, K.: Matrix multiplication on the connection machine. In: Proceedings of the 1989 ACM/IEEE Conference on Supercomputing, Supercomputing 1989, pp. 326–332. ACM, New York (1989), http://doi.acm.org/10.1145/76263.76298

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Informatics and Computing, Indiana University, Bloomington, IN, 47405, USA

    Jaliya Ekanayake & Geoffrey Fox

Authors
  1. Jaliya Ekanayake
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Geoffrey Fox
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. International Research, Institute on Autonomic Network Computing (IRINAC), Menradstr. 2, 80634, Munich, Germany

    Dimiter R. Avresky

  2. LAAS-CNRS, 7 Avenue du Colonel Roche, 31077, Toulouse Cedex 4, France

    Michel Diaz

  3. Leibmz-Rechenzentrum, Boltzmannstr. 1, 85748, Garching, Germany

    Arndt Bode

  4. Dipartimento di Informatica e Sistematica, Univesita di Roma La Spaienza, 00185, Roma, Italy

    Bruno Ciciani

  5. IBM Research Laboratory, Haifa, Israel

    Eliezer Dekel

Rights and permissions

Reprints and permissions

Copyright information

© 2010 ICST Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering

About this paper

Cite this paper

Ekanayake, J., Fox, G. (2010). High Performance Parallel Computing with Clouds and Cloud Technologies. In: Avresky, D.R., Diaz, M., Bode, A., Ciciani, B., Dekel, E. (eds) Cloud Computing. CloudComp 2009. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, vol 34. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12636-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-12636-9_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-12635-2

  • Online ISBN: 978-3-642-12636-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation