Skip to main content
SpringerLink
Log in

Validating and Scaling the MicroGrid: A Scientific Instrument for Grid Dynamics

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

Large-scale Grids that aggregate and share resources over wide-area networks present major challenges in understanding dynamic application and resource behavior for performance, stability, and reliability. Accurate study of the dynamic behavior of applications, middleware, resources, and networks depends on coordinated and accurate modeling of all four of these elements simultaneously.

We have designed and implemented a tool called the MicroGrid which enables accurate and comprehensive study of the dynamic interaction of applications, middleware, resource, and networks. The MicroGrid creates a virtual Grid environment – accurately modeling networks, resources, the information services (resource and network metadata) transparently. Thus, the MicroGrid enables users, Grid researchers, or Grid operators to study arbitrary collections of resources and networks. The MicroGrid includes the MaSSF online network simulator which provides packet-level accurate, but scalable network modeling.

We present experimental results with applications which validate the implementation of the MicroGrid, showing that it not only runs real Grid applications and middleware, but that it accurately models both their and underlying resource and network behavior. We also study a range of techniques for scaling a critical part of the online network simulator to the simulation of large networks. These techniques employ a sophisticated graph partitioner, and a range of edge and node weighting schemes exploiting a range of static network and dynamic application information. The best of these, profile-driven placement, scales well to online simulation of large networks of 6,000 nodes using 24 simulation engine nodes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. C. Kesselman and I. Foster, “The Globus Toolkit”, in I. Foster and C. Kesselman (eds.), The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers, pp. 259–278, 1999.

  2. A.S. Grimshaw, W.A. Wulf and the Legion Team, “The Legion Vision of a Worldwide Virtual Computer”, Communications of the ACM, Vol. 40, No. 1, 1997.

  3. D. Thain, T. Tannenbaum and M. Livny, “Condor and the Grid”, in A.J.G. Hey, F. Berman and G. Fox (eds.), Grid Computing: Making the Global Infrastructure a Reality, Wiley, 2003.

  4. S. Agrawal, J. Dongarra, K. Seymour et al., “NetSolve: Past, Present, and Future – a Look at a Grid Enabled Server”, in A.J.G. Hey, F. Berman and G. Fox (eds.), Grid Computing: Making the Global Infrastructure a Reality, Wiley, 2003.

  5. F. Berman, A. Chien, K. Cooper et al., “The GrADS Project: Software Support for High-Level Grid Application Development”, International Journal of High Performance Computing Applications, Vol. 15, No. 4, pp. 327–344, 2001.

    Google Scholar 

  6. V. Paxson and S. Floyd, “Wide-Area Traffic: The Failure of Poisson Modeling”, IEEE/ACM Transactions on Networking, Vol. 3, No. 3, pp. 226–244, 1995.

    Google Scholar 

  7. V. Misra, W. Gong and D. Towsley, “A Fluid-Based Analysis of a Network of AQM Routers Supporting TCP Flows with an Application to RED”, in ACM SIGCOMM’00, Stockholm, Sweden, 2000.

  8. J. Cowie, H. Liu, J. Liu et al., “Towards Realistic Million-Node Internet Simulations”, in Proceedings of the 1999 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA’99), Las Vegas, Nevada, June 28–July 1, 1999.

  9. R. Wolski, N. Spring and J. Hayes, “The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing”, Journal of Future Generation Computing Systems, Vol. 15, Nos. 5–6, pp. 757–768, October 1999.

    Google Scholar 

  10. J. Liu and D. Nicol, “DaSSF 3.1 User’s Manual”, 2001.

  11. SSFNet webpage, http://www.ssfnet.org

  12. “How to Write DML Network Models”, http://www.ssfnet.org/InternetDocs/ssfnetTutorial-1.html

  13. J.H. Cowie, Scalable Simulation Framework API Reference Manual, 1999.

  14. T.V. Lakshman and U. Madhow, “The Performance of TCP/IP for Networks with High Bandwidth-Delay Products and Random Loss”, IFIP Transactions C-26, High Performance Networking, pp. 135–150, 1994.

  15. “The GrADS project”, http://hipersoft.cs.rice.edu/grads

  16. A. Petitet, S. Blackford, J. Dongarra et al., “Numerical Libraries and the Grid: The GrADS Experiment with ScaLAPACK”, International Journal of High Performance Computing Applications, Vol. 15, No. 4, pp. 359–374, 2001.

    Google Scholar 

  17. H. Dail, F. Berman and H. Casanova, “A Decoupled Scheduling Approach for Grid Application Development Environments”, Journal of Parallel and Distributed Computing, 2003.

  18. O. Sievert and H. Casanova, “A Simple MPI Process Swapping Architecture for Iterative Applications”, International Journal of High Performance Computing Applications (IJHPCA), 2004.

  19. “FASTA Package of Sequence Comparison Programs” at ftp://ftp.virginia.edu/pub/fasta

  20. L.S. Blackford, J. Choi, A. Cleary et al., ScaLAPACK Users’ Guide, Society for Industrial and Applied Mathematics: Philadelphia, PA, 1997.

    Google Scholar 

  21. W.R. Pearson and D.J. Lipman, “Improved Tools for Biological Sequence Comparison”, in Proceedings of the National Academy of Science, 1988.

  22. R. Barrett, M.W. Berry, T.F. Chan et al., Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, 2nd edn., SIAM: Philadelphia, PA, 1994.

    Google Scholar 

  23. G.W. Flake, The Computational Beauty of Nature: Computer Explorations of Fractals, Chaos, Complex Systems, and Adaptation, MIT Press: Cambridge, MA, 1998.

    Google Scholar 

  24. K. Schloegel, G. Karypis and V. Kumar, “A New Algorithm for Multi-Objective Graph Partitioning”, in Euro-Par’99 Parallel Processing, Springer: Heidelberg, 1999.

    Google Scholar 

  25. Cisco Systems, NetFlow, 2001.

  26. H. Song, X. Liu, D. Jakobsen et al., “The MicroGrid: A Scientific Tool for Modeling Computational Grids”, in IEEE Supercomputing (SC 2000), Dallas, USA, November 4–10, 2000.

  27. A. Medina, A. Lakhina, I. Matta and J. Byers, “BRITE: An Approach to Universal Topology Generation”, in Proceedings of the International Workshop on Modeling, Analysis and Simulation of Computer and Telecommunications Systems (MASCOTS ‘01), Cincinnati, Ohio, 2001.

  28. P. Barford and M. Crovella, “Generating Representative Web Workloads for Network and Server Performance Evaluation”, in Measurement and Modeling of Computer Systems 1998, 1998.

  29. D.P. Olshefski, J. Nieh and D. Agrawal, “Inferring Client Response Time at the Web Server”, in Proceedings of the ACM International Conference on Measurement and Modeling of Computer Systems (SIGMETRICS 2002), Marina del Rey, CA, 2002.

  30. R. Pan, B. Prabhakar, K. Psounis and D. Wischik, “SHRINK: A Method for Scalable Performance Prediction and Efficient Network Simulation”, in IEEE INFOCOM, 2003.

  31. J. Jung, B. Krishnamurthy and M. Rabinovich, “Flash Crowds and Denial of Service Attacks: Characterization and Implications for CDNs and Web Sites”, in Proceeding of 11th World Wide Web Conference, Honolulu, Hawaii, 2002.

  32. R.F. Van Der Wijngaart and M. Frumkin, “NAS Grid Benchmarks Version 1.0”, 2002, NASA Ames Research Center.

  33. PlanetLab Website, http://www.planet-lab.org

  34. L. Breslau, D. Estrin, K. Fall et al., “Advances in Network Simulation”, IEEE Computer, Vol. 33, No. 5, pp. 59–67, May 2000.

    Google Scholar 

  35. L. Bajaj, M. Takai, R. Ahuja et al., “GloMoSim: A Scalable Network Simulation Environment”, UCLA Computer Science Department Technical Report 990027, May 1999.

  36. H. Schwetman, “CSIM: A C-based, Process Oriented Simulation Language”, in Proceedings of the 1986 Winter Simulation Conference, 1986.

  37. S. Toh, “SimC: A C Function Library for Discrete Simulation”, in Proceedings of the 11th Workshop in Parallel and Distributed Simulation, 1993.

  38. A. Miller, R. Nair and Z. Zhang, “JSIM: A Java-Based Simulation and Animation Environment”, in Proceedings of the 30th Annual Simulation Symposium (ANSS’97), 1997.

  39. F. Gomes, S. Franks, B. Unger et al., “SimKit: A High Performance Logical Process Simulation Class Library in C++”, in Proceedings of the 1995 Winter Simulation Conference, 1995.

  40. F. Howell and R. McNab, “SimJava: A Discrete Event Simulation Package for Java with Applications in Computer Systems Modelling”, in Proceedings of the First International Conference on Web-based Modelling and Simulation, 1998.

  41. R. Buyya and M. Murshed, “GridSim: A Toolkit for the Modeling and Simulation of Distributed Resource Management and Scheduling for Grid Computing”, The Journal of Concurrency and Computation: Practice and Experience (CCPE), Vol. 14, Nos. 13–15, 2002.

  42. A. Legrand, L. Marchal and H. Casanova, “Scheduling Distributed Applications: The SimGrid Simulation Framework”, in Proceedings of the 3rd IEEE International Symposium on Cluster Computing and the Grid (CCGrid’03), Tokyo, Japan, 2003.

  43. VMWare website, http://www.vmware.com

  44. A. Whitaker, M. Shaw and S.D. Gribble, “Scale and Performance in the Denali Isolation Kernel”, in 5th Symposium on Operating System Design and Implementation (OSDI 2002), Boston, MA, 2002.

  45. P. Barham, B. Dragovic, K. Fraser et al., “Xen and the Art of Virtualization”, in 19th ACM Symposium on Operating Systems Principles, Bolton Landing, NY, 2003.

  46. T. Kielmann, H. Bal, J. Maassen et al., “Programming Environments for High-Performance Grid Computing: The Albatross Project”, Future Generation Computer Systems, Vol. 18, No. 8, 2002.

  47. B. White, J. Lepreau, L. Stoller et al., “An Integrated Experimental Environment for Distributed Systems and Networks”, in Proceedings of 5th Symposium on Operating Systems Design and Implementation (OSDI), December 2002.

  48. A. Vahdat, K. Yocum, K. Walsh et al., “Scalability and Accuracy in a Large-Scale Network Emulator”, in Proceedings of 5th Symposium on Operating Systems Design and Implementation (OSDI), December 2002.

  49. L. Rizzo, “Dummynet and Forward Error Correction”, in Proceedings of the 1998 USENIX Anuual Technical Conference, USENIX Association: New Orleans, LA, 1998.

    Google Scholar 

  50. R. Simmonds, R. Bradford and B. Unger, “Applying Parallel Discrete Event Simulation to Network Emulation”, in 14th Workshop on Parallel and Distributed Simulation (PADS 2000), Bologna, Italy, May 28–31, 2000.

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering and Center for Networked Systems, University of California, San Diego, USA

    Xin Liu, Huaxia Xia & Andrew A. Chien

Authors
  1. Xin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huaxia Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrew A. Chien
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Liu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Liu, X., Xia, H. & Chien, A.A. Validating and Scaling the MicroGrid: A Scientific Instrument for Grid Dynamics. J Grid Computing 2, 141–161 (2004). https://doi.org/10.1007/s10723-004-4200-3

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-004-4200-3

Keywords

Search

Navigation