Managing Network-Aware Grid Services on Metropolitan Scale: The SCoPE Experience

Abstract

An increasing number of high-performance scientific and industrial applications, ranging from tera-scale data mining to complex computational genomics, weather forecast and financial modeling, are starting to take great advantages from the mesh of geographically distributed computing, network and data management resources, commonly referred to as “Grid”. In this paper, we discuss opportunities and challenges in the Grid resource management framework and network control plane design, critical to the provision of network-assisted extensible Grid services empowering a real high performance distributed computing system built on metropolitan-scale dark fiber transport networks, administered within a single domain and offering plenty of cheap bandwidth. Our primary goal is to enhance our existing grid computing infrastructures, currently focused on CPU and storage, to include the network as an integral Grid component that offers reliable, and if possible guaranteed, levels of service. As a proof of concept, we realized within the SCoPE High Performance Computing environment the prototype of a basic “Metropolitan Area” Grid architecture by implementing on the local scale a novel centralized network resource management service supporting a flexible Grid-application interface and several effective network resource reservation facilities.