Building Distributed, Wide-Area Applications with WheelFS

Abstract

It is a challenge to build applications that need to share data and are distributed across hundreds or thousands of computers in a wide-area network (e.g., PlanetLab or on a Grid). In order to cope with high latency, throughput bottlenecks, and temporary failures, such applications typically implement their own storage plan or use special-purpose storage solutions (e.g., DISC, Globus, Carbonite, etc.). Inspired by the success of the Google File System for cluster applications, this proposal investigates whether a general-purpose wide-area file system could simplify building distributed applications. In particular, this talk presents a preliminary design, called WheelFS. WheelFS’s goal is to ease the development of distributed applications such as cooperative Web caches, data-intensive Grid applications, and PlanetLab measurements, perhaps reducing the storage management code to a simple script around the application’s core logic. Towards this goal, WheelFS adopts many features from existing file systems, and adds two new ones: semantic cues, and write-locally-read-globally. This talk will also discuss several specific applications that can potentially benefit from using WheelFS.