DYNAMO - DirectorY, Net Archiver and MOver

Abstract

The Grid communities efforts on managing and transporting data have focused on very large data sets consisting of very large elements.We are interested in leveraging the benefits of solutions such GridFTP, in particular with respect to parallel data transfer and restartability (as well as security, third party control, etc.), for moving large data sets consisting of very large numbers of small objects, e.g., moving a file system subtree. In addition, we require a solution that 1) imposes constant memory overhead on the client and server systems, 2) is independent of the actual transfer mechanism used so we can easily take advantage of advances in technologies for transferring large files, 3) works well even for very large collections of very small files and 4) is a complete solution, i.e., reproduces the directory tree at the server. In this paper, we present Dynamo which is our tool built on GridFTP for transferring directory subtrees. In addition to describing the architecture and implementation of Dynamo, we present performance results on a range of networks showing that we have met our goals.