Journal of Grid Computing

, Volume 13, Issue 1, pp 95–113

The Case for Workflow-Aware Storage:An Opportunity Study

  • L. B. Costa
  • H. Yang
  • E. Vairavanathan
  • A. Barros
  • K. Maheshwari
  • G. Fedak
  • D. Katz
  • M. Wilde
  • M. Ripeanu
  • S. Al-Kiswany
Article

DOI: 10.1007/s10723-014-9307-6

Cite this article as:
Costa, L.B., Yang, H., Vairavanathan, E. et al. J Grid Computing (2015) 13: 95. doi:10.1007/s10723-014-9307-6

Abstract

This article evaluates the potential gains a workflow-aware storage system can bring. Two observations make us believe such storage system is crucial to efficiently support workflow-based applications: First, workflows generate irregular and application-dependent data access patterns. These patterns render existing generic storage systems unable to harness all optimization opportunities as this often requires enabling conflicting optimizations or even conflicting design decisions at the storage system level. Second, most workflow runtime engines make suboptimal scheduling decisions as they lack the detailed data location information that is generally hidden by the storage system. This paper presents a limit study that evaluates the potential gains from building a workflow-aware storage system that supports per-file access optimizations and exposes data location. Our evaluation using synthetic benchmarks and real applications shows that a workflow-aware storage system can bring significant performance gains: up to 3x performance gains compared to a vanilla distributed storage system deployed on the same resources yet unaware of the possible file-level optimizations.

Keywords

Large-scale storage system Workflow-aware storage system Workflow runtime engine 

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  • L. B. Costa
    • 1
  • H. Yang
    • 1
  • E. Vairavanathan
    • 1
  • A. Barros
    • 2
  • K. Maheshwari
    • 3
  • G. Fedak
    • 4
  • D. Katz
    • 3
  • M. Wilde
    • 3
  • M. Ripeanu
    • 1
  • S. Al-Kiswany
    • 1
  1. 1.Electrical and Computer EngineeringThe University of British ColumbiaVancouverCanada
  2. 2.Universidade Federal de Campina GrandeCampina GrandeBrazil
  3. 3.University of Chicago & Argonne National LaboratoryLemontUSA
  4. 4.INRIAValbonneFrance

Personalised recommendations