Declarative Rules for Inferring Fine-Grained Data Provenance from Scientific Workflow Execution Traces

  • Shawn Bowers
  • Timothy McPhillips
  • Bertram Ludäscher
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7525)


Fine-grained dependencies within scientific workflow provenance specify lineage relationships between a workflow result and the input data, intermediate data, and computation steps used in the result’s derivation. This information is often needed to determine the quality and validity of scientific data, and as such, plays a key role in both provenance standardization efforts and provenance query frameworks. While most scientific workflow systems can record basic information concerning the execution of a workflow, they typically fall into one of three categories with respect to recording dependencies: (1) they rely on workflow computation steps to declare dependency relationships at runtime; (2) they impose implicit assumptions concerning dependency patterns from which dependencies are automatically inferred; or (3) they do not assert any dependency information at all. We present an alternative approach that decouples dependency inference from workflow systems and underlying execution traces. In particular, we present a high-level declarative language for expressing explicit dependency rules that can be applied (at any time) to workflow trace events to generate fine-grained dependency information. This approach not only makes provenance dependency rules explicit, but allows rules to be specified and refined by different users as needed. We present our dependency rule language and implementation that rewrites dependency rules into relational queries over underlying workflow traces. We also demonstrate the language using common types of dependency patterns found within scientific workflows.


Data Item Abstract Model Execution Trace Dependency Information Dependency Pattern 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Altintas, I., Barney, O., Jaeger-Frank, E.: Provenance Collection Support in the Kepler Scientific Workflow System. In: Moreau, L., Foster, I. (eds.) IPAW 2006. LNCS, vol. 4145, pp. 118–132. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  2. 2.
    Amsterdamer, Y., Davidson, S.B., Deutch, D., Milo, T., Stoyanovich, J., Tannen, V.: Putting lipstick on pig: Enabling database-style workflow provenance. PVLDB 5(4) (2011)Google Scholar
  3. 3.
    Anand, M.K., Bowers, S., McPhillips, T.M., Ludäscher, B.: Efficient provenance storage over nested data collections. In: EDBT (2009)Google Scholar
  4. 4.
    Bowers, S., McPhillips, T., Riddle, S., Anand, M.K., Ludäscher, B.: Kepler/pPOD: Scientific Workflow and Provenance Support for Assembling the Tree of Life. In: Freire, J., Koop, D., Moreau, L. (eds.) IPAW 2008. LNCS, vol. 5272, pp. 70–77. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Davidson, S.B., Freire, J.: Provenance and scientific workflows: challenges and opportunities. In: SIGMOD (2008)Google Scholar
  6. 6.
    Gil, Y., et al.: Examining the challenges of scientific workflows. IEEE Computer 40(12), 24–32 (2007)CrossRefGoogle Scholar
  7. 7.
    Lee, E., Parks, T.: Dataflow process networks. Proc. of the IEEE 83(5), 773–799 (1995)CrossRefGoogle Scholar
  8. 8.
    Lim, C., Lu, S., Chebotko, A., Fotouhi, F.: Prospective and retrospective provenance collection in scientific workflow environments. In: IEEE SCC, pp. 449–456 (2010)Google Scholar
  9. 9.
    Ludäscher, B., Podhorszki, N., Altintas, I., Bowers, S., McPhillips, T.M.: From computation models to models of provenance: the rws approach. Concurrency and Computation: Practice and Experience 20(5), 507–518 (2008)CrossRefGoogle Scholar
  10. 10.
    Ludäscher, B., et al.: Scientific workflow management and the Kepler system. Concurrency and Computation: Practice and Experience 18(10) (2006)Google Scholar
  11. 11.
    McPhillips, T., Bowers, S., Zinn, D., Ludäscher, B.: Scientific workflow design for mere mortals. Future Generation Computer Systems 25(5) (2009)Google Scholar
  12. 12.
    Misra, A., Blount, M., Kementsietsidis, A., Sow, D., Wang, M.: Advances and Challenges for Scalable Provenance in Stream Processing Systems. In: Freire, J., Koop, D., Moreau, L. (eds.) IPAW 2008. LNCS, vol. 5272, pp. 253–265. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  13. 13.
    Missier, P., Paton, N., Belhajjame, K.: Fine-grained and efficient lineage querying of collection-based workflow provenance. In: EDBT (2010)Google Scholar
  14. 14.
    Missier, P., Soiland-Reyes, S., Owen, S., Tan, W., Nenadic, A., Dunlop, I., Williams, A., Oinn, T., Goble, C.: Taverna, Reloaded. In: Gertz, M., Ludäscher, B. (eds.) SSDBM 2010. LNCS, vol. 6187, pp. 471–481. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  15. 15.
    Moreau, L., et al.: The open provenance model core specification (v1.1). Future Generation Computer Systems 27(6), 743–756 (2011)MathSciNetCrossRefGoogle Scholar
  16. 16.
  17. 17.
    Simmhan, Y.L., et al.: A survey of data provenance in e-science. SIGMOD Record 34(3) (2005)Google Scholar
  18. 18.
    The W3C Provenance Working Group,

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Shawn Bowers
    • 1
  • Timothy McPhillips
    • 2
  • Bertram Ludäscher
    • 3
  1. 1.Dept. of Computer ScienceGonzaga UniversityUSA
  2. 2.Stanford Synchrotron Radiation Lightsource, SLAC National Accelerator LaboratoryStanford UniversityUSA
  3. 3.Dept. of Computer ScienceUniversity of California DavisUSA

Personalised recommendations