Skip to main content
SpringerLink
Log in

Unmanaged Workflows: Their Provenance and Use

  • Chapter
Data Provenance and Data Management in eScience

Part of the book series: Studies in Computational Intelligence ((SCI,volume 426))

Abstract

Provenance of scientific data will play an increasingly critical role as scientists are encouraged by funding agencies and grand challenge problems to share and preserve scientific data. But it is foolhardy to believe that all human processes, particularly as varied as the scientific discovery process, will be fully automated by a workflow system. Consequently, provenance capture has to be thought of as a problem applied to both human and automated processes. The unmanaged workflow is the full human-driven activity, encompassing tasks whose execution is automated by an orchestration tool, and tasks that are done outside an orchestration tool. In this chapter we discuss the implications of the unmanaged workflow as it affects provenance capture, representation, and use. Illustrations of capture include multiple experiences with unmanaged capture using the Karma tool. Illustrations of use include defining workflows by suggesting additions to workflow designs under construction, reconstructing process traces, and using analysis tools to assess provenance quality.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Droegemeier, K., Gannon, D., Reed, D., Plale, B., et al.: Service-oriented environments for dynamically interacting with mesoscale weather. Computing in Science and Engineering. IEEE Computer Society Press and American Institute of Physics 7(6), 12–29 (2005)

    Google Scholar 

  2. Folino, G., Forestiero, A., Papuzzo, G., Spezzano, G.: A grid portal for solving geoscience problems using distributed knowledge discovery services. Future Generation Computer Systems 26(1), 87–96 (2010)

    Article  Google Scholar 

  3. Deelman, E., Gannon, D., Shields, M., Taylor, I.: Workflows and e-Science: An overview of workflow system features and capabilities. Future Generation Computer Systems 25(5), 528–540 (2009)

    Article  Google Scholar 

  4. Simmhan, Y., Plale, B., Gannon, D.: A survey of data provenance in e-Science. ACM SIGMOD Record 34(3), 31–36 (2005)

    Article  Google Scholar 

  5. Simmhan, Y., Plale, B., Gannon, D.: Towards a Quality Model for Effective Data Selection in Collaboratories. In: IEEE Workshop on Workflow and Data Flow for Scientific Applications (SciFlow 2006), Held in Conjunction with ICDE, Atlanta, GA (2006)

    Google Scholar 

  6. Mukhi, N.K.: Monitoring Unmanaged Business Processes. In: 18th Int’l Conf. on Cooperative Information Systems, CoopIS (2010)

    Google Scholar 

  7. Moreau, L., Clifford, B., Freire, J., Futrelle, J., Gil, Y., Groth, P., Kwasnikowska, N., Miles, S., Missier, P., Myers, J., Plale, B., Simmhan, Y., Stephan, E., Bussche, J.V.D.: The Open Provenance Model core specification (v1.1). Future Generation Computer Systems 27(6), 743–756 (2010) ISSN 0167-739X, 10.1016/j.future.2010.07.005

    Article  Google Scholar 

  8. Groth, P., Miles, S., Moreau, L.: PReServ: Provenance Recording for Services. In: UK e-Science All Hands Meeting 2005, Nottingham, UK (September 2005)

    Google Scholar 

  9. Groth, P., Moreau, L.: Recording Process Documentation for Provenance. IEEE Trans. Parallel Distrib. Syst. 20(9), 1246–1259 (2009)

    Article  Google Scholar 

  10. Wen, L., Wang, J., van der Aalst, W.M.P., Huang, B., Sun, J.: Mining Process Models with Prime Invisible Tasks. Data and Knowledge Engineering 69(10), 999–1021 (2010)

    Article  Google Scholar 

  11. Zhang, J., Liu, Q., Xu, K.: Flow Recommender: a workflow recommendation technique for process provenance. In: Proceedings of the Eighth Australasian Data Mining Conference (AusDM 2009), Brisbane, Australia (December 2009)

    Google Scholar 

  12. Koop, D., Scheidegger, C.E., Callahan, S.P., Freire, J., Silva, C.T.: Viscomplete: Automating Suggesstions for Visualization Pipelines. IEEE Transactions on Visualisation and Computer Graphics 14(6), 1691–1698 (2008)

    Article  Google Scholar 

  13. Antonatos, S., Anagnostakis, K., Markatos, E.: Generating realistic workloads for network intrusion detection systems. In: ACM Workshop on Software and Performance, Redwood Shores, CA, USA (2004)

    Google Scholar 

  14. Noble, B.D., Satyanarayanan, M., Nguyen, G.T., Katz, R.H.: Trace-Based Mobile Network Emulation. In: Proceedings of SIG-COMM 1997, Cannes, France, pp. 51–61 (September 1997)

    Google Scholar 

  15. Curbera, F., Doganata, Y.N., Martens, A., Mukhi, N., Slominski, A.: Business provenance - a technology to increase tracibility of end-to-end operations. In: OTM Conferences, vol. (1), pp. 100–119 (2008)

    Google Scholar 

  16. Davidson, S.B., Freire, J.: Provenance and scientific workflows: challenges and opportunities. In: Proceedings of ACM SIGMOD, pp. 1345–1350 (2008)

    Google Scholar 

  17. Doganata, Y., Curbera, F.: Effect of Using Automated Auditing Tools on Detecting Compliance Failures in Unmanaged Processes. In: Dayal, U., Eder, J., Koehler, J., Reijers, H.A. (eds.) BPM 2009. LNCS, vol. 5701, pp. 310–326. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  18. Bodnarchuk, R.R., Bunt, R.B.: A synthetic workload model for a distributed systems file server. In: Proceedings of the SIGMETRICS Interational Conference on Measurement and Modeling of Computer Systems, pp. 50–59 (1991)

    Google Scholar 

  19. Mehra, P., Wah, B.: Synthetic Workload Generation for Load-balancing Experiments. IEEE Parallel and Distributed Technology 3(3), 4–19 (1995)

    Article  Google Scholar 

  20. Sreenivasan, K., Kleinman, A.J.: On the construction of a representative synthetic workload. Communications of the ACM, 127–133 (1974)

    Google Scholar 

  21. Freire, J., Koop, D., Santos, E., Silva, C.T.: Provenance for Computational Tasks: A Survey. Computing in Science and Engineering 10(3), 11–21 (2008)

    Article  Google Scholar 

  22. Missier, P., Sahoo, S.S., Zhao, J., Goble, C., Sheth, A.: Janus: From Workflows to Semantic Provenance and Linked Open Data. In: McGuinness, D.L., Michaelis, J.R., Moreau, L. (eds.) IPAW 2010. LNCS, vol. 6378, pp. 129–141. Springer, Heidelberg (2010), doi:10.1007/978-3-642-17819-1-16.

    Chapter  Google Scholar 

  23. Frew, J., Janée, G., Slaughter, P.: Automatic Provenance Collection and Publishing in a Science Data Production Environment—Early Results. In: McGuinness, D.L., Michaelis, J.R., Moreau, L. (eds.) IPAW 2010. LNCS, vol. 6378, pp. 27–33. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  24. Holland, D., Seltzer, M., Braun, U., Muniswamy-Reddy, K.: PASSing the provenance challenge. Concurrency and Computation: Practice and Experience 20(5), 531–540 (2008)

    Article  Google Scholar 

  25. Gu, W., Eisenhauer, G., Schwan, K.: Falcon: On-line Monitoring and Steering of Parallel Programs. Concurrency Practice and Experience 10(9), 699–736 (1998)

    Article  MATH  Google Scholar 

  26. Newhouse, S., Schopf, J., Richards, A., Atkinson, M.: Study of user priorities for e-infrastructure for e- research (SUPER). In: Proc. of the UK e-Science All Hands Conference (2007)

    Google Scholar 

  27. Scheidegger, C., Koop, D., Santos, E., Vo, H., Callahan, S., Freire, J., Silva, C.: Tackling the Provenance Challenge one layer at a time. Concurrency and Computation: Practice and Experience 20(5), 473–483 (2008)

    Article  Google Scholar 

  28. Bose, R., Frew, J.: Lineage retrieval for scientific data processing: a survey. ACM Comput. Survey 37(1), 1–28 (2005)

    Article  Google Scholar 

  29. Moreau, L.: The foundations for provenance on the web. Foundations and Trends in Web Science 2(2-3), 99–241 (2010)

    Article  MathSciNet  Google Scholar 

  30. Altintas, I., Barney, O., Jaeger-Frank, E.: Provenance Collection Support in the Kepler Scientific Workflow System. In: Moreau, L., Foster, I. (eds.) IPAW 2006. LNCS, vol. 4145, pp. 118–132. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  31. Oinn, T., Greenwood, M., Addis, M., Alpdemir, N., Ferris, J., Glover, K., Goble, C., Goderis, A., Hull, D., Marvin, D., Li, P., Lord, P., Pocock, M., Senger, M., Stevens, R., Wipat, A., Wroe, C.: Taverna: lessons in creating a workflow environment for the life sciences. Concurrency and Computation: Practice and Experience 18(10), 1067–1100 (2006)

    Article  Google Scholar 

  32. Zhao, J., Wroe, C., Goble, C., Stevens, R., Quan, D., Greenwood, M.: Using Semantic Web Technologies for Representing E-science Provenance. In: McIlraith, S.A., Plexousakis, D., van Harmelen, F. (eds.) ISWC 2004. LNCS, vol. 3298, pp. 92–106. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  33. Zhao, J., Goble, C., Stevens, R., Turi, D.: Mining Taverna’s semantic web of provenance. Concurrency and Computation: Practice and Experience 20(5), 463–472 (2008)

    Article  Google Scholar 

  34. Barga, R., Simmhan, Y., Withana, E.C., Sahoo, S., Jackson, J.: Provenance for Scientific Workflows Towards Reproducible Research. Bulletin of Technical Committee on Data Engineering, Special Issue on Data Provenance 33(3), 50–58 (2010)

    Google Scholar 

  35. Valerio, M., Sahoo, S., Barga, R., Jackson, J.: Capturing Workflow Event Data for Monitoring, Performance Analysis, and Management of Scientific Workflows. In: IEEE Fourth Int’l Conf. on e-Science 2008 (e-Science 2008), pp. 626–633 (2008)

    Google Scholar 

  36. Miles, S., Groth, P., Branco, M., Moreau, L.: The requirements of recording and using provenance in e-science experiments. Journal of Grid Computing 5(1), 1–25 (2007)

    Article  Google Scholar 

  37. PC3, http://twiki.ipaw.info/bin/view/Challenge/ThirdProvenanceChallenge (accessed December 20, 2009)

  38. Data to Insight Center, http://pti.iu.edu/d2i/provenance-karma (accessed January 2011)

  39. RabbitMQ Messaging System, http://www.rabbitmq.com (accessed July 2011)

  40. The WS-BPEL Extension for People (BPEL4People), Version 1.0 Specification, http://www.ibm.com/developerworks/webservices/library/specification/ws-bpel4people (accessed December 2011)

  41. Cao, B., Plale, B., Subramanian, G., Robertson, E., Simmhan, Y.: Provenance Information Model of Karma. In: IEEE Third Int’l Workshop on Scientific Workflows (SWF 2009), Los Angeles, CA (July 2009)

    Google Scholar 

  42. Mukhi, N.K.: Monitoring Unmanaged Business Processes. In: Meersman, R., Dillon, T.S., Herrero, P. (eds.) OTM 2010. LNCS, vol. 6426, pp. 44–59. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  43. Doganata, Y., Curbera, F.: Effect of Using Automated Auditing Tools on Detecting Compliance Failures in Unmanaged Processes. In: Dayal, U., Eder, J., Koehler, J., Reijers, H.A. (eds.) BPM 2009. LNCS, vol. 5701, pp. 310–326. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  44. Plale, B., Cao, B., Aktas, M.: Provenance Collection of Unmanaged Workflows with Karma. Journal Manuscript Accepted with Revisions (July 2011)

    Google Scholar 

  45. Shirasuna, S.: A Dynamic Scientific Workflow System for the Web Services Architecture. PhD thesis, Indiana University (September 2007)

    Google Scholar 

  46. Gil, Y., Ratnakar, V., Deelman, E., Mehta, G., Kim, J.: Wings for Pegasus: Creating Large-Scale Scientific Applications Using Semantic Representations of Computational Workflows, pp. 1767–1774. AAAI (2007)

    Google Scholar 

  47. Kim, J., Gil, Y., Spraragen, M.: Principles for interactive acquisition and validation of workflows. J. Exp. Theor. Artif. Intell. 22(2), 103–134 (2010)

    Article  MATH  Google Scholar 

  48. Leake, D.B.: Case-Based Reasoning in Context: The Present and Future. In: Leake, D.B. (ed.) Case-Based Reasoning: Experiences, Lessons, and Future Directions, pp. 1–35. AAAI Press/MIT Press (1996)

    Google Scholar 

  49. de Mántaras, R.L., McSherry, D., Bridge, D.G., Leake, D.B., Smyth, B., Craw, S., Faltings, B., Maher, M.L., Cox, M.T., Forbus, K.D., Keane, M.T., Aamodt, A., Watson, I.D.: Retrieval, reuse, revision and retention in case-based reasoning. Knowledge Eng. Review 20(3), 215–240 (2005)

    Article  Google Scholar 

  50. Leake, D.B., Kendall-Morwick, J.: Towards Case-Based Support for e-Science Workflow Generation by Mining Provenance. In: Althoff, K.-D., Bergmann, R., Minor, M., Hanft, A. (eds.) ECCBR 2008. LNCS (LNAI), vol. 5239, pp. 269–283. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  51. Leake, D., Kendall-Morwick, J.: Four Heads Are Better than One: Combining Suggestions for Case Adaptation. In: McGinty, L., Wilson, D.C. (eds.) ICCBR 2009. LNCS, vol. 5650, pp. 165–179. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  52. Cheah, Y.-W., Plale, B., Kendall-Morwick, J., Leake, D., Ramakrishnan, L.: A Noisy 10GB Provenance Database. In: Second International Workshop on Traceability and Compliance of Semi-Structured Processes, Clermont-Ferrand, France (2011) (in press)

    Google Scholar 

  53. Kendall-Morwick, J., Leake, D.: A Toolkit for Representation and Retrieval of Structured Cases. In: Proceedings of the ICCBR 2011 Workshop on Process-Oriented Case-Based Reasoning, Greenwich, U.K. (2011) (in press)

    Google Scholar 

  54. Ramakrishnan, L., Plale, B., Gannon, D.: WORKEM: Representing and Emulating Distributed Scientific Workflow Execution State. In: Proceedings of the 10th IEEE/ACM Int’l Symposium on Cluster, Cloud and Grid Computing (CCGrid 2010), Melbourne Australia (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Data to Insight Center, Indiana University, Bloomington, USA

    Mehmet S. Aktas & Beth Plale

  2. School of Informatics and Computing, Indiana University, Bloomington, USA

    Beth Plale & David Leake

  3. IBM T. J. Watson Research Center, Yorktown Heights, USA

    Nirmal K. Mukhi

Authors
  1. Mehmet S. Aktas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Beth Plale
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Leake
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nirmal K. Mukhi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehmet S. Aktas .

Editor information

Editors and Affiliations

  1. , Information and Communications, CSIRO, Hobart, 7001, Tasmania, Australia

    Qing Liu

  2. , School of Computing & Mathematical, Auckland University of Technology, Private Bag 92006, Auckland, 1142, New Zealand

    Quan Bai

  3. , Information and Communications, CSIRO, Hobart, 7001, Tasmania, Australia

    Stephen Giugni

  4. , Information Management and Technology, CSIRO, Acton, 2601, Aust Capital Terr, Australia

    Darrell Williamson

  5. Mathematics Informatics and Statistics, CSIRO, Acton, 2601, Aust Capital Terr, Australia

    John Taylor

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Aktas, M.S., Plale, B., Leake, D., Mukhi, N.K. (2013). Unmanaged Workflows: Their Provenance and Use. In: Liu, Q., Bai, Q., Giugni, S., Williamson, D., Taylor, J. (eds) Data Provenance and Data Management in eScience. Studies in Computational Intelligence, vol 426. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29931-5_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-29931-5_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-29930-8

  • Online ISBN: 978-3-642-29931-5

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation