Keywords

figure a

1 Introduction

Workflows have been widely used to model large-scale scientific workloads. The explicit definition of true dependencies between subsequent steps allows inferring concurrent execution strategies automatically, improving performances, and transferring input and output data wherever needed, fostering large-scale distributed executions. However, current Workflow Management Systems (WMSs) struggle to keep up with the ever-more demanding requirements of modern scientific applications, such as interoperability between different systems, distributed-by-design workflow models, and automatic optimisation of data movements.

With the advent of BigData, adopting a proper data management strategy has become a crucial aspect of large-scale workflow orchestration. Avoiding unnecessary data movements and coalescing data transfers are two established techniques for performance optimisation in distributed executions. Moving computation near data to remove the need for data transfers is the underlying principle of several modern approaches to large-scale executions, like Resilient Distributed Datasets [42] and in-situ workflows [4].

WMSs’ interoperability is an open problem in scientific workflows, which hinders reusability and composability. Despite several attempts to model product-agnostic workflow languages [11] and representations [28] present in the literature, these solutions capture only a subset of features, forcing WMSs to reduce their expressive power in the name of portability. The main issue in unifying workflow representations resides in the heterogeneity of different WMSs’ APIs and programming models tailored to the needs of a domain experts. Conversely, moving the interoperability efforts to the lower level of the workflow execution plan representation is a promising but still relatively unexplored alternative.

The heterogeneity in contemporary hardware resources and their features, further exacerbated by the end-to-end co-design approach [30], requires WMSs to support a large ecosystem of execution environments (from HPC to cloud, to the Edge), optimisation policies (performance vs. energy efficiency) and computational models (from classical to quantum). However, maintaining optimised executors for such diverse execution targets is an overarching effort. In this setting, a just-in-time compilation of target-specific execution bundles, optimised for a single workflow running in a single execution environment, would be a game-changing approach. Indeed, this approach allows for the efficient use of resources, as the compilation is done at the time of execution, taking into account the specific characteristics of the execution environment. It also ensures the effectiveness of the execution, as the compiled bundle is optimised for the specific workflow, leading to improved performance.

This work presents SWIRL, a “Scientific Workflow Intermediate Representation Language”. Unlike other product-agnostic workflow languages, SWIRL is not intended for human interaction but serves as a low-level compilation target for distributed workflow execution plans. It models the execution plan of a location-aware workflow graph as a distributed system with send/receive communication primitives. This work provides a formal method to encode a workflow instance into a distributed execution plan using these primitives, promoting interoperability and composability of different workflow models. It also includes a set of rewriting rules for automatic optimisation of data communications with correctness and consistency guarantees. The optimised SWIRL representation can then be compiled into one or more self-contained executable bundles, making it adaptable to specific execution environments and embracing heterogeneity.

The SWIRL implementation follows the same line as the theoretical approach, separating scientific workflows’ design and runtime phases. A SWIRL-based compiler translates a workflow system \(\texttt{W}\) to a high-performance, self-contained workflow execution bundle based on send/receive communication protocols and runtime libraries, which can easily be included in a Research Object [5], significantly improving reproducibility.

In detail, Sect. 2 introduces a generic formalism for representing distributed scientific workflow models, while the related work and the comparison with the SWIRL language is given in Sect. 2.1. Section 3 introduces the SWIRL semantics, and Sect. 4 derives the rewriting rules used for optimisation. Section 5 describes the implementation of the SWIRL compiler toolchain while Sect. 6 shows how to model the 1000 Genomes workflow [35], a Bioinformatics pipeline aiming at fetching, parsing and analysing data from the 1000 Genomes Project [39] into SWIRL system. Finally, Sect. 7 concludes the article. Full proofs and additional material can be found in [10] while the experiment is in [9].

2 Background and Related Work

This section gathers the related work (Sect. 2.1) and introduces a formal representation of scientific workflows (Sect. 2.2) and their mapping onto distributed and heterogeneous execution environments (Sect. 2.3).

2.1 Related Work

Location-Aware WMSs. Grid-native WMSs typically support distributed workflows out of the box, providing automatic scheduling and data transfer management across multiple execution locations. However, all the orchestration aspects are delegated to external, grid-specific technologies, limiting the spectrum of supported execution environments. For instance, Triana [36], Askalon [14] and Pegasus [12] delegate tasks offloading and data transfers to the GAP interface [37], the GLARE library [34], and HTCondor [38], respectively.

Recently, a new class of location-aware WMSs is bringing advantages in performance and costs of workflow executions on top of heterogeneous distributed environments. StreamFlow [8] allows users to explicitly map each step onto one or more locations in charge of its execution. It relies on a set of connectors to support several execution environments, from HPC queue managers to microservices orchestrators. Jupyter Workflow [7] transforms a sequential computational notebook into a distributed workflow by mapping each cell into one or more execution locations, semi-automatically extracting inter-cell data dependencies from the code, and delegating the runtime orchestration to StreamFlow. Mashup [32] automatically maps each workflow step onto the best-suited location, choosing between traditional Cloud VMs and serverless platforms.

Each tool has its own strategy to derive an execution plan from a workflow graph without relying on an explicit and consolidated intermediate representation. Moreover, none of them formalise this derivation process, hiding its details inside the WMS’s codebase. Instead, relying on a common intermediate language like SWIRL would allow interoperability between different tools and formal correctness guarantees on the adopted optimisation strategies.

Formal Models for Distributed Workflows. In the literature, the number of different WMSs is notable [3], however, up to our knowledge, there are only a few WMS for which formal models have been developed: Taverna [41], employing the lambda calculus [25] to define the workflow language in functional terms; Kepler [21] adopting Process Networks [18] and BPEL [26], where the workflow language is formalised with Petri Nets. YAWL [1] is another workflow language based on Petri Nets extended with constructs to address the multiple instances, advanced synchronisation, and cancellation patterns. It provides a detailed representation of workflow patterns [2] supported by an open-source environment.

Process algebra, in particular, different versions of \(\pi \)-calculus [33] are suited to model the workflow system due to the ability of processes to change their structure dynamically. A class of workflow patterns has been precisely defined using the execution semantics of \(\pi \)-calculus, in [29], while the basic control flow constructs modelled by \(\pi \)-calculus are given in [13]. A distributed extension of \(\pi \)-calculus [15] is examined as a formalisation for distributed workflow systems in [23], providing a discussion on the flexibility of the proposed representation. Aside from \(\pi \)-calculus, CCS (Calculus of Communication Systems) [24] models Web Service Choreography Interface descriptions.

2.2 Scientific Workflow Models

A generic workflow can be represented as a directed bipartite graph, where the nodes refer to either the computational steps of a modular application or the ports through which they communicate, and the edges encode dependency relations between steps.

Definition 1

A workflow is a directed bipartite graph \(W = (S, P, \mathcal {D})\) where S is the set of steps, P is the set of ports, and \(\mathcal {D}\subseteq (S \times P) \cup (P \times S)\) is the set of dependency links.

In the considered graph, one port can have multiple output edges meaning that more steps are dependent on it. The sets of input/output ports (steps) of a step (port) are defined with the following definition.

Definition 2

Given a workflow \(W=(S,P,\mathcal {D})\),a step \(s\in S\) and a port \(p\in P\), the sets of input and output ports of s are denoted with \( In (s)\) and \( Out (s)\), respectively, and defined as:

$$\begin{aligned} In (s)=\{p'\;|\;(p',s)\in \mathcal {D}\} \qquad Out (s)=\{p'\;|\;(s,p')\in \mathcal {D}\} \end{aligned}$$

while the sets of input and output steps of p are denoted with \( In (p)\) and \( Out (p)\), respectively, and defined as:

$$\begin{aligned} In (p)=\{s'\;|\;(s',p)\in \mathcal {D}\} \qquad Out (p)=\{s'\;|\;(p,s')\in \mathcal {D}\} \end{aligned}$$

Traditionally, scientific workflows are modelled using a dataflow approach, i.e., following token-pushing semantics in which tokens carry data values. The step executions are enabled by the presence of tokens in their input ports and produce new tokens in their output ports. In general, a single workflow model can generate infinite workflow instances. Different instances preserve the same graph structure but differ in the values carried by each token.

Definition 3

A workflow instance is a tuple \((W, D , \mathcal {I})\) where \(W=(S,P,\mathcal {D})\) is a workflow, \(D \) is a set of data elements, and \(\mathcal {I}\subseteq (D \times P)\) is a mapping relation connecting each data element \(d \in D \) to the port \(p \in P\) that contains it.

Definition 4

Given a workflow instance \((W,D ,\mathcal {I})\), where \(W=(S,P,\mathcal {D})\), and a step \(s \in {S}\), the sets of input and output data elements of s are denoted with \( In ^{D}(s)\) and \( Out ^{D}(s)\), respectively, and defined as:

$$\begin{aligned} In ^{D}(s) =\{d\;|\;(d,p)\in \mathcal {I}\;\wedge \; p\in In (s)\}\qquad Out ^{D}(s) =\{d\;|\;(d,p)\in \mathcal {I}\;\wedge \; p\in Out (s)\} \end{aligned}$$

Introducing more precise evaluation semantics, triggering strategies, or limitations on the dependencies structure can specialise this general definition to an actual workflow model (e.g., a Petri Net [31] or Coloured Petri Nets [16], a Kahn Processing Network [17], or a Synchronous Dataflow Graph [20]).

2.3 Distributed Workflow Models

A distributed workflow is a workflow whose steps can target different deployment locations in charge of executing them. To compute the step, the corresponding location must have access to or store all the input data elements, additionally, it will store all the output data elements on its local scope. Locations can be heterogeneous, exposing different hardware devices, software libraries, and security levels. Consequently, the steps are explicitly mapped onto execution locations depending on their computing requests. Given that, a distributed workflow model must contain a specification of the workflow structure, the set of available locations, and a mapping relation between steps and locations.

Definition 5

A distributed workflow is a tuple \(\left( W, L, \mathcal {M}\right) \), where \(W = \left( S, P, \mathcal {D}\right) \) is a workflow, L is the set of available locations, and \(\mathcal {M}\subseteq (S \times L)\) is a mapping relation stating which locations are in charge of executing each workflow step.

Each location can execute multiple steps on it, and a single step can be mapped onto multiple locations. Multiple steps related to a single location introduce a temporal constraint: all the involved steps compete to acquire the location’s resources. They can be serialised if the location does not have enough resources to execute all of them concurrently. Conversely, multiple locations related to a single step express a spatial constraint: all involved locations must collaborate to execute the step. This work does not impose any particular strategy for scheduling different step executions on a single location when temporal constraints arise. However, it is helpful to know the work queue of a given location l, i.e., the set of steps mapped onto it.

Definition 6

Given a distributed workflow \(\left( W, L, \mathcal {M}\right) \), where \(W = \left( S, P, \mathcal {D}\right) \), and a location \(l\in L\), the set of steps mapped onto l is called the work queue of l, denoted as \(Q(l)\) and defined as: \(Q(l) = \{s\;|\;l \in \mathcal {M}(s)\} \).

Similarly to what was discussed in Sect. 2.2, a single distributed workflow model can generate potentially infinite distributed workflow instances with different data elements and condition evaluations.

Definition 7

A distributed workflow instance is a tuple \(\texttt{I}=\left( W, L, \mathcal {M}, D , \mathcal {I}\right) \) where \((W,L,\mathcal {M})\) is a distributed workflow, D is a set of data elements, and \(\mathcal {I}\subseteq (D \times P)\) is a mapping relation connecting each data element \(d \in D \) to the port \(p \in P\) that contains it.

Example 1

Fig. 1 shows an example of a distributed workflow model. A step \(s_{1}\) produces two different output data elements \(d_1\) and \(d_2\), which are mapped to ports \(p_{1}\) and \(p_{2}\). The second and the third step, \(s_{2}\) and \(s_{3}\) depend on the data elements on the ports \(p_{1}\) and \(p_{2}\), respectively. None of them produces other outputs. This workflow is mapped onto four locations. Step \(s_{1}\) is executed on location \(l_{d}\), while \(s_{2}\) is offloaded to \(l_{1}\) and step \(s_{3}\) is mapped to two locations \(l_{2}\) and \(l_{3}\). Using definitions above, Fig. 1 can be written as follows:

$$\begin{aligned} \begin{aligned} W &= \left( \left\{ s_{1}, s_{2}, s_{3}\right\} \!, \left\{ p_{1}, p_{2}\right\} \!, \left\{ (s_{1}, p_{1}), (s_{1}, p_{2}), (p_{1}, s_{2}), (p_{2}, s_{3})\right\} \right) \\ L &= \left\{ l_{d}, l_{1}, l_{2}, l_{3}\right\} \qquad \qquad \qquad \mathcal {M}= \left\{ \left( s_{1}, l_{d}\right) \!, \left( s_{2}, l_{1}\right) \!, \left( s_{3}, l_{2}\right) \!, \left( s_{3}, l_{3}\right) \right\} \\ D & = \left\{ d_{1}, d_{2}\right\} \qquad \qquad \;\;\qquad \qquad \mathcal {I}= \left\{ \left( d_{1}, p_{1}\right) \!, \left( d_{2}, p_{2}\right) \right\} \end{aligned} \end{aligned}$$
Fig. 1.
figure 1

Example of a distributed workflow model. Steps are represented as squares and ports as circles. Dependency links between steps and ports are depicted as arrows with black-filled heads. Locations are represented as squashed rectangles. Mapping relations are expressed as dotted arrows. A potential instance of this model can be derived by adding data elements, denoted as sets of values, near their related port.

Full size image

3 The SWIRL Representation

This section introduces SWIRL, a “Scientific Workflow Intermediate Representation Language”. Given a distributed workflow instance (Sect. 2.3), SWIRL can model a decentralised execution plan, called workflow system, by inferring and projecting execution traces on each involved location and specifying inter-location communications using send/receive primitives. The following sections introduce SWIRL syntax and semantics and derive a procedure to formally encode a workflow instance \(\texttt{I}\) into a SWIRL workflow system \(\texttt{W}\).

Fig. 2.
figure 2

SWIRL structural congruence rules.

Full size image
Fig. 3.
figure 3

SWIRL reduction semantics rules.

Full size image

SWIRL models a distributed execution plan as a workflow system \(\texttt{W}\), which can be seen as a parallel composition of location configurations, tuples \(\left\langle l, D , e\right\rangle \), containing the location name l, the set \(D \) of data elements laying on l at a given time, and the execution trace \(e\) representing the actions to be executed on l.

Definition 8

The syntax of a workflow system \(\texttt{W}\) is defined by the following grammar:

$$\begin{aligned} \texttt{W}&::= \left\langle l, D , e\right\rangle \parallel (\texttt{W}_1 \;|\;\texttt{W}_2)\\ e&::= \mu \parallel e_1.e_2 \parallel (e_1\;|\;e_2)\parallel \textbf{0}\\ \mu &::= \texttt{exec}(s,F(s),\mathcal {M}(s)) \parallel \texttt{send}({d}\rightarrowtail {p},l,l')\parallel \texttt{recv}(p,l,l')\\ F(s) &::= In ^{D}(s)\mapsto Out ^{D}(s) \end{aligned}$$

Each execution trace \(e\) is constructed from the predicates \(\mu \), which can be composed using two operators: the sequential execution \(e_{1}.e_{2}\) and the parallel composition \(e_1\;|\;e_2\). The \(\textbf{0}\) symbol represents the empty trace.

A predicate \(\mu \) represents an action to be performed during workflow execution. Predicates \(\texttt{send}({d}\rightarrowtail {p},l,l')\) and \( \texttt{recv}(p,l,l')\) allow transferring the data element d over port p from location l to location \(l'\). Modelling ports and data separately seams redundant, but we prefer to keep them divided for the future extensions of the framework, as adding the loops. The \(\texttt{exec}(s,F(s),\mathcal {M}(s))\) action represents the execution of step s. Besides the name of the step, this predicate contains the set \(\mathcal {M}(s)\) of locations onto which s is mapped and the dataflow F(s), i.e., the set \( In ^{D}(s)\) of input data needed by s and the set \( Out ^{D}(s)\) of output data produced on each \(l \in \mathcal {M}(s)\) after the execution of s.

3.1 Semantics

The SWIRL semantics is defined in terms of a reduction semantics.

Definition 9

The SWIRL semantics is defined by the reduction relation \(\xrightarrow {\; \; \;}\) defined as a smallest relation closed under the rules of Figs. 2 and 3.

The structural congruence properties are reported in Fig. 2. The commutativity of the parallel composition in location and the execution trace level is defined with rule \((\textsc {Comt}_{u})\). For both operators, parallel composition and sequential execution, the identity element is \(\textbf{0}\) (rules \((\textsc {Id}_{\;|\;})\) and \((\textsc {Id}_{.})\)).

The rules of a SWIRL semantics are depicted in Fig. 3. The step execution is performed by the (Exec) rule. It collects all the locations \(\mathcal {M}(s)\) onto which step s is mapped and synchronises the execution action. The data \( Out ^{D}(s)\) produced by the step execution are added to the set \(D_{i}\) in all executing locations. Rule (L-Comm) describes local communication, while rule (Comm) represents a data transfer between two locations. In the latter case, the involved data element is copied to the targeted (receiving) location. Note that communications do not consume the data element on the sending location.

Assuming that configuration \(\left\langle l, D , e_1\right\rangle \) can be computed, rules (L-Par) and (Seq) allow for the execution of the parallel and the sequential composition inside the same location, respectively. The execution of the workflow sub-system as a part of a larger system is allowed by the rule (Par). The (Congr) rule allows the application of structural congruence.

When a step s is mapped onto multiple locations, each of them must contain an \(\texttt{exec}\) predicate with the set of involved locations. Such predicates introduce synchronisation points among different locations, as all involved execution traces must step forward in a single pass. Additionally, each location must have a copy of the input data \( In ^{D}(s)\), requiring multiple \(\texttt{send}\) operations for each element \(d \in In ^{D}(s)\), and will own a copy of \( Out ^{D}(s)\).

Example 2

The behaviour of the distributed workflow instance given in Fig. 1 can be modelled as a workflow system \(\texttt{W}\) with the following syntax:

$$\begin{aligned} &\qquad \qquad \texttt{W}= \left\langle l_{d}, \emptyset , e_{d}\right\rangle \;|\;\prod ^{3}_{i=1} \left\langle l_{i}, \emptyset , e_{i}\right\rangle \\ e_{d} &= \begin{aligned} &\texttt{exec}(s_1,{\emptyset }\mapsto {\{d_1,d_2\}},\{l_{d}\}).\big (\texttt{send}({d_1}\rightarrowtail {p_1},l_{d},l_1) \;|\;\\ &\texttt{send}({d_2}\rightarrowtail {p_2},l_{d},l_2)\;|\;\texttt{send}({d_2}\rightarrowtail {p_2},l_{d},l_3)\big ) \end{aligned}\\ e_{1} &= \texttt{recv}(p_1,l_{d},l_1).\texttt{exec}(s_2,{\{d_1\}}\mapsto {\emptyset },\{l_1\})\\ e_{2} &= \texttt{recv}(p_2,l_{d},l_2).\texttt{exec}(s_3,{\{d_2\}}\mapsto {\emptyset },\{l_2,l_3\})\\ e_{3} &= \texttt{recv}(p_2,l_{d},l_3).\texttt{exec}(s_3,{\{d_2\}}\mapsto {\emptyset },\{l_2,l_3\}) \end{aligned}$$

In the execution trace \(e_{d}\), step \(s_1\) is sending output data \(d_{2}\) to both locations \(l_2, l_3 \in \mathcal {M}(s_{3})\) through the same port \(p_2\).

3.2 Workflow Model Encoding

Example 2 describes the encoding of a distributed workflow instance \(\texttt{I}\) into a workflow system \(\texttt{W}\). This section introduces a formal methodology to perform this encoding automatically for any distributed workflow instance.

In SWIRL, the execution trace \(e_{l}\) of a location \(l \in L\) models the actions required to execute all the steps in its work queue \(Q(l)\). In this respect, \(e_{l}\) can be seen as the parallel composition of building blocks \(B_{l}(s)\), one for each \(s \in Q(l)\). Each building block \(B_{l}(s)\) contains the same sequence of actions: (i) receives all the necessary data elements for the step execution in which case it is necessary to determine all input data elements (\( In ^{D}(s)\)) and for each element to identify the step producing it (\( In (\mathcal {I}(d_i))\)) and the locations on which the steps are mapped to (\(\mathcal {M}( In (\mathcal {I}(d_i)))\)) (ii) executes the step s; (iii) sends the produced data elements (\( Out ^{D}(s)\)) to the locations onto which the receiving steps are mapped (one data element can be sent, over the same port, to the different steps/locations, therefore it is necessary to identify the steps data \(d_i\) is sent to with \( Out (\mathcal {I}(d_i))\) and the locations \(l_j\) on which each step is deployed).

Definition 10

Given a distributed workflow instance \(\texttt{I}=\left( W, L, \mathcal {M}, D , \mathcal {I}\right) \), a deployment location \(l\in L\) and a step \(s \in S\) s.t. \(l\in \mathcal {M}(s)\), the building block representing s in \(e_{l}\) is denoted by \(B_{l}(s)\) and defined as:

$$\begin{aligned} B_{l}(s)= & \left( \prod ^{\forall d_i \in In ^{D}(s)} \prod ^{\forall l_j \in \mathcal {M}( In (\mathcal {I}(d_i)))}\texttt{recv}(\mathcal {I}(d_i), l_j,l)\right) .\\ & \texttt{exec}(s, { In ^{D}(s)}\mapsto { Out ^{D}(s)},\mathcal {M}(s)).\\ &\left( \prod ^{\forall d_i \in Out ^{D}(s)} \prod ^{\forall s_k \in Out (\mathcal {I}(d_i))} \prod ^{\forall l_j \in \mathcal {M}(s_k)} \texttt{send}({d_{i}}\rightarrowtail {\mathcal {I}(d_i)}, l, l_j )\right) \end{aligned}$$

Definition 10 introduces the general form of \(B_{l}(s)\), which holds for steps connected to both input and output ports. If a step does not consume input data, as in the case of step \(s_{1}\) from Example 2, the receiving part of \(B_{l}(s)\) is modelled with \(\textbf{0}\), resulting in \(B_{l_{d}}(s_{1})=\textbf{0}.\texttt{exec}(s,{\emptyset }\mapsto {\{d_1\}},\{l_{d}\}).\texttt{send}({d_1}\rightarrowtail {p_1},l_{d},l_{1})\). The same applies to steps that do not produce output data.

The concept of building blocks \(B_{l}(s)\) allows for the modular construction of execution traces by processing one pair (s, l) at a time. Intuitively, for each mapping pair step-location (s, l), corresponding building blocks \(B_{l}(s)\) are made and added to the execution trace of the location l. Another important information is the instance data distribution on the locations, denoted by \(G(l) = \{d | d \in D_{l}\}\).

Definition 11

The encoding function \([\![{\cdot }]\!]: \mathcal {W}_{\texttt{I}} \xrightarrow {\; \;} \mathcal {W}_{\texttt{W}}\), where \(\mathcal {W}_{\texttt{I}}\) and \(\mathcal {W}_{\texttt{W}}\) are the sets of distributed workflow instances and workflow systems represented in SWIRL, respectively, is inductively defined as follows:

$$\begin{aligned} & [\![{\texttt{I}}]\!] = [\![{\texttt{I},\mathcal {M},G; \texttt{W}}]\!]\quad \text {where }\texttt{W}=\prod _{\forall l_i\in L} \left\langle l_i, \emptyset , e_l\right\rangle \\ &[\![{\texttt{I},\mathcal {M}\cup (s,l),G;\texttt{W}\;|\;\left\langle l, \emptyset , e_l\right\rangle }]\!] = [\![{\texttt{I},\mathcal {M},G;\texttt{W}\;|\;\left\langle l, \emptyset , e_l\;|\;B_{l}(s)\right\rangle }]\!]\\ &[\![{\texttt{I},\mathcal {M},G\cup G(l);\texttt{W}\;|\;\left\langle l, \emptyset , e_l\right\rangle }]\!] = [\![{\texttt{I},\mathcal {M},G;\texttt{W}\;|\;\left\langle l, G(l), e_l\right\rangle }]\!] \\ &[\![{\texttt{I},\emptyset ,\emptyset ;\texttt{W}}]\!] =\texttt{W}\end{aligned}$$

Formally, the encoding operator can be defined as a function with four input parameters: (i) a workflow instance \(\texttt{I}\) to be translated; (ii) the set of pairs (s, l) containing all mappings in \(\mathcal {M}\); (iii) the set \(G\) representing the distribution of the data over locations; (iv) placeholder to build the workflow system \(\texttt{W}\). The translation starts by adding the auxiliary parameters into the encoding process and inside a SWIRL placeholder, creating a workflow containing locations configurations for each location in the workflow instance \(\texttt{I}\) (for all \(l\in L\)). The iteration process is divided in two phases, first at each iteration, the encoding function takes the pair (s, l), identify the location l and add the building block \(B_{l}(s) \) into the execution trace to be executed on the location l. When all pairs are encoded, in the second phase, the iteration is on the distribution of the data over locations. Each set \(G(l)\subseteq G\) is encoded to the corresponding locations. In that way, the trace \(e_{l}\) is the parallel composition of building blocks \(B_{l}(s)\) for each \(s \in Q(l)\). The encoding finishes when both sets \(\mathcal {M}\) and \(G\) are empty.

3.3 Consistency of SWIRL Semantics

This section defines a concurrency relation on the derivations of a workflow system \(\texttt{W}\), which is then used to show the consistency of different execution diagrams through the semantics. As commonly done in the literature, this section only considers reachable workflow systems defined below.

Definition 12

Given a distributed workflow instance \(\texttt{I}=\left( W, L, \mathcal {M}, D , \mathcal {I}\right) \) and the function \([\![{\cdot }]\!]: \mathcal {W}_{\texttt{I}} \xrightarrow {\; \;} \mathcal {W}_{\texttt{W}} \), the initial state of a distributed workflow system is

$$ \texttt{W}_{Init}=[\![{\texttt{I}, L, *}]\!]= \prod _{l_j\in L} \left\langle l_{j}, \emptyset , \prod _{s\in Q(l_j)} B(s)\right\rangle $$

Definition 13

A state of a workflow system \(\texttt{W}\) is reachable if it can be derived from the initial state (\(\texttt{W}_{Init}\)) by applying the rules in Figs. 2 and 3.

Having a transition \(t:\texttt{W}\xrightarrow {\; \;}\texttt{W}'\), the workflow states \(\texttt{W}\) and \(\texttt{W}'\) are called source and target of the transition t, respectively. The concurrency relation is defined on the transitions having the same source. Formally:

Definition 14 (Concurrency relation)

Two different transitions \(t_1: \texttt{W}\xrightarrow {\; \;}\texttt{W}_1\) and \(t_2: \texttt{W}\xrightarrow {\; \;}\texttt{W}_2\) having the same source, are always concurrent, written \(t_1 \smile t_2\).

Following the standard notation, let \(t_2/t_1\) represent a transition \(t_2\) executed after the transition \(t_1\). The concurrency relation is used to prove the Church-Rosser property, which states that when two concurrent transitions execute at the same time, the ordering of the executions does not impact the eventual result. This finding shows that the concurrent semantics is confluent. Formally:

Lemma 1 (Church-Rosser property)

Given two concurrent transitions \(t_1: \texttt{W}\xrightarrow {\; \;} \texttt{W}_1\) and \(t_2: \texttt{W}\xrightarrow {\; \;} \texttt{W}_2\), there exist two transitions \(t_2/t_1: \texttt{W}_1\xrightarrow {\; \;} \texttt{W}_3\) and \(t_1/t_2: \texttt{W}_2\xrightarrow {\; \;} \texttt{W}_3\) having the same target.

4 Optimisation

This section introduces an optimisation function that scans the entire workflow system to remove redundant communications, improving performance. In particular, there are two cases in which execution traces can be optimised: (i) communications between steps deployed on the same location, which are always redundant; (ii) multiple communications of the same data element between a pair of locations, when different steps mapped onto the destination location require the same input data from the same ports.

The encoding function adds a communication to the workflow system \(\texttt{W}\) every time a data element is required for the execution of a step, no matter if it is already present at the destination location, creating unnecessary communications. For instance, consider a location \(\left\langle l, D, e\right\rangle \) where \(D = \emptyset \) and

$$\begin{aligned} e= \begin{aligned} &\texttt{recv}(p,l_1,l).\texttt{exec}(s,{\{d\}}\mapsto {\{d_1\}},\{l\}).\texttt{send}({d_1}\rightarrowtail {p_1},l,l)\;|\;\\ &\texttt{recv}(p_1,l,l).\texttt{exec}(s_1, {\{d_1\}}\mapsto {\emptyset },\{l\}) \end{aligned} \end{aligned}$$

After the execution of the step s, the data element \(d_1\) is saved on the location l (\(D\cup \{d_1\}\)), therefore the \(\texttt{send}\)/\(\texttt{recv}\) pair does not affect the state of \(\texttt{W}\). By removing the unnecessary communication, the trace \(e\) can be rewritten as:

$$\begin{aligned} e' = \begin{aligned} &\texttt{recv}(p,l_1,l).\texttt{exec}(s,{\{d\}}\mapsto {\{d_1\}},\{l\})\;|\;\texttt{exec}(s_1,{\{d_1\}}\mapsto {\emptyset }, \{l\}) \end{aligned} \end{aligned}$$

The rule (Exec) in Fig. 3, preserves dependency between steps s and \(s_1\) by ensuring that step \(s_1\) will not execute until the required data \(d_1\) is produced.

The second optimisation step is to remove redundant communications between different pairs of locations when the same data element is sent multiple times through the same port. For instance, consider two locations \(\left\langle l, D, e\right\rangle \) and \(\left\langle l', D', e'\right\rangle \) where \(D = D' = \emptyset \) and

$$\begin{aligned} e&= \texttt{recv}(p,l_1,l).\texttt{exec}(s,{\{d\}}\mapsto {\{d_1\}},\{l\}).(\prod ^{3}_{i=1}\texttt{send}({d_1}\rightarrowtail {p_1},l,l'))\\ e' &= \prod ^{3}_{i=1}\texttt{recv}(p_1,l,l').\texttt{exec}(s_i,{\{d_1\}}\mapsto {\emptyset },\{l'\}) \end{aligned}$$

The first location l sends the data element \(d_{1}\) to three steps mapped onto location \(l'\). Transferring the data element only once is enough, as the subsequent communications will not affect the state of \(\texttt{W}\), hence, there is:

$$\begin{aligned} e&= \texttt{recv}(p,l_1,l).\texttt{exec}(s,{\{d\}}\mapsto {\{d_1\}},\{l\}).\texttt{send}({d_1}\rightarrowtail {p_1},l,l')\\ e' &= \begin{aligned} &\texttt{recv}(p_1,l,l').\texttt{exec}(s_k,{\{d_1\}}\mapsto {\emptyset },\{l'\})\;|\;\prod ^{3}_{i=1, i\ne k}\texttt{exec}(s_i,{\{d_1\}}\mapsto {\emptyset },\{l'\}) \end{aligned} \end{aligned}$$

The optimisation of a workflow system \(\texttt{W}\) is defined in terms of three functions: the first and the secondFootnote 1 ones start the optimisation process and controls it till the end, by taking the additional parameter \(A\) (the set of all prefixes/actions) and calling the third function that actually rewrite the execution trace of a location. It goes through the execution traces of the workflow \(\texttt{W}\) and breaks them into single action (prefix) blocks. Analysing the blocks one by one, it performs the following actions: (i) if the predicate is a part of the communication on the same location, it is removed; (ii) if the predicate is already in the set \(A\), it is removed as well (meaning the same data element was already sent to the same location through the same port, just to different step); (iii) otherwise, the predicate is added to the set \(A\) and the drilling function moves to the next element.

Definition 15

Given the workflow system \(\texttt{W}\) and sets of workflow and optimised systems \(\mathcal {W}_{\texttt{W}}\) and \(\mathcal {W}_{\texttt{O}}\), respectively, the optimisation function \(\texttt{W}\), \(\llbracket {\cdot }\rrbracket :\mathcal {W}_{\texttt{W}}\xrightarrow {\; \;}\mathcal {W}_{\texttt{O}}\) is defined in terms of the auxiliary functions, \(\llbracket {\cdot }\rrbracket :\mathcal {W}_{\texttt{W}}\times A\xrightarrow {\; \;}\mathcal {W}_{\texttt{O}}\) and (where \(\texttt{o}\in \{|,.\}\) and \(A_{l,l}=\{\texttt{send}({d}\rightarrowtail {p},l,l), \texttt{recv}(p,l,l)\}\)) as follows:

figure c

The two workflow systems \(\texttt{W}\) and \(\texttt{O}=\llbracket {\texttt{W}}\rrbracket \) are modelling the same behaviour of the distributed workflow system, i.e. the computations of the workflow steps are executed in the same order in both systems with the difference in the number of communications. Therefore, the weak barbed bisimulation [33] is used to define the relation between the distributed workflow system and its optimised version.

To highlight that the executing action is a communication, it is labelled by \(\tau \). Therefore, \(\texttt{W}\xrightarrow {\; \tau \;}\texttt{W}'\) indicates that workflow system \(\texttt{W}\) can evolve into \(\texttt{W}'\) by performing the communication (transfer) action. The reflexive and transitive closure of \(\xrightarrow {\; \tau \;}\) is denoted with and the transition express the ability of the system \(\texttt{W}\) to evolve into \(\texttt{W}'\) by executing some number, possibly zero, of \(\tau \) actions (communications). Given the transition \(\texttt{W}\xrightarrow {\; \;}\texttt{W}'\) (any type of action, including the communication), if the same action can be executed after a certain number of communication actions, it is denoted as .

The observable elements in this setting are the executions of the steps and it is denoted by \(\texttt{W}\downarrow _{\nu }\) (resp. \(\texttt{O}\downarrow _{\nu }\)) where \(\nu =\texttt{exec}(s, F(s), \mathcal {M}(s))\) where the weak barb is denoted by \(\texttt{W}\Downarrow _{\nu }\) (resp. \(\texttt{O}\Downarrow _{\nu }\)), and it is defined as (resp. ). Hence, the barbed bisimulation will check that all the step executions in a workflow system can be matched by the executions in the optimised one.

Definition 16

A relation \(\mathcal {R}\subseteq \mathcal {W}\times \mathcal {O}\) is a weak barbed simulation if \(\texttt{W}\mathcal {R}\llbracket {\texttt{W}}\rrbracket \):

  • \(\texttt{W}\downarrow _{\nu }\) implies \(\llbracket {\texttt{W}}\rrbracket \Downarrow _{\nu }\)

  • \(\texttt{W}\xrightarrow {\; \;}\texttt{W}'\) implies \(\llbracket {\texttt{W}}\rrbracket \Rightarrow \llbracket {\texttt{W}'}\rrbracket \) with \(\texttt{W}' \mathcal {R}\llbracket {\texttt{W}'}\rrbracket \)

A relation \(\mathcal {R}\subseteq \mathcal {W}\times \mathcal {O}\) is a weak barbed bisimulation if \(\mathcal {R}\) and \(\mathcal {R}^{-1}\) are weak barbed simulations. Weak bisimilarity, \(\approx \), is the largest weak barbed bisimulation.

Fig. 4.
figure 4

The SWIRL compiler toolchain.

Full size image

The next theorem shows the operational correspondence between a distributed workflow system and its optimised term.

Theorem 1

For any distributed workflow system \(\texttt{W}\), \(\texttt{W}\approx \llbracket {\texttt{W}}\rrbracket \).

5 Implementation

The SWIRL compiler reference implementationFootnote 2, called swirlc, follows the same line as the theoretical approach, separating scientific workflows’ design and runtime phases. On the one hand, it allows the translation of high-level, product-specific workflow languages designed for direct human interaction to chains of low-level primitives easily understood by distributed runtime systems. A common representation fosters composability and interoperability among different workflow models, which can be easily combined into a single workflow system. Moreover, the translation process is performed with the formal consistency guarantees discussed in Sect. 3.2.

Finally, a SWIRL-based compiler can translate a workflow system \(\texttt{W}\) to a high-performance, self-contained workflow execution bundle based on send/receive communication protocols and runtime libraries, which can easily be included in a Research Object [5], improving reproducibility. An advanced compiler can also generate multiple execution bundles from the same workflow system, each optimised for a different execution environment (e.g., Cloud, HPC, or Edge), improving performance. As a bonus feature, the intrinsically distributed nature of SWIRL execution traces promotes decentralised runtime architectures, avoiding the single point of failure introduced by a centralised control plane.

Fig. 5.
figure 5

Graphical representation of the 1000 Genomes workflow contains five classes of steps mapped to diverse locations: (i) individuals (blue), number of steps n, mapped to locations \(l^{I}_{j}\), \(j\in [1, a]\); (ii) individuals_merge (violet), a single step mapped to location \(l^{IM}\); (iii) sifting (yellow), a single step mapped to location \(l^{IM}\); (iv) mutations_overlap (red), number of steps m, mapped to locations \(l^{MO}_{t}\), \(t\in [1,b],\) and (v) frequency (green) number of steps m, mapped to locations \(l^{F}_{k}\), \(k\in [1, c]\). The initial step \(s_{0}\), mapped to driver location \(l_{d}\) is a step that sends each input data element to the correct location for processing. The mapping between data elements and ports, where \(i\in [1, n]\) and \(h \in [1, m]\), is: \( \mathcal {I}= \left\{ \begin{aligned} & (d^{0}_i, p^{0}_i), (d^{P}_h, p^{P}_h), (d^{0}_{SF}, p^{0}_{SF}),\\ & (d^{I}_i, p^{I}_i),(d^{IM}, p^{IM}) , (d^{SF}, p^{SF}) \end{aligned} \right\} \) (Color figure online)

Full size image

Figure 4 sketches the SWIRL compiler toolchain. We implemented the SWIRL grammar using ANTLR [27], and we automatically generated Python3 parser classes to process the SWIRL syntax. All the components of the SWIRL toolchain rely on these parsers to process \(\mathtt {*.swirl}\) files. An abstract \(\texttt{SWIRLTranslator}\) class implements the encoding function, producing a SWIRL file from a workflow instance \(\texttt{I}\). A concrete implementation specialises the \(\texttt{SWIRLTranslator}\) logic to the semantics of a given workflow language, e.g., CWL [11], DAX (for Pegasus [12]), or the Galaxy Workflow Format (GWF) [40]Footnote 3. A \(\texttt{SWIRLOptimizer}^5\) class implements the optimisation function \(\llbracket {\cdot }\rrbracket :\mathcal {W}_{\texttt{W}}\xrightarrow {\; \;}\mathcal {W}_{\texttt{O}}\), generating an optimised \(\mathtt {*.swirl}\) file. Finally, an abstract \(\texttt{SWIRLCompiler}\) class produces an executable bundle from a \(\mathtt {*.swirl}\) file and a declarative metadata file, which contains additional information not currently modelled in the SWIRL semantics, e.g., step commands, data types and location IP addresses. We have implemented a simple compiler class that generates a multithreaded Python program for each location, relying on TCP sockets for send/receive communications.

6 Evaluation

This section tests the flexibility of the SWIRL representation on the 1000 Genomes workflow [35], a Bioinformatics pipeline aiming at fetching, parsing and analysing data from the 1000 Genomes Project [39] to identify mutational overlaps and provide a null distribution for rigorous statistical evaluation of potential disease-related mutations. However, we used the 1000 Genomes applications written in C++ [22]. Figure 5 shows a slightly simplified version of the 1000 Genomes workflow model. We removed some ports to simplify the notation, but their absence does not affect the reasoning reported in the rest of this Section. Note that the number of locations could be smaller than the number of steps. Hence, there could be a case when more steps are mapped to the same location.

The corresponding workflow system \(\texttt{W}\) can be constructed using the encoding function \([\![{\cdot }]\!] : \mathcal {W}_{\texttt{I}} \xrightarrow {\; \;} \mathcal {W}_{\texttt{W}}\) (Sect. 3.2). It can be written as follows:

$$\begin{aligned} \texttt{W}= &\prod _{i\in \{d,SF,IM\}}\left\langle l^{i}, \emptyset , e^{i}\right\rangle \;|\;\prod _{j=1}^{a} \left\langle l^{I}_{j}, \emptyset , e^{I}_{j}\right\rangle \;|\;\prod _{t=1}^{b} \left\langle l^{MO}_{t}, \emptyset , e^{MO}_{t}\right\rangle \;|\;\prod _{k=1}^{c} \left\langle l^{F}_{k}, \emptyset , e^{F}_{k}\right\rangle \end{aligned}$$

where each execution trace \(e^{*}_{*}\) defines the actions (steps and data transfers) depicted in Fig. 5 to be executed on the corresponding location. For instance, if the driver location is taken, the execution trace \(e^{d}\) is defines as:

$$\begin{aligned} e^{d}= &\prod _{i=1}^{n} \texttt{send}({d^{0}_i}\rightarrowtail {p^{0}_i},l^{d},l^{I}_{j})\;|\;\texttt{send}({d^{0}_{SF}}\rightarrowtail {p^{0}_{SF}},l^{d},l^{SF})\;|\;\\ &\prod _{h=1}^{m} (\texttt{send}({d^{P}_h}\rightarrowtail {p^{P}_h},l^{d},l^{MO}_{t})\;|\;\texttt{send}({d^{P}_h}\rightarrowtail {p^{P}_h},l^{d},l^{F}_{k})) \end{aligned}$$

The full representation of \(\texttt{W}\) is discussed in [10]. The 1000 Genomes workflow modelled above can be reproduced using the SWIRL implementation (Sect. 5). To keep the experiment small and ease reproducibility, the ten homogeneous execution locations and a single chromosome, i.e., a single workflow instance, are considered. The necessary installing package and instructions on how to run the experiment can be find in [9].

7 Conclusion

This work introduced SWIRL, a “Scientific Workflow Intermediate Representation Language” based on send/receive communication primitives. An encoding function maps any workflow instance onto a distributed execution plan \(\texttt{W}\), fostering interoperability and composability of different workflow models. A set of rewriting rules allows for automatic optimisation of data communications, improving performance with correctness and consistency guarantees. The optimised SWIRL representation can be compiled into one or more self-contained executable bundles addressing specific execution environments, ensuring reproducibility and embracing heterogeneity. SWIRL already proved itself to be flexible enough to model a real large-scale scientific workflow (even if still not supporting all features of modern WMSs).

The foundational contribution of SWIRL is to propose a novel direction to solve well-known problems in the field of scientific workflows. Indeed, SWIRL shifts the focus from high-level workflow languages, designed either for direct human interaction or to encode complex, product-specific features, to a low-level minimalistic representation of a workflow execution plan, which is far more manageable from both formalisation methods and compiler toolchains. In this context, we hope that SWIRL can pave the way to a novel, more formal approach to distributed workflow orchestration research.

The formal SWIRL representation gives the possibility to build the formally correct extensions, for instance, adding a type system where the multiparty sessions are enriched with security levels for messages (data in our case) [6] or deriving the causal-consistent reversible framework by applying the approach [19], that later can be used as a base to build fault-tolerance mechanism.