Pabble: parameterised Scribble

Abstract

Many parallel and distributed message-passing programs are written in a parametric way over available resources, in particular the number of nodes and their topologies, so that a single parallel program can scale over different environments. This article presents a parameterised protocol description language, Pabble, which can guarantee safety and progress in a large class of practical, complex parameterised message-passing programs through static checking. Pabble can describe an overall interaction topology, using a concise and expressive notation, designed for a variable number of participants arranged in multiple dimensions. These parameterised protocols in turn automatically generate local protocols for type checking parameterised MPI programs for communication safety and deadlock freedom. In spite of undecidability of endpoint projection and type checking in the underlying parameterised session type theory, our method guarantees the termination of end point projection and type checking.

1 Introduction

Message-passing is becoming a dominant programming model, as witnessed in application programs from high-performance computing scaling over thousands of cores or cloud-based scalable backends of popular web services. These are environments where services are dynamically provided, through choreography of interactions among numerous distributed components. Assuring safety of concurrent software in these environments is a vital concern: Many message-passing libraries, programs and systems are shared and long-lived, and some process sensitive data, so that safety violations such as deadlocks and incompatible messaging patterns or data payloads between senders and receivers can have catastrophic and unexpected consequences [10].

Our proposal for safety assurance for message-passing programs is based on multiparty session types [13]. The methodology considers the specification of a global interaction protocol among multiple participants, from which we can derive a local protocol for an individual participant. Once each program is type-checked against its local protocol, a set of typed programs is guaranteed to run without deadlock or communication mismatches. We based our work on [20], where the authors proposed a programming framework for message-passing parallel algorithms, centering on explicit, formal description of global protocols, and examined its effectiveness through an implementation of a toolchain for the C language. The toolchain uses a language Scribble  [12, 24] for describing the multiparty session types in a Java-like syntax. A simple example of a protocol in Scribble which represents a ring topology between four workers is given below:

A Scribble protocol starts from the keyword , followed by the protocol name, . The role declarations are then passed as parameters of the protocol, which are through to . The protocol describes a series of communications in which passes a message of type to by forwarding through and in that order and receives a message from . It is easy to notice that explicitly describing all interactions among distinct roles is verbose and inflexible: For example, when extending the protocol with an additional role , we must rewrite the whole protocol. On the other hand, we observe that these worker roles have identical communication patterns which can be logically grouped together: receives a message from and the last sends a message to . In order to capture these replicable patterns, we introduce an extension of Scribble with dependent types called Parameterised Scribble (Pabble). In Pabble, multiple participants can be grouped in the same role and indexed. This greatly enhances the expressive power and modularity of the protocols. Here ‘parameterised’ refers to the number of participants in a role that can be changed by parameters.

The following shows our ring example in the syntax of Pabble.

declares workers from to an arbitrary integer . The roles can be identified individually by their indices, for example, refers to the first and refers to the last. In the body of the protocol, the sender, , declares multiple s, bound by the bound variable , and iterates from to . The receivers, , are calculated on their indices for each instances of the bound variable . The second line is a message sent back from to .

The above code shows the local protocol of , projected with respect to the parameterised role. The projection for a parameterised role, such as , will give a parameterised local protocol. It represents multiple end points in the same logical grouping.

Challenges The main technical challenge for the design and implementation of parameterised session types is to develop a method to automatically project a parameterised global protocol to a parameterised local protocol ensuring termination and correctness of the algorithm.

Unfortunately, as in the indexed dependent type theory in the $\lambda$-calculus [2, 33], the underlying parameterised session type theory [9] has shown that the projection and type checking with general indices are undecidable. Hence, there is a tension between termination and expressiveness to enable concise specifications for complex parameterised protocols.

Our main approach to overcome these challenges is to make the theory more practical by extending Scribble with index notation originating from a widely used text book for modeling concurrent Java [16]. For example, notations and in the Ring protocol are from [16]. Interestingly, this compact notation is not only expressive enough to represent representative topologies ranging from parallel algorithms to distributed web services, but also offers a solution to cope with the undecidability of parameterised multiparty session types.

1.1 Overview

Figure 1 shows the relationships between the three layers: global protocols, local protocols and implementations. (1) A programmer first designs a global protocol using Pabble. (2) Then, our Pabble tool automatically projects the global protocol into its local protocols. (3) The programmer then either implement the parallel application using the local protocol as specification, or type-check existing parallel applications against the local protocol. If the communication interaction patterns in the implementations follow the local protocols generated from the global protocol, this method automatically ensures deadlock-free and type-safe communication in the implementation. In this work, we focus on the design and implementation of the language for describing parallel message-passing-based interaction as global and local protocols in (1) and (2) and outline how a Pabble local type checker for MPI (3) can be implemented.

Fig. 1

Pabble programming workflow

This article presents a full version of the work published in [19] which had a particular focus on modeling and expressing communication topologies in parallel applications. Apart from including the detailed proofs for the well-formedness conditions and a number of additional examples, we include use cases from web services and large-scale distributed cyber-infrastructures to show the flexibility of the Pabble language for compact parametric protocols outside the field of high-performance parallel applications. We also expand the related work for a more thorough survey and discussion on formal verification with MPI-based parallel applications.

The contributions of this article are:

• The first design and implementation of parameterised session types in a global protocol language (Pabble) (Sect.  2.2). The protocols can represent complex topologies with arbitrary number of participants, enhancing expressiveness and modularity for practical message-passing parallel programs.

• The projection algorithm for Pabble to check the well-formedness of parameterised global protocols (Sects.  2.3 and 2.4) and to generate parameterised local protocols from well-formed parameterised global protocols (Sect. 2.5). A correctness and termination proof of the projection algorithm is also presented (Sect. 2.7).

• A number of Pabble use cases in parallel programming and web services in Sect. 3.

Additional use cases of Pabble such as common interaction patterns for high-performance computing described in Dwarfs [1] can be found on the project web page [21]. We also outline a methodology for type checking source code written with MPI against Pabble protocols in Sect. 4.

2 Pabble: parameterised Scribble

Scribble  [24] is a developer-friendly notation for specifying application-level protocols based on the theory of multiparty session types [3, 13]. This section introduces an evolution of Scribble with parameterised multiparty session types (Pabble) defines its end point projection and proves its correctness.

2.1 The Pabble protocol language

The core elements of a Pabble protocol are interaction statements, choices and iterations. These are features common also to the Scribble language, which Pabble is extended from. Hence, Scribble protocols are compatible with Pabble, but the most expressive features such as role parameterisation can only be found in Pabble.

Interaction statements describe the messages passed between distributed participants of the protocol. For example, in the protocol below, ; is an interaction statement which sends a message from participant (called a role) to another participant . The participant are declared in the protocol as arguments of the protocol, . The subscripting notation of the roles are for indexing the participants and will be explained in details the next section. The message has a label, , which may be omitted from the interaction statement. The message also contains a type name as parameters to the label, e.g.,  , called a payload type. The payload type represents the data type of the message being sent.

Choice statements are written as

where each of the branches is an alternative interaction sub-pattern which the participants can collectively select. The deciding role sends a label (e.g., ) to other roles involved with the choice to notify them of the branch taken.

Iterations (loops) in the interaction patterns are written as recursion blocks , with statement to jump back to beginning of recursion.

2.2 Syntax of Pabble

2.2.1 Global protocols

Figure 2 lists the core syntax of Pabble, which consists of two protocol declarations, global and local. A global protocol is declared with the protocol name ($str$ denotes a string) with role and group parameters followed by the body $G$. Role $R$ is a name with argument expressions. The argument expressions are ranges or arithmetic expressions $h$, and the number of arguments corresponds to the dimension of the array of roles: for example, denotes a 2D array with size 4 and 2 in the two dimensions, respectively, forming a 4-by-2 array of roles.

Fig. 2

Pabble syntax

Declared roles can be grouped by specifying a named group using the keyword , followed by the group name and the set of roles. For example,

creates a group which consists of two s. A special built-in group, , is defined as all processes in a session. We can encode collective operators such as many-to-many and many-to-one communication with , which will be explained later.

Apart from specifying ranges explicitly, ranges can also be specified using expressions. Expression $e$ consists of the usual operators for numbers, logarithm, left and right logical shifts $(\mathtt{<}\mathtt{<}, \mathtt{>}\mathtt{>})$, numbers, variables ($i, j, k$), and constants $(\mathtt{M}, \mathtt{N})$. Constants are either bound outside the protocol declaration or are left free (unbound) to represent an arbitrary number. As in [16], when the constants are bound, they are declared by numbers outside the protocol, e.g.,  or lower and upper bounds, e.g., . We also allow leaving the declaration free (unbound), e.g., , as a shorthand to represent an arbitrary constant with lower and upper bounds and , respectively, i.e., , where is a special value representing the maximum possible value or practically unbounded. Binding range expression $b$ takes the form of $i:e_1..e_n$ which means $i$ is ranged from $e_1$ to $e_n$. Binding variables always bind to a range expression and not individual values. We shall explain the use of binding range expressions later in more details.

In a global protocol $G$, $l(T)$ $R_1$ $R_2$ is called an interaction statement, which represents passing a message with label $l$ and type $T$ from one role $R_1$ to another role $R_2$. $R_1$ is a sender role and $R_2$ is a receiver role. $\,R$ $\{G_1\}$ $\{G_{n} \}$ means the role $R$ will select one of the global types $G_1,\ldots ,G_n$. $\{G\}$ is recursion with the label $l$ which declares a label for statement. denotes a for-loop whose iteration is specified by $b$. For example, represents the iteration from $1$ to $n$ of $G$ where $G$ is parameterised by $i$.

Finally, $op_c(T)$ means all processes perform a distributed reduction in value with type $T$ with the operator $op_c$ (like in MPI). It takes a mandatory predefined operator $op_c$ where $op_c$ must be a commutative and associative arithmetic operation. Pabble currently supports sum and product.

We allow using simple expressions (e.g., ) to parameterise ranges. In addition, indices can also be calculated by expressions on bound variables (e.g., ) to refer to relative positions of roles.

These restrictions on indices such as bound variables and relative indices calculations ensure termination of the projection algorithm and type checking. The binding conditions are discussed in the next subsection.

2.2.2 Local protocols

Local protocol $L$ consists of the same syntax of the global type except the input from $R$ (receive) and the output to $R$ (send). The main declaration

means the protocol is located at role $R_e$. We call $R_e$ the endpoint role. In Pabble, multiple local protocol instances can reside in the same parameterised local protocol. This is because each local protocol is a local specification for a participant of the interaction. Where there are multiple participants with a similar interaction structure that fulfill the same role in the protocol, such as the s from our example from the introduction, the participants are grouped together as a single parameterised role. The local protocol for a collection of participants can be specified in a single parameterised local protocol, using conditional statements on the role indices to capture edge cases. For example, in a general case of a pipeline interaction, all participants receives from a neighbor and send to another neighbor, except the first participant which initiates the pipeline and is only a sender and the last participant which ends the pipeline and does not send. In these cases, we use conditional statements to guard the input or output statements. To express conditional statements in local protocols, $R$ may be prepended to input or output statement. $R$ input/output statement will be ignored if the local role does not match $R$. More complicated matches can be performed with a parameterised role, where the role parameter range of the condition is matched against the parameter of the local role. For example, will match but not . It is also possible to bind a variable to the range in the condition, e.g., , and can be used in the same statement.

2.3 Well-formedness conditions: index binding

As Pabble protocols include expressions in parameters, a valid Pabble protocol is subject to a few well-formedness conditions. Below, we show the conditions which ensure indices used in roles are correctly bounded. We use $\mathsf {fv}/\mathsf {bv}$ to denote the set of free/bound variables defined as $\mathsf {fv}(i)=\{i\}$, $\mathsf {fv}(\mathtt{N})=\mathsf {fv}( num )=\emptyset$ and $\mathsf {fv}(i:e_1 \dots e_n)=\cup \mathsf {fv}(e_j)$ and

Others are inductively defined.

• 1. In a global protocol role declaration, which are parameters of , indices outside of declared range are invalid, for example, a role is invalid if the role is declared .

• 2. Let

  

  • (a) Suppose an interaction statement $l$($T$) $R_1$ $R_2$ appears in $G$. Let $R_1 = Role _1[h_1]\dots [h_n]$ and $R_2 = Role _2[e'_1]\dots [e'_m]$ (we assume $n=0$ (resp. $m=0$) if $R_1$ (resp. $R_2$) is either a single participant or group).

    • (1) $n=m$ (i.e., the dimensions of the parameters are the same)

    • (2) $\mathsf {fv}(h_j)\subseteq \cup \mathsf {bv}(b_i)$ (i.e., the free variables in the sender roles are bound by the for-loops).

    • (3) $\mathsf {fv}(e'_j)\subseteq (\cup \mathsf {bv}(b_i))\cup \mathsf {bv}(h_j)$ (i.e., the free variables in the receiver roles are bound by either the for-loops or sender roles);

  • (b) Suppose a choice statement $R$

    

    appears in $G$. Then, $R$ is a single participant, i.e., either $Role$ or $Role [e]$ with $\mathsf {fv}(e)\subseteq (\cup \mathsf {bv}(b_i))$.

Condition 2(a)(1) ensures the number of sender parameters matches the number of receiver parameters. For example, the following is invalid:

Condition 2(a)(2) ensures variables used by a sender are declared by the enclosing for-loops.

Condition 2(a)(3) makes sure the receiver parameter at the j-th position is bound by the for-loops or the sender parameter at the j-th position (and not binders at other positions). For example, the following is valid:

But with the index swapped, it becomes invalid:

Condition 2(b) is similar for the case of statements where $R$ should be a single participant to satisfy the unique sender condition in [6, 8].

2.4 Well-formedness conditions: constants

In Pabble protocols, constants can be defined by

• (1) A single numeric value (); or

• (2) Lower and upper bound constraints not involving ().

Lower and upper bound constraints are designed for runtime constants, e.g., the number of processes spawned in a scalable protocol, which is unknown at design time and will be defined and immutable once the execution begins. To ensure, Pabble protocols are communication-safe in all possible values of constants, we must ensure that all parameterised role indices stay within their declared range. Such conditions prevent sending or receiving from an invalid (non-existent) role which will lead to communication mismatch at runtime.

In case (1), the check is trivial. In case (2), we require a general algorithm to check the validity between multiple constraints appeared in the regions. First, we formulate the constraints of the values of the constants as a series of linear inequalities. We then combine the linear inequalities and determine the feasible region using standard linear programming. The feasible region represents the pool of possible values in any combination of the constraints. The following explains how to determine whether the protocol will be valid for all combinations of constants:

The basic constraints from the constants are:

We then calculate the range of as . Since the objective is to ensure that the role parameters in the protocol body (i.e.,  and ) stay within the bounds of , we define a constraint set to be:

which are lower and upper bound inequalities of the two ranges. From them, we obtain this inequality as a result:

By comparing this against the basic constraints on the constants, we can check that not all outcomes belong to the regions, and thus, this is not a communication-safe protocol (an example of a unsafe case is and ). On the other hand, if we alter Line 4 to ;, the constraints are unconditionally true, and so, we can guarantee all combinations of constants and will not cause communication errors.

Arbitrary constants In addition to constant values and lower and upper bound constants, we also consider the use cases when the value of a constant can be any arbitrary value in the set of natural numbers. This is an extension of case (2) with the keyword, where we write to represent a range without upper bound.

In order to check that role indices are valid with unbounded ranges, we enforce two simple restrictions. First, only one constant can be defined with in one global protocol. Secondly, when the index is unbounded, its range calculation only uses addition or subtraction on integers (e.g., ).

A protocol with an invalid use of arbitrary constants is shown below:

If is instantiated to , then the role is declared to be . In the first interaction statement, is invalid, as is not in the range of . In the second statement, is also invalid, as it evaluates to and is out of range .

On the other hand, the following protocol is valid since the indices always stay between 0 and .

We have shown in [21], most of representative topologies with the arbitrary number of participants can be represented under these conditions.

2.5 Endpoint projection

In the next step, a Pabble protocol should be projected to a local protocol, which is a simplified Pabble protocol as viewed from the perspective of a given end point. The projection algorithm is explained below. To begin with the header of the global protocol

where the protocol name name and parameters param are preserved and the endpoint role $R_e$ is declared.

Table 1 shows the projection of the body of global protocol $G$ onto ${\mathbf {R}}$ at endpoint role $R_e$. The projection rules will be applied from top to bottom in the table, if a global protocol matches multiple rules, then there will be more than one line of projected protocol for a single global protocol. In Rules 1–4, we show the rule for the single argument as the same rule is applied to $n$-arguments. Each rule is applied if ${\mathbf {R}}$ meets the condition in the second column under the constraints given by the constant declarations. Rules 1 and 2 show the projection of the interaction statement when ${\mathbf {R}}$ appears in the receiver and the sender position, respectively. Since ${\mathbf {R}}$ is a single participant, it should satisfy ${\mathbf {R}}=R_e$ (i.e., the role is the endpoint role). The projection simply removes the reference to role ${\mathbf {R}}$ from the original interaction statement.

Table 1 Projection of $G$ onto ${\mathbf {R}}$ at the end point role $R_e$

Rules 3 and 4 show the projection of an interaction statement if role ${\mathbf {R}}$ is a parameterised single participant where ${\mathbf {R}}$ is an element of the endpoint role $R_e$. For example, if $R_e=$ , ${\mathbf {R}}$ can be either , or . In addition to removing the reference of role ${\mathbf {R}}$ in the receive and send statements, we also prepend the conditions which the role applies. The order of which the projection rules are applied ensure that an interaction statement will be localized to receive then send. In general, both receive–send or send–receive in the projected local protocol are correct, as long as the projection algorithm is consistent and the well-formedness conditions of the global protocol are satisfied. The global protocol will ensure, by session typing, that a send will have a matching receive at the same stage of the protocol.

Rule 5 is for all-to-all communication. Any role ${\mathbf {R}}$ will send a message with type $U$ to all other participants and will receive some value with type $U$ from all other participants. Since all participants start by first sending a message to all, no participant will block waiting to receive in the first phase, so no deadlock occurs.

Rules 6 and 7 are the projection rules for the case that we project onto a group. We need to check that a group is a subset of the endpoint role $R_e$ with respect to the group declarations in the global protocol. Then, the rules can be understood as Rules 3 and 4.

Rules 8 and 9 show the projection of interaction statements with parameterised roles using relative indexing (we show only one argument: the algorithm can be extended easily to multiple arguments using the same methods). Rule 8 uses two auxiliary transformations of expressions, and . Table 2 lists their examples. takes two arguments, a range with binding variable ($b$) and an expression using the binding variable ($e$). The expression is applied to both ends of the range to transform the relative expression into a well-defined range. calculates the inverse of a given expression, for example, the inverse of is and the inverse of is . In cases when an inverse expression cannot be derived, such as , the expression will be calculated by expanding to all values in the range and instantiating every value bound by its binding variable (e.g., ).

Table 2 Examples of and

A concrete example is given as follows, to project the statement

the statement will be expanded to

before applying the projection rules. In order to perform the range expansion above, the beginning and the end of the range must be known at projection time. For this reason, the projection algorithm returns failure if a statement uses parameterised roles with such expressions and the range of the expressions is defined with arbitrary constants (see Sect. 2.4). Otherwise, the expressions might expand infinitely and not terminate. This is the only situation which projection may fail, given a well-formed global protocol. The condition ${\mathbf {R}}[b] \subseteq R_e$ of Rule 9 means the range of $b$ is within the range of the endpoint role $R_e$. For example, $\subseteq$ .

If a projection role matches the choice role (${\mathbf {R}}$ in ${\mathbf {R}}$) (Rule 10), then it means a selection statement, whose action is selecting a branching by sending a label. The child or blocks ($L_1$...$L_N$) are recursively projected, whereas if a projection role does not match the choice role (Rule 11), then the choice statement represents a branch statement, which is the dual of the selection. For recursion (Rule 12), continue (Rule 13) and foreach (Rule 14) statements are just kept in the projected endpoint protocol.

2.6 Collective operations

In addition to point-to-point message-passing, collective operations can also be concisely represented by Pabble. End point message-passing statements are interpreted differently depending on the declarations (i.e., parameters) in the global type. Figures 3, 4, 5 and 6 list the four basic messaging patterns and the interpretations of their projections: point-to-point, scatter (distribution), gather (collection) and all-to-all (symmetric distribution and collection). As shown in the figures, the combination of projected local statements and the type (i.e., single participant or group role) of the local role being projected are unique and can identify the communication pattern in the global protocol.

Fig. 3

Point-to-point communication and Pabble representation

Fig. 4

Scatter pattern and Pabble representation

Fig. 5

Gather pattern and Pabble representation

Fig. 6

All-to-all pattern and Pabble representation

2.7 Correctness and termination of the projection

The parameterised session theory which Pabble is based on [9] has shown that, in the general case, projection and type checking are undecidable. Our first challenge for Pabble ’s design is to ensure the termination of well-formed checking and projection, without sacrificing the expressiveness. The theorems and proofs can be found in this section.

Theorem 1

(Termination) Given global protocol $G$, the well-formed checking terminates; and given a well-formed global type $G$ and an end point role $R_e$, projection $G$ on $R_e$ always terminates.

Proof

By the definition of the well-formedness conditions in Sect. 2.3, if a free variable appears in the range position, it is bound by either for-loops or the sender role in the interaction statement. In the case of the for-loop, we can apply the same reduction rules of the for-loop of the global types from Sect. 2 and apply the equality rules in [9, Fig. 15]. Hence, one can check, given $R_e$ and ${\mathbf {R}}$, all of the conditions (in the second column) in Table 1 are decidable. For the projection, the only non-trivial projection rule is Rule 8. The termination of this rule is ensured by the termination of

and

. If

is not defined, we first check $e$ has the finite range and use Rule 3 and 4 by expanding the interaction statements to all values in the range (as explained in Sect. 2.5). Hence, the projection algorithm always terminates. $\square$

Note that the above theorem implies the termination of type checking (see Theorem 4.4 in [9]).

One of the benefits of using Pabble is that it provides the expressiveness required to be able represent collective interactions in MPI. The correctness of projections of these protocols is ensured by the projection rule of the groups in [7]. The special case of $U$ follows the asynchronous subtyping rules in [18]. The correctness property which relates to ranges of Pabble follows:

Theorem 2

(Range) The indices of roles appearing in a local protocol body do not exceed the lower and upper bounds stated in the global protocol ProtocolName(para) in

.

Proof

If the range relies on case (2), the correctness is ensured by linear programming. Other cases are straightforward since each condition in Table 1 checks whether roles conform to the bounds in the global protocol. $\square$

3 Pabble examples

In Sect. 2.5, we describe how to obtain a local Pabble protocol by projection from a Pabble protocol. The local protocol can then be used as a blueprint to implement parallel programs. In this section, we run through two examples of local protocol projection, using a protocol in Sect. 3.1 and a protocol in Sect. 3.2, showing projection of protocols involving point-to-point and multicast collective applications, respectively.

Then, we present Pabble use cases in web services in Sect. 3.3 and remote procedure call (RPC) composition in Sect. 3.4, showing the capabilities of Pabble as a general-purpose parameterised protocol description language.

Finally, we show an implementation of a parallel linear equation solver Sect. 3.5 in MPI following a wraparound mesh protocol designed in Pabble, demonstrating how Pabble can be used in practical programming. Additional Pabble examples from the Dwarfs [1], evaluation metric is available from our web page [21].

3.1 Projection example: Ring protocol

We now run through the projection of the protocol in Sect. 1 as an example. Local protocols are generated from the global protocols. From the perspective of a projection tool, to write a protocol for an endpoint, we start with followed by the name of the protocol and the endpoint role it is projected for. Since the only role of the protocol is which is a parameterised role, we use the full definition of the parameterised role, . Then, we list the roles used in the protocol inside a pair of parentheses, similar to function arguments in a function definition in C. Note that if the projection role is in the list, we exclude it because the local protocol itself is in the perspective of that role; however, since parameterised roles can be used on multiple endpoint roles, we allow parameterised roles to appear in the list of roles in the protocol. The first line of the projected protocol is thus given as follows:

We then copy the recursion statement to the local protocol, which will be present in all projected protocols.

Next, we take the first interaction statement from protocol and project it with respect to , applying the rules listed in Table 1. As the first statement involves a parameterised destination role, we apply Rule 7 to extract the receive portion of the interaction statement. The function is applied to and the relative expression to obtain for the role condition. The of relative expression is , which will form the index of the sender role.

Since also matches the source parameterised role, Rule 8 is applied to get the send portion of the interaction statement.

Then, we move on to the second statement of the global protocol, ;. Similar to the previous statement, we apply Rule 3 and Rule 4 to obtain the respective receive and send statements in the local protocol.

Finally, we apply Rule 13 to trivially copy the statement to the local protocol.

The resulting local protocol is the following, as shown in

Sect. 1.

3.2 Projection example: MapReduce protocol

The following example shows another parameterised protocol, which represents the map–reduce pattern of work distribution and reduction. This example uses a common parallel programming idiom, collective operations. In contrast to the previous example, there are more than one declared role in the protocol, and one of the role is an ordinary nonparameterised role.

In this protocol, the statements involve two roles, one of which is an ordinary role (in the sense that it is non-parameterised), and the other is a parameterised role . The parameterised role represents a group of related roles, but do not expand to multiple explicit message-passing statements. We further declare a group role which include all the roles as members. The statement in Line 2 is a scatter operation by which the distributes a message of type to each of the named endpoints in group, to . The statement in Line 3 is a gather operation, the reverse of the scatter, which the role collects messages of type from the members of the group. Figure 7 depicts the interactions in the protocol.

Fig. 7

Topology of the MapReduce protocol

Listing 2 shows the local protocol of MapReduce at the role. Since is a nonparametric participant, Rule 2 and 1 are applied to get Line 2 and 3, respectively. This results in a protocol body without conditional interactions.

The local protocol of for is similarly derived by applying the projection rules. Since is a group role and a subset of , Rule 6 and 7 are applied to get Line 2 and 3.

3.3 Use case: web services

Pabble is inspired by applications in the domain of parallel programming, but the parametric nature of Pabble as a protocol language allows us to express interactions with more flexibility while keeping the protocols succinct.

Quote Request protocol specification (C-UC-002) is the most complex use case in [32] published by W3C Web Services Choreography Working Group [31].

Fig. 8

Web Services Quote Request Interaction

It describes the interaction between a buyer who interacts with multiple suppliers who in turn interact with multiple manufacturers in order to get a quote for some goods or services.

The basic steps of the interaction is as follows:

• 1. A buyer requests a quote from a set of suppliers

• 2. All suppliers forward the quote request of the items to their manufacturers

• 3. The suppliers interact with their manufacturers to build the quotes for the buyer, which is then sent back to the buyer

• 4.

  • (a) Either the buyer agrees with the quotes and place the orders

  • (b) Or the buyer modify the quote and send back to the suppliers

• 5. In the case, the supplier received an updated quote request (4b)

  • (a) Either the supplier respond to updated quote request by agreeing to it and sending a confirmation message back to buyer

  • (b) Or the supplier respond to the update quote request by modifying it and sending back to buyer and the buyer goes back to step 4

  • (c) Or the supplier respond to the update quote request by rejecting it

  • (d) Or the supplier renegotiate with the manufacturers, in which case we return to step 3

Figure 8 shows the interactions between different components in the Quote Request use case. We set the generic number S for suppliers and M for manufacturers. The interactions are described as a Pabble global protocol in Listing 4. In the protocol, we omitted the implicit requestIdType from the payload type in all of the messages which keeps track of states of each role in the stateless web transport.

The Buyer initiates the quote request on Line 2, when it broadcasts a Quote() message to all Suppliers. Then, on Line 4–7 each of the Supps forward the quote requests to their respective Manufacturers and get a reply from each of them by a series of gather and scatter interactions. Next, the Suppliers reply to the Buyer on Line 9, and the Buyer then decides between accepting the offer straight away (Line 14, outcome 4a), or sending a modified quote request (Line 17, outcome 4b). If a Supp received a modified quote, it decides between accepting the modified quote (Line 21, outcome 5a), rejecting the modified quote straight away (Line 29, outcome 5c) or modifying the quote and renegotiating with Buyer (Line 24, outcome 5b). It is also possible that the Supplier renegotiates with its Manufacturers again, so it notifies the Buyer and returns back to the initial negotiation phase (Line 32, outcome 5d). The projected endpoint protocol for Buyer is Listing 5.

3.4 Use case: RPC composition

We present a use case from the Ocean Observatories Initiative Project [28]. The use case describes a high-level Remote Procedural Call (RPC) request/response protocol between layers of proxy services. An application sends a request to a high-level service, and the service is expected to reply to the application with a result. If the service does not provide the requested service, then this high-level service will issue a request to a lower level service which can process the request. This request-response protocol is chained between services in each level until a low-level service is reached.

Figure 9 describes the chaining of RPC-style request/response protocol. A request is routed to the most relevant service provider through multiple proxy services hidden from higher level services. The request routes through a multi-hop path from the requester to the resources. The reply is routed in reverse through the same participant proxy services back to the requester.

Fig. 9

RPC request/response chaining

We represent this series of interactions using a Pabble protocol outlined below. The participants, , represents a proxy service in each of the levels. is the requester and is the actual service provider. A message is sent from a to the in the level directly below, until it reached which will process the request and reply to the higher level service with a . Using a loop with decrementing indices, the is cascaded to the originating service, . The Pabble protocol is shown in Listing 6.

Fig. 10

$N^2$-node wraparound mesh topology

As the request and response phase are symmetric and involve the same participants, we are able to compact the multi-layer protocol to only using two loops, each with one parameterised interaction statement. can be an arbitrary constant to allow maximum flexibility in the protocol. This simple and concise representation of complex RPC chaining protocol is possible because of the index notation in Pabble.

3.5 Implementation example: Linear equation solver

Listing 9 shows an example implementation outline for a linear equation solver using a wraparound mesh, which follows the Pabble protocol in Listing 7. The topology is illustrated in Fig. 10. The example is given in message-passing interface (MPI), the standardized API for developing message-passing applications in parallel computing.

The protocol above describes a wraparound mesh that performs a ring propagation between (for worker) in the same row (Line 3–4), and the result of each row is distributed to all s in the first column (i.e., ) using a group-to-group distribution on Line 7. The global protocol is then automatically projected into its local protocol shown in Listing 8 below. Developers can then implement the application using its local protocol as a guide.

Note the similarity of the local protocol and the structure of the MPI implementation in Listing 9. In particular, the conditional send and receive in MPI can directly correspond to the role conditions in the local protocol which was derived from the global protocol by projection.

4 Type checking

Given the local protocol and the implementation, we propose a session type checker to verify the conformance of the implementation against the projected local protocol. Conformance of end point programs against the projected protocol will yield communication-safe parallel programs.

Pabble local protocols have similar structure to that of MPI programs. Both Pabble protocols and MPI programs are designed such that a single source code representing multiple end points, a result of the single-program multiple-data (SPMD) parallel programming model. The core communication primitives of MPI can correspond to Pabble statements, as demonstrated in Listing 9. In addition, collective operations such as broadcast () or all-reduce () can be supported by the collective operation correspondence in Sect. 2.5.

Challenges for a complete MPI type checker In [20], Ng et al. introduced a session type checker for a nonparameterised protocol language and a simple session programming API. We face a number of challenges when building a complete type checker using the same methodology for Pabble, which is a dependent protocol language and MPI, which is a standard parameterised implementation API. The Pabble language with its well-formedness checks reduces the undecidability issues in the role representation by using integer instead of general indices. The type checking process will compare the protocol against a simplified, canonical local protocol extracted from the implementation, which still posts a challenge in the process of protocol extraction. In particular, inferring source and destination processes from parametric source code is non-trivial. MPI uses process IDs (or ranks) to identify processes, and it is valid to perform numeric operations on the ranks to efficiently calculate target processes. This allows ways of exploiting the C language features while remaining a valid program. For example, instead of using a conventional conditional statement, an MPI function call of this form may be used:

where the process ID, , is being used as a boolean, thus a straightforward analysis of usages would not be sufficient. In order to correctly calculate target processes of the interactions, it will be necessary to simulate rank calculations by techniques such as symbolic execution or combinations of runtime techniques.

5 Conclusion

This article introduced a new global protocol description language, Pabble, and applied it to ensure deadlock-free and type-safe communications in parallel programs. Local protocols projected from a parameterised global protocol and we outlined a methodology to specify and type-check MPI parallel programs for safe parallel programs. Our global protocols and local protocols bring the expressiveness of Scribble to new levels, overcoming the issue of the underlying parameterised multiparty session type theory [9] by a careful design choice for indices based on [16]. Combining with the multirole theory from [7], Pabble can represent and type-check representative MPI collective operators. We are not aware of any prior framework which is uniformly applicable to a safety guarantee for message-passing parallel programs which run over complex topologies, through static, low-cost type checking as compared to model checking.

Through our examples presented in this article, we have showed that the Pabble language is not limited to high-performance parallel applications. The examples, including web services and RPC, cover a broad category of interaction-centric scalable distributed applications. Our simple, formally based language provides an approach for designing services and applications with safe interaction patterns.

6 Related work

6.1 Formal verification for parallel applications

Formal verification for message-passing parallel programming has been actively studied in the area of MPI parallel applications. A recent survey [10] summarizes a wide range of model checking-based verification methods for MPI. Among them, ISP [29] is a dynamic verifier which applies model-checking techniques to identify potential communication deadlocks in MPI. Their tool uses a fixed test harness and in order to reduce the state space of possible thread interleavings of an execution, the tool exploits an independence between thread actions. Later in [30], the authors improved its scheduling policy to gain efficiency of the verification. While their approach aims to cover common deadlock patterns in MPI programs, it is still limited to a finite number of tests. Our approach does not rely on external testing, and all session typable programs are guaranteed communication-safe and deadlock-free by a low-cost static code generation and type checking.

TASS [26] is another tool that combines symbolic execution [25] and model-checking techniques to verify safety properties of MPI programs. The tool takes a C/MPI application and an input $n \ge 1$ which restricts the input space, then constructs an abstract model with $n$ processes and checks its functional equivalence and deadlocks by executing the model of the application. TASS does not verify properties for an unbounded number of communication participants nor treat parameterisation, whereas we can work with message-passing programs where the number of participants is unknown at compile time, if they are written in well-formed, projectable Pabble.

6.2 Formally based MPI languages

Pilot [5] is a parallel programming library built on standard MPI to provide a simplified parallel programming abstraction based upon CSP. The communication is synchronous and channels are untyped to facilitate reuse for different types. The implementation includes an analyser to detect communication deadlock at runtime. Our proposed typechecker is static and is able to detect and prevent deadlocks before execution.

Interprocedural control flow graph (ICFG) [27] and parallel control flow graph (pCFG) [4] are techniques to analyze MPI parallel programs for potential message leak errors. Their approach extends a traditional data-flow analysis by connecting control flow graphs of concurrent processes to their communication edges in order to derive the communication pattern and topology of a parallel program. They take a bottom-up engineering-based approach, in contrast to our formally based, top-down global protocol approach, which can give a high-level understanding of the overall communication by design, in addition to the communication safety assurance by multiparty session types.

6.3 Parameterised multiparty session types

Previous work from Ng et al. [20] introduces a C programming framework based on multiparty session types (MPSTs), but it does not treat parameterisation. Hence, the user needs to explicitly describe all interactions in the protocol, and the type checker does not work if the number of participants is unknown at compile time. Pabble ’s theoretical basis is developed in [9] where parameterised MPSTs are formalized using the dependent type theory of Gödel’s System ${\mathcal {T}}$. The main aim in [9] is to investigate the decidability and expressiveness of parameterisations of participants. Type checking in [9] is undecidable when the indices are not limited to decidable arithmetic subsets or the number of the loop in the parameterised types is infinite. The design of Pabble is inspired by the LTSA tool from a concurrency modeling text book used for the undergraduate teaching in the authors’ university over two decades [16]. The notations for parameterisations from the LTSA tool offers not only practical restrictions to cope with the undecidability of parameterised MPSTs [9], but also concise representations for parameterised parallel languages. Our work is the first to apply parameterised MPSTs in a practical environment and one foremost aim of our framework with Pabble and parameterised notation is to be developer-friendly [24] without compromising the strong formal basis of session types.

6.4 Dependent typing systems

Liquid Type [23] is a dependent typing system to automatically infer memory safety properties from program source code without using verbose annotations. The work [22] introduced an analyser for the C language in the low-level imperative environment based on Liquid Types and refinement types. The recent work on Liquid Types [15] applied the tool with SMT solvers to assist parallelisation of code regions by determining statically whether parallel threads will run on disjoint shared memory without races. Our work applies dependent session types to guarantee different kinds of safety, communication safety and deadlock freedom, in explicit message-passing based distributed and parallel programming rather than shared memory concurrency. It is an interesting future topic to integrate with model-checking tools to handle projectability with more complex indices in addition to functional correctness of session programs.

6.5 Session-based approaches to parallel programming

A recent work [11, 17] aims to use session types for deductive verification of MPI programs. A new type language is designed specifically for MPI and they used VCC, a concurrent C verifier tool to verify correctness of MPI against the type language. While the Pabble language was designed with influences from parallel programming APIs and parallel programming use cases, the language was designed to be an independent high-level abstraction over distributed interactions. As a result, our language makes no assumption about the execution environment (e.g., collective loops in MPI), and allows Pabble to represent general protocols from distributed systems or web services with distinct roles as shown in the examples.

7 Future work

Future works include extending Pabble and the underlying theory with support for modeling process creation and destroy, such as dynamic multirole approach described in [7].

A number of enhancements are planned for Pabble including support for annotations which can complement the protocol description to specify assertions. The type checking process can use the extra constraints or conditions provided to combine with model checkers to also assure functional correctness of the overall application. Annotations will also enable integration with runtime monitoring described in [14] for a combined static and dynamic approach to communication correct application using Pabble.

An approach to generate distributed parallel application is in the works, using a combination of Pabble protocol, which describes the interaction aspects of the application, and computation code, which describes the sequential computation behavior of the application.