1 Introduction

Many real-time systems need to support threads involving priorities and locking of resources. Locking of resources ensures mutual exclusion when accessing shared data or devices that cannot be preempted. Priorities allow scheduling of threads that need to finish their work within deadlines. Unfortunately, both features can interact in subtle ways leading to a problem, called Priority Inversion. Suppose three threads having priorities H(igh), M(edium) and L(ow). We would expect that the thread H blocks any other thread with lower priority and the thread itself cannot be blocked indefinitely by threads with lower priority. Alas, in a naive implementation of resource locking and priorities, this property can be violated. For this let L be in the possession of a lock for a resource that H also needs. H must therefore wait for L to exit the critical section and release this lock. The problem is that L might in turn be blocked by any thread with priority M, and so H sits there potentially waiting indefinitely (consider the case where threads with priority M continuously need to be processed). Since H is blocked by threads with lower priorities, the problem is called Priority Inversion. It was first described in [12] in the context of the Mesa programming language designed for concurrent programming.

If the problem of Priority Inversion is ignored, real-time systems can become unpredictable and resulting bugs can be hard to diagnose. The classic example where this happened is the software that controlled the Mars Pathfinder mission in 1997 [21]. On Earth, the software ran mostly without any problem, but once the spacecraft landed on Mars, it shut down at irregular, but frequent, intervals. This led to loss of project time as normal operation of the craft could only resume the next day (the mission and data already collected were fortunately not lost, because of a clever system design). The reason for the shutdowns was that the scheduling software fell victim to Priority Inversion: a low priority thread locking a resource prevented a high priority thread from running in time, leading to a system reset. Once the problem was found, it was rectified by enabling the Priority Inheritance Protocol (PIP) [24]Footnote 1 in the scheduling software.

The idea behind PIP is to let the thread L temporarily inherit the high priority from H until L leaves the critical section unlocking the resource. This solves the problem of H having to wait indefinitely, because L cannot be blocked by threads having priority M. While a few other solutions exist for the Priority Inversion problem, PIP is one that is widely deployed and implemented. This includes VxWorks (a proprietary real-time OS used in the Mars Pathfinder mission, in Boeing’s 787 Dreamliner, Honda’s ASIMO robot, etc.) and ThreadX (another proprietary real-time OS used in nearly all HP inkjet printers [28]), but also the POSIX 1003.1c Standard realised for example in libraries for FreeBSD, Solaris and Linux.

Two advantages of PIP are that it is deterministic and that increasing the priority of a thread can be performed dynamically by the scheduler. This is in contrast to Priority Ceiling [24], another solution to the Priority Inversion problem, which requires static analysis of the program in order to prevent Priority Inversion, and also in contrast to the approach taken in the Windows NT scheduler, which avoids this problem by randomly boosting the priority of ready low-priority threads (see for instance [2]). However, there has also been strong criticism against PIP. For instance, PIP cannot prevent deadlocks when lock dependencies are circular, and also blocking times can be substantial (more than just the duration of a critical section). Though, most criticism against PIP centres around unreliable implementations and PIP being too complicated and too inefficient. For example, Yodaiken writes in [30]:

“Priority inheritance is neither efficient nor reliable. Implementations are either incomplete (and unreliable) or surprisingly complex and intrusive.”

He suggests avoiding PIP altogether by designing the system so that no priority inversion may happen in the first place. However, such ideal designs may not always be achievable in practice.

In our opinion, there is clearly a need for investigating correct algorithms for PIP. A few specifications for PIP exist (in informal English) and also a few high-level descriptions of implementations (e.g. in the textbooks [15, Section 12.3.1] and [26, Section 5.6.5]), but they help little with actual implementations. That this is a problem in practice is proved by an email by Baker, who wrote on 13 July 2009 on the Linux Kernel mailing list:

“I observed in the kernel code (to my disgust), the Linux PIP implementation is a nightmare: extremely heavy weight, involving maintenance of a full wait-for graph, and requiring updates for a range of events, including priority changes and interruptions of wait operations.”

The criticism by Yodaiken, Baker and others suggests another look at PIP from a more abstract level (but still concrete enough to inform an implementation), and makes PIP a good candidate for a formal verification. An additional reason is that the original specification of PIP [24], despite being informally “proved” correct, is actually flawed.

Yodaiken [30] and also Moylan et al. [16] point to a subtlety that had been overlooked in the informal proof by Sha et al. They specify PIP in [24, Section III] so that after the thread (whose priority has been raised) completes its critical section and releases the lock, it “returns to its original priority level”. This leads them to believe that an implementation of PIP is “rather straightforward” [24]. Unfortunately, as Yodaiken and Moylan et al. point out, this behaviour is too simplistic. Moylan et al. write that there are “some hidden traps” [16]. Consider the case where the low priority thread L locks two resources, and two high-priority threads H and \(H'\) each wait for one of them. If L releases one resource so that H, say, can proceed, then we still have Priority Inversion with \(H'\) (which waits for the other resource). The correct behaviour for L is to switch to the highest remaining priority of the threads that it blocks. A similar error is made in the textbook [20, Section 2.3.1] which specifies for a process that inherited a higher priority and exits a critical section that “it resumes the priority it had at the point of entry into the critical section”. This error can also be found in the textbook [14, Section 16.4.1] where the authors write about this process: “its priority is immediately lowered to the level originally assigned”; and also in the more recent textbook [13, Page 119] where the authors state: “when [the task] exits the critical section that caused the block, it reverts to the priority it had when it entered that section”. The textbook [15, Page 286] contains a similar flawed specification and even goes on to develop pseudo-code based on this flawed specification. Accordingly, the operating system primitives for inheritance and restoration of priorities in [15] depend on maintaining a data structure called inheritance log. This log is maintained for every thread and broadly specified as containing “[h]istorical information on how the thread inherited its current priority” [15, Page 527]. Unfortunately, the important information about actually computing the priority to be restored solely from this log is not explained in [15] but left as an “exercise” to the reader. As we shall see, a correct version of PIP does not need to maintain this (potentially expensive) log data structure at all. Surprisingly also the widely read and frequently updated textbook [25] gives the wrong specification. On Page 254 the authors write: “Upon releasing the lock, the [low-priority] thread will revert to its original priority.” The same error is also repeated later in this popular textbook.

While [13,14,15, 20, 24, 25] are the only formal publications we have found that specify the incorrect behaviour, it seems also many informal descriptions of the PIP protocol overlook the possibility that another high-priority process might wait for a low-priority process to finish. A notable exception is the textbook [3], which gives the correct behaviour of resetting the priority of a thread to the highest remaining priority of the threads it blocks. This textbook also gives an informal proof for the correctness of PIP in the style of Sha et al. Unfortunately, this informal proof is too vague to be useful for formalising the correctness of PIP and the specification leaves out nearly all details in order to implement PIP efficiently.

Contributions There have been earlier formal investigations into PIP [8, 10, 29], but they employ model checking techniques. This paper presents a formalised and mechanically checked proof for the correctness of PIP. For this we needed to design a new correctness criterion for PIP. In contrast to model checking, our formalisation provides insight into why PIP is correct and allows us to prove stronger properties that, as we will show, can help with an efficient implementation of PIP. We illustrate this with an implementation of PIP in the educational operating system PINTOS [19]. For example, we found by “playing” with the formalisation that the choice of the next thread to take over a lock when a resource is released is irrelevant for PIP being correct—a fact that has not been mentioned in the literature and not been used in the reference implementation of PIP in PINTOS. This fact, however, is important for an efficient implementation of PIP, because we can give the lock to the thread with the highest priority so that it terminates more quickly. We are also able to generalise the scheduler of Sha et al. [24] to the practically relevant case where critical sections can overlap; see Fig. 1a for an example of this restriction. In the existing literature there is no proof and also no proof method that covers this generalised case.

Fig. 1
figure 1

Assume a process is over time locking and unlocking, say, three resources. The locking requests are labelled , , and respectively, and the corresponding unlocking operations are labelled , , and . Then graph a shows properly nested critical sections as required by Sha et al. [24] in their proof—the sections must either be contained within each other (the section – is contained in – ) or be independent ( – is independent from the other two). Graph b shows the general case where the locking and unlocking of different critical sections can overlap

Full size image

2 Formal Model of the Priority Inheritance Protocol

The Priority Inheritance Protocol, short PIP, is a scheduling algorithm for a single-processor system.Footnote 2 Following good experience in earlier work [27], our model of PIP is based on Paulson’s inductive approach for protocol verification [18]. In this approach a state of a system is given by a list of events that happened so far (with new events prepended to the list). Events of PIP fall into five categories defined as the Isabelle datatype:

figure m

whereby threads, priorities and (critical) resources are represented as natural numbers. In what follows we shall use as a name for critical resources. The event models the situation that a thread obtains a new priority given by the programmer or user (for example via the nice utility under UNIX). For states we define the following type-synonym:

figure p

As in Paulson’s work, we need to define functions that allow us to make some observations about states. One function, called , calculates the set of “live” threads that we have seen so far in a state:

figure r

In this definition stands for list-cons and for the empty list. We use to match any pattern, like in functional programming. Another function calculates the priority for a thread , which is defined as

figure w

In this definition we set as the default priority for threads that have not (yet) been created. The last function we need calculates the “time”, or index, at which time a thread had its priority last set.

figure y

In this definition stands for the length of the list of events . Again the default value in this function is for threads that have not been created yet. An actor of an event is defined as

figure ac

This allows us to filter out the actions a set of threads perform in a list of events , namely

figure af

where we use Isabelle’s notation for list-comprehensions. This notation is very similar to the notation used in Haskell for list-comprehensions. A precedence of a thread in a state is the pair of natural numbers defined as

figure ai

We also use the abbreviation

figure aj

for the precedences of a set of threads in state . The point of precedences is to schedule threads not according to priorities (because what should we do in case two threads have the same priority), but according to precedences. Precedences allow us to always discriminate between two threads with equal priority by taking into account the time when the priority was last set. We order precedences so that threads with the same priority get a higher precedence if their priority has been set earlier, since for such threads it is more urgent to finish their work. In an implementation this choice would translate to a quite straightforward FIFO-scheduling of threads with the same priority.

Moylan et al. [16] considered the alternative of “time-slicing” threads with equal priority, but found that it does not lead to advantages in practice. On the contrary, according to their work having a policy like our FIFO-scheduling of threads with equal priority reduces the number of tasks involved in the inheritance process and thus minimises the number of potentially expensive thread-switches.

Next, we introduce the concept of waiting queues. They are lists of threads associated with every resource. The first thread in this list (i.e. the head, or short ) is chosen to be the one that is in possession of the “lock” of the corresponding resource. We model waiting queues as functions, below abbreviated as . They take a resource as argument and return a list of threads. This allows us to define when a thread holds, respectively waits for, a resource given a waiting queue function .

figure aq

In this definition we assume that converts a list into a set. Note that in the first definition the condition about does not follow from , since the head of an empty list is undefined in Isabelle/HOL. At the beginning, that is in the state where no thread is created yet, the waiting queue function will be the function that returns the empty list for every resource.

figure au

Using and , we can introduce Resource Allocation Graphs (RAG), which represent the dependencies between threads and resources. We choose to represent s as relations using pairs of the form

figure ay

where the first stands for a waiting edge and the second for a holding edge ( and are constructors of a datatype for vertices). Given a waiting queue function, a is defined as the union of the sets of waiting and holding edges, namely

figure bc
Fig. 2
figure 2

An instance of a resource allocation graph (RAG)

Full size image

If there is no cycle, then every can be pictured as a forest of trees, as for example in Fig. 2.

Because of the s, we will need to formalise some results about graphs. It seems for our purposes the most convenient representation of graphs are binary relations given by sets of pairs shown in (2). The pairs stand for the edges in graphs. This relation-based representation has the advantage that the notions and are already defined in terms of relations amongst threads and resources. Also, we can easily re-use the standard notions for transitive closure operations and , as well as relation composition for our graphs. While there are a few formalisations for graphs already implemented in Isabelle, we choose to introduce our own library of graphs for PIP. The justification for this is that we wanted to have a more general theory of graphs which is capable of representing potentially infinite graphs (in the sense of infinitely branching and infinite size): the property that our s are actually forests of finitely branching trees having only a finite depth should be something we can prove for our model of PIP—it should not be an assumption we build already into our model. A forest is defined in our representation as the relation that is single valued and acyclic:

figure bl

The children, subtree and ancestors of a node in a graph can be easily defined relationally as

figure bm

Note that forests can have trees with infinite depth and containing nodes with infinitely many children. A finite forest is a forest whose underlying relation is well-foundedFootnote 3 and every node has finitely many children (is only finitely branching).

The locking mechanism ensures that for each thread node, there can be many incoming holding edges in the , but at most one out going waiting edge. The reason is that when a thread asks for a resource that is locked already, then the thread is blocked and cannot ask for another resource. Clearly, also every resource can only have at most one outgoing holding edge—indicating that the resource is locked. So if the is well-founded and finite, we can always start at a thread waiting for a resource and “chase” outgoing arrows leading to a single root of a tree, which must be a ready thread.

The use of relations for representing s allows us to conveniently define the Thread Dependants Graph (TDG):

figure bq

This definition is the relation that one thread is waiting for another to release a resource, but the corresponding resource is “hidden”. In Fig. 2 this means the connects and to , which both wait for resource to be released; and to , which cannot make any progress unless makes progress. Similarly for the other threads. If there is a circle of dependencies in a (and thus ), then clearly we have a deadlock. Therefore when a thread requests a resource, we must ensure that the resulting and are not circular. In practice, the programmer has to ensure this. Our model will enforce that critical resources can only be requested provided no circularity can arise (but critical sections can overlap, see Fig 1).

Next we introduce the notion of the current precedence of a thread in a state . It is defined as

figure cf

While the precedence of any thread is determined statically (for example when the thread is created), the point of the current precedence is to dynamically boost this precedence, if needed according to PIP. Therefore the current precedence of is given as the maximum of the precedences of all threads in its subtree (which includes by definition itself). Since the notion of current precedence is defined as the transitive closure of the dependent threads in the , we deal correctly with the problem in the informal algorithm by Sha et al. [24] where a priority of a thread is lowered prematurely (see Introduction). We again introduce an abbreviation for current precedences of a set of threads, written .

figure cl

The next function, called , defines the behaviour of the scheduler. It will be defined by recursion on the state (a list of events); this function returns a schedule state, which we represent as a record consisting of two functions:

figure cn

The first function is a waiting queue function (that is, it takes a resource and returns the corresponding list of threads that lock or wait for it); the second is a function that takes a thread and returns its current precedence [see the in (5)]. We assume the usual getter and setter methods for such records.

In the initial state, the scheduler starts with all resources unlocked [the corresponding function is defined in (1)] and the current precedence of every thread is initialised with ; that means . Therefore we have for the initial schedule state

figure cs

The cases for , and are also straightforward: we calculate the waiting queue function of the (previous) state ; this waiting queue function is unchanged in the next schedule state—because none of these events lock or release any resource; for calculating the next , we use and defined above. This gives the following three clauses for :

figure dc

More interesting are the cases where a resource, say , is requested or released. In these cases we need to calculate a new waiting queue function. For the event , we have to update the function so that the new thread list for is the old thread list plus the thread appended to the end of that list (remember the head of this list is assigned to be in the possession of this resource). This gives the clause

figure dh

The clause for event is similar, except that we need to update the waiting queue function so that the thread that possessed the lock is deleted from the corresponding thread list. For this list transformation, we use the auxiliary function . A simple version of would just delete this thread and return the remaining threads, namely

figure dl

In practice, however, often the thread with the highest precedence in the list will get the lock next. We have implemented this choice, but later found out that the choice of which thread is chosen next is actually irrelevant for the correctness of PIP. Therefore we prove the stronger result where is defined as

figure dn

where stands for Hilbert’s epsilon and implements an arbitrary choice for the next waiting list. It just has to be a list of distinct threads and contains the same elements as (essentially can be any reordering of the list ). This gives for the clause:

figure dt

Having the scheduler function at our disposal, we can “lift”, or overload, the notions , , , , and to operate on states only.

figure ea

With these abbreviations in place we can derive the following two facts about s and , which are more convenient to use in subsequent proofs.

figure ed

Next we can introduce the notion of a thread being in a state (i.e. threads that do not wait for any resource, which are the roots of the trees in the , see Fig. 2). The thread is then the thread with the highest current precedence of all ready threads.

figure eh

In the second definition stands for the image of a set under a function. Note that in the initial state, that is where the list of events is empty, the set is empty and therefore there is neither a thread ready nor running. If there is one or more threads ready, then there can only be one thread running, namely the one whose current precedence is equal to the maximum of all ready threads. We use sets to capture both possibilities. We can now also conveniently define the set of resources that are locked by a thread in a given state and also when a thread is detached in a state (meaning the thread neither holds nor waits for a resource—in the RAG this would correspond to an isolated node without any incoming and outgoing edges, see Fig. 2):

figure ek

Finally we can define what a valid state is in our model of PIP. For example we cannot expect to be able to exit a thread, if it was not created yet. These validity constraints on states are characterised by the inductive predicate and . We first give five inference rules for relating a state and an event that can happen next.

figure eo

The first rule states that a thread can only be created, if it is not alive yet. Similarly, the second rule states that a thread can only be terminated if it was running and does not lock any resources anymore (this simplifies slightly our model; in practice we would expect the operating system releases all locks held by a thread that is about to exit). The event can happen if the corresponding thread is running.

figure eq

This is because the event is for a thread to change its own priority—therefore it must be running.

If a thread wants to lock a resource, then the thread needs to be running and also we have to make sure that the resource lock does not lead to a cycle in the RAG (the purpose of the second premise in the rule below). In practice, ensuring the latter is the responsibility of the programmer. In our formal model we brush aside these problematic cases in order to be able to make some meaningful statements about PIP.Footnote 4

figure es

Similarly, if a thread wants to release a lock on a resource, then it must be running and in the possession of that lock. This is formally given by the last inference rule of .

figure eu

Note, however, that apart from the circularity condition, we do not make any assumption on how different resources can be locked and released relative to each other. In our model it is possible that critical sections overlap. This is in contrast to Sha et al. [24] who require that critical sections are properly nested (recall Fig. 1).

A valid state of PIP can then be conveniently be defined as follows:

figure ev

This completes our formal model of PIP. In the next section we present a series of desirable properties derived from this model of PIP. This can be regarded as a validation of the correctness of our model.

3 The Correctness Proof

Sha et al. state their first correctness criterion for PIP in terms of the number of low-priority threads [24, Theorem 3]: if there are low-priority threads, then a blocked job with high priority can only be blocked a maximum of times. Their second correctness criterion is given in terms of the number of critical resources [24, Theorem 6]: if there are critical resources, then a blocked job with high priority can only be blocked a maximum of times. Both results on their own, strictly speaking, do not prevent indefinite, or unbounded, Priority Inversion, because if a low-priority thread does not give up its critical resource (the one the high-priority thread is waiting for), then the high-priority thread can never run. The argument of Sha et al. is that if threads release locked resources in a finite amount of time, then indefinite Priority Inversion cannot occur—the high-priority thread is guaranteed to run eventually. The assumption is that programmers must ensure that threads are programmed in this way. However, even taking this assumption into account, the correctness properties of Sha et al. are not true for their version of PIP—despite being “proved”. As Yodaiken [30] and Moylan et al. [16] pointed out: If a low-priority thread possesses locks to two resources for which two high-priority threads are waiting for, then lowering the priority prematurely after giving up only one lock, can cause indefinite Priority Inversion for one of the high-priority threads, invalidating their two bounds (recall the counter example described in the Introduction).

Even when fixed, their proof idea does not seem to go through for us, because of the way we have set up our formal model of PIP. One reason is that we allow critical sections, which start with a -event and finish with a corresponding -event, to arbitrarily overlap (something Sha et al. explicitly exclude). Therefore we have designed a different correctness criterion for PIP. The idea behind our criterion is as follows: for all states , we know the corresponding thread with the highest precedence; we show that in every future state (denoted by ) in which is still alive, either is running or it is blocked by a thread that was alive in the state and was waiting for or in the possession of a lock in . Since in , as in every state, the set of alive threads is finite, can only be blocked by a finite number of threads.

However, the theorem we are going to prove hinges upon a number of natural assumptions about the states and , the thread and the events happening in . We list them next:

Assumptions on the statesand We need to require that and are valid states:

figure ft

Assumptions on the thread The thread must be alive in and has the highest precedence of all alive threads in . Furthermore the priority of is (we need this in the next assumptions).

figure ga

Assumptions on the events in To make sure has the highest precedence we have to assume that events in can only create (respectively set) threads with equal or lower priority than of . For the same reason, we also need to assume that the priority of does not get reset and all other reset priorities are either less or equal. Moreover, we assume that does not get “exited” in . This can be ensured by assuming the following three implications.

figure gj

The locale mechanism of Isabelle helps us to manage conveniently such assumptions [9]. Under these assumptions we shall prove the following correctness property:

Theorem 1

Given the assumptions about states and , the thread and the events in , then either

  • or

  • there exists a thread with and such that , and .

This theorem ensures that the thread , which has the highest precedence in the state , is either running in state , or can only be blocked in the state by a thread that already existed in and is waiting for a resource or had a lock on at least one resource—that means the thread was not detached in . As we shall see shortly, that means there are only finitely many threads that can block in this way.

The next lemma is part of the proof for Theorem 1: Given our assumptions (on ), the first property we show that a running thread must either wait for or hold a resource in state .

Lemma 1

If and then .

Proof

Let us assume otherwise, that is is detached in state , then, according to the definition of detached, does not hold or wait for any resource. Hence the -value of in is not boosted, that is , and is therefore lower than the precedence (as well as the -value) of . This means will not run as long as is a live thread. In turn this means cannot take any action in state to change its current status; therefore is still detached in state . Consequently is also not boosted in state and would not run. This contradicts our assumption. \(\square \)

Proof (of Theorem 1)

If , then there is nothing to show. So let us assume otherwise. Since the is well-founded, we know there exists an ancestor of that is the root of the corresponding subtree and therefore is ready (it does not request any resources). Let us call this thread . Since in PIP the -value of any thread equals the maximum precedence of all threads in its -subtree, and is in the subtree of , the -value of cannot be lower than the precedence of . But, it can also not be higher, because the precedence of is the maximum among all threads. Therefore we know that the -value of is the same as the precedence of . The result is that must be running. This is because -value of is the highest of all ready threads. This follows from the fact that the -value of any ready thread is the maximum of the precedences of all threads in its subtrees (with having the highest of all threads and being in the subtree of ). We also have that since we assumed is not running. By Lemma 1 we have that . If is not detached in , that is either holding or waiting for a resource, it must be that .

This concludes the Proof of Theorem 1. \(\square \)

4 A Finite Bound on Priority Inversion

Like in the work by Sha et al. our result in Theorem 1 does not yet guarantee the absence of indefinite Priority Inversion. For this we further need the property that every thread gives up its resources after a finite amount of time. We found that this property is not so straightforward to formalise in our model. There are mainly two reasons for this: First, we do not specify what “running the code” of a thread means, for example by giving an operational semantics for machine instructions. Therefore we cannot characterise what are “good” programs that contain for every locking request for a resource also a corresponding unlocking request. Second, we need to distinguish between a thread that “just” locks a resource for a finite amount of time (even if it is very long) and one that locks it forever (there might be an unbounded loop in between the locking and unlocking requests).

Because of these problems, we decided in our earlier paper [31] to leave out this property and let the programmer take on the responsibility to program threads in such a benign manner (in addition to causing no circularity in the RAG). This leave-it-to-the-programmer approach was also taken by Sha et al. in their paper. However, in this paper we can make an improvement by establishing a finite bound on the duration of Priority Inversion measured by the number of events. The events can be seen as a rough(!) abstraction of the “runtime behaviour” of threads and also as an abstract notion of “time”—when a new event happens, some time must have passed.

What we will establish in this section is that there can only be a finite number of states after state in which the thread is blocked (recall for this that a state is a list of events). For this finiteness bound to exist, Sha et al. informally make two assumptions: first, there is a finite pool of threads (active or hibernating) and second, each of these threads will give up its resources after a finite amount of time. However, we do not have this concept of active or hibernating threads in our model. In fact we can dispense with the first assumption altogether and allow that in our model we can create new threads or exit existing threads arbitrarily. Consequently, the absence of indefinite priority inversion we are trying to establish in our model is not true, unless we stipulate an upper bound on the number of threads that have been created during the time leading to any future state after . Otherwise our PIP scheduler could be “swamped” with -requests of lower priority threads. So our first assumption states:

Assumption on the number of threads created after the state : Given the state , in every “future” valid state , we require that the number of created threads is less than a bound , that is

whereby is a list of events.

Note that it is not enough to just state that there are only finite number of threads created up until a single state after . Instead, we need to put this bound on the events for all valid states after . This ensures that no matter which “future” state is reached, the number of -events is finite. This bound is assumed with respect to all future states of , not just a single one.

For our second assumption about giving up resources after a finite amount of “time”, let us introduce the following definition about threads that can potentially block :

figure jt

This set contains all threads that are not detached in state . According to our definition of , this means a thread in either holds or waits for some resource in state . Our Theorem 1 implies that only these threads can all potentially block after state . We need to make the following assumption about the threads in the -set:

Assumptions on the threads : For each such there exists a finite bound such that for all future valid states , we have that if , then

By this assumption we enforce that any thread potentially blocking must become detached (that is it owns no resource anymore) after a finite number of events in . Again we have to state this bound to hold in all valid states after . The bound reflects how each thread is programmed: Though we cannot express what instructions a thread is executing, the events in our model correspond to the system calls made by a thread. Our bounds the number of these “calls”.

The main reason for these two assumptions is that we can prove the following: The number of states after in which the thread is not running (that is where Priority Inversion occurs) can be bounded by the number of actions the threads in perform (i.e. events) and how many threads are newly created. To state our bound formally, we need to make a definition of what we mean by intermediate states between a state and a future state after ; they will be the list of states starting from up to the state . For example, suppose \(\textit{es} = [\textit{e}_n, \textit{e}_{n-1}, \ldots , \textit{e}_2, \textit{e}_1]\), then the intermediate states from upto are

figure ku

This list of intermediate states can be defined by the following recursive function

figure kv

Our theorem can then be stated as follows:

Theorem 2

Given our assumptions about bounds, we have that

This theorem uses Isabelle’s list-comprehension notation, which lists all intermediate states between and , and then filters this list according to states in which is not running. By calculating the number of elements in the filtered list using the function , we have the number of intermediate states in which is not running and which by the theorem is bounded by the term on the right-hand side.

Proof

There are two characterisations for the number of events in : First, in each state in , clearly either is running or not running. Together with , that implies

(7)

The actions in can be partitioned into the actions of and the actions of threads other than . The latter can further be divided into actions of existing threads and the actions to create new ones. Moreover, the actions of existing threads other than are by Thm 1 the actions of blockers. This gives rise to

(8)

Furthermore we know that an action of in the intermediate states can only be taken when is running. Therefore

holds. Substituting this into (7) gives

into which we can substitute (8) yielding

By our first assumption we know that the number of -events are bounded by the bound . By our second assumption we can prove that the actions of all blockers is bounded by the sum of bounds of the individual blocking threads, that is

With this in place we can conclude our theorem. \(\square \)

This theorem is the main conclusion we obtain for the Priority Inheritance Protocol. It is based on the fact that the set of is fixed at state when becomes the thread with the highest priority. Then no additional blocker of can appear after the state . And in this way we can bound the number of states where the thread with the highest priority is prevented from running. Our bound does not depend on the restriction of well-nested critical sections in the Priority Inheritance Protocol as imposed by Sha et al.

5 Properties for an Implementation

While our formalised proof gives us confidence about the correctness of our model of PIP, we found that the formalisation can even help us with efficiently implementing it. For example Baker complained that calculating the current precedence in PIP is quite “heavy weight” in Linux (see the Introduction). In our model of PIP the current precedence of a thread in a state depends on the precedences of all threads in its subtree—a “global” transitive notion, which is indeed heavy weight [see the equation for shown in (6)]. We can however improve upon this. For this recall the notion of of a thread defined in (3). There a child is a thread that is only one “hop” away from the thread in the (and waiting for to release a resource). Using children, we can prove the following lemma for more efficiently calculating of a thread .

Lemma 2

figure mg

That means the current precedence of a thread can be computed by considering the static precedence of and the current precedences of the children of . Their s, in general, need to be computed by recursively descending into deeper “levels” of the . However, the current precedence of a thread , say, only needs to be recomputed when its static precedence is re-set or when one of its children changes its current precedence or when the children set changes (for example in a -event). If only the static precedence or the children-set changes, then we can avoid the recursion and compute the of locally. In such cases the recursion does not need to descend into the corresponding subtree. Once the current precedence is computed in this more efficient manner, the selection of the thread with highest precedence from a set of ready threads is a standard scheduling operation and implemented in most operating systems.

Below we outline how our formalisation guides the efficient calculation of in response to each kind of events.

We assume that the current state and the next state , whereby , are both valid (meaning the event is allowed to occur in ). In this situation we can show that

figure na

This means in an implementation we do not have to recalculate the and also none of the current precedences of the other threads. The current precedence of the created thread is just its precedence, namely the pair .

We again assume that the current state and the next state , whereby this time , are both valid. We can show that

figure ni

This means again we do not have to recalculate the and also not the current precedences for the other threads. Since is not alive anymore in state , there is no need to calculate its current precedence.

We assume that and with are both valid. We can show that

figure nq

The first property is again telling us we do not need to change the . The second shows that the -values of all threads other than are unchanged. The reason for this is more subtle: Since must be running, then it does not wait for any resource to be released and it cannot be in any subtree of any other thread. So all current precedences of other threads are unchanged.

We assume that and with being are both valid. We have to consider two subcases: one where there is a thread to “take over” the released resource , and one where there is not. Let us consider them in turn. Suppose in state , the thread takes over resource from thread . We can prove

figure of

which shows how the needs to be changed. The next lemmas suggest how the current precedences need to be recalculated. For threads that are not and nothing needs to be changed, since we can show

figure oj

For and we need to use Lemma 2 to recalculate their current precedence since their children have changed. However, neither and is element of the respective children, which is shown by the following two facts:

figure oo

This means the recalculation of the of and can be done independently and also done locally by only looking at the children: according to (9) and (10) none of the of the children changes, just the children-sets changes by a -event.

In the other case where there is no thread that takes over , we can prove that the updated merely deletes the relevant edge and that no current precedence needs to be recalculated for any thread .

figure ox

We assume that and with are both valid. We again have to analyse two subcases, namely the one where is not locked, and one where it is. We treat the former case first by showing that

figure pd

This means we need to add a holding edge to the . However, note that while the changes the corresponding does not change. Together with the fact that the precedences of all threads are unchanged, no value is changed. Therefore, no recalculation of the value of any thread is needed.

In the second case we know that resource is locked. We can show that

figure pl

That means we have to add a waiting edge to the . Furthermore the current precedence for all threads that are not ancestors of (in the new or ) are unchanged. For the ancestors of we need to follow the edges in the and recompute the . Whereas in all other event we might have to make modifications to the , no recalculation of depends on the . This is the only case where the recalculation needs to take the connections in the into account. To do this we can start from and follow the -edges to recompute the of every thread encountered on the way using Lemma 2. This means the recomputation can be done locally (level-by-level) in a bottom-up fashion. Since the , and thus , are loop free, this procedure will always stop. The following lemma shows, however, that this procedure can actually stop often earlier without having to consider all ancestors.

figure qc

This property states that if an intermediate -value does not change (in this case the -value of ), then the procedure can also stop, because none of ancestor-threads will have their current precedence changed.

As can be seen, a pleasing byproduct of our formalisation is that the properties in this section closely inform an implementation of PIP, namely whether the needs to be reconfigured or current precedences need to be recalculated for an event. This information is provided by the lemmas we proved. We confirmed that our observations translate into practice by implementing our version of PIP on top of PINTOS, a small operating system written in C and used for teaching at Stanford University [19].Footnote 5 While there is no formal connection between our formalisation and the C-code shown below, the results of the formalisation clearly shine through in the design of the code.

To implement PIP in PINTOS, we only need to modify the kernel functions corresponding to the events in our formal model. The events translate to the following function interface in PINTOS:

Event

PINTOS function

Our implicit assumption that every event is an atomic operation is ensured by the architecture of PINTOS (which allows disabling of interrupts when some operations are performed). The case where an unlocked resource is given next to the waiting thread with the highest precedence is realised in our implementation by priority queues. We implemented them as Braun trees [17], which provide efficient -operations for accessing and updating. In the code we shall describe below, we use the function , for inserting a new element into a priority queue, and the function , for updating the position of an element that is already in a queue. Both functions take an extra argument that specifies the comparison function used for organising the priority queue.

Apart from having to implement relatively complex datastructures in C using pointers, our experience with the implementation has been very positive: our specification and formalisation of PIP translates smoothly to an efficient implementation in PINTOS. Let us illustrate this with the C-code for the function , shown in Fig. 3. This function implements the operation of requesting and, if free, locking of a resource by the current running thread. The convention in the PINTOS code is to use the terminology locks rather than resources. A lock is represented as a pointer to the structure lock (Line 1). Lines 2–4 are taken from the original code of in PINTOS. They contain diagnostic code: first, there is a check that the lock is a “valid” lock by testing whether it is not NULL; second, a check that the code is not called as part of an interrupt—acquiring a lock should only be initiated by a request from a (user) thread, not from an interrupt; third, it is ensured that the current thread does not ask twice for a lock. These assertions are supposed to be satisfied because of the assumptions in PINTOS about how this code is called. If not, then the assertions indicate a bug in PINTOS and the result will be a “kernel panic”.

Fig. 3
figure 3

Our version of the lock_acquire function for the small operating system PINTOS. It implements the operation corresponding to a -event

Full size image

Lines 6 and 7 of lock_acquire make the operation of acquiring a lock atomic by disabling all interrupts, but saving them for resumption at the end of the function (Line 31). In Line 8, the interesting code with respect to scheduling starts: we first check whether the lock is already taken (its value is then 0 indicating “already taken”, or 1 for being “free”). In case the lock is taken, we enter the if-branch inserting the current thread into the waiting queue of this lock (Line 9). The waiting queue is referenced in the usual C-way as . Next, we record that the current thread is waiting for the lock (Line 10). Thus we established two pointers: one in the waiting queue of the lock pointing to the current thread, and the other from the current thread pointing to the lock. According to our specification in Sect. 2 and the properties we were able to prove for , we need to “chase” all the ancestor threads in the and update their current precedence; however we only have to do this as long as there is change in the current precedence.

The “chase” is implemented in the while-loop in Lines 13–24. To initialise the loop, we assign in Lines 11 and 12 the variable to the owner of the lock. Inside the loop, we first update the precedence of the lock held by (Line 14). Next, we check whether there is a change in the current precedence of . If not, then we leave the loop, since nothing else needs to be updated (Lines 15 and 16). If there is a change, then we have to continue our “chase”. We check what lock the thread is waiting for (Lines 17 and 18). If there is none, then the thread is ready (the “chase” is finished with finding a root in the ). In this case we update the ready-queue accordingly (Lines 19 and 20). If there is a lock is waiting for, we update the waiting queue for this lock and we continue the loop with the holder of that lock (Lines 22 and 23). After all current precedences have been updated, we finally need to block the current thread, because the lock it asked for was taken (Line 25).

If the lock the current thread asked for is not taken, we proceed with the else-branch (Lines 26–30). We first decrease the value of the lock to 0, meaning it is taken now (Line 27). Second, we update the reference of the holder of the lock (Line 28), and finally update the queue of locks the current thread already possesses (Line 29). The very last step is to enable interrupts again thus leaving the protected section.

Similar operations need to be implemented for the function, which we however do not show. The reader should note though that we did not verify our C-code. This is in contrast, for example, to the work on seL4, which actually verified in Isabelle/HOL that their C-code satisfies its specification, though this specification does not contain anything about PIP [11]. Our verification of PIP however provided us with (formally proven) insights on how to design the C-code. It gave us confidence that leaving the “chase” early, whenever there is no change in the calculated current precedence, does not break the correctness of the algorithm.

6 Conclusion

The Priority Inheritance Protocol (PIP) is a classic textbook algorithm used in many real-time operating systems in order to avoid the problem of Priority Inversion. Although classic and widely used, PIP does have its faults: for example it does not prevent deadlocks in cases where threads have circular lock dependencies.

We had two goals in mind with our formalisation of PIP: One is to make the notions in the correctness proof by Sha et al. [24] precise so that they can be processed by a theorem prover. The reason is that a mechanically checked proof avoids the flaws that crept into their informal reasoning. We achieved this goal: The correctness of PIP now only hinges on the assumptions behind our formal model. The reasoning, which is sometimes quite intricate and tedious, has been checked by Isabelle/HOL. We can also confirm that Paulson’s inductive method for protocol verification [18] is quite suitable for our formal model and proof. The traditional application area of this method is security protocols.

The second goal of our formalisation is to provide a specification for actually implementing PIP. Textbooks, for example Vahalia [26, Section 5.6.5], explain how to use various implementations of PIP and abstractly discuss their properties, but surprisingly lack most details important for a programmer who wants to implement PIP (similarly Sha et al. [24]). That this is an issue in practice is illustrated by the email from Baker we cited in the Introduction. We achieved also this goal: The formalisation allowed us to efficiently implement our version of PIP on top of PINTOS, a simple instructional operating system for the x86 architecture implemented by Pfaff [19]. It also gives the first author enough data to enable his undergraduate students to implement PIP (as part of their OS course). A byproduct of our formalisation effort is that nearly all design choices for the implementation of PIP scheduler are backed up with a proved lemma. We were also able to establish the property that the choice of the next thread which takes over a lock is irrelevant for the correctness of PIP. Moreover, we eliminated a crucial restriction present in the proof of Sha et al.: they require that critical sections nest properly, whereas our scheduler allows critical sections to overlap. What we are not able to do is to mechanically “synthesise” an actual implementation from our formalisation. To do so for C-code seems quite hard and is beyond current technology available for Isabelle. Also our proof-method based on events is not “computational” in the sense of having a concrete algorithm behind it: our formalisation is really more about the specification of PIP and ensuring that it has the desired properties (the informal specification by Sha et al. did not).

PIP is a scheduling algorithm for single-processor systems. We are now living in a multi-processor world. Priority Inversion certainly occurs also there, see for example work by Brandenburg, and Davis and Burns [1, 6]. However, there is very little “foundational” work about PIP-algorithms on multi-processor systems. We are not aware of any correctness proofs, not even informal ones. There is an implementation of a PIP-algorithm for multi-processors as part of the “real-time” effort in Linux, including an informal description of the implemented scheduling algorithm given by Rostedt in [23]. We estimate that the formal verification of this algorithm, involving more fine-grained events, is a magnitude harder than the one we presented here, but still within reach of current theorem proving technology. We leave this for future work.

To us, it seems sound reasoning about scheduling algorithms is fiendishly difficult if done informally by “pencil-and-paper”. We infer this from the flawed proof in the paper by Sha et al. [24] and also from [22] where Regehr points out an error in a paper about Preemption Threshold Scheduling by Wang and Saksena [28]. The use of a theorem prover was invaluable to us in order to be confident about the correctness of our reasoning (for example no corner case can be overlooked). The most closely related work to ours is the formal verification in PVS of the Priority Ceiling Protocol done by Dutertre [7]—another solution to the Priority Inversion problem, which however needs static analysis of programs in order to avoid it. There have been earlier formal investigations into PIP [8, 10, 29], but they employ model checking techniques. The results obtained by them apply, however, only to systems with a fixed size, such as a fixed number of events and threads. In contrast, our result applies to systems of arbitrary size. Moreover, our result is a good witness for one of the major reasons to be interested in machine checked reasoning: gaining deeper understanding of the subject matter.

Our formalisation consists of around 600 lemmas and overall 9200 lines of readable and commented Isabelle/Isar code with a few apply-scripts interspersed. The formal model of PIP is 310 lines long; our graph theory implementation using relations is 1615 lines; the basic properties of PIP take around 5000 lines of code; and the formal correctness proof 1250 lines.

The properties relevant for an implementation require 1000 lines. The code of our formalisation can be downloaded from the Mercurial repository at http://talisker.inf.kcl.ac.uk/cgi-bin/repos.cgi/pip.