figure a
figure b

1 Introduction

Requirements elicitation is a proactive process which, by capturing the intended behavior of a system at an early stage, safeguards against decisions that could lead to increased development costs and even catastrophic failures. Formal requirements analysis can solidify engineers’ confidence in the expressed specification. Our work is concerned with ensuring requirements consistency for system components, as a pre-requisite for subsequent system-level analysis. In particular, we focus on the notion of realizability: a realizable set of requirements guarantees that an implementation exists, such that it always behaves in a manner consistent with the specification, no matter what input it receives from its environment. The notion of realizability, first described as implementability by Pnueli and Rosner [47], has since then shaped an entire research area over the specification and synthesis of reactive systems.

This paper presents the realizability analysis framework that we have developed as part of NASA’s open source tool FRET  [3] for writing, understanding, and formalizing requirements. FRET is designed with a strong focus on usability, and is used by several NASA projects to explore the benefits of writing requirements that can be processed by formal analysis tools [10, 17, 42, 45]. Additionally, FRET has been used by external (to NASA) industrial and research teams, e.g., for the formalization of aircraft engine controller requirements [19]. FRET ’s realizability framework has two main goals: 1) to implement efficient algorithms for checking realizability, and 2) to provide user support in understanding and correcting sources of unrealizability. With these features, FRET provides an end-to-end solution to capturing, analyzing, and diagnosing requirements.

FRET ’s realizability framework provides a user-friendly interface for analyzing the requirements of system components. We have designed a graphic environment, in which the user can observe a (potentially) decomposed version of the specification that is sound with respect to realizability, as well as further dive into the task of diagnosing unrealizable requirements. Compositional analysis is based on our theoretical framework for checking realizability of a global specification through smaller, more tractable parts [25, 43]. The diagnosis process is based on the theoretical work by Könighofer et al. [33, 34] on generating minimal conflicts of unrealizability. We adjusted the diagnosis algorithm to support the discovery of all minimal conflicts in a contract, accompanied by a counterexample of unrealizability. The computed artifacts can be visualized as an interactive diagram that depicts the dependencies between requirements and conflicts. Counterexample traces that originate from these conflicts can also be simulated to enhance the understanding of unrealizability sources. For the analysis, we have integrated in FRET state-of-the-art tools with respect to realizability checking modulo infinite theories.

In particular, the contributions of this work are:

  • The design and implementation of a realizability checking framework in FRET that tightly integrates the JKind  [23] and Kind 2 [35] analysis tools;

  • a diagnosis feature for unrealizability that returns all minimal conflicts and their counterexamples in an easy-to-use, graphical user interface;

  • the extension of the simulator component in FRET, to be used for the simulation of conflicting requirements in unrealizable specifications; and

  • improvements of the algorithms in our in-house fork of the JKind model checker, following recent work from the Kind 2 and GenSys [48] tools.

2 Related Work

Table 1 provides a comparison between prominent requirements specification tools that support realizability checking with respect to various aspects, such as support for liveness properties, specification decomposition, algorithms.

Table 1. Comparison of requirements specification tools w.r.t. realizability checking.
Full size table

Spectra Tools [37] and RATSY [8] are requirements specification tools for reactive synthesis over the General Reactivity of Rank 1 (GR(1)) fragment of LTL. The GR(1) fragment is particularly appealing, because it subsumes a subset of requirements that may appear in real world problems, adheres to the popular Assume-Guarantee paradigm, and a polynomial-time synthesis algorithm exists for it [9, 46]. Both tools are limited to finite-state problems, and provide the ability to diagnose unrealizable specifications, primarily through the computation of minimal unrealizable cores [33, 40] and counterstrategy synthesis, where an implementation for the environment is generated, such that its actions always lead to the violation of the specification [34, 39]. Furthermore, Spectra Tools provide the ability to repair unrealizable specifications  [38].

SpeAR [22] and AGREE [14] are tools developed at Collins Aerospace for the purpose of requirements specification and analysis. Realizability checking is provided as a feature in both tools with limited support. Both tools depend on JKind ’s k-induction algorithm for realizability checking, which supports infinite-state problems, but is not sound with respect to unrealizable results [24].

EARS-CTRL [36] is yet another requirements specification platform that enables analysis of requirements written in Easy Approach to Requirements Syntax (EARS) [41]. Its realizability checking implementation relies upon autoCode4 [13], and is limited to the GXW subset of LTL [12]. Similar to Spectra Tools and RATSY, its analysis is limited to finite-state problems.

FRET ’s realizability-checking framework encapsulates desirable features of the aforementioned tools into an interface that is designed for users of varying backgrounds in formal methods. Additionally, it is the only requirements specification tool that provides a powerful decomposition approach to help with analysis performance [25, 43]. FRET ’s realizability framework is powered by the algorithms in JKind and Kind 2. As such, it can analyze requirements that are as expressive as arbitrary discrete past-time metric LTL (pmLTL) formulas, and which may involve arithmetic expressions over the Linear Integer and Real Arithmetic SMT-LIB logics [7]. In practice, the framework targets analysis of formulas corresponding to requirements written in FRETish, as presented in the next section. FRETish requirements correspond to templates that form only a subset of all pmLTL formulas. As long as future FRETish extensions can be translated into pmLTL, analysis will be supported by the realizability backend.

Table 2. Two FSM requirements in FRETish and pmLTL from Katis et al. [32].
Full size table

3 The FRETish Language

In FRET, requirements are written in a restricted natural language called FRETish  [27]. FRET formalizes FRETish requirements in pmLTL and then into Lustre. A FRETish requirement is described using up to six distinct fields (the * symbol designates mandatory fields): 1) specifies the time intervals where the requirement is enforced, 2) is a Boolean expression that triggers the to occur at the time the expression’s value becomes true, or is true at the beginning of the scope interval, 3) is the system component that the requirement is levied upon, 4) shall* is used to express that the component’s behavior must conform to the requirement, 5) specifies when the response shall happen, subject to the constraints defined in and and 6) is the Boolean expression that the component’s behavior must satisfy.

FRETish provides 8 scopes: , , , , , , , and . The scope means always; the others are with respect to when the system is in a mode or satisfies a Boolean expression. For example, means the requirement is enforced when the system is in mode M, as determined by the Boolean variable M. Also allowed for scope in place of a single Boolean variable is a Boolean expression, except for which in the expression case is written with ; e.g., . In FRETish, the optional condition field is introduced by the words , , or , which are synonymous in FRETish, or the word , which is the same as . FRETish provides 10 timings: , , , , , N time steps, N time steps, N time steps, bool_expr, and bool_expr. When the scope is omitted it is taken as ; when the condition is omitted, it is taken as true; when the timing is omitted, it is taken as . If we consider the condition being omitted as a separate case, there are \(8\times 2\times 10 = 160\) possible combinations of \(\langle \), , \(\rangle \), each formalized as a distinct pmLTL formula template. The templates are generated by an algorithm that has been formally proven to generate formalizations with the intended semantics [15].

Boolean expressions can use the standard logical connectives (!, &, \(\mid \)) and can involve arithmetic relations (=,\(\texttt {!\!=}\),\(\texttt {<}\),\(\texttt {<=}\),\(\texttt {>}\),\(\texttt {>=}\)) and operators (\(+\),−,\(*\),/) over integer and real variables. There are two predefined predicates preInt and preReal that refer to previous values: the expression \(\textit{preInt}(\textit{init},n)\), for integer expression n, returns the value of n at the previous timepoint; if at the beginning of the trace where there is no previous value, then the value of init is returned. Currently, FRETish does not allow arbitrary nesting of temporal operators, e.g. “In mode m, before q the system shall ...”. Timed operators with intermediate bounds are also not currently expressible; e.g., the equivalent of H[i,j] p, where i \(\ne \) 0.

For the remainder of the paper we use a running example, namely Finite State Machine (FSM), to demonstrate the various aspects of our framework. FSM contains 13 requirements for an abstracted version of an advanced autopilot system, and is part of the Lockheed-Martin Cyber-Physical Challenge Problems [18, 32, 42]. The requirements capture safety expectations with regards to the autopilot system’s state transitions. Table 2 contains two FSM requirements written in FRETish and their pmLTL formulas, which are generated by FRET.

4 Implementation

Figure 1a shows the architectural components of FRET that communicate with or belong to the Realizability Analysis framework.Footnote 1 Grayed components illustrate the contributions of this paper. The asterisks in Simulator and JKind indicate that their existing implementation and features were considerably extended for this work. Arrows show the flow of data between components. All components are implemented in JavaScript using the React, Material-UI and D3 libraries [2, 5, 6].

Fig. 1.
figure 1

Implementation views for realizability checking in FRET.

Full size image

FRET requirements are written using the Editor/Elicitor component, which also provides semantic explanations in various forms to assist users to clarify subtle semantic issues. The Simulator component provides an interactive visualizer based on graphical signal representation. Given a FRET requirement, it shows temporal traces of each of the variables involved as well as the valuation of the requirement for each point in time. The user can interactively modify the input signals, which results in automatically updating the valuation of the requirement and thus, visually inspecting the temporal behavior of the requirement. As part of this work, we extended the Simulator with the following features: 1) the ability to import and export simulation traces, 2) support for numerical expressions, and 3) simultaneous visualization of multiple requirements. We integrated the Simulator in our realizability analysis workflow, to provide the ability to inspect and interact with counterexample traces in unrealizable specifications.

The Variable Mapping component collects essential information provided by the user regarding the variables of the requirements, e.g., data types and correspondence to system inputs or outputs. Realizability Analysis consists of three sub-components. The Realizability Checking Engine is responsible for checking realizability of requirement sets either monolithically or compositionally. Given an unrealizable set of requirements, the Realizability Diagnosis Engine implements the algorithm proposed by Könighofer et al. [33, 34] to compute all minimal unrealizable sets of requirements, called minimal unrealizable cores. For each such core, a counterexample trace is computed that depicts a case under which the environment can lead the system into a deadlocking state. For the computation of minimal conflicts, our implementation uses the delta-debugging algorithm [49]. The Visualizer implements the user interface that displays analysis results as well as diagnostic results in the case of unrealizable specifications. These results are typically hard to digest in their original form. As such, the visualizer translates the information into an interactive diagram that allows the user to focus on unrealizable cores and inspect or simulate conflicting requirements.

We have integrated into FRET the JKind  [23] and Kind 2 [11] tools for checking realizability. We actively maintain a fork of JKind  [30], because the original repository lacks an implementation for the fixpoint algorithm by Katis et al. [31]. Formerly, the fork implementation relied on the AE-VAL solver’s Model-Based Projection algorithm to perform quantifier elimination over forall-exists formulas [20, 21]. As part of this work, we have improved its performance by utilizing Z3’s [16] quantifier elimination tactics. For instance, for the analysis of FSM the version of JKind using AE-VAL took 1524.82 s [43], whereas our optimization through Z3 dramatically decreased the time to 0.6 s.

The flow of usage of our framework is as follows (Fig. 1b). Once requirements are written in FRETish and variable information is provided, the user may start the analysis. Realizability can be performed through two different modes: 1) monolithic and 2) compositional, i.e., through the computation of independent sub-specifications, namely connected components. Each connected component is an undirected dependency graph with requirements as vertices and system outputs as edges. Compositional analysis has been proved faster and more prone to return result, compared to the monolithic option [43]. At the next step, the specification is translated to Lustre [29] and fed into JKind and Kind 2 to perform realizability checking. If the specification is unrealizable, the user can diagnose it using the generated counterexamples, and the FRET simulator.

5 Features Walkthrough

We next demonstrate the features of framework through our running example.

Fig. 2.
figure 2

The realizability checking interface in FRET.

Full size image

Realizability Checking. Figure 2 provides a snapshot of the overall graphical user interface (GUI) for realizability checking in FRET. As soon as the system component is selected, its connected components (CC) are computed. In the case of FSM, three CCs are identified. The GUI provides a focused view for each one (‘CCX’ tabs, with X being the corresponding index value), where the user can see which requirements participate in each CC via a table that dynamically grays out unrelated requirements. As soon as the CCs are computed, the realizability checking options become available, i.e., compositional and monolithic.

To check realizability, the user clicks the ‘Check’ button. Depending on the input specification, four possible answers may be given i.e., the specification is realizable, unrealizable, inconsistent, or the analysis is inconclusive (“unknown” result). Figure 2 shows the results of a compositional check for FSM, where connected components CC0 and CC1 are unrealizable, and CC2 is realizable.

Diagnosing Unrealizability. The compositional results above suggest that the FSM requirements are, as a whole, unrealizable. The next step in the process is to try and understand the source(s) of unrealizability. Since only CC0 and CC1 are unrealizable, it suffices to diagnose these independently. Following Fig. 2, the user selects the ‘CC0’ tab and clicks the ‘Diagnose’ button. The computation of minimal unrealizable cores kicks in, as outlined in Sect. 4, identifying 4 cores.

Visualizing Unrealizability. The raw artifacts produced by realizability checking and diagnosis are difficult for the users to digest. Therefore, the ability to visualize data in a user-friendly format is necessary, especially for unrealizable specifications. The core of our proposed solution to visualize unrealizability relies on the use of chord diagrams [1]. A chord diagram is a graphic representation of interrelationships between data, where each individual element is placed along the perimeter of a circular construct and relationships are depicted through edges between elements. An important feature of chord diagrams is the ability to maintain a clear representation of dependencies through hierarchical edge bundling [28], even when the size of data is large.

Fig. 3.
figure 3

(a) Chord Diagram for connected component CC0 in FSM. (b) Chord Diagram for Infusion_Manager. (c) Focused view (one core) for Infusion_Manager. (d) Focused view (one requirement) for Infusion_Manager.

Full size image

Figure 3a shows the chord diagram that is generated for connected component CC0 in FSM. Requirements and conflicts (i.e., unrealizable cores) define the input data to the chord diagram, which depicts each set using a distinguishable arc on the circular pattern (left and right arc, respectively). Chords, i.e., edges, connect each requirement to the conflicts that it appears in, with each edge being assigned a distinct color that matches the color-coded conflicts.

Table 3. Counterexample for conflicting requirements [FSM-006] and [FSM-007].
Full size table

While hierarchical edge bundling helps us maintain a clear total view, it may be the case that the engineer would like to focus on a particular subset of dependencies, related to either a particular requirement or a specific conflict. We enable this through interactive means where parts of the interface that are not related to the selected element can be filtered out. Figure 2 shows an instance where the user has already interacted with the chord diagram for CC0, focusing on the unrealizable core containing [FSM-006] and [FSM-007]. The table of requirements is dynamically sorted so that relevant requirements appear on the top, and are outlined with the color of the corresponding conflict. Additionally, a counterexample witnessing the unrealizability of the conflict is displayed. Table 3 shows the counterexample for requirements [FSM-006] and [FSM-007].

Fig. 4.
figure 4

Simulation of conflicting requirements [FSM-006] and [FSM-007].

Full size image

Simulating Conflicting Requirements. Our experience with counterexamples has indicated that a single execution trace is not enough to truly understand interactions between requirements. Therefore, we provide the ability for the user to interact with the set of conflicting requirements by using the FRET simulator, which we have substantially extended to meet our needs in visualizing conflicting requirements. Figure 4 shows how the counterexample (Table 3) for [FSM-006] and [FSM-007] is displayed in the simulator window: each line shows the values of the input signals as well as the valuation of each of the requirements.

The counterexample in Table 3 is not the only witness to the unrealizability of these requirements. Another example is a trace where requirement [FSM-006] holds for 5 consecutive ticks, leading to a violation of requirement [FSM-007] at the last tick, assuming that the antecedent of the latter was true at least once within the last 5 ticks. By modifying the values of the input variables, a user may identify additional witnesses to unrealizability causes. Combined with the ability to store and review traces, the simulator makes for an integral element towards understanding and repairing unrealizable specifications.

6 Case Studies

6.1 Lift Plus Cruise Aircraft

This study reports preliminary results on requirements for an autonomous ‘lift plus cruise’ concept aircraft.Footnote 2 This aircraft has a hovering vehicle mode, using its lifting rotors. From the hover mode, it can transition to a flying forward mode, eventually using its rear pusher propeller, and where lift is provided by the wing instead of the lifting rotors. Inbetween the hover and forward modes is a transitional mode which is a phase of concern for the aircraft engineers.

Table 4. FRETish requirements for Lift Plus Cruise from Katis et al. [32].
Full size table

As of this paper, 11 requirements have been formalized in FRET  [32]. A subset is shown in Table 4, describing the transition relations and constraints among various vehicle modes and vehicle motion. Requirement [LPC01] states that starts in hover mode. Requirement [LPC04] specifies that if the previous mode is hover, and ground speed is greater than 15 knots, then enters transitional mode. Requirement [LPC09] states the conditions for transitioning to forward mode. Variables hover, transitional and forward are specified as distinct integer constants. All of the other variables, e.g., airspeed, rear_propeller, are outputs.

The first complete set of FRETish requirements raised concerns, as realizability checking yielded non-sensical counterexamples, where at least one requirement between [LPC04] and [LPC09] was violated in the initial state. We quickly identified the issue: both requirements were written using a version of the ‘previous’ operator pre which is undefined at the initial state. We addressed this by introducing the preInt and preReal operators, which at the initial state return the value of their first argument.

The resulting 11 requirements are in one CC, so we ran analysis in monolithic mode. The requirements are shown to be realizable in about 8 s. As a sanity check for realizability, we experimented with various subsets of the original requirements, as well as adding contradictions. A notable example was omitting [LPC01], while modifying [LPC03] so that in hover mode, must fly faster than 30 knots. This experiment, unexpectedly to us, led to realizability. Further inspection quickly revealed how omitting [LPC01] allows the controlled variable vehicle_mode to never enter the hover mode. Including [LPC01] led to unrealizability with minimal conflict [LPC01], [LPC02] and [LPC03].

6.2 Generic Infusion Pump

This study explores 12 formalized requirements, proven unrealizable by Gacek et al. [24], of the Infusion_Manager subcomponent for a Generic Patient Controlled Analgesic (GPCA) infusion pump [44]. The GPCA system originates from the Generic Infusion Pump Research project, a joint effort to identify best software engineering practices in the development of medical devices [4].

Taking advantage of FRETish ’s support for system modes ( field), we derived 26 requirements, as opposed to the original 12  [32]. The increased number is a direct product of the declaration of 8 distinct modes, stemming from the system variable Current_System_Mode, which was originally of integer type. For example, requirement \(\mathbf {G1}\) from Gacek et al.:

$$\begin{aligned} {\textbf {G1}} \mathrel {{\mathop {=}\limits ^{\text {def}}}}~&({Current\_System\_Mode'} \ge 0)~\wedge ~ ({Current\_System\_Mode'} \le 8)~\wedge ~ \\&({Current\_System\_Mode'} = {0} \Rightarrow ~ {Commanded\_Flow\_Rate'} = 0)~\wedge ~ \\&({Current\_System\_Mode'} = 1 \Rightarrow ~ {Commanded\_Flow\_Rate'} = 0) \end{aligned}$$

was rewritten into three requirements: \(\mathbf {G1_{1}}\) ensures that the system is in at least one of the 8 modes at any time, while requirements \(\mathbf {G1_{2}}\) and \(\mathbf {G1_{3}}\) ensure that the pump’s flow rate is equal to 0 when the system is in mode 0 or 1, respectively. We additionally introduced requirements to ensure mutual exclusion between modes, something that was not needed with a single mode variable. We used Kind 2 to show equivalence between our requirements and the original specification.

Gacek et al. had already shown that the Infusion_Manager requirements are unrealizable, verbally attributing unrealizability to a conflict between G1 and requirement G7:

$$\begin{aligned} \begin{aligned}&{\textbf {G7}} \mathrel {{\mathop {=}\limits ^{\text {def}}}}({System\_On} \wedge {Highest\_Level\_Alarm} = 3) \Rightarrow \\&\;\;\;\;({Commanded\_Flow\_Rate'} = {Flow\_Rate\_KVO}) \end{aligned} \end{aligned}$$

The authors claimed that the requirements are unrealizable because they disagree on the value of output Commanded_Flow_Rate under specific conditions. However, FRET ’s diagnostic procedure provided a different answer, identifying 8 minimal unrealizable cores. Furthermore, the assumed conflict between requirements G1 and G7 does not really exist. While the two requirements do disagree on the value for the system output Commanded_Flow_Rate under specific circumstances, a realization still exists: one which would never exercise modes 0 or 1! Nevertheless, the report by Gacek et al. was still on the right track, as part of G1 (FRETish requirement \(\mathbf {G1_3}\)) and G7 participate in at least one minimal unrealizable core with requirement G11, the latter enforcing the system to enter mode 1, given specific system input values:

$$\begin{aligned} \begin{aligned} {\textbf {G11}} \mathrel {{\mathop {=}\limits ^{\text {def}}}}({System\_On} \wedge {Configured} < 1) \Rightarrow {Current\_System\_Mode' = 1} \end{aligned} \end{aligned}$$

Figure 3b shows the chord diagram for Infusion_Manager, depicting the 8 minimal unrealizable cores. Figures 3c and 3d show resulting states of the diagram after the user interacted with it in order to focus on a specific core, or a specific requirement, respectively.

7 Conclusion

We presented the realizability analysis framework in FRET and demonstrated its interactive GUI, which helps users diagnose unrealizable specifications through visualizations and simulation of conflicts. The framework employs state-of-the-art analysis algorithms that support infinite theories. In the future, we plan to extend the tool with recommendations in the form of environment assumptions.