1 Overview

As model checking becomes more integrated into the standard design and verification process for safety-critical systems, the platforms for model-checking research have become more limited (e.g., for the SMV language [47], neither CadenceSMV [46] nor NuSMV [24] are actively maintained; only closed source nuXmv  [15] remains). Continuing advances in the field require utilizing higher-level languages that offer sufficient expressive power to describe modern, complex systems and enable validation by industrial system designers. At the same time, contributing advances to back-end model-checking algorithms requires the ability to compare across the full range of state-of-the-art algorithms without regard for which open- or closed-source model checkers implement them or what input languages those tools accept. Comparing new advances in model-checking algorithms to state-of-the-art algorithms requires re-implementing entire model checkers, e.g., [30]. We need a sustainable tool flow that can model the system in the most domain-appropriate high-level modeling language, analyze it with the full range of state-of-the-art model-checking algorithms, and return counterexamples or certificates in the original modeling language.

Our tool suite represents an initial step in unifying model-checking research platforms. We seed an extensible framework designed around a model-checking intermediate language, MoXI (Model eXchange Interlingua). MoXI aims to serve as a common language for the international research community that can connect popular front-end modeling languages with the state of the art in back-end model-checking algorithms. Our vision is that MoXI will enable researchers to model-check a new or extended modeling language simply by writing translators to and from MoXI. Similarly, developing a new backend model-checking algorithm will only require writing a translator to and from MoXI to enable comparisons with existing algorithms and evaluations on every benchmark model, regardless of its original modeling language.

Our initial tool suite accepts models in the higher-level language SMV [47] and efficiently interfaces with the back-end model checkers that competed in the last Hardware Model Checking Competition (HWMCC) [13]. We choose SMV because it is a popular, expressive modeling language successfully used in a wide range of industrial verification efforts [14, 17, 23, 29, 30, 33, 34, 36, 42, 45, 48, 49, 54, 61, 63,64,65]. SMV is important because, uniquely from other model-checking input languages, it includes high-level constructs critically required for modeling and validating safety-critical systems, such as many aerospace operational systems from Boeing’s Wheel Braking System [14] to NASA’s Automated Airspace Concept [34, 45, 64, 65] to a variety of Unmanned Aerial Systems [55, 59]. SMV has been used extensively by the hardware model-checking community as well (e.g., at FMCAD [38]) and has appealing qualities that could further the integration of formal methods with the embedded-systems community. Two freely available model checkers, CadenceSMV [46] and NuSMV [24] (which is integrated into today’s nuXmv [52]), previously provided viable research platforms. However, today, CadenceSMV’s 32-bit pre-compiled binary and nuXmv ’s closed-source releases are no longer suitable for research, e.g., into improved model-checking algorithms. We provide accessibility to continue the progression of high-level language model checking in SMV via an open-source research platform that allows the use of new algorithms under the hood.

Pushing the state of the art are several open-source, award-winning model-checking tools, including AVR  [35], Pono  [44], BtorMC  [51], and ABC [18]. These tools support a hardware-oriented bit-level input language like Aiger or a bit-precise, word-level format like Btor2. Unfortunately, such languages do not enable the direct modeling of modern complex systems as SMV does, hindering validation efforts. For instance, it is challenging to convince industrial system designers that Aiger models correctly capture their higher-level systems. Perhaps driven by HWMCC, most systems for translating from high-level models to Aiger currently focus on hardware designs, without providing a natural way to describe other computational systems, e.g., embedded systems. Also, the problem of translating counterexamples produced by low-level model-checking algorithms back into meaningful counterexamples for a non-hardware-centric higher-level language model, such as one in SMV, remains a challenge.

Section 2 provides a basic introduction to MoXI, sufficient to enable understanding of the tool suite functionality; a description of the full language and its semantics appears in [57, 58]. Section 3 details the extensible research and verification suite of tools, including translators between the languages SMV, MoXI (in concrete and JSON dialects), and Btor2; utilities for validation; and a full model-checking implementation. Here, we provide a detailed example of behaviorally equivalent models in SMV, MoXI, and Btor2. Our efforts to validate their correctness appear in Sect. 4. Section 5 demonstrates the efficiency of model checking SMV-language models with a tool portfolio including nuXmv and via translation through MoXI, which performs better than checking with nuXmv alone. The toolFootnote 1 and all of the benchmarksFootnote 2 used in this experiment are available online for others to utilize in building additional translators to extend our tool suite and the use of MoXI as an intermediate language for symbolic model checking. Section 6 concludes with a discussion of future work.

2 Intermediate Language

MoXI (detailed in [57]) is an intermediate language designed to serve as a common input and output standard for model checkers for finite- and infinite-state systems. It is general enough to encode high-level modeling languages like SMV yet simple enough to enable efficient model checking, including through low-level languages such as Btor2 or SAT/SMT-based engines. Key features include a simple and easily parsable syntax, a rich set of data types, minimal syntactic sugar (at least for now), well-understood formal semantics, and a small but comprehensive set of commands.

MoXI maximizes machine-readability. Therefore, it does not support several human-interface features found in high-level languages such as SMV, TLA+ [43], PROMELA [37], Simulink [27], SCADE [28], and Lustre [19]; nor does it directly support the full features of hardware modeling languages such as VHDL [40], or Verilog [39]. However, many models and queries expressed in these languages can be reduced to MoXI representations. MoXI development was directly informed by previous intermediate formats for formal verification, their successful applications, and their limitations. The eventual form of MoXI stems from a combination of previous work as well as direct conversations with model checking and SMT researchers, including the developers of Aiger  [2,3,4], Btor2  [51], Kind 2 [22], NuSMV [21], nuXmv  [16, 20], SAL/SALLY [9, 32, 50], VMT [26, 41], and SMT-LIB (the standard I/O language for SMT solvers) [6, 7]. MoXI also benefited from the feedback from a technical advisor board of prominent researchers and practitioners in academia and industry [58].

MoXI ’s base logic is the same as that of SMT-LIB Version 2: many-sorted first-order logic with equality, quantifiers, let binders, and algebraic datatypes. MoXI extends this logic to (first-order) temporal logic while adopting a discrete and linear notion of time with standard finite and infinite trace-based semantics. MoXI also extends the SMT-LIB language with new commands for defining and verifying multi-component reactive systems. For the latter, it focuses on the specification and checking of reachability conditions (or, indirectly, state and transition invariants) and deadlocks, possibly under fairness conditions on system inputs. Each system definition command defines a transition system by specifying an initial state condition, a transition relation, and system invariants. These are provided as SMT formulas, with minimal syntactic restrictions, for flexibility and future extensibility. Each defined system is parameterized by a state signature, provided as a sequence of typed variables, and can be expressed as the synchronous composition of other systems.Footnote 3 The signature partitions state variables into input, output, and local variables. Each system verification command expresses one or more reachability queries over a previously defined system. The queries can be conditional on environmental assumptions on the system’s inputs and fairness conditions on its executions. Together with the ability to write observer systems, this allows the expression of arbitrary LTL specifications via standard encodings [56]. Responses to a system verification command can contain (finite or lasso) witness traces for reachable properties or proof certificates for unreachable ones.

Figure 1 contains an example (adapted from [5]) of a three-bit counter and its modular definition in MoXI, together with a reachability query and a sample response to the query. Figure 2 contains an extension of that model with an observer system and a query for checking the observational equivalence of the three-bit counter with a bit-vector counter of matching width. The various components of each system definition or check command are provided as attribute-value pairs, following the syntax of SMT-LIB annotations. Transition predicates use primed variables to denote next-state values.

Fig. 1.
figure 1

(Top) The three-bit counter circuit composes three one-bit counters together, where each counter uses a latch to store that counter’s current value. (Middle) A MoXI implementation of the circuit uses define-system (lines 1–20) to describe and compose each counter component. It then queries (lines 21–26) whether the counter can output 2. (Bottom) A possible query response provides a trace showing that the counter outputs 2 within 3 execution steps. We write Bool values as integers here for compactness.

Full size image
Fig. 2.
figure 2

(Top) Extending the MoXI model shown in Fig. 1, a Monitor (lines 33–40) computes the output for a ThreeBitCounter and a bit-vector-based counter (lines 28–32). The function to_bv3 (definition is omitted for space constraints) converts bit values to the corresponding bit-vector value. The check-system command (lines 40–45) queries whether their outputs can possibly differ. (Bottom) The check-system-response reports an unsatisfiable query, proving the two counters equivalent.

Full size image

3 Tool Suite

We provide a suite of tools for translating into and out of MoXI and validating MoXI scripts. The tools are implemented in type-annotated Python with a focus on finite-state systems (for now). Figure 3 illustrates the end-to-end toolchain for model checking using MoXI, including relationships between the various tools.

Fig. 3.
figure 3

Starting with a nuXmv model, smv2moxi generates a behaviorally equivalent MoXI model in either the MoXI concrete syntax or a JSON dialect syntax. moxi2btor translates this MoXI model to a set of Btor2 models, one for each query, which an off-the-shelf model checker (e.g., AVR  [35], Pono  [44], BtorMC  [51]) solves. Then, btorwit2moxiwit creates a MoXI witness from the Btor2 witness using the Btor2 model to map variable names properly, and similarly for moxiwit2smvwit. The sort checker validates MoXI input against any of the SMT-LIB logics listed in Sect. 3.2. The validator checks JSON dialect input against our provided schema.

Full size image

3.1 Translators

The tool suite provides four translators that take as input a model, query, or witness specified in a source language and output a behaviorally equivalent model, query, or witness in the configured target language.

(1) smv2moxi translates specifications written in (a common subset of) the SMV language into MoXI. Broadly, this tool supports Finite State Machine (FSM) definitions (nuXmv manual, Sect. 2.3 [16]). It currently supports only statically typed expressions; for example, all module instantiations of the same defined module must share the same signature. (For a module M with parameters p1 and p2, the types of p1, p2 must be the same across all instantiations of M.) Fig. 4 shows that the translation preserves the hierarchy between the SMV modules and submodule instantiations.

The MoXI encoding captures SMV macro and function declarations (DEFINE, FUN), variable declarations (VAR, IVAR, FROZENVAR), state machine declarations (INIT, TRANS, INVAR, ASSIGN), invariant specifications (AG [property], INVARSPEC) and fairness constraints (FAIRNESS, JUSTICE, COMPASSION). To support LTL specifications (LTLSPEC), smv2moxi runs PANDA [56], an open-source tool offering a portfolio of LTL-to-symbolic automaton translations in SMV format.

The smv2moxi tool consists of (1) preprocessing that renames identifiers deviating from the SMV grammar (discussed in Sect. 4); (2) running the C preprocessor (SMV supports C-style macros) and PANDA [56] (for LTL specifications); (3) parsing via a SLY-generated [8] parser; (4) running an SMV type checker; (5) translating to MoXI. We emphasize that tool guarantees apply to well-formed SMV models as determined by nuXmv.

(2) moxi2btor translates MoXI to Btor2 by creating a Btor2 file for each :query attribute in each check-system command. Some crucial differences between MoXI and Btor2 present non-trivial challenges. Firstly, Btor2 does not support hierarchical models. moxi2btor flattens the system hierarchy in its translation as a result. Secondly, MoXI allows for declarative-style initial, transition, and invariant conditions while Btor2 allows only assignment-style. Figure 4 shows how moxi2btor encodes each system’s conditions using three variants of each variable. Thirdly, a MoXI query with multiple reachability properties asks for a trace that eventually satisfies each property. In Btor2, multiple bad properties in a file ask for a trace that eventually satisfies at least one such property. Figure 4 again shows how the translation resolves this difference. The moxi2btor tool’s workflow consists of (1) parsing via a SLY-generated parser [8]; (2) running sortcheck (Sect. 3.2); (3) translating to a set of Btor2 files, each behaviorally equivalent to its corresponding :query.

(3) btorwit2moxiwit translates Btor2 witnesses to MoXI witnesses using the check-system-response syntax. It assumes moxi2btor created the Btor2 input files used to generate the witness and uses information that moxi2btor encodes in the comments of each Btor2 file, e.g., to map bit vectors to enumeration values for variables of such sorts.

(4) moxiwit2smvwit translates MoXI witnesses to SMV-language witnesses.

Fig. 4.
figure 4

The toolchain translates the SMV model for a delay circuit on the left to the MoXI model in the center by creating a define-system command for each MODULE. It then generates the Btor2 model on the right, introducing three variants of each check-system variable (.init, .cur, .next) and setting constraints such as the :init and :next of Delay on lines 13 and 15 respectively. The Btor2 “flag” variable F_rch (line 27) encodes if formula rch has been true at least once during the execution; the presence of multiple Btor2 bad properties asks for a trace where at least one such property is eventually true, we conjunct the flag variables to ask for a trace where every property is eventually true.

Full size image

3.2 Utilities

sortcheck We provide a sort-checker for MoXI that supports the following SMT-LIB logics: QF_BV, QF_ABV, QF_LIA, QF_NIA, QF_LRA, and QF_NRA.

validate We define a JSON Schema for MoXI and support a JSON dialect for MoXI in our tools. Given the evolving nature of new languages and their standards, tool writers often pay an unnecessary overhead keeping front-end tools up to date. By supporting the representation of MoXI constructs in the JSON dialect, we expect to facilitate tool development, improve tool interoperability, and ensure conformance to the language standard. Tool writers can use off-the-shelf JSON parsers (e.g., simdjson, RapidJSON) to obtain industrial-strength MoXI parsers in the language they choose “for free.” We plan to include a JSON schema for each MoXI release, enabling seamless front-end compatibility with the latest MoXI standard along with language/platform independence. The validate utility invokes a JSON validator from Python’s jsonschema package to validate a MoXI script (in the JSON dialect) against the MoXI JSON schema.

4 Tool Suite Validation

We validate our tools using a combination of manual inspection, sort checking of translated output, and comparing witnesses between those generated by nuXmv and our end-to-end tool suite. We use catbtor  [51] for sort checking and BtorMC, AVR, and Pono for bounded model checking (BMC) of Btor2 files. For benchmark generation, we use the set of nuXmv input files provided in the most recent release of nuXmv (Fig. 5).

Fig. 5.
figure 5

The witness translation after model checking the Btor2 file in Fig. 4 works right to left: it maps each Btor2 .cur variable to its MoXI counterpart and discards the last frame of the witness due to the delay caused by using flag variables. Similarly, it maps each MoXI variable to its SMV counterpart.

Full size image

Manual Inspection. We provide an initial set of hand-written MoXI benchmarks to perform manual validation. Each benchmark is well-sorted according to sortcheck, generates well-sorted Btor2 via moxi2btor according to catbtor, and generates correct, manually-inspected witnesses via BtorMC and btorwit2moxiwit.Footnote 4

Sort Checked Translations. Using the benchmarks distributed with nuXmv as input, we check that the output of smv2moxi and moxi2btor are well-sorted according to sortcheck and catbtor. We discovered discrepancies in benchmarks distributed with nuXmv while developing these utilities, where the benchmarks did not conform to the grammar defined in Chap. 2 of the nuXmv User Manual [16] but were accepted by nuXmv nonetheless, particularly concerning identifiers. The preprocessor of smv2moxi transforms these identifiers into valid ones. There were also numerous ill-typed benchmarks that smv2moxi ’s type checker correctly rejects.

Output Comparison. Using the nuXmv benchmarks again as input, we run nuXmv and our tool suite to generate witnesses for each specification. Both nuXmv and our tool suite agree on the result of every model-checking query. Section 5 describes how our toolchain (using BtorMC, AVR, or Pono as its back end) shows a similar number of timeouts compared with nuXmv when the latter is set to use BMC or k-induction.

5 Benchmarks

We provide an initial set of MoXI benchmarks for the model-checking community generated from the set of SMV input files provided in the most recent release of nuXmv. Noting that many of the SMV benchmarks are results of a Btor2 to nuXmv translation themselves, we stress that this set of benchmarks is intended to be an initial set. We expect to achieve greater benchmark diversity with continued toolchain development and increased adoption of MoXI by other researchers.

Experimental Evaluation. We compare the end-to-end performance of model-checking SMV-language models with a portfolio comprising nuXmv and Btor2 model checkers: AVR, Pono, and BtorMC, on a set of 960 QF_ABV-compatible SMV benchmarks, i.e., SMV models with boolean, word or array types. We use the HWMCC 2020 versions of AVR and Pono, the version of BtorMC from the latest version of Boolector [51], and the latest public release of nuXmv (version 2.0.0). Each checker is configured with a 1-h time limit and 8GB memory limit and runs BMC [12] and k-induction [60] with a max bound of 1000. (We do not run BtorMC with k-induction due to a bug in its implementation.)

Figure 6 shows our evaluation, with portfolio performance depicted as virtual-best (vb). While we consider this a proof-of-concept evaluation, we observe that SMV-language model checking using Btor2 model checkers, enabled via a translation through MoXI, delivers superior performance on unsafe queries compared to model checking with nuXmv alone: vb-bmc solves 57% more benchmarks than nuXmv-bmc while ensuring all Btor2 witnesses are correctly translated to SMV traces. We measure competitive performance with vb-kind solving 6% more benchmarks than nuXmv-kind for safe queries. The vb performance gains are due to its ability to use a variety of model checkers with different SMT solver backends of varying strengths, e.g., nuXmv uses MathSAT [25], AVR uses Yices [31], and Pono uses Boolector [51], while ensuring correct model and witness translation through MoXI. Section 4 of Rozier et al. [57] includes experimental data using each tool’s IC3-based algorithms.

Fig. 6.
figure 6

Performance comparison on unsafe and safe queries with BMC and k-induction across different model checkers. vb-* represents the virtual best solver. Wall-clock time for the non-nuXmv plots includes translation time.

Full size image

6 Conclusion and Future Work

The presented tool suite provides the foundational step in developing an open-source, state-of-the-art symbolic model-checking framework for the research community. It constitutes the first tool support for the new intermediate language MoXI, the first experimental evidence of the potential for efficient translation through MoXI, and a basis upon which the hardware and software model-checking communities can build. Adding support for checking models in a high-level modeling language is now as easy as adding a translator between that language and MoXI to this tool suite. Similarly, experimenting with a novel back-end model-checking algorithm to check all supported input modeling languages only requires writing a new MoXI translator interfacing with that algorithm. Benchmarking against other model-checking algorithms no longer require re-implementing existing tools to achieve an apples-to-apples comparison.

Connecting this toolchain to existing tools enables the immediate application of verification techniques for Btor2 to MoXI beyond just hardware model checkers. For example, a software model checker can verify a MoXI model via Btor2C  [11], making at least 59 other backend verifiers for MoXI available [10].

This release enables future instantiations of HWMCC [13] to add competition tracks centered around MoXI, with extensions from the model-checking research community. Specifying, proving correct, and extracting efficient C code for our translation using a theorem prover such as PVS [53] would provide an additional trusted translation between languages beyond the validation techniques in Sect. 4. We are writing a back end to Yosys [62], the open-source RTL synthesis framework, to generate files directly from Verilog designs and facilitate a more extensive set of realistic benchmarks to add to the initial set in Sect. 5. Additionally, once MoXI certificates are fully defined, we can translate Btor2-Cert  [1] certificates back to MoXI from Btor2-Cert-supported verifiers. Finally, we expect developers of model checkers for higher-level modeling languages than a language like Btor2 may choose to support MoXI directly. We have work in this direction underway for the Kind 2 checker [22].