Timescales: A Benchmark Generator for MTL Monitoring Tools

Ulus, Dogan

doi:10.1007/978-3-030-32079-9_25

Dogan Ulus¹⁰

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11757))

Included in the following conference series:

International Conference on Runtime Verification

856 Accesses
9 Citations

Abstract

This article presents a benchmark generator, Timescales, which can be used to evaluate the performance and scalability of runtime verification tools using Metric Temporal Logic (MTL) formulas as their specifications. We mainly target runtime verification of cyber-physical systems and generate traces similar to the qualitative behavior of sensor readings and state variables of such systems that are observed/sampled continuously. Since such systems are composed of many heterogeneous components that work over very different time scales, it is crucial to measure the performance of the MTL monitoring tool for a wide range of timing parameters in specifications. Hence, Timescales supports the generation of benchmarks for 10 typical timed properties for any given trace length and timing parameters with several other useful features. Finally, we include some default benchmark suites generated by Timescales.

Download conference paper PDF

A Benchmark Library for Parametric Timed Model Checking

MLTL Multi-type (MLTLM): A Logic for Reasoning About Signals of Different Types

Multi-head Monitoring of Metric Temporal Logic

1 Introduction

Cyber physical systems (CPS) refer to large-scale interconnected control and communication systems that incorporate physical and computing components at various levels. Real-time decision making is the defining characteristic of cyber physical systems such that it does not only mean responding to soft and hard deadlines but also making decisions in a timely manner based on an ongoing interaction between the system and environment. The design of CPS often involves heterogeneous components that operate on different temporal scales and the correct operation requires a high level of coordination and cooperation among such components. Overall the design results in very complex artifacts to verify the correctness and evaluate the performance. Therefore, we usually resort to conventional simulation and model based testing methods for the analysis of CPS. Current research efforts include developing fast, scalable, and versatile runtime verification techniques and tools that handle complex timing requirements of CPS and provide an additional level of rigor and effectiveness over conventional testing methods.

Metric Temporal Logic (MTL) [6] is a popular formalism to specify temporal properties with timing constraints over the behavior of cyber physical systems. Several existing runtime verification (RV) tools support MTL as their specification and employ different techniques and algorithms to monitor MTL formulas over temporal behaviors (traces) [1, 2, 7,8,9]. From any MTL monitoring tool, we typically expect a linear-time performance in both the length of the trace and the size of the formula. Furthermore, the insensitivity to the base time unit and numeric time values in formulas is a very much desired feature for MTL monitoring tools. This becomes especially crucial for CPS applications since we usually have to use small and large numerical constants in the same formula when specifying timing requirements of components that operate on different timescales. Hence, in this paper, we present Timescales benchmark generator, which can be used to evaluate the performance and scalability of MTL monitoring tools. In particular, Timescales generates temporal behaviors for a predefined set of MTL formulas with parameterized timing constraints. We consider the past and future fragments of MTL separately and our benchmarks consist of one trace file in the comma-separated values (CSV^{Footnote 1}) format and two (past and future) specification files in the YAML^{Footnote 2} format. Besides we provide ANTLR^{Footnote 3} grammar files to parse our MTL formulas. These file formats are widely supported and their implementations are already available for many programming languages.

Timescales is essentially developed to help measure the typical performance of MTL monitoring tools. Therefore, we, first and foremost, consider the most common types of timed properties encountered in real system designs. Such typical properties have been studied in the papers [4, 5], which employs the template system developed by Dwyer et al. for untimed specifications [3]. In this template system, a property consists of (1) a pattern, which describes what must be observed, and (2) a scope, which describes the temporal extent of the pattern. Each property further contains one or two timing parameters, which can be controlled by the user. In Timescales, we support a total of 10 typical timed properties over 4 pattern (absence, universality, recurrence, response) and 4 temporal scopes (before, after, between, globally) for the benchmark generation. Then, the tool generates a discrete time behavior that satisfy the property for a given property, duration, and values of timing parameters. We also provide an option to generate dense time behaviors where the maximum density is controlled by the user. Since we usually want to generate a collection of benchmarks by varying all these parameters over a grid, we include some example scripts to generate such benchmark suites with some predefined values.

The structure of this paper is as follows. Section 2 overviews the syntax and semantics of MTL as used in this paper. In Sect. 3, we describe supported timed properties and corresponding MTL formulas for the benchmark generation. Section 4 explains the benchmark generation generally and gives further details about the implementation including trace generation, output formats, and default benchmark suites generated by Timescales.

2 Metric Temporal Logic

Metric Temporal Logic (MTL) [6] is an extension of linear temporal logic (LTL) in which temporal operators are endowed with timing constraints. In this paper, we interpret MTL formulas over a bounded discrete time domain $\mathbb {T} = [1, N]$ of total duration N and use so-called non-strict (reflexive) semantics of temporal operators, timed since ($\mathcal {S}_{I}$) and timed until ($\mathcal {U}_{I}$). Given a finite set P of atomic propositions, the formulas of MTL are defined by the following grammar:

$$\begin{aligned} \varphi \ =\ p \ |\ \lnot \, \varphi \ |\ \varphi _{1} \vee \varphi _{2}\ |\ \varphi _{1}\, \mathcal {S}_{I}\, \varphi _{2}\ |\ \varphi _{1}\, \mathcal {U}_{I}\, \varphi _{2} \end{aligned}$$

where $p\in P$ and $I \subseteq [0, \infty )$. Then the satisfaction relation $(w, t) \vDash \varphi $ indicates that the Boolean temporal behavior $w : \mathbb {T} \rightarrow \mathbb {B}^{P}$ satisfies the formula $\varphi $ at the time point $t \in \mathbb {T}$ as follows.

where we use $w_p : \mathbb {T} \rightarrow \mathbb {B}$ to denote the projection of w onto its component p. Other timed modalities include time constrained variants of sometime in the past ($\blacklozenge _{I}$) and always in the past ($\blacksquare _{I}$) as well as sometime in the future ($\lozenge _{I}$) and always in the future ($\square _{I}$). For any temporal operator, we usually omit the time bound if there is no constraint and call such an operator untimed. The past fragment of MTL, or past MTL in short, is defined as a restriction of MTL without the until operator. Similarly we do not use the since operator in the future fragment. In this paper, we write MTL formulas either in the past or future fragment of MTL.

3 Supported Properties

This section overviews supported timed properties in Timescales. By default a single benchmark consists of a logical formula that capture the property either in the past or future fragment of MTL and a temporal behavior that satisfies the formula. We select 10 typical timed properties for the benchmark generation and we cover in these properties all types of temporal operators and different types of intervals for timing constraints. Each timed property is parameterized with one or two timing parameters in order to generate benchmarks with different timing characteristics. Note that we do not claim or seek the logical equivalence between the past and future MTL formulas in the following but capture the intention in both fragments.

Bounded Absence After Q. This property has one timing parameter u. Intuitively it means that it is always the case that the event P does not occur at least for U time units after the event Q occurs. We capture this property by the formula