Keywords

1 Introduction

Over the past decades, the scope of runtime verification has grown from an essentially boolean check, indicating whether or not a program execution satisfies a given formal specification, towards the real-time computation of more and more expressive statistical data. A typical example are Unmanned Aerial Systems (UAS), where the monitor continuously collects and aggregates inputs from sensors and on-board components to provide the human operator with critical information in hand-over situations [1, 3, 26].

Traditionally, the interaction between runtime verification tools and their users has largely been based on textual interfaces, such as “alert” messages that are issued in case of a violation of the specification. In applications like UAS, however, such a simple user interface is often no longer sufficient. In addition to understanding that the monitor has detected a problem, the human operator must quickly grasp the situation and decide on potentially time-critical corrective action. A well-designed visual presentation of the available data is therefore of critical importance for the safe operation of the system.

Fig. 1.
figure 1

Screenshot of a visualization displaying the battery status, velocity, and GPS coordinates of a UAS.

Full size image

Generating useful visualizations is far from trivial. First and foremost, the visualization must ensure that important or dangerous information is clearly visible to the user; because of the abundance of available data, data must be prioritized, and less important data must be hidden in favor of more important data. Similarly, the frequency of data points must be adjusted to provide meaningful information avoiding overlaps and adjusting for discrepancies in the availability of the raw data. All these computations have to be adjusted in response to actions by the user, who may look at multidimensional data from different angles or zoom into data areas of particular interest.

In this paper, we report on our recent effort in extending the RTLola monitoring framework with real-time visualization capabilities. RTLola  [4, 11] is a stream-based monitoring framework for cyber-physical systems and networks. RTLola processes, evaluates, and aggregates streams of input data, such as sensor readings, and provides a real-time analysis in the form of comprehensive statistics and logical assessments of the system’s health. An RTLola monitor is generated from a formal description given in the RTLola specification language. The specifications consist of stream equations that translate input streams into output streams. RTLola specifications are statically analyzed to determine the required memory and are then either directly executed by the RTLola interpreter, or compiled onto an FPGA.

The fundamental insight of our approach is that the data processing needed to generate the visualization should be the responsibility of the monitor itself, rather than that of some external visualization tool. The monitor has access to all information and is therefore in the best position to determine the relevancy of individual data points. Because of the expressive power of the monitoring language, the monitor also has the computational means to interpolate and aggregate the raw data as required. Finally, keeping all data manipulations in one place reduces redundancy and avoids errors and misinterpretations.

We organize the RTLola specification of the data processing for the visualization into three functional areas: 1. Data Synchronization: This part of the specification guarantees synchronous data updates for the visualization. This is important because the visualization combines data from different streams into a single entry in the plot: for example, a point indicating the position of a drone might be annotated with the speed of the drone. If different attributes have different timing, for example because of the variations in the frequency of the sensors, the monitor interpolates the missing data. 2. Filtering and Aggregation: This part of the specification avoids overlapping markers in the visualization, which are caused if readings arrive at a high rate. The monitor smoothes the input signal and adjusts the rate according to the current visualization. 3. Priorization and Attention Management: This part of the specification determines the criticality of the available information and ensures that the human operator does not miss important information.

The monitor and the visualization component are connected via a synchronous interface. The responsibility of the visualization component is to create the graphical display and to react to user requests. Since this user interaction may affect the visibility of plots or change the scaling, a backchannel provides this information to the monitor in the form of additional input streams. Fig. 1 shows a screenshot of our prototype implementation, which is based on the monitoring framework RTLola and the visualization framework cgv [16]. The monitor interacts with configurable 2D and 3D plots that support time-series plots, scatter plots, trajectory plots and multi-variate visualization through a flexible mapping of data attributes to the visual attributes color, opacity and size. We have applied our approach to the real-time visualization of UAS and other cyber-physical systems, based on existing RTLola case studies; our experience suggests that adding the visualization specification inside the monitor leads to highly informative visualizations as well as to understandable and easily maintainable monitoring code.

1.1 Related Work

This paper connects two traditionally separate areas of research: runtime monitoring and visualization. Somewhat surprisingly, visualization has not played a major role in monitoring research before. Despite a wide range of monitoring approaches, from formal logic [9, 12, 17, 25] to stream-based specification languages [6, 7, 10, 15], most tools have in common that they rely on textual, rather than visual, methods for data presentation. This paper shows that stream-based monitoring languages like RTLola are very well suited to carry out the needed data processing for useful visualizations. Our focus on RTLola is motivated by recent work on RTLola-based monitoring for UAS [2, 3] and other cyber-physical systems [4, 5, 11]. However, the approach of the paper is clearly transferrable to other monitoring tools for CPS [22,23,24].

In the area of visualization, research on streaming visualization is also still in an early stage. A notable result are streaming processing models for data [28] and techniques for kernel density estimation in aggregated views over 2D maps [19]. There has been a systematic discussion of the suitability and problems of traditional visual analysis techniques [8, 18, 27]. Additionally, visualization frameworks [14] and visualization techniques that allow the user to better cope with changes over time have been developed, such as zoomable navigation [21], paged views [13], and transformation-based smooth transitions [20]. These approaches differ substantially from the approach taken in this paper, in that these are independent visualization tools, which prepare the data for the visualization independently of the monitor. By contrast, our setup tightly integrates the monitor with the visualization, keeping all data manipulations in one place.

2 RTLola

RTLola is a stream-based monitoring framework for cyber-physical systems and networks. An RTLola monitor is generated from a formal specification description given in the RTLola specification language. The specification consists of stream equations that describe the transformation of incoming data streams into output streams, and a set of trigger conditions that result in notifications to the user. The RTLola framework includes automatic static analysis methods that ensure the predictability of the monitor with respect to memory consumption and other relevant properties. We illustrate the RTLola specification language with a small example; for more details, we refer the reader to [4, 11].

figure a

The specification declares three input streams: The first stream represents readings received by the GNSS (global navigation satellite system) module, the second stream shows the battery charge status, and the third one contains the current time. Next, the specification declares the output stream, which filters the stream to timestamps of newly received battery readings. For this, it binds the timing of the stream to the timing of , indicated by the annotation. In RTLola, such a filter is called a static filter. As might have a different timing, the value is accessed via a 0-order hold interpolation. The next stream uses a dynamic filtering to exclude noisy sensor readings. In our example, the GNSS sends (0.0, 0.0) coordinates during initialization that the specification can discard. The last two lines contain triggers checking if there is an unusual drop in the battery status and if the coordinates do not exceed some thresholds.

In RTLola, the static and dynamic filters are combined into a pacing type, which defines the timing of each stream. This type is either inferred or explicitly annotated. RTLola ’s type checker verifies the timing of the streams and RTLola provides different operators to interpolate data if the timing constraints cannot be guaranteed in the stream expression.

3 Generating Visualization Data

We now describe in more detail the generation of visualization data with RTLola stream equations. For the communication from the monitor to the visualization component, the specification contains output streams that are mapped to plot coordinates and visual attributes such as size and color. It also contains an output stream per axis, setting its displayed range. For the backchannel, the specification has input streams that receive the data from the visualization reflecting the interaction of the visualization component with the user. For each 2D-plot, we include one input stream to transfer the current scale factors; for each 3D-plot we include two input streams, representing the projection matrix and window size. Additionally, the specification has an input stream for each plot indicating which plot is visible.

Fig. 2.
figure 2

RTLola specification demonstrating the interplay.

Full size image

We structure the generation of the visualization data into three areas: Data Synchronization and Interpolation, Filtering and Aggregation, and Prioritization. For each area, we shortly describe the problem, then describe the mechanism of how RTLola solves the task and explain the solution in more detail with our running example shown in Fig. 2. We display the coordinates of a GNSS in a 2-dimensional plot, and the remaining battery charge is mapped onto the color of the marker. The input streams and represent the sensor readings followed by the streams and implementing the backchannel. The output streams and compute the upper and lower bound per axis which is in our case the global minimum and maximum. Alternatively, our plot could represent with the following stream expression the data over a time-period \(\sigma \):

figure r
Fig. 3.
figure 3

Screenshot of the prototype with different monitors. On the left side, we use a monitor forwarding all data, whereas the right monitor filters the data using the specification in Fig. 2.

Full size image

The next streams are helper streams to filter the data and the last output stream contains all information needed for a new marker in the plot.

3.1 Data Synchronization and Interpolation

For drawing a marker, the visualization needs to know all its visual attributes which might be based on sensors with different frequencies. We use RTLola ’s type system to guarantee that the monitor sends synchronized updates per plot. For this task, we use the concept of pacing types as introduced in Sect. 2. We define a pacing for each plot and annotate the streams for this plot with the desired pacing. Then, we use RTLola ’s type checker to verify that the data is available. In our example, we want to create a marker whenever at least one sensor sends an update and therefore use the disjunction of the two input streams as the pacing type. This pacing type is annotated to the stream . Similarly, we want to update the axis limits, represented by the streams and , whenever we get a new GPS sensor reading.

We cannot directly access the current value of each stream because they may have different timings. Instead, we specify how missing data is interpolated. RTLola offers different approaches for this task, e.g., by using data aggregations, zero-order hold operations, or even different forms of data interpolations. In our example, we use a zero-order hold on the missing data.

3.2 Data Filtering and Aggregation

This section shows how a monitor prepares data to provide more understandable updates to the user. Figure 3 shows two plots from the same execution. On the left side, the monitor forwards all data to the visualization resulting in overlapping markers. These overlapping markers overload the plot with unrequired information and even overlap some information, as in our example, the color illustrating the battery status. On the right side, the monitor tailors the data for visualization and transfers the prepared data, so we do not have this problem.

The monitor on the right uses RTLola ’s dynamic filtering mechanism to prepare the data. With this filtering approach, the monitor can dynamically adapt the throughput to the visualization. Dependent on the scenario, different filters are helpful: For example, a filter forwards the GPS data dependent on the current velocity or increasing the sample rate if the system violates a property could be easily expressed in RTLola. In our running example, we forward the markers only if the difference between the new marker and the previous marker exceeds a threshold and if the plot is visible. For this, the output stream has two such filters connected by a disjunction. Both filters use the information provided by the visualization to describe the current state of the plot – visibility and scaling. With the second disjunct, we ensure that a marker is transmitted only if the plot is visible. The other filter is encoded by the stream . In this stream, we decide whether the difference to the previously transmitted marker is sufficient to warrant a new marker. For this, we first compute the distance between the pixel coordinates of the candidate marker and the last marker based on the current bounds and scaling, and compare it with a defined threshold that prevents overlapping. We also check whether the difference in the charging level warrants a new marker. A similar approach also applies to 3D plots. Instead of the , we use the projecting matrix that encodes besides the scaling, the viewing rotation, and perspective.

Depending on the scenario and the size of the plot, it can be useful to aggregate values (such as by computing the average, minimum, and maximum of the data since the last marker) instead of dropping values. In RTLola, this can easily be done using the corresponding aggregation functions.

Fig. 4.
figure 4

On the left side, one marker with irrelevant information covers a critical marker, whereas the monitor on the right encodes a form of priority.

Full size image

3.3 Attention Management

In Sect. 3.2, we have already discussed how the monitor can filter data points if they do not contain relevant information. Often, however, this is not sufficient, and we need to prioritize information: Fig. 4 shows two plots containing a critical state that the operator should recognize, illustrated by the red marker. This marker is partially covered on the left by plotting a new marker that does not contain this information anymore. The operator can easily miss this information, so the monitor on the right prioritizes them and thus does not send the candidate marker to the visualization tool.

Again, we use RTLola ’s dynamic filtering mechanism to prevent the coverage of higher prioritized information. We also introduce new streams encoding the priority of information and checking the coverage of markers. In our running example, we need to change the stream expression of and add the following streams to the specification:

figure ab

The stream encodes the priority of a marker and the next stream stores the values of the last critical marker. The changed stream now also determines if a potential new marker would overlap the last critical marker by computing the distance between the markers with the help of and and then checking if this distance is sufficient.

Preventing covering markers with less relevant information is only one example of how we can encode the priority of information. Another example occurs when the specification aggregates data points to make the plots more readable: With aggregations, we lose information about the system. In general, this behavior is intended because the human supervisor cannot process all information from every sensor. In critical situations, however, the operator usually is focused on the part that misbehaves. In these situations, the monitor can switch to transferring each data point instead of aggregating them, or it might reduce the required difference for new markers, so the supervisor gets all the required information. Such a property can be expressed in RTLola by adapting the timing of a stream or by using different aggregation functions.

4 Conclusions

In this paper, we have introduced a principled approach to the real-time visualization of stream-based monitoring data. The key contributions are the novel design principle, which shifts the responsibility for the data preparation from the visualization component to the monitor; the organization of the approach into three major functional areas; and the solution of the visualization challenges with the mechanisms of a stream-based monitoring language.

Our practical experience with the approach of the paper has been very positive. We have used the approach to visualize stream-based monitoring data from recent case studies that use RTLola for UAS [2, 3] and other cyber-physical systems [4, 5, 11]. The visual tools provided by cgv have proven very useful for the type of data produced by our monitors. For example, we have visualized the failure of the GPS module in a drone, which was recognized by the system health check in the existing monitor specification, by adding a halo to the markers of the estimated position and by increasing the marker frequency. While clearly more research is needed in order to determine the best visualizations, our experience already indicates that this type of visualization is very helpful in quickly understanding complicated situations.

We hope that this paper will inspire other developers of runtime verification tools to invest in real-time visualization as well. We believe that our “monitoring-oriented” visualization approach provides a significant step towards meaningful visualizations that exploit the wealth of information available within the monitor. In future work, it might even be possible to integrate explicit visualization operators into monitoring languages like RTLola, and thus largely automate the visualization process presented in this paper.