Optimising Data-Driven Safety Related Systems

Abstract

The operation of many safety related systems is dependent upon a number of interacting parameters. Frequently these parameters must be ‘tuned’ to the particular operating environment to provide the best possible performance. We focus on the Short Term Conflict Alert (STCA) system, which warns of airspace infractions between aircraft, as an example of a safety related system that must raise an alert to dangerous situations, but should not raise false alarms. Current practice is to ‘tune’ by hand the many parameters governing the system in order to optimise the operating point in terms of the true positive and false positive rates, which are frequently associated with highly imbalanced costs.

We regard the tuning of safety related systems as a multi-objective optimisation problem. We show how a region of the optimal receiver operating characteristic (ROC) curve may be obtained, permitting the system operators to select the operating point. We apply this methodology to the STCA system, showing that we can improve upon the current hand-tuned operating point, as well as providing the salient ROC curve describing the true positive versus false positive trade-off. We also address the robustness of the optimal ROC curve to perturbations of the data used to learn it. Bootstrap resampling is used to evaluate the uncertainty in the optimal operating curve and show how the probability of a particular operating point can be estimated.