1 Introduction

Safe planning and decision making under uncertainty are widely regarded as central challenges in enabling robots to successfully operate in real-world environments. By far the most common conceptual framework for addressing these challenges is to assign costs to stochastic outcomes and then to use the expected value of the resulting cost distribution as a quantity that “summarizes” the value of a decision. Such a quantity can then be optimized, or bounded within a constrained formulation. However, in settings where risk has to be accounted for, this choice is rarely well justified beyond the mathematical convenience it affords. For example, imagine a safety-critical application such as autonomous driving; would a passenger riding in an autonomous car be happy to do so if she was told that the average behavior of the car is not to crash? While one can introduce some degree of risk sensitivity (i.e., sensitivity to the tails of the cost distribution) in the expected cost framework by simply shaping the cost function, this can quickly turn into an exercise in “cost function hacking”. Unless one is careful about the way one shapes the cost function, this can lead to the robot behaving in an irrational manner [1]. The common alternative approach, aimed at promoting risk sensitivity, is to consider a worst-case assessment of the distribution of stochastic outcomes. In practice, however, such an assessment can often be quite conservative: an autonomous car whose goal is to never crash would never leave the garage.

The expected value operator and the worst-case assessment are examples of risk metrics. Informally, a risk metric is a mapping from a random variable corresponding to costs to a real number. The expected cost corresponds to risk neutrality while the worst-case assessment corresponds to extreme risk aversion. For practical applications, we would like to explore risk metrics that lie in between these extremes. This raises the following question: is there a class of risk metrics that lie between these extremes while still ensuring that the robot quantifies risk (and hence safety) in a rational and trustworthy manner? This question is central to the problem of decision making under uncertainty since the choice of a risk metric is one that must be made in any framework that assigns costs to outcomes. Yet, despite the role of safe decision making under uncertainty as a core theme in practically all areas of robotics, this question has received very little attention within the robotics community. As a result, there is arguably no firm theoretical foundation for making an informed decision about what risk metric to use for a given robotics application.

The goal of this paper is to provide a first step towards such a principled framework. More precisely, we describe axioms (properties) that a risk metric employed by a robot should satisfy in order to be considered sensible. To our knowledge, this is the first attempt to provide such an axiomatic framework for evaluating risk in robotics applications. Our effort is inspired by a similar effort in the finance community that led to the identification of coherent risk metrics [2, 13] as a class of risk metrics that have desirable properties for assessing the risk associated with a financial asset (e.g., a portfolio of stocks). The influence that these ideas have had can be gauged by the fact that in 2014 the Basel Committee on Banking Supervision changed its guidelines for banks to replace the Value at Risk (VaR) (a non-coherent risk metric) with the Conditional Value at Risk (CVaR) (a coherent risk metric) for assessing market risk [11]. We refer the reader to the extended version of our paper [8] for a more thorough discussion of approaches for quantifying risk in finance, along with related approaches in other fields.

We believe that the question of what properties a risk metric should satisfy in robotics applications is a fundamental one: a robot’s inability to assess risks in a rational way could lead to behavior that is harmful both to itself and humans or other autonomous agents around it. Our hope is that the ideas presented in this paper can help the community converge upon a set of properties that risk metrics must satisfy in order for the robot’s decision-making system to be considered rational and trustworthy, paralleling a similar effort in the financial industry. Indeed, it is conceivable that in the not-so-distant future, robots such as autonomous cars or unmanned aerial vehicles (UAVs) deployed in safety-critical scenarios will be subject to regulatory frameworks that mandate the use of “officially-approved” risk metrics.

Outline: The outline of this paper is as follows. Section 2 formally introduces the notion of a risk metric (Sect. 2.1) and proposes an interpretation of risk in robotics applications (Sect. 2.2). In Sect. 3 we advocate axioms that risk metrics in robotics applications should satisfy in order to be considered sensible (Sects. 3.1), provide examples of metrics that satisfy them, and discuss pitfalls stemming from using risk metrics that do not satisfy these axioms (Sect. 3.2). Section 4 concludes the paper with a number of points for discussion and directions for future research.

2 Assessing Risk: Preliminaries

In this section, we formally introduce risk metrics and propose an intuitive interpretation of risk quantification in robotics contexts. This interpretation will form the basis for the axioms we advocate in Sect. 3.

2.1 Risk Metrics

We denote the set of possible outcomes that may occur when a robot operates in uncertain settings as \(\varOmega \). In order to avoid heavy use of measure theoretic notions, we take \(\varOmega \) to be finite. Denote by \(\mathbb {P}\) a probability mass function that assigns probabilities \(\mathbb {P}(\omega )\) to outcomes \(\omega \in \varOmega \). Consider a cost function \(Z: \varOmega \rightarrow \mathbb {R}\) that assigns costs \(Z(\omega )\) to outcomes. The cost Z is then a random variable, namely the cost random variable. Let \(\mathscr {Z}\) denote the set of all random variables on \(\varOmega \). A risk metric is a mapping \(\rho : \mathscr {Z} \rightarrow \mathbb {R}\), i.e., a risk metric maps a cost random variable to a real number.

2.2 Interpretation of Risk in Robotics Applications

Imagine a fictional government agency known as the Robot Certification Agency (RCA) that is responsible for certifying if a given robot is safe to operate in the real world. How should the RCA quantify the risk faced by this robot? As an example, consider an autonomous car driving from one city to another. While performing this task, the robot will incur random costs Z (e.g., due to fuel consumption, time, crashes, mechanical wear and tear, etc.). In order to provide clear interpretations of the axioms for risk metrics discussed below, we will take the following axiom as a key starting point.

A0. Monetary costs. The costs Z are expressed in monetary terms.

This axiom ensures that the costs assigned to outcomes have a tangible and interpretable value, which will be instrumental in defining a meaningful notion of risk below. Such an axiom may also provide a handle on reasoning about insurance policies for safety-critical robots (e.g., autonomous cars). We note that our starting point contrasts with one where one considers a more abstract and subjective notion of cost (e.g., quadratic state and control costs for Linear Quadratic Regulator problems).

Given this assumption, suppose that the RCA demands that the robot’s owner must deposit an amount of money \(\rho (Z)\) before the robot is deployed such that the RCA is satisfied that the owner will be able to cover the potential costs incurred during operations (e.g., making repairs to the robot due to an accident) with the amount \(\rho \). We define the amount \(\rho (Z)\) as the perceived risk from operating the robot. The particular risk metric \(\rho : \mathscr {Z} \rightarrow \mathbb {R}\) the RCA uses will depend on its attitude towards risk and may depend on the application under consideration. For example, the RCA may ask for a deposit \(\rho (Z) = \mathbb {E}[Z]\) if it is risk neutral. If it wants to be highly conservative, the RCA may demand a deposit equal to the worst-case cost outcome. The question we will pursue in Sect. 3 is the following: what properties must \(\rho \) satisfy in order for it to be considered sensible?

We note that we are using the RCA here as a pedagogical tool to provide an interpretation of risk in robotics applications and to motivate the axioms described in Sect. 3. In reality, the robot’s decision-making system will assess risks and make decisions based on those assessments.

3 An Axiomatization of Risk Metrics for Robotics Applications

3.1 Axioms and Their Interpretations

We now parallel results in finance [3, Chap. 4] [13] and propose six axioms for risk metrics. Specifically, we make the case that these axioms should be fulfilled by any risk metric used in a robotics application in order for it to be considered a sensible assessment of risk. For each axiom, we first provide a formal statement and then an intuitive interpretation based on the interpretation of risk from Sect. 2.2.

A1. Monotonicity. Let \(Z, Z' \in \mathscr {Z}\) be two cost random variables. Suppose \(Z(\omega ) \le Z'(\omega )\) for all \(\omega \in \varOmega \). Then \(\rho (Z) \le \rho (Z')\).

Interpretation: If a random cost \(Z'\) is guaranteed to be greater than or equal to a random cost Z no matter what random outcome occurs, then \(Z'\) must be deemed at least as risky as Z. One can think of the random costs as corresponding to two different robots, or the same robot performing different tasks, or executing different controllers. Given our interpretation of risk, this axiom states that the RCA must demand at least as large a deposit for covering costs for the robot (or task) corresponding to \(Z'\) as for the robot (or task) corresponding to Z. This is a sensible requirement since we are guaranteed to incur at least as high a cost in the second scenario as the first no matter which outcome \(\omega \in \varOmega \) is realized.

A2. Translation invariance. Let \(Z \in \mathscr {Z}\) and \(c \in \mathbb {R}\). Then \(\rho (Z+c) = \rho (Z) + c\).

Interpretation: If one is charged a deterministic cost c (in addition to random costs incurred when the robot is operated), then the RCA should demand that this amount c be set aside in addition to money for covering the other costs from operating the robot. Note that this axiom also implies that \(\rho (Z - \rho (Z)) = 0\). Thus, \(\rho (Z)\) is the smallest amount that must be deducted from the costs to make the task risk-free.

A3. Positive homogeneity. Let \(Z \in \mathscr {Z}\) and \(\beta \ge 0\) be a scalar. Then \(\rho (\beta Z) = \beta \rho (Z)\).

Interpretation: If all the costs incurred by the robot (regardless of the random outcome) are scaled by \(\beta \), the RCA demands that the deposit is scaled commensurately. This is reasonable since this corresponds to simply changing the units of cost (recall that we assumed that the costs are expressed in monetary terms).

A4. Subadditivity. Let \(Z, Z' \in \mathscr {Z}\). Then \(\rho (Z + Z') \le \rho (Z) + \rho (Z')\).

Interpretation: This axiom encourages diversification of risk. For example, imagine a system with two robots. Suppose that Z and \(Z'\) are costs incurred by Robot 1 and Robot 2 respectively. The left-hand side (LHS) of the inequality corresponds to the deposit that the RCA demands when both robots are run simultaneously, while the right-hand side (RHS) corresponds to the sum of the deposits when the robots are deployed separately. Axiom A4 then states that deploying both robots simultaneously is at most as risky as deploying them separately.

This captures the intuition that one robot acts as a hedge against the other robot failing (i.e., if one of the robot fails in some way, then the other will make up for this loss). Another interpretation is that this axiom promotes redundancy in the system. The exact interpretation of the LHS and RHS of the inequality in A4 will depend on the particular application under consideration. For example, the two sub-costs Z and \(Z'\) may correspond to two separate sub-tasks that the robot must perform. In this case, the LHS corresponds to the robot performing both tasks simultaneously while the RHS corresponds to performing them independently. A4 encodes the intuition that performing both tasks simultaneously is less risky since one sub-task can act as a hedge against the other.

We note that A3 and A4 together imply convexity:

$$ \rho (\lambda Z + (1 - \lambda ) Z') \le \lambda \rho (Z) + (1 - \lambda ) \rho (Z'), \text { for all } \lambda \in [0,1]. $$

A5. Comonotone additivity. Suppose Z and \(Z'\) are comonotone, i.e., \((Z(\omega ) - Z(\omega '))(Z'(\omega ) - Z'(\omega ')) \ge 0\), \(\forall \) \((\omega ,\omega ') \in \varOmega \times \varOmega \). Then \(\rho (Z + Z') = \rho (Z) + \rho (Z')\).

Interpretation: This axiom supplements A4. In particular, A5 states that if two costs rise and fall together, then there is no benefit from diversifying (e.g., if one robot always performs poorly at a task when the other does or when a robot performs poorly at a sub-task when it also performs poorly at another one).

A6. Law invariance. If Z and \(Z'\) are identically distributed, then \(\rho (Z) = \rho (Z')\).

Interpretation: If two tasks have the same distribution of costs, then the RCA demands an equal deposit in both cases. For example, suppose \(\varOmega = \{\omega , \omega ' \}\) with both outcomes having probability 0.5. Further, suppose \(Z(\omega ) = 1\), \(Z(\omega ') = 10\), \(Z'(\omega ) = 10\), \(Z'(\omega ') = 1\). The two situations must be considered equally risky even though the assignment of costs to events is different.

Taken together, Axioms A1–A6 capture a fairly exhaustive set of essential properties that we believe any reasonable quantification of risk in robotics should obey given our interpretation of risk in Sect. 2.2. A hypothetical RCA that quantifies risk in a manner that is consistent with these axioms would be considered a sensible one. Moreover, robots that assess risks according to risk metrics that fail to satisfy some of these axioms can behave in a manner that would be considered extremely unreasonable and arguably very unsafe, as illustrated in Sect. 3.2. We thus advocate risk metrics that satisfy Axioms A1–A6 for use in robotics applications. We note that risk metrics satisfying Axioms A1–A6 have also been studied in the context of portfolio optimization in finance and are known as distortion risk metrics [3, 14]. These metrics enjoy an elegant characterization in terms of the CVaR metric. We refer the reader to [8] for a discussion of this characterization.

We note, however, that the axioms highlighted here only form a starting point for a complete axiomatization of risk metrics in robotics applications. For example, the sequential nature of many decision-making tasks in robotics gives rise to additional important considerations beyond the ones discussed here. We refer the reader to [8] for a discussion of these additional axioms.

3.2 Examples and Pitfalls of Commonly Used Risk Metrics

In this section, we first discuss examples of existing risk metrics that fulfill Axioms A1–A6. We then discuss commonly used risk metrics that do not fulfill some of these axioms, along with pitfalls stemming from their use.

An important risk metric that satisfies Axioms A1–A6 is the Conditional Value at Risk (CVaR) [12]. The \(\text {CVaR}_\alpha \) for a random cost Z at level \(\alpha \) is defined as:

$$\begin{aligned} \text {CVaR}_{\alpha } (Z) := \frac{1}{\alpha } \int _{1-\alpha }^{1} \text {VaR}_{1-\tau } (Z) \ d \tau , \end{aligned}$$
(1)

where \(\text {VaR}_\alpha (Z)\) is the Value at Risk (VaR) at level \(\alpha \), i.e., simply the \((1-\alpha )\)-quantile of the cost random variable Z:

$$\begin{aligned} \text {VaR}_\alpha (Z) := \min \{z \ | \ \mathbb {P}[Z > z] \le \alpha \}. \end{aligned}$$
(2)

Intuitively, \(\text {CVaR}_{\alpha }\) is the expected value of Z in the conditional distribution of Z’s upper \((1-\alpha )\)-tail. It can thus be interpreted as a risk metric that quantifies “how bad is bad.” We note that the expected cost and worst-case assessment also satisfy A1–A6. Figure 1 provides a visualization of the expected cost, worst case, VaR, and CVaR. Axioms A1–A6 thus define a broad class of risk metrics that capture a wide spectrum of risk assessments from risk-neutral to worst-case.

However, there are many examples of popular risk metrics that do not fulfill the axioms we advocate. For example, a very popular metric to quantify risk in robotics applications is the mean-variance risk metric: \(\mathbb {E}[Z] + \beta \text {Variance}[Z]\) (see, e.g., [4, 6, 10]). The mean-variance metric satisfies A6 but fails to satisfy the other axioms. This can lead to a robot that utilizes the mean-variance metric making decisions that would be considered unreasonable. Consider the setup in Table 1 (based on [7]), where \(\omega _1,\omega _2,\omega _3,\omega _4\) are disturbance outcomes, and Z and \(Z'\) are the costs resulting from executing two different controllers \(\pi \) and \(\pi '\). Which controller should the robot execute? Controller \(\pi \) results in lower costs no matter what the disturbance outcome is and hence should be preferred by any sensible decision maker. However, computing the mean-variance risk metric with \(\beta = 1\), we see that:

$$\mathbb {E}[Z] + \text {Variance}[Z] = 3.75 > \mathbb {E}[Z'] + \text {Variance}[Z'] = 3.4375.$$

The robot would hence strictly prefer \(\pi '\). This unreasonable behavior is a result of the mean-variance risk metric failing to satisfy Axiom A1 (monotonicity).

Fig. 1
figure 1

An illustration of four important risk metrics, namely: expected cost, worst case, Value at Risk (VaR), and Conditional Value at Risk (CVaR)

Full size image
Table 1 Issues with the mean-variance risk metric. Any rational agent would choose controller \(\pi \) (with associated costs Z) since it always results in lower costs. But, using the mean-variance risk metric results in choosing \(\pi '\) (with associated costs \(Z'\))
Full size table

We note that other commonly used risk metrics in robotics also suffer from similar issues. We refer the reader to [8] for a discussion of the pitfalls of using the Value at Risk (VaR) (defined in Eq. (2)), which fails to satisfy the subadditivity axiom (A4). Table 2 lists the axioms satisfied by popular risk metrics in the literature (we refer the reader to [13] for definitions). We note that the standard semi-deviation is widely used in finance [13], while the entropic risk metric has been popular in control theory for risk-averse control [5, 15].

To summarize our discussion, based on our arguments in this section, we advocate the use of risk metrics satisfying Axioms A1–A6 (i.e., distortion risk metrics) in robotics applications. This is in contrast to popular risk metrics used in the robotics literature (e.g., mean-variance, or VaR).

4 Discussion and Conclusions

Our goal in this paper has been to provide preliminary directions towards an axiomatic theory of risk for robotics applications. We have advocated properties that risk metrics employed by robots should satisfy in order for them to be considered sensible. These axioms define a class of risk metrics, known as distortion risk metrics, which have been previously used in finance. We end with some questions that highlight areas for future thought.

Discussion 1

(Axioms A1–A6) The axioms of monotonicity (A1), translation invariance (A2), positive homogeneity (A3), and law invariance (A6) should arguably be applicable in any robotics application. Subadditivity (A4) and comonotone additivity (A5) are also intuitively appealing, particularly for applications that involve some degree of high-level decision making (since a high-level decision making system should ideally diversify risks). However, the interpretation of diversification is somewhat unclear in certain applications. For example, imagine a humanoid robot whose goal is to minimize a cost function that is a sum of two components Z and \(Z'\), where Z penalizes one aspect of the robot’s motion (e.g., deviations of the robot’s torso from the vertical orientation) while \(Z'\) penalizes another aspect (e.g., deviation of the robot’s gaze from a target). For such low-level control tasks, the interpretation of \(\rho (Z) + \rho (Z')\) is not entirely clear since it is not possible to perform the different subtasks corresponding to Z and \(Z'\) independently of each other. The following is thus a question for discussion and future work: should A4 and A5 be abandoned (or replaced by other axioms) for such tasks?

Table 2 Axioms satisfied by popular risk metrics
Full size table

Discussion 2

(Further axioms) While we have highlighted a number of axioms that we believe are particularly important, the identification of other axioms is an important direction for future work. These may depend on the particular domain of application. Moreover, for certain applications it may not be necessary to impose all the axioms described here. For example, A4 and A5 (concerning diversification of risks) will generally be relevant to high-level decision making tasks where it is possible to diversify risks and may not be relevant for low-level control tasks where diversification may not be possible.

Discussion 3

(Choosing a particular risk metric) For a given application, we may wish to choose a particular risk metric from the class of metrics described here. How should such a metric be chosen? One possibility is to learn a distortion risk metric that explains how humans evaluate risk in the given application domain and then employ the learned risk metric. We describe first steps towards this in [9], where we have introduced a framework for risk-sensitive inverse reinforcement learning for learning humans’ risk preferences from the class of coherent risk metrics.

Discussion 4

(Legal frameworks) The question of safety for AI systems has received significant attention recently (see, e.g., [1] for a recent review). An important component of this discussion has been the consideration of legal frameworks and guidelines that must be placed on AI systems to ensure that they do not pose a threat to our safety. Such considerations for robots such as unmanned aerial vehicles are already extremely pressing for government agencies such as the Federal Aviation Administration (FAA). It is not difficult to imagine a future where the Robot Certification Agency (RCA) is in fact a real entity that certifies the safety of new robotic systems. How can we effectively engage lawmakers and government officials in discussions on how to evaluate risks in robotic applications?

Our hope is that the ideas presented in this paper will spur further work on this topic and eventually lead to a convergence upon a particular class of risk metrics that form the standard for assessing risk in robotics.