Towards a Framework for Comparing the Complexity of Robotic Tasks

Abstract

We are motivated by the problem of comparing the complexity of one robotic task relative to another. To this end, we define a notion of reduction that formalizes the following intuition: Task 1 reduces to Task 2 if we can efficiently transform any policy that solves Task 2 into a policy that solves Task 1. We further define a quantitative measure of the relative complexity between any two tasks for a given robot. We prove useful properties of our notion of reduction (e.g., reflexivity, transitivity, and antisymmetry) and relative complexity measure (e.g., nonnegativity and monotonicity). In addition, we propose practical algorithms for estimating the relative complexity measure. We illustrate our framework for comparing robotic tasks using (i) examples where one can analytically establish reductions, and (ii) reinforcement learning examples where the proposed algorithm can estimate the relative complexity between tasks.

M. Ho and A. Farid—Equal contribution.

Appendices

Appendix

A Proof of Properties

Proposition

1(Task Reduction is a Non-Strict Partial Ordering Relation). Suppose that $\forall \ (\tau _\xi , \tau _\zeta ) \in \mathcal {T}^2$, $H_{\xi , \zeta }$ and $G_{\zeta , \xi }$ include the identity and are closed under composition on $\mathcal {T}$. Then, task reductions satisfy the following properties and thus define a non-strict partial ordering relation.

Property

1.a. Reflexivity: $\tau _1 \preceq \tau _1$.

Property

1.b. Antisymmetry: $\tau _1 \prec \tau _2 \implies \lnot (\tau _2 \preceq \tau _1)$, where $\tau _1 \prec \tau _2$ is defined as $(\tau _1 \preceq \tau _2) \wedge \lnot (\tau _1 \equiv \tau _2)$.

Property

1.c. Transitivity: $(\tau _1 \preceq \tau _2) \wedge (\tau _2 \preceq \tau _3) \implies \tau _1 \preceq \tau _3$.

Proof

Property 1.a: $\tau _1 \preceq \tau _1 \implies \exists \ g \in G_{1,1}$, $h \in H_{1,1}$ such that

$$\begin{aligned} g \circ \pi ^\star _1 \circ h \in \varPi ^\star _1. \end{aligned}$$

(6)

If g and h are the identity function, then $g \circ \pi ^\star _1 \circ h = \pi ^\star _1 \ \forall \ \pi ^\star _1 \in \varPi ^\star _1$. Thus, $\tau _1 \preceq \tau _1$ when $G_{1,1}$ and $H_{1,1}$ include their respective identity functions.

Property 1.b: Suppose $\tau _1 \prec \tau _2$ and thus $(\tau _1 \preceq \tau _2) \wedge \lnot (\tau _1 \equiv \tau _2)$. Note that $\lnot (\tau _1 \equiv \tau _2) \implies \lnot \big ((\tau _1 \preceq \tau _2) \wedge (\tau _2 \preceq \tau _1) \big ) \implies \lnot (\tau _1 \preceq \tau _2) \vee \lnot (\tau _2 \preceq \tau _1)$. We assumed $(\tau _1 \preceq \tau _2)$, so we must have that $\lnot (\tau _2 \preceq \tau _1)$.

Property 1.c: Suppose $\tau _1 \preceq \tau _2$ and $\tau _2 \preceq \tau _3$. By Definition 4, $\exists \ g_1 \in G_{2,1}$, $g_2 \in G_{3,2}$, $h_1 \in H_{1,2}$, and $h_2 \in H_{2,3}$ such that $g_1 \circ \pi ^\star _2 \circ h_1 \in \varPi ^\star _1 \ \forall \ \pi ^\star _2 \in \varPi ^\star _2$ and $g_2 \circ \pi ^\star _3 \circ h_2 \in \varPi ^\star _2 \ \forall \ \pi ^\star _3 \in \varPi ^\star _3$. Consider

$$\begin{aligned} \underbrace{g_1 \circ \overbrace{g_2 \circ \pi ^\star _3 \circ h_2}^{\in \varPi ^\star _2} \circ h_1}_{\in \varPi ^\star _1} \end{aligned}$$

(7)

for all $\pi ^\star _3 \in \varPi ^\star _3$. Let $g_3 := g_1 \circ g_2$ and $h_3 := h_2 \circ h_1$ so that $g_3 \circ \pi ^\star _3 \circ h_3 \in \varPi ^\star _1$ for all $\pi ^\star _3 \in \varPi ^\star _3$. If $G_{3,1}$ and $H_{1,3}$ are closed under composition on $\mathcal {T}$, then $g_3 \in G_{3,1}$ and $h_3 \in H_{1,3}$ and $\tau _1 \preceq \tau _3$. Thus, task reductions are transitive if $G_{3,1}$ and $H_{1,3}$ are closed under composition on $\mathcal {T}$.    $\square$

Proposition 6

(Strict Task Reduction is a Strict Partial Ordering Relation). Suppose that $\forall \ (\tau _\xi , \tau _\zeta ) \in \mathcal {T}^2$, $H_{\xi , \zeta }$ and $G_{\zeta , \xi }$ include the identity and are closed under composition on $\mathcal {T}$. Then, strict task reductions satisfy the following properties and thus define a strict partial ordering relation.

Property

6.a. Irreflexivity: $\lnot (\tau _1 \prec \tau _1)$.

Property

6.b. Asymmetry: $\tau _1 \prec \tau _2 \implies \lnot (\tau _2 \prec \tau _1)$.

Property

6.c. Transitivity: $\tau _1 \prec \tau _2 \wedge \tau _2 \prec \tau _3 \implies \tau _1 \prec \tau _3$.

Proof

Property 6.a: Suppose $\tau _1 \prec \tau _1 \implies \tau _1 \preceq \tau _1 \wedge \lnot (\tau _1 \equiv \tau _1)$. $\tau _1 \equiv \tau _1$ by Property 2.b. $\Rightarrow \!\Leftarrow \implies \lnot (\tau _1 \prec \tau _1)$ when $H_{1,1}$ and $G_{1,1}$ include their respective identity functions.

Property 6.b: $\tau _1 \prec \tau _2 \implies \lnot (\tau _2 \preceq \tau _1)$ by Property 1.b, since $\tau _1 \prec \tau _2 \implies \lnot (\tau _1 \equiv \tau _2)$. $\lnot (\tau _2 \preceq \tau _1) \iff \lnot (\tau _2 \preceq \tau _1) \vee \tau _2 \equiv \tau _1 \implies \lnot \big (\tau _2 \preceq \tau _1 \wedge \lnot (\tau _2 \equiv \tau _1)\big ) \implies \lnot (\tau _2 \prec \tau _1).$

Property 6.c: $\tau _1 \prec \tau _2 \wedge \tau _2 \prec \tau _3 \implies \tau _1 \preceq \tau _2 \wedge \tau _2 \preceq \tau _3 \wedge \lnot (\tau _1 \equiv \tau _2) \wedge \lnot (\tau _2 \equiv \tau _3) \implies \tau _1 \preceq \tau _3 \wedge \lnot (\tau _1 \equiv \tau _3)$ by Properties 1.c and 2.c. $\implies \tau _1 \prec \tau _3$ when $H_{1,3}$ and $G_{3,1}$ are closed under composition on $\mathcal {T}$.    $\square$

Proposition

2(Task Equivalence is an Equivalence Relation). Suppose that $\forall \ (\tau _\xi , \tau _\zeta ) \in \mathcal {T}^2$, $H_{\xi , \zeta }$ and $G_{\zeta , \xi }$ include the identity and are closed under composition on $\mathcal {T}$. Then, task equivalence satisfies the following properties and thus defines an equivalence relation.

Property

2.a. Reflexivity: $\tau _1 \equiv \tau _1$.

Property

2.b. Symmetry: $\tau _1 \equiv \tau _2 \implies \tau _2 \equiv \tau _1$.

Property

2.c. Transitivity: $\tau _1 \equiv \tau _2 \wedge \tau _2 \equiv \tau _3 \implies \tau _1 \equiv \tau _3$.

Proof

Property 2.a: $\tau _1 \equiv \tau _1 \implies \tau _1 \preceq \tau _1$ by Property 1.a when $G_{1,1}$ and $H_{1,1}$ include the identity. Thus, task equivalence is reflexive if $G_{1,1}$ and $H_{1,1}$ include the identity.

Property 2.b: $\tau _1 \equiv \tau _2 \implies (\tau _1 \preceq \tau _2) \wedge (\tau _2 \preceq \tau _1)$ by Definition 5 $\implies (\tau _2 \preceq \tau _1) \wedge (\tau _1 \preceq \tau _2)$ $\implies \tau _2 \equiv \tau _1$.

Property 2.c: $(\tau _1 \equiv \tau _2) \wedge (\tau _2 \equiv \tau _3) \implies (\tau _1 \preceq \tau _2) \wedge (\tau _2 \preceq \tau _3) \wedge (\tau _3 \preceq \tau _2) \wedge (\tau _2 \preceq \tau _1)$ by Definition 5. $(\tau _3 \preceq \tau _2) \wedge (\tau _2 \preceq \tau _1) \implies (\tau _3 \preceq \tau _1)$ by Property 1.c when $G_{3,1}$ and $H_{1,3}$ are closed under composition on $\mathcal {T}$. Similarly, $(\tau _1 \preceq \tau _2) \wedge (\tau _2 \preceq \tau _3) \implies (\tau _1 \preceq \tau _3)$. Thus $(\tau _1 \preceq \tau _3) \wedge (\tau _3 \preceq \tau _1) \implies \tau _1 \equiv \tau _3$. Thus task equivalence is transitive if $G_{3,1}$ and $H_{1,3}$ are closed under composition on $\mathcal {T}$.    $\square$

Proposition

5(Properties of the Relative Complexity). Relative Complexity satisfies the following properties:

Property

5.a. Nonnegativity and boundedness: $C_{\tau _1 / \tau _2} \in [0,1]$.

Property

5.b. Monotonicity with respect to H and G: If $H \subseteq H'$ and $G \subseteq G'$, then $C_{\tau _1 / \tau _2}(H', G') \preceq C_{\tau _1 / \tau _2}(H, G)$.

Assume that the supremum and infimum in Definition 7 are attained by functions in $\varPi ^\star _2, H, G$. Then:

Property

5.c. Equivalence between reduction and 0 relative complexity: $C_{\tau _1 / \tau _2} = 0 \iff \tau _1 \preceq \tau _2$.

Property

5.d. Equivalence between no reduction and positive relative complexity: $C_{\tau _1 / \tau _2} \in (0,1] \iff \lnot (\tau _1 \preceq \tau _2)$.

Proof

Property 5.a: $R_1(g \circ \pi ^\star _2 \circ h) \in [0, R^\star _1]$. Therefore, $R_1(g \circ \pi ^\star _2 \circ h) / R^\star _1 \in [0, 1] \implies C_{\tau _1/\tau _2} \in [0, 1]$ for any H, G.

Property 5.b: Consider $H, H'$ such that $H \subseteq H'$ and $G, G'$ such that $G \subseteq G'$. For any function f, the following is true $\forall \pi _2^\star$:

$$\begin{aligned} \inf _{h\in H', g\in G'} f(h, g, \pi ^\star _2) \le \inf _{h\in H, g\in G} f(h,g, \pi ^\star _2). \end{aligned}$$

(8)

This implies the following:

$$\begin{aligned} \sup _{\pi _2 \in \varPi ^\star _2} \inf _{h\in H', g\in G'} \bigg [1 - \frac{R_1(g\circ \pi ^\star _2 \circ h)}{R^\star _1} \bigg ] \le \sup _{\pi ^\star _2 \in \varPi ^\star _2} \inf _{h\in H, g\in G} \bigg [1 - \frac{R_1(g\circ \pi ^\star _2 \circ h)}{R^\star _1} \bigg ]. \end{aligned}$$

(9)

Property 5.c: Assume $C_{\tau _1 / \tau _2} = 0$ for some H and G $\iff$ for any $\pi ^\star _2 \in \varPi ^\star _2 \ \exists \ g\in G$ and $h \in H$ such that $R_1(g\circ \pi ^\star _2 \circ h) = R^\star _1$. $R_1(\pi _1) = R^\star _1$ $\iff$ $\pi _1 \in \varPi ^\star _1$. Thus, for all $\pi ^\star _2 \in \varPi ^\star _2 \ \exists \ g \in G$ and $h \in H$ such that $g\circ \pi ^\star _2 \circ h \in \varPi ^\star _1$ $\iff \tau _1 \preceq \tau _2$.

Property 5.d: The contrapositive of Property 5.c is $\lnot (\tau _1 \preceq \tau _2) \iff C_{\tau _1 / \tau _2} \ne 0$. By Property 5.a, the complexity measure is $C_{\tau _1 / \tau _2} \in [0,1]$, therefore, $C_{\tau _1 / \tau _2} \in (0,1] \iff C_{\tau _1 / \tau _2} \ne 0$. Thus $C_{\tau _1 / \tau _2} \in (0,1] \iff \lnot (\tau _1 \preceq \tau _2)$.    $\square$

B Additional Experimental Details

Approximating Relative Complexity using Q-learning. We apply Q-learning to Algorithm 1 by letting the loss functions $L_1$ and $L_2$ correspond to a Q-learning loss: $L_\xi (\pi _\xi ) = -\frac{1}{B}\sum _{b=1}^{B}[Q^{\pi _\xi }(s_b,a_b) \log p(a_b)]$, where $p(a_b)$ corresponds to the probability of an action for policy $\pi _\xi$ (which may be a transformation of another policy such as $\pi _\xi = g \circ \pi _{\zeta } \circ h$), $Q^{\pi _\xi }(s_b,a_b)$ are the Q-values, and B is the batch size. We run Algorithm 1 for 1000 iterations and use a batch size B of 1000 transitions.

Approximating Relative Complexity using SAC. We modify Algorithm 1 to use SAC for approximating the relative complexity. Let $Q_2^{\pi _2}$ be a critic of $\pi _2$ on task $\tau _2$ and $Q_1^{g \circ \pi _2 \circ h}$ be a critic of $g \circ \pi _2 \circ h$ on task $\tau _1$. We add an additional step to the algorithm for updating the critics on task $\tau _1$ and $\tau _2$. The critics are then used in the updates for the policy $\pi _2$ and the encoder/decoder. The resulting method is presented in Algorithm 2. We run Algorithm 2 for 50,000 iterations and use a batch size of 200 transitions.