Performance Limit Evaluation Strategy for Automated Driving Systems

Abstract

Efficient detection of performance limits is critical to autonomous driving. As autonomous driving is difficult to be realized under complicated scenarios, an improved genetic algorithm-based evolution test is proposed to accelerate the evaluation of performance limits. It conducts crossover operation at all positions and mutation several times to make the high-quality chromosome exist in candidate offspring easily. Then the normal offspring is selected statistically based on the scenario complexity, which is designed to measure the difficulty of realizing autonomous driving through the Analytic Hierarchy Process. The benefits of modified cross/mutation operators on the improvement of scenario complexity are analyzed theoretically. Finally, the effectiveness of improved genetic algorithm-based evolution test is validated after being applied to evaluate the collision avoidance performance of an automatic parallel parking system.

Appendix 1 Definitions and Proofs

1.1 Definition of Symbols

${g}_{i}$: The objective function value corresponding to ${{\varvec{T}}}_{i}$;

${g}_{\mathrm{th}}$: The threshold of the objective function value;

G: The generation of population;

${G}_{\mathrm{th}}$: The maximum number of interaction test process;

${{\varvec{P}}}_{i}^{\mathrm{C}}$: The i-th parent individual pair in the natural selected population ${{\varvec{X}}}^{\mathrm{S}}$ from ${{\varvec{X}}}_{G}$;

${{\varvec{P}}}_{i,j}^{\mathrm{C}}$: The offspring pair generated by single-point crossover at the j-th position of ${{\varvec{P}}}_{i}^{\mathrm{C}}$;

${{\varvec{P}}}_{i}^{\mathrm{C}\boldsymbol{*}}$: The selected offspring pair from ${{\varvec{P}}}_{i,j}^{\mathrm{C}},\boldsymbol{ }j=1,\cdots ,L$, where $L$ is the number of genes;

${{\varvec{T}}}^{*}$: The best test case evaluated by ${g}_{i}$, and its objective function value is ${g}^{*}$;

${{\varvec{X}}}_{G}=\left\{{{\varvec{T}}}_{i},i=1,\cdots ,2m\right\}$: The G-th population compose of 2 m individuals;

${{\varvec{X}}}^{\mathrm{C}}=\left\{{{\varvec{P}}}_{1}^{\mathrm{C}*},\cdots ,{{\varvec{P}}}_{m}^{\mathrm{C}*}\right\}$: The offspring population generated by crossover operation;

${{\varvec{X}}}_{i}^{\mathrm{M}}$: The population generated by conducting the i-th mutation on ${{\varvec{X}}}^{\mathrm{C}}$;

${{\varvec{X}}}^{\mathrm{M}*}$: The selected population from ${{\varvec{X}}}_{i}^{\mathrm{M}},\boldsymbol{ }i=1,\cdots ,N$, where $N$ is the mutation times.

$\overline{D}\left( {\mathbf{X}} \right) = \frac{1}{n}\sum\limits_{{i{ = 1}}}^{n} {D\left( {{\varvec{T}}_{i} } \right)}$: The average complexity of a population ${\mathbf{X}} = \left\{ {{\varvec{T}}_{1} ,{\varvec{T}}_{2} , \ldots ,{\varvec{T}}_{n} } \right\}$;

${P}_{\mathrm{r}}\left(\bullet \right)$: The occurrence probability of an event.

1.2 Proof of Theorem 1

From Eqs. (4) and (5):

$$E\left(\overline{D }\left(C\left({{\varvec{P}}}_{i}^{\mathrm{C}}\right)\right)\right)-E\left(\overline{D }\left(\widehat{C}\left({{\varvec{P}}}_{i}^{\mathrm{C}}\right)\right)\right)=\sum_{j=1}^{L}\left({D}_{i,j}^{\mathrm{C}}-\frac{{p}_{\mathrm{C}}}{L}\right)\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)-(1-{p}_{\mathrm{C}}){\overline{D }({\varvec{P}}}_{i}^{\mathrm{C}}),$$

(11)

where $D_{i,j}^{C} = e^{{d \times \overline{D}\left( {P_{i,j}^{C} } \right)}} /\sum\limits_{k = 1}^{L} {e^{{d \times \overline{D}\left( {P_{i,j}^{C} } \right)}} }$ is the normalized complexity of ${\varvec{P}}_{i,j}^{C}$. According to whether the full crossover can increase the complexity of offspring, Eq. (11) can be re-written as

$$E\left(\overline{D }\left(C\left({{\varvec{P}}}_{i}^{\mathrm{C}}\right)\right)\right)-E\left(\overline{D }\left(\widehat{C}\left({{\varvec{P}}}_{i}^{\mathrm{C}}\right)\right)\right)=\sum_{j\in {\Omega }_{+}^{\mathrm{C}}}\left({D}_{i,j}^{\mathrm{C}}-\frac{{p}_{\mathrm{C}}}{L}\right)\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)+\sum_{j\in {\Omega }_{-}^{\mathrm{C}}}\left({D}_{i,j}^{\mathrm{C}}-\frac{{p}_{\mathrm{C}}}{L}\right)\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)-(1-{p}_{\mathrm{C}}){\overline{D }({\varvec{P}}}_{i}^{\mathrm{C}})\ge \underset{j\in {\Omega }_{+}^{\mathrm{C}}}{\mathrm{min}}\left(\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)\right)\sum_{j\in {\Omega }_{+}^{C}}\left({D}_{i,j}^{\mathrm{C}}-\frac{{p}_{\mathrm{C}}}{L}\right)+\underset{j\in {\Omega }_{-}^{\mathrm{C}}}{\mathrm{max}}\left(\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)\right)\sum_{j\in {\Omega }_{-}^{C}}\left({D}_{i,j}^{\mathrm{C}}-\frac{{p}_{\mathrm{C}}}{L}\right)-(1-{p}_{\mathrm{C}}){\overline{D }({\varvec{P}}}_{i}^{\mathrm{C}})$$

(12)

where ${\Omega }_{+}^{\mathrm{C}}=\left\{j|{D}_{i,j}^{\mathrm{C}}\ge \frac{{p}_{\mathrm{C}}}{L}\right\}$ and ${\Omega }_{-}^{\mathrm{C}}=\left\{j|j=1,\cdots ,L\right\}-{\Omega }_{+}^{\mathrm{C}}$.

From Eq. (5) and the definition of ${D}_{i,j}^{\mathrm{C}}$:

$$\sum_{j=1}^{L}\frac{{p}_{\mathrm{C}}}{L}+1-{p}_{\mathrm{C}}=1 \mathrm{and }\sum_{j=1}^{L}{D}_{i,j}^{\mathrm{C}}=1\Rightarrow \sum_{j\in {\Omega }_{+}^{\mathrm{C}}}\left({D}_{i,j}^{\mathrm{C}}-\frac{{p}_{\mathrm{C}}}{L}\right)+\sum_{j\in {\Omega }_{-}^{\mathrm{C}}}\left({D}_{i,j}^{\mathrm{C}}-\frac{{p}_{\mathrm{C}}}{L}\right)-\left(1-{p}_{\mathrm{C}}\right)=0.$$

(13)

when ${p}_{\mathrm{C}}=1$, the following equation establishes from Eq. (13):

$$\sum_{j\in {\Omega }_{-}^{\mathrm{C}}}\left({D}_{i,j}^{\mathrm{C}}-\frac{1}{L}\right)=-\sum_{j\in {\Omega }_{+}^{\mathrm{C}}}\left({D}_{i,j}^{\mathrm{C}}-\frac{1}{L}\right)$$

(14)

Then substituting Eq. 14 and ${p}_{\mathrm{C}}=1$ to Eq. (12) yields

$$E\left(\overline{D }\left(C\left({{\varvec{P}}}_{i}^{\mathrm{C}}\right)\right)\right)-E\left(\overline{D }\left(\widehat{C}\left({{\varvec{P}}}_{i}^{\mathrm{C}}\right)\right)\right)\ge \left[\underset{j\in {\Omega }_{+}^{\mathrm{C}}}{\mathrm{min}}\left(\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)\right)\right.-\left.\underset{j\in {\Omega }_{-}^{\mathrm{C}}}{\mathrm{max}}\left(\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)\right)\right]\sum_{j\in {\Omega }_{+}^{\mathrm{C}}}\left({D}_{i,j}^{\mathrm{C}}-\frac{1}{L}\right)$$

(15)

According to the definition of ${\Omega }_{+}^{\mathrm{C}}$ and ${\Omega }_{-}^{\mathrm{C}}$:

$${D}_{i,j}^{\mathrm{C}}-\frac{1}{L}\ge 0, \forall j\in {\Omega }_{+}^{\mathrm{C}}\,\mathrm{ and }\,\underset{j\in {\Omega }_{-}^{\mathrm{C}}}{\mathrm{max}}\left(\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)\right)\frac{1}{d}\mathrm{ln}\left(\frac{{p}_{\mathrm{C}}}{L}\sum_{k=1}^{L}{\mathrm{e}}^{d\times \overline{D }\left({{\varvec{P}}}_{i,k}^{\mathrm{C}}\right)}\right)\le \underset{j\in {\Omega }_{+}^{\mathrm{C}}}{\mathrm{min}}\left(\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)\right)$$

(16)

Conclusion C1 is proved by substituting Eq. (16) to Eq. (15).

When $0\le {p}_{\mathrm{C}}<1$, the following equation is deduced by substituting Eq. (13) to Eq. (11):

$$E\left(\overline{D }\left(C\left({{\varvec{P}}}_{i}^{\mathrm{C}}\right)\right)\right)-E\left(\overline{D }\left(\widehat{C}\left({{\varvec{P}}}_{i}^{\mathrm{C}}\right)\right)\right)=\sum_{j=1}^{L}{D}_{i,j}^{\mathrm{C}}\left(\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)-{\overline{D }({\varvec{P}}}_{i}^{\mathrm{C}})\right) -\frac{{p}_{\mathrm{C}}}{L}\sum_{j=1}^{L}\left(\overline{D }\left({{\varvec{P}}}_{i,j}^{\mathrm{C}}\right)-{\overline{D }({\varvec{P}}}_{i}^{\mathrm{C}})\right)$$

(17)

Equation (17) is monotonically decreasing with ${p}_{\mathrm{C}}$ as the variable when Eq. (7) establishes, and so C2 establishes with the conclusion derived from C1.

1.3 Proof of Lemma 1

Since the N times canonical mutations are independent, the following equations are obtained:

$${P}_{\mathrm{r}}\left(\tilde{M }\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)=\sum_{i=0}^{N}\left[\frac{i{\mathrm{C}}_{N}^{i}}{N}{\mathrm{P}}_{\mathrm{r}}^{i}\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)\right.\times \left.{\left(1-{P}_{\mathrm{r}}\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)\right)}^{N-i}\right]={P}_{\mathrm{r}}\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)\times \sum_{i=1}^{N-1}\left[{\mathrm{C}}_{N-1}^{i-1}{\mathrm{P}}_{\mathrm{r}}^{i-1}\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)\right.\times \left.{\left(1-{P}_{\mathrm{r}}\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)\right)}^{N-i}\right]$$

(18)

where ${\mathrm{C}}_{N}^{i}=\frac{N!}{i!\left(N-i\right)!}$ denotes the combination number.

According to the binomial expansion

$$\sum_{i=1}^{N-1}\left[{\mathrm{C}}_{N-1}^{i-1}{\mathrm{P}}_{\mathrm{r}}^{i-1}\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)\right.\times \left.{\left(1-{P}_{\mathrm{r}}\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)\right)}^{N-i}\right]=\left[{P}_{\mathrm{r}}\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)+1\right.{\left.-{P}_{\mathrm{r}}\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)={{\varvec{X}}}^{\mathrm{M}*}\right)\right]}^{N-1}=1$$

(19)

Lemma 1: Is proved by substituting Eq. (19) to Eq. (18).

1.4 Proof of Theorem 2

According to Step 6 and Lemma 1:

$$E\left(\overline{D }\left(M\left({{\varvec{X}}}^{\mathrm{C}}\right)\right)\right)-E\left(\overline{D }\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)\right)\right)=\sum_{i=1}^{N}\left({D}_{i}^{\mathrm{M}}-\frac{1}{N}\right)\times \overline{D }\left({{\varvec{X}}}_{i}^{\mathrm{M}}\right)$$

(20)

Being similar to the analysis procedure of Eq. (12), Eq. (20) is re-written as

$$E\left(\overline{D }\left(M\left({{\varvec{X}}}^{\mathrm{C}}\right)\right)\right)-E\left(\overline{D }\left(\widehat{M}\left({{\varvec{X}}}^{\mathrm{C}}\right)\right)\right)\ge \underset{i\in {\Omega }_{+}^{\mathrm{M}}}{\mathrm{min}}\left(\overline{D }\left({{\varvec{X}}}_{i}^{\mathrm{M}}\right)\right)\times \sum_{i\in {\Omega }_{+}^{\mathrm{M}}}\left({D}_{i}^{\mathrm{M}}-\frac{1}{N}\right)+\underset{i\in {\Omega }_{-}^{\mathrm{M}}}{\mathrm{max}}\left(\overline{D }\left({{\varvec{X}}}_{i}^{\mathrm{M}}\right)\right)\times \sum_{i\in {\Omega }_{-}^{\mathrm{M}}}\left({D}_{i}^{\mathrm{M}}-\frac{1}{N}\right)$$

(21)

where ${\Omega }_{+}^{\mathrm{M}}=\left\{i|{D}_{i}^{\mathrm{M}}\ge \frac{1}{N}\right\}$, ${\Omega }_{-}^{\mathrm{M}}=\left\{i|i=1,\cdots ,N\right\}-{\Omega }_{+}^{\mathrm{M}}$ and $D_{i}^{M} = e^{{d \times \overline{D}\left( {X_{i}^{M} } \right)}} /\sum\limits_{k = 1}^{N} {e^{{d \times \overline{D}\left( {X_{k}^{M} } \right)}} }$ is the normalized complexity. Equation (9) is derived referring to the analysis process from (15) to (16) with the fact that $\sum\nolimits_{i = 1}^{N} {D_{i}^{M} = 1}$.