An integrated model of autonomous topological spatial cognition

Abstract

This paper is focused on endowing a mobile robot with topological spatial cognition. We propose an integrated model—where the concept of a ‘place’  is defined as a collection of appearances or locations sharing common perceptual signatures or physical boundaries. In this model, as the robot navigates, places are detected in a systematic manner via monitoring coherency in the incoming visual data while pruning out uninformative or scanty data. Detected places are then either recognized or learned along with mapping as necessary. The novelties of the model are twofold: First, it explicitly incorporates a long-term spatial memory where the knowledge of learned places and their spatial relations are retained in place and map memories respectively. Second, the processing modules operate together so that the robot is able to build its spatial memory in an organized, incremental and unsupervised manner. Thus, the robot’s long-term spatial memory evolves completely on its own while learned knowledge is organized based on appearance-related similarities in a manner that is amenable for higher-level semantic reasoning, As such, the proposed model constitutes a step forward towards having robots that are capable of interacting with their environments in an autonomous manner.

Appendices

Appendix

The summary of the most commonly used symbols in the paper, their definitions and the sections where they are defined are presented in Table 5 for convenience.

Table 5 List of symbols

Fig. 15

Representation of visual data from sample bases in Fr, Sa, Lj and NC sites. a Visual data from sample bases in the Fr, Lj , Sa and NC sites. b Corresponding bubble surfaces for each of (color, Cartesian, non-Cartesian and intensity) features

Bubble space

This section presents a brief summary of bubble space representation for completeness. The interested reader is referred to Erkent and Bozma (2013) for further details. The bubble space ${\mathcal {B}} = {\mathcal {X}} \times {\mathcal {F}}$ is an abstract representation of the robot’s base along with its viewing directions (pan and tilt) ${\mathcal {F}} \subset S^2$ with $b \in {\mathcal {B}}$ defined as $b = \left[ x \, f \right] ^T$ where $x \in {\mathcal {X}}$ and $f \in {\mathcal {F}}$. Bubble surfaces $B_i(x,t): Im(h(x)) \times R^{\ge 0} \rightarrow R^{\ge 0}$ are hypothetical spherical surfaces surrounding the robot defined as:

$$\begin{aligned} B_i(x,t) = \left\{ \left[ \begin{array}{l} f \\ \rho _i(b,t) \end{array} \right] \mid \forall f \in {\mathcal {F}} \,\, \text{ and } b=\left[ x \,f\right] ^T \right\} \end{aligned}$$

(12)

where the image of a section h—namely Im(h(x))—is the set of viewing directions from a given base x with the section $h : {\mathcal {X}} \rightarrow {\mathcal {B}}$ defined as a continuous map such that $\forall x \in {\mathcal {X}}$, $\pi (h(x))=x$ and $\pi : {\mathcal {B}} \rightarrow {\mathcal {X}}$ defined as the projection of b onto ${\mathcal {X}}$ as $\pi (b)=x$. Finally, the function $\rho _i: {\mathcal {B}} \times R^{\ge 0} \rightarrow R^{\ge 0}$ is a Riemannian metric that encodes the observed values of ${ v }_i^{th}$ sensory feature. For simplification of notation, the second argument is omitted whenever time dependency is clear. Each bubble surface is initialized to be a $S^2$ sphere with radius $\rho _0 \in R^{\ge 0}$—namely $\rho _i(b,0)=\rho _0$. As the robot looks around, for each viewing direction $f \in {\mathcal {F}}$, it computes each feature value $q_{i}(b,t) \ge 0$. Next, each bubble surface $B_i(x,t)$ is deformed at the viewing direction f by an amount that depends on the associated sensory feature value $q_i(b,t)$ as:

$$\begin{aligned} \rho _i\big (b,t^+\big ) = q_i\big (b,t\big ) \end{aligned}$$

(13)

where the superscript $t^+$ denotes time just after t. As this is done for each feature ${ v }_i \in {\mathcal {V}}$ where $\left| {\mathcal {V}} \right| = N_v$, a set of $N_v$ bubble surfaces is generated. In the experiments, the robot computes seven bubble surfaces corresponding to seven visual features (hue, Cartesian, non-Cartesian and intensity). For the sample scenes as shown in Fig. 15a, the bubble surfaces are as shown Fig. 15b. The intensity bubble surface is used for checking reliability of sensory data in place detection.

Bubble descriptors are holistic (vector) representations of bubble surfaces. They are constructed using the double Fourier series representation of bubble surfaces as:

$$\begin{aligned} \rho _i\big (b,t\big ) = \sum ^{H_1}_{h_1=0} \sum ^{H_2}_{h_2=0}\lambda _{h_1h_2} z_{xi,h_1h_2}^T (t) e_{h_1h_2}(f) \end{aligned}$$

If $f \in {\mathcal {F}}$ is defined as $f =\left[ f_1 \, f_2\right] ^T$, for each $(h_1,h_2)$, the vector $e_{h_1h_2}(f) \in R^4$ consists of an orthonormal set of trigonometric basis functions as:

$$\begin{aligned} e_{h_1h_2}(f) = \left[ \begin{array}{l} cos\left( h_1 f_1\right) cos\left( h_2 f_2\right) \\ sin\left( h_1 f_1\right) cos\left( h_2 f_2\right) \\ cos\left( h_1 f_1\right) sin\left( h_2 f_2\right) \\ sin\left( h_1 f_1\right) sin\left( h_2 f_2\right) \end{array}\right] \end{aligned}$$

(14)

The corresponding vector $z_{xi,h_1h_2}(t) \in R^4$ is defined as:

$$\begin{aligned} { z_{xi,h_1h_2}(t) = \frac{1}{\pi ^{2}} \left[ \begin{array}{l} \displaystyle \smallint _{0}^{2\pi }\smallint _{0}^{\pi } {\rho _i(b,t) cos(h_1 f_1)cos(h_2 f_2)df_1 df_2} \\ \displaystyle \smallint _{0}^{2\pi }\smallint _{0}^{\pi } {\rho _i(b,t) sin(h_1 f_1)cos(h_2 f_2))df_1 df_2} \\ \displaystyle \smallint _{0}^{2\pi }\smallint _{0}^{\pi } {\rho _i(b,t)cos(h_1 f_1)sin(h_2 f_2))df_1 df_2} \\ \displaystyle \smallint _{0}^{2\pi }\smallint _{0}^{\pi } {\rho _i(b,t) sin(h_1 f_1)sin(h_2 f_2) df_1 df_2} \end{array} \right] } \end{aligned}$$

(15)

The parameters $\lambda _{h_1h_2}$ are defined as:

$$\begin{aligned} \lambda _{h_1h_2} = \left\{ \begin{array}{ll} \frac{1}{4} &{}\quad \text {if } h_1 = 0, h_2 = 0\\ \frac{1}{2} &{}\quad \text {if } h_1 > 0, h_2 = 0 \ or \ h_1 = 0, h_2 > 0\\ 1 &{}\quad \text {if } h_1 > 0, h_2 > 0 \end{array} \right. \end{aligned}$$

(16)

A bubble descriptor $I(x,t) \in R^{N_I}$ is a $N_I-$dimensional vector with $N_I = N_v(H_1+1)(H_2+1)$ defined as:

$$\begin{aligned} I(x,t) = \Big [I_{1,00}(x,t), \ldots , I_{N_v,H_1H_2}(x,t) \Big ]^T \end{aligned}$$

(17)

where

$$\begin{aligned} I_{i, h_1h_2}(x,t) = z_{xi,h_1h_2}^T(t) z_{xi,h_1h_2}(t) \end{aligned}$$

(18)

Bubble descriptors have been shown to be rotationally invariant with respect to heading changes while being computable in an incremental manner—as new observations are made. Furthermore, they are flexible integrating visual features since their dimensionality are independent of the number of observations. Furthermore, no data association (Williams et al. 2009) is required for finding correspondences among observations taken at different times.

In the experiments, the bubble descriptors are constructed using the first six features with the number of harmonics $H_1=H_2=9$. The intensity bubble surface is not used in order have decreased sensitivity to illumination level. Thus, the length of each bubble descriptor is $N_I=600$.

Sensory data reliability

This section presents a brief discussion of how reliability is measured. Reliability depends on the informativeness, coherency and plenitude of the sensory data. Since sensory data are internally represented using descriptors, they can be measured via processing the descriptors appropriately. In our case, this processing uses the bubble descriptors. The interested reader is kindly referred to Karaoguz and Bozma (2014) for further details.

Informativeness measures whether an incoming sensory data is semantically rich or not. For example, in case of low illumination conditions, everything in the image will look dark. Similarly, if the robot field of view is comprised of an extended object such as a door, again there won’t be much variation in the visual or depth images. An indication of both cases may be detected by computing the average deformation $\mu _i(x_k)$ or variance $\sigma _i(x_k)$ of the associated (intensity or depth) bubble surfaces $B_i(x_k,t_k)$:

$$\begin{aligned} \mu _i\big ({x_k}\big )= & {} \frac{1}{\pi ^{2}}\int _{0}^{2\pi }\int _{0}^{\pi } {\rho _i\big (b,t_k\big ) df_1 df_2} \\ \sigma _i\big ({x_k}\big )= & {} \int _{0}^{2\pi }\int _{0}^{\pi } \big ({\rho _i(b,t_k)} - \mu _i(x_k)\big )^2 df_1 df_2 \end{aligned}$$

Low values indicate minimal surface deformation which implies that the data is not informative. Hence, the informativeness decision is based on a binary valued function $\varsigma : k\rightarrow \left\{ 0,1\right\}$:

$$\begin{aligned} \varsigma (x_k) =\left\{ \begin{array}{ll} 1 &{}\quad \mu _i({x_k}) \le \tau _{\eta } \\ 1 &{} \quad \sigma _i({x_k}) \le \tau _{\sigma } \\ 0 &{} \quad \text{ otherwise } \end{array} \right. \end{aligned}$$

where $\tau _{\eta }$ and $\tau _{\sigma }$ are a priori selected threshold parameters. Sensory data from a particular base point $x_k$ is used if and only if $\varsigma (x_k) = 0$. In this work, the bubble surface associated with intensity feature ($i=7$) is used.

The coherency of data from two consecutive base points $x_k$ and $x_{k-1}$ is measured by comparing the similarity of their respective bubble descriptors $I(x_{k})$ and $I(x_{k-1})$ using a $\chi ^2$-distance. For example, in case of jagged robot head or body motion, sensory data from consecutive bases will be quite unrelated. Thus, the incoherency decision is based on a binary valued function $\kappa : k\rightarrow \left\{ 0,1\right\}$:

$$\begin{aligned} \kappa (x_k) = \left\{ \begin{array}{ll} 0 &{} \quad \left\| I(x_k),I(x_{k-1}) \right\| _{\chi ^2} \le \tau _\kappa \\ 1 &{}\quad \text{ otherwise } \end{array} \right. \end{aligned}$$

A low similarity value as compared with the incoherency threshold $\tau _{\kappa }$ is an indicator of incoherency.

Finally, the pool of data associated with each place should be of sufficient amount. For example, sensory data from just a few base points—even if informative and coherent—will not in general be indicative of a particular place. The plenitude decision is based on the extent of the detected places $D_m$—namely those with extent less than a plenitude threshold $\tau _p$ are considered to have insufficient amount of data. The values of the informativeness thresholds $\tau _{\eta }$ and $\tau _{\sigma }$, incoherency threshold $\tau _{\kappa }$ and the plenitude threshold $\tau _p$ affect the place detection performance. In this work, they are adjusted manually based on the camera type and nature of incoming sensory data.