A Robot Learns the Facial Expressions Recognition and Face/Non-face Discrimination Through an Imitation Game

Abstract

In this paper, we show that a robotic system can learn online to recognize facial expressions without having a teaching signal associating a facial expression with a given abstract label (e.g., ‘sadness’, ‘happiness’). Moreover, we show that recognizing a face from a non-face can be accomplished autonomously if we imagine that learning to recognize a face occurs after learning to recognize a facial expression, and not the opposite, as it is classically considered. In these experiments, the robot is considered as a baby because we want to understand how the baby can develop some abilities autonomously. We model, test and analyze cognitive abilities through robotic experiments. Our starting point was a mathematical model showing that, if the baby uses a sensory motor architecture for the recognition of a facial expression, then the parents must imitate the baby’s facial expression to allow the online learning. Here, a first series of robotic experiments shows that a simple neural network model can control a robot head and can learn online to recognize the facial expressions of the human partner if he/she imitates the robot’s prototypical facial expressions (the system is not using a model of the face nor a framing system). A second architecture using the rhythm of the interaction first allows a robust learning of the facial expressions without face tracking and next performs the learning involved in face recognition. Our more striking conclusion is that, for infants, learning to recognize a face could be more complex than recognizing a facial expression. Consequently, we emphasize the importance of the emotional resonance as a mechanism to ensure the dynamical coupling between individuals, allowing the learning of increasingly complex tasks.

Appendix

1.1 Visual Processing

The visual attention is controlled by a reflex mechanism that allows the robot to focus its gaze on potentially interesting regions. The focus points are the result of a local competition performed on the convolution between a DOG (difference of Gaussians) filter and the norm of the gradient of the input image (we use the same architecture for place and object recognition [19, 35]). This process allows the system to focus more on the corners and ends of the lines in the image (e.g., eyebrows, corners of the lips). The main advantages of this process over the SIFT (Scale Invariant Feature Transform) [33] method are its computational speed and a smaller number of extracted focus points. The intensity of the focus points is directly linked to their level of interest. Through a recurrent inhibition of the already selected points, a sequence of exploration is defined (a scan path). A short-term memory allows maintaining the inhibition of the already explored points. This memory is reset after each new image acquisition.⁹

To reduce the computational load (and to simplify the recognition), the image is sub-sampled from $720 \times 580$ to $256 \times 192$ pixels. For each focus point in the image, a local view centered on the focus point is extracted: either a log-polar transform or Gabor filters are applied (Fig. 18) to obtain an input image or a vector that is more robust to the rotations and distance variations. In our case, the log-polar transform has a radius of 20 pixels and projects to a $32 \times 32$ input image. The Gabor filtering is used to extract the mean and the standard deviation for the convolution of the 24 Gabor filters (see “Appendix”). Hence, the input vector obtained from the Gabor filter has only 48 components. It is a smaller vector than the result of the log-polar, which transform induces an a priori better generalization but also has a lower discrimination capability (Figs. 17, 19, 20, 21).

Fig. 17

Visual processing: this visual system is based on a sequential exploration of the image focus points. A gradient extraction is performed on the input image ($256 \times 192$ pixels). A convolution with a Difference Of Gaussian (DOG) provides the focus points. Last, the local views are extracted around each focus point

Fig. 18

Visual features: a the local log-polar transform increases the robustness of the extracted local views to small rotations and scale variations (a log-polar transform centered on the focus point is performed to obtain an image that is more robust to small rotations and distance variations. Its radius is 20 pixels). b Gabor filters are applied to obtain a signature that is more robust than a log-polar transform (the Gabor filters are $60 \times 60$); the features extracted for each convolution with a Gabor filter are the mean and the standard deviation

Fig. 19

A Gabor filter with these parameters $\sigma =8, \gamma =3$, and $\theta =\pi /3$

Fig. 20

Gabor filters for different frequencies and orientations

Fig. 21

Result of the convolution with 24 Gabor filters (different frequencies and orientations)

1.2 Gabor Filters

A Gabor filter has the following equation:

$$\begin{aligned}&G(x,y) = \frac{1}{2 \pi \sigma } \exp \left( - \frac{x^{2}}{+}y^{2}{2 \sigma ^{2}}\right) \cos (2 \pi f (x\cos (\theta )\nonumber \\&\qquad \qquad +y\sin (\theta ))) \end{aligned}$$

(10)

$$\begin{aligned} f=\frac{1}{\sigma \gamma } \end{aligned}$$

(11)

1.3 Model for the Rhythm Prediction

The neural network uses three groups of neurons. The Derivation Group ($DG$) receives the input signal, and the Temporal Group ($TG$) is a battery of neurons (15 neurons) with different temporal activities. The Prediction Group ($PG$) learns the conditioning between $DG$ (the present) and $TG$ (the past) information. In this model (Fig. 13), a $PG$ neuron learns and also predicts the delay between two events from $DG$. A $DG$ activation sets to zero the $TG$ neurons because of the links between $DG$ and $TG$. The neuronal activity ($DG$) involves instantaneous modifications in the weights between $PG$ and $TG$. After each reset (set to zero) by $DG$, the $TG$ neurons have the following activity:

$$\begin{aligned} Act^{TG}_{l}\left( t \right) = \frac{1}{m} \cdot \exp { - \frac{\left( t-m\right) ^2}{2 \cdot \sigma }} \end{aligned}$$

(12)

$l$ corresponds to the cell subscript, $m$ is a time constant, $\sigma$ is the standard deviation, and $t$ is the time. The $TG$ activity presents an activity trace $DG$ (of the past). $PG$ receives the $PG$ and $TG$ links, and the $TG$ information corresponds to the elapsed time since the last event, whereas the $DG$ information corresponds to the instant of the appearance of a new event. $PG$ sums the $TG$ and $DG$ inputs with the following equation:

$$\begin{aligned} Pot^{PG} = \sum _{l}{W_{pg}^{tg(l)} \cdot Act^{TG}_l} + W^{dg}_{pg} \cdot Act^{DG} \end{aligned}$$

(13)

$Act^{TG}_{l}$ is the $l$ cells activity of $TG, W_{pg}^{tg(l)}$ are the weight values between $TG$ and $PG, Act^{DG}$ is the $DG$ neuron activity and $W_{pg}$ is the weight value between $DG$ and $PG$. The $PG$ activation is triggered by the maximal value detection of its potential (the maximal value is equal to the potential’s derivative when it is equal to zero):

$$\begin{aligned}&Act^{PG} = f_{PG}\left( Pot^{PG}\right) \end{aligned}$$

(14)

$$\begin{aligned}&f_{PG} \left( x\left( t\right) \right) = \left\{ \begin{array}{ll} 1 &{}\quad if \frac{dx\left( t\right) }{dt}<0 \ and \ \frac{dx\left( t-1\right) }{dt} >0 \\ 0 &{}\quad else \end{array} \right. \end{aligned}$$

(15)

Finally, only the $W_{PG}^{TG}$ are learned, and we perform a one-shot learning process. The learning rule is the following:

$$\begin{aligned} W_{PG}^{TG(l)} = \left\{ \begin{array}{ll} \frac{Act^{TG}_{l}}{\sum _{l} \left( Act^{TG}_{l} \right) ^2} &{}\quad if \ Act^{TG} \ne 0 \\ inchange &{}\quad else \end{array} \right. \end{aligned}$$

(16)

Thereby, the $PG$ is activated if and only if the $TG$ cells’ activity sum is equal to that of learning.