Sensorimotor Visual Perception on Embodied System Using Free Energy Principle

Abstract

We propose an embodied system that is based on the free energy principle (FEP) for sensorimotor visual perception (SMVP). Although the FEP mathematically describes the rule that living things obey, limitation by embodiment is required to model SMVP. The proposed system is configured by a body, which partially observes the environment, and memory, which retains classified knowledge about the environment as a generative model, and executes active and perceptual inferences. Evaluation using the MNIST dataset showed that the proposed system recognizes characters by active and perceptual inferences, and the intentionality corresponding to human confirmation bias is reproduced on the system.

Appendix

The generative model, described in this paper, is a combination of a variational autoencoder (VAE) and a fully connected neural network (FNN). The architecture is shown in Fig. 7. The encoder consists of four 2D convolutional layers and each layer is followed by a batch normalization and a rectified linear unit. The bottleneck consists of two linear transformation layers for calculating the average and the variance with reparameterizing function. The decoder consists of four 2D transposed convolutional layers and each layer is followed by a batch normalization and a rectified linear unit (sigmoid unit for the last layer). The classifier consists of a linear transformation layer followed by a rectified linear unit and a linear transformation layer followed by a softmax unit. The model was trained using Adam optimizer (learning rate: 0.001) with the sum of VAE loss and FNN loss.

Fig. 7.

Architecture of generative model

Algorithm 1 shows the pseudo code of the processing flow. The ${p}_{{a}_{t-1}}\left({s}_{t},{x}_{t}\right)$ is pre-trained using training data of $\left({s}_{t},{x}_{t}\right)$. All the training data of ${s}_{t}$ are pre-processed so that the center of gravity of an image is shifted to the center position. During the operation of the proposed embodied system, the process from the 2nd line to the 12th line is repeated. First, an attention image $s^{\prime}_{t}$ is obtained from the vision sensor. The past sensory input images are composed with the obtained $s^{\prime}_{t}$ while maintaining each relative attention position. The center of gravity of the composed image is calculated, and the composed image is shifted so that the center of gravity is located at the center position of the image. The shifted composed image is an ${s}_{t}$. Then, $q\left({x}_{t}|{\phi }_{{x}_{t}}\right)$ is calculated by inputting ${s}_{t}$ to ${p}_{{a}_{t-1}}\left({s}_{t},{x}_{t}\right)$. After that, the sub-function starting from the 14th line is called to generate expected sensory input images ${s}_{t+1}$. In the sub-function, an ${q}_{img}\left({x}_{t}|{\phi }_{{x}_{t}}\right)$ is calculated by inputting ${s}_{t}$ to ${p}_{{a}_{t-1}}\left({s}_{t},{x}_{t}\right)$. A template image is generated by detecting a bounding rectangle area of non-zero pixels in ${s}_{t}$ and extracting the area from ${s}_{t}$. Template matching is carried out in the ${q}_{img}\left({x}_{t}|{\phi }_{{x}_{t}}\right)$, and the representative position of the current $s_{t}$, $u_{cur}$, is obtained. To calculate the next candidate attention positions ${u}_{next}$, a candidate region of ${u}_{next}$ is set. The candidate region is a region obtained by adding a fixed margin pixel to a region of ${s}_{t}$ in ${q}_{img}\left({x}_{t}|{\phi }_{{x}_{t}}\right)$. The region of ${s}_{t}$ is defined by ${u}_{cur}$ and the size of the template image. The ${u}_{next}$ are calculated by sliding the window with the fixed stride pixel in the candidate region. The window is the size of $s^{\prime}_{t}$. The representative positions of all the window positions during sliding are ${u}_{next}$. The ${s}_{t+1}$ are generated by extracting the region of the ${s}_{t}$ and the region of the next candidate attention images $s^{\prime}_{t + 1}$ from ${q}_{img}\left({x}_{t}|{\phi }_{{x}_{t}}\right)$. The region of the ${s}_{t}$ is defined by ${u}_{cur}$ and the size of the template image, as mentioned above. The region of the $s^{\prime}_{t + 1}$ are defined by ${u}_{next}$ and the size of $s^{\prime}_{t + 1}$. The extracted images are clipped or applied with zero-padding to have the same size as ${s}_{t}$. Each approximate posterior distribution $q\left({x}_{t+1}|{\phi }_{{x}_{t+1}}\right)$ is calculated by inputting each image included in ${s}_{t+1}$ to ${p}_{{a}_{t-1}}\left({s}_{t},{x}_{t}\right)$. Note that $q\left({x}_{t}|{\phi }_{{x}_{t}}\right)$ is calculated using the current ${s}_{t}$, while $q\left({x}_{t+1}|{\phi }_{{x}_{t+1}}\right)$ is calculated using ${s}_{t+1}$. The entropy of each $q\left({x}_{t+1}|{\phi }_{{x}_{t+1}}\right)$ is calculated and added to the uncertainty map $M$. Finally, the attention position having the minimum value in $M$ is defined as the next attention position.