A Method to Construct Visual Recognition Algorithms on the Basis of Neural Activity Data

Abstract

Visual recognition by animals significantly outperforms man-made algorithms. The brain’s intelligent choice of visual features is considered to be underlying this performance gap. In order to attain better performance for man-made algorithms, we suggest using the visual features that are used in the brain in these algorithms. For this goal, we propose to obtain visual features correlated with the brain activity by applying a kernel canonical correlation analysis (KCCA) method to pairs of image data and neural data recorded from the brain of an animal exposed to the images. It is expected that only the visual features that are highly correlated with the neural activity provide useful information for visual recognition. Applied to hand-written digits as image data and activity data of a multi-layer neural network model as a model for a brain, the method successfully extracted visual features used in the neural network model. Indeed, the use of these visual features in the support vector machine (SVM) made it possible to discriminate the hand-written digits. Since this discrimination required to utilize the knowledge possessed in the neural network model, a simple application of the usual SVM without the use of these features could not discriminate them. We further demonstrate that even the use of non-digit hand-written characters for the KCCA extracts visual features which enable the SVM to discriminate the hand-written digits. This indicates the versatile applicability of our method.