Abstract
In this paper, we propose a trainable multiplication layer (TML) for a neural network that can be used to calculate the multiplication between the input features. Taking an image as an input, the TML raises each pixel value to the power of a weight and then multiplies them, thereby extracting the higher-order local auto-correlation from the input image. The TML can also be used to extract co-occurrence from the feature map of a convolutional network. The training of the TML is formulated based on backpropagation with constraints to the weights, enabling us to learn discriminative multiplication patterns in an end-to-end manner. In the experiments, the characteristics of the TML are investigated by visualizing learned kernels and the corresponding output features. The applicability of the TML for classification and neural network interpretation is also evaluated using public datasets.
Supported by Qdai-jump Research Program and JSPS KAKENHI Grant Number JP17K12752 and JP17K19402.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amodei, D., et al.: Deep speech 2: end-to-end speech recognition in English and Mandarin. In: International Conference on Machine Learning, pp. 173–182. IMLS, New York (2016)
Brodatz, P.: Textures: A Photographic Album for Artists and Designers. Dover Publications, New York (1966)
Fujino, K., Mitani, Y., Fujita, Y., Hamamoto, Y., Sakaida, I.: Liver cirrhosis classification on M-mode ultrasound images by higher-order local auto-correlation features. J. Med. Bioeng. 3(1), 29–32 (2014)
Greenspan, H., van Ginneken, B., Summers, R.M.: Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35(5), 1153–1159 (2016)
Hayashi, H., Shibanoki, T., Shima, K., Kurita, Y., Tsuji, T.: A recurrent probabilistic neural network with dimensionality reduction based on time-series discriminant component analysis. IEEE Trans. Neural Networks Learn. Syst. 26(12), 3021–3033 (2015)
Hu, E., Nosato, H., Sakanashi, H., Murakawa, M.: A modified anomaly detection method for capsule endoscopy images using non-linear color conversion and Higher-order Local Auto-Correlation (HLAC). In: Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5477–5480. IEEE, Osaka (2013)
Kobayashi, T.: Trainable co-occurrence activation unit for improving ConvNet. In: Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1273–1277. IEEE, Calgary (2018)
Kobayashi, T., Otsu, N.: Action and simultaneous multiple-person identification using cubic higher-order local auto-correlation. In: Proceedings of the 17th International Conference on Pattern Recognition (ICPR), vol. 4, pp. 741–744. IEEE, Cambridge (2004)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)
Kylberg, G.: The kylberg texture dataset v. 1.0. External report (Blue series) 35, Centre for Image Analysis, Swedish University of Agricultural Sciences and Uppsala University, Uppsala, Sweden (2011)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Lin, M., Chen, Q., Yan, S.: Network in network. In: Proceedings of the International Conference on Learning Representations (ICLR), Banff, Canada (2014)
Ma, L., Lu, Z., Li, H.: Learning to answer questions from image using convolutional neural network. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, Arizona, pp. 3567–3573 (2016)
Majumder, N., Poria, S., Gelbukh, A., Cambria, E.: Deep learning-based document modeling for personality detection from text. IEEE Intell. Syst. 32(2), 74–79 (2017)
Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2015)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML), Haifa, Israel, pp. 807–814 (2010)
Nakayama, H., Harada, T., Kuniyoshi, Y.: Global Gaussian approach for scene categorization using information geometry. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2336–2343. IEEE, San Francisco (2010)
Nosato, H., Sakanashi, H., Takahashi, E., Murakawa, M.: An objective evaluation method of ulcerative colitis with optical colonoscopy images based on higher order local auto-correlation features. In: Proceedings of the 11th International Symposium on Biomedical Imaging (ISBI), pp. 89–92. IEEE, Beijing (2014)
Otsu, N., Kurita, T.: A new scheme for practical flexible and intelligent vision systems. In: Proceedings of the IAPR Workshop on Computer Vision, pp. 431–435. IAPR, Tokyo (1988)
Shih, Y.F., Yeh, Y.M., Lin, Y.Y., Weng, M.F., Lu, Y.C., Chuang, Y.Y.: Deep co-occurrence feature learning for visual object recognition. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu (2017)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA (2015)
Tsuji, T., Bu, N., Fukuda, O., Kaneko, M.: A recurrent log-linearized Gaussian mixture network. IEEE Trans. Neural Networks 14(2), 304–316 (2003)
Tsuji, T., Fukuda, O., Ichinobe, H., Kaneko, M.: A log-linearized Gaussian mixture network and its application to EEG pattern classification. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 29(1), 60–72 (1999)
Uehara, K., Sakanashi, H., Nosato, H., Murakawa, M., Miyamoto, H., Nakamura, R.: Object detection of satellite images using multi-channel higher-order local autocorrelation, pp. 1339–1344 (2017)
Valle-Lisboa, J.C., Reali, F., Anastasía, H., Mizraji, E.: Elman topology with sigma-pi units: an application to the modeling of verbal hallucinations in schizophrenia. Neural Networks 18(7), 863–877 (2005)
Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 Dataset. Technical report (2011)
Wang, Q., Li, P., Zhang, L.: G2DeNet: global Gaussian distribution embedding network and its application to visual recognition. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2730–2739. IEEE, Honolulu (2017)
Weber, C., Wermter, S.: A self-organizing map of sigma-pi units. Neurocomputing 70(13), 2552–2560 (2007)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
Zhang, T., Kahn, G., Levine, S., Abbeel, P.: Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search. In: Proceedings of the International Conference on Robotics and Automation (ICRA), pp. 528–535. IEEE, Stockholm (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Hayashi, H., Uchida, S. (2019). A Trainable Multiplication Layer for Auto-correlation and Co-occurrence Extraction. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11362. Springer, Cham. https://doi.org/10.1007/978-3-030-20890-5_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-20890-5_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20889-9
Online ISBN: 978-3-030-20890-5
eBook Packages: Computer ScienceComputer Science (R0)