Abstract
In this paper we investigate the use of Fourier Neural Operators (FNOs) for image classification in comparison to standard Convolutional Neural Networks (CNNs). Neural operators are a discretization-invariant generalization of neural networks to approximate operators between infinite dimensional function spaces. FNOs—which are neural operators with a specific parametrization—have been applied successfully in the context of parametric PDEs. We derive the FNO architecture as an example for continuous and Fréchet-differentiable neural operators on Lebesgue spaces. We further show how CNNs can be converted into FNOs and vice versa and propose an interpolation-equivariant adaptation of the architecture.
This work was supported by the European Union’s Horizon 2020 programme, Marie Skłodowska-Curie grant agreement No. 777826. TR and MB acknowledge the support of the BMBF, grant agreement No. 05M2020. SK and MB acknowledge the support of the DFG, project BU 2327/19-1. This work was carried out while MB was with the FAU Erlangen-Nürnberg.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Our code is available online: github.com/samirak98/FourierImaging.
- 2.
This dataset consists of 60, 000 training and 10, 000 test \(28\times 28\) images (grayscale).
- 3.
We employ a former version of the data set, which consists of 76, 262 RGB images for training and 2, 250 images for testing of size \(224\times 224\), where the task is to classify birds out of 450 possible classes.
References
Ambrosetti, A., Prodi, G.: A Primer of Nonlinear Analysis. Cambridge University Press, Cambridge (1993)
Briand, T.: Trigonometric polynomial interpolation of images. Image Process. On Line 9, 291–316 (2019)
Cai, D., Chen, K., Qian, Y., Kämäräinen, J.K.: Convolutional low-resolution fine-grained classification. Pattern Recogn. Lett. 119, 166–171 (2019)
Chi, L., Jiang, B., Mu, Y.: Fast Fourier convolution. Adv. Neural. Inf. Process. Syst. 33, 4479–4488 (2020)
Fukushima, K.C.: Cognitron: a self-organizing multilayered neural network. Biol. Cybern. 20, 121–136 (1975)
Goldberg, H., Kampowsky, W., Tröltzsch, F.: On NEMYTSKIJ operators in Lp-spaces of abstract functions. Math. Nachr. 155(1), 127–140 (1992)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)
Grafakos, L.: Classical Fourier Analysis. Graduate Texts in Mathematics, 3rd edn. Springer, New York (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE CVPR, pp. 770–778 (2016)
Hendrycks, D., Gimpel, K.: Gaussian error linear units (GELUs). arXiv:1606.08415 (2016)
Johnny, W., Brigido, H., Ladeira, M., Souza, J.C.F.: Fourier neural operator for image classification. In: 2022 17th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6 (2022)
Kovachki, N.B., Lanthaler, S., Mishra, S.: On universal approximation and error bounds for Fourier neural operators. J. Mach. Learn. Res. 22(1), 13237–13312 (2022)
Kovachki, N.B., et al.: Neural operator: Learning maps between function spaces. arXiv:2108.08481 (2021)
Koziarski, M., Cyganek, B.: Impact of low resolution on image recognition with deep neural networks: an experimental study. Int. J. Appl. Math. Comput. Sci. 28(4), 735–744 (2018)
Li, Z., Kovachki, N.B., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A.M., Anandkumar, A.: Fourier neural operator for parametric partial differential equations. In: 9th International Conference on Learning Representations (ICLR) (2021)
Li, Z., et al.: Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794 (2021)
Peng, X., Hoffman, J., Stella, X.Y., Saenko, K.: Fine-to-coarse knowledge transfer for low-res image classification. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3683–3687. IEEE (2016)
Piosenka, G.: Birds 500 - species image classification (2021). https://www.kaggle.com/datasets/gpiosenka/100-bird-species
Rao, Y., Zhao, W., Zhu, Z., Lu, J., Zhou, J.: Global filter networks for image classification. Adv. Neural. Inf. Process. Syst. 34, 980–993 (2021)
Remmert, R.: Theory of Complex Functions. Springer, New York (1991)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Tröltzsch, F.: Optimal Control of Partial Differential Equations: Theory, Methods, and Applications, Graduate Studies in Mathematics, vol. 112. American Mathematical Society, Providence, Rhode Island (2010)
Vaĭnberg, M.M.: Variational method and method of monotone operators in the theory of nonlinear equations. No. 22090, John Wiley & Sons, Hoboken (1974)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747 (2017)
Zhou, M., et al.: Deep Fourier up-sampling. arxiv:2210.05171 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kabri, S., Roith, T., Tenbrinck, D., Burger, M. (2023). Resolution-Invariant Image Classification Based on Fourier Neural Operators. In: Calatroni, L., Donatelli, M., Morigi, S., Prato, M., Santacesaria, M. (eds) Scale Space and Variational Methods in Computer Vision. SSVM 2023. Lecture Notes in Computer Science, vol 14009. Springer, Cham. https://doi.org/10.1007/978-3-031-31975-4_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-31975-4_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31974-7
Online ISBN: 978-3-031-31975-4
eBook Packages: Computer ScienceComputer Science (R0)