Skip to main content
SpringerLink
Log in

A survey of quaternion neural networks

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Quaternion neural networks have recently received an increasing interest due to noticeable improvements over real-valued neural networks on real world tasks such as image, speech and signal processing. The extension of quaternion numbers to neural architectures reached state-of-the-art performances with a reduction of the number of neural parameters. This survey provides a review of past and recent research on quaternion neural networks and their applications in different domains. The paper details methods, algorithms and applications for each quaternion-valued neural networks proposed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Abbreviations

ML:

Machine learning

AI:

Artificial intelligence

(R, G, B):

Red, green, blue

Q{Model}:

Quaternion{Model}

CVNN:

Complex-valued neural network

NN:

Neural network

MLP:

Multilayer perceptron

DNN:

Deep neural network

RNN:

Recurrent neural network

CNN:

Convolutional neural network

DAE:

Denoising autoencoder

CAE:

Convolutional autoencoder

HNN:

Hopfield neural network

SVM:

Support vector machine

PCA:

Principal component analysis

LDA:

Latent Dirichlet allocation

ReLU:

Rectified linear unit

tanh:

Hyperbolic tangent

eLU:

Exponential linear unit

CRF:

Cauthy–Riemann–Fueter

MSE:

Mean squared error

GAN:

Gaussian angular noise

PSNR:

Peak signal to noise ratio

ABr:

Average brightness

HOG:

Histograms oriented gradient

PolSAR:

Polarimetric synthetic aperture radar

CCS:

Customer care service

References

  • Adavanne S, Politis A, Nikunen J, Virtanen T (2018) Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE J Sel Top Signal Process 13:34–48

    Google Scholar 

  • Aizenberg IN, Gonzalez A (2018) Image recognition using MLMVN and frequency domain features. In: 2018 International joint conference on neural networks (IJCNN), pp 1–8

  • Aizenberg I, Alexander S, Jackson J (2011) Recognition of blurred images using multilayer neural network based on multi-valued neurons. In: 2011 41st IEEE International symposium on multiple-valued logic. IEEE, pp 282–287

  • Arena P, Fortuna L, Re R, Xibilia MG (1993) On the capability of neural networks with complex neurons in complex valued functions approximation. In: 1993 IEEE International symposium on circuits and systems, ISCAS’93. IEEE, pp 2168–2171

  • Arena P, Fortuna L, Occhipinti L, Xibilia MG (1994) Neural networks for quaternion-valued function approximation. In: 1994 IEEE International symposium on circuits and systems, ISCAS’94, vol 6. IEEE, pp 307–310

  • Arena P, Fortuna L, Muscato G, Xibilia MG (1997) Multilayer perceptrons to approximate quaternion valued functions. Neural Netw 10(2):335–342

    Google Scholar 

  • Bayro-Corrochano E, Lechuga-Gutiérrez L, Garza-Burgos M (2018) Geometric techniques for robotics and hmi: Interpolation and haptics in conformal geometric algebra and control using quaternion spike neural networks. Robot Auton Syst 104:72–84

    Google Scholar 

  • Bechet F, Maza B, Bigouroux N, Bazillon T, El-Beze M, De Mori R, Arbillot E (2012) Decoda: a call-centre human–human spoken conversation corpus. In: LREC, pp 1343–1347

  • Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3(Jan):993–1022

    MATH  Google Scholar 

  • Buchholz S, Sommer G (2000) Quaternionic spinor MLP. CiteSeer, Princeton

    Google Scholar 

  • Buchholz S, Le Bihan N (2006) Optimal separation of polarized signals by quaternionic neural networks. In: 2006 14th European signal processing conference. IEEE, pp 1–5

  • Chou JC (1992) Quaternion kinematic and dynamic differential equations. IEEE Trans Robot Autom 8(1):53–64

    Google Scholar 

  • Comminiello D, Lella M, Scardapane S, Uncini A (2018) Quaternion convolutional neural networks for detection and localization of 3D sound events. arXiv:181206811

  • Cui Y, Takahashi K, Hashimoto M (2013) Design of control systems using quaternion neural network and its application to inverse kinematics of robot manipulator. In: 2013 IEEE/SICE International symposium on system integration (SII). IEEE, pp 527–532

  • Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Computer society conference on computer vision and pattern recognition, CVPR 2005, vol 1. IEEE, pp 886–893

  • De Boer PT, Kroese DP, Mannor S, Rubinstein RY (2005) A tutorial on the cross-entropy method. Ann Oper Res 134(1):19–67

    MathSciNet  MATH  Google Scholar 

  • De Leo S, Rotelli P (1997) Local hypercomplex analyticity. arXiv preprint arXiv:9703002 [funct-an]

  • Diebel J (2006) Representing attitude: Euler angles, unit quaternions, and rotation vectors. Matrix 58(15–16):1–35

    Google Scholar 

  • Dornaika F, Horaud R (1998) Simultaneous robot-world and hand-eye calibration. IEEE Trans Robot Autom 14(4):617–622

    Google Scholar 

  • Fortuna L, Muscato G, Xibilia M (1996) An hypercomplex neural network platform for robot positioning. In: 1996 IEEE International symposium on circuits and systems, ISCAS’96. Connecting the World, vol 3. IEEE, pp 609–612

  • Fortuna L, Muscato G, Xibilia MG (2001) A comparison between hmlp and hrbf for attitude control. IEEE Trans Neural Netw 12(2):318–328

    Google Scholar 

  • Garofolo JS, Lamel LF, Fisher WM, Fiscus JG, Pallett DS (1993) Darpa timit acoustic-phonetic continous speech corpus CD-ROM. NIST speech disc 1-1.1. NASA STI/Recon technical report no. 93

  • Gaudet CJ, Maida AS (2018) Deep quaternion networks. In: 2018 International joint conference on neural networks (IJCNN). IEEE, pp 1–8

  • Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: the KITTI dataset. Int J Robot Res (IJRR) 32:1231–1237

    Google Scholar 

  • Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249–256

  • Graves A, Mohamed Ar, Hinton G (2013) Speech recognition with deep recurrent neural networks. In: 2013 IEEE International conference on acoustics, speech and signal processing. IEEE, pp 6645–6649

  • Greenblatt A, Mosquera-Lopez C, Agaian S (2013) Quaternion neural networks applied to prostate cancer gleason grading. In: 2013 IEEE International conference on systems, man, and cybernetics (SMC). IEEE, pp 1144–1149

  • Hamilton WR (1844) Ii. on quaternions; or on a new system of imaginaries in algebra. Lond Edinb Dublin Philos Mag J Sci 25(163):10–13

    Google Scholar 

  • Hearst MA, Dumais ST, Osuna E, Platt J, Scholkopf B (1998) Support vector machines. IEEE Intell Syst Appl 13(4):18–28

    Google Scholar 

  • He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  • Higham NJ (1990) Analysis of the Cholesky decomposition of a semi-definite matrix. Oxford University Press, Oxford

    MATH  Google Scholar 

  • Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    MathSciNet  MATH  Google Scholar 

  • Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    MathSciNet  MATH  Google Scholar 

  • Hinton G, Deng L, Yu D, Dahl GE, Mohamed A, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97

    Google Scholar 

  • Hirose A (2012) Complex-valued neural networks, vol 400. Springer, Berlin

    MATH  Google Scholar 

  • Hitzer EM (2007) Quaternion fourier transform on quaternion fields and generalizations. Adv Appl Clifford Algebras 17(3):497–517

    MathSciNet  MATH  Google Scholar 

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Google Scholar 

  • Hopfield JJ, Tank DW (1985) “Neural” computation of decisions in optimization problems. Biol Cybern 52(3):141–152

    MATH  Google Scholar 

  • Huang FJ, LeCun Y (2006) Large-scale learning with SVM and convolutional for generic object categorization. In: null. IEEE, pp 284–291

  • Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:150203167

  • Isokawa T, Kusakabe T, Matsui N, Peper F (2003) Quaternion neural network and its application. In: International conference on knowledge-based and intelligent information and engineering systems. Springer, pp 318–324

  • Isokawa T, Nishimura H, Kamiura N, Matsui N (2006) Fundamental properties of quaternionic hopfield neural network. In: 2006 International joint conference on neural networks, IJCNN’06. IEEE, pp 218–223

  • Isokawa T, Nishimura H, Kamiura N, Matsui N (2008) Associative memory in quaternionic hopfield neural network. Int J Neural Syst 18(02):135–145

    Google Scholar 

  • Isokawa T, Matsui N, Nishimura H (2009) Quaternionic neural networks: fundamental properties and applications. In: Complex-valued neural networks: utilizing high-dimensional parameters. IGI global. pp 411–439

  • Isokawa T, Nishimura H, Matsui N (2012) Quaternionic multilayer perceptron with local analyticity. Information 3(4):756–770

    Google Scholar 

  • Jolliffe I (2011) Principal component analysis. In: Lovric M (ed) International encyclopedia of statistical science. Springer, Berlin, pp 1094–1096

  • Karney CF (2007) Quaternions in molecular modeling. J Mol Graph Model 25(5):595–604

    Google Scholar 

  • Kinugawa K, Shang F, Usami N, Hirose A (2018) Isotropization of quaternion-neural-network-based PolSAR adaptive land classification in Poincare-sphere parameter space. IEEE Geosci Remote Sens Lett 15:1234–1238

    Google Scholar 

  • Kobayashi M (2015) Hybrid quaternionic hopfield neural network. IEICE Trans Fundam Electron Commun Comput Sci 98(7):1512–1518

    Google Scholar 

  • Kobayashi M, Nakajima A (2012) Twisted quaternary neural networks. IEEJ Trans Electr Electron Eng 7(4):397–401

    Google Scholar 

  • Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  • Krizhevsky A, Nair V, Hinton G (2014) The cifar-10 dataset. http://www.cs.toronto.edu/kriz/cifar html

  • Kusamichi H, Isokawa T, Matsui N, Ogawa Y, Maeda K (2004) A new scheme for color night vision by quaternion neural network. In: Proceedings of the 2nd international conference on autonomous robots and agents, vol 1315. Citeseer

  • Lin JS, Cheng KS, Mao CW (1996) A fuzzy hopfield neural network for medical image segmentation. IEEE Trans Nucl Sci 43(4):2389–2398

    Google Scholar 

  • Mandic DP, Goh VSL (2009) Complex valued nonlinear adaptive filters: noncircularity, widely linear and neural models, vol 59. Wiley, New York

    Google Scholar 

  • Mandic DP, Jahanchahi C, Took CC (2011) A quaternion gradient operator and its applications. IEEE Signal Process Lett 18(1):47–50

    Google Scholar 

  • Matsui N, Isokawa T, Kusamichi H, Peper F, Nishimura H (2004) Quaternion neural network with geometrical operators. J Intell Fuzzy Syst 15(3, 4):149–164

    MATH  Google Scholar 

  • Mikolov T, Karafiát M, Burget L, Černockỳ J, Khudanpur S (2010) Recurrent neural network based language model. In: Eleventh annual conference of the international speech communication association

  • Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp 807–814

  • Nitta T (1995) A quaternary version of the back-propagation algorithm. In: IEEE International conference on neural networks, 1995. Proceedings, vol 5. IEEE, pp 2753–2756

  • Nitta T (2004) A solution to the 4-bit parity problem with a single quaternary neuron. Neural Inf Process Lett Rev 5(2):33–39

    Google Scholar 

  • Nitta T, de Garis H (1992) A 3D vector version of the back-propagation algorithm. In: Proceedings of international joint conference on neural networks, pp 511–516

  • Ogawa T (2016) Neural network inversion for multilayer quaternion neural networks. Comput Technol Appl 7:73–82

    Google Scholar 

  • Parcollet T, Morchid M, Bousquet PM, Dufour R, Linarès G, De Mori R (2016) Quaternion neural networks for spoken language understanding. In: 2016 IEEE Spoken language technology workshop (SLT). IEEE, pp 362–368

  • Parcollet T, Morchid M, Linares G (2017a) Deep quaternion neural networks for spoken language understanding. In: 2017 IEEE Automatic speech recognition and understanding workshop (ASRU). IEEE, pp 504–511

  • Parcollet T, Morchid M, Linares G (2017b) Quaternion denoising encoder–decoder for theme identification of telephone conversations. Proceedings of Interspeech 2017, pp 3325–3328

  • Parcollet T, Morchid M, Linarès G (2018a) Quaternion convolutional neural networks for heterogeneous image processing. arXiv preprint arXiv:181102656

  • Parcollet T, Ravanelli M, Morchid M, Linarès G, Trabelsi C, Mori RD, Bengio Y (2018b) Quaternion recurrent neural networks. arXiv preprint arXiv:1806.04418

  • Parcollet T, Zhang Y, Morchid M, Trabelsi C, Linarès G, de Mori R, Bengio Y (2018c) Quaternion convolutional neural networks for end-to-end automatic speech recognition. In: Interspeech 2018, 19th Annual conference of the international speech communication association, Hyderabad, India, 2–6 September 2018, pp 22–26. https://doi.org/10.21437/Interspeech.2018-1898

  • Pascanu R, Mikolov T, Bengio Y (2013) On the difficulty of training recurrent neural networks. In: International conference on machine learning, pp 1310–1318

  • Platt J, et al. (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifiers. MIT Press, pp 61–74

  • Pletinckx D (1989) Quaternion calculus as a basic tool in computer graphics. Vis Comput 5(1–2):2–13

    MATH  Google Scholar 

  • Popa CA (2018) Learning algorithms for quaternion-valued neural networks. Neural Process Lett 47(3):949–973

    Google Scholar 

  • Sangwine SJ (1996) Fourier transforms of colour images using quaternion or hypercomplex, numbers. Electron Lett 32(21):1979–1980

    Google Scholar 

  • Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681

    Google Scholar 

  • Shang F, Hirose A (2014) Quaternion neural-network-based PolSAR land classification in poincare-sphere-parameter space. IEEE Trans Geosci Remote Sensing 52(9):5693–5703

    Google Scholar 

  • Shoemake K (1985) Animating rotation with quaternion curves. In: ACM SIGGRAPH computer graphics, vol 19. ACM, pp 245–254

  • Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556

  • Soulard R, Carré P (2011) Quaternionic wavelets for texture classification. Pattern Recognit Lett 32(13):1669–1678

    Google Scholar 

  • Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  • Takahashi K, Takahashi S, Cui Y, Hashimoto M (2014) Remarks on computational facial expression recognition from HOG features using quaternion multi-layer neural network. In: International conference on engineering applications of neural networks. Springer, pp 15–24

  • Takahashi K, Isaka A, Fudaba T, Hashimoto M (2017) Remarks on quaternion neural network-based controller trained by feedback error learning. In: 2017 IEEE/SICE International symposium on system integration (SII), pp 875–880

  • Tokuda, K., Zen, H., Kitamura, T. (2003) Trajectory modeling based on HMMs with the explicit relationship between static and dynamic features. In Eighth European conference on speech communication and technology

  • Trabelsi C, Bilaniuk O, Zhang Y, Serdyuk D, Subramanian S, Santos JF, Mehri S, Rostamzadeh N, Bengio Y, Pal CJ (2017) Deep complex networks. arXiv preprint arXiv:170509792

  • Ujang BC, Jahanchahi C, Took CC, Mandic D (2010) Quaternion valued neural networks and nonlinear adaptive filters.

  • Ujang BC, Took CC, Mandic DP (2011) Quaternion-valued nonlinear adaptive filtering. IEEE Trans Neural Netw 22(8):1193–1206

    Google Scholar 

  • Valle ME, de Castro FZ (2018) On the dynamics of hopfield neural networks on unit quaternions. IEEE Trans Neural Netw Learn Syst 29(6):2464–2471

    Google Scholar 

  • Vincent P, Larochelle H, Bengio Y, Manzagol PA (2008) Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th international conference on machine learning. ACM, pp 1096–1103

  • Willmott CJ, Matsuura K (2005) Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Clim Res 30(1):79–82

    Google Scholar 

  • Xu D, Jahanchahi C, Took CC, Mandic DP (2015) Enabling quaternion derivatives: the generalized HR calculus. R Soc Open Sci 2(8):150255

    MathSciNet  Google Scholar 

  • Xu D, Zhang L, Zhang H (2017) Learning algorithms in quaternion neural networks using GHR calculus. Neural Netw World 27(3):271

    Google Scholar 

  • Yoshida M, Kuroe Y, Mori T (2005) Models of hopfield-type quaternion neural networks and their energy functions. Int J Neural Syst 15:129–135

    Google Scholar 

  • Yun X, Bachmann ER (2006) Design, implementation, and experimental results of a quaternion-based kalman filter for human body motion tracking. IEEE Trans Robot 22(6):1216–1227

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratoire Informatique d’Avignon (LIA), Université d’Avignon, Avignon, France

    Titouan Parcollet, Mohamed Morchid & Georges Linarès

  2. ORKIS, Aix-en-Provence, France

    Titouan Parcollet

Authors
  1. Titouan Parcollet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Morchid
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Georges Linarès
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Titouan Parcollet.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Parcollet, T., Morchid, M. & Linarès, G. A survey of quaternion neural networks. Artif Intell Rev 53, 2957–2982 (2020). https://doi.org/10.1007/s10462-019-09752-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-019-09752-1

Keywords

Search

Navigation