A survey of quaternion neural networks

Abstract

Quaternion neural networks have recently received an increasing interest due to noticeable improvements over real-valued neural networks on real world tasks such as image, speech and signal processing. The extension of quaternion numbers to neural architectures reached state-of-the-art performances with a reduction of the number of neural parameters. This survey provides a review of past and recent research on quaternion neural networks and their applications in different domains. The paper details methods, algorithms and applications for each quaternion-valued neural networks proposed.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3

Abbreviations

ML:

Machine learning

AI:

Artificial intelligence

(R, G, B):

Red, green, blue

Q{Model}:

Quaternion{Model}

CVNN:

Complex-valued neural network

NN:

Neural network

MLP:

Multilayer perceptron

DNN:

Deep neural network

RNN:

Recurrent neural network

CNN:

Convolutional neural network

DAE:

Denoising autoencoder

CAE:

Convolutional autoencoder

HNN:

Hopfield neural network

SVM:

Support vector machine

PCA:

Principal component analysis

LDA:

Latent Dirichlet allocation

ReLU:

Rectified linear unit

tanh:

Hyperbolic tangent

eLU:

Exponential linear unit

CRF:

Cauthy–Riemann–Fueter

MSE:

Mean squared error

GAN:

Gaussian angular noise

PSNR:

Peak signal to noise ratio

ABr:

Average brightness

HOG:

Histograms oriented gradient

PolSAR:

Polarimetric synthetic aperture radar

CCS:

Customer care service

References

  1. Adavanne S, Politis A, Nikunen J, Virtanen T (2018) Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE J Sel Top Signal Process 13:34–48

    Google Scholar 

  2. Aizenberg IN, Gonzalez A (2018) Image recognition using MLMVN and frequency domain features. In: 2018 International joint conference on neural networks (IJCNN), pp 1–8

  3. Aizenberg I, Alexander S, Jackson J (2011) Recognition of blurred images using multilayer neural network based on multi-valued neurons. In: 2011 41st IEEE International symposium on multiple-valued logic. IEEE, pp 282–287

  4. Arena P, Fortuna L, Re R, Xibilia MG (1993) On the capability of neural networks with complex neurons in complex valued functions approximation. In: 1993 IEEE International symposium on circuits and systems, ISCAS’93. IEEE, pp 2168–2171

  5. Arena P, Fortuna L, Occhipinti L, Xibilia MG (1994) Neural networks for quaternion-valued function approximation. In: 1994 IEEE International symposium on circuits and systems, ISCAS’94, vol 6. IEEE, pp 307–310

  6. Arena P, Fortuna L, Muscato G, Xibilia MG (1997) Multilayer perceptrons to approximate quaternion valued functions. Neural Netw 10(2):335–342

    Google Scholar 

  7. Bayro-Corrochano E, Lechuga-Gutiérrez L, Garza-Burgos M (2018) Geometric techniques for robotics and hmi: Interpolation and haptics in conformal geometric algebra and control using quaternion spike neural networks. Robot Auton Syst 104:72–84

    Google Scholar 

  8. Bechet F, Maza B, Bigouroux N, Bazillon T, El-Beze M, De Mori R, Arbillot E (2012) Decoda: a call-centre human–human spoken conversation corpus. In: LREC, pp 1343–1347

  9. Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3(Jan):993–1022

    MATH  Google Scholar 

  10. Buchholz S, Sommer G (2000) Quaternionic spinor MLP. CiteSeer, Princeton

    Google Scholar 

  11. Buchholz S, Le Bihan N (2006) Optimal separation of polarized signals by quaternionic neural networks. In: 2006 14th European signal processing conference. IEEE, pp 1–5

  12. Chou JC (1992) Quaternion kinematic and dynamic differential equations. IEEE Trans Robot Autom 8(1):53–64

    Google Scholar 

  13. Comminiello D, Lella M, Scardapane S, Uncini A (2018) Quaternion convolutional neural networks for detection and localization of 3D sound events. arXiv:181206811

  14. Cui Y, Takahashi K, Hashimoto M (2013) Design of control systems using quaternion neural network and its application to inverse kinematics of robot manipulator. In: 2013 IEEE/SICE International symposium on system integration (SII). IEEE, pp 527–532

  15. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Computer society conference on computer vision and pattern recognition, CVPR 2005, vol 1. IEEE, pp 886–893

  16. De Boer PT, Kroese DP, Mannor S, Rubinstein RY (2005) A tutorial on the cross-entropy method. Ann Oper Res 134(1):19–67

    MathSciNet  MATH  Google Scholar 

  17. De Leo S, Rotelli P (1997) Local hypercomplex analyticity. arXiv preprint arXiv:9703002 [funct-an]

  18. Diebel J (2006) Representing attitude: Euler angles, unit quaternions, and rotation vectors. Matrix 58(15–16):1–35

    Google Scholar 

  19. Dornaika F, Horaud R (1998) Simultaneous robot-world and hand-eye calibration. IEEE Trans Robot Autom 14(4):617–622

    Google Scholar 

  20. Fortuna L, Muscato G, Xibilia M (1996) An hypercomplex neural network platform for robot positioning. In: 1996 IEEE International symposium on circuits and systems, ISCAS’96. Connecting the World, vol 3. IEEE, pp 609–612

  21. Fortuna L, Muscato G, Xibilia MG (2001) A comparison between hmlp and hrbf for attitude control. IEEE Trans Neural Netw 12(2):318–328

    Google Scholar 

  22. Garofolo JS, Lamel LF, Fisher WM, Fiscus JG, Pallett DS (1993) Darpa timit acoustic-phonetic continous speech corpus CD-ROM. NIST speech disc 1-1.1. NASA STI/Recon technical report no. 93

  23. Gaudet CJ, Maida AS (2018) Deep quaternion networks. In: 2018 International joint conference on neural networks (IJCNN). IEEE, pp 1–8

  24. Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: the KITTI dataset. Int J Robot Res (IJRR) 32:1231–1237

    Google Scholar 

  25. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249–256

  26. Graves A, Mohamed Ar, Hinton G (2013) Speech recognition with deep recurrent neural networks. In: 2013 IEEE International conference on acoustics, speech and signal processing. IEEE, pp 6645–6649

  27. Greenblatt A, Mosquera-Lopez C, Agaian S (2013) Quaternion neural networks applied to prostate cancer gleason grading. In: 2013 IEEE International conference on systems, man, and cybernetics (SMC). IEEE, pp 1144–1149

  28. Hamilton WR (1844) Ii. on quaternions; or on a new system of imaginaries in algebra. Lond Edinb Dublin Philos Mag J Sci 25(163):10–13

    Google Scholar 

  29. Hearst MA, Dumais ST, Osuna E, Platt J, Scholkopf B (1998) Support vector machines. IEEE Intell Syst Appl 13(4):18–28

    Google Scholar 

  30. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034

  31. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  32. Higham NJ (1990) Analysis of the Cholesky decomposition of a semi-definite matrix. Oxford University Press, Oxford

    Google Scholar 

  33. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    MathSciNet  MATH  Google Scholar 

  34. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    MathSciNet  MATH  Google Scholar 

  35. Hinton G, Deng L, Yu D, Dahl GE, Mohamed A, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97

    Google Scholar 

  36. Hirose A (2012) Complex-valued neural networks, vol 400. Springer, Berlin

    Google Scholar 

  37. Hitzer EM (2007) Quaternion fourier transform on quaternion fields and generalizations. Adv Appl Clifford Algebras 17(3):497–517

    MathSciNet  MATH  Google Scholar 

  38. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Google Scholar 

  39. Hopfield JJ, Tank DW (1985) “Neural” computation of decisions in optimization problems. Biol Cybern 52(3):141–152

    MATH  Google Scholar 

  40. Huang FJ, LeCun Y (2006) Large-scale learning with SVM and convolutional for generic object categorization. In: null. IEEE, pp 284–291

  41. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:150203167

  42. Isokawa T, Kusakabe T, Matsui N, Peper F (2003) Quaternion neural network and its application. In: International conference on knowledge-based and intelligent information and engineering systems. Springer, pp 318–324

  43. Isokawa T, Nishimura H, Kamiura N, Matsui N (2006) Fundamental properties of quaternionic hopfield neural network. In: 2006 International joint conference on neural networks, IJCNN’06. IEEE, pp 218–223

  44. Isokawa T, Nishimura H, Kamiura N, Matsui N (2008) Associative memory in quaternionic hopfield neural network. Int J Neural Syst 18(02):135–145

    Google Scholar 

  45. Isokawa T, Matsui N, Nishimura H (2009) Quaternionic neural networks: fundamental properties and applications. In: Complex-valued neural networks: utilizing high-dimensional parameters. IGI global. pp 411–439

  46. Isokawa T, Nishimura H, Matsui N (2012) Quaternionic multilayer perceptron with local analyticity. Information 3(4):756–770

    Google Scholar 

  47. Jolliffe I (2011) Principal component analysis. In: Lovric M (ed) International encyclopedia of statistical science. Springer, Berlin, pp 1094–1096

  48. Karney CF (2007) Quaternions in molecular modeling. J Mol Graph Model 25(5):595–604

    Google Scholar 

  49. Kinugawa K, Shang F, Usami N, Hirose A (2018) Isotropization of quaternion-neural-network-based PolSAR adaptive land classification in Poincare-sphere parameter space. IEEE Geosci Remote Sens Lett 15:1234–1238

    Google Scholar 

  50. Kobayashi M (2015) Hybrid quaternionic hopfield neural network. IEICE Trans Fundam Electron Commun Comput Sci 98(7):1512–1518

    Google Scholar 

  51. Kobayashi M, Nakajima A (2012) Twisted quaternary neural networks. IEEJ Trans Electr Electron Eng 7(4):397–401

    Google Scholar 

  52. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  53. Krizhevsky A, Nair V, Hinton G (2014) The cifar-10 dataset. http://www.cs.toronto.edu/kriz/cifar html

  54. Kusamichi H, Isokawa T, Matsui N, Ogawa Y, Maeda K (2004) A new scheme for color night vision by quaternion neural network. In: Proceedings of the 2nd international conference on autonomous robots and agents, vol 1315. Citeseer

  55. Lin JS, Cheng KS, Mao CW (1996) A fuzzy hopfield neural network for medical image segmentation. IEEE Trans Nucl Sci 43(4):2389–2398

    Google Scholar 

  56. Mandic DP, Goh VSL (2009) Complex valued nonlinear adaptive filters: noncircularity, widely linear and neural models, vol 59. Wiley, New York

    Google Scholar 

  57. Mandic DP, Jahanchahi C, Took CC (2011) A quaternion gradient operator and its applications. IEEE Signal Process Lett 18(1):47–50

    Google Scholar 

  58. Matsui N, Isokawa T, Kusamichi H, Peper F, Nishimura H (2004) Quaternion neural network with geometrical operators. J Intell Fuzzy Syst 15(3, 4):149–164

    MATH  Google Scholar 

  59. Mikolov T, Karafiát M, Burget L, Černockỳ J, Khudanpur S (2010) Recurrent neural network based language model. In: Eleventh annual conference of the international speech communication association

  60. Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp 807–814

  61. Nitta T (1995) A quaternary version of the back-propagation algorithm. In: IEEE International conference on neural networks, 1995. Proceedings, vol 5. IEEE, pp 2753–2756

  62. Nitta T (2004) A solution to the 4-bit parity problem with a single quaternary neuron. Neural Inf Process Lett Rev 5(2):33–39

    Google Scholar 

  63. Nitta T, de Garis H (1992) A 3D vector version of the back-propagation algorithm. In: Proceedings of international joint conference on neural networks, pp 511–516

  64. Ogawa T (2016) Neural network inversion for multilayer quaternion neural networks. Comput Technol Appl 7:73–82

    Google Scholar 

  65. Parcollet T, Morchid M, Bousquet PM, Dufour R, Linarès G, De Mori R (2016) Quaternion neural networks for spoken language understanding. In: 2016 IEEE Spoken language technology workshop (SLT). IEEE, pp 362–368

  66. Parcollet T, Morchid M, Linares G (2017a) Deep quaternion neural networks for spoken language understanding. In: 2017 IEEE Automatic speech recognition and understanding workshop (ASRU). IEEE, pp 504–511

  67. Parcollet T, Morchid M, Linares G (2017b) Quaternion denoising encoder–decoder for theme identification of telephone conversations. Proceedings of Interspeech 2017, pp 3325–3328

  68. Parcollet T, Morchid M, Linarès G (2018a) Quaternion convolutional neural networks for heterogeneous image processing. arXiv preprint arXiv:181102656

  69. Parcollet T, Ravanelli M, Morchid M, Linarès G, Trabelsi C, Mori RD, Bengio Y (2018b) Quaternion recurrent neural networks. arXiv preprint arXiv:1806.04418

  70. Parcollet T, Zhang Y, Morchid M, Trabelsi C, Linarès G, de Mori R, Bengio Y (2018c) Quaternion convolutional neural networks for end-to-end automatic speech recognition. In: Interspeech 2018, 19th Annual conference of the international speech communication association, Hyderabad, India, 2–6 September 2018, pp 22–26. https://doi.org/10.21437/Interspeech.2018-1898

  71. Pascanu R, Mikolov T, Bengio Y (2013) On the difficulty of training recurrent neural networks. In: International conference on machine learning, pp 1310–1318

  72. Platt J, et al. (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifiers. MIT Press, pp 61–74

  73. Pletinckx D (1989) Quaternion calculus as a basic tool in computer graphics. Vis Comput 5(1–2):2–13

    MATH  Google Scholar 

  74. Popa CA (2018) Learning algorithms for quaternion-valued neural networks. Neural Process Lett 47(3):949–973

    Google Scholar 

  75. Sangwine SJ (1996) Fourier transforms of colour images using quaternion or hypercomplex, numbers. Electron Lett 32(21):1979–1980

    Google Scholar 

  76. Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681

    Google Scholar 

  77. Shang F, Hirose A (2014) Quaternion neural-network-based PolSAR land classification in poincare-sphere-parameter space. IEEE Trans Geosci Remote Sensing 52(9):5693–5703

    Google Scholar 

  78. Shoemake K (1985) Animating rotation with quaternion curves. In: ACM SIGGRAPH computer graphics, vol 19. ACM, pp 245–254

  79. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556

  80. Soulard R, Carré P (2011) Quaternionic wavelets for texture classification. Pattern Recognit Lett 32(13):1669–1678

    Google Scholar 

  81. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  82. Takahashi K, Takahashi S, Cui Y, Hashimoto M (2014) Remarks on computational facial expression recognition from HOG features using quaternion multi-layer neural network. In: International conference on engineering applications of neural networks. Springer, pp 15–24

  83. Takahashi K, Isaka A, Fudaba T, Hashimoto M (2017) Remarks on quaternion neural network-based controller trained by feedback error learning. In: 2017 IEEE/SICE International symposium on system integration (SII), pp 875–880

  84. Tokuda, K., Zen, H., Kitamura, T. (2003) Trajectory modeling based on HMMs with the explicit relationship between static and dynamic features. In Eighth European conference on speech communication and technology

  85. Trabelsi C, Bilaniuk O, Zhang Y, Serdyuk D, Subramanian S, Santos JF, Mehri S, Rostamzadeh N, Bengio Y, Pal CJ (2017) Deep complex networks. arXiv preprint arXiv:170509792

  86. Ujang BC, Jahanchahi C, Took CC, Mandic D (2010) Quaternion valued neural networks and nonlinear adaptive filters.

  87. Ujang BC, Took CC, Mandic DP (2011) Quaternion-valued nonlinear adaptive filtering. IEEE Trans Neural Netw 22(8):1193–1206

    Google Scholar 

  88. Valle ME, de Castro FZ (2018) On the dynamics of hopfield neural networks on unit quaternions. IEEE Trans Neural Netw Learn Syst 29(6):2464–2471

    Google Scholar 

  89. Vincent P, Larochelle H, Bengio Y, Manzagol PA (2008) Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th international conference on machine learning. ACM, pp 1096–1103

  90. Willmott CJ, Matsuura K (2005) Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Clim Res 30(1):79–82

    Google Scholar 

  91. Xu D, Jahanchahi C, Took CC, Mandic DP (2015) Enabling quaternion derivatives: the generalized HR calculus. R Soc Open Sci 2(8):150255

    MathSciNet  Google Scholar 

  92. Xu D, Zhang L, Zhang H (2017) Learning algorithms in quaternion neural networks using GHR calculus. Neural Netw World 27(3):271

    Google Scholar 

  93. Yoshida M, Kuroe Y, Mori T (2005) Models of hopfield-type quaternion neural networks and their energy functions. Int J Neural Syst 15:129–135

    Google Scholar 

  94. Yun X, Bachmann ER (2006) Design, implementation, and experimental results of a quaternion-based kalman filter for human body motion tracking. IEEE Trans Robot 22(6):1216–1227

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Titouan Parcollet.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Parcollet, T., Morchid, M. & Linarès, G. A survey of quaternion neural networks. Artif Intell Rev 53, 2957–2982 (2020). https://doi.org/10.1007/s10462-019-09752-1

Download citation

Keywords

  • Hypercomplex numbers
  • Quaternion neural networks
  • Deep Learning