Direct Incorporation of \(L_1\)-Regularization into Generalized Matrix Learning Vector Quantization

  • Falko LischkeEmail author
  • Thomas Neumann
  • Sven Hellbach
  • Thomas Villmann
  • Hans-Joachim Böhme
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10841)


Frequently, high-dimensional features are used to represent data to be classified. This paper proposes a new approach to learn interpretable classification models from such high-dimensional data representation. To this end, we extend a popular prototype-based classification algorithm, the matrix learning vector quantization, to incorporate an enhanced feature selection objective via \(L_1\)-regularization. In contrast to previous work, we propose a framework that directly optimizes this objective using the alternating direction method of multipliers (ADMM) and manifold optimization. We evaluate our method on synthetic data and on real data for speech-based emotion recognition. Particularly, we show that our method achieves state-of-the-art results on the Berlin Database of Emotional speech and show its abilities to select relevant dimensions from the eGeMAPS set of audio features.


  1. 1.
    Absil, P.A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2009)zbMATHGoogle Scholar
  2. 2.
    Ali, H., Hariharan, M., Yaacob, S., Adom, A.H.: Facial emotion recognition using empirical mode decomposition. Expert Syst. Appl. 42(3), 1261–1277 (2015)CrossRefGoogle Scholar
  3. 3.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bi, J., Bennett, K., Embrechts, M., Breneman, C., Song, M.: Dimensionality reduction via sparse support vector machines. JMLR 3(Mar), 1229–1243 (2003)zbMATHGoogle Scholar
  5. 5.
    Biehl, M., Hammer, B., Villmann, T.: Prototype-based models in machine learning. Wiley Interdisc. Rev.: Cogn. Sci. 7(2), 92–111 (2016)CrossRefGoogle Scholar
  6. 6.
    Biehl, M., Hammer, B., Schleif, F.M., Schneider, P., Villmann, T.: Stationarity of matrix relevance learning vector quantization. Mach. Learn. Rep. 3, 1–17 (2009)Google Scholar
  7. 7.
    Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, New York (2006)zbMATHGoogle Scholar
  8. 8.
    Bojer, T., Hammer, B., Schunk, D., Von Toschanowitz, K.: Relevance determination in learning vector quantization. In: Proceedings of ESANN (2001)Google Scholar
  9. 9.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefGoogle Scholar
  10. 10.
    Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W., Weiss, B.: A database of German emotional speech. In: Interspeech, vol. 5, pp. 1517–1520 (2005)Google Scholar
  11. 11.
    Chandrashekar, G., Sahin, F.: A survey on feature selection methods. Comput. Electr. Eng. 40(1), 16–28 (2014)CrossRefGoogle Scholar
  12. 12.
    Donoho, D.L.: De-noising by soft-thresholding. IEEE TIT 41(3), 613–627 (1995)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Donoho, D.L.: For most large underdetermined systems of linear equations the minimal \(\ell \)1-norm solution is also the sparsest solution. CPAMA 59(6), 797–829 (2006)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Eyben, F., Scherer, K.R., Schuller, B.W., Sundberg, J., André, E., Busso, C., Devillers, L.Y., Epps, J., Laukka, P., Narayanan, S.S., Truong, K.P.: The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE TAC 7(2), 190–202 (2016)Google Scholar
  15. 15.
    Eyben, F., Weninger, F., Gross, F., Schuller, B.: Recent developments in openSMILE, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM, pp. 835–838. ACM (2013)Google Scholar
  16. 16.
    Hammer, B., Villmann, T.: Generalized relevance learning vector quantization. Neural Netw. 15(8), 1059–1068 (2002)CrossRefGoogle Scholar
  17. 17.
    Hsu, C.W., Lin, C.J.: A comparison of methods for multiclass support vector machines. IEEE TNN 13(2), 415–425 (2002)Google Scholar
  18. 18.
    Kaden, M., Lange, M., Nebel, D., Riedel, M., Geweniger, T., Villmann, T.: Aspects in classification learning - review of recent developments in learning vector quantization. Found. Comput. Decis. Sci. 39(2), 79–105 (2014)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Kanth, N.R., Saraswathi, S.: Efficient speech emotion recognition using binary support vector machines multiclass SVM. In: 2015 IEEE ICCIC, December 2015Google Scholar
  20. 20.
    Kim, J., Truong, K.P., Englebienne, G., Evers, V.: Learning spectro-temporal features with 3D CNNs for speech emotion recognition. arXiv preprint arXiv:1708.05071 (2017)
  21. 21.
    Kohonen, T.: Learning vector quantization. In: Kohonen, T. (ed.) Self-Organizing Maps. SSINF, vol. 30, pp. 175–189. Springer, Heidelberg (1995). Scholar
  22. 22.
    Korkmaz, O.E., Atasoy, A.: Emotion recognition from speech signal using mel-frequency cepstral coefficients. In: 2015 9th ELECO, pp. 1254–1257, November 2015Google Scholar
  23. 23.
    Lee, J., Tashev, I.: High-level feature representation using recurrent neural network for speech emotion recognition. In: Interspeech 2015. ISCA, September 2015Google Scholar
  24. 24.
    Mao, Q., Dong, M., Huang, Z., Zhan, Y.: Learning salient features for speech emotion recognition using convolutional neural networks. IEEE Trans. Multimedia 16(8), 2203–2213 (2014)CrossRefGoogle Scholar
  25. 25.
    Murty, K.G., Kabadi, S.N.: Some NP-complete problems in quadratic and nonlinear programming. Math. Program. 39(2), 117–129 (1987)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Ng, A.Y.: Feature selection, L1 vs. L2 regularization, and rotational invariance. In: Proceedings of the 21th ICML, ICML 2004, p. 78. ACM, New York (2004)Google Scholar
  27. 27.
    Obozinski, G., Taskar, B., Jordan, M.: Multi-task feature selection. Statistics Department, UC Berkeley, Technical report 2 (2006)Google Scholar
  28. 28.
    Ofodile, I., Kulkarni, K., Corneanu, C.A., Escalera, S., Baro, X., Hyniewska, S., Allik, J., Anbarjafari, G.: Automatic recognition of deceptive facial expressions of emotion (2017)Google Scholar
  29. 29.
    Palo, H., Mohanty, M., Chandra, M.: Efficient feature combination techniques for emotional speech classification. IJST 19(1), 135–150 (2016)Google Scholar
  30. 30.
    Riedel, M., Rossi, F., Kästner, M., Villmann, T.: Regularization in relevance learning vector quantization using \(l_1\)-norms. In: Verleysen, M. (ed.) Proceedings of ESANN 2013, pp. 17–22 (2013)Google Scholar
  31. 31.
    Saeys, Y., Abeel, T., Van de Peer, Y.: Robust feature selection using ensemble feature selection techniques. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008. LNCS (LNAI), vol. 5212, pp. 313–325. Springer, Heidelberg (2008). Scholar
  32. 32.
    Sato, A., Yamada, K.: Generalized learning vector quantization. In: Advances in Neural Information Processing Systems, pp. 423–429 (1996)Google Scholar
  33. 33.
    Schneider, P., Biehl, M., Hammer, B.: Adaptive relevance matrices in learning vector quantization. Neural Comput. 21(12), 3532–3561 (2009)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Schneider, P., Bunte, K., Stiekema, H., Hammer, B., Villmann, T., Biehl, M.: Regularization in matrix relevance learning. IEEE Trans. Neural Netw. 21(5), 831–840 (2010)CrossRefGoogle Scholar
  35. 35.
    Schuller, B., Steidl, S., Batliner, A.: The INTERSPEECH 2009 emotion challenge. In: 10th Annual Conference of the ISCA (2009)Google Scholar
  36. 36.
    Schuller, B., Steidl, S., Batliner, A., Schiel, F., Krajewski, J.: The INTERSPEECH 2011 speaker state challenge. In: 12th Annual Conference of the ISCA (2011)Google Scholar
  37. 37.
    Schuller, B., Steidl, S., Batliner, A., et al.: The INTERSPEECH 2017 computational paralinguistics challenge: addressee, cold and snoring. In: ComParE, Interspeech 2017, pp. 3442–3446 (2017)Google Scholar
  38. 38.
    Sinith, M.S., Aswathi, E., Deepa, T.M., Shameema, C.P., Rajan, S.: Emotion recognition from audio signals using support vector machine. In: 2015 IEEE RAICS, pp. 139–144, December 2015Google Scholar
  39. 39.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. Roy. Stat. Soc. Ser. B (Methodol.) 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  40. 40.
    Townsend, J., Koep, N., Weichwald, S.: Pymanopt: a python toolbox for optimization on manifolds using automatic differentiation. J. Mach. Learn. Res. 17(137), 1–5 (2016)MathSciNetzbMATHGoogle Scholar
  41. 41.
    Villmann, T., Bohnsack, A., Kaden, M.: Can learning vector quantization be an alternative to SVM and deep learning? JAISCR 7(1), 65–81 (2017)Google Scholar
  42. 42.
    Wang, K., An, N., Li, B.N., Zhang, Y., Li, L.: Speech emotion recognition using fourier parameters. IEEE TAC 6(1), 69–75 (2015)Google Scholar
  43. 43.
    Wen, G., Li, H., Huang, J., Li, D., Xun, E.: Random deep belief networks for recognizing emotions from speech signals. Comput. Intell. Neurosci. 2017 (2017)CrossRefGoogle Scholar
  44. 44.
    Zhang, Y., Zhang, L., Hossain, M.A.: Adaptive 3D facial action intensity estimation and emotion recognition. Expert Syst. Appl. 42(3), 1446–1464 (2015)CrossRefGoogle Scholar
  45. 45.
    Zhu, J., Rosset, S., Tibshirani, R., Hastie, T.J.: 1-norm support vector machines. In: Advances in Neural Information Processing Systems, pp. 49–56 (2004)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Falko Lischke
    • 1
    Email author
  • Thomas Neumann
    • 1
  • Sven Hellbach
    • 1
  • Thomas Villmann
    • 2
  • Hans-Joachim Böhme
    • 1
  1. 1.HTW DresdenDresdenGermany
  2. 2.Saxony Institute for Computational Intelligence and Machine LearningUniversity of Applied Sciences MittweidaMittweidaGermany

Personalised recommendations