Advertisement

Computational Experience with Pseudoinversion-Based Training of Neural Networks Using Random Projection Matrices

  • Luca Rubini
  • Rossella Cancelliere
  • Patrick Gallinari
  • Andrea Grosso
  • Antonino Raiti
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8722)

Abstract

Recently some novel strategies have been proposed for neural network training that set randomly the weights from input to hidden layer, while weights from hidden to output layer are analytically determined by Moore-Penrose generalised inverse; such non-iterative strategies are appealing since they allow fast learning. Aim of this study is to investigate the performance variability when random projections are used for convenient setting of the input weights: we compare them with state of the art setting i.e. weights randomly chosen according to a continuous uniform distribution. We compare the solutions obtained by different methods testing this approach on some UCI datasets for both regression and classification tasks; this results in a significant performance improvement with respect to conventional method.

Keywords

random projections weights setting pseudoinverse matrix 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Rumellhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Parallel Distrib. Process.: Exploration in the Microstructure of Cognition, vol. 1, pp. 318–362. MIT Press, Cambridge (1986)Google Scholar
  2. 2.
    LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient backProp. In: Orr, G.B., Müller, K.-R. (eds.) NIPS-WS 1996. LNCS, vol. 1524, pp. 9–50. Springer, Heidelberg (1998)Google Scholar
  3. 3.
    Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y.: An empirical evaluation of deep architectures on problems with many factors of variation. In: 24th ICML (2007)Google Scholar
  4. 4.
    Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: 25th ICML (2008)Google Scholar
  5. 5.
    Collobert, R., Weston, J.: A unified architecture for language processing: Deep neural networks with multitask learning. In: 25th ICML (2008)Google Scholar
  6. 6.
    Mnih, A., Hinton, G.E.: A scalable hierarchical distributed language model. In: 23rd NIPS, pp. 1081–1088 (2009)Google Scholar
  7. 7.
    Poggio, T., Girosi, F.: Networks for approximation and learning. IEEE 78(9), 1481–1497 (1990)CrossRefGoogle Scholar
  8. 8.
    Cancelliere, R.: A High Parallel Procedure to Initialize the Output Weights of a Radial Basis Function or BP Neural Network. In: Sørevik, T., Manne, F., Moe, R., Gebremedhin, A.H. (eds.) PARA 2000. LNCS, vol. 1947, pp. 384–390. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  9. 9.
    Huang, G.-B., Zhu, Q.-Y., Siew, C.-K.: Extreme Learning Machine: Theory and applications. Neurocomputing 70, 489–501 (2006)CrossRefGoogle Scholar
  10. 10.
    Halawa, K.: A method to improve the performance of multilayer perceptron by utilizing various activation functions in the last hidden layer and the least squares method. Neural Processing Letters 34, 293–303 (2011)CrossRefGoogle Scholar
  11. 11.
    Nguyen, T.D., Pham, H.T.B., Dang, V.H.: An efficient Pseudo Inverse matrix-based solution for secure auditing. In: IEEE International Conference on Computing and Communication Technologies, Research, Innovation, and Vision for the Future (2010)Google Scholar
  12. 12.
    Kohno, K., Kawamoto, M., Inouye, Y.: A Matrix Pseudoinversion Lemma and Its Application to Block-Based Adaptive Blind Deconvolution for MIMO Systems. IEEE Transactions on Circuits and Systems I: Regular Papers 57(7), 1449–1462 (2010)CrossRefMathSciNetGoogle Scholar
  13. 13.
    Ajorloo, H., Manzuri-Shalmani, M.T., Lakdashti, A.: Restoration of damaged slices in images using matrix pseudo inversion. In: 22nd International Symposium on Computer and Information Sciences (2007)Google Scholar
  14. 14.
    Wang, X.-Z., Wang, D., Huang, G.-B.: Special Issue on Extreme Learning Machines. Editorial. Soft Comput. 16(9), 1461–1463 (2012)CrossRefGoogle Scholar
  15. 15.
    Wang, X.: Special Issue on Extreme Learning Machine with Uncertainty. Editorial. Int. J. Unc. Fuzz. Knowl. Based Syst. 21(supp. 02), v–vi (2013)Google Scholar
  16. 16.
    Arriaga, R.I., Vempala, S.: An algorithmic theory of learning: robust concepts and random projection. In: 40th Annual Symp. on Foundations of Computer Science, pp. 616–623. IEEE Computer Society Press (1999)Google Scholar
  17. 17.
    Vempala, S.: Random projection: a new approach to VLSI layout. In: 39th Annual Symp. on Foundations of Computer Science. IEEE Computer Society Press (1998)Google Scholar
  18. 18.
    Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: 30th Symp. on Theory of Computing, pp. 604–613. ACM (1998)Google Scholar
  19. 19.
    Penrose, R.: On best approximate solution of linear matrix equations. Proceedings of the Cambridge Philosophical Society 52, 17–19 (1956)CrossRefzbMATHMathSciNetGoogle Scholar
  20. 20.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)zbMATHGoogle Scholar
  21. 21.
    Badeva, V., Morosov, V.: Problemes incorrectements posès, thèorie et applications (in French). Masson, Paris (1991)Google Scholar
  22. 22.
    Cancelliere, R., De Luca, R., Gai, M., Gallinari, P., Artières, T.: Pseudoinversion for neural training: tuning the regularisation parameter. Technical report n. 149/13, Dep. of Computer Science, University of Turin (2013)Google Scholar
  23. 23.
    Tikhonov, A.N., Arsenin, V.Y.: Solutions of Ill-Posed Problems. Winston, Washington, DC (1977)Google Scholar
  24. 24.
    Tikhonov, A.N.: Solution of incorrectly formulated problems and the regularization method. Soviet Mathematics 4, 1035–1038 (1963)Google Scholar
  25. 25.
    Gallinari, P., Cibas, T.: Practical complexity control in multilayer perceptrons. Signal Processing 74, 29–46 (1999)CrossRefzbMATHGoogle Scholar
  26. 26.
    Poggio, T., Girosi, F.: Regularization algorithms that are equivalent to multilayer networks. Science 247, 978–982 (1990)CrossRefzbMATHMathSciNetGoogle Scholar
  27. 27.
    Girosi, F., Jones, M., Poggio, T.: Regularization theory and neural networks architectures. Neural Computation 7(2), 219–269 (1995)CrossRefGoogle Scholar
  28. 28.
    Haykin, S.: Neural Networks, a comprehensive foundation. Prentice Hall, U.S.A. (1999)Google Scholar
  29. 29.
    Fuhry, M., Reichel, L.: A new Tikhonov regularization method. Numerical Algorithms 59, 433–445 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  30. 30.
    Hecht-Nielsen, R.: Context vectors: general purpose approximate meaning representations self-organized from raw data. In: Zurada, J.M., Marks II, R.J., Robinson, C.J. (eds.) Computational Intelligence: Imitating Life, pp. 43–56. IEEE Press (1994)Google Scholar
  31. 31.
    Bingham, E., Mannila, H.: Random projection in dimensionality reduction: Applications to image and text data. In: Conference on Knowledge Discovery and Data Mining, KDD 2001, San Francisco, CA, USA (2001)Google Scholar
  32. 32.
    Johnson, W.B., Lindenstrauss, J.: Extensions of Lipshitz mapping into Hilbert space. In: Conference in Modern Analysis and Probability. Contemporary Mathematics, vol. 26, pp. 189–206. Amer. Math. Soc. (1984)Google Scholar
  33. 33.
    Dasgupta, S., Gupta, A.: An elementary proof of the Johnson-Lindenstrauss lemma. Technical report TR-99-006, International Computer Science Institute, Berkeley, California, USA (1999)Google Scholar
  34. 34.
    Achlioptas, D.: Database-friendly random projections. In: ACM Symp. on the Principles of Database Systems, pp. 274–281 (2001)Google Scholar
  35. 35.
    Asuncion, A., Newman, D.J.: UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences (2007), http://www.ics.uci.edu/~mlearn/MLRepository.html

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Luca Rubini
    • 1
  • Rossella Cancelliere
    • 1
  • Patrick Gallinari
    • 2
  • Andrea Grosso
    • 1
  • Antonino Raiti
    • 1
  1. 1.Department of Computer ScienceUniversità di TorinoTurinItaly
  2. 2.Laboratory of Computer Sciences, LIP6Université Pierre et Marie CurieParisFrance

Personalised recommendations