Compressive ELM: Improved Models through Exploiting Time-Accuracy Trade-Offs

  • Mark van Heeswijk
  • Amaury Lendasse
  • Yoan Miche
Part of the Communications in Computer and Information Science book series (CCIS, volume 459)

Abstract

In the training of neural networks, there often exists a trade-off between the time spent optimizing the model under investigation, and its final performance. Ideally, an optimization algorithm finds the model that has best test accuracy from the hypothesis space as fast as possible, and this model is efficient to evaluate at test time as well. However, in practice, there exists a trade-off between training time, testing time and testing accuracy, and the optimal trade-off depends on the user’s requirements. This paper proposes the Compressive Extreme Learning Machine, which allows for a time-accuracy trade-off by training the model in a reduced space. Experiments indicate that this trade-off is efficient in the sense that on average more time can be saved than accuracy lost. Therefore, it provides a mechanism that can yield better models in less time.

Keywords

Extreme Learning Machine ELM random projection compressive sensing Johnson-Lindenstrauss approximate matrix decompositions 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Huang, G.-B., Zhu, Q.-Y., Siew, C.-K.: Extreme learning machine: Theory and applications. Neurocomputing 70(1-3), 489–501 (2006)CrossRefGoogle Scholar
  2. 2.
    Huang, G.-B., Chen, L., Siew, C.-K.: Universal Approximation Using Incremental Constructive Feedforward Networks with Random Hidden Nodes. IEEE Transactions on Neural Networks 17(4), 879–892 (2006)CrossRefGoogle Scholar
  3. 3.
    Deng, W.-Y., Zheng, Q.-H., Chen, L.: Regularized extreme learning machine. In: IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2009, pp. 389–395 (2009)Google Scholar
  4. 4.
    van Heeswijk, M., Miche, Y., Oja, E., Lendasse, A.: GPU-accelerated and parallelized ELM ensembles for large-scale regression. Neurocomputing 74(16), 2430–2437 (2011)CrossRefGoogle Scholar
  5. 5.
    Miche, Y., van Heeswijk, M., Bas, P., Simula, O., Lendasse, A.: TROP-ELM: A double-regularized ELM using LARS and Tikhonov regularization. Neurocomputing 74(16), 2413–2421 (2011)CrossRefGoogle Scholar
  6. 6.
    Nelder, J., Mead, R.: A simplex method for function minimization. The Computer Journal 7(4), 308–313 (1965)CrossRefMATHGoogle Scholar
  7. 7.
    Lagarias, J.C., Reeds, J.A., Wright, M.H., Wright, P.E.: Convergence Properties of the Nelder–Mead Simplex Method in Low Dimensions. SIAM Journal on Optimization 9, 112–147 (1998)CrossRefMATHMathSciNetGoogle Scholar
  8. 8.
    Miche, Y., Sorjamaa, A., Bas, P., Simula, O., Jutten, C., Lendasse, A.: OP-ELM: optimally pruned extreme learning machine. IEEE Transactions on Neural Networks 21(1), 158–162 (2010)CrossRefGoogle Scholar
  9. 9.
    van Heeswijk, M., Miche, Y.: Binary/Ternary Extreme Learning Machines. Neurocomputing (to appear)Google Scholar
  10. 10.
    Neumann, K., Steil, J.J.: Batch intrinsic plasticity for extreme learning machines. In: Honkela, T. (ed.) ICANN 2011, Part I. LNCS, vol. 6791, pp. 339–346. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  11. 11.
    Asuncion, A., Newman, D.J.: UCI Machine Learning Repository (2007)Google Scholar
  12. 12.
    Halko, N., Martinsson, P.-G., Tropp, J.: Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions (September 2011) arXiv:0909.4061Google Scholar
  13. 13.
    Achlioptas, D.: Database-friendly random projections: Johnson-Lindenstrauss with binary coins. Journal of Computer and System Sciences 66(4), 671–687 (2003)CrossRefMATHMathSciNetGoogle Scholar
  14. 14.
    Matoušek, J.: On variants of the Johnson-Lindenstrauss lemma. Random Structures & Algorithms, 142–156 (2008)Google Scholar
  15. 15.
    Ailon, N., Chazelle, B.: Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform. In: Proceedings of the Thirty-Eighth Annual ACM Symposium on Theory of Computing, STOC 2006, pp. 557–563. ACM Press, New York (2006)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Mark van Heeswijk
    • 1
  • Amaury Lendasse
    • 1
    • 2
    • 3
  • Yoan Miche
    • 1
  1. 1.Department of Information and Computer ScienceAalto University School of ScienceAaltoFinland
  2. 2.Arcada University of Applied SciencesHelsinkiFinland
  3. 3.Department of Mechanical and Industrial EngineeringThe University of IowaIowa CityUSA

Personalised recommendations