Sampling Hidden Parameters from Oracle Distribution

  • Sho Sonoda
  • Noboru Murata
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8681)


A new sampling learning method for neural networks is proposed. Derived from an integral representation of neural networks, an oracle probability distribution of hidden parameters is introduced. In general rigorous sampling from the oracle distribution holds numerical difficulty, a linear-time sampling algorithm is also developed. Numerical experiments showed that when hidden parameters were initialized by the oracle distribution, following backpropagation converged faster to better parameters than when parameters were initialized by a normal distribution.


Integral representation neural networks sampling learning oracle distribution backpropagation weight initialization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE Trans. on PAMI 35(8), 1798–1828 (2013)CrossRefGoogle Scholar
  2. 2.
    Denoeux, T., Lengellé, R.: Initializing back propagation networks with prototypes. Neural Networks 6(3), 351–363 (1993)CrossRefGoogle Scholar
  3. 3.
    De Freitas, J.F.G., Niranjan, M., Gee, A.H., Doucet, A.: Sequential Monte Carlo methods to train neural network models. Neural Computation 12(4), 955–993 (2000)CrossRefGoogle Scholar
  4. 4.
    Kůrková, V.: Complexity estimates based on integral transforms induced by computational units. Neural Networks 33, 160–167 (2012)CrossRefzbMATHGoogle Scholar
  5. 5.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 11, 2278–2324 (1998)CrossRefGoogle Scholar
  6. 6.
    LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient Backprop. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) NN: Tricks of the Trade, 2nd edn. LNCS, vol. 7700, pp. 9–48. Springer, Heidelberg (2012)Google Scholar
  7. 7.
    LeCun, Y., Cortes, C.: The MNIST database of handwritten digits,
  8. 8.
    Murata, N.: An integral representation of functions using three-layered networks and their approximation bounds. Neural Networks 9(6), 947–956 (1996)CrossRefGoogle Scholar
  9. 9.
    Sprecher, D.A.: A numerical implementation of Kolmogorov’s superpositions. Neural Networks 9(5), 765–772 (1996)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Sho Sonoda
    • 1
  • Noboru Murata
    • 1
  1. 1.Schools of Advanced Science and EngineeringWaseda UniversityShinjuku-kuJapan

Personalised recommendations