Forward and Backward Selection in Regression Hybrid Network

  • Shimon Cohen
  • Nathan Intrator
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2364)


We introduce a Forward Backward and Model Selection algorithm (FBMS) for constructing a hybrid regression network of radial and perceptron hidden units. The algorithm determines whether a radial or a perceptron unit is required at a given region of input space. Given an error target, the algorithm also determines the number of hidden units. Then the algorithm uses model selection criteria and prunes unnecessary weights. This results in a final architecture which is often much smaller than a RBF network or a MLP. Results for various data sizes on the Pumadyn data indicate that the resulting architecture competes and often outperform best known results for this data set.


Hybrid Network Architecture SMLP Clustering Regularization Nested Models Model Selection 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. The Wadsworth Statistics/Probability Series, Belmont, CA, 1984.MATHGoogle Scholar
  2. [2]
    C.E. Rasmussen, R.M. Neal, G.E. Hinton, D. Van Camp, Z. Ghahrman M. Revow, R. Kustra, and R. Tibshirani. The delve manual. 1996.Google Scholar
  3. [3]
    S. Cohen and N. Intrator. Automatic model selection of ridge and radial functions. In Second International workshop on Multiple Classifier Systems, 2001.Google Scholar
  4. [4]
    S. Cohen and N. Intrator. A hybrid projection based and radial basis function architecture: Initial values and global optimization. To appear in Special issue of PAA on Fusion of Multiple Classifiers, 2001.Google Scholar
  5. [5]
    D. L. Donoho and I. M. Johnstone. Projection-based approximation and a duality with kernel methods. Annals of Statistics, 17:58–106, 1989.MATHMathSciNetCrossRefGoogle Scholar
  6. [6]
    G.W. Flake. Square unit augmented, radially extended, multilayer percpetrons. In G. B. Orr and K. Müller, editors, Neural Networks: Tricks of the Trade, pages 145–163. Springer, 1998.Google Scholar
  7. [7]
    J. H. Friedman. Mutltivariate adaptive regression splines. The Annals of Statistics, 19:1–141, 1991.MATHMathSciNetCrossRefGoogle Scholar
  8. [8]
    B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In C. L. Giles, S. J. Hanson, and J. D. Cowan, editors, Advances in Neural Information Processing Systems, volume 5. Morgan Kaufmann, San Mateo, CA, 1993.Google Scholar
  9. [9]
    R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991.CrossRefGoogle Scholar
  10. [10]
    N. Sugie K. Suzuki, I. Horiba. A simple neural network algorithm with application to filter synthesis. Neural Processing Letters, Kluwer Academic Publishers, Netherlands, 13:43–53, 2001.Google Scholar
  11. [11]
    R. E. Kass and A. E. Raftery. Bayes factors. Journal of The American Statistical Association, 90:773–795, 1995.MATHCrossRefGoogle Scholar
  12. [12]
    Y.C. Lee, G. Doolen, H.H. Chen, G.Z. Sun, T. Maxwell, H.Y. Lee, and C.L. Giles. Machine learning using higher order correlation networks. Physica D, pages 22-D: 276–306, 1986.MathSciNetGoogle Scholar
  13. [13]
    R. M. Neal. Bayesian Learning for Neural Networks. Springer, New York, 1996.MATHGoogle Scholar
  14. [14]
    S. J. Nowlan. Soft competitive adaptation: Neural network learning algorithms basd on fitting statistical mixtures. Ph.D. dissertation, Carnegie Mellon University, 1991.Google Scholar
  15. [15]
    A. Papoulis. Probbaility, Random Variables, and Stochastic Process, volume 1. McGRAW-HILL, New York, third edition, 1991.Google Scholar
  16. [16]
    D. G. Stork R. O. Duda, P. E. Hart. Pattern Classification. John Wiley Sons, INC., New York, 2001.MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Shimon Cohen
    • 1
  • Nathan Intrator
    • 1
  1. 1.School of Computer ScienceTel Aviv UniversityIsrael

Personalised recommendations