Abstract
This chapter investigates a specific ensemble learning approach by negative correlation learning (NCL) [21, 22, 23]. NCL is an ensemble learning algorithm which considers the cooperation and interaction among the ensemble members. NCL introduces a correlation penalty term into the cost function of each individual learner so that each learner minimizes its mean-square-error (MSE) error together with the correlation with other ensemble members.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
μi, i = 1, 2, …M, is the inverse variance of the Gaussian distribution of weights for network i.
- 2.
Since we optimize αi for each individual networks in the ensemble, in this figure we only show the mean αi value.
- 3.
To generate an initial RBF network population: Generate an initial population of M RBF Networks, the number of hidden nodes K for each network is specified randomly restricted by the maximal number of hidden nodes. The centers μk are initialized with randomly selected data points from the training set and the width σk are determined as the Euclidian distance between μk and the closest μj(j≠k, j ∈ { 1, …, K}).
- 4.
Choose parents based on roulette wheel selection algorithm and perform crossover. Then perform a few number of updates for weights, centers, and widths. Compare the children with parents and keep the better ones.
- 5.
The raw fitness values depend on their ranked layers (fronts) in the population. If they are in the same layer (front), e.g., they are both nondominant solutions, the one in the less-crowded area will receive greater fitness according to the fitness sharing algorithm.
- 6.
Negative correlation was used to indicate the correlation between on individual’s error with the error of the rest of the ensemble. By minimizing the correlation term, i.e., \(-{\sum \nolimits }_{n=1}^{N}{({f}_{i}({\mathbf{x}}_{n}) - {f}_{\mathrm{ens}}({\mathbf{x}}_{n}))}^{2}\), the individual in the population will be more diverse, i.e., the term \({\sum \nolimits }_{n=1}^{N}{({f}_{i}({\mathbf{x}}_{n}) - {f}_{\mathrm{ens}}({\mathbf{x}}_{n}))}^{2}\) increases. Therefore, the average training error term \({\sum \nolimits }_{n=1}^{N}{({f}_{i}({\mathbf{x}}_{n}) - {y}_{n})}^{2}\) will increase.
References
C. M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, USA, 1996.
G. Brown, J. Wyatt, R. Harris, and X. Yao. Diversity creation methods: A survey and categorisation. Journal of Information Fusion, 6(1):5–20, 2005.
G. Brown, J. Wyatt, and P. Ti\(\mathrm{\check{n}}\)o. Managing diversity in regression ensembles. Journal of Machine Learning Research, 6:1621–1650, 2005.
A. Chandra and X. Yao. Ensemble learning using multi-objective evolutionary algorithms. Journal of Mathematical Modelling and Algorithms, 5(4):417–445, 2006.
A. Chandra and X. Yao. Ensemble learning using multi-objective evolutionary algorithms. Journal of Mathematical Modelling and Algorithms, 5(4):417–445, 2006.
H. Chen, P. Ti\(\mathrm{\check{n}}\)o, and X. Yao. Predictive ensemble pruning by expectation propagation. IEEE Transactions on Knowledge and Data Engineering, 21(7):999–1013, 2009.
H. Chen and X. Yao. Evolutionary random neural ensemble based on negative correlation learning. In Proceedings of IEEE Congress on Evolutionary Computation (CEC’07), pp. 1468–1474, 2007.
H. Chen and X. Yao. Regularized negative correlation learning for neural network ensembles. IEEE Transactions on Neural Networks, 20(12):1962–1979, 2009.
H. Chen and X. Yao. Multiobjective regularized negative correlation learning for neural network ensembles. IEEE Transactions on Knowledge and Data Engineering, 22(12): 1738–1751, 2010.
H. H. Dam, H. A. Abbass, C. Lokan, and X. Yao. Neural-based learning classifier systems. IEEE Transactions on Knowledge and Data Engineering, 20(1):26–39, 2008.
P. Darwen and X. Yao. Every niching method has its niche: fitness sharing and implicit sharing compared. In Proceedings of Parallel Problem Solving from Nature (PPSN) IV, volume 1141, pp. 398–407, Berlin, Germany, 1996.
S. Geman, E. Bienenstock, and R. Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4(1):1–58, 1992.
T. Van Gestel, J. A. K. Suykens, G. Lanckriet, A. Lambrechts, B. De Moor, and J. Vandewalle. Bayesian framework for least-squares support vector machine classifiers, gaussian processes, and kernel fisher discriminant analysis. Neural Computation, 14(5):1115–1147, 2002.
T. K. Ho. The random subspace method for constructing decision forests. IEEE Transaction on Pattern Analysis and Machine Intelligence, 20(8):832–844, 1998.
M. M. Islam, X. Yao, and K. Murase. A constructive algorithm for training cooperative neural network ensembles. IEEE Transaction on Neural Networks, 14(4):820–834, 2003.
Y. Jin, T. Okabe, and B. Sendhoff. Neural network regularization and ensembling using multi-objective evolutionary algorithms. In Proceedings of IEEE Congress on Evolutionary Computation (CEC’04), pages 1–8, 2004.
A. Krogh and J. A. Hertz. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems, volume 4, pp. 950–957, 1992.
A. Krogh and J. A. Hertz. A simple weight decay can improve generalization. Advances in Neural Information Processing Systems, pp. 950–950, 1993.
A. Krogh and J. Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems 7, pp. 231–238, Denver, Colorado, USA, 1995.
Y. Liu and X. Yao. Negatively correlated neural networks can produce best ensembles. In Australian Journal of Intelligent Information Processing Systems 4(3/4), pp. 176–185, 1997.
Y. Liu and X. Yao. Ensemble learning via negative correlation. Neural Networks, 12(10): 1399–1404, 1999.
Y. Liu and X. Yao. Simultaneous training of negatively correlated neural networks in an ensemble. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 29(6):716–725, 1999.
Y. Liu, X. Yao, and T. Higuchi. Evolutionary ensembles with negative correlation learning. IEEE Transaction on Evolutionary Computation, 4(4):380–387, 2000.
D. J. C. MacKay. Bayesian interpolation. Neural Computation, 4(3):415–447, 1992.
R. McKay and H. Abbass. Analyzing anticorrelation in ensemble learning. In Proceedings of 2001 Conference on Australian Artificial Neural Networks and Expert Systems, pp. 22–27, 2001.
F. L. Minku, H. Inoue, and X. Yao. Negative correlation in incremental learning. Natural Computing, 8(2):289–320, 2009.
L. L. Minku, A. White, and X. Yao. The impact of diversity on on-line ensemble learning in the presence of concept drift. IEEE Transactions on Knowledge and Data Engineering, 22(5): 730–742, 2010.
M. F. Møller. A scaled conjugate gradient algorithm for fast supervised learning. Neural Network, 6(4):525–533, 1993.
R. M. Neal. Bayesian learning for neural networks. Springer, New York, 1996.
B. D. Ripley. Pattern Recognition and Neural Networks. Cambridge University Press, UK, 1996.
N. Srinivas and K. Deb. Multiobjective function optimization using nondominated sorting genetic algorithms. Evolutionary Computation, 2(3):221–248, 1995.
V. N. Vapnik. The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995.
S. Wang, H. Chen, and X. Yao. Negative correlation learning for classification ensembles. In Proceedings of the 2010 International Joint Conference on Neural Networks (IJCNN’10), pp. 2893–2900, 2010.
S. Wang and X. Yao. Diversity analysis on imbalanced data sets by using ensemble models. In Proceedings of the 2009 IEEE Symposium on Computational Intelligence and Data Mining (CIDM’09), pp. 324–331, 2009.
S. Wang and X. Yao. Diversity exploration and negative correlation learning on imbalanced data sets. In Proceedings of the 2009 International Joint Conference on Neural Networks (IJCNN’09), pp. 3259–3266, 2009.
X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423–1447, 1999.
Acknowledgements
This work has been funded by the European Commission’s 7th Framework Program, under grant Agreement INSFO-ICT-270428 (iSense).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Chen, H., Cohn, A.G., Yao, X. (2012). Ensemble Learning by Negative Correlation Learning. In: Zhang, C., Ma, Y. (eds) Ensemble Machine Learning. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-9326-7_6
Download citation
DOI: https://doi.org/10.1007/978-1-4419-9326-7_6
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4419-9325-0
Online ISBN: 978-1-4419-9326-7
eBook Packages: EngineeringEngineering (R0)