Lazy Meta-Learning: Creating Customized Model Ensembles on Demand

  • Piero P. Bonissone
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7311)


In the not so distant future, we expect analytic models to become a commodity. We envision having access to a large number of data-driven models, obtained by a combination of crowdsourcing, crowdservicing, cloud-based evolutionary algorithms, outsourcing, in-house development, and legacy models. In this new context, the critical question will be model ensemble selection and fusion, rather than model generation. We address this issue by proposing customized model ensembles on demand, inspired by Lazy Learning. In our approach, referred to as Lazy Meta-Learning, for a given query we find the most relevant models from a DB of models, using their meta-information. After retrieving the relevant models, we select a subset of models with highly uncorrelated errors. With these models we create an ensemble and use their meta-information for dynamic bias compensation and relevance weighting. The output is a weighted interpolation or extrapolation of the outputs of the models ensemble. Furthermore, the confidence interval around the output is reduced as we increase the number of uncorrelated models in the ensemble. We have successfully tested this approach in a power plant management application.


Machine learning lazy learning meta-learning computational intelligence fusion ensemble entropy Pareto set neural networks coal-fired power plant management 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bonissone, P.: The life cycle of a fuzzy knowledge-based classifier. In: Proc. North American Fuzzy Information Processing Society (NAFIPS 2003), Chicago, IL, pp. 488–494 (2003)Google Scholar
  2. 2.
    Patterson, A., Bonissone, P., Pavese, M.: Six Sigma Quality Applied Throughout the Lifecycle of and Automated Decision System. Journal of Quality and Reliability Engineering International 21(3), 275–292 (2005)CrossRefGoogle Scholar
  3. 3.
    Bonissone, P., Varma, A., Aggour, K.: An Evolutionary Process for Designing and Maintaining a Fuzzy Instance-based Model (FIM). In: Proc. First Workshop of Genetic Fuzzy Systems (GFS 2005), Granada, Spain (2005)Google Scholar
  4. 4.
    Bonissone, P., Varma, A., Aggour, K., Xue, F.: Design of local fuzzy models using evolutionary algorithms. Computational Statistics and Data Analysis 51, 398–416 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Bonissone, P.: Soft Computing: A Continuously Evolving Concept. Int. J. Computational Intelligence Systems 3(2), 237–248 (2010)CrossRefGoogle Scholar
  6. 6.
    Bonissone, P., Cadenas, J.M., Garrido, M.C., Diaz, R.A.: A Fuzzy Random Forest. International Journal of Approximate Reasoning 51(7), 729–747 (2010), doi:10.1016/j.ijar.2010.02.003MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Bonissone, P., Xue, F., Subbu, R.: Fast Meta-models for Local Fusion of Multiple Predictive Models. Applied Soft Computing Journal 11(2), 1529–1539 (2008), doi:10.1016/j.asoc.2008.03.006CrossRefGoogle Scholar
  8. 8.
    Bonissone, P., Eklund, N., Goebel, K.: Using an Ensemble of Classifiers to Audit a Production Classifier. In: Oza, N.C., Polikar, R., Kittler, J., Roli, F. (eds.) MCS 2005. LNCS, vol. 3541, pp. 376–386. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Evangelista, P., Embrechts, M., Bonissone, P., Szymanski, B.: Fuzzy ROC Curves for Unsupervised Nonparametric Ensemble Techniques. In: IJCNN 2005, Montreal, Canada, pp. 3040–3045 (2005)Google Scholar
  10. 10.
    Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., Zaharia, M.: Above the Clouds: A Berkeley View of Cloud Computing. Technical Report EECS-2009-28, EECS Department, University of California, Berkeley (2009)Google Scholar
  11. 11.
    Howe, J.: Crowdsourcing- Why the Power of the Crowd Is Driving the Future of Business. Random House, New York (2008)Google Scholar
  12. 12.
    Vance, A.: Kaggle’s Contests: Crunching Numbers for Fame and Glory. Businessweek, January 04 (2012)Google Scholar
  13. 13.
    Bonissone, P., Subbu, R., Aggour, K.: Evolutionary Optimization of Fuzzy Decision Systems for Automated Insurance Underwriting. In: Proc. FUZZ-IEEE 2002, Honolulu, HI, pp. 1003–1008 (2002)Google Scholar
  14. 14.
    Aggour, K., Bonissone, P., Cheetham, W., Messmer, R.: Automating the Underwriting of Insurance Applications. AI Magazine 27(3), 36–50 (2006)Google Scholar
  15. 15.
    Bennett III, F.H., Koza, J.R., Shipman, J., Stiffelman, O.: Building a parallel computer system for $18,000 that performs a half peta-flop per day. In: Banzhaf, W., Daida, J., Eiben, A.E., Garzon, M.H., Honavar, V., Jakiela, M., Smith, R.E. (eds.) GECCO 1999: Proceedings of the Genetic and Evolutionary Computation Conference, Orlando, FL, pp. 1484–1490. Morgan Kaufmann, San Francisco (1999)Google Scholar
  16. 16.
    Sherry, D., Veeramachaneni, K., McDermott, J., O’Reilly, U.-M.: FlexGP: Genetic Programming on the Cloud. To Appear in Parallel Implementation of Evolutionary Algorithms, EvoStar 2012, Malaga, Spain (2012)Google Scholar
  17. 17.
    Bonissone, P.P., Iyer, N.: Soft Computing Applications to Prognostics and Health Management (PHM): Leveraging Field Data and Domain Knowledge. In: Sandoval, F., Prieto, A.G., Cabestany, J., Graña, M. (eds.) IWANN 2007. LNCS, vol. 4507, pp. 928–939. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  18. 18.
    Roli, F., Giacinto, G., Vernazza, G.: Methods for Designing Multiple Classifier Systems. In: Kittler, J., Roli, F. (eds.) MCS 2001. LNCS, vol. 2096, pp. 78–87. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  19. 19.
    Kuncheva, L.: Switching between selection and fusion in combining classifiers: An experiment. IEEE Transactions on Systems, Man, and Cybernetics, Part B 32(2), 146–156 (2002)CrossRefGoogle Scholar
  20. 20.
    Tumer, K., Ghosh, J.: Error correlation and error reduction in ensemble classifiers. Connection Science 8, 385–404 (1996)CrossRefGoogle Scholar
  21. 21.
    Kuncheva, L., Whitaker, C.: Ten measures of diversity in classifier ensembles: Limits for two classifiers. In: Proceedings of IEE Workshop on Intelligent Sensor Processing, Birmingham, p. 10/1-6 (2001)Google Scholar
  22. 22.
    Wolpert, D.H.: Stacked generalization. Neural Networks 5, 241–259 (1992)CrossRefGoogle Scholar
  23. 23.
  24. 24.
    Duch, W., Grudzinski, K.: Meta-learning: searching in the model space. In: Proc. of the Int. Conf. on Neural Information Processing (ICONIP), Shanghai, China (2001)Google Scholar
  25. 25.
    Schaul, T., Schmidhuber, J.: Meta-learning. Scholarpedia 5(6), 4650 (2010)CrossRefGoogle Scholar
  26. 26.
    Utgoff, P.: Shift of bias for inductive concept learning. In: Michalski, R., Carbonell, J., Mitchell, T. (eds.) Machine Learning, pp. 163–190 (1986)Google Scholar
  27. 27.
    Schmidhuber, J.: Evolutionary principles in self-referential learning. Diploma thesis, Institut für Informatik, Technische Universität München (1987)Google Scholar
  28. 28.
    Atkeson, C.G.: Memory-based approaches to approximating continuous functions. In: Casdagli, M., Eubank, S. (eds.) Nonlinear Modeling and Forecasting, pp. 503–521. Addison Wesley, Harlow (1992)Google Scholar
  29. 29.
    Atkeson, C.G., Moore, A., Schaal, S.: Locally Weighted Learning. Artificial Intelligence Review 11(1-5), 11–73 (1997)CrossRefGoogle Scholar
  30. 30.
    Bersini, H., Bontempi, G., Birattari, M.: Is readability compatible with accuracy? From neuro-fuzzy to lazy learning. In: Freksa, C. (ed.) Proceedings in Artificial Intelligence 7, pp. 10–25. Infix/Aka, Berlin (1998)Google Scholar
  31. 31.
    Deb, K.: Multi-objective optimization using evolutionary algorithms. J. Wiley (2001)Google Scholar
  32. 32.
    Coello Coello, C.A., Van Veldhuizen, D.A., Lamont, G.B.: Evolutionary Algorithm MOP Approaches, Evolutionary Algorithms for Solving Multi-Objective Problems. Kluwer Academic (2002)Google Scholar
  33. 33.
    Bonissone, P., Subbu, R., Lizzi, J.: Multi Criteria Decision Making (MCDM): A Framework for Research and Applications. IEEE Computational Intelligence Magazine 4(3), 48–61 (2009)CrossRefGoogle Scholar
  34. 34.
    Breiman, L., Friedman, J., Olshen, R.A., Stone, C.J.: Classification and regression trees. Wadsworth (1984)Google Scholar
  35. 35.
    Bonissone, P., Cheetham, W.: Fuzzy Case-Based Reasoning for Decision Making. In: Proc. FUZZ-IEEE 2001, Melbourne, Australia, vol. 3, pp. 995–998 (2001)Google Scholar
  36. 36.
    Xue, F., Subbu, R., Bonissone, P.: Locally Weighted Fusion of Multiple Predictive Models. In: IEEE International Joint Conference on Neural Networks (IJCNN 2006), Vancouver, BC, Canada, pp. 2137–2143 (2006), doi:10.1109/IJCNN.2006.246985Google Scholar
  37. 37.
    Yan, W., Xue, F.: Jet Engine Gas Path Fault Diagnosis Using Dynamic Fusion of Multiple Classifiers. In: IJCNN 2008, Hong Kong, pp. 1585–1591 (2008)Google Scholar
  38. 38.
    Bentley, J.L.: Multidimensional binary search trees used for associative searching. Communications of the ACM 18(9), 509–517 (1975)CrossRefzbMATHGoogle Scholar
  39. 39.
    Gray, A., Moore, A.: N-Body Problems in Statistical Learning. In: Proc. Advances in Neural Information Processing Systems, NIPS (2001)Google Scholar
  40. 40.
    Subbu, R., Bonissone, P., Eklund, N., Yan, W., Iyer, N., Xue, F., Shah, R.: Management of Complex Dynamic Systems based on Model-Predictive Multi-objective Optimization. In: CIMSA 2006, La Coruña, Spain, pp. 64–69 (2006)Google Scholar
  41. 41.
    Subbu, R., Bonissone, P., Bollapragada, S., Chalermkraivuth, K., Eklund, N., Iyer, N., Shah, R., Xue, F., Yan, W.: A review of two industrial deployments of multi-criteria decision-making systems at General Electric. In: First IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making (MCDM 2007), Honolulu, Hawaii (2007), doi:10.1109/MCDM.2007.369428Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Piero P. Bonissone
    • 1
  1. 1.GE Global Research CenterNiskayunaUSA

Personalised recommendations