Meta-mining Evaluation Framework: A Large Scale Proof of Concept on Meta-learning

  • William RaynautEmail author
  • Chantal Soule-Dupuy
  • Nathalie Valles-Parlangeau
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9992)


This paper aims to provide a unified framework for the evaluation and comparison of the many emergent meta-mining techniques. This framework is illustrated on the case study of the meta-learning problem in a large scale experiment. The results of this experiment are then explored through hypothesis testing in order to provide insight regarding the performance of the different meta-learning schemes, advertising the potential of our approach regarding meta-level knowledge discovery.


  1. 1.
    Brazdil, P., Gama, J., Henery, B.: Characterizing the applicability of classification algorithms using meta-level learning. In: Bergadano, F., Raedt, L. (eds.) ECML 1994. LNCS, vol. 784, pp. 83–102. Springer, Heidelberg (1994). doi: 10.1007/3-540-57868-4_52 CrossRefGoogle Scholar
  2. 2.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Dong, L., Frank, E., Kramer, S.: Ensembles of balanced nested dichotomies for multi-class problems. In: Jorge, A.M., Torgo, L., Brazdil, P., Camacho, R., Gama, J. (eds.) PKDD 2005. LNCS (LNAI), vol. 3721, pp. 84–95. Springer, Heidelberg (2005). doi: 10.1007/11564126_13 CrossRefGoogle Scholar
  4. 4.
    Giraud-Carrier, C., Vilalta, R., Brazdil, P.: Introduction to the special issue on meta-learning. Mach. Learn. 54(3), 187–193 (2004)CrossRefGoogle Scholar
  5. 5.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: an update. ACM SIGKDD Explor. Newsl. 11(1), 10–18 (2009)CrossRefGoogle Scholar
  6. 6.
    Hilario, M., Nguyen, P., Do, H., Woznica, A., Kalousis, A.: Ontology-based meta-mining of knowledge discovery workflows. In: Jankowski, N., Duch, W., Gra̧bczewski, K. (eds.) Meta-Learning in Computational Intelligence. Studies in Computational Intelligence, vol. 358, pp. 273–315. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  7. 7.
    Kalousis, A., Hilario, M.: Model selection via meta-learning: a comparative study. Int. J. Artif. Intell. Tools 10(04), 525–554 (2001)CrossRefzbMATHGoogle Scholar
  8. 8.
    Kalousis, A., Hilario, M.: Feature selection for meta-learning. In: Cheung, D., Williams, G.J., Li, Q. (eds.) PAKDD 2001. LNCS (LNAI), vol. 2035, pp. 222–233. Springer, Heidelberg (2001). doi: 10.1007/3-540-45357-1_26 CrossRefGoogle Scholar
  9. 9.
    King, R.D., Feng, C., Sutherland, A.: StatLog: comparison of classification algorithms on large real-world problems. Int. J. Appl. Artif. Intell. 9(3), 289–333 (1995)CrossRefGoogle Scholar
  10. 10.
    Kononenko, I., Bratko, I.: Information-based evaluation criterion for classifier’s performance. Mach. Learn. 6(1), 67–80 (1991)Google Scholar
  11. 11.
    Leite, R., Brazdil, P., Vanschoren, J.: Selecting classification algorithms with active testing. In: Perner, P. (ed.) MLDM 2012. LNCS (LNAI), vol. 7376, pp. 117–131. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-31537-4_10 CrossRefGoogle Scholar
  12. 12.
    Leyva, E., Gonzalez, A., Perez, R.: A set of complexity measures designed for applying meta-learning to instance selection. IEEE Trans. Knowl. Data Eng. 27(2), 354–367 (2015)CrossRefGoogle Scholar
  13. 13.
    Michie, D., Spiegelhalter, D.J., Taylor, C.C.: Machine Learning, Neural and Statistical Classification. Ellis Horwood, Upper Saddle River (1994)zbMATHGoogle Scholar
  14. 14.
    Misir, M., Sebag, M.: Algorithm selection as a collaborative filtering problem, p. 43 (2013). hal-00922840
  15. 15.
    Nguyen, P., Hilario, M., Kalousis, A.: Using meta-mining to support data mining workflow planning and optimization. J. Artif. Intell. Res. 51, 605–644 (2014)Google Scholar
  16. 16.
    Peng, Y., Flach, P.A., Brazdil, P., Soares, C.: Decision tree-based data characterization for meta-learning. In: IDDM-2002 p. 111 (2002)Google Scholar
  17. 17.
    Pfahringer, B., Bensusan, H., Giraud-Carrier, C.: Tell me who can learn you and i can tell you who you are: landmarking various learning algorithms. In: Proceedings of the 17th International Conference on Machine Learning, pp. 743–750 (2000)Google Scholar
  18. 18.
    Platt, J., et al.: Sequential minimal optimization: A fast algorithm for training support vector machines (1998)Google Scholar
  19. 19.
    Robnik-Šikonja, M., Kononenko, I.: Theoretical and empirical analysis of relieff and rrelieff. Mach. Learn. 53(1–2), 23–69 (2003)CrossRefzbMATHGoogle Scholar
  20. 20.
    Serban, F., Vanschoren, J., Kietz, J.U., Bernstein, A.: A survey of intelligent assistants for data analysis. ACM Comput. Surv. (CSUR) 45(3), 31 (2013)CrossRefGoogle Scholar
  21. 21.
    Sun, Q., Pfahringer, B.: Pairwise meta-rules for better meta-learning-based algorithm ranking. Mach. Learn. 93(1), 141–161 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Sun, Q., Pfahringer, B., Mayo, M.: Full model selection in the space of data mining operators. In: Proceedings of the 14th Annual Conference Companion on Genetic and Evolutionary Computation, pp. 1503–1504. ACM (2012)Google Scholar
  23. 23.
    Todorovski, L., Brazdil, P., Soares, C.: Report on the experiments with feature selection in meta-level learning. In: Proceedings of the PKDD 2000 Workshop on Data Mining, Decision Support, Meta-learning and ILP: Forum For Practical Problem Presentation and Prospective Solutions, pp. 27–39. Citeseer (2000)Google Scholar
  24. 24.
    Uno, T., Asai, T., Uchida, Y., Arimura, H.: An efficient algorithm for enumerating closed patterns in transaction databases. In: Suzuki, E., Arikawa, S. (eds.) DS 2004. LNCS (LNAI), vol. 3245, pp. 16–31. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-30214-8_2 CrossRefGoogle Scholar
  25. 25.
    Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: networked science in machine learning. SIGKDD Explor. 15(2), 49–60 (2013). CrossRefGoogle Scholar
  26. 26.
    Vilalta, R., Drissi, Y.: A perspective view and survey of meta-learning. Artif. Intell. Rev. 18(2), 77–95 (2002). CrossRefGoogle Scholar
  27. 27.
    Xu, L., Hutter, F., Shen, J., Hoos, H.H., Leyton-Brown, K.: SATzilla2012: improved algorithm selection based on cost-sensitive classification models, pp. 57–58 (2012)Google Scholar
  28. 28.
    Zakova, M., Kremen, P., Zelezny, F., Lavrac, N.: Automating knowledge discovery workflow composition through ontology-based planning. IEEE Trans. Autom. Sci. Eng. 8(2), 253–264 (2011)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • William Raynaut
    • 1
    Email author
  • Chantal Soule-Dupuy
    • 1
  • Nathalie Valles-Parlangeau
    • 1
  1. 1.IRIT UMR 5505, UT1, UT3, Universite de ToulouseToulouseFrance

Personalised recommendations