Data Mining and Knowledge Discovery

, Volume 28, Issue 5–6, pp 1503–1529 | Cite as

A peek into the black box: exploring classifiers by randomization

  • Andreas Henelius
  • Kai Puolamäki
  • Henrik Boström
  • Lars Asker
  • Panagiotis Papapetrou


Classifiers are often opaque and cannot easily be inspected to gain understanding of which factors are of importance. We propose an efficient iterative algorithm to find the attributes and dependencies used by any classifier when making predictions. The performance and utility of the algorithm is demonstrated on two synthetic and 26 real-world datasets, using 15 commonly used learning algorithms to generate the classifiers. The empirical investigation shows that the novel algorithm is indeed able to find groupings of interacting attributes exploited by the different classifiers. These groupings allow for finding similarities among classifiers for a single dataset as well as for determining the extent to which different classifiers exploit such interactions in general.


Classifiers Randomization 



AH and KP were partly supported by the Revolution of Knowledge Work project, funded by Tekes. HB and LA were partly supported by the project High-Performance Data Mining for Drug Effect Detection at Stockholm University, funded by Swedish Foundation for Strategic Research under grant IIS11-0053.


  1. Andrews R, Diederich J, Tickle AB (1995) Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowl Based Syst 8(6):373–389CrossRefGoogle Scholar
  2. Bache K, Lichman M (2014) UCI machine learning repository.
  3. Breiman L (2001) Random forests. Mach Learn 45(1):5–32CrossRefzbMATHGoogle Scholar
  4. Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression trees. Wadsworth, BelmontzbMATHGoogle Scholar
  5. Chanda P, Cho YR, Zhang A, Ramanathan M (2009) Mining of attribute interactions using information theoretic metrics. In: IEEE International Conference on Data Mining Workshops, pp 350–355Google Scholar
  6. Clark P, Niblett T (1989) The CN2 induction algorithm. Mach Learn 3(4):261–283Google Scholar
  7. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273–293zbMATHGoogle Scholar
  8. De Bie T (2011a) An information theoretic framework for data mining. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, New York, NY, KDD ’11, pp 564–572Google Scholar
  9. De Bie T (2011b) Maximum entropy models and subjective interestingness: an application to tiles in binary databases. Data Min Knowl Discov 23(3):407–446CrossRefzbMATHMathSciNetGoogle Scholar
  10. Domingos P, Pazzani MJ (1997) On the optimality of the simple bayesian classifier under zero-one loss. Machine Learning 29(2–3):103–130CrossRefzbMATHGoogle Scholar
  11. Freitas AA (2001) Understanding the crucial role of attribute interaction in data mining. Artif Intell Rev 16(3):177–199CrossRefzbMATHMathSciNetGoogle Scholar
  12. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. ACM SIGKDD Explor Newsl 11(1):10–18CrossRefGoogle Scholar
  13. Hanhijärvi S, Ojala M, Vuokko N, Puolamäki K, Tatti N, Mannila H (2009) Tell me something i don’t know: Randomization strategies for iterative data mining. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, New York, NY, KDD ’09, pp 379–388Google Scholar
  14. Henelius A, Korpela J, Puolamäki K (2013) Explaining interval sequences by randomization. In: Blockeel H, Kersting K, Nijssen S, Z̆elezný Filip (eds) Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, vol 8188, pp 337–352Google Scholar
  15. Hornik K, Buchta C, Zeileis A (2009) Open-source machine learning: R meets Weka. Comput Stat 24(2):225–232. doi: 10.1007/s00180-008-0119-7 CrossRefzbMATHMathSciNetGoogle Scholar
  16. Ishibuchi H, Nojima Y (2007) Analysis of interpretability-accuracy tradeoff of fuzzy systems by multiobjective fuzzy genetics-based machine learning. Int J Approx Reason 44(1):4–31CrossRefzbMATHMathSciNetGoogle Scholar
  17. Jakulin A, Bratko I, Smrke D, Demsar J, Zupan B (2003) Attribute interactions in medical data analysis. In: 9th Conference on Artificial Intelligence in Medicine in Europe, pp 229–238Google Scholar
  18. Janitza S, Strobl C, Boulesteix AL (2013) An auc-based permutation variable importance measure for random forests. BMC Bioinform 14:119CrossRefGoogle Scholar
  19. Johansson U, König R, Niklasson L (2003) Rule extraction from trained neural networks using genetic programming. In: 13th International Conference on Artificial Neural Networks, pp 13–16Google Scholar
  20. Liaw A, Wiener M (2002) Classification and regression by randomforest. R News 2(3):18–22,
  21. Lijffijt J, Papapetrou P, Puolamäki K (2014) A statistical significance testing approach to mining the most informative set of patterns. Data Min Knowl Discov 28:238–263. doi: 10.1007/s10618-012-0298-2 CrossRefzbMATHMathSciNetGoogle Scholar
  22. Misra G, Golshan B, Terzi E (2012) A Framework for Evaluating the Smoothness of Data-Mining Results. In: Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases, vol II, pp 660–675Google Scholar
  23. Ojala M, Garriga GC (2010) Permutation tests for studying classier performance. J Mach Learn Res 11:1833–1863zbMATHMathSciNetGoogle Scholar
  24. Plate T (1999) Accuracy versus interpretability in flexible modeling: implementing a tradeoff using gaussian process models. Behaviormetrika 26:29–50CrossRefGoogle Scholar
  25. Pulkkinen P, Koivisto H (2008) Fuzzy classifier identification using decision tree and multiobjective evolutionary algorithms. Int J Approx Reason 48(2):526–543CrossRefGoogle Scholar
  26. Quinlan JR (1986) Induction of decision trees. Mach Learn 1:81–106Google Scholar
  27. R Core Team (2014) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria,
  28. Segal MR, Cummings MP, Hubbard AE (2001) Relating amino acid sequence to phenotype: analysis of peptide-binding data. Biometrics 57(2):632–643CrossRefzbMATHMathSciNetGoogle Scholar
  29. Strobl C, Boulesteix AL, Zeileis A, Hothorn T (2007) Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics 8(25):Google Scholar
  30. Strobl C, Boulesteix AL, Kneib T, Augustin T, Zeileis A (2008) Conditional variable importance for random forests. BMC Bioinform 9:307CrossRefGoogle Scholar
  31. Wickham H, Chang W (2014) devtools: Tools to make developing R code easier., r package version 1.5
  32. Zacarias OP, Boström H (2013) Comparing support vector regression and random forests for predicting malaria incidence in Mozambique. In: International conference on advances in ICT for Emerging regions, IEEE, pp 217–221Google Scholar
  33. Zhao Z, Liu H (2007) Searching for interacting features. In: Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp 1156–1161Google Scholar
  34. Zhao Z, Liu H (2009) Searching for interacting features in subset selection. Intell Data Anal 13(2):207–228Google Scholar

Copyright information

© The Author(s) 2014

Authors and Affiliations

  • Andreas Henelius
    • 1
  • Kai Puolamäki
    • 1
  • Henrik Boström
    • 2
  • Lars Asker
    • 2
  • Panagiotis Papapetrou
    • 2
  1. 1.Finnish Institute of Occupational HealthHelsinkiFinland
  2. 2.Department of Computer and Systems SciencesStockholm UniversityKistaSweden

Personalised recommendations