Advertisement

Random Forests

Living reference work entry
Part of the Springer Reference Naturwissenschaften book series (SRN)

Zusammenfassung

Random Forests und deren Varianten gehören zu den erfolgreichsten Methoden des maschinellen Lernens. Ihre Einfachheit, Effizienz, Robustheit, Genauigkeit und Allgemeinheit führten sowohl zu mannigfaltigen Adaptionen des zugrunde-liegenden Konzepts als auch zu vielen erfolgreichen Anwendungen auf verschiedene Problemstellungen. Dieser Artikel versucht einen Überblick über Random Forests zu schaffen, stellt deren Ursprünge dar, erklärt grundlegende Konzepte und potentielle Erweiterungen, diskutiert Vor- und Nachteile und erwähnt einige der einflussreichsten Anwendungen im Bereich der digitalen Bildanalyse.

Schlüsselwörter:

Random forests Ensemble learning Classification and regression trees Tree ensembles Semantic image analysis 

Literatur

  1. 1.
    Ah-Pine, J.: Data fusion in information retrieval using consensus aggregation operators. In: International Conference on Web Intelligence and Intelligent Agent Technology, Bd. 1, S. 662–668 (2008)Google Scholar
  2. 2.
    Ahmad, A., Brown, G.: Random ordinality ensembles – a novel ensemble method for multi-valued categorical data. In: Proceedings of the 8th International Workshop on Multiple Classifier Systems, S. 222–231 (2009)Google Scholar
  3. 3.
    Alpaydin, E., Jordan, M.I.: Local linear perceptrons for classification. IEEE Trans. Neural Netw. 7(3), 788–792 (1996)CrossRefGoogle Scholar
  4. 4.
    Amit, Y., Geman, D.: Shape quantization and recognition with randomized trees. Neural Comput. 9, 1545–1588 (1997)CrossRefGoogle Scholar
  5. 5.
    Banfield, R., Hall, L., Bowyer, K., Kegelmeyer, W.: Ensemble diversity measures and their application to thinning. Inf. Fusion 6(1), 49–62 (2005)CrossRefGoogle Scholar
  6. 6.
    Battiti, R., Colla, A.: Democracy in neural nets: voting schemes for classification. Neural Netw. 7, 691–707 (1994)CrossRefGoogle Scholar
  7. 7.
    Bauer, E., Kohavi, R.: An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach. Learn. 36, 105–139 (1999)CrossRefGoogle Scholar
  8. 8.
    Benediktsson, J., Swain, P.: Consensus theoretic classification methods. IEEE Trans. Syst. Man Cybern. 22, 688–704 (1992)CrossRefGoogle Scholar
  9. 9.
    Benediktsson, J., Sveinsson, J., Ingimundarson, J.I., Sigurdsson, H., Ersoy, O.: Multistage classifiers optimized by neural networks and genetic algorithms. Nonlinear Anal. Theory Methods Appl. 30, 1323–1334 (1997)CrossRefGoogle Scholar
  10. 10.
    Biau, G., Devroye, L., Lugosi, G.: Consistency of random forests and other averaging classifiers. J. Mach. Learn. Res. 9, 2015–2033 (2008)Google Scholar
  11. 11.
    Bloch, I.: Information combination operators for data fusion: a comparative review with classification. IEEE Trans. Syst. Man Cybern. – Part A: Syst. Hum. 26, 52–67 (1996)CrossRefGoogle Scholar
  12. 12.
    Bosch, A., Zisserman, A., Munoz, X.: Image classification using random forests and ferns. In: IEEE 11th International Conference on Computer Vision, S. 1–8 (2007)Google Scholar
  13. 13.
    Breiman, L.: Bagging predictors. Technical Report No. 421, Department of Statistics, University of California (1994)Google Scholar
  14. 14.
    Breiman, L.: Arcing classifiers. Technical Report, Department of Statistics, University of California (1996)Google Scholar
  15. 15.
    Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996)Google Scholar
  16. 16.
    Breiman, L.: Some infinity theory for predictor ensembles. Technical Report No. 577, Department of Statistics, University of California (2000)Google Scholar
  17. 17.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefGoogle Scholar
  18. 18.
    Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: CART: Classification and Regression Trees (1984)Google Scholar
  19. 19.
    Brown, G., Wyatt, J., Harris, R., Yao, X.: Diversity creation methods: A survey and categorisation. J. Inf. Fusion 6(1), 5–20 (2005)CrossRefGoogle Scholar
  20. 20.
    Bühlmann, P.: Bagging, boosting and ensemble methods. In: Handbook of Computational Statistics, S. 985–1022 (2012)Google Scholar
  21. 21.
    Buntine, W.: Learning classification trees. In: Artificial Intelligence Frontiers in Statistics, S. 182–2011 (1991)Google Scholar
  22. 22.
    Caruana, R., Karampatziakis, N., Yessenalina, A.: An empirical evaluation of supervised learning in high dimensions. In: Proceedings International Conference on Machine Learning, S. 96–103 (2008)Google Scholar
  23. 23.
    Caruana, R., Niculescu-Mizil, A.: An empirical comparison of supervised learning algorithms. In: Proceedings of the 23rd International Conference on Machine Learning (2006)Google Scholar
  24. 24.
    Chiang, C.C., Fu, H.C.: A divide-and-conquer methodology for modular supervised neural network design. In: IEEE International Conference on Neural Networks, S. 119–124 (1994)Google Scholar
  25. 25.
    Cho, S.B., Kim, J.H.: Combining multiple neural networks by fuzzy integral for robust classification. IEEE Trans. Syst. Man Cybern. 25(2), 380–384 (1995)CrossRefGoogle Scholar
  26. 26.
    Ciresan, D., Meier, U., Masci, J., Schmidhuber, J.: A committee of neural networks for traffic sign classification. In: The 2011 International Joint Conference on Neural Networks, IJCNN 2011, S. 1918–1921 (2011)Google Scholar
  27. 27.
    Cutler, A.: Fast classification using perfect random trees. Technical Report 5/99/99, Department of Mathematics and Statistics, Utah State University (1999)Google Scholar
  28. 28.
    Cutler, A., Zhao, G.: Pert – perfect random tree ensembles. In: Computing Science and Statistics (2001)Google Scholar
  29. 29.
    Dasarathy, B.V., Sheela, B.V.: Composite classifier system design: concepts and methodology. In: Proceedings of the IEEE, Bd. 67, S. 708–713 (1979)Google Scholar
  30. 30.
    Dietterich, T.G.: Ensemble methods in machine learning. In: Proceedings of the First International Workshop on Multiple Classifier Systems, S. 1–15. Springer (2000)Google Scholar
  31. 31.
    Dietterich, T.G., Bakiri, G.: Solving multiclass learning problems via error-correcting output codes. J. Artif. Intell. Res. 2, 263–286 (1995)Google Scholar
  32. 32.
    Dietterich, T.G., Fisher, D.: An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Mach. Learn. 40, 139–157 (2000)CrossRefGoogle Scholar
  33. 33.
    Drucker, H., Cortes, C., Jackel, L., LeCun, Y., Vapnik, V.: Boosting and other ensemble methods. Neural Comput. 6, 1289–1301 (1994)CrossRefGoogle Scholar
  34. 34.
    Duan, K., Keerthi, S.: Which is the best multiclass SVM method? An empirical study. In: Multiple Classifier Systems, S. 732–760 (2005)Google Scholar
  35. 35.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification (2000)Google Scholar
  36. 36.
    Filippi, E., Costa, M., Pasero, E.: Multi-layer perceptron ensembles for increased performance and fault-tolerance in pattern recognition tasks. In: IEEE International Conference on Neural Networks, S. 2901–2906 (1994)Google Scholar
  37. 37.
    Freund, Y.: Boosting a weak learning algorithm by majority. In: Proceedings of the Third Annual Workshop on Computational Learning Theory, S. 202–216 (1990)Google Scholar
  38. 38.
    Gader, P., Mohamed, M., Keller, J.: Fusion of handwritten word classifiers. Pattern Recognit. Lett. 17, 577–584 (1996)CrossRefGoogle Scholar
  39. 39.
    Gal-Or, M., May, J., Spangler, W.: Assessing the predictive accuracy of diversity measures with domain-dependent, asymmetric misclassification costs. Inf. Fusion 6(1), 37–48 (2005)CrossRefGoogle Scholar
  40. 40.
    Gall, J., Lempitsky, V.: Class-specific hough forests for object detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2009), S. 1022–1029 (2009)Google Scholar
  41. 41.
    Geurts, P., Wehenkel, D.E.L.: Extremely randomized trees. Mach. Learn. 63(1), 3–42 (2006)CrossRefGoogle Scholar
  42. 42.
    Giacinto, G., Roli, F.: Approach to the automatic design of multiple classifier systems. Pattern Recognit. Lett. 22(1), 25–33 (2001)CrossRefGoogle Scholar
  43. 43.
    Hall, L.O., Bowyer, K.W., Banfield, R.E., Bhadoria, D., Philip, W., Eschrich, S.: Comparing pure parallel ensemble creation techniques against bagging. In: The Third IEEE International Conference on Data Mining, S. 533–536 (2003)Google Scholar
  44. 44.
    Hänsch, R.: Generic object categorization in PolSAR images – and beyond, Technische Universität Berlin, Germany. Ph.D. thesis (2014)Google Scholar
  45. 45.
    Hänsch, R., Hellwich, O.: Performance assessment and interpretation of random forests by three-dimensional visualizations. In: International Conference on Information Visualization Theory and Applications (IVAPP 2015), S. 149–156 (2015)Google Scholar
  46. 46.
    Hansen, L.K., Salamon, P.: Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell. 12(10), 993–1001 (1990)CrossRefGoogle Scholar
  47. 47.
    Ho, T.K.: Random decision forest. In: Proceedings of the Third International Conference on Document Analysis and Recognition, S. 278–282 (1995)Google Scholar
  48. 48.
    Ho, T.K.: The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell. 20(8), 832–844 (1998)CrossRefGoogle Scholar
  49. 49.
    Ho, T.K., Hull, J., Srihari, S.: Decision combination in multiple classifier systems. IEEE Trans. Pattern Anal. Mach. Intell. 16(1), 66–75 (1994)CrossRefGoogle Scholar
  50. 50.
    Hunt, E.B.: Concept Learning: An Information Processing Problem (1962)CrossRefGoogle Scholar
  51. 51.
    Jacobs, R.A.: Methods for combining experts probability assessments. Neural Comput. 7, 867–888 (1995)CrossRefGoogle Scholar
  52. 52.
    Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural Comput. 3, 79–87 (1991)CrossRefGoogle Scholar
  53. 53.
    Jordan, M., Xu, L.: Convergence results for the em approach to mixtures of experts architectures. Neural Netw. 8, 1409–1431 (1995)CrossRefGoogle Scholar
  54. 54.
    Kass, G.V.: An exploratory technique for investigating large quantities of categorical data. Appl. Stat. 29(2), 119–127 (1980)CrossRefGoogle Scholar
  55. 55.
    Keller, J.M., Gader, P., Tahani, H., Chiang, J.H., Mohamed, M.: Advances in fuzzy integration for pattern recognition. Fuzzy Sets Syst. 65(2–3), 273–283 (1994)CrossRefGoogle Scholar
  56. 56.
    Kittler, J., Hatef, M., Duin, R., Mates, J.: On combining classifiers. IEEE Trans. Pattern Anal. Mach. Intell. 20(3), 226–239 (1998)CrossRefGoogle Scholar
  57. 57.
    Kohavi, R., Kunz, C.: Option decision trees with majority votes. In: Proceedings of the Fourteenth International Conference on Machine Learning, S. 161–169 (1997)Google Scholar
  58. 58.
    Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation and active learning. In: Advances in Neural Information Processing Systems, S. 231–238. MIT, Cambridge (1995)Google Scholar
  59. 59.
    Kuncheva, L.: A theoretical study on six classifier fusion strategies. IEEE Trans. Pattern Anal. Mach. Intell. 24(2), 281–286 (2002)CrossRefGoogle Scholar
  60. 60.
    Kuncheva, L.: Combining Pattern Classifiers, Methods and Algorithms (2004)Google Scholar
  61. 61.
    Kuncheva, L., Bezdek, J., Duin, R.: Decision templates for multiple classifier fusion: an experimental comparison. Pattern Recognit. 34(2), 299–314 (2001)CrossRefGoogle Scholar
  62. 62.
    Kuncheva, L., Whitaker, C.: Measures of diversity in classifier ensembles and their relationship with ensemble accuracy. Mach. Learn. 51(2), 181–207 (2003)CrossRefGoogle Scholar
  63. 63.
    Lam, L., Suen, C.: Optimal combination of pattern classifiers. Pattern Recognit. Lett. 16, 945–954 (1995)CrossRefGoogle Scholar
  64. 64.
    Leibe, B., Leonardis, A., Schiele, B.: Combined object categorization and segmentation with an implicit shape model. In: ECCV’04 Workshop on Statistical Learning in Computer Vision, S. 1–16 (2004)Google Scholar
  65. 65.
    Lepetit, V., Fua, P.: Keypoint recognition using randomized trees. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1465–1479 (2006)CrossRefGoogle Scholar
  66. 66.
    Lepetit, V., Lagger, P., Fua, P.: Randomized trees for real-time keypoint recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), Bd. 2, S. 775–781 (2005)Google Scholar
  67. 67.
    Lin, Y., Jeon, Y.: Random forests and adaptive nearest neighbors. J. Am. Stat. Assoc. 101–474 (2002)Google Scholar
  68. 68.
    Liu, Y., Yao, X.: Ensemble learning via negative correlation. Neural Netw. 12, 1399–1404 (1999)CrossRefGoogle Scholar
  69. 69.
    Masoudnia, S., Ebrahimpour, R.: Mixture of experts: a literature survey. Artif. Intell. Rev. 1–19 (2012)Google Scholar
  70. 70.
    Melville, P., Mooney, R.J.: Creating diversity in ensembles using artificial data. Inf. Fusion 6, 99–111 (2004)CrossRefGoogle Scholar
  71. 71.
    Messenger, R., Mandell, L.: A modal search technique for predictive nominal scale multivariate analysis. J. Am. Stat. Assoc. 67, 768–772 (1972)Google Scholar
  72. 72.
    Morgan, J., Sonquist, J.: Problems in the analysis of survey data, and a proposal. J. Am. Stat. Assoc. 58, 415–434 (1963)CrossRefGoogle Scholar
  73. 73.
    Moosmann, F., Nowak, E., Jurie, F.: Randomized clustering forests for image classification. IEEE Trans. Pattern Anal. Mach. Intell. 30(9), 1632–1646 (2008)CrossRefGoogle Scholar
  74. 74.
    Ng, K.C., Abramson, B.: Consensus diagnosis: a simulation study. IEEE Trans. Syst. Man Cybern. 22, 916–928 (1992)CrossRefGoogle Scholar
  75. 75.
    Nowlan, S., Hinton, G.: Evaluation of adaptive mixtures of competing experts. Adv. Neural Inf. Process. Syst. 3, 774–780 (1991)Google Scholar
  76. 76.
    Opitz, D., Shavlik, J.: Actively searching for an effective neural-network ensemble. Connect. Sci. 8, 337–354 (1996)CrossRefGoogle Scholar
  77. 77.
    Quinlan, J.: Discovering rules from large collections of examples: a case study (1979)Google Scholar
  78. 78.
    Quinlan, J.: Inductive knowledge acquisition: a case study (1987)Google Scholar
  79. 79.
    Quinlan, J.: C4.5: Programs for Machine Learning (1993)Google Scholar
  80. 80.
    Rodner, E., Denzler, J.: Learning with few examples by transferring feature relevance. In: Proceedings of the 31st DAGM Symposium on Pattern Recognition, S. 252–261 (2009)Google Scholar
  81. 81.
    Rodriguez, J.J., Kuncheva, L.I., Alonso, C.J.: Rotation forest: a new classifier ensemble method. IEEE Trans. Pattern Anal. Mach. Intell. 28(10), 1619–1630 (2006)CrossRefGoogle Scholar
  82. 82.
    Rogova, G.: Combining the results of several neural network classifiers. Neural Netw. 7(5), 777–781 (1994)CrossRefGoogle Scholar
  83. 83.
    Roli, F., Giacinto, G.: Design of multiple classifier systems. In: Hybrid Methods Pattern Recognition (2002)CrossRefGoogle Scholar
  84. 84.
    Rosen, B.: Ensemble learning using decorrelated neural networks. Connect. Sci. 8, 373–384 (1996)CrossRefGoogle Scholar
  85. 85.
    Ruta, D., Gabrys, B.: An overview of classifier fusion methods. Comput. Inf. Syst. 7, 1–10 (2000)Google Scholar
  86. 86.
    Ruta, D., Gabrys, B.: Classifier selection for majority voting. Inf. Fusion 6(1), 63–81 (2005)CrossRefGoogle Scholar
  87. 87.
    Schapire, R.E.: The strength of weak learnability. Mach. Learn. 5(2), 197–227 (1990)Google Scholar
  88. 88.
    Schlimmer, J., Fisher, D.: A case study of incremental concept induction. In: Proceedings of the Fifth National Conference on Artificial Intelligence, S. 135–141 (1986)Google Scholar
  89. 89.
    Schroff, F., Criminisi, A., Zisserman, A.: Object class segmentation using random forests. In: Proceedings of the British Machine Vision Conference, S. 1–10 (2008)Google Scholar
  90. 90.
    Shotton, J., Johnson, M., Cipolla, R.: Semantic texton forests for image categorization and segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2008), S. 1–8 (2008)Google Scholar
  91. 91.
    Strobl, C., Boulesteix, A.L., Kneib, T., Augustin, T., Zeileis, A.: Conditional variable importance for random forests. BMC Bioinform. 9(307), 1–11 (2008)Google Scholar
  92. 92.
    Tsymbal, A., Pechenizkiy, M., Cunningham, P.: Diversity in search strategies for ensemble feature selection. Inf. Fusion 6(1), 83–98 (2005)CrossRefGoogle Scholar
  93. 93.
    Utgoff, P.: Incremental induction of decision trees. Mach. Learn. 4(2), 161–186 (1989)CrossRefGoogle Scholar
  94. 94.
    Wang, W., Brakensiek, A., Rigoll, G.: Combination of multiple classifiers for handwritten word recognition. In: Proceedings of International Workshop on Frontiers in Handwriting Recognition (IWFHR 2002), S. 117–122 (2002)Google Scholar
  95. 95.
    Windeatt, T.: Diversity measures for multiple classifier system analysis and design. Inf. Fusion 6, 21–36 (2005)CrossRefGoogle Scholar
  96. 96.
    Wolpert, D.H.: Stacked generalization. Neural Netw. 5(3), 241–259 (1992)CrossRefGoogle Scholar
  97. 97.
    Wolpert, D., Macready, W.: No free lunch theorems for search. Technical Report SFI-TR-95-02-010 (Santa Fe Institute) (1995)Google Scholar
  98. 98.
    Wolpert, D., Macready, W.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)CrossRefGoogle Scholar
  99. 99.
    Woods, K., Kegelmeyer, W., Bowyer, K.: Combination of multiple classifiers using local accuracy estimates. IEEE Trans. Pattern Anal. Mach. Intell. 19(4), 405–410 (1997)CrossRefGoogle Scholar
  100. 100.
    Xu, L., Krzyzak, A., Suen, C.: Methods for combining multiple classifiers and their appli-cations to handwriting recognition. IEEE Trans. Syst. Man Cybern. 22(3), 418–435 (1992)CrossRefGoogle Scholar
  101. 101.
    Yang, W., Zou, T., Dai, D., Shuai, Y.: Supervised land-cover classification of TerraSAR-X imagery over urban areas using extremely randomized clustering forests. In: Joint Urban Remote Sensing Event, S. 1–6 (2009)Google Scholar
  102. 102.
    Zadrozny, B., Elkan, C.: Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2002), S. 694–699 (2002)Google Scholar
  103. 103.
    Zhang, C., Ma, Y.: Ensemble Machine Learning – Methods and Applications (2012)CrossRefGoogle Scholar
  104. 104.
    Zhou, Z.H.: Ensemble Methods: Foundations and Algorithms (2012)Google Scholar
  105. 105.
    Zou, T., Yang, W., Dai, D., Sun, H.: Polarimetric SAR image classification using multifeatures combination and extremely randomized clustering forests. EURASIP J. Adv. Signal Process. 2010(1), 1–9 (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Technische Universität BerlinBerlinDeutschland
  2. 2.Technische Universität BerlinBerlinDeutschland

Personalised recommendations