Ensembles of Ensembles: Combining the Predictions from Multiple Machine Learning Methods
The rapid growth of machine learning (ML) has resulted in an almost overwhelmingly large number of modelling techniques, demanding better elucidation of their strengths and weaknesses in applied contexts. Tree-based methods such as Random Forests (RF) and Boosted Regression Trees (BRT) are powerful ML approaches that make no assumptions about the functional forms of the relationship with predictors, are flexible in handling missing data, and can easily capture complex, non-linear interactions. As with many ML methods, however, RF and BRT are potentially vulnerable to overfitting and a subsequent loss of generalizability.
The combination of predictions from multiple modelling methods, often referred to as the generation of “ensemble” or “consensus” predictions, is well established in fields such as meteorology. Recent ecological research suggests that combining the predictions from multiple methods is straightforward to implement, and can result in higher predictive accuracy than the results from single algorithms alone.
Using an example dataset involving satellite-derived fishing-vessel traffic information, we iteratively constructed 500 RF, BRT and ensemble (ENS) models and assessed the resulting mean-square prediction error (MSE) using cross-validated testing data. Performance depended upon the range of “parameter space”, with RF, BRT, and ENS approaches performing best under certain conditions. In general, variation in the number of trees used to train RF models seemed unimportant, but for this particular dataset, cross-validated error was lowest when tree depth was set to low values. BRT models were most sensitive to the size of the predictor subset used to train the model when faster “burn in” periods were selected (i.e., a higher shrinkage value). At slower “burn in” periods (i.e., shrinkage of 0.001) BRT tended to outperform both RF and ENS models except at the lowest values of predictor subset size. ENS models tended to exhibit lower prediction error with less variance under most conditions, and for faster “burn in” settings, yielded the lowest prediction errors of all.
We discuss how an approach like “committee averaging” can be used to determine a combined prediction from multiple methods, potentially improving predictive accuracy while also allowing a wider range of potentially useful algorithms to be employed.
KeywordsEnsemble models Fishing traffic Model comparison Boosted regression trees Random forests Committee averaging
- Brawn C (2016) Marine traffic risk density in Atlantic Canada. Unpubl. ReportGoogle Scholar
- Breiman L (2002) Manual on setting up, using, and understanding random forests v3.1. https://www.stat.berkeley.edu/~breiman/Using_random_forests_V3.1.pdf. Accessed 31 July 2015
- Caruana R, Niculescu-Mizil A (2006) Proceedings of the 23rd international conference on machine learning, Pittsburgh, PAGoogle Scholar
- Elith J, Burgman M (2002) Predictions and their validation: rare plants in the central highlands, Victoria, Australia. In: Scott JM, Heglund PJ, Morrison ML, Raphael MG, Wall WA, Samson FB (eds) Predicting species occurrences: issues of accuracy and scale. Island Press, Covelo, pp 303–314Google Scholar
- Elith J, Graham CH, Anderson RP, Dudik M, Ferrier S, Guisan A, Hijmans RJ, Huettmann F, Leathwick JR, Lehmann A, Li J, Lohmann LG, Loiselle RA, Manion G, Moritz C, Nakamura M, Nakazawa Y, Overton JM, Peterson AT, Phillips SJ, Richardson K, Scachetti-Pereira R, Schapiro RE, Soberón J, Williams S, Wisz MS, Zimmermann NE (2006) Novel methods improve prediction of species’ distributions from occurrence data. Ecography 29:129–151CrossRefGoogle Scholar
- Fernández-Delgado M, Cernadas E, Barro S, Amorim D (2014) Do we need hundreds of classifiers to solve real-world classification problems? J Mach Learn Res 15:3133–3181Google Scholar
- Huettmann F, Schmid M (2015) Climate change predictions of pelagic biodiversity components. In: De Broyer C, Koubbi P, Griffiths HJ, Raymond B, Udekem d’Acoz C, Van de Putte AP, Danis B, David B, Grant S, Gutt J, Held C, Hosie G, Huettmann F, Post A, Ropert-Coudert Y (eds) Biogeographic Atlas of the Southern Ocean. Scientific Committee on Antarctic Research, Cambridge, pp 390–396Google Scholar
- Liaw A, Wiener M (2002) Classification and regression by Random forest. R News 2(3):18–22Google Scholar
- Lieske DJ, Schmid M, Mahoney M (2018) Data and analysis script. https://doi.org/10.5281/zenodo.1318352
- Ridgeway G (2012) Generalized boosted models: a guide to the gbm package. URL: http://gradientboostedmodels.googlecode.com/git/gbm/inst/doc/gbm.pdf (downloaded January 8, 2016)
- Salford Systems, Inc. (2016) https://www.salford-systems.com/
- Schmid MS, Aubry C, Grigor J, Fortier L (2016) The LOKI underwater imaging system and an automatic identification model for the detection of zooplankton taxa in the Arctic Ocean. Methods Oceanogr In presGoogle Scholar