CaDET: interpretable parametric conditional density estimation with decision trees and forests
- 514 Downloads
We introduce CaDET, an algorithm for parametric Conditional Density Estimation (CDE) based on decision trees and random forests. CaDET uses the empirical cross entropy impurity criterion for tree growth, which incentivizes splits that improve predictive accuracy more than the regression criteria or estimated mean-integrated-square-error used in previous works. CaDET also admits more efficient training and query procedures than existing tree-based CDE approaches, and stores only a bounded amount of information at each tree leaf, by using sufficient statistics for all computations. Previous tree-based CDE techniques produce complicated uninterpretable distribution objects, whereas CaDET may be instantiated with easily interpretable distribution families, making every part of the model easy to understand. Our experimental evaluation on real datasets shows that CaDET usually learns more accurate, smaller, and more interpretable models, and is less prone to overfitting than existing tree-based CDE approaches.
KeywordsParametric models Random forests Sufficient statistics
- Casella, G., & Berger, R. L. (2002). Statistical inference. Pacific Grove, CA: Duxbury.Google Scholar
- Di Mauro, N., Vergari, A., Basile, T. M., & Esposito, F. (2017). Fast and accurate density estimation with extremely randomized cutset networks. In: Joint European conference on machine learning and knowledge discovery in databases (pp. 203–219). Berlin: Springer.Google Scholar
- Hothorn, T., & Zeileis, A. (2017). Transformation forests. arXiv preprint arXiv:1701.02110.
- Lahman, S. (2018). Sean Lahman’s baseball archive. http://www.seanlahman.com/baseball-archive/statistics/.
- Molina, A., Vergari, A., Di Mauro, N., Natarajan, S., Esposito, F., & Kersting, K. (2018). Mixed sum-product networks: A deep architecture for hybrid domains. In: Thirty-second AAAI conference on artificial intelligence.Google Scholar
- Pospisil, T., & Lee, A. B. (2018). RFCDE: Random forests for conditional density estimation. arXiv preprint arXiv:1804.05753.
- Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1(1), 81–106.Google Scholar
- Rahman, T., Kothalkar, P., & Gogate, V. (2014). Cutset networks: A simple, tractable, and scalable approach for improving the accuracy of Chow–Liu trees. In: Joint European conference on machine learning and knowledge discovery in databases (pp. 630–645).Google Scholar
- Rosenblatt, M. (1969). Conditional probability density and regression estimators. Multivariate Analysis II, 25, 31.Google Scholar
- Zhu, J., & Hastie, T. (2002). Kernel logistic regression and the import vector machine. In Advances in neural information processing systems (pp. 1081–1088).Google Scholar