Abstract
A dynamic programming algorithm for constructing optimal dyadic decision trees was recently introduced, analyzed, and shown to be very effective for low dimensional data sets. This paper enhances and extends this algorithm by: introducing an adaptive grid search for the regularization parameter that guarantees optimal solutions for all relevant trees sizes, replacing the dynamic programming algorithm with a memoized recursive algorithm whose run time is substantially smaller for most regularization parameter values on the grid, and incorporating new data structures and data pre-processing steps that provide significant run time enhancement in practice.
Article PDF
Similar content being viewed by others
References
Asuncion, A., & Newman, D. J. (2007). UCI machine learning repository.
Blanchard, G., Schäfer, C., Rozenholc, Y., & Müller, K.-R. (2007). Optimal dyadic decision trees. Machine Learning, 66(2–3), 209–241.
Breiman, L., Friedman, J., Olshen, J., & Stone, C. (1984). Classification and regression trees. Belmont: Wadsworth.
Cormen, T. H., Leiserson, C. E., & Rivest, R. L. (1990). Introduction to algorithms. New York: McGraw-Hill.
Devroye, L., Györfi, L., & Lugosi, G. (1996). A probabilistic theory of pattern recognition. New York: Springer.
Donoho, D. (1997). Cart and best-ortho-basis: a connection. Annals of Statistics, 25, 1870–1911.
Garey, M. R. (1972). Optimal binary identification procedures. SIAM Journal on Applied Mathematics, 23(2), 173–186.
Lew, A. (1978). Optimal conversion of extended-entry decision tables with general cost criteria. Communications of the ACM, 21(4), 269–279.
Meisel, W. S., & Michalopoulos, D. A. (1973). A partitioning algorithm with application in pattern classification and the optimization of decision trees. IEEE Transactions on Computers, 22(1), 93–103.
Moret, B. M. E. (1982). Decision trees and diagrams. Computing Surveys, 14(4), 593–623.
Murthy, S. K. (1998). Automatic construction of decision trees from data: A multi-disciplinary survey. Data Mining and Knowledge Discovery, 2, 345–389.
Nijssen, S., & Fromont, E. (2007). Mining optimal decision trees from itemset lattices. In Proceedings of the 13th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 530–539).
Payne, H. J., & Meidel, W. S. (1977). An algorithm for constructing optimal binary decision trees. IEEE Transactions on Computers, 26(9), 905–916.
Pooch, U. W. (1974). Translation of decision tables. Computing Surveys, 6(2), 125–151.
Quinlan, J. R. (1993). C4.5: Programs for machine learning. San Mateo: Morgan Kaufmann.
Quinlan, J. R. (1996). Improved use of continuous attributes in C4.5. Journal of Artificial Intelligence Research, 4, 77–90.
Reinwald, L. T., & Soland, R. M. (1966). Conversion of limited-entry decision tables to optimal computer programs I: minimum average processing time. Journal of the ACM, 13(3), 339–358.
Reinwald, L. T., & Soland, R. M. (1967). Conversion of limited-entry decision tables to optimal computer programs II: minimum storage requirement. Journal of the ACM, 14(4), 742–755.
Safavian, S. R., & Landgrebe, D. (1991). A survey of decision tree classifier methodology. IEEE Transactions on Systems, Man, and Cybernetics, 21(3), 660–674.
Schumacher, H., & Sevcik, K. C. (1976). The synthetic approach to decision table conversion. Communications of the ACM, 19(6), 343–351.
Scott, C., & Nowak, R. (2003). Dyadic classification trees via structural risk minimization. In S. Thrun, S. Becker, & K. Obermayer (Eds.), Advances in neural information processing system (Vol. 15, pp. 359–366). Cambridge: MIT Press.
Scott, C., & Nowak, R. (2004). Near-minimax optimal classification with dyadic classification trees. In S. Thrun, L. Saul, & B. Schölkopf (Eds.), Advances in neural information processing systems (Vol. 16, pp. 1117–1124). Cambridge: MIT Press.
Scott, C., & Nowak, R. (2006). Minimax optimal classification with dyadic decision trees. IEEE Transactions on Information Theory, 52(4), 1335–1353.
Author information
Authors and Affiliations
Corresponding author
Additional information
Editor: Johannes Fürnkranz.
The work was supported by the Laboratory Directed Research and Development (LDRD) office at Los Alamos National Laboratory.
Rights and permissions
About this article
Cite this article
Hush, D., Porter, R. Algorithms for optimal dyadic decision trees. Mach Learn 80, 85–107 (2010). https://doi.org/10.1007/s10994-010-5167-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-010-5167-x