Cost-Sensitive Decision Tree Learning for Forensic Classification

  • Jason V. Davis
  • Jungwoo Ha
  • Christopher J. Rossbach
  • Hany E. Ramadan
  • Emmett Witchel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4212)


In some learning settings, the cost of acquiring features for classification must be paid up front, before the classifier is evaluated. In this paper, we introduce the forensic classification problem and present a new algorithm for building decision trees that maximizes classification accuracy while minimizing total feature costs. By expressing the ID3 decision tree algorithm in an information theoretic context, we derive our algorithm from a well-formulated problem objective. We evaluate our algorithm across several datasets and show that, for a given level of accuracy, our algorithm builds cheaper trees than existing methods. Finally, we apply our algorithm to a real-world system, Clarify. Clarify classifies unknown or unexpected program errors by collecting statistics during program runtime which are then used for decision tree classification after an error has occurred. We demonstrate that if the classifier used by the Clarify system is trained with our algorithm, the computational overhead (equivalently, total feature costs) can decrease by many orders of magnitude with only a slight (<1%) reduction in classification accuracy.


Decision Tree Information Gain Function Counting Decision Tree Algorithm Splitting Criterion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bradford, J., Kunz, C., Kohavi, R., Brunk, C., Brodley, C.: Pruning decision trees with misclassification costs. In: European Conference on Machine Learning (1998)Google Scholar
  2. 2.
    Brun, Y., Ernst, M.D.: Finding latent code errors via machine learning over program executions. In: ICSE (2004)Google Scholar
  3. 3.
    Cohen, I., Zhang, S., Goldszmidt, M., Symons, J., Kelly, T., Fox, A.: Capturing, indexing, clustering, and retrieving system history. In: SOSP (2005)Google Scholar
  4. 4.
    Cover, T.M., Thomas, J.A.: Elements of information theory. Wiley Series in Telecommunications (1991)Google Scholar
  5. 5.
    Blake, C.L., Newman, D.J., Hettich, S., Merz, C.J.: UCI repository of machine learning databases (1998)Google Scholar
  6. 6.
    Elkan, C.: The foundations of cost-sensitive learning. In: International joint conference on artifical intelligence (2001)Google Scholar
  7. 7.
    Ha, J., Ramadan, H., Davis, J., Rossbach, C., Roy, I., Witchel, E.: Navel: Automating software support by classifying program behavior. Technical Report TR-06-11, University of Texas at Austin (2006)Google Scholar
  8. 8.
    Hangal, S., Lam, M.S.: Tracking down software bugs using automatic anomaly detection. In: ICSE (2002)Google Scholar
  9. 9.
    Liblit, B., Naik, M., Zheng, A.X., Aiken, A., Jordan, M.I.: Scalable statistical bug isolation. In: PLDI (2005)Google Scholar
  10. 10.
    Mitchell, T.: Machine Learning. In: WCB. McGraw-Hill, New York (1997)Google Scholar
  11. 11.
    Norton, S.W.: Generating better decision trees. In: International joint conference on artifical intelligence (1989)Google Scholar
  12. 12.
    Nunez, M.: The use of background knowledge in decision tree induction. Machine Learning (1991)Google Scholar
  13. 13.
    Quinlan, R.: C4.5: programs for machine learning. Morgan Kaufmann Publishers, San Francisco (1992)Google Scholar
  14. 14.
    Turney, P.: Cost-sensitive classification: Empirical evaluation of a hybrid genetic decision tree induction algorithm. Journal of artificial intelligence research (1995)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jason V. Davis
    • 1
  • Jungwoo Ha
    • 1
  • Christopher J. Rossbach
    • 1
  • Hany E. Ramadan
    • 1
  • Emmett Witchel
    • 1
  1. 1.Dept. of Computer SciencesThe University of Texas at Austin 

Personalised recommendations