Data Mining and Knowledge Discovery

, Volume 17, Issue 2, pp 253–282 | Cite as

Maximizing classifier utility when there are data acquisition and modeling costs

  • Gary M. WeissEmail author
  • Ye Tian


Classification is a well-studied problem in data mining. Classification performance was originally gauged almost exclusively using predictive accuracy, but as work in the field progressed, more sophisticated measures of classifier utility that better represented the value of the induced knowledge were introduced. Nonetheless, most work still ignored the cost of acquiring training examples, even though this cost impacts the total utility of the data mining process. In this article we analyze the relationship between the number of acquired training examples and the utility of the data mining process and, given the necessary cost information, we determine the number of training examples that yields the optimum overall performance. We then extend this analysis to include the cost of model induction—measured in terms of the CPU time required to generate the model. While our cost model does not take into account all possible costs, our analysis provides some useful insights and a template for future analyses using more sophisticated cost models. Because our analysis is based on experiments that acquire the full set of training examples, it cannot directly be used to find a classifier with optimal or near-optimal total utility. To address this issue we introduce two progressive sampling strategies that are empirically shown to produce classifiers with near-optimal total utility.


Data mining Machine learning Induction Decision trees Utility-based data mining Cost-sensitive learning Active learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Berry M and Linoff G (2004). Data mining techniques for marketing, sales and customer relationship management. Wiley Publishing, Indianapolis, IN Google Scholar
  2. Breiman L, Friedman JH, Olshen RA, Stone CJ (1983) Classification and regression trees. WadsworthGoogle Scholar
  3. Caruna R, Joachims T and Backstrom L (2004). KDD-CUP 2004: results and analysis. SIGKDD Explor 6(2): 95–108 CrossRefGoogle Scholar
  4. Cohn D, Atlas L and Ladner R (1994). Improving generalization with active learning. Mach Learn 15(2): 201–221 Google Scholar
  5. Drummond C and Holte R (2006). Cost curves: an improved method for visualizing classifier performance. Mach Learn 65(1): 95–130 CrossRefGoogle Scholar
  6. Elkan C (2001) The foundations of cost-sensitive learning. In: Proceedings of the Seventeenth International Joint Conference on artificial intelligence, Seattle, WA, pp 973–978Google Scholar
  7. Esposito F, Malerba D and Semeraro G (1997). A comparative analysis of methods for pruning decision trees. IEEE Trans Pattern Anal Mach Intell 19(5): 476–491 CrossRefGoogle Scholar
  8. Fayyad U, Piatetsky-Shapiro G and Smyth P (1996). From data mining to knowledge discovery in databases.. AI Mag 17: 37–54 Google Scholar
  9. Greiner R, Grove A and Roth D (2002). Learning cost-sensitive active classifiers. Artif Intell 39: 137–174 CrossRefMathSciNetGoogle Scholar
  10. Hettich S, Bay SD (1999) The UCI KDD archive []. University of California, Dept. of Information and Computer Science, Irvine, CAGoogle Scholar
  11. Hoehn B, Southey F, Holte R, Bulitko V (2005) Effective short-term opponent exploitation in simplified poker. In: Proceedings of the Twentieth National Conference on artificial intelligence, Pittsburgh, PA, pp 783–788Google Scholar
  12. Kapoor A, Greiner R (2005) Learning and classifying under hard budgets. In: Proceedings of the Sixteenth European Conference on machine learning, Porto, Portugal, pp 170–181Google Scholar
  13. Lewis D, Catlett J (1994) Heterogeneous uncertainty sampling for supervised learning. In: Proceedings of the Eleventh International Conference on machine learning, New Brunswick, NJ, pp 148–156Google Scholar
  14. Li R, Belford G (2002) Instability of decision tree classification algorithms. In: Proceedings of the Eighth ACM SIGKDD International Conference on knowledge discovery and data mining, Edmonton, Canada, pp 570–575Google Scholar
  15. Martin JK, Hirschberg DS (1996) On the complexity of learning decision trees. In: Proceedings of the fourth International Symposium on artificial intelligence and mathematics, Fort Lauderdale, FloridaGoogle Scholar
  16. Melville P, Saar-Tsechansky M, Provost F, Mooney R (2005) Economical active-feature value acquisition through expected utility estimation. In: Proceedings of the First International Workshop on Utility-Based Data Mining, Chicago, IL, pp 10–16Google Scholar
  17. Newman DJ, Hettich S, Blake CL, Merz CJ (1998) UCI Repository of machine learning databases []. University of California, Department of Information and Computer Science, Irvine, CAGoogle Scholar
  18. Provost F and Fawcett T (2001). Robust classification for imprecise environments. Mach Learn 42: 203–231 zbMATHCrossRefGoogle Scholar
  19. Provost F, Jensen D, Oates T (1999) Efficient progressive sampling. In: Proceedings of the Fifth International Conference on knowledge discovery and data mining, San Diego, CA, pp 23–32Google Scholar
  20. Quinlan JR (1993). C4.5: programs for machine learning. Morgan Kaufmann, San Mateo, CA Google Scholar
  21. Snedecor GW and Cochran WG (1989). Statistical methods. Iowa State University Press, Ames, OH zbMATHGoogle Scholar
  22. Turney P (2000) Types of cost in inductive concept learning. In: Workshop on Cost-Sensitive Learning at the Seventeenth International Conference on machine learning, Stanford, CAGoogle Scholar
  23. Van Rijsbergen CJ (1979) Information retrieval, 2nd edn. Butterworth, London Google Scholar
  24. Veeramachaneni S, Avesani P (2003) Active sampling for feature selection. In: Proceedings of the Third IEEE International Conference on data mining, Melbourne, Florida, pp 665–668Google Scholar
  25. Weiss GM and Provost F (2003). Learning when training data are costly: the effect of class distribution on tree induction. J Artif Intell Res 19: 315–354 zbMATHGoogle Scholar
  26. Weiss GM, Saar-Tsechansky M and Zadrozny B (2005). Report on UBDM-05: workshop on utility-based data mining. SIGKDD Explor 17(2): 145–147 CrossRefGoogle Scholar
  27. Zadrozny B, Weiss GM and Saar-Tsechasnky M (2006). UBDM-2006: utility-based data mining workshop report. SIGKDD Explor 8(2): 98–101 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Department of Computer and Information ScienceFordham UniversityBronxUSA

Personalised recommendations