Finding Robust Models Using a Stratified Design

  • Rohan A. Baxter
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4304)


Predictive performance in model selection is often estimated using out-of-sample validation and test datasets. The assumption is that the test and validation datasets are from the same population as the training dataset. This assumption may not apply in the common application context where the model is applied to scoring of future data. This paper proposes a sample design which can lead to better model performance and robust estimates of model generalization error. The sample design is shown applied to a collection scoring application.


Test Dataset Credit Risk Validation Dataset Challenger Design Data Mining Application 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Breiman, L.: Statistical modeling: The two cultures. Statistical Sciences 16(3), 199–231 (2001)MATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Hand, D., Mannila, H., Smyth, P.: Principles of data mining. MIT Press, Cambridge (2001)Google Scholar
  3. 3.
    Han, J., Kimber, M.: Data Mining: Concepts and techniques. Morgan Kaufmann, San Francisco (2001)Google Scholar
  4. 4.
    Witten, I., Frank, E.: Data mining: Machine Learning Tools and Techniques. Morgan Kaufmann, San Francisco (2005)MATHGoogle Scholar
  5. 5.
    Maindonald, J.: The role of models in predictive validation (statistics for budding data miners). In: ISIS (2003)Google Scholar
  6. 6.
    Dhar, V., Stein, R.: Finding robust and usable models from data mining. In: PCAI (1998)Google Scholar
  7. 7.
    Zadrozny, B.: Learning and Evaluating Classifiers under Sample Selection Bias (2004)Google Scholar
  8. 8.
    Elkan, C.: Foundations of cost-sensitive learning (2001)Google Scholar
  9. 9.
    Blake, C., Keogh, E., Merz, C.: Uci repository of machine learning databases. UCI website (2001)Google Scholar
  10. 10.
    Hand, D.: Classifier technology and the illusion of progress. Statistical Science 21, 1–15 (2006)MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Rohan A. Baxter
    • 1
  1. 1.Analytics Project, Office of the Chief Knowledge OfficerAustralian Taxation Office

Personalised recommendations