Lazy Overfitting Control

Abstract

A machine learning model is said overfit the training data relative to a simpler model if the first model is more accurate on the training data but less accurate on the test data. Overfitting control—selecting an appropriate complexity fit—is a central problem in machine learning. Previous overfitting control methods include penalty methods, which penalize a model for complexity, cross-validation methods, which experimentally determine when overfitting occurs on the training data relative to the test data, and ensemble methods, which reduce overfitting risk by combining multiple models. These methods are all eager in that they attempt to control overfitting at training time, and they all attempt to improve the average accuracy, as computed over the test data. This paper presents an overfitting control method which is lazy—it attempts to control overfitting at prediction time for each test case. Our results suggest that lazy methods perform well because they exploit the particulars of each test case at prediction time rather than averaging over all possible test cases at training time.