Empirical Risk Minimization

Definition

The goal of learning is usually to find a model which delivers good generalization performance over an underlying distribution of the data. Consider an input space $\mathcal{X}$ and output space $\mathcal{Y}$. Assume the pairs $(X \times Y ) \in \mathcal{X}\times \mathcal{Y}$ are random variables whose (unknown) joint distribution is P _XY . It is our goal to find a predictor $f : \mathcal{X}\mapsto \mathcal{Y}$ which minimizes the expected risk:

$$\displaystyle{P(f(X)\neq Y ) = \mathsf{E}_{(X,Y )\sim P_{XY }}\ \left [\delta (f(X)\neq Y )\right ],}$$

where δ(z) = 1 if z is true, and 0 otherwise.

However, in practice we only have n pairs of training examples (X _i , Y _i ) drawn identically and independently from P _XY . Since P _XY is unknown, we often use the risk on the training set (called empirical risk) as a surrogate of the expected risk on the underlying distribution:

$$\displaystyle{ \frac{1} {n}\sum _{i=1}^{n}\delta (f(X_{ i})\neq Y _{i}).}$$

Empirical Risk...