Unsupervised Learning without Overfitting: Empirical Risk Approximation as an Induction Principle for Reliable Clustering

Abstract

Unsupervised learning algorithms are designed to extract structure from data samples on the basis of a cost function for structures. For a reliable and robust inference process, the unsupervised learning algorithm has to guarantee that the extracted structures are typical for the data source. In particular, it has to reject all structures where the inference is dominated by the arbitrariness of the sample noise and which, consequently, can be characterized as overfitting in unsupervised learning. This paper summarizes an inference principle called Empirical Risk Approximation which allows us to quantitatively measure the overfitting effect and to derive a criterion as a saveguard against it. The crucial condition for learning is met if (i) the empirical risk of learning uniformly converges towards the expected risk and if (ii) the hypothesis class retains a minimal variety for consistent inference. Parameter selection of learnable data structures is demonstrated for the case of k-means clustering and Monte Carlo simulations are presented to support the selection principle.