Data Complexity, Margin-Based Learning, and Popper’s Philosophy of Inductive Learning

Summary

This chapter provides a characterization of data complexity using the framework of Vapnik’s learning theory and Karl Popper’s philosophical ideas that can be readily interpreted in the context of empirical learning of inductive models from finite data. We approach the notion of data complexity under the setting of predictive learning, where this notion is directly related to the flexibility of a set of possible models used to describe available data. Hence, any characterization of data complexity is related to model complexity control. Recent learning methods [such as support vector machines (SVM), aka kernel methods] introduced the concept of margin to control model complexity. This chapter describes the characterization of data complexity for such margin-based methods. We provide a general philosophical motivation for margin-based estimators by interpreting the concept of margin as the degree of a model’s falsifiability. This leads to a better understanding of two distinct approaches to controlling model complexity: margin-based, where complexity is controlled by the size of the margin (or adaptive empirical loss function); and model-based, where complexity is controlled by the parameterization of admissible models. We describe SVM methods that combine margin-based and model-based complexity control, and show the effectiveness of the SVM strategy via empirical comparisons using synthetic data sets. Our comparisons clarify the difference between SVM methods and regularization methods. Finally, we introduce a new index of data complexity for margin-based classifiers. This new index effectively measures the degree of separation between the two classes achieved by margin-based methods (such as SVMs). The data sets with a high degree of separation (hence, good generalization) are characterized as simple, as opposed to complex data sets with heavily overlapping class distributions.