Regression Models for Machine Learning

Abstract

This chapter investigates the regression models and methods for machine learning in engineering computations, from both non-Bayesian and Bayesian perspectives. The non-Bayesian regression models, including the least square regression, ridge regression, and support vector regression, equipped or not equipped with kernel trick, are first examined as they share the same principle, which is to find an element in the parametrically indexed hypothesis set by minimizing specific empirical loss functions. The Bayesian regression models are then investigated from the views of parametric space, and then functional space, all of which aim at assigning probabilistic beliefs on the elements of the hypothesis set using Bayesian inference. It is highlighted that, with specific assumptions, the Bayesian models provide measures of the prediction errors for unobserved points. This feature allows to develop a unique learning skill, i.e. active learning, which aims at devising optimal design strategies for minimizing the number of simulator calls, especially when each call is computationally cumbersome. This is shown to be effective when applied to cutting-edge research on Bayesian numerical analysis such as Bayesian cubature and Bayesian reliability assessment examined in this chapter. Both theoretical details and numerical examples are presented for examining each method.