Gradient Boosting with Extreme Learning Machines for the Optimization of Nonlinear Functionals

Abstract

In this paper we investigate the use of the Extreme Learning Machine (ELM) paradigm for the approximate minimization of a general class of functionals which arise routinely in operations research, optimal control and statistics problems. The ELM and, in general, neural networks with random hidden weights, have proved to be very efficient tools for the optimization of costs typical of machine learning problems, due to the possibility of computing the optimal outer weights in closed form. Yet, this feature is possible only when the cost is a sum of squared terms, as in regression, while more general cost functionals must be addressed with other methods. Here we focus on the gradient boosting technique combined with the ELM to address important instances of optimization problems such as optimal control of a complex system, multistage optimization and maximum likelihood estimation. Through the application of a simple gradient boosting descent algorithm, we show how it is possible to take advantage of the accuracy and efficiency of the ELM for the approximate solution of this wide family of optimization problems.