Mathematical Programming

, Volume 134, Issue 1, pp 127–155

Sample size selection in optimization methods for machine learning

  • Richard H. Byrd
  • Gillian M. Chin
  • Jorge Nocedal
  • Yuchen Wu
Full Length Paper Series B

DOI: 10.1007/s10107-012-0572-5

Cite this article as:
Byrd, R.H., Chin, G.M., Nocedal, J. et al. Math. Program. (2012) 134: 127. doi:10.1007/s10107-012-0572-5

Abstract

This paper presents a methodology for using varying sample sizes in batch-type optimization methods for large-scale machine learning problems. The first part of the paper deals with the delicate issue of dynamic sample selection in the evaluation of the function and gradient. We propose a criterion for increasing the sample size based on variance estimates obtained during the computation of a batch gradient. We establish an \({O(1/\epsilon)}\) complexity bound on the total cost of a gradient method. The second part of the paper describes a practical Newton method that uses a smaller sample to compute Hessian vector-products than to evaluate the function and the gradient, and that also employs a dynamic sampling technique. The focus of the paper shifts in the third part of the paper to L1-regularized problems designed to produce sparse solutions. We propose a Newton-like method that consists of two phases: a (minimalistic) gradient projection phase that identifies zero variables, and subspace phase that applies a subsampled Hessian Newton iteration in the free variables. Numerical tests on speech recognition problems illustrate the performance of the algorithms.

Mathematics Subject Classification

49M15 49M37 65K05 

Copyright information

© Springer and Mathematical Optimization Society 2012

Authors and Affiliations

  • Richard H. Byrd
    • 1
  • Gillian M. Chin
    • 2
  • Jorge Nocedal
    • 2
  • Yuchen Wu
    • 3
  1. 1.Department of Computer ScienceUniversity of ColoradoBoulderUSA
  2. 2.Department of Industrial Engineering and Management SciencesNorthwestern UniversityEvanstonUSA
  3. 3.Google Inc.Mountain ViewUSA

Personalised recommendations