Reweighted l ₂-Regularized Dual Averaging Approach for Highly Sparse Stochastic Learning

Abstract

Recent advances in dual averaging schemes for primal-dual subgradient methods and stochastic learning revealed an ongoing and growing interest in making stochastic and online approaches consistent and tailored towards sparsity inducing norms. In this paper we focus on the reweighting scheme in the $l_2$-Regularized Dual Averaging approach which favors properties of a strongly convex optimization objective while approximating in a limit the $l_0$-type of penalty. In our analysis we focus on a regret and convergence criteria of such an approximation. We derive our results in terms of a sequence of strongly convex optimization objectives obtained via the smoothing of a sub-differential and non-smooth loss function, e.g. hinge loss. We report an empirical evaluation of the convergence in terms of the cumulative training error and the stability of the selected set of features. Experimental evaluation shows some improvements over the $l_1$-RDA method in the generalization error as well.