Block coordinate descent algorithms for large-scale sparse multiclass classification

3.1 Multiclass classification: direct vs. indirect formulations

Classifying an object into one of several categories is an important problem arising in many applications such as document classification and object recognition. Machine learning approaches to this problem can be roughly divided into two categories: direct and indirect approaches. While direct approaches formulate the multiclass problem directly, indirect approaches reduce the multiclass problem to multiple independent binary classification or regression problems. Because support vector machines (SVMs) (Boser et al. 1992) were originally proposed as a binary classification model, they have frequently been used in combination with indirect approaches to perform multiclass classification. Among them, one of the most popular is “one-vs-rest” (Rifkin and Klautau 2004), which consists of learning to separate one class from all the others, independently for all m possible classes. Direct multiclass SVM extensions were later proposed by Weston and Watkins (1999), Lee et al. (2004) and Crammer and Singer (2002). They were all formulated as constrained problems and solved in the dual. An unconstrained (non-differentiable) form of the Crammer-Singer formulation is popularly used with stochastic subgradient descent algorithms such as Pegasos (Shalev-Shwartz et al. 2010). Another popular direct multiclass (smooth) formulation, which is an intuitive extension of traditional logistic regression, is multiclass logistic regression. In this paper, we propose an efficient direct multiclass formulation.

3.2 Sparse multiclass classification

Recently, mixed-norm regularization has attracted much interest (Yuan and Lin 2006; Meier et al. 2008; Duchi and Singer 2009a, 2009b; Obozinski et al. 2010) due to its ability to impose sparsity at the feature group level. Few papers, however, have investigated its application to multiclass classification. Zhang et al. (2006) extend Lee et al.’s multiclass SVM formulation (Lee et al. 2004) to employ ℓ ₁/ℓ _∞ regularization and formulate the learning problem as a linear program (LP). However, they experimentally verify their method only on very small problems (both in terms of n and d). Duchi and Singer (2009a) propose a boosting-like algorithm specialized for ℓ ₁/ℓ ₂-regularized multiclass logistic regression. In another paper, Duchi and Singer (2009b) derive and analyze FOBOS, a stochastic subgradient descent framework based on forward-backward splitting and apply it, among other things, to ℓ ₁/ℓ ₂-regularized multiclass logistic regression. In this paper, we choose ℓ ₁/ℓ ₂ regularization, since it can be optimized more efficiently than ℓ ₁/ℓ _∞ regularization (see Sect. 4.7).

3.3 Coordinate descent methods

Although coordinate descent methods were among the first optimization methods proposed and studied in the literature (see Bertsekas 1999 and references therein), it is only recently that they regained popularity, thanks to several successful applications in the machine learning (Fu 1998; Shevade and Keerthi 2003; Friedman et al. 2007, 2010b; Yuan et al. 2010; Qin et al. 2010) and optimization (Tseng and Yun 2009; Wright 2012; Richtárik and Takáč 2012a) communities. Conceptually and algorithmically simple, (block) coordinate descent algorithms focus at each iteration on updating one block of variables while keeping the others fixed, and have been shown to be particularly well-suited for minimizing objective functions with non-smooth separable regularization such as ℓ ₁ or ℓ ₁/ℓ ₂ (Tseng and Yun 2009; Wright 2012; Richtárik and Takáč 2012a).

Coordinate descent algorithms have different trade-offs: expensive gradient-based greedy block selection as opposed to cheap cyclic or randomized selection, use of line search (Tseng and Yun 2009; Wright 2012) or not (Richtárik and Takáč 2012a). For large-scale linear classification, and we confirm in this paper, cyclic and randomized block selection schemes have been shown to achieve excellent performance (Yuan et al. 2010, 2011; Chang et al. 2008; Richtárik and Takáč 2012a). The most popular loss function for ℓ ₁-regularized binary classification is arguably logistic regression, due to its smoothness (Yuan et al. 2010). Binary logistic regression was also successfully combined with ℓ ₁/ℓ ₂ regularization in the case of user-defined feature groups (Meier et al. 2008). However, recent work (Yuan et al. 2010, 2011; Chang et al. 2008) using coordinate descent indicate that logistic regression is substantially slower to train than ℓ ₂-loss (squared hinge) SVMs. This is because, contrary to ℓ ₂-loss SVMs, logistic regression requires expensive log and exp computations (equivalent to dozens of multiplications) to compute the gradient or objective value (Yuan et al. 2011). Motivated by this background, we propose a novel efficient direct multiclass formulation. Compared to multiclass logistic regression, which suffers from the same problems as its binary counterpart, our formulation can be optimized very efficiently by block coordinate descent and lends itself to large-scale and high-dimensional problems such as document classification.

4.1 Objective function

Our objective, Eq. (2), is similar in spirit to Weston and Watkins’ multiclass SVM formulation (Weston and Watkins 1999), in that it ensures that the correct class’s score is greater than all the other classes by at least 1. However, it has the following differences: it is unconstrained (rather than constrained), it is ℓ ₁/ℓ ₂-regularized (rather than \(\ell_{2}^{2}\)-regularized) and it penalizes misclassifications quadratically (rather linearly), which ensures differentiability of L(W).

4.2 Optimization by block coordinate descent

A key property of F(W) is the separability of its non-smooth part R(W) over groups j=1,2,…,d. This calls for an algorithm which minimizes F(W) by updating W group by group. In this paper, to minimize F(W), we thus employ block coordinate descent. We consider two variants, one with line search (Tseng and Yun 2009) and the other without (Richtárik and Takáč 2012a). We present the two variants in a unified manner.

Algorithm 1 outlines block coordinate descent for minimizing F(W). At each iteration, Algorithm 1 selects a block W _j:∈R ^m of coefficients and updates it, keeping all other blocks fixed (how to choose the block is delayed to Sect. 4.4). This procedure is repeated several times until a suitable stopping criterion is met or the maximum number of outer iterations K is reached. The main difficulty arising in Algorithm 1 is how to solve the sub-problem associated with each weight block W _j:. Let W ^t be the weight matrix at iteration t. The key idea of block coordinate descent frameworks for non-smooth separable minimization (Tseng and Yun 2009; Richtárik and Takáč 2012a) is to update each block by solving the following quadratic approximation of F around W ^t :

$$ \boldsymbol{W}^*_{j:} = \underset{\boldsymbol{W}_{j:} \in \mathbf{R}^m}{\mathrm{argmin}} ~ G\bigl(\boldsymbol{W}^t \bigr)_{j:}^\mathrm{T} \bigl(\boldsymbol{W}_{j:} - \boldsymbol{W}_{j:}^t\bigr) + \frac{1}{2} \bigl( \boldsymbol{W}_{j:} - \boldsymbol{W}_{j:}^t \bigr)^\mathrm{T} \boldsymbol{H}^t \bigl(\boldsymbol{W}_{j:} - \boldsymbol{W}_{j:}^t\bigr) + \lambda\| \boldsymbol{W}_{j:}\|_2, $$

(3)

where we used G(W) _j:∈R ^m to denote the jth row of the gradient of L(W) and H ^t is a m×m matrix. If we choose \(\boldsymbol{H}^{t}=\mathcal{L}_{j}^{t} \boldsymbol{I}\) where \(\mathcal{L}_{j}^{t}\) is a scalar (we discuss its choice in Sect. 4.4) and I is the identity matrix, Eq. (3) can be rewritten as:

$$\boldsymbol{W}^*_{j:} = \underset{\boldsymbol{W}_{j:} \in \mathbf{R}^m}{\mathrm{argmin}} ~ \frac{1}{2} \bigl\| \boldsymbol{W}_{j:} - \boldsymbol{V}_{j:}^t \bigr\|^2 + \mu_j^t \|\boldsymbol{W}_{j:} \|_2 $$

where we defined \(\boldsymbol{V}_{j:}^{t} = \boldsymbol{W}^{t}_{j:} - \frac{1}{\mathcal{L}_{j}^{t}} G(\boldsymbol{W}^{t})_{j:}\) and \(\mu_{j}^{t} = \frac{\lambda}{\mathcal{L}_{j}^{t}}\). This problem takes a form which is well-known in the signal-processing literature and whose solution is called proximity operator (Combettes and Wajs 2005). The proximity-operator associated with the ℓ ₂ norm takes a closed form (see e.g. Duchi and Singer 2009b for a derivation):

$$ \boldsymbol{W}^*_{j:} = \mathrm{Prox}_{\mu_j^t \|\cdot\|_2}\bigl( \boldsymbol{V}^t_{j:}\bigr) = \max\biggl\{1 - \frac{\mu_j^t}{\|\boldsymbol{V}^t_{j:}\|_2}, 0\biggr\} \boldsymbol{V}^t_{j:}. $$

(4)

This operator is known as vectorial soft-thresholding operator (Wright et al. 2009), owing to the fact that \(\boldsymbol{W}^{*}_{j:}\) becomes entirely zero when \(1 - \frac{\mu^{t}}{\|\boldsymbol{V}^{t}_{j:}\|_{2}} < 0\). Summarizing, we obtain \(\boldsymbol{W}^{*}_{j:}\) by taking a partial gradient step with step size \(\frac{1}{\mathcal{L}^{t}_{j}}\) and then projecting the result by \(\mathrm{Prox}_{\mu_{j}^{t} \|\cdot\|_{2}}\). Finally, let \(\boldsymbol{\delta }^{t}_{j} = \boldsymbol{W}^{*}_{j:} - \boldsymbol{W}^{t}_{j:}\). The last step consists in setting \(\boldsymbol{W}^{t+1}_{j:} = \boldsymbol{W}^{t}_{j:} + \alpha_{j}^{t} \boldsymbol{\delta}^{t}_{j}\). We discuss the choice of \(\alpha_{j}^{t}\) in Sect. 4.4. Algorithm 2 summarizes how to solve the block sub-problem associated with W _j: (we drop the superscript t since there is no ambiguity).

Open image in new window

Open image in new window

4.3 Efficient partial gradient computation

Since computing A(W) _ir from scratch would be computationally prohibitive, we instead initialize A(W) to 1 _n×m at the beginning of Algorithm 1, then when a weight block is updated by W _j:←W _j:+α _j δ _j , we update A(W) by \(A(\boldsymbol{W})_{ir} \leftarrow A(\boldsymbol{W})_{ir} + \alpha_{j} (\boldsymbol{\delta}_{jr} - \boldsymbol{\delta}_{j{y_{i}}}) \boldsymbol{x}_{ij}\) for all i such that x _ij ≠0 and all r≠y _i . Thanks to this implementation technique, denoting \(\hat{n}\) the average number of non-zero values per feature, the cost of computing Eq. (5) is only \(O(\hat{n}(m-1))\). We summarize how to efficiently compute G(W) _j: in Algorithm 3. When using sparse data, the compressed sparse column (CSC) format can be used for fast access to all non-zero values of feature j (inner loop in Algorithm 3).

Open image in new window

4.4 Choice of block, \(\mathcal{L}_{j}^{t}\) and \(\alpha_{j}^{t}\)

We now discuss how to choose, at every iteration, the block W _j: to update, \(\mathcal{L}_{j}^{t}\) and \(\alpha_{j}^{t}\), depending on whether a line search is used or not.

4.5 Stopping criterion

We would like to develop a stopping criterion for Algorithm 1 which can be checked at almost no extra computational cost. Proposition 1 characterizes an optimal solution of Eq. (2).

Proposition 1

W is an optimal solution of Eq. (2) if and only if ∀j:

Open image in new window

4.6 Global convergence properties

We discuss convergence properties for the two block coordinate descent variants we considered: cyclic block coordinate descent with line search (Tseng and Yun 2009) and randomized block coordinate descent without line search (Richtárik and Takáč 2012a). To have finite termination of the line search, Tseng and Yun (Lemma 5.1), require that L has Lipschitz continuous gradient, which we prove with Lemma 1 in Appendix A. For asymptotic convergence, Tseng and Yun assume that each block is cyclically visited (Eq. (12)). They further assume (Assumption 1) that H ^t is upper-bounded by some value and lower-bounded by 0, which is guaranteed by our choice \(\boldsymbol {H}^{t} = \mathcal{L}_{j}^{t} \boldsymbol{I}\). Richtárik and Takáč also assume (Sect. 2) that the blockwise gradient is Lipschitz. They show (Theorem 4) that using their algorithm, there exists a finite iteration t such that P(F(W ^t )−F(W ^∗)≤ϵ)≥1−ρ, where ϵ>0 is the accuracy of the solution and 0<ρ<1 is the target confidence.

4.7 Extensions

Another possible extension consists in replacing ℓ ₁/ℓ ₂ regularization by ℓ ₁/ℓ _∞ regularization or ℓ ₁+ℓ ₁/ℓ ₂ regularization (sparse group lasso, Friedman et al. 2010a). This requires changing the proximity operator, Eq. (4), as well as reworking the stopping criterion developed in Sect. 4.5. Similarly to ℓ ₁/ℓ ₂ regularization, ℓ ₁/ℓ _∞ regularization leads to group sparsity. However, the proximity operator associated with the ℓ _∞ norm requires a projection on an ℓ ₁-norm ball (Bach et al. 2012) and is thus computationally more expensive than the proximity operator associated with the ℓ ₂ norm, which takes a closed form, Eq. (4). For ℓ ₁+ℓ ₁/ℓ ₂ regularization (sparse group lasso), the group-wise proximity operator can readily be computed by applying first the proximity operator associated with the ℓ ₁ norm and then the one associated with the ℓ ₂ norm (Bach et al. 2012). However, sparse group lasso regularization requires the tuning of an extra hyperparameter, which balances between ℓ ₁/ℓ ₂ and ℓ ₁ regularizations. For this reason, we do not consider it in our experiments.

5.1 Datasets

5.2 Comparison of block coordinate descent with other solvers

All solvers are used to minimize the same objective: our proposed multiclass formulation, Eq. (2).

Figures 2 and 3 compare the relative objective value difference \(\frac{F(\boldsymbol{W}) - F(\boldsymbol{W}^{*})}{F(\boldsymbol{W^{*}})}\) (lower is better) and test accuracy (higher is better) of the above solvers as a function of training time, when λ=10⁻³ and λ=10⁻⁵, respectively. For FOBOS, we used the step size \(\eta_{t} = \frac{\eta_{0}}{\sqrt{t}}\), where we chose η ₀ beforehand from 10⁻³, 10⁻², …, 10³ with a held-out validation set.

Open image in new window

Open image in new window

Open image in new window

Open image in new window

Figure 4 compares the number of non-zero rows of the solution (lower is better) as a function of training time for the different solvers, when λ=10⁻³ (left) and λ=10⁻⁵ (right).

Open image in new window

Open image in new window

5.3 Comparison with other multiclass and multitask objectives

For both multitask squared hinge and multiclass logistic regression, we computed the partial gradient using an efficient implementation technique similar to the one described in Sect. 4.3. For the multiclass and multitask squared hinge formulations, we used BCD with line search. For the multiclass logistic regression formulation, we used BCD without line search, since we observed faster training times (see Sect. 5.3.1). For multiclass logistic regression, the partial gradient’s Lipschitz constant is \(\mathcal{K}_{j} = \frac{1}{2} \sum_{i} \boldsymbol{x}_{ij}^{2}\) (Duchi and Singer 2009a).

Figure 5 compares the row-sparsity/test-accuracy trade-off of the above objectives. We generated 10 log-spaced values between λ=10⁻² and λ=10⁻⁴ (Sector), and between λ=10⁻³ and λ=10⁻⁵ (other datasets). For each λ value, we computed the solution and measured the percentage of non-zero rows as well as test accuracy, so as to obtain Fig. 5. Table 2 shows the time (minimum, median and maximum) that was needed to compute the solutions of each objective. The results reported are averages obtained from 3 different train-test splits. We set τ=10⁻³ and K=200.

Open image in new window

Table 2

Minimum, median and maximum training times (in seconds) of different ℓ ₁/ℓ ₂-regularized objective functions, when computing solutions for 10 log-space values between λ=10⁻² and λ=10⁻⁴ (Sector), and between λ=10⁻³ and λ=10⁻⁵ (other datasets)

Dataset	MC Squared Hinge	MT Squared Hinge	MC Logistic
Amazon7	285.93	655.53	4,137.39
	486.54	909.84	5,822.83
	1,118.65	1,740.90	6,128.27
RCV1	2,962.82	5,089.91	10,413.47
	3,901.58	6,300.77	11,540.80
	4,581.33	6,556.67	15,055.57
MNIST	32.55	42.74	141.54
	55.25	79.56	260.30
	61.67	99.37	645.04
News20	50.46	71.42	172.94
	70.73	93.82	218.04
	75.08	98.15	244.62
Sector	62.01	251.57	149.09
	167.01	259.50	532.30
	191.52	273.85	743.45