A deterministic rescaled perceptron algorithm

Abstract

The perceptron algorithm is a simple iterative procedure for finding a point in a convex cone $F\subseteq \mathbb {R}^m$. At each iteration, the algorithm only involves a query to a separation oracle for $F$ and a simple update on a trial solution. The perceptron algorithm is guaranteed to find a point in $F$ after $\mathcal O(1/\tau _F^2)$ iterations, where $\tau _F$ is the width of the cone $F$. We propose a version of the perceptron algorithm that includes a periodic rescaling of the ambient space. In contrast to the classical version, our rescaled version finds a point in $F$ in $\mathcal O(m^5 \log (1/\tau _F))$ perceptron updates. This result is inspired by and strengthens the previous work on randomized rescaling of the perceptron algorithm by Dunagan and Vempala (Math Program 114:101–114, 2006) and by Belloni et al. (Math Oper Res 34:621–641, 2009). In particular, our algorithm and its complexity analysis are simpler and shorter. Furthermore, our algorithm does not require randomization or deep separation oracles.

Appendix: Proof of (7) and (8)

Let $v \in \text {int}(\mathbb {B}^{m-1})$ be given. The volumes ${\left| \left| \left| (\Psi \circ \Phi )'(v) \right| \right| \right| }$ and ${\left| \left| \left| \Phi '(v) \right| \right| \right| }$ are $\sqrt{\det \left( (\Psi \circ \Phi )'(v)^\mathrm{T}(\Psi \circ \Phi )'(v)\right) }$ and $\sqrt{\det \left( \Phi '(v)^\mathrm{T}\Phi '(v)\right) }$ respectively. Thus (7) and (8) are equivalent to

$$\begin{aligned} \det \left( (\Psi \circ \Phi )'(v)^\mathrm{T}(\Psi \circ \Phi )'(v)\right) = \dfrac{\alpha ^2}{(1-\Vert v\Vert ^2)\left( \alpha ^2 + (1-\alpha ^2) \Vert v\Vert ^2\right) ^m}, \end{aligned}$$

(15)

and

$$\begin{aligned} \det \left( \Phi '(v)^\mathrm{T}\Phi '(v)\right) = \dfrac{1}{1-\Vert v\Vert ^2}. \end{aligned}$$

(16)

We first prove (15). To simplify notation, put $t^2 := \alpha ^2+\left( 1-\alpha ^2\right) \Vert v\Vert ^2$. Computing the partial derivatives of $(\Psi \circ \Phi )(v) = \frac{\left( v, \alpha \sqrt{1-\Vert v\Vert ^2}\right) }{\sqrt{\alpha ^2 + (1-\alpha ^2)\Vert v\Vert ^2}}$ we get

$$\begin{aligned} (\Psi \circ \Phi )'(v)&= \begin{bmatrix} \dfrac{\partial (\Psi \circ \Phi )(v)}{\partial v_1}&\cdots&\dfrac{\partial (\Psi \circ \Phi )(v)}{\partial v_{m-1}} \end{bmatrix} \\&= \begin{bmatrix} \dfrac{1}{t}I_{m-1} - \dfrac{(1-\alpha ^2)}{t^3}vv^\mathrm{T}\\ -\dfrac{\alpha }{t^3\sqrt{1-\Vert v\Vert ^2}}v^\mathrm{T}\end{bmatrix}, \end{aligned}$$

where $I_{m-1}$ is the $(m-1)\times (m-1)$ identity matrix. Hence,

$$\begin{aligned} {(\Psi \circ \Phi )'(v) }^\mathrm{T}(\Psi \circ \Phi )'(v) = \dfrac{1}{t^2}I_{m-1} + F vv^\mathrm{T}, \end{aligned}$$

where

$$\begin{aligned} F&= -\dfrac{2(1-\alpha ^2)}{t^4} + \dfrac{(1-\alpha ^2)^2\Vert v\Vert ^2}{t^6} + \dfrac{\alpha ^2}{t^6(1-\Vert v\Vert ^2)} \\&= \frac{-2(1-\alpha ^2)t^2(1-\Vert v\Vert ^2) + (1-\alpha ^2)^2\Vert v\Vert ^2(1-\Vert v\Vert ^2)+\alpha ^2}{t^6(1-\Vert v\Vert ^2)} \\&= \frac{(1-\alpha ^2)(1-\Vert v\Vert ^2)(-2t^2 + (1-\alpha ^2)\Vert v\Vert ^2)+\alpha ^2}{t^6(1-\Vert v\Vert ^2)} \\&= \frac{(1-t^2)(-t^2-\alpha ^2)+\alpha ^2}{t^6(1-\Vert v\Vert ^2)} \\&= \frac{\alpha ^2-1+t^2}{t^4(1-\Vert v\Vert ^2)}. \end{aligned}$$

The fourth step above follows from $t^2 = 1-(1-\alpha ^2)(1-\Vert v\Vert ^2) = \alpha ^2 + (1-\alpha ^2)\Vert v\Vert ^2$.

Hence we have

$$\begin{aligned} \det \left( {(\Psi \circ \Phi )'(v) }^\mathrm{T}(\Psi \circ \Phi )'(v)\right)&= \det \left( \dfrac{1}{t^{2}}\left( I_{m-1} + \dfrac{\alpha ^2-1+t^2}{t^2(1-\Vert v\Vert ^2)} vv^\mathrm{T}\right) \right) \\&= \dfrac{1}{t^{2(m-1)}} \det \left( I_{m-1} + \dfrac{\alpha ^2-1+t^2}{t^2(1-\Vert v\Vert ^2)} vv^\mathrm{T}\right) \\&= \frac{1}{t^{2(m-1)}} \left( 1+\frac{(\alpha ^2 - 1 +t^2)\Vert v\Vert ^2}{t^2(1-\Vert v\Vert ^2)}\right) \\&= \frac{t^2(1-\Vert v\Vert ^2) + (\alpha ^2 - 1 +t^2)\Vert v\Vert ^2}{t^{2m}(1-\Vert v\Vert ^2)} \\&= \frac{\alpha ^2}{(\alpha ^2+ \left( 1-\alpha ^2\right) \Vert v\Vert ^2)^{m}(1-\Vert v\Vert ^2)}. \end{aligned}$$

The last step follows because

$$\begin{aligned} t^2(1-\Vert v\Vert ^2) + (\alpha ^2 - 1 +t^2)\Vert v\Vert ^2&= t^2 - (1- \alpha ^2)\Vert v\Vert ^2 \\&= \alpha ^2+(1-\alpha ^2)\Vert v\Vert ^2 - (1- \alpha ^2)\Vert v\Vert ^2 \\&= \alpha ^2. \end{aligned}$$

The proof of (16) is similar but easier. Computing partial derivatives of $\Phi (v) = (v,\sqrt{1-\Vert v\Vert ^2})$ we get

$$\begin{aligned} \Phi '(v) = \begin{bmatrix} I_{m-1}\\ \dfrac{1}{\sqrt{1-\Vert v\Vert ^2}}v^\mathrm{T}\end{bmatrix}. \end{aligned}$$

Hence

$$\begin{aligned} {\Phi '(v)}^\mathrm{T}\Phi '(v) = I_{m-1} + \dfrac{1}{1-\Vert v\Vert ^2} vv^\mathrm{T}. \end{aligned}$$

Therefore,

$$\begin{aligned} \det \left( {\Phi '(v)}^\mathrm{T}\Phi '(v)\right) = \det \left( I_{m-1} + \dfrac{1}{1-\Vert v\Vert ^2} vv^\mathrm{T}\right) = 1+\dfrac{\Vert v\Vert ^2}{1-\Vert v\Vert ^2} = \dfrac{1}{1-\Vert v\Vert ^2}. \end{aligned}$$