Statistical Active Learning Algorithms for Noise Tolerance and Differential Privacy

Abstract

We describe a framework for designing efficient active learning algorithms that are tolerant to random classification noise and are differentially-private. The framework is based on active learning algorithms that are statistical in the sense that they rely on estimates of expectations of functions of filtered random examples. It builds on the powerful statistical query framework of Kearns (JACM 45(6):983–1006, 1998). We show that any efficient active statistical learning algorithm can be automatically converted to an efficient active learning algorithm which is tolerant to random classification noise as well as other forms of “uncorrelated” noise. The complexity of the resulting algorithms has information-theoretically optimal quadratic dependence on $1/(1-2\eta )$, where $\eta$ is the noise rate. We show that commonly studied concept classes including thresholds, rectangles, and linear separators can be efficiently actively learned in our framework. These results combined with our generic conversion lead to the first computationally-efficient algorithms for actively learning some of these concept classes in the presence of random classification noise that provide exponential improvement in the dependence on the error $\epsilon$ over their passive counterparts. In addition, we show that our algorithms can be automatically converted to efficient active differentially-private algorithms. This leads to the first differentially-private active learning algorithms with exponential label savings over the passive case.

Appendices

Appendix 1: Passive SQ Learning of Halfspaces

The first SQ algorithm for learning general halfspaces was given by Blum et al. [12]. This algorithm requires access to unlabeled samples from the unknown distribution and therefore is only label-statistical. This algorithm can be used as a basis for our active SQ algorithm but the resulting active algorithm will also be only label-statistical. As we have noted in Sect. 2, this is sufficient to obtain our RCN tolerant active learning algorithm given in Corollary 3.7. However our differentially-private simulation needs the algorithm to be (fully) statistical. Therefore we base our algorithm on the algorithm of Dunagan and Vempala for learning halfspaces [29]. While [29] does not contain an explicit statement of the SQ version of the algorithm it is known and easy to verify that the algorithm has a SQ version [49]. This follows from the fact that the algorithm in [29] relies on a combination on the Perceptron [45] and the modified Perceptron algorithms [12] both of which have SQ versions [12, 17]. Another small issue that we need to take care of to apply the algorithm is that the running time and tolerance of the algorithm depend polynomially (in fact, linearly) on $\log (1/\rho _0)$, where $\rho _0$ is the margin of the points given to the algorithm. Namely, $\rho _0 = \min _{x \in S}\frac{|w \cdot x|}{\Vert x\Vert }$, where $h_w$ is the target homogeneous halfspace and $S$ is the set of points given to the algorithm. We are dealing with continuous distributions for which the margin is 0 and therefore we make the following observation. In place of $\rho _0$ we can use any margin $\rho _1$ such that the probability of being within margin $\le \rho _1$ around the target hyperplane is small enough that it can be absorbed into the tolerance of the statistical queries of the Dunagan–Vempala algorithm for margin $\rho _1$. Formally,

Definition 7.1

For positive $\delta <1$ and distribution $D$, we denote

$$\begin{aligned} \gamma (D,\delta ) = \inf _{\Vert w\Vert =1}\sup _{\gamma >0} \left\{ \gamma \ \left| \ \mathop {\Pr }\limits _D\left[ \frac{|w \cdot x|}{\Vert x\Vert }\le \gamma \right] \le \delta \ \right. \right\} , \end{aligned}$$

namely the smallest value of $\gamma$ such that for every halfspace $h_w,\,\gamma$ is the largest such that the probability of being within margin $\gamma$ of $h_w$ under $D$ is at most $\delta$. Let $\tau _{{\mathtt {DV}}}(\rho ,\epsilon )$ be the tolerance of the SQ version of the Dunagan–Vempala algorithm when the initial margin is equal to $\rho$ and error is set to $\epsilon$. Let

$$\begin{aligned} \rho _1(D,\epsilon ) = \frac{1}{2} \sup _{\rho \ge 0}\left\{ \rho \ |\ \gamma (D,\tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)/3) \ge \rho \right\} . \end{aligned}$$

Now, for $\rho _1 = \rho _1(D,\epsilon )$ we know that $\gamma (D,\tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)/3) \ge \rho$. Let $D'$ be defined as distribution $D$ conditioned on having margin at least $\rho$ around the target hyperplane $h_w$. By the definition of the function $\gamma$, the probability of being within margin $\le \rho$ is at most $\tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)/3$. Therefore for any query function $g:X \times \{-1,1\}\rightarrow [-1,1]$,

$$\begin{aligned} \left| \mathop {\mathbf {E}}_D[g(x,h_w(x))] - (1-\tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)/3) \mathop {\mathbf {E}}_{D'}[g(x,h_w(x))] \right| \le \tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)/3 \end{aligned}$$

and hence $|\mathop {\mathbf {E}}\nolimits _D[g(x,h_w(x))] - \mathop {\mathbf {E}}\nolimits _{D'}[h_w(x,f(x))]| \le 2 \tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)/3$. This implies that we can obtain an answer to any SQ relative to $D'$ with tolerance $\tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)$ by using the same SQ relative to $D$ with tolerance $\tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)/3$. This means that by running the Dunagan–Vempala algorithm in this way we will obtain a hypothesis with error at most $\epsilon /2$ relative to $D'$. This hypothesis has error at most $\epsilon /2 + 2 \tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)/3$ which, without loss of generality, is at most $\epsilon$. Combining these observations about the Dunagan–Vempala algorithm, we obtain the following statement.

Theorem 7.2

([29]) There exists a SQ algorithm LearnHS-DV that learns ${\mathcal {H}}_d$ to accuracy $1-\epsilon$ over any distribution $D$. Further LearnHS outputs a homogeneous halfspace, runs in time polynomial in $d,\,1/\epsilon$ and $\log (1/\rho _1)$ and uses SQs of tolerance $\ge 1/\text{ poly }(d,1/\epsilon ,\log (1/\rho _1))$, where $\rho _1 = \rho _1(D,\epsilon )$.

To apply Theorem 7.2 we need to obtain bounds on $\rho _1(D,\epsilon )$ for any distribution $D$ on which we might run the Dunagan–Vempala algorithm.

Lemma 7.3

Let $D$ be an isotropic log-concave distribution. Then for any $\delta \in (0,1/20),\,\gamma (D,\delta ) \ge \delta /(6 \ln (1/\delta ))$.

Proof

Let $\gamma \in (0,1/16)$ and $w$ be any unit vector. We first upper-bound $\mathop {\Pr }\nolimits _D\left[ \frac{|w \cdot x|}{\Vert x\Vert }\le \gamma \right]$.

$$\begin{aligned} \mathop {\Pr }\limits _D\left[ \frac{|w \cdot x|}{\Vert x\Vert }\le \gamma \right]&\le \mathop {\Pr }\limits _D\left[ \Vert x\Vert \le \ln (1/\gamma ) \text{ and } \frac{|w \cdot x|}{\Vert x\Vert }\le \gamma \right] + \mathop {\Pr }\limits _D\left[ \Vert x\Vert > \ln (1/\gamma ) \right] \nonumber \\&\le \mathop {\Pr }\limits _D\left[ |w \cdot x| \le \gamma \cdot \ln (1/\gamma ) \right] + \mathop {\Pr }\limits _D\left[ \Vert x\Vert > \ln (1/\gamma )\right] . \end{aligned}$$

(8)

By Lemma 5.7 in [43], for an isotropic log-concave $D$ and any $R > 1,\,\mathop {\Pr }\nolimits _D[\Vert x\Vert > R] < e^{-R+1}$. Therefore

$$\begin{aligned} \mathop {\Pr }\limits _D\left[ \Vert x\Vert > \ln (1/\gamma ) \right] \le e \cdot \gamma . \end{aligned}$$

Further, by Lemma 3.2,

$$\begin{aligned} \mathop {\Pr }\limits _D\left[ |w \cdot x| \le \gamma \cdot \ln (1/\gamma ) \right] \le 2 \gamma \cdot \ln (1/\gamma ). \end{aligned}$$

Substituting, these inequalities into Eq. (8) we obtain that for $\gamma \in (0,1/16)$,

$$\begin{aligned} \mathop {\Pr }\limits _D\left[ \frac{|w \cdot x|}{\Vert x\Vert }\le \gamma \right] \le 2 \gamma \cdot \ln (1/\gamma ) + e \cdot \gamma \le 3 \gamma \cdot \ln (1/\gamma ). \end{aligned}$$

This implies that for $\gamma = \delta /(6 \ln (1/\delta ))$ and any unit vector $w$,

$$\begin{aligned} \mathop {\Pr }\limits _D\left[ \frac{|w \cdot x|}{\Vert x\Vert }\le \gamma \right] \le 3 \delta /(6 \ln (1/\delta )) \cdot \left( \ln (1/\delta ) + \ln (6 \ln (1/\delta )\right) \le \delta , \end{aligned}$$

where we used that for $\delta < 1/20,\,6 \ln (1/\delta ) \le 1/\delta$. By definition of $\gamma (D,\delta )$, this implies that $\gamma (D,\delta ) \ge \delta /(6 \ln (1/\delta ))$. $\square$

We are now ready to prove Theorem 3.5. There exists a SQ algorithm LearnHS that learns ${\mathcal {H}}_d$ to accuracy $1-\epsilon$ over any distribution $D_{|\chi }$, where $D$ is an isotropic log-concave distribution and $\chi :\mathbb {R}^d \rightarrow [0,1]$ is a filter function. Further LearnHS outputs a homogeneous halfspace, runs in time polynomial in $d,\,1/\epsilon$ and $\log (1/\lambda )$ and uses SQs of tolerance $\ge 1/\text{ poly }(d,1/\epsilon ,\log (1/\lambda ))$, where $\lambda = \mathop {\mathbf {E}}\nolimits _D[\chi (x)]$.

Proof of Theorem 3.5

To prove the theorem we bound $\rho _1 = \rho _1(D_{|\chi },\epsilon )$ and then apply Theorem 7.2. We first observe that for any event $\Lambda$,

$$\begin{aligned} \mathop {{\mathbf {Pr}}}\limits _{D_{|\chi }}[\Lambda ] \le \mathop {{\mathbf {Pr}}}\limits _D[\Lambda ]/\mathop {\mathbf {E}}_D[\chi ]. \end{aligned}$$

Applying this to the event $\frac{|w \cdot x|}{\Vert x\Vert } \le \gamma$ in Definition 7.1 we obtain that $\gamma (D_{|\chi },\delta ) \ge \gamma (D,\delta \cdot \mathop {\mathbf {E}}\nolimits _D[\chi ])$. By Lemma 7.3, we get that $\gamma (D_{|\chi },\delta ) = \Omega (\lambda \delta /\log (1/(\lambda \delta )))$.

In addition, by Theorem 7.2, $\tau _{{\mathtt {DV}}}(\rho ,\epsilon ) \ge 1/p(d,1/\epsilon ,\log (1/\rho ))$ for some polynomial $p$. This implies that

$$\begin{aligned} \gamma (D_{|\chi },\tau _{{\mathtt {DV}}}(\rho ,\epsilon /2)/3)&\le \gamma \left( D_{|\chi },\Omega \left( \frac{1}{p(d,1/\epsilon ,\log (1/\rho ))}\right) \right) \\&= \tilde{\Omega }\left( \frac{\lambda }{p(d,1/\epsilon ,\log (1/\rho ))}\right) . \end{aligned}$$

Therefore, we will obtain that,

$$\begin{aligned} \rho _1(D_{|\chi }, \epsilon ) =\tilde{\Omega }\left( \frac{\lambda }{p(d,1/\epsilon ,1)}\right) . \end{aligned}$$

$\square$

By plugging this bound into Theorem 7.2 we obtain the claim.

Appendix 2: Proofs from Sect. 4

We now prove Lemmas 4.6 and 4.7 which we restate for convenience.

Lemma 8.1

(Lemma 4.6 restated) For any $v,w\in S_{d-1}$ such that $\Vert v-w\Vert = \Delta \le \sqrt{2}$ and $\gamma > 0$,

$$\begin{aligned} {\mathbf {Pr}}[h_v(x)&\ne h_w(x) \ |\ |\langle v, x \rangle | \le \gamma ]\\&= \frac{A_{d-3} \int _0^\gamma (1-r^2)^{(d-3)/2}\int _{\frac{r \cdot \sqrt{2-\Delta ^2}}{\Delta \cdot \sqrt{1-r^2}}}^1 (1-s^2)^{(d-4)/2}ds\cdot dr }{A_{d-2} \int _0^\gamma (1-r^2)^{(d-3)/2}dr}. \end{aligned}$$

We denote the probability by ${\mathsf {cp}}_d(\gamma ,\Delta )$.

Proof

By using spherical symmetry, we can assume without loss of generality that $v = (1,0,0,\ldots ,0)$ and $w=(\sqrt{1-\Delta ^2/2}, \Delta /\sqrt{2},0,0,\ldots ,0)$. We now examine the surface area of the points that satisfy $h_w(x) = -1$ and $0 \le \langle v, x \rangle \le \gamma$ (which is a half of the error region at distance at most $\gamma$ from $v$). To compute it we consider the points on $S_{d-1}$ that satisfy $\langle v, x \rangle = r$. These points form a hypersphere $\sigma$ of dimension $d-2$ and radius $\sqrt{1-r^2}$. In this hypersphere points that satisfy $h_w(x) = -1$ are points $(r,s,x_3,\ldots ,x_d) \in S_d$ for which $r\sqrt{1-\Delta ^2/2} + s \Delta /\sqrt{2} \le 0$. In other words, $s \ge r\sqrt{2-\Delta ^2}/\Delta$ or points of $\sigma$ which are at least $r\sqrt{2-\Delta ^2}/\Delta$ away from hyperplane $(0,1,0,0,\ldots ,0)$ passing through the origin of $\sigma$ (also referred to as a hyperspherical cap). As in the Eq. (6), we obtain that its $d-2$-dimensional surface area is:

$$\begin{aligned} (1-r^2)^{(d-2)/2}\int _{\frac{r \cdot \sqrt{2-\Delta ^2}}{\Delta \cdot \sqrt{1-r^2}}}^1 A_{d-3} (1-s^2)^{(d-4)/2}ds \end{aligned}$$

Integrating over all $r$ from $0$ to $\gamma$ gives the surface area of the region $h_w(x) = -1$ and $0 \le \langle v, x \rangle \le \gamma$:

$$\begin{aligned} \int _0^\gamma (1-r^2)^{(d-3)/2}\int _{\frac{r \cdot \sqrt{2-\Delta ^2}}{\Delta \cdot \sqrt{1-r^2}}}^1 A_{d-3} (1-s^2)^{(d-4)/2}ds\cdot dr. \end{aligned}$$

Hence the conditional probability is as claimed. $\square$

Lemma 8.2

(Lemma 4.7 restated) For $\Delta \le \sqrt{2}$, any $d\ge 4$, and $\gamma \ge \Delta /(2\sqrt{d}),\,\partial _\Delta {\mathsf {cp}}_d(\gamma ,\Delta ) \ge 1/(56\gamma \cdot \sqrt{d})$.

Proof

First note that

$$\begin{aligned} \tau (\gamma )=\frac{A_{d-3}}{A_{d-2}\int _0^\gamma (1-r^2)^{(d-3)/2} dr} \end{aligned}$$

is independent of $\Delta$ and therefore it is sufficient to differentiate

$$\begin{aligned} \theta (\gamma ,\Delta ) = \int _0^\gamma (1-r^2)^{(d-3)/2}\int _{\frac{r \cdot \sqrt{2-\Delta ^2}}{\Delta \cdot \sqrt{1-r^2}}}^1 (1-s^2)^{(d-4)/2}ds\cdot dr. \end{aligned}$$

Let $\gamma ' = \Delta /(2\sqrt{d})$ (note that by our assumption $\gamma ' \le \gamma$). By the Leibnitz integral rule,

$$\begin{aligned} \partial _\Delta \theta (\gamma ,\Delta )&= \int _0^\gamma (1-r^2)^{(d-3)/2} \partial _\Delta \left( \int _{\frac{r \cdot \sqrt{2-\Delta ^2}}{\Delta \cdot \sqrt{1-r^2}}}^1(1-s^2)^{(d-4)/2}ds \right) \cdot dr \\&= \int _0^\gamma (1-r^2)^{(d-3)/2} \left( 1- \frac{r^2 (2-\Delta ^2)}{\Delta ^2(1-r^2)}\right) ^{\frac{d-4}{2}} \cdot \frac{2r}{\Delta ^2 \sqrt{1-r^2} \sqrt{2-\Delta ^2}} \cdot dr \\&\ge \int _0^\gamma (1-r^2)^{(d-4)/2} \left( 1- \frac{2 r^2}{\Delta ^2 (1-r^2)}\right) ^{\frac{d-4}{2}} \cdot \frac{2r}{\sqrt{2} \Delta ^2} \cdot dr \\&\ge \int _0^{\gamma '} (1-r^2)^{(d-4)/2} \left( 1- \frac{2 r^2}{\Delta ^2(1-r^2)}\right) ^{\frac{d-4}{2}} \cdot \frac{\sqrt{2} \cdot r}{\Delta ^2} \cdot dr. \end{aligned}$$

Now using the conditions $\Delta \le \sqrt{2},\,d\ge 4$, we obtain that $\gamma ' \le 1/(2\sqrt{2})$ and hence for all $r \in [0,\gamma '],\,1-r^2 \ge 7/8$ and $r^2/\Delta ^2 \le \gamma '^2/\Delta ^2 = 1/(4d)$. This implies that for all $r\in [0,\gamma ']$,

$$\begin{aligned} 1- \frac{2 r^2}{\Delta ^2(1-r^2)} \ge 1- \frac{2}{\frac{7}{8} 4d} = 1-\frac{4}{7d}. \end{aligned}$$

Now,

$$\begin{aligned} \left( 1-\frac{4}{7d}\right) ^{(d-4)/2} \ge 1- \frac{4 (d-4)}{14 d} \ge \frac{5}{7}. \end{aligned}$$

Substituting this into our expression for $\partial _\Delta \theta (\gamma ,\Delta )$ we get

$$\begin{aligned}&\partial _\Delta \theta (\gamma ,\Delta )\\&\quad \ge \int _0^{\gamma '} (1-r^2)^{(d-4)/2} \frac{\sqrt{2} \cdot 5r}{7 \Delta ^2} \cdot dr \ge \frac{1}{\Delta ^2}\int _0^{\gamma '} (1-r^2)^{(d-4)/2} \cdot r \cdot dr \\&\quad = \frac{1}{\Delta ^2(d-2)} \left( 1 - (1-\gamma '^2)^{(d-2)/2}\right) \ge \frac{1}{\Delta ^2(d-2)} \left( 1 -e^{-\gamma '^2 (d-2)/2}\right) \\&\quad \ge ^{(*)}\frac{1}{\Delta ^2(d-2)} \left( 1 -\left( 1-\frac{(d-2)\gamma '^2}{4}\right) \right) = \frac{\gamma '^2}{4 \Delta ^2} =\frac{1}{16d}, \end{aligned}$$

where to derive $(*)$ we use the fact that $e^{-\gamma '^2 (d-2)/2} \le 1-\gamma '^2 (d-2)/4$ since $e^{-x} \le 1-x/2$ for every $x \in [0,1]$ and $\gamma '^2 (d-2)/2 \le \frac{\Delta ^2 (d-2)}{8d} < 1$. At the same time, $\int _0^\gamma (1-r^2)^{(d-3)/2} dr \le \gamma$ and therefore,

$$\begin{aligned} \partial _\Delta {\mathsf {cp}}_d(\gamma ,\Delta ) = \tau (\gamma ) \cdot \partial _\Delta \theta (\gamma ,\Delta ) \ge \frac{A_{d-3}}{16 d \gamma A_{d-2}} \ge \frac{1}{32 \sqrt{3} \gamma \sqrt{d} } > \frac{1}{56 \gamma \sqrt{d}}, \end{aligned}$$

where we used that $\frac{A_{d-3}}{A_{d-2}} \ge \sqrt{d}/(2\sqrt{3})$ (e.g. [27]). $\square$