Verifying the Quality of Outsourced Training on Clouds

Abstract

Deep learning training is often outsourced to clouds due to its high computation overhead. However, clouds may not perform model training correctly due to the potential violations on Service Level Agreement (SLA) and attacks, incurring low quality of outsourced training. It is challenging for customers to understand the quality of outsourced training on clouds. They cannot measure the quality by simply testing the trained models because the testing performance is impacted by various factors, e.g., the quality of training and testing data. In order to address these issues, in this paper, we propose a novel framework that allows customers to verify the quality of outsourced training without modifying the processes of model training. Particularly, our framework achieves black-box verification by utilizing an extra training task that can be learned by the model only after the model converges on the original training task. We construct well-designed extra training tasks according to the original tasks, and develop a training quality verification method to measure the model performance on the extra task with a hypothesis testing-based threshold. The experiment results show that the models passing the quality verification achieve at least 96% of their best performance with negligible accuracy loss, i.e., less than 0.25%.

A A Proof of Theorem 1

Proof:

The main difference between the cleanly labeled samples and randomly labeled samples is that the former can be generalized while the latter can only be “memorized” by the model [7]. Specifically, the gradients for optimizing a cleanly labeled sample also contribute to optimize the other cleanly labeled samples, when the model is not convergent on the cleanly labeled samples. In contrast, optimizing any randomly labeled sample has almost no contribution to the other randomly labeled samples or cleanly labeled samples. In other words, the gradient correlation of clean label samples is much stronger than of gradients of random label samples:

$$\begin{aligned} \underset{\begin{array}{c} \boldsymbol{x}_i\in D_c, \boldsymbol{x}_j\in D_c , \boldsymbol{x}_i\ne \boldsymbol{x}_j \end{array}}{\text {average}} \frac{\boldsymbol{g}(\boldsymbol{x}_i) \cdot \boldsymbol{g}(\boldsymbol{x}_j)}{\left\| \boldsymbol{g}(\boldsymbol{x}_i)\right\| \left\| \boldsymbol{g}(\boldsymbol{x}_j)\right\| } \gg \underset{\begin{array}{c} \boldsymbol{x}_i\in D_r, \boldsymbol{x}_j\in D_r , \boldsymbol{x}_i\ne \boldsymbol{x}_j \end{array}}{\text {average}} \frac{\boldsymbol{g}(\boldsymbol{x}_i) \cdot \boldsymbol{g}(\boldsymbol{x}_j)}{\left\| \boldsymbol{g}(\boldsymbol{x}_i)\right\| \left\| \boldsymbol{g}(\boldsymbol{x}_j)\right\| }, \end{aligned}$$

(7)

where $\boldsymbol{g}(\boldsymbol{x})$ denote the gradient of $\boldsymbol{x}$, i.e., $\boldsymbol{g}(\boldsymbol{x}) := \nabla _{\theta } {\mathcal {L}(\boldsymbol{x}, f_{\theta })}$. The norm of the gradient of a individual sample can be approximated to a constant C since the direction of gradients has a more important effect than the norm of gradients. Then, we have:

$$\begin{aligned} \underset{\begin{array}{c} \boldsymbol{x}_i\in D_c, \boldsymbol{x}_j\in D_c , \boldsymbol{x}_i\ne \boldsymbol{x}_j \end{array}}{\text {average}} {\boldsymbol{g}(\boldsymbol{x}_i) \cdot \boldsymbol{g}(\boldsymbol{x}_j)} \gg \underset{\begin{array}{c} \boldsymbol{x}_i\in D_r, \boldsymbol{x}_j\in D_r , \boldsymbol{x}_i\ne \boldsymbol{x}_j \end{array}}{\text {average}} {\boldsymbol{g}(\boldsymbol{x}_i) \cdot \boldsymbol{g}(\boldsymbol{x}_j)} . \end{aligned}$$

(8)

Suppose there are $n_c$ and $n_r$ samples in $\mathcal {D}_c$ and $\mathcal {D}_r$ respectively, i.e., $D_{c}=\{(\boldsymbol{x}_i,y_i)_{i=1}^{n_c}\}$ and $D_{r}=\{(\boldsymbol{x}_i,y_i)_{i=n_c+1}^{n_c+n_r}\}$. The left term of Eq. 2 can be rewritten as:

$$\begin{aligned} \left\| \nabla _{{\theta }} {\mathcal {L}(\mathcal {D}_c, {\theta })} \right\| ^2 =&\left\| \sum _{i=1}^{n_c} \boldsymbol{g}(x_i) \right\| ^2 \end{aligned}$$

(9)

$$\begin{aligned} =&\sum _{i=1}^{n_c} \left\| \boldsymbol{g}(x_i) \right\| ^2 + 2 \sum _{\begin{array}{c} j=1 \\ j\ne i \end{array}}^{n_c}\sum _{i=1}^{n_c} \boldsymbol{g}(x_i)\cdot \boldsymbol{g}(x_j)\end{aligned}$$

(10)

$$\begin{aligned} =&n_c\cdot C + 2 \sum _{\begin{array}{c} j=1 \\ j\ne i \end{array}}^{n_c}\sum _{i=1}^{n_c} \boldsymbol{g}(x_i)\cdot \boldsymbol{g}(x_j) . \end{aligned}$$

(11)

Similarly, for the right term of Eq. 2, we have:

$$\begin{aligned} \left\| \nabla _{{\theta }} {\mathcal {L}(\mathcal {D}_r, {\theta })} \right\| ^2 = n_r\cdot C + 2 \sum _{\begin{array}{c} j=n_c+1 \\ j\ne i \end{array}}^{n_c+n_r}\sum _{i=n_c+1}^{n_c+n_r} \boldsymbol{g}(x_i)\cdot \boldsymbol{g}(x_j) . \end{aligned}$$

(12)

Since $|\mathcal {D}_r|>|\mathcal {D}_c|$, we have $n_c>n_r$. Then the conclusion can be deductively reasoned from Eq. 11, 12 and 8.    $\square$