Deep Learning

Abstract

This chapter is focused on a specific topic in modern machine learning, i.e., deep learning. First of all, we introduce a few fundamental aspects, including perceptron and why we need multi-layer structures, how the convolutional neural networks extract features layer by layer, the back-propagation learning algorithm, and the functional layers of convolutional neural networks. Then, we will focus on the safety and security vulnerabilities of deep learning, explaining uncertainty estimation, adversarial attack on the robustness, poisoning attack, model stealing, membership inference and model inversion. Unlike traditional machine learning models that we discussed in the previous chapters where the structure or the training algorithm of a machine learning model may be considered for the design of the attacks, most attacks for deep learning do not consider the internal structure of a deep learning model, although the gradient may be used in some cases. These black-box or grey-box attacks suggest that many of the attack algorithms for deep learning may also be applicable to traditional machine learning models, which explains why we frequently refer to algorithms in the chapter when discussing safety threads for traditional machine learning models.

Exercises

Question 1

Consider the example dataset in Example 5.1, please compute the below expressions (up to 2 decimal places)

• GainRatio(Humidity, PlayTennis) = 0.16

• GainRatio(Temperature, PlayTennis) = 0.03

• GainRatio(Wind, PlayTennis) = 0.05

• InfoGain(Humidity, PlayTennis) = 0.15

• InfoGain(Temperature, PlayTennis) = 0.03

• InfoGain(Wind, PlayTennis) = 0.05 □

Question 2

Consider part of the Iris dataset in Table 10.5, please compute the below expressions (up to 2 decimal places)

• GainRatio(SepalLength, IrisClass) = 

  Table 10.5 Part of Iris dataset

• GainRatio(SepalWidth, IrisClass) = 

• GainRatio(PedalLength, IrisClass) = 

• GainRatio(PedalWidth, IrisClass) = 

• InfoGain(SepalLength, IrisClass) = 

• InfoGain(SepalWidth, IrisClass) = 

• InfoGain(PedalLength, IrisClass) = 

• InfoGain(PedalWidth, IrisClass) = 

Question 3

Understand the basic idea of random forest by conducting research on the literature, and implement a random forest algorithm based on the decision tree algorithm to see if random forest performs better than decision tree on Iris dataset. □

Question 4

Following the last newquestion, please give an adversarial attack algorithm for random forest. □

Question 5

Consider the four data samples in Example 7.1 (also provided in Table 10.6) and the mean square error, if we have the following two functions:

Table 10.6 A small dataset

• $f_{{\mathbf {w}}_1}=2X_1+1X_2+20X_3-330$

• $f_{{\mathbf {w}}_2}=X_1-2X_2+23X_3-332$

please newanswer the following newquestions:

• 1. which model is better for linear regression?

• 2. which model is better for linear classification by considering 0-1 loss for y ^T = (0, 1, 1, 0)?

• 3. which model is better for logistic regression for y ^T = (0, 1, 1, 0)?

• 4. According to the logistic regression of the first model, what is the prediction result of the first model on a new input (181, 92, 12.4)? □

Answer

• 1. Because

  $$\displaystyle \begin{aligned} \begin{array}{rll} f_{{\mathbf{w}}_1}({\mathbf{x}}_1)-y_1 = & 2*182+1*87+20*11.3-330-325 & = 22 \\ f_{{\mathbf{w}}_1}({\mathbf{x}}_2)-y_2 = & 2*189+1*92+20*12.3-330-344 & = 42 \\ f_{{\mathbf{w}}_1}({\mathbf{x}}_3)-y_3 = & 2*178+1*79+20*10.6-330-350 & = -33 \\ f_{{\mathbf{w}}_1}({\mathbf{x}}_4)-y_4 = & 2*183+1*90+20*12.7-330-320 & = 60 \\ f_{{\mathbf{w}}_2}({\mathbf{x}}_1)-y_1 = & 182-2*87+23*11.3-332-325 & = -389.1 \\ f_{{\mathbf{w}}_2}({\mathbf{x}}_2)-y_2 = & 189-2*92+23*12.3-332-344 & = -388.1 \\ f_{{\mathbf{w}}_2}({\mathbf{x}}_3)-y_3 = & 178-2*79+23*10.6-332-350 & = -418.2 \\ f_{{\mathbf{w}}_2}({\mathbf{x}}_4)-y_4 = & 183-2*90+23*12.7-332-320 & = -356.9 \\ \end{array} \end{aligned} $$

  (10.89)

  the model $f_{{\mathbf {w}}_1}$ is better for linear regression, according to the loss function (Eq. (7.3));

• 2. Because

  $$\displaystyle \begin{aligned} \begin{array}{cc} step(f_{{\mathbf{w}}_1}({\mathbf{x}}_1))=1\\ step(f_{{\mathbf{w}}_1}({\mathbf{x}}_2))=1\\ step(f_{{\mathbf{w}}_1}({\mathbf{x}}_3))=1\\ step(f_{{\mathbf{w}}_1}({\mathbf{x}}_4))=1\\ step(f_{{\mathbf{w}}_2}({\mathbf{x}}_1))=0\\ step(f_{{\mathbf{w}}_2}({\mathbf{x}}_2))=0\\ step(f_{{\mathbf{w}}_2}({\mathbf{x}}_3))=0\\ step(f_{{\mathbf{w}}_2}({\mathbf{x}}_4))=0\\ \end{array} \end{aligned} $$

  (10.90)

  we have that both models are the same, according to the loss function (Eq. (7.13));

• 3. Because

  $$\displaystyle \begin{aligned} \begin{array}{rcl} y_1*\log(\sigma(f_{{\mathbf{w}}_1}({\mathbf{x}}_1)))+(1-y_1)\log((1-\sigma(f_{{\mathbf{w}}_1}({\mathbf{x}}_1))))&=&-M\\ y_2*\log(\sigma(f_{{\mathbf{w}}_1}({\mathbf{x}}_2)))+(1-y_2)\log((1-\sigma(f_{{\mathbf{w}}_1}({\mathbf{x}}_2))))&=&0\\ y_3*\log(\sigma(f_{{\mathbf{w}}_1}({\mathbf{x}}_3)))+(1-y_3)\log((1-\sigma(f_{{\mathbf{w}}_1}({\mathbf{x}}_3))))&=&0\\ y_4*\log(\sigma(f_{{\mathbf{w}}_1}({\mathbf{x}}_4)))+(1-y_4)\log((1-\sigma(f_{{\mathbf{w}}_1}({\mathbf{x}}_4))))&=&-M\\ y_1*\log(\sigma(f_{{\mathbf{w}}_2}({\mathbf{x}}_1)))+(1-y_1)\log((1-\sigma(f_{{\mathbf{w}}_2}({\mathbf{x}}_1))))&=&0\\ y_2*\log(\sigma(f_{{\mathbf{w}}_2}({\mathbf{x}}_2)))+(1-y_2)\log((1-\sigma(f_{{\mathbf{w}}_2}({\mathbf{x}}_2))))&=&-44.1\\ y_3*\log(\sigma(f_{{\mathbf{w}}_2}({\mathbf{x}}_3)))+(1-y_3)\log((1-\sigma(f_{{\mathbf{w}}_2}({\mathbf{x}}_3))))&=&-68.2\\ y_4*\log(\sigma(f_{{\mathbf{w}}_2}({\mathbf{x}}_4)))+(1-y_4)\log((1-\sigma(f_{{\mathbf{w}}_2}({\mathbf{x}}_4))))&=&-1.1\\ \end{array} \end{aligned} $$

  (10.91)

  where M represents a large number, so we have

  $$\displaystyle \begin{aligned} \begin{array}{c} \hat{L}(f_{{\mathbf{w}}_1}) = -\frac{1}{4}(-M+0+0-M)=M/2\\ \hat{L}(f_{{\mathbf{w}}_2}) = -\frac{1}{4}(0-44.1-68.2-1.1)=28.35 \end{array} \end{aligned} $$

  (10.92)

  according to Eq. (7.18). Therefore, $f_{{\mathbf {w}}_2}$ is better.

• 4. According to Eq. (7.16), we have

  $$\displaystyle \begin{aligned} \begin{array}{c} P_{{\mathbf{w}}_1}(y=1|\mathbf{x})=\sigma(2*181+1*92+20*12.4-330)=1 \\ P_{{\mathbf{w}}_1}(y=0|\mathbf{x})=1-\sigma(2*181+1*92+20*12.4-330)=0 \end{array} \end{aligned} $$

  (10.93)

  Therefore, it is predicted to 1. □

Question 6

Understand the basic idea of Bayesian linear regression by conducting research on the literature, and implement a Bayesian linear regression algorithm to compare its performance with linear regression. □

Question 7

Write a program for the adversarial attack for logistic regression. □

Question 8

Given a function f(x) = e ^x∕(1 + e ^x), how many critical points? □

Answer

0 □

Question 9

Given a function $f(x_1,x_2)= 9x_1^2+3x_2+4$, how many critical points? □

Answer

0, because there is no assignment to x ₁ and x ₂ that can make the gradient of f(x ₁, x ₂) equal to 0. □

Question 10

Consider the dataset in Table 10.7, please newanswer the following newquestion:

Table 10.7 A small dataset

• P(Wealthy = Y ) = 4∕7

• P(Wealthy = N) = 3∕7

• P(Gender = F|Wealthy = Y ) = 3∕4

• P(Gender = M|Wealthy = Y ) = 1∕4

• P(HrsWorked > 40.5|Wealthy = Y ) = 1∕4

• P(HrsWorked < 40.5|Wealthy = Y ) = 3∕4

• P(Gender = F|Wealthy = N) = 1∕3

• P(Gender = M|Wealthy = N) = 2∕3

• P(HrsWorked > 40.5|Wealthy = N) = 2∕3

• P(HrsWorked < 40.5|Wealthy = N) = 1∕3

Based on the above, please use Classify a new instance with Naive Bayes algorithm (Gender = F, HrsWorked = 44). □

Answer

Because

$$\displaystyle \begin{aligned} \begin{array}{l} P(Wealthy=Y)*P(Gender=F | Wealthy = Y)*\\ P(HrsWorked > 40.5 | Wealthy = Y) = 3/28\\ P(Wealthy=N)*P(Gender=F | Wealthy = N)*\\ P(HrsWorked > 40.5 | Wealthy = n) = 2/21\\ \end{array} \end{aligned} $$

(10.94)

we have that it will be predicted as Wealthy = Y  according to Eq. (8.11). □

Question 11

Implement the perceptron learning algorithm in Sect. 10.1, and compare the obtained binary classifier with the one obtained from logistic regression. □

Question 12

Understand how learning rate in the perceptron learning algorithm affects the learning results. Draw a curve to exhibit the change of accuracy with respect to the learning rate. □

Question 13

Use the perceptron learning algorithm to work with the XOR dataset in Example 10.3, and check the accuracy. □

Question 14

Assuming that all weights in Fig. 10.5, i.e., those numbers 1 and -1, need to be learned. Can you adapt the perceptron learning algorithm to learn the weights? □

Question 15

Understand the equivalence of applying matrix expression to compute the outputs for a dataset and the computation of outputs for individual inputs. □