Optimal frequency-hopping anti-jamming strategy based on multi-step prediction Markov decision process

Abstract

With the increasingly fierce competition for electromagnetic spectrum resources and the rapid development in artificial intelligence, frequency-hopping (FH) communication systems are facing serious intelligent jamming attacks. Under this background, game theory has become an effective tool to solve the optimal decision-making problem in communication countermeasures. In this paper, focusing on defending against sweep jamming attacks and considering the inevitable transmission delay of an actual communication system, we propose a multi-step prediction Markov decision process (MPMDP) and set up a multi-step prediction Bellman iterative equation (MPBIE). To solve the MPBIE, multi-step state transition probabilities are derived, and a simplified algorithm is designed. Combing the complete strategy we proposed, the MPMDP strategy to maximize the utility of the FH communication system is obtained. With the assumption that the jammer can learn the MPMDP strategy of the FH communication system, we furtherly study its optional sweep jamming attack strategy. The MPMDP strategy and the optional intelligent sweep jamming attack strategy as a Nash equilibrium of the communication system and the jammer are proved. Lastly, simulation results show that the property and theorems proposed in this paper are valid, and our MPMDP strategy outperforms similar algorithms in terms of the applicable range and performance.

Appendix

1.1 Proof of Property 1

With action h being adopted, the MPBIE in (5) is expanded as follows:

$$\begin{aligned} \begin{aligned}&\!\!Q(S_1,A_d^{(h_{\tau })},h)\!=\!p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})[U(C_{d-\tau +1},h)\\&\qquad \qquad \!+\!\delta (p(C_1|C_{\!d-\tau +1\!},h)V^*(C_1,A_d^{(h_{\tau })})\\&\qquad \qquad \!+\!p(J|C_{\!d-\tau +1\!},\!h)V^*(J,\!A_d^{\!(h_{\tau })\!}))]\!+\!p(J|S,A_d^{(h_{\tau })})[U(J,h)\\&\qquad \qquad \!+\!\delta (p(C_1|J,h)V^*\!(C_1,A_d^{(h_{d})})\!+\!p(J|J,\!h)V^*(J,\!A_d^{\!(h_{d})\!}))]. \end{aligned} \end{aligned}$$

(12)

In (12), $Q(S_1,A_d^{(h_{\tau })},h)$ is only related to the d-step state transition probabilities $p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})$ and $p(J|S,A_d^{(h_{\tau })})$, and the values in the two $[\cdot ]$ are equal for any $S_1\in \mathbf {S}$ and $\tau \in \{1,2,\ldots ,k\}$. Subsequently, combining (10) and (11), we find that $p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})$ and $p(J|S,A_d^{(h_{\tau })})$ are not related to the initial state $S_1$, but only related to $\tau$. Therefore, (13) can be deduced easily:

$$\begin{aligned} Q(J,A_d^{(h_{\tau })}\!,\!h)\!=\!Q(C_1,A_d^{(h_{\tau })}\!,\!h)\!=\!\ldots \!=\!Q(C_{\bar{k}}\!,\!A_d^{(h_{\tau })},h). \end{aligned}$$

(13)

On the other hand, with action s being adopted, the MPBIE in (5) is expanded as follows:

$$\begin{aligned} \begin{aligned} \!Q(S_1,\!A_d^{(h_{\tau })},\!s)&\!=p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})[U(C_{d-\tau +1},s)\\&~+\!\delta (p(C_1|C_{d-\tau +1},s)V^*(C_1,A_d^{(h_{d})})\\&~+\!p(J|C_{d-t+1},s)V^*(J,A_d^{(h_{d})}))]\\&~+\!p(J|S,\!A_d^{(h_{\tau }\!)})[U(J,s)\!+\!\delta V^*\!(J,\!A_d^{(\!h_{d}\!)})]. \end{aligned} \end{aligned}$$

(14)

Similar to the inference of (13), we obtain (15) from (14):

$$\begin{aligned} Q(J,A_d^{(h_{\tau })}\!,\!s)\!=\!Q(C_1,A_d^{(h_{\tau })}\!,\!s)\!=\ldots =\!Q(C_{\bar{k}},\!A_d^{(h_{\tau })},s). \end{aligned}$$

(15)

Combining (13), (15), and (5), we obtain (16):

$$\begin{aligned} V^*(J,A_d^{(h_{\tau })})\!=\!V^*(C_1,A_d^{(h_{\tau })})\!=\ldots =\!V^*(C_{\bar{k}},A_d^{(h_{\tau })}). \end{aligned}$$

(16)

Hence, Property 1 is proven. This property is the basis for the proofs of other theorems.

1.2 Proof of Theorem 1

According to Property 1 and combining (12) and (3), we obtain (17):

$$\begin{aligned} \begin{aligned} \!\!Q(S_1,\!A_d^{(h_{\tau })}\!,h)&\!=p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})U(C_{d-\tau +1},h)\\&\qquad +\,p(J|S,A_d^{(h_{\tau })})U(J,s)+\delta V^*(J,A_d^{(h_{\tau })})\\&=p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})R_s(T_h-T_p)\\&\qquad +\,p(J|S,A_d^{(\!h_{\tau }\!)})L_s(T_h\!-\!T_p)\!+\!\delta V^*\!(J,A_d^{(\!h_{\tau })}\!). \end{aligned} \end{aligned}$$

(17)

By calculating the difference of $Q(S_1,A_d^{(h_{\tau })},h)$ to $\tau$, we obtain (18):

$$\begin{aligned} \begin{aligned}&\!Q(S_1,A_d^{(h_{\tau })},\!h)-Q(S_1,A_d^{(h_{\tau -1})},\!h)\\ \!=&[p(C_{\!d\!-\!\tau \!+\!1\!}|S,\!A_d^{\!(h_{\tau }\!)}\!)\!-\!p(C_{\!d\!-\!\tau \!+\!2\!}|S,\!A_d^{\!(h_{\!\tau -1\!})\!})]R_s(T_h\!-\!T_p)\\&\quad +\,[p(J|S,A_d^{(h_{\tau })})-p(J|S,A_d^{(h_{\tau -1})})]L_s(T_h-T_p)\\ \!=&p(C_{\!d-\!\tau \!+\!1\!}|S,\!A_d^{(h_{\tau }\!)})p(J|C_{\!d\!-\!\tau \!+\!1\!},\!s)[(R_s\!-\!L_s)(T_h\!-\!T_p)]\\&\ge 0. \end{aligned} \end{aligned}$$

(18)

In (18), $p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})$ increases monotonically with $\tau$, $R_s(T_h\!-\!T_p)-L_s(T_h\!-\!T_p)$ and $p(J|C_{d-\tau +1},s)$ are both positive constants. Therefore, $Q(S_1,A_d^{(h_{\tau })},h)$ increases monotonically with $\tau$.

According to Property 1 and combining (14) and (3), we obtain (19):

$$\begin{aligned} \begin{aligned}&Q(S_1,A_d^{(h_{\tau })}\!,\!s)\\&=p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})[U(C_{d-\tau +1},s)\\&~+\delta (p(C_{d-\tau +2}|C_{d-\tau +1},s))V^*(C_{d-\tau +2},A_d^{(h_{\tau -1})})\\&~+p(J|C_{d-\tau +1},s)V^*(J,A_d^{(h_{\tau -1})})]\\&~+p(J|S,\!A_d^{(h_{\tau })})[U(J,\!s)\!+\!\delta (p(J|J,s))V^*\!(J,A_d^{(h_{\!\tau -1\!})})]\\&=p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})R_sT_h+p(J|S,A_d^{(h_{\tau })})L_sT_h\\&~+\delta V^*(J|S,A_d^{(h_{\tau -1})}). \end{aligned} \end{aligned}$$

(19)

By calculating the difference of $Q(S_1,A_d^{(h_{\tau })},s)$ to $\tau$, we obtain (20):

$$\begin{aligned} \begin{aligned}&Q(S_1,A_d^{(h_{\tau })},s)-Q(S_1,A_d^{(h_{\tau -1})},s)\\&=p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})p(J|C_{d-\tau +1},s)R_sT_h\\&\quad -\,p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})p(J|C_{d-\tau +1},s)L_sT_h\\&\quad +\,\delta [V^*(J,A_d^{(h_{\tau -1})})-V^*(J,A_d^{(h_{\tau -2})})]\\&=p(C_{d-\tau +1}|S,A_d^{(h_{\tau })})p(J|C_{d-\tau +1},s)[R_sT_h-L_sT_h]\\&\quad +\,\delta [V^*(J,A_d^{(h_{\tau -1})})-V^*(J,A_d^{(h_{\tau -2})})]. \end{aligned} \end{aligned}$$

(20)

Equation (20) is a recursive expression, which can be proven using the recursive method. With $\tau =1,2$ and action h being adopted, we obtain (21):

$$\begin{aligned} \begin{aligned} Q(J,A_d^{(h_{1})}\!,\!h)&=p(C_d|J,A_d^{(h_{\tau })})U(C_d,h)\\&~+p(J|J,A_d^{(h_{\tau })})U(J,h)\!+\!\delta V^*(J,A_d^{(h_{d})}),\\ \!Q(J,A_d^{(s)}\!,\!h)&=U(J,h)+\delta V^*(J,A_d^{(h_{d})}). \end{aligned} \end{aligned}$$

(21)

As a result of $U(C_d,h)>0$ and $U(J,h)<0$, we obtain (22) form (21):

$$\begin{aligned} Q(J,A_d^{(h_{1})},h)>Q(J,A_d^{(s)},h). \end{aligned}$$

(22)

Similarly, With $\tau =1,2$ and action s being adopted, we obtain (23):

$$\begin{aligned} \begin{aligned} Q(J,A_d^{(h_{1})}\!,\!s)&=p(C_d|J,A_d^{(h_{\tau })})U(C_d,s)\\&~+p(J|J,A_d^{(h_{\tau })})U(J,s)\!+\!\delta V^*(J,A_d^{(s)}),\\ Q(J,A_d^{(s)}\!,\!s)&=U(J,s)+\delta V^*(J,A_d^{(s)}). \end{aligned} \end{aligned}$$

(23)

Similarly, given that $U(C_d,h)>0$ and $U(J,h)<0$, we obtain (24):

$$\begin{aligned} Q(J,A_d^{(h_{1})},s)>Q(J,A_d^{(s)},s)=Q(J,A_d^{(h_0)},s). \end{aligned}$$

(24)

Combining (22), (24), and (5), we obtain (25):

$$\begin{aligned} V^*(J,A_d^{(h_{1})})>V^*(J,A_d^{(s)}). \end{aligned}$$

(25)

Substituting (25) back into (20), we obtain (26):

$$\begin{aligned} Q(S_1,A_d^{(h_{2})},s)-Q(S_1,A_d^{(h_1)},s)>0. \end{aligned}$$

(26)

Given that $Q(S_1,A_d^{(h_{\tau })},h)$ increases monotonically with $\tau$, we obtain (27):

$$\begin{aligned} V^*(S_1,A_d^{(h_{2})})>V^*(S_1,A_d^{(h_{1})}). \end{aligned}$$

(27)

Substituting (27) back into (20), (28) can be deduced easily:

$$\begin{aligned} Q(S_1,A_d^{(h_{3})},s)>Q(S_1,A_d^{(h_2)},s). \end{aligned}$$

(28)

Continuing the recursive process according to the above rules until $\tau =d$, we obtain that $Q(S_1,A_d^{(h_{\tau })},s)>Q(S_1,A_d^{(h_{\tau -1})},s)$ is true for any $1\le \tau \le d$. $Q(S_1,A_d^{(h_{\tau })},s)$ increases monotonically with $\tau$. The first term on the right side of the equation in (20) also increases monotonically with $\tau$, and its increasing ratio is always greater than that of $Q(S_1,A_d^{(h_{\tau })},h)$ for any $\tau$ (reference (18)). Besides, the increasing ratio of the second term on the right side of the equation in (20) is always greater than 0.

In conclusion, both $Q(S_1,A_d^{(h_{\tau })},s)$ and $Q(S_1,A_d^{(h_{\tau })},h)$ increase monotonically with $\tau$, and the increasing ratio of $Q(S_1,A_d^{(h_{\tau })},s)$ is always greater than that of $Q(S_1,A_d^{(h_{\tau })},h)$. Therefore, there is at most one intersection $\tau ^*$ for which $a_{d+1}=h$ when $\tau \le \tau ^*$; otherwise, $a_{d+1}=s$.

Hence, Theorem 1 is proven.

1.3 Proof of Theorem 2

According to (5), we can obtain (29):

$$\begin{aligned} \begin{aligned} Q(J,A_d^{({s})}\!,\!s)&=p(J|J,A_d^{(s)})[U(J,s)\!+\!\delta V^*(J,A_d^{(s)})]\\&=U(J,s)+\delta V^*(J,A_d^{(s)}),\\ Q(J,A_d^{({s})}\!,\!h)&=p(J|J,A_d^{(s)})[U(J,h)\!+\!\delta V^*(J,A_d^{(h_d)})]\\&=U(J,h)+\delta V^*(J,A_d^{(h_d)}). \end{aligned} \end{aligned}$$

(29)

Utilizing the property of the second term in (20) is always greater than 0, we find that $V^*(J,A_d^{(h_{d})})>V^*(J,A_d^{(h_{0})})=V^*(J,A_d^{(s)})$. therefore, (30) can be obtained as follows:

$$\begin{aligned} \begin{aligned} Q(J,A_d^{(s)},h)&=U(J,h)+\delta V^*(J,A_d^{(h_d)})\\&>U(J,h)+\delta V^*(J,A_d^{(s)})\\&>U(J,s)+\delta V^*(J,A_d^{(s)})\\&=Q(J,A_d^{(s)},s).\\ \end{aligned} \end{aligned}$$

(30)

Hence, Theorem 2 is proven.

1.4 Proof of Theorem 3

With action h being adopted and utilizing property 1, we obtain (31) by expanding (5):

$$\begin{aligned} \begin{aligned}&Q(C_k,A_d^{(s)},h)\\&\!=\!p(C_{\!k+d}|C_k,\!A_d^{\!(s)\!})[U(C_{\!k+d},h)\!+\!\delta (p(C_1|C_{\!k+d},h)V^*\!(C_1\!,\!A_d^{\!(h_d\!)})\\&\quad \!+\!p(J|C_{k+d},h)V^*(J,,A_d^{(h_d)}))]+p(J|C_k,A_d^{(s)})[U(J,h)\\&\quad \!+\!\delta (p(C_1|C_{\!k+d},h)V^*(C_1\!,\!A_d^{\!(h_d)\!})+p(J|C_{\!k+d},h)V^*(J\!,\!A_d^{\!(h_d)\!}))]\\&\!=\!p(C_{k+d}|C_k,A_d^{(s)})U(C_{k+d},h)+p(J|C_k,A_d^{(s)})U(J,h)\\&\quad \!+\!\delta V^*(J,A_d^{(h_d)})). \end{aligned} \end{aligned}$$

(31)

The difference $Q(C_k,A_d^{(s)},h)$ to k is calculated as follows:

$$\begin{aligned} \begin{aligned}&Q(C_k,A_d^{(s)},h)-Q(C_{k-1},A_d^{(s)},h)\\&~=[p(C_{d+k}|C_k,A_d^{(s)})-p(C_{d+k-1}|C_{k-1},A_d^{(s)})]U(C_{k},h)\\&\quad +[p(J|C_k,A_d^{(s)})-p(J|C_{k-1},A_d^{(s)})]U(J,h)\\&~=[p(C_{\!d\!+\!k}|C_k,\!A_d^{\!(s)\!})\!-\!p(C_{\!d+k-1\!}|C_{\!k-1\!},\!A_d^{\!(s)\!})]\\&\quad \times [U(C_{k},\!h)\!-\!U(J,\!h)]<0. \end{aligned} \end{aligned}$$

(32)

In (32), $Q(C_k,A_d^{(s)},h)$ decreases monotonically with k. In addition, comparing (30) and (31), we obtain (33):

$$\begin{aligned} V^*\!(J,A_d^{\!(s)\!})\!=\!Q(J,A_d^{\!(s)\!},h)\!<\!Q(C_k,A_d^{\!(s)\!},h)\!\le \! V^*\!(C_k,A_d^{\!(s)\!}). \end{aligned}$$

(33)

With action s being adopted and utilizing of property 1, we obtain (34) by expanding (5):

$$\begin{aligned} \begin{aligned}&Q(C_k,A_d^{(s)},s)=p(C_{k+d}|C_k,A_d^{(s)})[U(C_{k+d},s)\\&~\!+\!\delta p(C_{\!k\!+\!d\!+\!1\!}|C_{\!k\!+\!d},\!s)V^*\!(C_{\!k\!+\!d\!+\!1\!},\!A_d^{\!(s)\!})\!+\!\delta p(J|C_{k\!+\!d},\!s)V^*\!(J,\!A_d^{\!(s\!)})]\\&\quad +p(J|C_{k},A_d^{(s)})[U(J,s)+\delta V^*(J,A_d^{(s)})]\\&\!=\!p(C_{k+d}|C_k,A_d^{(s)})U(C_{k+d},s)\\&\quad +\delta p(C_{\!k+d+1\!}|C_k,\!A_{\!d+1\!}^{\!(s)\!})V^*\!(C_{k+d+1\!},A_d^{\!(s)\!})\!+\!p(J|C_k,A_d^{\!(s)\!})U(J,s)\\&\quad +\delta (1\!-\!p(C_{\!k+d+1\!}|C_k,\!A_{d+1}^{\!(s)})) V^*(J,A_d^{\!(s)})\\&\!<\!p(C_{\!k\!+\!d}|C_k,\!A_d^{\!(s)\!})U(C_{\!k\!+\!d},\!s)\!+\!p(J|C_k,\!A_{d}^{\!(s)\!})U(J,\!s)\!+\!\delta V^*\!(J,\!A_d^{\!(s)}\!). \end{aligned} \end{aligned}$$

(34)

The inequality in (34) makes use of the result of (33). In (34), $U(J,s)<0$ and $U(C_{k+s},s)>0$, $p(C_{k+d}|C_k,A_d^{(s)})$ and $p(J|C_k,A_d^{(s)})$ decreases and increases monotonically with k. Therefore, $Q(C_k,A_d^{(s)},h)$ decreases monotonically with k. Combining the property that $Q(C_k,A_d^{(s)},h)$ decreases monotonically with k in (32), we find that $V^*(C_k,A_d^{(s)})$ decreases monotonically with k.

So far, we have proved that all $Q(C_k,A_d^{(s)},s)$, $Q(C_k,A_d^{(s)},h)$, and $V^*(C_k,A_d^{(s)})$ decrease monotonically with k. However, we have not determined the number of crossings between $Q(C_k,A_d^{(s)},s)$ and $Q(C_k,A_d^{(s)},h)$. Therefore, it is necessary to furtherly compare the monotonically decreasing ratio of $Q(C_k,A_d^{(s)},s)$ and $Q(C_k,A_d^{(s)},h)$. Calculate the difference of $Q(C_k,A_d^{(s)},s)$ to k as follows:

$$\begin{aligned} \begin{aligned}&Q(C_k,A_d^{(s)},s)-Q(C_{k-1},A_d^{(s)},s)\\ =&[p(C_{k+d}|C_k,A_d^{(s)})-p(C_{k+d-1}|C_{k-1},A_d^{(s)})]U(C_{k},s)\\&\quad +\,\delta p(C_{k+d+1}|C_{k},A_{d+1}^{(s)})[V^*(C_{k+d+1},A_d^{(s)})-V^*(J,A_d^{(s)})]\\&\quad +\,[p(J|C_{k},A_d^{(s)})-p(J|C_{k-1},A_d^{(s)})]U(J,s)\\&\quad +\,\delta p(C_{k+d}|C_{k-1},A_{d+1}^{(s)})[V^*(J,A_d^{(s)})-V^*(C_{k+d},A_d^{(s)})]\\<&[p(C_{\!k+d}|C_k,\!A_d^{\!(s)\!})\!-\!p(C_{\!k+d-1\!}|C_{\!k-1},\!A_d^{\!(s)\!})][U(C_{k},s)\!-\!U(J,\!s)]\\&\quad +\,\delta [p(C_{\!k\!+\!d\!+\!1\!}|C_k,\!A_{\!d\!+\!1}^{(s)})\!-\!p(C_{\!k\!+\!d}|C_{\!k\!-\!1},\!A_{d+1}^{(s)})][V^*\!(C_{k+d},\!A_d^{\!(s)\!})\\&\quad -\,V^*(J,A_d^{(s)})]\\ <&[p(C_{\!k+d}|C_k,A_d^{\!(s)\!})\!-\!p(C_{\!k+d-1\!}|C_{\!k-1\!},\!A_d^{(s)})][U(C_{k},\!s)\!-\!U(J,s)]. \end{aligned} \end{aligned}$$

(35)

The first inequality in (35) utilizes the property that $V^*(C_{k}|C_k,A_{d}^{(s)})$ decreases monotonically with k. The second inequality in (35) utilizes the property that $p(C_{k+d}|C_k,A_{d}^{(s)})$ decreases monotonically with k and the result of (33).

Comparing (35) and (32), we find the relationship as (36):

$$\begin{aligned} \begin{aligned}&Q(C_k,A_d^{\!(s)\!},s)\!-\!Q(C_{\!k-1},A_d^{\!(s)\!},s)\\&~~~~<Q(C_k,A_d^{\!(s)\!},h)\!-\!Q(C_{k-1},A_d^{\!(s)\!},h) <0. \end{aligned} \end{aligned}$$

(36)

In other words, $Q(C_k,A_d^{(s)},s)$, $Q(C_k,A_d^{(s)},h)$, and $V^*(C_k,A_d^{(s)})$ all decrease monotonically with k, and the decreasing ratio of $Q(C_k,A_d^{(s)},s)$ is always greater than $Q(C_k,A_d^{(s)},h)$. Therefore, there is at most one intersection k, such that when $k>k^*$, $a_{d+1}=h$; otherwise, $a_{d+1}=s$ .

Hence, Theorem 3 is proven.