Performance Analysis for Secure Cooperative Systems Under Unreliable Backhaul Over Nakagami-m Channels

Abstract

In this paper, the secrecy performance of cooperative heterogeneous networks with unreliable backhaul over Nakagami-m fading channels is investigated. To secure the proposed system, a friendly jammer is considered to confuse eavesdroppers. To transmit the signals from the source to the destination, a two-phase transmitter/relay selection scheme is proposed. The best transmitter is selected when the signal-to-noise ratio at the relays is maximized. In the second phase, the best relay is chosen when the jamming signal-to-interference-plus-noise ratio of the eavesdroppers is minimized. To investigate the system performance, closed -form expressions are derived for the secrecy outage probability, ergodic capacity and non-zero achievable secrecy rate. In order to gain an insight into the system, asymptotic analysis is also provided. The results show that the degree of cooperative transmission and backhaul reliability are key parameters in the system and these parameters determine the secrecy performance.

1 Introduction

Due to the increasing wireless data traffic demand, future networks will become more dense and heterogeneous. In heterogeneous networks (HetNets), macro cells and small cells will be used to increase the capacity needed for this rise in traffic demand and to offload traffic. A backhaul link will connect these small cells with the core network. The traditional backhaul is wired and can ensure the connection. However, the cost of the deployment and maintenance is high, especially when a large number of small cells is needed to cover dense scenarios. Moreover, small cells may not need such a highly reliable backhaul as traditional macro cells do [27]. This is because small cells serve a lower traffic capacity than macro cells. In this way, wireless backhaul has emerged as an alternative and attractive approach due to its low cost and flexibility. However, wireless backhaul is not as reliable as its wired backhaul counterpart due to wireless channel impairments such as non-line-of-sight (nLOS) propagation and channel fading [10].

The reliability of the backhaul is an important factor in HetNets and research has been undertaken to investigate how the backhaul reliability can affect the system performance [2, 10,11,12,13,14,15, 19,20,21,22, 31]. In [2], the authors considered co-channel inter-cell interference (ICI) in HetNets with unreliable backhaul and coordinated multi-point (CoMP) transmission was considered to reduce the interference. In [12,13,14,15, 22, 31], the impact of unreliable backhaul on cooperative relay systems was investigated. The outage probability of finite-sized selective relaying systems with unreliable backhaul was studied in [15]. A cognitive network with unreliable backhaul was investigated in [21], and asymptotic analysis showed that performance was mainly decided by backhaul reliability. For all the research mentioned on unreliable backhaul connections, backhaul reliability is a key factor in the system performance. Therefore, in a HetNet context it is essential for us to investigate backhaul reliability.

Another aspect that cannot be ignored is security [5]. For a complete study in HetNets system, security needs to be considered. The traditional way to enhance security is deploying cryptographic techniques across upper layers. However, it consumes significant power to encrypt and decrypt data [25]. In addition, with the development of quantum computing, key schemes can be broken and the key infrastructure become insecure [30]. In recent years, physical layer security (PLS) has become increasingly popular to secure wireless communications [9, 26, 28]. The main idea of PLS is that the wireless channels are random and unpredictable, thus this can be exploited to keep the information confidential from eavesdroppers. Wyner proposed that when the main channel has better propagation conditions than the eavesdroppers’, the communication between legitimate users could be secure [29]. However, when the wiretap channel is better, the secrecy rate can even drop to zero [7]. Various PLS techniques have been investigated to tackle this problem and enhance the security of the main channel; one of the main techniques is using cooperative jamming to generate artificial noise to confuse eavesdroppers [1, 4, 6, 7, 16, 17, 23, 32]. In [6, 32], the authors considered PLS and energy harvesting with a friendly jammer, and the authors in [6] also proposed joint jammer and relay selection schemes. In related work [16], a jammer is assumed to be an energy constrained node with no power of its own and can harvest power from the source node, but cooperative relaying was not considered in this work. However, all of the research above ignored the reliability of the backhaul. As discussed an unreliable wireless backhaul results in poor performance. It is essential to consider backhaul reliability when studying PLS in a small cell HetNet contexts.

Research in [13, 14, 22, 31] has taken into account PLS in relay systems with unreliable backhaul. In [31], the authors studied PLS and energy harvesting. In [14], the authors studied PLS in full-duplex cooperative relay systems. In [13], multiple eavesdroppers that can wiretap information from relay and transmitters are considered in a finite-sized cooperative system. In [22], a friendly jammer was used to confuse eavesdroppers in single carrier systems.

Cooperative jamming in the presence of wireless backhaul over Nakagami-m fading channels has not been studied yet. In this research, we investigate the cooperative jamming in a small cells HetNet context with unreliable backhaul over Nakagami-m fading channel. In our system model, an outage occurs when the system is either not reliable or not secure, hence we assess the secrecy outage probability as a performance parameter. Our main contributions are summarized as below:

• We investigate the secrecy performance of cooperative systems by exploiting cooperative relay and jamming signals in the presence of unreliable backhaul links between macro-cells and small-cells over Nakagami-m fading channels.

• A two-phase transmitter/relay selection scheme is proposed. The achievable SNR at the relays is maximized by applying the best small cell transmitter selection in the first phase. The relay selection scheme is deployed in the second phase to minimize the instantaneous signal-to-interference-plus-noise ratio (SINR) at the eavesdroppers.

• Analytical expressions to evaluate the secrecy outage probability, non-zero achievable secrecy rate, and ergodic capacity are derived in closed-form. The asymptotic secrecy expressions are also attained to gain full insights into the impact of backhaul reliability on the network secrecy performance in the high SNR regime.

• The effect of the number of small-cell transmitters, relays, eavesdroppers and backhaul reliability on the system performance is investigated.

The remainder of the paper is organized as follows. System and channel models are described in Section 2. Derivation of the SNR distributions in the proposed system is obtained in Section 3. The closed-form expressions for outage probability, ergodic capacity and symbol error rate as well as the asymptotic analysis are carried out in Section 4, while numerical results are presented in Section 5. Finally, the paper is concluded in Section 6.

Notation:P[⋅] is the probability of occurrence of an event. For a random variable X, F_X(⋅) denotes its cumulative distribution function (CDF) and f_X(⋅) denotes the corresponding probability density function (PDF). max(⋅) and min(⋅) denote the maximum and minimum of their arguments, respectively.

2 System model

We consider a HetNet system with a macro base station, S, K small cells, T_{{1,⋯ ,K}}, M relays, R_{{1,⋯ ,M}}, a jammer, J, N eavesdroppers, E_{{1,⋯ ,N}} and a destination, D, as shown in Fig. 1. S is connected to T_k via wireless backhaul. We assume that there is no direct link between T_k and D because of the poor channel condition. T_k sends information to D with the help of R_m. A single J transmits jamming signals in the system, and we assume that the jamming signals can be nulled out at D [6]. E_n wiretaps the information transmitted from R_m. All of the nodes are equipped with single antenna and operate in half-duplex. We assume that all the channels are Nakagami-m fading, and the channel power gains are gamma distributed. The cumulative distribution function (CDF) and probability density function (PDF) of the random variable X can be written as

$$\begin{array}{@{}rcl@{}} &F_{{X}}\left( {x}\right)= 1-\exp\left( -\frac{x}{\theta_{X}} \right)\sum\limits_{i = 0}^{m_{X}-1}\frac{1}{i!}\left( \frac{x}{\theta_{X}}\right)^{i}. \end{array} $$

(1)

$$\begin{array}{@{}rcl@{}} &f_{{X}}\left( {x}\right)= \frac{x^{m_{X} -1}}{{\Gamma}{(m_{X})}\theta_{X}^{m_{X}}}\exp\left( -\frac{x}{\theta_ X}\right). \end{array} $$

(2)

Fig. 1

A cooperative heterogeneous network with multiple small cell transmitters, relays and a friendly jammer in the presence of eavesdroppers

Where Γ(⋅,⋅) is the incomplete gamma function [8, Eq. (8.352.6)].

Backhaul reliability is modeled as a Bernoulli process $\mathbb {I}_k$ with success probability s_k where $\mathbb {P}(\mathbb {I}_{k^{*}}= 1)=s_{k}$ and $\mathbb {P}(\mathbb {I}_{k^{*}}= 0)= 1-s_{k}$. This indicates that T_k is participating in the transmission if the message is successfully delivered over its dedicated backhaul with probability s_k whereas it defers its transmission with probability 1 − s_k.

We assume the global channel state information (CSI) is available, which is a common assumption in PLS [22]. The CSI of the eavesdroppers can be known when eavesdroppers are active in the network and their status can be monitored [3].

In the first hop, the received signals at R_m are of the form

$$\begin{array}{@{}rcl@{}} &\textit{\textbf{y}}_{R} = \sqrt{\mathcal{P}_{t} \alpha_T^{k,m}}h_{T_k R_m} \mathbb{I}_k \textbf{x} + z , \end{array} $$

(3)

where $\mathcal {P}_{t}$ is the transmit power at T_k and $h_{T_k R_m}$ is the channel coefficient of the link T_k − R_m, x is the unit power transmitted symbol and z is the complex additive white Gaussian noise (AWGN) with zero mean and variance σ, i.e., z ∼ CN(0,σ). The path loss component corresponding to $h_{T_{k} R_{m}}$ is denoted as $\alpha _T^{k,m}$, respectively.

In the second hop, the received signals at D are of the form

$$\begin{array}{@{}rcl@{}} \textit{\textbf{y}}_{D} = \sqrt{\mathcal{P}_{r} \alpha_D^{m}} h_{R_m D} \textbf{x} + z , \end{array} $$

(4)

where $\mathcal {P}_{r}$ is the transmit power at relays and $h_{R_m D}$ is the channel coefficient of the link R_m − D. The path loss component corresponding to $h_{R_m D}$ is represented by $\alpha _D^{m}$, respectively.

Similarly, the received signals at E are of the form

$$\begin{array}{@{}rcl@{}} \textit{\textbf{y}}_{E} = \sqrt{\mathcal{P}_{r} \alpha_E^{m,n}} h_{R_m E_n} \textbf{x} + \sqrt{\mathcal{P}_{j} \alpha_J^{n}} h_{J E_n} +z , \end{array} $$

(5)

where $\mathcal {P}_{j}$ is the transmit power at the jammer, $h_{R_m E_n}$ is the channel coefficient of the link R_m − E_n and $h_{J E_n}$ is the channel coefficient of the link J − E_n. The path loss component corresponding to $h_{R_m E_n}$ and $h_{J E_n}$ are denoted as $\alpha _E^{m,n}$ and $\alpha _J^{n}$, respectively.

We assume that the unreliable backhaul links are independent from the indices of the K transmitters, i.e., s_k = s,∀k.

3 SNR distributions

In this section, SNR distributions are derived firstly which are necessary for system secrecy performance analysis in the next section.

From Eq. 3, the SNR from T_k and R_m can be given as

$$\begin{array}{@{}rcl@{}} SNR_{T_{k} R_{m}}&=&\frac{\mathcal{P}_{t} \alpha_T^{k,m} |h_{T_k R_m}|^2}{\sigma_n^2} \mathbb{I}_k =\widetilde{\alpha}_R |h_{T_k R_m}|^2 \mathbb{I}_k \\ &=& \lambda^{k,m} \mathbb{I}_k, \end{array} $$

(6)

where $\widetilde {\alpha }_R = \frac {\mathcal {P}_{t} \alpha _T^{k,m}} {\sigma _n^2}$. λ^k,m ∼Ga(m_R,𝜃_R).

Similarly, according to Eqs. 4 and 5, the SNR between R_m and D and the SINR between R_m and E_n can be obtained as

$$\begin{array}{@{}rcl@{}} SNR_{R_{m} D}= \frac{\mathcal{P}_{r} \alpha_D^{m} |h_{R_m D}|^2}{\sigma_n^2} = \widetilde{\alpha}_D |h_{R_m D}|^2={\lambda_{D}^{m}}, \end{array} $$

(7)

$$\begin{array}{@{}rcl@{}} SINR_{R_{m} E_{n}} &=& \frac{\mathcal{P}_{r} \alpha_E^{m,n} |h_{R_m E_n}|^2} {\sigma_n^2 + \mathcal{P}_{j} \alpha_J^{n} |h_{J E_n}|^2} = \frac{\widetilde{\alpha}_E |h_{R_m E_n}|^2} {1+ \widetilde{\alpha}_J |h_{J E_n}|^2} \\&=&\frac{\lambda^{m,n}}{1+ {\lambda_{J}^{n}}}, \end{array} $$

(8)

where $\widetilde {\alpha }_D=\frac {\mathcal {P}_{r} \alpha _D^{m}} {\sigma _n^2}$, $\widetilde {\alpha }_E = \frac {\mathcal {P}_{r} \alpha _E^{m,n}}{\sigma _n^2}$ and $\widetilde {\alpha }_J = \frac {\mathcal {P}_{j} \alpha _J^{n}}{\sigma _n^2}$. ${\lambda _{D}^{m}} \sim \text {Ga}(m_{D},\theta _{D})$, λ^m,n ∼Ga(m_E,𝜃_E) and ${\lambda _{J}^{n}} \sim \text {Ga}(m_{J},\theta _{J})$.

In order to achieve a high performance of the considered system, our selection scheme is to maximize the performance at the relays and destination and minimize the performance at the eavesdroppers.

3.1 Distribution of the link $T_{k^{*}}-R_{m}$

In the first hop, each relay selects a small cell that can achieve the best performance of the link T_k − R_m. In this way, the best small cell is selected as

$$\begin{array}{@{}rcl@{}} k^{*} = \arg \max\limits_{k = 1,...,K} SNR_{T_{k} R_{m}}, \end{array} $$

(9)

Corresponding CDF of $SNR_{T_{K^{*}} R_{m}}$ is given as

$$\begin{array}{@{}rcl@{}} F_{{SNR_{T_{k^{*}} R_{m}}}}\left( {x}\right) &=& 1+ \sum\limits_{k = 1}^{K} \sum\limits_{\omega_{1},...,\omega_{m_{R}}}^{k} \binom{K}{k} \left( \frac{k!}{\omega_{1}!...\omega_{m_{R}}!} \right) \\ && \frac{ (-1)^{k} s^{k}} {{\prod}_{t = 0}^{m_{R}-1} \left( t!(\theta_R)^{t} \right)^{\omega_{t + 1}}}\\ &&\times x^{{\sum}_{t = 0}^{m_{R}-1}t \omega_{t + 1}} e^{-k x/\theta_R}. \end{array} $$

(10)

The proof is given in Appendix A.

3.2 Distribution of the link $R_{m} -E_{n^{*}}$

In the second hop, the eavesdropper is selected when the SINR between R_m and E_n is maximum to enhance its performance,

$$\begin{array}{@{}rcl@{}} n^{*} = \arg \max_{k = 1,...,K} SINR_{R_{m} E_{n}}. \end{array} $$

(11)

Corresponding CDF of $SINR_{R_{m} E_{n^{*}}}$ can be derived according to Appendix B.

3.3 Distribution of the link $R_{m^{*}}-E_{n^{*}}$

The relay is selected when the SNR of the link $R_{m}-E_{n^{*}}$ is minimized. It can be formulated as

$$\begin{array}{@{}rcl@{}} m^{*} = \arg \min_{m = 1,...,M} SINR_{R_{m} E_{n^{*}}}, \end{array} $$

(12)

corresponding PDF of $SINR_{R_{m^{*}} E_{n^{*}}}$ can be derived according to Appendix C and corresponding CDF of $SINR_{R_{m^{*}} E_{n^{*}}}$ can be obtained as

$$\begin{array}{@{}rcl@{}} F_{{SINR_{R_{m^{*}} E_{n^{*}}}}}\left( {x}\right) = {\int}_{0}^{x} f_{{SINR_{R_{m^{*}} E_{n^{*}}}}}\left( {t}\right) dt. \end{array} $$

(13)

3.4 Distribution of the end-to-end SNR

The relays use decode-and-forward (DF) protocol for its high system performance. This is because the interference is lower in DF protocol compared with amplify-and-forward (AF) protocol [31]. In this way, the end-to-end SNR of the considered system at the destination is given by

$$\begin{array}{@{}rcl@{}} SNR_{DF} = \min(SNR_{T_{k^{*}} R_{m^{*}}}, SNR_{R_{m^{*}} D}), \end{array} $$

(14)

where $SNR_{T_{k^{*}} R_{m^{*}}}$ is the SNR from the selected small cell transmitter to the selected relay, and $SNR_{R_{m^{*}} D}$ is the SNR from the selected relay to the destination.

Corresponding CDF of the end-to-end SNR_DF can be obtained as

$$\begin{array}{@{}rcl@{}} F_{{SNR_{DF}}}\left( {x}\right) &=& 1 - [1- F_{{SNR_{T_{k^{*}} R_{m^{*}}}}}\left( {x}\right)]\\ &&\times[1-F_{{SNR_{R_{m^{*}} D}}}\left( {x}\right)]. \end{array} $$

(15)

According to Eq. 15 and by applying binomial and multinomial theorems, the CDF of the end-to-end SNR obtained at D can be given as

$$\begin{array}{@{}rcl@{}} F_{{SNR_{DF}}}\left( {x}\right) &=& 1+ \sum\limits_{k = 1}^{K} \sum\limits_{q = 0}^{m_{D}-1} \sum\limits_{\omega_{1},...,\omega_{m_{R}}}^{k} \binom{K}{k} \left( \frac{k!}{\omega_{1}!...\omega_{N_{R}}!} \right) \\ &&\times(-1)^{k} s^{k} \frac{ 1} {q! {\prod}_{t = 0}^{m_{R}-1} \left( t!(\theta_R)^{t} \right)^{\omega_{t + 1}}} \left( \frac{1}{\theta_D} \right)^{q} \\ &&\times x^{{\sum}_{t = 0}^{m_{R}-1}t \omega_{t + 1} + q} e^{(-k /\theta_R + 1/\theta_D) x}. \end{array} $$

(16)

4 Secrecy performance analysis

This section derives the performances of secrecy outage probability, non-zero achievable secrecy rate and ergodic capacity utilizing the SNR distributions obtained in the previous section. Towards deriving these performances, secrecy rate is required to be defined first. Secrecy capacity is equal to the difference between main channel and the wiretap channel, which is given by [31]

$$\begin{array}{@{}rcl@{}} \mathcal{C}_{S} = \frac{1}{2} \left[\log_{2} (1 + SNR_{DF}) - \log_{2} (1 + SINR_{R_{m^{*}} E_{n^{*}}}) \right]^{+} ,\\ \end{array} $$

(17)

where [x]⁺ = max(x,0). In addition, log2(1 + SNR_DF) is the instantaneous capacity obtained at D from selected relay and $\log _{2} (1+ SNR_{R_{m^{*}} E_{n^{*}}})$ is the instantaneous capacity of the channel from selected relay to the selected eavesdropper.

4.1 Secrecy outage probability

The secrecy outage probability is introduced to evaluate the system security and it is defined as the probability that the instantaneous secrecy capacity is less than a positive target secrecy rate 𝜃 [31], i.e.,

$$\begin{array}{@{}rcl@{}} \mathcal{P}_{out} (\theta) &=& Pr(\mathcal{C}_{S} < \theta) \\ &=& {\int}_{0}^{\infty} F_{{SNR_{DF}}}\left( {2^{2 \theta}(1+x)-1}\right) \\ && \times f_{{SINR_{R_{m^{*}} E_{n^{*}}}}}\left( {x}\right) dx. \end{array} $$

(18)

Substitute (15) and (32) into (18), and with the help of [24, Eq. (2.3.6.9)], [8, Eq. (9.211.4)], the expression for secrecy outage probability can be given as (19).

$$\begin{array}{@{}rcl@{}} \mathcal{P}_{out} (\theta) &=& 1+J \widetilde{\sum\limits_{D}} \widetilde{\sum\limits_{E}} \sum\limits_{\alpha= 0}^{\beta} \binom{\beta}{\alpha} ({\Upsilon} - 1)^{\beta - \alpha} \\ &&\times({\Upsilon})^{\alpha} {\theta_E}^{\widetilde{\varphi}_{3}} e^{-{\Phi} ({\Upsilon} - 1)} (J_{1} -J_{2} +J_{3} ), \end{array} $$

(19)

where

$$ \left\{\begin{array}{lllll} J_{1} &= Q_{1} {\Gamma}(\widetilde{\varphi}_{2}+ \alpha + 1) {\epsilon}^{\widetilde{\varphi}_{2}+ \alpha + 1 -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \alpha + 1, \widetilde{\varphi}_{2}+ \alpha + 2 -\widetilde{\varphi}_{3}, \epsilon({\Phi} {\Upsilon} + \widetilde{\varphi}_{1}) \right), \\ J_{2} &= Q_{2} {\Gamma}(\widetilde{\varphi}_{2}+ \alpha) {\epsilon}^{\widetilde{\varphi}_{2}+ \alpha -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \alpha , \widetilde{\varphi}_{2}+ \alpha + 1 -\widetilde{\varphi}_{3}, \epsilon({\Phi} {\Upsilon} + \widetilde{\varphi}_{1}) \right), \\ J_{3} &=Q_{3} {\Gamma}(\widetilde{\varphi}_{2}+ \alpha + 2) {\epsilon}^{\widetilde{\varphi}_{2}+ \alpha + 2 -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \alpha + 2, \widetilde{\varphi}_{2}+ \alpha + 3 -\widetilde{\varphi}_{3}, \epsilon({\Phi} {\Upsilon} + \widetilde{\varphi}_{1}) \right). \end{array}\right. $$

______________________________________________________________________________________________

and $\beta = {\sum }_{t = 0}^{m_{R}-1}t$, Υ = 2^2𝜃, $\epsilon = \frac {\theta _E}{\theta _J}$$J = \frac {M N}{(\theta _J)^{m_{J}} (m_{J}-1)!}$.

$$\begin{array}{@{}rcl@{}} Q_{1} &=& \frac{1/\theta_J + m_{J} + j -i}{\theta_E}, Q_{2} \stackrel{\triangle}{=} \frac{i}{\theta_J}, Q_{3} = \frac{1}{(\theta_E)^{2}}.\\ {\varphi_{1}^{N}} &=& n/\theta_E, {\varphi_{2}^{N}} \stackrel{\triangle}{=} \sum\nolimits_{t = 0}^{m_{E}-1}t \vartheta_{t + 1}, {\varphi_{3}^{N}} \\ &=& \sum\nolimits_{\eta_{1}= 0}^{0} (m_{J}+\eta_{1}) \mu_{1,\eta_{1} + 1} + \sum\nolimits_{\eta_{2}= 0}^{1} (m_{J}+\eta_{2}) \\ &&\times \mu_{2,\eta_{2} + 1} +...+ \sum\nolimits_{\eta_{m_{E}}= 0}^{m_{E}-1} (m_{J}+\eta_{m_{E}}) \mu_{m_{E},\eta_{m_{E}} + 1}.\\ \end{array} $$

$$\begin{array}{@{}rcl@{}} \widetilde{\varphi}_{1} &=& 1/\theta_E +\varphi_{1}^{N-1}+ \varphi_{1}^{mN}, \widetilde{\varphi}_{2} = \varphi_{2}^{N-1}+ \varphi_{2}^{mN} + i, \widetilde{\varphi}_{3} \\ &=& \varphi_{3}^{N-1}+ \varphi_{3}^{mN} + m_{J}+j + 1. \end{array} $$

In addition, $\widehat {\sum \limits _{N,n,m_{E}}}$, $\widetilde {\sum \limits _{E}}$, $\widetilde {\sum \limits _{D}}$ are the shorthand notations of

$$\begin{array}{@{}rcl@{}} \left\{\begin{array}{lllllllll} \widetilde{\sum \limits_{D}} &= {\sum}_{k = 1}^{K} {\sum}_{q = 0}^{m_{D}-1} {\sum}_{\omega_{1},...,\omega_{m_{R}}}^{k} \binom{K}{k} \left( \frac{k!}{\omega_{1}!...\omega_{m_{R}}!} \right) (-1)^{k} \lambda^{k} \\ &\quad\frac{ 1} {q! {\prod}_{t = 0}^{m_{R}-1} \left( t!(\theta_R)^{t} \right)^{\omega_{t + 1}}} \left( \frac{1}{\theta_D} \right)^{q} . \\\widetilde{\sum \limits_{E}} & = \widehat{\sum\limits_{N-1,l,m_{E}}} \widehat{\sum\limits_{mN,r,m_{E}}} {\sum}_{m = 0}^{M-1} {\sum}_{i = 0}^{m_{E}-1} {\sum}_{j = 0}^{i} \binom{M-1}{m} \binom{i}{j} (-1)^{m} \frac{1}{i! (\theta_E )^{i}} {\Gamma}(m_{J}+j). \\\widehat{\sum\limits_{N,n,m_{E}}} &= {\sum}_{n = 0}^{N} {\sum}_{\vartheta_{1},...,\vartheta_{m_{E}}}^{n} {\sum}_{\mu_{1,1}}^{\vartheta_{1}} {\sum}_{\mu_{2,1},\mu_{2,2}}^{\vartheta_{2}} ... {\sum}_{\mu_{m_{E},1},...,\mu_{m_{E},m_{E}}}^{\vartheta_{m_{E}}} \binom{N}{n} (-1)^{n} \\&\quad\left( \frac{n!}{\vartheta_{1}!...\vartheta_{m_{E}}!} \right) \left( \frac{\vartheta_{1}!}{\mu_{1,1}!} \right) \left( \frac{\vartheta_{2}!}{\mu_{2,1}!\mu_{2,2}!} \right) ... \left( \frac{\vartheta_{m_{E}}!}{\mu_{m_{E},1}!...\mu_{m_{E},m_{E}}!} \right) \\&\quad\left( \frac{1}{(\theta_J)^{m_{J}} (m_{J}-1)!} \right)^{n} \frac{1}{{\prod}_{t = 0}^{m_{E}-1} \left( t!(\theta_E)^{t} \right)^{\vartheta_{t + 1}}} {\prod}_{\eta_{1}= 0}^{0} \left[ \binom{0}{\eta_{1}} {\Gamma}(m_{J}+\eta_{1})\right]^{\mu_{1,\eta_{1} + 1}} \\ &\quad{\prod}_{\eta_{2}= 0}^{1} \left[ \binom{1}{\eta_{2}} {\Gamma}(m_{J}+\eta_{2})\right]^{\mu_{2,\eta_{2} + 1}} ... \\ &\quad{\prod}_{\eta_{m_{E}}= 0}^{m_{E}-1} \left[ \binom{m_{E}-1}{\eta_{m_{E}}} {\Gamma}(m_{J}+\eta_{m_{E}})\right]^{\mu_{m_{E},\eta_{m_{E}} + 1}}. \end{array}\right. \end{array} $$

______________________________________________________________________________________________

To provide full insights into the impact of unreliable backhaul connections, the asymptotic expression for the secrecy outage probability can be obtained as Eq. 20.

$$\begin{array}{@{}rcl@{}} \mathcal{P}_{out}^{\infty} (\theta) &\overset{\theta_D \rightarrow \infty}{=}& 1 +J\widetilde{\sum\limits_{D^{\infty}}} \widetilde{\sum\limits_{E}} \sum\limits_{\alpha= 0}^{\tilde{\beta}} \binom{\tilde{\beta}}{\alpha} ({\Upsilon} - 1)^{\tilde{\beta} - \alpha}({\Upsilon})^{\alpha} {\theta_E}^{\widetilde{\varphi}_{3}} e^{-\tilde{\Phi} ({\Upsilon} - 1)} (J_{4} - J_{5} + J_{6}), \end{array} $$

(20)

______________________________________________________________________________________________where

$$\begin{array}{@{}rcl@{}} \left\{\begin{array}{llll} J_{4} &= Q_{1} {\Gamma}(\widetilde{\varphi}_{2}+ \alpha + 1) {\epsilon}^{\widetilde{\varphi}_{2}+ \alpha + 1 -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \alpha + 1, \widetilde{\varphi}_{2}+ \alpha + 2 -\widetilde{\varphi}_{3}, \epsilon(\tilde{\Phi} {\Upsilon} + \widetilde{\varphi}_{1}) \right), \\J_{5} &= Q_{2} {\Gamma}(\widetilde{\varphi}_{2}+ \alpha) {\epsilon}^{\widetilde{\varphi}_{2}+ \alpha -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \alpha , \widetilde{\varphi}_{2}+ \alpha + 1 -\widetilde{\varphi}_{3}, \epsilon(\tilde{\Phi} {\Upsilon} + \widetilde{\varphi}_{1}) \right), \\J_{6} &=Q_{3} {\Gamma}(\widetilde{\varphi}_{2}+ \alpha + 2) {\epsilon}^{\widetilde{\varphi}_{2}+ \alpha + 2 -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \alpha + 2, \widetilde{\varphi}_{2}+ \alpha + 3 -\widetilde{\varphi}_{3}, \epsilon(\tilde{\Phi}{\Upsilon} + \widetilde{\varphi}_{1}) \right). \end{array}\right. \end{array} $$

______________________________________________________________________________________________

and $\tilde {\beta } = {\sum }_{t = 0}^{m_{R}-1}t \omega _{t + 1}$, $\tilde {\Phi } = \frac {k}{\theta _R}$, $\widetilde {\sum \limits _{D^{\infty }}} ={\sum }_{k = 1}^{K} {\sum }_{\omega _{1},...,\omega _{m_{R}}}^{k}$$\binom {K}{k} \left (\frac {k!}{\omega _{1}!...\omega _{m_{R}}!} \right ) \frac { (-1)^{k-1} s^{k}} {{\prod }_{t = 0}^{m_{R}-1} \left (t!(\theta _R)^{t} \right )^{\omega _{t + 1}}}$

4.2 Probability of non-zero secrecy rate

The probability of non-zero secrecy rate is the probability that secrecy rate is more than zero, or another way $SNR_{T_{k^{*}}D}$ is higher than $SINR_{R_{m^{*}} E_{n^{*}}}$. The probability of none-zero secrecy rate can be obtained as [31]

$$\begin{array}{@{}rcl@{}} Pr(C_{S} > 0) &=& 1- \mathcal{P}_{out} (0) \\ &=&1- {\int}_{0}^{\infty} F_{{SNR_{DF}}}\left( {x}\right) f_{{SINR_{R_{m^{*}} E_{n^{*}}}}}\left( {x}\right) dx,\\ \end{array} $$

(21)

Using Eqs. 15 and 32 and with the help of [24, Eq. (2.3.6.9)]. The closed-form expression for the probability of non-zero achievable secrecy rate is given as, Eq. 22.

$$\begin{array}{@{}rcl@{}} Pr(\mathcal{C}_{S} > 0) = - J \widetilde{\sum \limits_{D}} \widetilde{\sum \limits_{E}} {\theta_E}^{\widetilde{\varphi}_{3}} (J_{7} - J_{8} +J_{9} ), \end{array} $$

(22)

where

$$\begin{array}{@{}rcl@{}} \left\{\begin{array}{llll} J_{7} &= Q_{1} {\Gamma}(\widetilde{\varphi}_{2}+ \beta + 1) {\epsilon}^{\widetilde{\varphi}_{2}+ \beta + 1 -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \beta + 1, \widetilde{\varphi}_{2}+ \beta + 2 -\widetilde{\varphi}_{3}, \epsilon({\Phi} + \widetilde{\varphi}_{1}) \right), \\J_{8} &= Q_{2} {\Gamma}(\widetilde{\varphi}_{2}+ \beta) {\epsilon}^{\widetilde{\varphi}_{2}+ \beta -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \beta , \widetilde{\varphi}_{2}+ \beta + 1 -\widetilde{\varphi}_{3}, \epsilon({\Phi} + \widetilde{\varphi}_{1}) \right), \\ J_{9} &=Q_{3} {\Gamma}(\widetilde{\varphi}_{2}+ \beta + 2) {\epsilon}^{\widetilde{\varphi}_{2}+ \beta + 2 -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \beta + 2, \widetilde{\varphi}_{2}+ \beta + 3 -\widetilde{\varphi}_{3}, \epsilon({\Phi} + \widetilde{\varphi}_{1}) \right). \end{array}\right. \end{array} $$

______________________________________________________________________________________________

To investigate the asymptotic behavior of the probability of non-zero achievable secrecy rate in high SNR regime, the asymptotic expression is given as Eq. 23.

$$\begin{array}{@{}rcl@{}} Pr(\mathcal{C}_{S}^{\infty} > 0) \overset{\theta_D \rightarrow \infty}{=} -J \widetilde{\sum \limits_{D^{\infty}}} \widetilde{\sum \limits_{E}} {\theta_E}^{\widetilde{\varphi}_{3}} (J_{10} - J_{11} +J_{12}),\\ \end{array} $$

(23)

where

$$\begin{array}{@{}rcl@{}} \left\{\begin{array}{llll} J_{10} &= Q_{1} {\Gamma}(\widetilde{\varphi}_{2}+ \tilde{\beta} + 1) {\epsilon}^{\widetilde{\varphi}_{2}+ \tilde{\beta} + 1 -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \tilde{\beta} + 1, \widetilde{\varphi}_{2}+ \tilde{\beta} + 2 -\widetilde{\varphi}_{3}, \epsilon(\tilde{\Phi} + \widetilde{\varphi}_{1}) \right), \\ J_{11} &= Q_{2} {\Gamma}(\widetilde{\varphi}_{2}+ \tilde{\beta}) {\epsilon}^{\widetilde{\varphi}_{2}+ \tilde{\beta} -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \tilde{\beta} , \widetilde{\varphi}_{2}+ \tilde{\beta} + 1 -\widetilde{\varphi}_{3}, \epsilon(\tilde{\Phi} + \widetilde{\varphi}_{1}) \right), \\J_{12} &= Q_{3} {\Gamma}(\widetilde{\varphi}_{2}+ \tilde{\beta} + 2) {\epsilon}^{\widetilde{\varphi}_{2}+ \tilde{\beta} + 2 -\widetilde{\varphi}_{3}} {\Psi} \left( \widetilde{\varphi}_{2}+ \tilde{\beta} + 2, \widetilde{\varphi}_{2}+ \tilde{\beta} + 3 -\widetilde{\varphi}_{3}, \epsilon(\tilde{\Phi} + \widetilde{\varphi}_{1}) \right). \end{array}\right. \end{array} $$

______________________________________________________________________________________________

4.3 Ergodic capacity

The ergodic capacity is defined as the average secrecy rate averaged over all the SNR distributions.

Ergodic capacity (nat/s/Hz) is expressed as [31]

$$\begin{array}{@{}rcl@{}} \mathcal{C}_{erg} &= \frac{1}{2\ln(2)}{\int}_{0}^{\infty} \frac{F_{{SINR_{E_{n^{*},m^{*}}}}}\left( {x}\right)}{1+x}[1-F_{{SNR_{DF}}}\left( {x}\right)] dx \end{array} $$

(24)

Substitute (13) and (15) into (24), and with the help of [24, Eq. (2.3.6.9)], [18, Eq. (1.1.1)], [18, Eq. (2.6.2)], [18, Appendix A7], ergodic capacity can be evaluated as Eq. 25.

$$\begin{array}{@{}rcl@{}} \mathcal{C}_{erg} &=& -\frac{1}{2\ln(2)} \left( \widetilde{\sum \limits_{D}} {\Gamma}(\beta+ 1) {\Psi} \left( \beta+ 1, \beta+ 1, {\Phi} \widetilde{\sum \limits_{D}} \sum\limits_{h = 0}^{M} \binom{M}{h} (-1)^{h} {\theta_J}^{\varphi_{3}^{hN}} \right) \right. \\&&\qquad\quad\qquad\left. -\widehat{\sum\limits_{hN,v,N_{E}}} \frac{({\Phi}+\varphi_{1}^{hN})^{-\varphi_{2}^{hN}-\beta-1}}{{\Gamma}(\varphi_{3}^{hN})} H_{1,(1:1),0,(1:1)}^{1,1,1,1,1} \left[ {\frac{1}{\Phi+\varphi_{1}^{hN}} \atop \frac{1}{\epsilon ({\Phi}+\varphi_{1}^{hN})}} \ \left\vert \ { {(1+\varphi_{2}^{hN}+\beta,1) \atop {(0,1);(1-\varphi_{3}^{hN},1)}} \atop {{-} \atop {(0,1);(0,1)}}} \right.\right] \right). \end{array} $$

(25)

______________________________________________________________________________________________

To gain the full insights of the system, the asymptotic ergodic capacity is given as Eq. 26.

$$\begin{array}{@{}rcl@{}} \mathcal{C}_{erg}^{\infty} &\overset{\theta_D \rightarrow \infty}{=}& -\frac{1}{2\ln(2)} \left( \widetilde{\sum \limits_{D^{\infty}}} {\Gamma}(\tilde{\beta}+ 1) {\Psi} \left( \tilde{\beta}+ 1, \tilde{\beta}+ 1, \tilde{\Phi} \widetilde{\sum \limits_{D^{\infty}}} {\sum}_{h = 0}^{M} \binom{M}{h} (-1)^{h} {\theta_J}^{\varphi_{3}^{hN}} \right) \right. \\&&\left. -\widehat{\sum\limits_{hN,v,m_{E}}} \frac{(\tilde{\Phi}+\varphi_{1}^{hN})^{-\varphi_{2}^{hN}-\tilde{\beta}-1}}{{\Gamma}(\varphi_{3}^{hN})} H_{1,(1:1),0,(1:1)}^{1,1,1,1,1} \left[ {\frac{1}{\tilde{\Phi}+\varphi_{1}^{hN}} \atop \frac{1}{\epsilon (\tilde{\Phi}+\varphi_{1}^{hN})}} \ \left\vert \ { {(1+\varphi_{2}^{hN}+\tilde{\beta},1) \atop {(0,1);(1-\varphi_{3}^{hN},1)}} \atop {{-} \atop {(0,1);(0,1)}}} \right] \right.\right). \end{array} $$

(26)

______________________________________________________________________________________________

where $H_{p q}^{m n} \left [ . \right ]$ denotes the Fox H-function [18, Eq. (1.1.1)].

5 Numerical results

In this section, numerical results along with simulations are shown for the analysis carried out on the proposed system. The threshold of secrecy outage probability is fixed at 𝜃 = 1 bits/s/Hz. The binary phase-shift keying (BPSK) modulation is adopted in the simulations with transmission block size S = 64 symbols. In figures, “Sim” represents the simulation results, “Ana” represents the analytical results and “Asy” represents the asymptotic analysis results. We investigate the network performance with various parameters to examine the effects of the degrees of cooperative transmission and backhaul reliability.

5.1 Secrecy outage probability

Fig. 2 investigates the secrecy outage probability for various M and N. The network parameters are set as K = 3,s = 0.998,{m_R,m_E,m_J,m_D} = {2,2,2,3}, and {𝜃_R,𝜃_E,𝜃_J} = {10,10,10} dB. We can observe that the number of relays and eavesdroppers strongly affects the secrecy outage probability. Specifically, when N = 1, the secrecy outage probability decreases when the number of relay increases, thus achieving a better system performance. By contrast, when M = 1, the secrecy outage probability increases when the number of eavesdropper increases. We can also observe that our results approach asymptotic results in the high SNR regime.

Fig. 2

Secrecy outage probability for various M,N

Figure 3 plots the secrecy outage probability with various K and s. We set the parameters as M = 2,N = 1,{m_R,m_E,m_J,m_D} = {2,2,3,2}, and {𝜃_R,𝜃_E,𝜃_J} = {10,10,10} dB. We can observe from the figures that when K = 1, the secrecy outage probability decreases when the backhaul reliability gets higher. If the system has more reliable backhaul, it performs better.

Fig. 3

Secrecy outage probability for various K,s

In addition, when s = 0.95 and the number of small-cell transmitters increases from K = 1 to K = 3, the secrecy outage probability decreases due to the increased received signal power at D.

Figure 4 shows the effects of dense networks on secrecy outage probability versus number of relays M with K = 10,s = 0.998,{m_R,m_E,m_J,m_D} = {2,2,2,3},and {𝜃_R,𝜃_E,𝜃_J,𝜃_D} = {10,10,10,10} dB. We can observe that for all N = {1,5,10}, secrecy outage probability decreases when there are more relays due to the cooperative communication. In addition, with the increase of N from N = 1 to N = 10, the secrecy outage probability increases. This is because the achievable capacity in the wiretap channels gets higher.

Fig. 4

Impact of the dense networks on the secrecy outage probability

5.2 Ergodic capacity

Figures 5 and 6 depict the effect of the number of small-cell transmitters, the number of relays and eavesdroppers and backhaul reliability on the ergodic capacity. Parameters are the same in the corresponding figures of the ergodic capacity and secrecy outage probability. Effects of the degree of cooperative transmission and reliability of backhaul are complementary on ergodic secrecy rate and secrecy outage probability. When secrecy outage probability decreases, the ergodic secrecy rate would increase. We can observe in the figures that when N = 1, the ergodic capacity increases when the number of relay grows. Also, when M = 1, ergodic capacity decreases when the number of eavesdroppers increases. In addition, when K = 1, the ergodic capacity increases when the backhaul is more reliable, and when s = 0.95, the ergodic capacity increases when the number of small-cell transmitters increases.

Fig. 5

Ergodic capacity for various M,N

Fig. 6

Ergodic capacity for various K,s

5.3 Non-zero achievable secrecy rate

Figures 7 and 8 show the same performances as of secrecy outage probability and ergodic secrecy rate with the same parameters, correspondingly. The observations are similar in all the cases.

Fig. 7

Non-zero achievable secrecy rate probability for various M,N

Fig. 8

Non-zero achievable secrecy rate probability for various K,s

In the figure, simulation results match well with the numerical results, thus, validating the analysis presented in the paper.

6 Conclusions

This paper investigates the secrecy performance of cooperative heterogeneous networks with unreliable backhaul links. A two phase transmitter/relay selection scheme was proposed to maximize the SNR at the relays and minimize the SINR at the eavesdroppers. Closed-form expressions are derived and asymptotic expressions are also provided. Results show that when the number of small-cell transmitters and relays increases, the system can achieve a better performance. However, the increase of eavesdroppers can significantly degrade the system performance. Moreover, backhaul reliability is a key parameter for the improvement of secrecy performance.

Appendices

Appendix A

Since all the channels follow Nakagami-m fading, and the CDF and PDF can be found in Eqs. 1 and 2. The CDF and PDF of $SNR_{T_{k} R_{m}}$ can be written as

$$\begin{array}{@{}rcl@{}} f_{{SNR_{T_{k} R_{m}}}}\left( {x}\right) &=& (1-s) \delta(x) + \frac{s}{(\theta_R)^{m_{R}} (m_{R}-1)!}\\ &&x^{m_{R}-1} e^{-x/\theta_R}, \\ F_{{SNR_{T_{k} R_{m}}}}\left( {x}\right) &=& 1 - s e^{-x/\theta_R} \sum\limits_{l = 0}^{m_{R}-1} \frac{1}{l!} \left( \frac{x}{\theta_R} \right)^{l}, \end{array} $$

(27)

where δ(.) denotes the Dirac delta function. Since the best transmitter T$_{k^{*}}$ is selected, the CDF of $SNR_{T_{k^{*}} R_{m}}$ can be given as

$$\begin{array}{@{}rcl@{}} F_{{SNR_{T_{k^{*}} R_{m}}}}\left( {x}\right)= \left[ F_{{SNR_{T_{k} R_{m}}}}\left( {x}\right) \right]^{K}. \end{array} $$

(28)

After some simple manipulations, we can derive (10).

Appendix B

The CDF of $SINR_{R_{m} E_{n}}$ can be obtained as

$$\begin{array}{@{}rcl@{}} F_{{SINR_{R_{m} E_{n}}}}\left( {x}\right) &=& 1 - \frac{1}{(\theta_J)^{m_{J}} (m_{J}-1)!} \\ &&\times\sum\limits_{i = 0}^{m_{E}-1} \sum\limits_{j = 0}^{i} \binom{i}{j} \frac{ {\Gamma}(m_{J}+j)}{i! (\theta_E )^{i}}, \\ &&\times x^{i} e^{-x/\theta_E} \left( \frac{1}{\theta_J}+ \frac{x}{\theta_E} \right)^{-(m_{J}+j)}. \end{array} $$

(29)

Since the frequency selective fading channels between the particular relay to the eavesdroppers are independent and identically distributed, the CDF of the $SINR_{E_{n^{*}}}$ is given as

$$\begin{array}{@{}rcl@{}} F_{{SINR_{R_{m} E_{n^{*}}}}}\left( {x}\right)= [ F_{{SINR_{R_{m} E_{n}}}}\left( {x}\right) ]^{N}. \end{array} $$

(30)

Appendix C

We first derive the PDF of $SINR_{R_{m} E_{n^{*}}}$,

$$\begin{array}{@{}rcl@{}} f_{{SINR_{R_{m} E_{n^{*}}}}}\left( {x}\right)= \frac{\partial F_{{SINR_{R_{m} E_{n^{*}}}}}\left( {x}\right)} {\partial x}. \\ \end{array} $$

(31)

Since (12), we derive the PDF of ${SINR_{R_{m^{*}} E_{n^{*}}}}$,

$$\begin{array}{@{}rcl@{}} f_{{SINR_{R_{m^{*}} E_{n^{*}}}}}\left( {x}\right) &=& M f_{{SINR_{R_{m} E_{n^{*}}}}}\\ &&\times \left( {x}\right) \left[1-F_{{SINR_{R_{m} E_{n^{*}}}}}\left( {x}\right)\right]^{M-1}. \end{array} $$

(32)