1 Introduction

As one of the promising key techniques of the fifth-generation (5G) wireless networks, non-orthogonal multiple access (NOMA) has been applied in various areas [1, 2]. NOMA utilizes the power domain to achieve multiple access strategies, which is unlike the conventional orthogonal multiple access (OMA) structures. Since the NOMA technique exploits the new dimension of the power domain, it has the potential to be integrated with existing MA paradigms. On the other hand, the technique of cooperative transmission can form a virtual multiple-input multiple-output (MIMO) scheme to process data cooperatively, which can enhance the communication reliability for the users who are in poor channel conditions. It is reasonable to integrate NOMA into relaying networks. The authors in [3] proposed a new cooperative NOMA scheme and analyzed the outage probability and diversity gain of the system.

In the above references, all the relays are in half-duplex (HD) mode which requires additional bandwidth or time slot and reduces spectral efficiency of the system because the relay isolates the transmit and receive signals in two orthogonal time slots or frequency channels. To avoid this issue, full-duplex (FD) is proposed as a promising technique for the next generation wireless systems, because it can double the spectrum efficiency by realizing transmission and reception on the same carrier frequency, simultaneously [4]. Two main types of FD relay techniques, namely FD amplify-and-forward (AF) relaying and FD decode-and-forward (DF) relaying, have been discussed in [5, 6]. The authors in [7] propose an energy-efficient oriented algorithm for FD cooperative NOMA systems and demonstrate both significant energy-efficient and throughput enhancements over the HD cooperative NOMA and the prefixed FD cooperative NOMA regime. However, the significant amount of self-interference (SI) between transmitting and receiving radio frequency chains means that the performance of FD relay is degraded by the residual SI [8]. In [9], the authors provide a characterization of the distribution of the self-interference before and after active cancellation mechanisms. In [10], the author analyzed the outage probability, ergodic rate and energy efficiency of FD user relaying with imperfect SI cancellation.

Alternatively, we may instead consider a distributed version of a FD system, called virtual FD (VFD), where a pair of HD relays receive and transmit signals successively to imitate the operation of FD relay. This idea inherits the benefits of the FD system, while each relay requires only a single antenna, and the physical separation between the two HD relays eliminates the SI problem perfectly [8]. The combination of NOMA and VFD will further improve the spectrum efficiency. In [11], the authors found that VFD-NOMA system achieves a lower outage probability and a higher ergodic rate than the existing successive relaying OMA system and ideal FD-NOMA system. However, secrecy transmission of VFD cooperative NOMA system still remains unexplored.

In future networks, the ubiquitous connectivity makes our privacy and secrecy expose to radio space. Security becomes a major concern when designing communication systems. Due to the broadcast nature of radio propagation, wireless communication is vulnerable to be eavesdropped [12]. In relay-assisted NOMA networks, when the relay is untrusted, it may exploit these weaknesses to its benefit and behave as a passive eavesdropper who tries to listen in on an ongoing transmission [13]. Cooperative jamming has been become a promising technique to protect the transmission in untrusted relay OMA networks [14]. Authors in [15] describe the evolution of cooperative jamming transmission strategies from point-to-point channels to multiple antenna systems, followed by generalizations to larger multiuser networks. Choi and Lee [16] proposed a new cooperative jamming technique for an AF relay network with multiple relays, and the simulation results showed that the network with the proposed technique provides lower secrecy outage probability than a conventional scheme. The internal jamming strategy is employed in [17], where the legitimate user is enlisted as a jammer to distract the untrusted relay and adopts SI cancellation to eliminate the jamming. Yuan et al. [18] analyzed cooperative NOMA scheme, where the source actively sends jamming signals, while the relay is forwarding, thereby enhancing the security of intended communication links. In [19], a two-stage FD jamming scheme is proposed to ensure the secure transmission of cooperative NOMA users. Although beamforming of relay was designed in this paper to cancel the SI, when the estimation error of the channel state information (CSI) cannot be ignored, or the number of antennas at the relay is not enough to perform the beamforming, the residual self-interference channel still has to be considered. In this case, VFD-NOMA system is more practical because physical separation of relays eliminates the SI problem perfectly. But until now, there are few studies on the security performance of VFD-NOMA system.

According to the above references, this paper studies the cooperative VFD-NOMA transmission and investigated its secrecy performance with the presentence of an untrusted relay. The VFD system consists an untrusted AF relay and a trusted DF relay. Two users are served by the trusted relay using cooperative NOMA, and only one user is served by the untrusted relay because the untrusted relay is an eavesdropper. We assume that the untrusted relay can obtain messages of all users. A new cooperative jamming strategy is adopted to secure the transmission of untrusted relay link. The jamming signal transmitted by only one user is utilized to confuse the untrusted relay. The exact effective secrecy throughput (EST) expressions for NOMA users and approximate EST expression for non-NOMA user are derived, which shows that the jamming is indispensable for achieving nonzero EST. Our principal motivations and contributions are summarized as follows.

  • As mentioned above, the combination of NOMA and VFD can improve the frequency efficiency and eliminate the residual SI of FD relaying simultaneously, but when we use VFD in a NOMA system, a pair of relays must be employed to form a VFD system and one of the relays may be an eavesdropper. There is still a paucity of research contributions on investigating the security issues of NOMA in a VFD system, which is the motivation of this paper. In order to improve the spectral efficiency, eliminate the residual SI of FD relaying and prevent untrusted relay eavesdropping simultaneously, we integrate NOMA, VFD and untrusted relay together and introduce a VFD relaying NOMA scheme in this study. The application scenario of this study is that multiple relays with one antenna can be employed to forward information, but some relays have poor credibility and could be eavesdroppers. We assume that in VFD relay system, one relay is trusted and the other is untrusted. To guarantee the secure transmission, a new cooperative jamming strategy is adopted to distract the untrusted relay. It is differing from [20] where jamming is transmitted by both the two users and our scheme saves the jamming power.

  • The novelty of this study is that a novel cooperative jamming strategy is adopted to secure the untrusted relay VFD-NOMA networks and the EST expressions are derived to obtain further insights into parameters on system performance, which shows that there exist the optimal values of source power, relay power and jamming power. These findings are of great theoretical value for practical secure communication.

The rest of this paper is organized as follows. In Sect. 2, we introduce the system model. The closed-form expressions of EST are derived in Sect. 3. Numerical and simulations results are presented in Sect. 4, and conclusions are drawn in Sect. 5.

2 System model

As shown in Fig. 1, we consider a cooperative VFD-NOMA system, which consists of a source (S), a trusted DF relay (R), an untrusted AF relay (E) and three users (m, n, p). m is close to n, but they are far away from p. We assume that there do not exist direct links between S and users. R and E are willing to help the transmission. R can forward the information for m and n using NOMA, but it cannot forward information for p because they are too far apart. E is closer to p than R and can forward the information for p. But E is untrusted and can be regarded as potential Eves, so there is risk of leaking information of all users to E. All nodes are configured with one antenna and operate in HD mode. The channels between any transmitter and receiver are assumed to experience independent block Rayleigh fading, and the additive white Gaussian noise of all links has a zero mean and equal variance \({N_0}\). The channel coefficients between nodes are known as follows: \({h_{SR}}\) denotes channel coefficient between S and R, \({h_{SE}}\) denotes channel coefficient between S and E, \({h_{RE}}\) denotes channel coefficient between R and E, \({h_{Sm}}\) denotes channel coefficient between R and m, \({h_{Sn}}\) denotes channel coefficient between R and n, \({h_{Ep}}\) and denotes channel coefficient between E and p. The channel power gains of those hops \({\left| {{h_{SR}}} \right| ^2}\), \({\left| {{h_{SE}}} \right| ^2}\), \({\left| {{h_{RE}}} \right| ^2}\), \({\left| {{h_{Rm}}} \right| ^2}\), \({\left| {{h_{Rn}}} \right| ^2}\)and \({\left| {{h_{Ep}}} \right| ^2}\) are assumed to be exponentially distribution, and the corresponding instantaneous signal-to-noise ratios (SNRs) \({{{P_S}{{\left| {{h_{SR}}} \right| }^2}}/ {{N_0}}}\), \({{{P_S}{{\left| {{h_{SE}}} \right| }^2}}/ {{N_0}}}\), \({{{P_R}{{\left| {{h_{RE}}} \right| }^2}} / {{N_0}}}\), \({{{P_R}{{\left| {{h_{Rm}}} \right| }^2}} / {{N_0}}}\), \({{{P_R}{{\left| {{h_{Rn}}} \right| }^2}} / {{N_0}}}\) and \({{{P_E}{{\left| {{h_{Ep}}} \right| }^2}} / {{N_0}}}\) are also exponential random variables with parameters \({\lambda _1}\), \({\lambda _2}\), \({\lambda _3}\), \({\lambda _4}\), \({\lambda _5}\) and \({\lambda _6},\), respectively. \(P_S\), \(P_R\) and \(P_E\) are the transmission power at S, R and E, respectively. We also assume that without loss of generality, channel gains are sorted as \({\left| {{h_{Rm}}} \right| ^2} < {\left| {{h_{Rn}}} \right| ^2}\). The protocol of the VFD-NOMA scheme is divided into three equal time slots which is stated as follows.

During the first time slot, S broadcasts a superimposed mixture

$$\begin{aligned} {x_s} = \sqrt{{a_m}{P_S}} {x_m} + \sqrt{{a_n}{P_S}} {x_n}, \end{aligned}$$
(1)

where \({x_m}\) and \({x_n}\) are the signals for m and n, respectively, and \({\text{E}}\left( {x_m^2} \right) = {\text{E}}\left( {x_n^2} \right) = 1.\) (\({\text{E}}\left( \cdot \right)\) denotes the expectation of a random variable.) Based on well-known Wyner secrecy code theorem [21], \({x_m}\) and \({x_n}\) are coded as \(\left( {{\tau _{m,t}},{\tau _{m,s}}} \right)\) and \(\left( {{\tau _{n,t}},{\tau _{n,s}}} \right)\), respectively, where \({\tau _{m,t}}\) and \({\tau _{n,t}}\) are the codeword rates, and \({\tau _{m,s}}\) and \({\tau _{n,s}}\) are secrecy rates. \(a_m\) and \(a_n\) are the power split factors of signals \({x_m}\) and \({x_n}\), respectively. Note that \({a_m} + {a_n} = 1\) and \({a_m} > {a_n}\) according to the principle of NOMA. In order to reduce the risk of leaking the information of m and n to E, at the same time, p is chosen to transmit a jamming signal to confuse E which is shown in the first part of Fig. 1. The observations at R and E are given by

$$\begin{aligned} {y_R}&= {h_{SR}}\left( {\sqrt{{a_m}{P_S}} {x_m} + \sqrt{{a_n}{P_S}} {x_n}} \right) + {n_R}, \end{aligned}$$
(2)
$$\begin{aligned} {y_{E,1}}&= {h_{SE}}\left( {\sqrt{{a_m}{P_S}} {x_m} + \sqrt{{a_m}{P_S}} {x_n}} \right) + \sqrt{{P_I}} {x_0} + {n_E}, \end{aligned}$$
(3)

where \(n_R\) and \(n_E\) are the additive white Gaussian noises, \(x_0\) is the jamming signal, and \(P_I\) is the power of the received jamming at E. (The setting of \(P_I\) in our model is the same as [22]. We can also set an jamming transmission power at p and then consider the impacts of jamming power and jamming channel fading on the performance, but it will increase analysis complexity and we will discuss this setting in the future. In the numerical results part of this paper, we will investigate the impact of different values of \(P_I\) on the system performance.) As p and R are too far apart, there is no jamming signal in (2).

Fig. 1
figure 1

Illustration of a VFD-NOMA network in the presence of the security threat from untrusted relay

Full size image

R is assumed to have the ability to apply successive interference cancellation (SIC). According to the principle of SIC, \(x_m\) is demodulated with the interference of \(x_n\), because \(x_m\) is allocated more power, and when \(x_m\) is demodulated successfully, \(x_n\) can be directly demodulated after subtracting the interference of \(x_m\) [23]. The received signal-to-interference-and-noise ratio (SINR) for decoding \(x_m\) at R can be written as

$$\begin{aligned} {\gamma _{R \rightarrow m}} = \frac{{{a_m}{{\left| {{h_{SR}}} \right| }^2}{P_S}}}{{{a_n}{{\left| {{h_{SR}}} \right| }^2}{P_S} + {N_0}}}. \end{aligned}$$
(4)

After R successfully demodulates the signal of m, the interference from m can be eliminated by SIC. The SINR for decoding \(x_n\) at R can be written as

$$\begin{aligned} {\gamma _{R \rightarrow n}} = \frac{{{a_n}{P_S}{{\left| {{h_{SR}}} \right| }^2}}}{{{N_0}}}. \end{aligned}$$
(5)

We consider the worst case that E has the ability to apply SIC and can always eliminate the interference from \(x_m\) while demodulating \(x_n\). The SINRs of E wiretapping the message of m and n in first time slot can be written as

$$\begin{aligned} {\gamma _{E \rightarrow m,1}}&= \frac{{{a_m}{P_S}{{\left| {{h_{SE}}} \right| }^2}}}{{{a_n}{P_S}{{\left| {{h_{SE}}} \right| }^2} + {P_I} + {N_0}}}, \end{aligned}$$
(6)
$$\begin{aligned} {\gamma _{E \rightarrow n,1}}&= \frac{{{a_n}{P_S}{{\left| {{h_{SE}}} \right| }^2}}}{{{P_I} + {N_0}}}. \end{aligned}$$
(7)

During the second time slot, R re-encodes \(x_m\) and \(x_n\) and forwards it to m and n. The observations at m and n are given by

$$\begin{aligned} {y_m}&= {h_{Rm}}\left( {\sqrt{{a_m}{P_R}} {x_m} + \sqrt{{a_n}{P_R}} {x_n}} \right) + {n_0}, \end{aligned}$$
(8)
$$\begin{aligned} {y_n}&= {h_{Rn}}\left( {\sqrt{{a_m}{P_R}} {x_m} + \sqrt{{a_n}{P_R}} {x_n}} \right) + {n_0}. \end{aligned}$$
(9)

When m and n demodulate the message of m, the SINRs can be expressed as

$$\begin{aligned} {\gamma _{m \rightarrow m}}&= \frac{{{a_m}{{\left| {{h_{Rm}}} \right| }^2}{P_R}}}{{{a_n}{{\left| {{h_{Rm}}} \right| }^2}{P_R} + {N_0}}}, \end{aligned}$$
(10)
$$\begin{aligned} {\gamma _{n \rightarrow m}}&= \frac{{{a_m}{{\left| {{h_{Rn}}} \right| }^2}{P_R}}}{{{a_n}{{\left| {{h_{Rn}}} \right| }^2}{P_R} + {N_0}}}. \end{aligned}$$
(11)

After n successfully demodulates the signal of m, the received SINR at n to decode its own signal is

$$\begin{aligned} {\gamma _{n \rightarrow n}} = \frac{{{a_n}{P_R}{{\left| {{h_{Rn}}} \right| }^2}}}{{{N_0}}}. \end{aligned}$$
(12)

When R forwards the messages to m and n in this time slot, S transmits a new signal \({x_p}\) of p to E with transmission power of \(P_S\) at the same time. Based on Wyner secrecy code theorem, \(x_p\) is coded as \(\left( {{\tau _{p,t}},{\tau _{p,s}}} \right)\), where \({\tau _{p,t}}\) is the codeword rate and \({\tau _{p,s}}\) is secrecy rate. E can also receive the signal from R at this phase which is shown in the second part of Fig. 1. Similar to the first time slot, p also transmits the jamming signal \(x_0\) to E at this time slot. The observation at E is given by

$$\begin{aligned} {y_{E,2}} = {h_{SE}}\sqrt{{P_S}} {x_p} + {h_{RE}}\left( {\sqrt{{a_m}{P_R}} {x_m} + \sqrt{{a_m}{P_R}} {x_n}} \right) + \sqrt{{P_I}} {x_0} + {n_0}. \end{aligned}$$
(13)

With this SINR, E can wiretap the message of m, n and p. \(x_p\) is decoded first because it has the most power, then \(x_m\) is decoded, and \(x_n\) is decoded at the end. The SINRs of E wiretapping the messages in this time slot can be written as

$$\begin{aligned} {\gamma _{E \rightarrow p}}&= \frac{{{P_S}{{\left| {{h_{SE}}} \right| }^2}}}{{{P_R}{{\left| {{h_{RE}}} \right| }^2} + {P_I} + {N_0}}}, \end{aligned}$$
(14)
$$\begin{aligned} {\gamma _{E \rightarrow m,2}}&= \frac{{{a_m}{P_R}{{\left| {{h_{RE}}} \right| }^2}}}{{{a_n}{P_R}{{\left| {{h_{RE}}} \right| }^2} + {P_I} + {N_0}}}, \end{aligned}$$
(15)
$$\begin{aligned} {\gamma _{E \rightarrow n,2}}&= \frac{{{a_n}{P_R}{{\left| {{h_{RE}}} \right| }^2}}}{{{P_I} + {N_0}}}. \end{aligned}$$
(16)

We assume that \(x_0\) is known by m and n and the corresponding term related to the jamming signal can be eliminated in (8) and (9). (This is a common assumption in the jamming literature [24,25,26,27] where the pre-defined jamming signal can be generated by using some pseudorandom codes that are known to m and n but not available to the curious untrusted relay. Transmission of the jamming signal from p to m and n can be accomplished by implementing the destination-based cooperative jamming technique.)

During the third time slot, E relays \({y_{E,2}}\) to p using AF protocol with amplifying coefficient \(G = \sqrt{{{{P_E}} / {\left( {{P_S}{{\left| {{h_{SE}}} \right| }^2} + {P_R}{{\left| {{h_{RE}}} \right| }^2} + {P_I} + {N_0}} \right) }}}\). The received signal at p can be written as

$$\begin{aligned} {y_p} = G{h_{Ep}}{y_{E,2}} + {n_0}. \end{aligned}$$
(17)

Following the principle of SIC, the jamming signal is demodulated and deducted from superposition signal by p completely, before it demodulates its intended messages. The SINR for p to decode its own message is given by

$$\begin{aligned} {\gamma _{p \rightarrow p}} = \frac{{{P_S}{P_E}{{\left| {{h_{Ep}}} \right| }^2}{{\left| {{h_{SE}}} \right| }^2}}}{{{P_R}{P_E}{{\left| {{h_{Ep}}} \right| }^2}{{\left| {{h_{RE}}} \right| }^2} + {P_E}{N_0}{{\left| {{h_{Ep}}} \right| }^2} + {P_S}{{\left| {{h_{SE}}} \right| }^2}{N_0} + {P_R}{{\left| {{h_{RE}}} \right| }^2}{N_0} + {P_I}{N_0} + N_0^2}}. \end{aligned}$$
(18)

Remarks:

(1) From the above description of the proposed protocol, we can see that in order to complete a secure communication, R needs to decode twice and recode once, m and p need to decode once and n needs to decode twice. The computational complexity of the proposed protocol is not very high.

(2) In our protocol, the source is responsible for the algorithm design. As there are only two relays in our system, common training processes of individual channels from source to relay and then to destination can be used to acquire the required CSI [28,29,30]. In addition, we use jamming not encryption in our protocol and no routing is required, so there is no overhead other than training processes.

3 Security performance analysis

In this section, we adopt EST as the metric of secrecy performance which considers the reliability performance and secrecy performance holistically. The exact EST expressions of m and n and approximate EST expression of p are derived in the following parts. The EST of m is expressed as

$$\begin{aligned} {\eta _m} = {\tau _{m,s}}\Pr \left( {{\gamma _{R \rightarrow m}}> {2^{3{\tau _{m,t}}}} - 1,{\gamma _{m \rightarrow m}} > {2^{3{\tau _{m,t}}}} - 1,{\gamma _{E \rightarrow m}} < {2^{3\left( {{\tau _{m,t}} - {\tau _{m,s}}} \right) }} - 1} \right) , \end{aligned}$$
(19)

where \(\Pr \left( \cdot \right)\) represents the probability that the expressions in brackets hold and \({\gamma _{E \rightarrow m}}\) is expressed as

$$\begin{aligned} {\gamma _{E \rightarrow m}} = \max \left( {{\gamma _{E \rightarrow m,1}},{\gamma _{E \rightarrow m,2}}} \right) . \end{aligned}$$
(20)

(20) indicates that the SINR of E wiretapping the message of m is the maximum eavesdropping SINR of the first time slot and the second time slot. Substituting (4), (6), (10) and (15) into (19), we can obtain

$$\begin{aligned}&\Pr \left( {{\gamma _{R \rightarrow m}}> {2^{3{\tau _{m,t}}}} - 1,{\gamma _{m \rightarrow p}}> {2^{3{\tau _{m,t}}}} - 1,{\gamma _{E \rightarrow m}}< {2^{3\left( {{\tau _{m,t}} - {\tau _{m,s}}} \right) }} - 1} \right) \\&\quad = \Pr \left( {\frac{{{a_m}{{\left| {{h_{SR}}} \right| }^2}{P_S}}}{{{a_n}{{\left| {{h_{SR}}} \right| }^2}{P_S} + {N_0}}}> {\theta _1},\frac{{{a_m}{{\left| {{h_{Rm}}} \right| }^2}{P_R}}}{{{a_n}{{\left| {{h_{Rm}}} \right| }^2}{P_R} + {N_0}}} > {\theta _1},} \right. \\&\quad \left. \max \left( {\frac{{{a_m}{P_S}{{\left| {{h_{SE}}} \right| }^2}}}{{{a_n}{P_S}{{\left| {{h_{SE}}} \right| }^2} + {P_I} + {N_0}}},\frac{{{a_m}{P_R}{{\left| {{h_{RE}}} \right| }^2}}}{{{a_n}{P_R}{{\left| {{h_{RE}}} \right| }^2} + {P_I} + {N_0}}}} \right) < {\theta _2} \right) , \end{aligned}$$
(21)

where \({\theta _1} = {2^{3{\tau _{m,t}}}} - 1\) and \({\theta _2} = {2^{3\left( {{\tau _{m,t}} - {\tau _{m,s}}} \right) }} - 1\). Following the channel assumption previously, we can get

$$\begin{aligned} & \Pr (\gamma _{{R \to m}} > \theta _{1} ,\gamma _{{m \to p}} > \theta _{1} ,\gamma _{{E \to m}} < \theta _{2} ) \\ & \quad = \exp \frac{{ - \lambda _{1} \theta _{1} }}{{(a_{m} - \theta _{1} a_{n} )}}\exp \frac{{ - \lambda _{4} \theta _{1} }}{{(a_{m} - \theta _{1} a_{n} )}} \\ & \quad \times \left( {1 - \exp \left( {\frac{{ - \lambda _{2} \theta _{2} (\rho _{I} + 1)}}{{(a_{m} - \theta _{2} a_{n} )}}} \right)} \right)\left( {1 - \exp \left( {\frac{{ - \lambda _{3} \theta _{2} (\rho _{I} + 1)}}{{P_{S} (a_{m} - \theta _{2} a_{n} )}}} \right)} \right), \\ \end{aligned}$$
(22)

where \({\rho _I} = {{{P_I}} / {{N_0}}}\). We have to notice that \({a_m} - {\theta _1}{a_n} > 0\) must be satisfied, if not, \({\eta _m}\) will be zero. Submitting (22) into (19), we can get the closed-form EST expression of m which can be written as

$$\begin{aligned} {\eta _m}&={\tau _{m,s}} \exp \frac{{ - {\lambda _1}{\theta _1}}}{{\left( {{a_m} - {\theta _1}{a_n}} \right) }}\exp \frac{{ - {\lambda _4}{\theta _1}}}{{\left( {{a_m} - {\theta _1}{a_n}} \right) }}\\&\times \left( {1 - \exp \left( {\frac{{ - {\lambda _2}{\theta _2}\left( {{\rho _I} + 1} \right) }}{{\left( {{a_m} - {\theta _2}{a_n}} \right) }}} \right) } \right) \left( {1 - \exp \left( {\frac{{ - {\lambda _3}{\theta _2}\left( {{\rho _I} + 1} \right) }}{{{P_S}\left( {{a_m} - {\theta _2}{a_n}} \right) }}} \right) } \right) . \\ \end{aligned}$$
(23)

Similarly, the EST of n is expressed as

$$\begin{aligned} {\eta _n}&= {\tau _{n,s}}\Pr \left( {{\gamma _{R \rightarrow m}}> {\theta _1},{\gamma _{R \rightarrow n}}> {2^{3{\tau _{n,t}}}} - 1,{\gamma _{n \rightarrow m}}> {\theta _1},} \right. \\&\quad \left. {{\gamma _{n \rightarrow n}} > {2^{3{\tau _{n,t}}}} - 1,{\gamma _{E \rightarrow n}} < {2^{3\left( {{\tau _{n,t}} - {\tau _{n,s}}} \right) }} - 1} \right) . \\ \end{aligned}$$
(24)

where \({\gamma _{E \rightarrow n}} = \max \left( {{\gamma _{E \rightarrow n,1}},{\gamma _{E \rightarrow n,2}}} \right)\). Similar to the analysis above, (24) can be expressed as

$$\begin{aligned} {\eta _n}&= {\tau _{n,s}}\exp \left( { - {\lambda _1}g} \right) \exp \left( { - {\lambda _4}g} \right) \left( {1 - \exp \left( {\frac{{ - {\lambda _2}{\theta _4}\left( {{\rho _I} + 1} \right) }}{{{a_n}}}} \right) } \right) \\&\quad \times \left( {1 - \exp \left( {\frac{{ - {\lambda _3}{\theta _4}\left( {{\rho _I} + 1} \right) }}{{{a_n}}}} \right) } \right) , \end{aligned}$$
(25)

where \(g = \max \left( {\frac{{{\theta _1}}}{{\left( {{a_m} - {\theta _1}{a_n}} \right) }},\frac{{{\theta _3}}}{{{a_n}}}} \right)\), \({\theta _3} = {2^{3{\tau _{n,t}}}} - 1\) and \({\theta _4} = {2^{3\left( {{\tau _{n,t}} - {\tau _{n,s}}} \right) }} - 1\).

3.1 Approximate expression of p

The EST of p is expressed as

$$\begin{aligned} {\eta _p} = {\tau _{p,s}}\Pr \left( {{\gamma _{p \rightarrow p}} > {2^{3{\tau _{p,t}}}} - 1,{\gamma _{E \rightarrow p}} < {2^{3\left( {{\tau _{p,t}} - {\tau _{p,s}}} \right) }} - 1} \right) . \end{aligned}$$
(26)

Before deriving the approximate expression of p, we set \(X = {{{P_S}{{\left| {{h_{SE}}} \right| }^2}} / {{N_0}}}\), \(Y = {{{P_E}{{\left| {{h_{Ep}}} \right| }^2}} / {{N_0}}}\) and \(Z = {{{P_R}{{\left| {{h_{RE}}} \right| }^2}} / {{N_0}}}\). According to (14) and (18), (26) can be rewritten as

$$\begin{aligned} {\eta _p} = {\tau _{p,s}}\Pr \left( {\frac{{XY}}{{YZ + X + Y + Z + {\rho _I} + 1}} > {\theta _5},\frac{X}{{Z + {\rho _I} + 1}} < {\theta _6}} \right) , \end{aligned}$$
(27)

where \({\theta _5} = {2^{3{\tau _{p,t}}}} - 1\) and \({\theta _6} = {2^{3\left( {{\tau _{p,t}} - {\tau _{p,s}}} \right) }} - 1\). The first part in the brackets of (27) can be written as

$$\begin{aligned} X > \frac{{{\theta _5}\left( {Y + YZ + {\rho _I} + 1} \right) }}{{Y - {\theta _5}}}. \end{aligned}$$
(28)

We have to notice that \(Y - {\theta _5} > 0\) must be satisfied, if not, \({\eta _p}\) will be zero. The second part in the brackets of (27) can be written as

$$\begin{aligned} X < {\theta _6}\left( {Z + {\rho _I} + 1} \right) . \end{aligned}$$
(29)

With (28) and (29), we can get the bounds of X and according to the bounds, \(\frac{{{\theta _5}\left( {Y + YZ + {\rho _I} + 1} \right) }}{{Y - {\theta _5}}} < {\theta _6}\left( {Z + {\rho _I} + 1} \right)\) must be satisfied which can be rewritten as

$$\begin{aligned} \left( {\left( {{\theta _5} - {\theta _6}} \right) Y + {\theta _5}{\theta _6}} \right) Z < \left( {{\theta _6}\left( {{\rho _I} + 1} \right) - {\theta _5}} \right) Y - \left( {{\theta _5}{\theta _6} + {\theta _5}} \right) \left( {{\rho _I} + 1} \right) . \end{aligned}$$
(30)

As \(\left( {\left( {{\theta _5} - {\theta _6}} \right) Y + {\theta _5}{\theta _6}} \right) > 0\) is always satisfied, we can get the bounds of Z as

$$\begin{aligned} 0< Z < \frac{{\left( {{\theta _6}\left( {{\rho _I} + 1} \right) - {\theta _5}} \right) Y - \left( {{\theta _5}{\theta _6} + {\theta _5}} \right) \left( {{\rho _I} + 1} \right) }}{{\left( {{\theta _5} - {\theta _6}} \right) Y + {\theta _5}{\theta _6}}}. \end{aligned}$$
(31)

It is noticed that \(\left( {{\theta _6}\left( {{\rho _I} + 1} \right) - {\theta _5}} \right) Y - \left( {{\theta _5}{\theta _6} + {\theta _5}} \right) \left( {{\rho _I} + 1} \right) > 0\) must be satisfied, if not, \({\eta _p}\) will be zero. Then, we can get the bounds of Y as

$$\begin{aligned} Y > \frac{{\left( {{\theta _5}{\theta _6} + {\theta _5}} \right) \left( {{\rho _I} + 1} \right) }}{{{\theta _6}\left( {{\rho _I} + 1} \right) - {\theta _5}}}. \end{aligned}$$
(32)

Compared (32) with the previous bound \(Y - {\theta _5} > 0\), we can easily get that (32) is the bond of Y. It is noticed that \({\theta _6}\left( {{\rho _I} + 1} \right) - {\theta _5} > 0\) must be satisfied in (32), or \({\eta _p} = 0\); this inequality holds easily when \({\rho _I}\) is large enough. Submitting the bounds expressions of (28), (29), (31) and (32) into (27), we can get the probability in (27) as

$$\begin{aligned}&\Pr \left( {\frac{{XY}}{{YZ + X + Y + Z + {\rho _I} + 1}} > {\theta _5},\frac{X}{{Z + {\rho _I} + 1}} < {\theta _6}} \right) \\&\quad = \int _{\frac{{\left( {{\theta _5}{\theta _6} + {\theta _5}} \right) \left( {{\rho _I} + 1} \right) }}{{{\theta _6}\left( {{\rho _I} + 1} \right) - {\theta _5}}}}^\infty {f\left( y \right) } dy\int _0^{\frac{{\left( {{\theta _6}\left( {{\rho _I} + 1} \right) - {\theta _5}} \right) y - \left( {{\theta _5}{\theta _6} + {\theta _5}} \right) \left( {{\rho _I} + 1} \right) }}{{\left( {{\theta _5} - {\theta _6}} \right) y + {\theta _5}{\theta _6}}}} {f\left( z \right) dz} \int _{\frac{{{\theta _5}\left( {y + yz + {\rho _I} + 1} \right) }}{{y - {\theta _5}}}}^{{\theta _6}\left( {z + {\rho _I} + 1} \right) } {f\left( x \right) dx}. \\ \end{aligned}$$
(33)

Following the channel assumption previously, we can get

$$\begin{aligned} & \Pr \left( {\frac{{XY}}{{YZ + X + Y + Z + \rho _{I} + 1}} > \theta _{5} ,\frac{X}{{Z + \rho _{I} + 1}} < \theta _{6} } \right) \\ & \quad = \lambda _{3} \int_{{\frac{{(\theta _{5} \theta _{6} + \theta _{5} )(\rho _{I} + 1)}}{{\theta _{6} (\rho _{I} + 1) - \theta _{5} }}}}^{\infty } {f(y)} dy\left[ {\frac{{y - \theta _{5} }}{{(\lambda _{3} + \lambda _{2} \theta _{5} )y - \lambda _{3} \theta _{5} }}} \right. \\ & \quad \times \left( {1 - \exp \left( { - \frac{{\left( {(\lambda _{3} + \lambda _{2} \theta _{5} )y - \lambda _{3} \theta _{5} } \right)}}{{y - \theta _{5} }}\frac{{\left( {\theta _{6} (\rho _{I} + 1) - \theta _{5} } \right)y - (\theta _{5} \theta _{6} + \theta _{5} )(\rho _{I} + 1)}}{{(\theta _{5} - \theta _{6} )y + \theta _{5} \theta _{6} }}} \right)} \right) \\ & \quad \times \exp \left( {\frac{{ - \lambda _{2} \theta _{5} (y + \rho _{I} + 1)}}{{y - \theta _{5} }}} \right) - \frac{1}{{\lambda _{3} + \lambda _{2} \theta _{6} }} \\ & \quad \times \left( {1 - \exp \left( { - (\lambda _{3} + \lambda _{2} \theta _{6} )\frac{{\left( {\theta _{6} (\rho _{I} + 1) - \theta _{5} } \right)y - \left( {\theta _{5} \theta _{6} + \theta _{5} } \right)\left( {\rho _{I} + 1} \right)}}{{(\theta _{5} - \theta _{6} )y + \theta _{5} \theta _{6} }}} \right)} \right) \\ & \quad \times \left. {\exp ( - (\lambda _{2} \theta _{6} (\rho _{I} + 1)))} \right]. \\ \end{aligned}$$
(34)

When Y is large enough (in most cases, Y is very large because of the large jamming power), we can get \({{\left( {\left( {{\lambda _3} + {\lambda _2}{\theta _5}} \right) y - {\lambda _3}{\theta _5}} \right) } / {y - {\theta _5}}} \approx \left( {{\lambda _3} + {\lambda _2}{\theta _5}} \right)\) and \(\frac{{\left( {{\theta _6}\left( {{\rho _I} + 1} \right) - {\theta _5}} \right) y - \left( {{\theta _5}{\theta _6} + {\theta _5}} \right) \left( {{\rho _I} + 1} \right) }}{{\left( {{\theta _5} - {\theta _6}} \right) y + {\theta _5}{\theta _6}}} \approx {{\left( {{\theta _6}\left( {{\rho _I} + 1} \right) - {\theta _5}} \right) } / {\left( {{\theta _5} - {\theta _6}} \right) }}\). Thus, invoking those two approximations into (34) and with the help of [31, Equation (3.352.2)], (34) can be expressed as

$$\begin{aligned} & \Pr \left( {\frac{{XY}}{{YZ + X + Y + Z + \rho _{I} + 1}} > \theta _{5} ,\frac{X}{{Z + \rho _{I} + 1}} < \theta _{6} } \right) \\ & \quad \approx \frac{{\lambda _{3} \exp ( - \lambda _{2} \theta _{5} )}}{{(\lambda _{3} + \lambda _{2} \theta _{5} )}}\left( {1 - \exp \left( { - \frac{{(\lambda _{3} + \lambda _{2} \theta _{5} )(\theta _{6} (\rho _{I} + 1) - \theta _{5} )}}{{(\theta _{5} - \theta _{6} )}}} \right)} \right) \\ & \quad \times \left( {\exp \left( { - \frac{{\lambda _{6} (\theta _{5} \theta _{6} + \theta _{5} )(\rho _{I} + 1)}}{{\theta _{6} (\rho _{I} + 1) - \theta _{5} }}} \right) + \left( {\frac{{\lambda _{3} \theta _{5} }}{{(\lambda _{3} + \lambda _{2} \theta _{5} )}} - \theta _{5} } \right)\exp \left( { - \frac{{\lambda _{3} \theta _{5} \lambda _{6} }}{{(\lambda _{3} + \lambda _{2} \theta _{5} )}}} \right)} \right. \\ & \quad \left. { \times Ei\left( { - \lambda _{6} \frac{{(\theta _{5} \theta _{6} + \theta _{5} )(\rho _{I} + 1)}}{{\theta _{6} (\rho _{I} + 1) - \theta _{5} }} + \frac{{\lambda _{3} \theta _{5} \lambda _{6} }}{{(\lambda _{3} + \lambda _{2} \theta _{5} )}}} \right)} \right) - \frac{{\lambda _{3} \exp ( - (\lambda _{1} \theta _{6} (\rho _{I} + 1)))}}{{\lambda _{3} + \lambda _{2} \theta _{6} }} \\ & \quad \times \left( {1 - \exp \left( { - (\lambda _{3} + \lambda _{2} \theta _{6} )\frac{{(\theta _{6} (\rho _{I} + 1) - \theta _{5} )}}{{(\theta _{5} - \theta _{6} )}}} \right)} \right)\exp \left( { - \frac{{\lambda _{6} (\theta _{5} \theta _{6} + \theta _{5} )(\rho _{I} + 1)}}{{\theta _{6} (\rho _{I} + 1) - \theta _{5} }}} \right). \\ \end{aligned}$$
(35)

where \({{\mathrm{Ei}}}\left( \cdot \right)\) denotes the exponential integral function [31, Equation (8.211. 2)]. Submitting (35) into (27), we can get the approximate EST expression of p which can be expressed as

$$\begin{aligned} & \eta _{p} \approx \tau _{{m,s}} \frac{{\lambda _{3} \exp ( - \lambda _{2} \theta _{5} )}}{{(\lambda _{3} + \lambda _{2} \theta _{5} )}}\left( {1 - \exp \left( { - \frac{{(\lambda _{3} + \lambda _{2} \theta _{5} )(\theta _{6} (\rho _{I} + 1) - \theta _{5} )}}{{(\theta _{5} - \theta _{6} )}}} \right)} \right) \\ & \quad \times \left( {\exp \left( { - \frac{{\lambda _{6} (\theta _{5} \theta _{6} + \theta _{5} )(\rho _{I} + 1)}}{{\theta _{6} (\rho _{I} + 1) - \theta _{5} }}} \right) + \left( {\frac{{\lambda _{3} \theta _{5} }}{{(\lambda _{3} + \lambda _{2} \theta _{5} )}} - \theta _{5} } \right)\exp \left( { - \frac{{\lambda _{3} \theta _{5} \lambda _{6} }}{{(\lambda _{3} + \lambda _{2} \theta _{5} )}}} \right)} \right. \\ & \quad \left. { \times Ei\left( { - \lambda _{6} \frac{{(\theta _{5} \theta _{6} + \theta _{5} )(\rho _{I} + 1)}}{{\theta _{6} (\rho _{I} + 1) - \theta _{5} }} + \frac{{\lambda _{3} \theta _{5} \lambda _{6} }}{{(\lambda _{3} + \lambda _{2} \theta _{5} )}}} \right)} \right) - \frac{{\lambda _{3} \exp ( - (\lambda _{1} \theta _{6} (\rho _{I} + 1)))}}{{\lambda _{3} + \lambda _{2} \theta _{6} }} \\ & \quad \times \left( {1 - \exp \left( { - (\lambda _{3} + \lambda _{2} \theta _{6} )\frac{{(\theta _{6} (\rho _{I} + 1) - \theta _{5} )}}{{(\theta _{5} - \theta _{6} )}}} \right)} \right)\exp \left( { - \frac{{\lambda _{6} (\theta _{5} \theta _{6} + \theta _{5} )(\rho _{I} + 1)}}{{\theta _{6} (\rho _{I} + 1) - \theta _{5} }}} \right). \\ \end{aligned}$$
(36)

4 Numerical results and discussion

In this section, computer simulations are performed to validate the above theoretical analysis. We introduce the channel model \({\text{E}}\left( {{{\left| {{h_{ij}}} \right| }^2}} \right) = d_{ij}^{ - \alpha }\), where \({d_{ij}}\) is the distance between node i and j, and \(\alpha = 3\) is the path loss exponent. The other parameters are set as follows: \({d_{SR}} = 20\) m, \({d_{SE}} = 30\) m, \({d_{RE}} = 20\) m, \({d_{Rm}} = 40\) m, \({d_{Rn}} = 20\) m, \({d_{Ep}} = 40\) m, \({R_{m,t}}={R_{n,t}}=0.8\) bits/s/Hz, \({R_{p,t}}=0.4\) bits/s/Hz, \({R_{m,s}}={R_{n,s}}=0.4\) bits/s/Hz, \({R_{p,s}}=0.2\) bits/s/Hz, \({a_m}=9/10\), \({a_n}=1/10\) and \({N_0}=-50\) dB.

Figure 2 illustrates the performance of the proposed scheme and HD-NOMA versus \(P_S\) with \({P_R} = - 18\,{\text{dB}}\), \({P_I} = 24\,{\text{dB}}\) and \({P_E} = 15\,{\text{dB}}\). The HD-NOMA system with two relays requires a total of 4 equal orthogonal time slots which is similar to [32]. The transmission detail of HD-NOMA is shown in Table 1. It can be observed that simulation and theoretical results match very well with each other for m and n and the approximate analysis for p is accurate because Y is very large in our setting which is very consistent with the actual communication scenario. Significantly, the EST of all users first increases and then decreases due to the increase of \(P_S\), which demonstrates that increasing \(P_S\) cannot always improve EST as high \(P_S\) increases the risk of eavesdropping. It also shows that there exists an optimal \(P_S\) of our scheme to maximize EST for all users and this optimal value can easily be obtained by a one-dimensional exhaustive search. Compared with HD-NOMA, the user p attains worse performance for any \(P_S\), but one time slot can be saved in our scheme and this performance gap narrows when the optimal value of \(P_S\) is obtained. In addition, our scheme outperforms HD-NOMA for users m and n when \(P_S\) is small. All of the above statements illustrate the practical feasibility of our scheme.

Fig. 2
figure 2

The EST of users m, n and p versus \(P_S\)

Full size image
Table 1 Transmission of HD-NOMA
Full size table

Figure 3 shows the EST of users versus \(P_R\) with \({P_S} = - 6\,{\text{dB}}\), \({P_I} = 24\,{\text{dB}}\) and \({P_E} = 15\,{\text{dB}}\). For m and n of VFD-NOMA, it is shown that the analytical curves perfectly match the Monte Carlo simulation results and EST of m and n first increases and then decreases as high \(P_R\) increases the risk of eavesdropping for both two schemes. The optimal value of \(P_R\) to maximize the EST of m and n can also easily be obtained by a one-dimensional exhaustive search. For p, Fig. 3 confirms the close agreement between the simulation and approximate analytical results. We can also see that the performance of p in HD-NOMA does not affect by \(P_R\), but the performance of p in our scheme decreases as \(P_R\) increases which is because \(P_R\) is considered as interference when E forwards the information of p. In addition, it is seen that when \(P_R\) is small, the change of \(P_R\) does not affect the performance of p in our scheme. Although the existing HD-NOMA outperforms our protocol in some cases, our scheme can save one time slot and this performance gap is acceptable in actual communication scenario. Since the theoretical analyses agree well with the simulations, we will only plot the analytical results in the remaining figure.

Fig. 3
figure 3

The EST of users m, n and p versus \(P_R\)

Full size image

Figure 4 plots the performance of our scheme and HD-NOMA versus \(P_I\) with \({P_S} = - 6\,{\text{dB}}\), \({P_R} = - 18\,{\text{dB}}\) and \({P_E} = 15\,{\text{dB}}\). We can observe that for m and n, increasing \(P_I\) can improve the EST, but 28dB is enough and no more power is needed. This result can help us saving interference signal power in actual communication scenario. We can also find that for p, increasing \(P_I\) cannot always improve EST as high \(P_I\) reduces the efficiency of the amplifier and the optimal value of \(P_I\) for p is basically the same as m and n.

Fig. 4
figure 4

The EST of users m, n and p versus \(P_I\)

Full size image

5 Conclusion

In this paper, we studied the secrecy performance in VFD-NOMA networks. A VFD relaying scheme with an untrusted AF-HD relay and a trusted DF-HD relay is used in this system to improve the spectral efficiency where cooperative jamming is designed to confuse the untrusted relay. The exact and approximate expressions for EST are derived which shows that jamming is necessary for secure transmission and the powers of S, R and jamming should be carefully considered to improve secrecy performance and save the system energy consumption. From the simulation results, we can see that the proposed VFD-NOMA scheme is more practical than existing HD-NOMA scheme. As future work, the imperfect SIC receiver and adaptive power split factors can be further investigated.