1 Introduction

With the rapid popularization of smart terminals and multimedia services, data traffic in wireless communication networks is growing exponentially, which demands a huge amount of spectrum resources. However, the remaining available spectrum is scarce while efforts are made to explore new frequency bands and to improve spectrum utilization efficiency. Cognitive relay network was considered to have a great potential to mitigate the spectrum scarcity problem and has become a hot topic in the field of wireless communications in recent years. For example, authors in [1] proposed an energy-efficient relay selection and power allocation scheme in cooperative cognitive radio networks. Based on the analysis of outage probabilities of primary users and secondary users in cognitive relay networks, novel cooperative relay selection schemes were proposed in [2, 3]. In [4], authors took the secondary user as potential cooperator for the primary user, in which the secondary transmitter acts as cooperative relay for the primary transmitter to enhance their outage performances.

Except for the research on the performance of cognitive relay networks, the transmission mode of cooperative relays also has attracted a lot of research efforts. Depending on whether the transmission and reception can be done simultaneously in the same frequency band, relay transmission mode can be generally categorized into half duplex (HD) and full duplex (FD) [5]. Because of the low complexity of relay design in the HD mode, a large number of works have been done for it, such as works on the outage probability, the resource allocation for multi-carrier non-orthogonal multiple access systems, and the channel capacity were reported in [6,7,8], respectively. In [9], a new transmission protocol for the HD multi-hop relaying system was proposed, which selected optimal states of nodes and corresponding optimal transmission rates such that the achievable average rate from the source to the destination was maximized. Though the HD mode is very popular, it requires two orthogonal phases for receiving and transmitting signals which causes a waste of spectrum resources.

Recently, with the continuous improvement of antenna technology and signal processing capability, self-interference in the FD mode can be well eliminated [10,11,12]. In particular, the basic cause of self-interference cancelation bottlenecks in the FD mode has been studied in [13], which indicated that self-interference can be further suppressed. According to that the FD mode has received a lot of research interest from both industry and academia [14]. For example, multi-objective optimization for power-efficient and secure FD communication systems was studied in [15], while joint user selection and power allocation in FD multicell networks was investigated in [16]. Considering different system models with FD relays, authors in [17,18,19] discussed power allocation and energy efficiency. In addition, a new device-to-device (D2D) communication scheme was proposed which allowed D2D links to underlay the cellular downlink by assigning D2D transmitters as FD relays to assist the cellular downlink transmission [20]. Besides, the energy efficiency and outage performance were also deeply studied in FD D2D communications [21, 22]. Although the FD mode has been widely studied, with the increase of self-interference power, its performance will be greatly degraded, which may even be worse than that of the HD mode [23].

To avoid disadvantages of HD and FD modes while utilizing advantages of them, some earlier works have investigated the combination of HD and FD. For example, a novel scheme consisting of opportunistic mode selection between HD and FD was proposed in [23], which was compared with either the HD mode or the FD mode. The results showed that the instantaneous and average spectral efficiency were improved a lot. Authors of [24] proposed an optimal transmission scheduling scheme for a hybrid HD/FD relaying system, which could achieve a higher spectral efficiency than a single duplex relaying system. A hybrid duplex scheme was proposed in a random-access wireless network and heterogeneous wireless networks [25, 26], which could get a high throughput. In [27,28,29], a hybrid HD/FD relaying mode was proposed for wireless ad hoc networks, heterogeneous networks, and cognitive relay networks, which aimed to maximize the security performance or the sum rate. In addition, a joint relay mode selection and power allocation model was proposed in [30], with the aim to maximize the sum rate of a multi-carrier relay network with hybrid relay modes on a per sub-carrier basis. Moreover, a novel parallel hybrid radio frequency/free-space optical relaying system with both the non-buffer-aided and the buffer-aided schemes has been studied in [31], and optimal relay selection policies were used to maximize the end-to-end throughput. Authors in [32] adopted an adaptive antenna which can automatically select receiving or transmitting signals to maximize the end-to-end signal-to-interference-plus-noise ratio in the FD relaying system. In a relay-aided cellular network, an opportunistic HD/FD relaying mode selection scheme based on the received signal-to-interference-plus-noise ratio was introduced, which can achieve a high energy efficiency [33].

Most of existing works focused on the performance of primary system and secondary system in cognitive relay networks or opportunistic relaying mode selection in a cooperative relay network, but rarely consider opportunistic relaying mode selection in cognitive relay networks. [29] studied the opportunistic relaying mode selection in an underlay spectrum sharing system, but focused on the outage performance. In this work, an opportunistic HD/FD relaying mode selection criterion in a cognitive relay network is proposed where a secondary transmitter acts as a cooperative relay to assist the transmission of primary system, in order to get a high throughput of secondary system under the interference constraint of primary system and transmission power constraints. This opportunistic relaying mode selection criterion not only utilizes the throughput advantage of FD but also weakens the negative effect of self-interference. In case of reliable communications in the primary system, the secondary system can achieve maximum throughput. Though both this work and earlier works [23, 29, 33] studied opportunistic HD/FD relaying mode selection through mode switching, they are quite different in several aspects. Firstly, in this work, we consider a cognitive relay network but in [23, 29, 33] authors considered a three-node relay network, another kind of cognitive relay network, and a relay-aided cellular network, respectively. Secondly, in this work, the switching between HD and FD is based on the residual self-interference power but in [23, 29] and [33] the switching was based on the maximum channel capacity and the signal-to-interference-plus-noise ratio, respectively. Thirdly, in this work the goal is to maximize the throughput of secondary system but in [23, 29, 33], the goals were to maximize the spectral efficiency, the outage probability, and the energy efficiency, respectively.

The remainder of this paper is organized as follows. Section 2 describes the cognitive relay network, the HD transmission mode, the FD transmission mode, and the opportunistic HD/FD relaying mode selection criterion. Then, the secondary throughput derivation and the problem of maximizing the throughput of secondary system is formulated and solved in Section 3. Section 4 presents simulation results of secondary throughput. Finally, concluding remarks are made in Section 5.

2 System model

The cognitive relay network under consideration is shown in Fig. 1. The primary system consists of a primary transmitter (PT) and a primary receiver (PR), while the secondary system consists of a secondary transmitter (ST) and a secondary receiver (SR). It should be noted that the primary system (PT, PR) has absolute control over the use of its licensed frequency band, while the secondary system (ST, SR) is willing to offer assistance for the primary system and in return gets the opportunity to use the licensed frequency band. To be more specific, ST acts as cooperative relay to assist the transmission in the primary system, in order to ensure the communication quality-of-service of the primary system. In return, ST is allowed to transmit its own signal in the licensed frequency band. This system model may be applied in the scenario where there are no dedicated relay nodes but a secondary system shares the frequency band of a primary system through cooperative relaying [4]. All of the nodes are half-duplex except for ST that has the full-duplex capability. ST is equipped with a transmitting antenna and a receiving antenna while other nodes are equipped with a single antenna. ST can opportunistically switch between the HD mode and the FD mode according to a certain criterion. Through checking out the literature, earlier works addressing the integration of HD and FD modes in a terminal was rarely found. For example, HD/FD mode switching boundaries were discussed in [23] but the way of implementation was not mentioned. A possible strategy to integrate HD and FD modes in ST is to dynamically control its receiving and transmitting antennas to achieve HD/FD mode switching with a software-defined radio module. The whole transmission process consists of multiple transmission slots. In the initialization of each transmission slot, the default FD mode is tested and the residual self-interference power is measured, like pilot training in channel estimation. The principle of HD and FD mode switching is elaborated as follows. In the first transmission slot, if the residual self-interference power is below a given threshold, ST retains the FD mode and its receiving and transmitting antennas are activated. This slot is not divided into two equal sub-slots. The transmitting antenna of ST transmits the signal received by the receiving antenna of ST immediately. In the second transmission slot, if the residual self-interference power is above a given threshold, ST switches to the HD mode and one of its antennas is deactivated. This slot is divided into two equal sub-slots. The activated antenna of ST receives a signal in the first sub-slot and transmits the received signal in the second sub-slot. The HD and FD mode switching occurs in successive transmission slots following the above principle. Note that the focus of this work is on the signal transmission process while operations of antennas and signal processing are beyond the scope.

Fig. 1
figure 1

A cognitive relay network with HD/FD mode switching

The simple amplify-and-forward relaying protocol is adopted. In the HD mode, a transmission slot is divided into two equal sub-slots. In the first sub-slot, PT transmits a signal to PR, while ST and SR also receive the signal. In the second sub-slot, ST combines the received signal with its own signal and then amplifies the composite signal and forwards the amplified signal to PR and SR. PR and SR recover their desired signals from their received signals, respectively. In the FD mode, each transmission slot is not divided into two equal sub-slots. ST receives the signal from PT and immediately amplifies and forwards the composite signal to PR and SR.

All the channels are assumed to experience Rayleigh flat fading and the channel state information is perfectly known through channel estimation, and remains constant within a transmission slot. Channel coefficients of links PT → PR, PT → ST, PT → SR, ST → PR, and ST → SR are denoted by h1, h2, h3, h4, and h5, respectively. Moreover, it is assumed that \( {h}_i\sim CN\left(0,{d}_i^{-v}\right)\ \left(i=1,2,3,4,5\right) \), which means that h i is a circularly symmetric complex Gaussian random variable with variance \( {d}_i^{-v} \). Here, d i represents the normalized distance between two nodes and v represents the path loss exponent. That is to say, h1, h2, h3, h4, and h5 denote normalized distances between PT and PR, PT and ST, PT and SR, ST and PR, and ST and SR, respectively. This distance normalization is done with respect to the distance between PT and PR, i.e., d1 = 1. It is worth noting that the self-interference can be mitigated and its residual part also can be considered as following circularly symmetric complex Gaussian distribution (see [17, 18, 21, 23] and references therein). So, the self-interference channel coefficient from the transmitting antenna to the receiving antenna of ST can be modeled as \( {h}_q\sim CN\left(0,{d}_q^{-v1}\right) \), where d q is the distance between transmitting and receiving antennas of ST and v1 is the path loss exponent of the self-interference channel.

2.1 HD transmission mode

When the half-duplex (HD) transmission mode is used, although the self-interference does not exist, but the utilization efficiency of a frequency band is only half of the FD transmission mode. Only one antenna at ST is activated to receive and transmit signals in two sub-slots. In the first sub-slot, PT transmits the primary signal x P (m) (with zero mean and normalized variance 1) with a transmission power P P . PR, ST, and SR receive the signal. Received signals of PR, ST, and SR are denoted by y11, y21, y31, respectively, which are written as

$$ {y}_{a1}(m)=\sqrt{P_p}{h}_a{x}_p(m)+{n}_{a1}(m), $$
(1)

where a = 1, 2, and 3. Here, h a is a channel coefficient, and na1~N(0, σ2) is an additive white Gaussian noise (AWGN) with zero mean and variance σ2. The signal-to-noise ratio (SNR) at ST can be calculated as

$$ {SNR}_{1, ST}^{HD}=\frac{P_P{\left|{h}_2\right|}^2}{\sigma^2}. $$
(2)

In the second sub-slot, ST combines the signal \( {x}_{c1}(m)=\sqrt{P_P}{h}_2{x}_p(m)+{n}_{21}(m) \) received in the first sub-slot with its own signal x s (m) and generates the amplified composite signal x c (m). In order to simplify the signal transmission process, weighting of signals is not considered and these signals are normalized to the same order of magnitudes. The amplified signal is then forwarded to PR and SR. The amplified composite signal can be represented by

$$ {x}_c(m)=\sqrt{\beta}\left({x}_{c1}(m)+{x}_s(m)\right), $$
(3)

where β is the amplification gain of ST in the HD mode, x s (m) is the secondary signal with zero mean and normalized variance 1. Received signals at PR and SR are denoted by \( {\tilde{y}}_1(m) \) and \( {\tilde{y}}_3(m) \), respectively, which can be expressed as

$$ {\tilde{y}}_1(m)={h}_4{x}_c(m)+{v}_1(m), $$
(4)
$$ {\tilde{y}}_3(m)={h}_5{x}_c(m)+{v}_3(m). $$
(5)

Here, v1 and v3 are AWGN with zero mean and variance σ2. PR decodes the primary signal x p (m) from signals received in these two sub-slots, and the interference power from the secondary system to the primary system can be expressed as

$$ {P}_I=\beta {\left|{h}_4\right|}^2. $$
(6)

SR decodes the secondary signal x s (m) from the signal received in the second sub-slot, and the throughput can be expressed as

$$ {R}_{half}=\frac{1}{2}W{\log}_2\left(1+\frac{\beta {\left|{h}_5\right|}^2}{\beta {\left|{h}_5\right|}^2\left({P}_P{\left|{h}_2\right|}^2+{\sigma}^2\right)+{\sigma}^2}\right). $$
(7)

Here, W is the bandwidth and the value \( \frac{1}{2} \) is due to the fact that the transmission slot is divided into two equal sub-slots. Note that the amplify-and-forward relaying protocol is adopted and SR does not attempt to decode the primary signal, thus it is treated as an interference in (7). In addition, the goal of this work is to maximize the throughput of secondary system while satisfying the interference constraint of primary system, so maximizing the throughput of primary system is not concerned, but the maximal-ratio combining can be applied to improve the diversity gain of primary system [4].

2.2 FD transmission mode

When the full-duplex (FD) transmission mode is used, a transmission slot is not divided into two equal sub-slots. It allows simultaneous transmission and reception in the same frequency band. However, the communication quality-of-service is degraded by the severe self-interference from the transmitting antenna to the receiving antenna of ST. In the FD transmission mode, ST will amplify and forward the composite signal received at the previous time instant m − 1. Specifically, the signal received by ST at the time instant m can be written as \( {\tilde{x}}_{c1}(m)=\sqrt{P_P}{h}_2{x}_p(m)+{n}_{21}(m)+{v}_r(m) \), where \( {v}_r(m)=\sqrt{P_q}{h}_qx(m) \) is the residual self-interference due to the FD operation, x(m) = x lin (m) + x imp (m) is the transmitted SI containing the known linear part x lin (m) and transmitter impairments x imp (m), and P q is the residual self-interference power after interference cancelation. Moreover, it is worth noting that the analysis of these two parts has been done in [13]. But in order to simplify the following analysis and focus on the residual self-interference power, x imp (m) is not considered.

Next, ST will forward the signal \( {\tilde{x}}_{c1} \) received at the previous time instant along with its own signal x s . The signal forwarded by ST at the time instant m can be represented by

$$ {\tilde{x}}_c(m)=\sqrt{\beta^{\hbox{'}}}\left({\tilde{x}}_{c1}\left(m-1\right)+{x}_s(m)\right), $$
(8)

where β is the amplification gain of ST in the FD transmission mode. In order to analyze the effect of residual SI on the system performance, it is assumed that the residual SI signal v r is zero mean, additive, and white Gaussian, which is denoted as v r ~CN(0, V) [20, 21]. Received signals at PR and SR from ST are denoted by \( {\tilde{z}}_1(m) \) and \( {\tilde{z}}_3(m) \), respectively, which can be expressed as

$$ {\tilde{z}}_1(m)={h}_4{\tilde{x}}_c(m)+{v}_1(m), $$
(9)
$$ {\tilde{z}}_3(m)={h}_5{\tilde{x}}_c(m)+{v}_3(m). $$
(10)

Note that due to the FD operation, PR and SR also receive signals from PT at the time instant m. The received signals at PR and SR are \( {\tilde{z}}_1(m)+\sqrt{P_p}{h}_1{x}_p(m) \) and \( {\tilde{z}}_3(m)+\sqrt{P_p}{h}_3{x}_p(m) \), respectively. PR decodes the primary signal x p (m) from the signal received at the time instant m, and the interference power from the secondary system to the primary system can be expressed as

$$ {\tilde{P}}_I={\beta}^{\hbox{'}}{\left|{h}_4\right|}^2\left(1+{P}_q{\left|{h}_q\right|}^2\right). $$
(11)

SR decodes the secondary signal x s (m) from the signal received at the time instant m, and the throughput can be expressed as

$$ {R}_{full}=W{\log}_2\left(1+\frac{\beta^{\prime }{\left|{h}_5\right|}^2\left(1+{P}_q{\left|{h}_q\right|}^2\right)}{\beta {\left|{h}_5\right|}^2\left({P}_P{\left|{h}_2\right|}^2+{\sigma}^2\right)\left(1+{P}_q{\left|{h}_q\right|}^2\right)+{P}_P{\left|{h}_3\right|}^2+{\sigma}^2}\right). $$
(12)

Note that in [23, 29], transmission powers in the FD mode were halved in order to make a fair comparison between the FD mode and the HD mode. But in this work, the target is to maximize the throughput of secondary system when FD and HD modes are opportunistically selected. Therefore, the transmission power in the FD mode is not necessarily halved.

2.3 Opportunistic HD/FD relaying mode selection

It is difficult to find a general way for HD/FD relaying mode selection. Therefore, an opportunistic HD/FD relaying mode selection criterion is designed as follows: ST selects the FD mode or the HD mode based on the residual self-interference power. When the residual self-interference power at ST in the FD transmission mode is below a given threshold Γ, the FD mode is preferred in order to get a high throughput. However, when the residual self-interference power is above a given threshold Γ, the FD mode suffers a great performance loss and the HD mode becomes a better option. In practice, the residual self-interference power may be measured at ST by using the method in [34] or other methods.

3 Secondary throughput in the opportunistic mode selection

In this section, firstly, the throughput of secondary system in the opportunistic mode selection is derived. Then, the throughput maximization of secondary system under the interference constraint of primary system and transmission power constraints is formulated as a constrained optimization problem. From the description of opportunistic mode selection, the secondary throughput can be expressed as

$$ R=P\left({P}_q>\Gamma \right){R}_{half}+P\left({P}_q\le \Gamma \right){R}_{full}. $$
(13)

Here, P(P q  > Γ) denotes the probability that P q is greater than Γ, the same as P(P q  ≤ Γ). From (13), we know that R is calculated in the statistical average sense, in order to reflect the average throughput that can be obtained through mode switching. It is worth noting that since it is assumed that the channel state information is constant within a transmission slot and can be perfectly obtained by channel estimation, R represents the throughput of secondary system when ST selects a relaying mode according to the residual self-interference power in a transmission slot. From the perspective of multiple transmission slots, due to the variation of channel state information over slots, R represents the throughput of secondary system when ST selects a relaying mode according to the residual self-interference power in every slot.

3.1 Secondary throughput derivation

There are many practical models of self-interference which greatly affects the throughput of FD transmission. As the self-interference cancelation itself is not concerned, the early and commonly used model in [35] is adopted. Based on experimental results in [35], the variance of the residual SI is modeled as \( V=\frac{1}{\omega }{P_S}^{\lambda } \), that is to say, the residual self-interference power can be expressed as

$$ {P}_q=\frac{1}{\omega }{P_S}^{\lambda }, $$
(14)

where 1/ω and λ(0 ≤ λ ≤ 1) are constants, and 1/ω indicates the attenuation of residual self-interference power which reflects the effectiveness of selected interference cancelation technique. This model generally includes two cases: the optimistic case in which the self-interference variance is simply a constant and is not a function of the transmission power (λ = 0) [14, 36], and the other case in which the variance increases linearly with the transmission power (λ = 1) [32, 37]. Without loss of generality, the second case is considered in which P q  = P S /ω, and 1/ω plays an important role in the FD transmission mode which characterizes the quality of self-interference cancelation. Here, P S is the transmission power of ST in the FD mode, which can be expressed as

$$ {P}_S={\beta}^{\hbox{'}}\left({P}_p{\left|{h}_2\right|}^2+{P}_p{\left|{h}_q\right|}^2+{\sigma}^2+1\right). $$
(15)

Substituting (15) into (14), we can get the expression of P q as

$$ {P}_q=\frac{\beta^{\prime}\left({P}_P{\left|{h}_2\right|}^2+{\sigma}^2+1\right)}{\omega -{\beta}^{\prime }{\left|{h}_q\right|}^2}. $$
(16)

In order to facilitate the calculation, denote r i  = |h i |2(i = 1, 2, 3, 4, 5, q). r i follows exponential distribution with the parameter \( {\lambda}_i={d}_i^v\left(i=1,2,3,4,5,q\right) \). The probability P(P q  > Γ) can be further calculated as \( P\left\{\frac{\beta^{\prime}\left({P}_P{r}_2+{\sigma}^2+1\right)}{\omega -{\beta}^{\prime }{r_q}^2}>\Gamma \right\} \). The probability P(P q  > Γ) is difficult to be obtained directly. So the above fraction is decomposed into two parts and their properties are observed, respectively. It is assumed that there is a probability P1 which can be described as f1(t) = P1(βP P r2 + (βσ2 + β) < t). Here, in addition to r2, other parameters can be regarded as constants. So, we can get the following form \( {P}_1\left({r}_2<\frac{t\hbox{-} \left({\beta}^{\prime }{\sigma}^2+{\beta}^{\prime}\right)}{\beta^{\prime }{P}_P}\right)\left(t\ge \left({\beta}^{\prime }{\sigma}^2+{\beta}^{\prime}\right)\right) \). We can also know that r2~ exp(λ2). So, we can get the probability distribution function of r2 as\( F\left({r}_2\right)=1-\exp \left(-{\lambda}_2\frac{t\hbox{-} \left({\beta}^{\prime }{\sigma}^2+{\beta}^{\prime}\right)}{\beta^{\prime }{P}_P}\right) \). Next, the first-order derivative of F(r2) with respect to t will be found. The probability density function of P1 is obtained and can be expressed as

$$ {f}_y={F}_t^{\hbox{'}}\left({r}_2\right)=A\exp \left(- By\right)\left(y\ge \left({\beta}^{\hbox{'}}{\sigma}^2+{\beta}^{\hbox{'}}\right)\right). $$
(17)

Here, A is \( {e}^{\frac{\lambda_2\left({\beta}^{\prime }{\sigma}^2+{\beta}^{\prime}\right)}{\beta^{\prime }{P}_P}}\frac{\lambda_2}{\beta^{\prime }{P}_P} \) and B is \( \frac{\lambda_2}{\beta^{\prime }{P}_P} \). The same as before, we can get the probability density function of f2(t) = P2(ω − βr q  < t), which is expressed as

$$ {f}_x={F}_t^{\hbox{'}}\left({r}_q\right)=C\exp \left(- Dx\right)\left(x<\omega \right). $$
(18)

Here, C is \( {e}^{\frac{-{\lambda}_q\omega }{\beta^{\prime }}}\frac{\lambda_q}{\beta^{\prime }} \) and D is \( \frac{-{\lambda}_q}{\beta^{\prime }} \).

From the previous derivation, we can get the probability P(P q  ≤ Γ), which can be expressed as

$$ P\left({P}_q\le \Gamma \right)=P\left\{\frac{\beta^{\prime}\left({P}_P{r}_2+{\sigma}^2+1\right)}{\omega -{\beta}^{\prime }{r_q}^2}\le \Gamma \right\}=P\left\{\frac{Y}{X}\le Z\right\}, $$
(19)

where Y is β(P P r2 + σ2 + 1), X is ω − βr q 2, and Z denotes Γ. From the knowledge of probability and statistics, the probability \( P\left\{\frac{Y}{X}\le Z\right\} \) can be derived as

$$ {\displaystyle \begin{array}{l}P\left\{\frac{Y}{X}\le Z\right\}={P}_{Y/X}(Z)=\underset{s}{\iint }f\left(x,y\right) dxdy={\iint}_{Y/X\le Z,y\ge \left({\beta}^{\hbox{'}}{\sigma}^2+{\beta}^{\hbox{'}}\right),x<\omega }f\left(x,y\right) dxdy\\ {}=\underset{-\infty }{\overset{0}{\int }} dx\underset{\beta \hbox{'}\left({\sigma}^2+1\right)}{\overset{\infty }{\int }} AC\exp \left(- By- D x\right) dy+\underset{\frac{\beta^{\hbox{'}}\left({\sigma}^2+1\right)}{Z}}{\overset{\omega }{\int }} dx\underset{\beta \hbox{'}\left({\sigma}^2+1\right)}{\overset{Zx}{\int }} AC\exp \left(- By- D x\right) dy\\ {}=\frac{AC}{B D}\exp \left(-{B\beta}^{\hbox{'}}\left({\sigma}^2+1\right)\right)\left[\exp \left(-\frac{{D\beta}^{\hbox{'}}\left({\sigma}^2+1\right)}{Z}\right)-\exp \left(- D\omega \right)-1\right]+\frac{AC}{B}\frac{1}{B Z+D}\\ {}\left[\exp \left(-\left( BZ+D\right)\omega \right)-\exp \left(-\left(B+\frac{D}{Z}\right){\beta}^{\hbox{'}}\left({\sigma}^2+1\right)\right)\right]\end{array}} $$
(20)

Next, values of A, B, C, D, and Z are substituted into (20) and the expression of the probability P(P q  ≤ Γ) is obtained as

$$ P\left({P}_q\le \Gamma \right)=\kern0.5em \frac{\lambda_2\Gamma}{\lambda_q{P}_P-{\lambda}_2\Gamma}\exp \left(\frac{\lambda_q{\beta}^{\prime}\left({\sigma}^2+1\right)-{\lambda}_q\Gamma \omega }{\beta^{\prime}\Gamma}\right)+\frac{\lambda_q{P}_P}{\lambda_2\Gamma -{\lambda}_q{P}_P}\exp \left(\frac{\lambda_2{\beta}^{\prime}\left({\sigma}^2+1\right)-{\lambda}_2\Gamma \omega }{\beta^{\prime }{P}_P}\right)+\exp \left(\frac{-{\lambda}_q\omega }{\beta^{\prime }}\right)+1. $$
(21)

When (7), (12), and (21) are substituted into (13), we can obtain the expression of the secondary throughput in the opportunistic mode selection, which can be expressed as

$$ {\displaystyle \begin{array}{l}R=P\left({P}_q>\Gamma \right){R}_{half}+P\left({P}_q\le \Gamma \right){R}_{full}\\ {}\kern0.75em =\left\{\left(\frac{\lambda_2\Gamma}{\lambda_q{P}_P-{\lambda}_2\Gamma}\exp \left(\frac{\lambda_q{\beta}^{\prime}\left({\sigma}^2+1\right)-{\lambda}_q\Gamma \omega }{\beta^{\prime}\Gamma}\right)-\frac{\lambda_q{P}_P}{\lambda_2\Gamma -{\lambda}_q{P}_P}\exp \left(\frac{\lambda_2{\beta}^{\prime}\left({\sigma}^2+1\right)-{\lambda}_2\Gamma \omega }{\beta^{\prime }{P}_P}\right)-\exp \left(\frac{-{\lambda}_q\omega }{\beta^{\prime }}\right)\right)\ast \frac{1}{2}W{\log}_2\left(1+\frac{\beta {\left|{h}_5\right|}^2}{\beta {\left|{h}_5\right|}^2\left({P}_P{\left|{h}_2\right|}^2+{\sigma}^2\right)+{\sigma}^2}\right)\right\}\\ {}\kern0.75em +\left\{\left[\frac{\lambda_2\Gamma}{\lambda_q{P}_P-{\lambda}_2\Gamma}\exp \left(\frac{\lambda_q{\beta}^{\prime}\left({\sigma}^2+1\right)-{\lambda}_q\Gamma \omega }{\beta^{\prime}\Gamma}\right)+\frac{\lambda_q{P}_P}{\lambda_2\Gamma -{\lambda}_q{P}_P}\exp \left(\frac{\lambda_2{\beta}^{\prime}\left({\sigma}^2+1\right)-{\lambda}_2\Gamma \omega }{\beta^{\prime }{P}_P}\right)+\exp \left(\frac{-{\lambda}_q\omega }{\beta^{\prime }}\right)+1\right]\ast W{\log}_2\left(1+\frac{\beta^{\prime }{\left|{h}_5\right|}^2\left(1+{P}_q{\left|{h}_q\right|}^2\right)}{\beta^{\prime }{\left|{h}_5\right|}^2\left({P}_P{\left|{h}_2\right|}^2+{\sigma}^2\right)\left(1+{P}_q{\left|{h}_q\right|}^2\right)+{P}_P{\left|{h}_3\right|}^2+{\sigma}^2}\right)\right\}.\end{array}} $$
(22)

3.2 Secondary throughput maximization

In the opportunistic mode selection, the objective is to seek optimal amplification gains β and β in order to maximize R while keeping P I and \( {\tilde{P}}_I \) below a threshold and the transmission power of ST does not exceed its limit. The secondary throughput maximization problem can be formulated as

$$ {\displaystyle \begin{array}{l}\underset{\beta, {\beta}^{\hbox{'}}}{\max}\kern1em R\\ {}\mathrm{s}.\mathrm{t}.\kern0.5em 0<{P}_I\le \Lambda, \\ {}\kern1.5em 0<{\tilde{P}}_I\le \Lambda, \\ {}\kern1.5em 0<\beta \le {\beta}_{\mathrm{max}},\\ {}\kern1.5em 0<{\beta}^{\hbox{'}}\le {\beta}_{\mathrm{max}}^{\hbox{'}},\end{array}} $$
(23)

where Λ is the interference threshold at PR, βmax and \( {\beta}_{\mathrm{max}}^{\prime } \) are the maximum allowed amplification gains of ST in the HD mode and the FD mode, respectively, which are βmax = Ps, max/(P P r2 + σ2 + 1) and \( {\beta}_{\mathrm{max}}^{\prime }={P}_{s,\max }/\left({P}_P{r}_2+{\mathrm{P}}_q{r}_q+{\sigma}^2+1\right) \), and Ps, max is the maximum allowed transmission power of PT. Note that the interference endured by the primary system is limited by constraints in (23) and the secondary system will not cause harmful interference to the primary system.

Unfortunately, the joint optimization over β and β is very hard due to the fact that R is not concave in β and β jointly. To overcome this difficulty, we can first optimize over one variable, and let the other variable be fixed. That is, we can optimize over β for a fixed β, and optimize over β for a fixed β, separately. Then, we can consider the joint optimization by utilizing separate optimization results. A one-dimensional search (ODS) method and an alternate optimization (AOP) method are proposed to find the solution to the optimization problem in (23). In the following, details of the AOP method and the ODS method are given.

Optimization over β for a fixed β: Given β, the optimization over β can be formulated as

$$ \underset{\beta }{\max }\ R\kern0.5em \mathrm{s}.\mathrm{t}.\kern0.5em 0<{P}_I\le \Lambda, 0<\beta \le {\beta}_{\mathrm{max}}. $$
(24)

From the domain of the function P I and the inequality constraint in (24), we can obtain the feasible region of β as \( \beta \in \left(0,{\widehat{\beta}}_{\mathrm{max}}\right] \), where \( {\widehat{\beta}}_{\mathrm{max}}=\min \left\{\frac{\Lambda}{r_4},{\beta}_{\mathrm{max}}\right\} \).

Theorem 1: R is strictly quasi-concave in β for β ∈ [0, +∞).

Proof: See Appendix 1.

From Theorem 1, there are only three cases for the curve R versus β for \( \beta \in \left(0,{\widehat{\beta}}_{\mathrm{max}}\right] \).

Case 1: R1 strictly increases with β for \( \left(0,{\widehat{\beta}}_{\mathrm{max}}\right] \) if \( {\left.{dR}_1/ d\beta\ \right|}_{\beta ={\widehat{\beta}}_{\mathrm{max}}}\ge 0 \), where dR/ is given by (26) in Appendix 1. The solution to the optimization problem (24) is achieved at \( \overset{\smile }{\beta }={\widehat{\beta}}_{\mathrm{max}} \).

Case 2: R strictly decreases with β for \( \left(0,{\widehat{\beta}}_{\mathrm{max}}\right] \) if \( {\left. dR/ d\beta\ \right|}_{\beta =\left(0,{\widehat{\beta}}_{\mathrm{max}}\right]}\le 0 \). The optimal solution is achieved at \( \overset{\smile }{\beta}\approx 0 \).

Case 3: R first strictly increases and then strictly decreases with β for \( \left(0,{\widehat{\beta}}_{\mathrm{max}}\right] \) ifdR/ |β = 0 > 0 and \( {\left. dR/ d\beta\ \right|}_{\beta ={\widehat{\beta}}_{\mathrm{max}}}<0 \). The optimal solution is achieved at \( \overset{\smile }{\beta }={\beta}^{\ast } \), where β is the point at which R reaches its maximum when \( \beta \in \left(0,{\widehat{\beta}}_{\mathrm{max}}\right] \) for a fixed β and is obtained by solving the equation dR/ = 0.

Optimization over βfor a fixed β: Given β, the optimization over β can be formulated as

$$ \underset{\beta^{\hbox{'}}}{\max }\ R\kern0.5em \mathrm{s}.\mathrm{t}.\kern0.5em 0<{\tilde{P}}_I\le \Lambda, 0<{\beta}^{\hbox{'}}\le {\beta}_{\mathrm{max}}^{\hbox{'}}, $$
(25)

From the domain of the function \( {\tilde{P}}_I \) and the inequality constraint in (25), we can obtain the feasible region of β as \( {\beta}^{\prime}\in \left(0,{{\widehat{\beta}}^{\prime}}_{\mathrm{max}}\right] \), where \( {{\widehat{\beta}}^{\prime}}_{\mathrm{max}}=\left(-b+\sqrt{b^2-4 ac}\right)/2a \), a = P q r q r2r4 + r q r4,b = ωr4 + Λr q , and c =  − ωΛ.

Theorem 2: R is strictly quasi-concave in β for β ∈ [0, +∞).

Proof: See Appendix 2.

Similar to the previous analysis, from Theorem 2, there are only three cases for the curve R2 versus β in \( \left({{\widehat{\beta}}^{\prime}}_{\mathrm{min}},{\widehat{\beta}}_{\mathrm{max}}^{\prime}\right] \).

Case 1: R strictly increases with β for \( \left({{\widehat{\beta}}^{\prime}}_{\mathrm{min}},{\widehat{\beta}}_{\mathrm{max}}^{\prime}\right] \) if \( {\left. dR/d{\beta}^{\prime }\ \right|}_{\beta^{\prime }={\widehat{\beta}}_{\mathrm{max}}^{\prime }}\ge 0 \), where dR/ is given by (29) in Appendix 2. The solution to the optimization problem (25) is achieved at \( {\overset{\smile }{\beta}}^{\prime }={\widehat{\beta}}_{\mathrm{max}} \).

Case 2: R strictly decreases with β for \( \left({{\widehat{\beta}}^{\prime}}_{\mathrm{min}},{\widehat{\beta}}_{\mathrm{max}}^{\prime}\right] \) if \( {\left. dR/d{\beta}^{\prime }\ \right|}_{\beta^{\prime }={{\widehat{\beta}}^{\prime}}_{\mathrm{min}}}\le 0 \). The optimal solution is achieved at \( {\overset{\smile }{\beta}}^{\prime }={{\widehat{\beta}}^{\prime}}_{\mathrm{min}} \).

Case 3: R first strictly increases and then strictly decreases with β for\( \left({{\widehat{\beta}}^{\prime}}_{\mathrm{min}},{\widehat{\beta}}_{\mathrm{max}}^{\prime}\right] \) if \( {\left. dR/d{\beta}^{\prime }\ \right|}_{\beta^{\prime }={{\widehat{\beta}}^{\prime}}_{\mathrm{min}}}>0 \) and \( {\left. dR/d{\beta}^{\prime }\ \right|}_{\beta ={\widehat{\beta}}_{\mathrm{max}}^{\prime }}<0 \). The optimal solution is achieved at \( {\overset{\smile }{\beta}}^{\prime }={\beta^{\prime}}^{\ast } \), where β′∗ is the point at which R reaches its maximum when \( {\beta}^{\prime}\in \left({{\widehat{\beta}}^{\prime}}_{\mathrm{min}},{\widehat{\beta}}_{\mathrm{max}}^{\prime}\right] \) for a fixed β and is obtained by solving the equation dR/ = 0.

ODS: First of all, one can enumerate all values of β in the feasible region and obtain corresponding optimal solutions to the problem in (23). Next, by comparing all possible optimal solutions, finally, one can get the optimal solution to the problem in (23).

AOP: The optimization problems (24) and (25) can be alternately repeated by letting the output of one of the optimization problems be the input of the other. The specific procedure of the AOP algorithm is listed in Table 1.

Table 1 Iterative optimization algorithm AOP

4 Simulation results

The MATLAB tool is used to simulate the secondary throughput related to various system parameters and verify the effectiveness of the proposed opportunistic mode selection criterion. As other HD/FD relaying mode selection criterion is not seen in literature, the proposed opportunistic HD/FD relaying mode selection criterion is compared with the HD or FD mode to show its merit. The bandwidth W is normalized to 1, i.e., W = 1 Hz. The topology of the cognitive relay network is constructed like this: PT, PR, ST, and SR are collinear. Based on this, nodes can be rendered in the 2D plane and PT, PR, and SR are located at points (0, 0), (0, 1), and (1, 0), respectively. ST moves along the positive X axis between PT and SR. Other simulation parameters are set as pS, max = 0.1W, σ2 = 1, ω = 60dB, Γ = 10−6, Λ = 0.1W, v = 4, v1 = 0.5, d1 = d3 = 1, \( {d}_4=\sqrt{1+{d_2}^2} \), d5 = 1 − d2, d q  = 0.1, and ε0 = 10−3.

As shown in Fig. 2, we can observe the probability of opportunistic mode selecting HD versus the power of PTP P . In this simulation, the distance between PT and ST is set as d2 = 0.5. From (16), with the increase of P P , the residual self-interference power will increase. This means when the residual self-interference power is low, the performance of FD mode is better than that of the HD mode; thus, the probability of opportunistic mode selecting HD will be low. But with the increase of P P , the performance of FD mode will become worse and worse; thus, the probability of opportunistic mode selecting HD will be high.

Fig. 2
figure 2

Probability of opportunistic mode selecting HD versus power of PT P p

Figure 3 shows the effect of power of PT P P on the throughput of secondary system when d2 = 0.5. It is clear that, when the power of PT is low, from (16), we can know that the residual self-interference power at ST is also low. In this case, the throughput in the FD mode is nearly twice as much as the HD mode. Moreover, the opportunistic mode will select the FD mode basically and takes the throughput advantage of FD. But with the increase of P P , the residual self-interference power will become higher and higher and will seriously affect the performance of FD. About at P P  = 0.23W, the throughput of HD will become better than FD. At this point, the opportunistic mode selection criterion will select the HD mode which can achieve a higher throughput than FD. Due to the statistical average simulation of getting P q by (16), there is a small throughput gap between the opportunistic mode selection and either the FD or HD mode. Nevertheless, Fig. 3 clearly shows the advantage of the opportunistic mode selection in terms of selecting either the FD mode or the HD mode under different residual self-interference power regimes.

Fig. 3
figure 3

Secondary throughput R versus power of PT P p

Note that in [23, 29] the HD/FD mode switching was executed according to maximizing the channel capacity; thus, the opportunistic mode selection outperforms both the HD mode and the FD mode. But in this work, the secondary throughput in the opportunistic mode selection is composed of the throughput of HD and FD modes weighted by statistical average; thus, the secondary throughput in the opportunistic mode selection equals that either in the FD mode or in the HD mode, depending on the residual self-interference power.

To gain more insights, the throughput of primary system in the opportunistic mode selection is also simulated. Figure 4 shows the effect of power of PT P P on the throughput of primary system, where two received signals at PR are combined with maximal-ratio combining. In this simulation, all parameter settings are the same as those in Fig. 3. It is clear that, with the increase of P P , the throughput of primary system in all the three modes increases. In other words, the interference constraint of secondary system to the primary system is well satisfied. It is worth noting that, since there is a direct link in the primary system, the throughput of primary system increases with P P even in the FD mode where self-interference exists. However, as P P increases, self-interference in the relay link slows down the speed of increasing the throughput of primary system. From the line trends in Fig. 3, it can be conjectured that although the opportunistic mode selection is adopted at ST, the throughput of primary system in the opportunistic mode selection also can be equal to that in either the FD mode or the HD mode under different residual self-interference power regimes.

Fig. 4
figure 4

Primary throughput versus power of PT P p

Figure 5 shows the optimal throughput of secondary system R versus the distance d2. From this figure, we can see the optimal throughput of secondary system first increases and then decreases with d2. When d2 ≈ 0.5, which means ST is near the central point between PT and SR, the optimal secondary throughput R reaches its maximum. On the other hand, from the above figure, it is easy to know that optimization results by ODS and AOP methods are nearly the same, which indicates that the AOP method can attain a near-optimal solution.

Fig. 5
figure 5

Optimal secondary throughput R* versus distance d2

Figure 6 illustrates the convergence of the AOP algorithm in iteration when d2 = 0.5. It can be seen from Fig. 6 that the AOP algorithm converges only after five iterations, which indicates that its computational complexity is lower than that of the ODS method.

Fig. 6
figure 6

Optimal secondary throughput R* in iteration

To show the uniformity of the opportunistic mode selection and the AOP algorithm, the impact of the self-interference threshold Γon the probability of opportunistic mode selecting HD and the secondary throughput is further simulated.

The probability of opportunistic mode selecting HD versus the self-interference threshold Γis shown in Fig. 7. In this simulation, the transmission power of PT is set as P P  = 0.23W while other parameter settings remain unchanged. In this case, the residual self-interference power is a fixed value. From (13) and (16), we can know that with the increase of Γ, the probability of opportunistic mode selecting HD will decrease, as shown in Fig. 7. It also shows that derivations from (16) are valid.

Fig. 7
figure 7

Probability of opportunistic mode selecting HD versus self-interference threshold Γ

Figures 8 and 9 show the impact of self-interference threshold Γon the throughput of secondary system under settings of P P  = 0.15W and P P  = 0.35W, respectively. Combining Figs. 8 and 9 and (13), we can know that with the change of the self-interference threshold Γ, the secondary throughput in the opportunistic mode selection will also change. However, it still approaches either in the FD mode or in the HD mode, which further validates the opportunistic mode selection criterion. Moreover, combining Figs. 7, 8, and 9, when Γincreases, the probability of an opportunistic mode selecting FD will increase. However, the performance of FD is not superior to that of HD in all cases, as shown in Fig. 9. Therefore, choosing a proper self-interference threshold to switch between HD and FD modes can effectively enhance the throughput of secondary system. However, it is difficult to obtain the optimal self-interference threshold through solving the optimization problem in (23), as Γ is involved in exponential functions in (22).

Fig. 8
figure 8

Secondary throughput R versus self-interference threshold Γ when P p = 0.15W

Fig. 9
figure 9

Secondary throughput R versus self-interference threshold Γ when P p = 0.35W

5 Conclusions

In this paper, opportunistic HD/FD relaying mode selection in a cognitive relay network has been studied, which takes the residual self-interference power at the cooperative relay (secondary transmitter) as switching criterion. The problem of maximizing the throughput of secondary system under the interference constraint of primary system and transmission power constraints was formulated. Moreover, an alternate optimization method was introduced to solve this optimization problem, and optimum amplification gains in the HD mode and the FD mode were obtained. Numerical results illustrated that the proposed opportunistic mode selection criterion selects either the HD mode or the FD mode depending on the residual self-interference power, which flexibly utilizes respective advantages of HD and FD and achieves a higher throughput than either the FD mode or the HD mode under different residual self-interference power regimes. This result can be applied to help the adaptive transmission protocol design in cognitive relay networks.