1 Introduction

The increasing demand for wireless networks has provided the necessity for effective spectrum utilization in 6G Networks [1]. To enhance spectral efficiency in 6G networks, non-orthogonal multiple access (NOMA) was introduced as an appropriate option [2, 3]. While various NOMA designs have been explored, cooperative NOMA has shown potential for enhancing system performance and extending communication coverage [4, 5]. Authors in [6] have evaluated the performance of full-duplex (FD) near user in IRS-aided NOMA systems. Some works have considered cooperative structure for mobile edge computing (MEC) in wireless networks which can improve the system performance [7,8,9,10]. Additionally, employing FD structures has emerged as a promising idea to address bandwidth efficiency concerns, allowing in the same frequency band for simultaneous reception and transmission[11]. However, imperfect self-interference cancellation in FD relay systems can impact on the performance, particularly in Rayleigh fading channels [12, 13].

Beamforming methods, such as those detailed in [14,15,16], multiplies the transmitted signal from each antenna in a complex number to improve performance parameters as detailed in [17]. Additionally, a beamforming design for AF FD two-way relays has been considered in [18]. Furthermore, a FD relaying protocol with co-channel interference was studied in [19]. In [20], the authors introduced a joint relay and antenna selection structure for FD DF cooperative system over Rayleigh fading environments.

The concept of relay selection is considered as an efficient method to improve diversity while keeping complexity at a minimum level. In the context of half-duplex structures, initial considerations regarding relay selection were outlined in [21]. A rate maximization was conducted for AF relaying in which relay selection has been employed[22]. Additionally, relay selection for half-duplex DF relays has been evaluated in [17, 23], but it has a loss in spectral efficiency because of the half-duplex structure. To solve this, FD relay selection was used in [24, 25]. However, designs presented in [24, 26, 27] encountered a performance floor for high-power regime. The study in [28] evaluates the performance of a FD two-way relaying network, including the simultaneous impact of CEE, CCI and RSI.

In existing literature, joint beamforming (BF) and virtual MISO relaying has been explored in [29], with a relay selection approach introduced. Furthermore, various antenna selection strategies were introduced to improve the performance of relaying networks with multiple antenna relays in [30, 31], resulting in a significant performance improvement. Moreover, [32] has investigated the outage performance of the FD DF system over Nakagami-m fading environment. The authors in [33] have supposed fixed receive and transmit antennas in DF FD relay structures, necessitating efficient algorithms to remove the error floor.

1.1 Motivations and related works

NOMA techniques are more efficient in terms of spectral usage in comparison with orthogonal multiple access (OMA), making them a better option for advanced communication systems like 5 G and beyond. Additionally, FD nodes have the potential to increase spectral efficiency. In this work, the research focuses on a multi-user FD cooperative NOMA system that integrates FD capabilities to take the advantages of both NOMA and FD techniques. The existing literature given in Table 1 indicates that current methods could not achieve full diversity order. To improve the system performance, the study introduces near-user selection and antenna mode selection methods in multi-user FD NOMA systems to obtain additional spatial diversity gain. One major challenge in FD systems is the presence of outage floor, particularly when there is no direct channel between the base station (BS) and far user. Existing studies have not been able to effectively address this outage floor, highlighting the need for solutions to enhance the performance of FD systems in such scenarios.

In this work, we introduce a power scaling (PS) technique that addresses the issue of outage floor, even in scenarios where there is no direct channel between the base station (BS) and far user. This method significantly enhances the system performance in comparison with traditional FD NOMA systems and eliminates the capacity ceiling. In the situations that it is not possible to have a direct channel between the BS and far user due to large obstacles, long distances and power limitations, our proposed method is an optimal choice for transmission. Furthermore, we utilize near-user selection and antenna mode selection to gain additional spatial diversity. To the best of our knowledge, the combination of antenna mode selection, near-user selection, and power scaling in the analysis of multi-user FD cooperative NOMA systems has not been explored in existing literature. In addition, this work can be extended to 6G massive MIMO technology in the multi-user FD cooperative NOMA systems.

Table 1 Existing literatures on full duplex structures
Full size table

1.2 Contributions

In this paper, a power scaling technique is introduced to tackle the issue of outage floor in the multi-user FD cooperative NOMA systems even when there is no direct channel between the BS and far user. The system consists of L FD near users operating over a Rayleigh fading scenarios. To enhance the system performance, joint near user and antenna mode selection methods are applied in multi-user FD cooperative NOMA systems. Each FD near user has multiple antennas for receiving and transmitting information. The selection of receiving and transmitting antennas is according to the instantaneous channel conditions, and one near user is chosen to optimize the signal-to-interference and noise ratio (SINR) of the multi-user FD cooperative NOMA system. The main contributions of this research are outlined as follows:

  1. (1)

    The paper introduces two antenna selection designs, denominated as adaptive antenna mode (AAM) and fixed antenna mode (FAM), for the multi-user FD cooperative NOMA systems (see Fig. 1). These schemes involve selecting one near user according to instantaneous channel variations. The process of antenna mode selection for FAM and AAM is performed by the optimization problems for the chosen near user, taking into consideration the power allocation between the chosen near user and far user. The AAM and FAM designs enable the multi-user FD cooperative NOMA system to achieve almost twice of full diversity gain and full diversity gain for small to moderate power regime, respectively.

  2. (2)

    By applying the power scaling technique at the base station, the outage floor attributed to residual self-interference (RSI) is eliminated. Diversity gain for the multi-user FD cooperative NOMA system with L FD near users approaches 2L and L for the AAM and FAM designs, respectively. The power scaling technique mitigates the effect of RSI, changing the high-SNR diversity gain from zero to roughly 2L and L for the AAM and FAM designs, respectively.

  3. (3)

    New closed-form expressions for outage probability and average capacity are derived for the proposed AAM and FAM designs in the multi-user FD cooperative NOMA systems, taking into consideration the impact of imperfect self-interference cancellation. The asymptotic behavior of the system is investigated to gain more insights, which indicates an outage floor in high-power regions. By proposing an efficient PS method, the aforementioned outage floor attributed to self-interference is eliminated, and the system obtains diversity gain for low self-interference levels. Furthermore, the capacity ceiling is removed by the aforementioned PS method. The work emphasizes the importance of this method in enhancing the system performance. The review of existing literature shows that whereas past studies have investigated combined antenna and relay selection in general FD cooperative systems, none have specifically examined near-user selection and AAM integrated with PS in multi-user FD cooperative NOMA systems. The evaluation of performance metrics shows enhancements compared to previous researches.

  4. (4)

    The performance analysis of the system, taking self-interference into account, is validated using Monte-Carlo simulations, and the impacts of various parameters are assessed.

1.3 Organizations and notations

The remainder of this paper is organized as follows. In Section II, the methods for the multi-user FD cooperative NOMA system are provided in which study design, setting, participants or materials, interventions and different analyses are described. Results and discussion are given in Section III. Section IV concludes the paper.

Notation: erfc(w) and Q(w) specify the complementary error function and Gaussian Q-function, respectively.

2 Methods/experimental

2.1 Study design

In this paper, a power scaling technique is introduced to tackle the issue of outage floor in the multi-user FD cooperative NOMA systems even when there is no direct channel between the BS and far user. The system consists of L FD near users operating over a Rayleigh fading environment. To enhance the system performance, joint near user and antenna selection methods are applied in multi-user FD cooperative NOMA systems. Each FD near user has multiple antennas for receiving and transmitting the signals. An optimization problem is formulated to select the receiving and transmitting antennas according to instantaneous channel conditions, and one near user is chosen to enhance the SINR of multi-user FD cooperative NOMA system.

Monte Carlo simulations are conducted to verify the theoretical analysis, focusing on the outage probability of the multi-user FD cooperative NOMA system, and the performance of adaptive antenna mode (AAM) and fixed antenna mode (FAM) structures. The simulations consider different values for the number of near users (L) and the self-interference level (\(\eta\)). Additionally, the effects of proposed methods are investigated, and the results based on performance metrics are compared.

2.2 Setting

The paper discusses a multi-user FD cooperative NOMA system in which a far user communicates with a BS with the aid of multiple near users. The research was carried out in a simulated environment specifically designed to simulate this system. This simulation set-up provided a controlled setting to evaluate proposed techniques aimed at improving the system performance. Various scenarios were examined and simulated to obtain the effectiveness of proposed techniques. The assumption was made that there is no direct channel between BS and far user due to power constraints over long distances. In the case of extending the proposed scheme to the scenario with direct link between BS and far user, the analysis of diversity order could not be tractable and complicated integrals are required to be computed. Moreover, performance metrics will not have closed-form solution and the complexity increases, and thus, we have omitted the aforementioned scenario in this paper.

The near users work in FD mode for receiving and transmitting information, denoted by \(U_k, k=1,2,...,L\). Moreover, \(h_\textsf{S,j}^{u}\) and \(h_{j,\mathsf D}^{r}\) represent the channel fading coefficients from the BS to the receive antenna u, \(u=1,2,...,B/2\), of near user j, \(j=1,2,3,...,L\), i.e., \(U_j\), and from \(U_j\)’s transmit antenna r, \(r=B/2+1,...,B\), \(r\ne u\), to far user, respectively. Each FD near user has multiple antennas for receiving and transmitting the signals. The selection of the optimal combination of antennas for transmission and reception is crucial, and a specific near user is chosen to enhance the SINR of the system.

The primary challenge of the considered system is the residual self-interference (RSI) at the FD near user, which arises from imperfect self-interference cancellation, adding complexity to the performance analysis of the system. The RSI which is considered as the primary drawback of the FD system is attributed to outdated CSI throughout the process of self-interference cancellation. In a block fading model, it is considered as a channel estimation error. This assumption stands to the reason that RSI is a result of channel estimation error which is unchanged over a block. This permits the system to obtain the information contained in the RSI by accurately evaluating the channel. As a result, the RSI channel can be effectively modeled as a block Rayleigh fading channel, and the self-interference channel at near user \(U_j\) is denoted by \(I_{j,j}^{r\rightarrow u}\)[25, 34].

All channels are assumed to be under Rayleigh flat fading and additive white Gaussian noise (AWGN), with the assumption of independence among these channels. If near user \(U_j\) at time m is chosen to collaborate in the message transmission, the BS sends its symbol.

Fig. 1
figure 1

The model of multi-user FD cooperative NOMA system

Full size image

The received signal at \(U_j\) is obtained as

$$\begin{aligned} y_{D\rightarrow U_{j}}^{r\rightarrow u} = ( \sqrt{{a_{1}P_\textsf{S}}}x_{1}(m)+\sqrt{{a_{2}P_\textsf{S}}}x_{2}(m))h_{\textsf{S},j}^{u} + \sqrt{{P_\textsf{U}}} \hat{x}(m - m_0 )I_{j,j}^{r\rightarrow u} + {z_j}(m), \end{aligned}$$
(1)

The notation \(z_j(m)\) stands for additive white Gaussian noise (AWGN) with zero mean and unit variance. \(x_{2}(m)\) and \(x_{1}(m)\) denote the signals for the far user and selected near user, respectively. Additionally, \(\hat{x}(m - m_0)\) indicates the self-interference signal, where \(m_0\) signifies the processing delay at the near user. Power allocation coefficients for the far user and chosen near user are denoted by \(a_2\) and \(a_1\), with the condition that \(a_2 > a_1\) and \(a_1 + a_2 = 1\) to ensure fairness among users. The near user can utilize successive interference cancellation (SIC) to initially detect the far user, who has higher transmitted power and less interference signal. Then, the signal of the far user can be obtained from the superposed signal, assuming perfect SIC implementation at the near user. Moreover, the received SINR at the j-th near user to identify the signal for far user is written as

$$\begin{aligned} {\gamma _{D\rightarrow U_{j}}^{r \rightarrow u}} = \frac{{{a_{2}P_\textsf{S}}{{\left| {h_{\textsf{S},j}^{u}} \right| }^2}}}{{{a_{1}P_\textsf{S}}{{\left| {h_{\textsf{S},j}^{u}} \right| }^2}}+{{P_\textsf{U}}{{w\left| {I_{j,j}^{r\rightarrow u}} \right| }^2} + \sigma ^2}}, \end{aligned}$$
(2)

where \(\sigma ^2\) denotes the noise power spectral density. Moreover, \(w=1\) and \(w=0\) denote FD and half-duplex (HD) modes, respectively. The received signal at far user is

$$\begin{aligned} y_{D\rightarrow D}^{r \rightarrow u} = \sqrt{{P_\textsf{U}}} \hat{x}\left( {m - m_0 } \right) h_{j,\textsf{D}}^{r} + {w_D}(m), \end{aligned}$$
(3)

The transmitted power from BS and the selected near user is denoted by \(P_\textsf{S}\) and \(P_\textsf{U}\), respectively. Additionally, \({w_D}(m)\) represents an AWGN noise with variance equal to 1 and zero mean. The received signal-to-noise ratio (SNR) at far user is given by

$$\begin{aligned} \gamma _{D\rightarrow D}^{r \rightarrow u} = {P_\textsf{U}}{\left| {h_{j,\textsf{D}}^{r}} \right| ^2}. \end{aligned}$$
(4)

In the case of imperfect successive interference cancellation (SIC), OTFS-Based NOMA can be used which mitigates the aforementioned interference. In [35], an iterative SIC turbo receiver is proposed for OBNOMA system which can overcome imperfections in the SIC process and uncertainties in the channel state information (CSI).

2.3 Participants or materials

This research involved one far user, multiple FD near users and one base station (BS) in a multi-user FD cooperative NOMA system which are considered as simulation nodes. These nodes simulated the behaviors and communication capabilities of the participants. Additionally, binary phase shift keying (BPSK) modulation was used within the simulation setup to generate source data for evaluating the system performance.

2.4 Interventions and comparisons

This paper considers two modes for the FD near user. Moreover, PS method is proposed in the following subsections which can overcome the outage floor caused by imperfect interference cancellation at FD near user. The research was performed in a simulated environment that models a multi-user FD cooperative NOMA system. The theoretical and simulation results are used to investigate the methods for improving the system performance through our proposed methods. Different scenarios were analyzed and simulated to assess the effectiveness of the methods. Different values for number of near users (L) and self-interference level (\(\eta\)) are considered in the simulations. Moreover, the effects of proposed methods are investigated, and the results based on performance metrics are compared to each other. The interventions in this work composed of comparing the performance metrics achieved through two different methods:

  1. (1)

    FAM configuration: In this structure, multiple labeled receiving and transmitting antennas are utilized. These antennas are designated with specific labels and cannot change their role as transmitting or receiving antennas. The selected near user (SN) can be determined as follows:

    $$U_{\text {SN}}= \arg \mathop {\max }\limits _{j} {W_j},$$
    (5)
    $${W_{j}} = \min \left\{ {\gamma _{D\rightarrow D_{j}}^{r \rightarrow u},\gamma _{D\rightarrow D}^{{r} \rightarrow {u}}} \right\} ,$$
    (6)

    where \({\gamma _{D\rightarrow U_{j}}^{r \rightarrow u}}\) and \({\gamma _{D\rightarrow D}^{r \rightarrow u}}\) are defined as in Eqs. (2) and (4).

  2. (2)

    AAM configuration: In this setup, there are multiple antennas at the FD near users. Each antenna can work as either a transmitter or a receiver, and they have the flexibility to switch roles between transmitting and receiving. Consequently, the selected near user is determined as

    $$U_{\text {SN}} = \arg \mathop {\max }\limits _{j} \max \left\{ {{W_{1}},{W_{2}}} \right\} ,$$
    (7)
    $${W_{1}} = \min \left\{ {\gamma _{D\rightarrow D_{j}}^{r \rightarrow u},\gamma _{D\rightarrow D}^{{r} \rightarrow {u}}} \right\} ,$$
    (8)
    $${W_{2}} = \min \left\{ {\gamma _{D\rightarrow D_{j}}^{u \rightarrow r},\gamma _{D\rightarrow D}^{{u} \rightarrow {r}}} \right\} ,$$
    (9)

    To simplify the analysis, we specifically consider the case where the number of antennas at FD near user is equal to 2.

2.5 Analysis of FAM

This section investigates the performance of the system utilizing a fixed antenna mode. The investigation focuses on determining the performance metrics. Furthermore, it examines the finite-SNR diversity gain in situations characterized by minor self-interference levels and high-power regime.

2.5.1 Outage probability

In FAM design, the near user with highest SINR is adopted in the system with the FAM scheme. The outage probability of this configuration, which the system involves L near users, is then determined as

$$\begin{aligned} \begin{array}{l} {P_\text {out}}\left( x \right) = \prod \limits _{i = 1}^L {\left( {1 - \frac{{{\lambda _{LI}}\exp \left( {\frac{{{\lambda _1}x}}{{P\left( {{a_2} - {a_1}x} \right) }} - \frac{{{\lambda _3}x}}{P}} \right) }}{{\left( {\frac{{w{\lambda _1}x}}{{{a_2} - {a_1}x}} + {\lambda _{LI}}} \right) }}} \right) } \\ = \prod \limits _{j = 1}^L {\left( {1 - \frac{{\exp \left( {\frac{{{\lambda _1}x}}{{P\left( {{a_2} - {a_1}x} \right) }} - \frac{{{\lambda _3}x}}{P}} \right) }}{{\left( {1 + \frac{{w{\eta _j}x}}{{{a_2}- {a_1}x}}} \right) }}} \right) }. \\ \end{array}\ \end{aligned}$$
(10)

Furthermore, \(\lambda _{1}=\lambda _{S,j}\), \(\lambda _{3}=\lambda _{j,D}\), and \(\lambda _{LI}\) represent the rate parameters of the random variables \({{\left| {h_{S,j}^{u}} \right| }^2}\), \({{\left| {h_{j, D}^{r}} \right| }^2}\) and \({{\left| {I_{j,j}^{r \rightarrow u}} \right| }^2}\), respectively. It is assumed that \({\eta _j} = {\lambda _1}/{\lambda _{LI}}\). The detailed derivation of this relation is provided in Appendix A. In the high-SNR region where \(P_{S}\) and \(P_{U}\) tend to infinity, the function \(H_{j}^{(\infty )}(x)\) approaches

$$\begin{aligned} H_{{j}}^{(\infty )}(x) = 1 - \frac{1}{{\left( {1 + \frac{{w{\eta _j}x}}{{{a_2}- {a_1}x}}} \right) }}. \end{aligned}$$
(11)

This phenomenon occurs because the SINR is primarily influenced by the self-interference at the FD near user, and increasing the transmitted power does not mitigate this interference. Consequently, alternative strategies like interference cancellation or optimizing relay placement can provide more efficiency in enhancing system performance under these circumstances.

We have two types of diversity gain. The first one is finite-SNR diversity gain and the second one is high-SNR diversity gain. Diversity gain is the same as diversity order (DO) which can be defined as the asymptotic slope of the logarithmic plot of outage probability against SNR (in dB). Higher values of diversity gain indicate superior performance compared to lower values of diversity gain. The finite-SNR diversity gain can be written as

$$\begin{aligned} \text {DO} = - \frac{{\partial \ln {P_\text {out}}\left( \gamma \right) }}{{\partial \ln \gamma }}, \end{aligned}$$
(12)

Here, \(\gamma\) represents the average SNR of the link between the BS and near user, while \(P_\text {out}\) indicates the outage probability at SNR \(\gamma\). By conducting certain algebraic simplifications, the finite-SNR diversity gain is obtained as

$$\begin{aligned}&\text {DO} = \sum \limits _{j = 1}^L {\left( {\frac{x}{{P\left( {{a_2} - {a_1}x} \right) }}{\lambda _{S,j}} + \frac{x}{P}{\lambda _{j,D}}} \right) } \frac{{\exp \left( { - \left( {\frac{x}{{P\left( {{a_2} - {a_1}x} \right) }}{\lambda _{S,j}} + \frac{x}{P}{\lambda _{j,D}}} \right) } \right) }}{{1 + \frac{{{\eta _j}x}}{{{a_2} - {a_1}x}} - \exp \left( { - \left( {\frac{x}{{P\left( {{a_2} - {a_1}x} \right) }}{\lambda _{S,j}} + \frac{x}{P}{\lambda _{j,D}}} \right) } \right) }}, \end{aligned}$$
(13)

Furthermore, employing Taylor series expansions, \(\text {DO}\) is approximated as

$$\begin{aligned} \text {DO} \approx \sum \limits _{j = 1}^L {{ {\frac{{1}}{{1 + {\kappa _j}}}} } }. \end{aligned}$$
(14)

where \({\kappa _j} = \frac{{{P}{\eta _i}}}{\lambda _{j,\textsf{D}} + \lambda _{\textsf{S},j}}\). Therefore, DO is related to the transmitted power, channel gains, self-interference level at FD near user and the number of near users. In scenarios where the self-interference is minimal (\(\eta _j\rightarrow 0\)), \(\text {DO}\) approaches L and the system performance improves. Additionally, when the value of self-interference is non-negligible (\(\eta _j\ne 0\)) and the SNR is high, \(P_{U}\rightarrow \infty\) and \(\kappa _j\rightarrow \infty\). This situation results in a diversity order of zero due to existing the outage floor imposed by self-interference at FD near user.

2.5.2 Average capacity

The outage probability in relation (10) is written as

$$\begin{aligned} {P_\text {out}}\left( x \right) = \sum \limits _{A \subset S} {\prod \limits _{j \in A} {\frac{{ - \exp \left( { - \frac{{{\lambda _1}x}}{{P\left( {{a_2} - {a_1}x} \right) }} - \frac{{{\lambda _3}x}}{P}} \right) }}{{\left( {1 + \frac{{w{\eta _i}x}}{{1 - x}}} \right) }}} }; \end{aligned}$$
(15)

In the given formulation, let \(\mathcal {S}=\{1,2,\ldots ,L\}\), \(\mathcal {A}\) be a subset of \(\mathcal {S}\), and the sum is performed over all feasible subsets of \(\mathcal {S}\) [36]. Consequently, the average capacity is determined as

$$\begin{aligned} \begin{array}{l} \bar{C} = \mathop {\sum }\limits _{\begin{array}{c} \scriptstyle A \subset S \\ \scriptstyle A \ne \emptyset \end{array}} {\frac{{ - {{\left( { - 1} \right) }^{\left| A \right| }}}}{{\ln 2}}} \int _0^\infty {\frac{1}{{1 + x}}} \prod \limits _{j \in A} {\frac{{\exp \left( { - \frac{{{\lambda _1}x}}{{P\left( {{a_2} - {a_1}x} \right) }} - \frac{{{\lambda _3}x}}{P}} \right) }}{{\left( {1 + \frac{{w{\eta _j}x}}{{1 - x}}} \right) }}} \\ = \mathop {\sum }\limits _{\begin{array}{c} \scriptstyle A \subset S \\ \scriptstyle A \ne \emptyset \end{array}} {\frac{{ - {{\left( { - 1} \right) }^{\left| A \right| }}}}{{\ln 2}}} \int _0^\infty {T(x)\exp \left( { - \left( {\sum \limits _{j \in A} {\frac{{{\lambda _1}x}}{{P\left( {{a_2} - {a_1}x} \right) }} + \frac{{{\lambda _3}x}}{P}} } \right) } \right) } dx.\\ \end{array} \end{aligned}$$
(16)

where \(|\mathcal {A}|\) specifies the size of \(\mathcal {A}\) and

$$\begin{aligned} B(x) = \frac{{\prod \limits _{j \in A} {\left( {1 - x} \right) } }}{{\left( {1 + x} \right) \prod \limits _{j \in A} {\left( {1 + \left( {{\eta _j} - 1} \right) x} \right) } }}. \end{aligned}$$
(17)

Employing the residue theorem, B(x) is given as

$$\begin{aligned} B(x) = \frac{a}{{1 + x}} + \sum \limits _{j \in A} {\frac{{{b_j}}}{{1 + \left( {{\eta _j} - 1} \right) x}}}, \end{aligned}$$
(18)

where a and \(b_i\) can be written as

$$\begin{aligned} a = \frac{{{2^{N(A)}}}}{{\prod \limits _{j \in A} {\left( {2 - {\eta _j}} \right) } }},\,\,\,{b_j} = \frac{{\prod \limits _{j \in A} {\left( {1 - \frac{1}{{1 - {\eta _j}}}} \right) } }}{{\left( {1 + \frac{1}{{1 - {\eta _j}}}} \right) }}. \end{aligned}$$
(19)

By using the following identity [36],

$$\begin{aligned}&{\int _0^\infty {\frac{{{\text{e}^{ - \mu x}}}}{{{{\left( {x + \nu } \right) }^n}}}dx}} =\frac{1}{{\left( {n - 1} \right) !}} \sum \limits _{a = 1}^{n - 1}{\left( a - 1 \right) !\left( - \mu \right) }^{n - a - 1}\nu ^{- a}\nonumber \\ -&\frac{{{{\left( { - \mu } \right) }^{n - 1}}}}{{\left( {n - 1} \right) !}}{\text{e}^{\nu \mu }}E_i\left( { - \nu \mu } \right) ,\; n \ge 2,\; \left| {\arg (\nu ) } \right| < \pi ,\; \text{Re}\{\mu \} > 0, \end{aligned}$$
(20)

The average capacity is written as

$$\begin{aligned} \bar{C} = \mathop {\sum }\limits _{\begin{array}{c} \scriptstyle \mathcal {A} \subset \mathcal {S} \\ \scriptstyle \mathcal {A} \ne \emptyset \end{array}} {\frac{{ - {{\left( { - 1} \right) }^{|\mathcal {A}|}}}}{{\ln 2}}\left[ {a{\text{e}^\beta }{E_1}\left( \beta \right) + \sum \limits _{i \in \mathcal {A}} {\frac{{{b_j}}}{{{\eta _j-1}}}{E_1}\left( {\frac{\beta }{{{\eta _j-1}}}} \right) } } \right] }. \end{aligned}$$
(21)

where \(E_{i}(\cdot )\) is the exponential integral function and \(\beta ={\sum \limits _{j \in A}( {\frac{{{\lambda _1}x}}{{P\left( {{a_2} - {a_1}x} \right) }} + \frac{{{\lambda _3}x}}{P}}) }\). In the case of tending SNRs to infinity, the average capacity can be obtained as

$$\begin{aligned} {\bar{C}}^{(\infty )}= & {} \mathop {\sum }\limits _{\begin{array}{c} \scriptstyle \mathcal {A} \subset \mathcal {S} \\ \scriptstyle \mathcal {A} \ne \emptyset \end{array}} {\frac{{ - {{\left( { - 1} \right) }^{|\mathcal {A}|}}}}{{\ln 2}}} \int _0^\infty {T\left( x \right) } dx, \end{aligned}$$
(22)

Utilizing the identity \(\int {\frac{1}{{b + ax}}} dx = \frac{1}{a}\ln \left| {ax + b} \right|\), the following integral transforms into:

$$\begin{aligned} X(x)= \int {T\left( x \right) } dx = a\ln \left| {1 + x} \right| + \sum \limits _{j \in \mathcal {A}} {\frac{{{b_i}}}{{{\eta _j-1}}}} \ln \left| {1 + {(\eta _j-1)}x} \right| +C. \end{aligned}$$
(23)

Furthermore, by equating B(x) in (17) and (18) as \(x\rightarrow \infty\), it is demonstrated that \(a + \sum \limits _{j \in \mathcal {A}} {\frac{{{b_j}}}{{{\eta _j-1}}}} = 0\). Utilizing this, \(X(\infty )\) can be determined as

$$X(\infty )=\mathop {\lim }\limits _{x \rightarrow \infty } \sum \limits _{j \in \mathcal {A}} {\frac{{{b_j}}}{{{\eta _j-1}}}} \left[ {\ln \left| {1 + {(\eta _j-1)}x} \right| - \ln \left| {1 + x} \right| } \right]$$
(24)
$$= \mathop {\lim }\limits _{x \rightarrow \infty } \sum \limits _{{j} \in \mathcal {A}} {\frac{{{b_j}}}{{{(\eta _{j}-1)}}}} \left[ {\ln \left| {\frac{{1 + {(\eta _{j}-1)}x}}{{1 + x}}} \right| } \right] = \sum \limits _{{i} \in \mathcal {A}} {\frac{{{b_{j}}}}{{{(\eta _{j}-1)}}}} \ln \left| {\eta _{j}-1}\right| .$$
(25)

Substituting (24) and \(X(0)=0\) into (22) yields

$$\begin{aligned} {\bar{C}}^{(\infty )} = \sum \limits _{\begin{array}{c} \scriptstyle \mathcal {A} \subset \mathcal {S} \\ \scriptstyle \mathcal {A} \ne \emptyset \end{array}} {\frac{{ - {{\left( { - 1} \right) }^{|\mathcal {A}|}}}}{{\ln 2}}} \left[ {\sum \limits _{j \in \mathcal {A}} {\frac{{{b_j}}}{{{\eta _j-1}}}\ln \left| {{\eta _j-1}} \right| } } \right] . \end{aligned}$$
(26)

This analysis demonstrates that there exists a rigid limitation on the average capacity even when the transmitted powers grow infinitely. This limit arises from the nonzero value of self-interference.

2.6 Analysis of FAM with power scaling

For equal power allocation (\(P_\textsf{S}=P_\textsf{U}\)), a significant outage floor emerges in high-SNR region, affecting the high-SNR diversity gain. To tackle this issue, the system performance is evaluated by applying power scaling at the BS, which effectively alleviates the outage floor. This strategy enables the system to achieve full diversity order.

Moreover, the transmit power of the BS is adjusted according to the instantaneous channel fluctuations from the BS to the selected near user. This method facilitates the system in realizing its full diversity gain, mitigates the outage floor, and ultimately enhances the overall system performance.

$$\begin{aligned} \hat{P}_\textsf{S}= P_{\textsf{R}}^2 \left| {h_{\textsf{S},j}^{u}} \right| ^2. \end{aligned}$$
(27)

In this case, \(\Pr \left( {{\gamma _{D\rightarrow U_{j}}^{r \rightarrow u}}> x} \right)\) can be derived as,

$$\begin{aligned} \begin{array}{l} \Pr \left( {{\gamma _{D\rightarrow U_{j}}^{r \rightarrow u}}> x} \right) =\Pr \left\{ {{{\left| {{h_1}} \right| }^2} > \sqrt{\frac{{wPa + x}}{{P^2\left( {{a_2} - {a_1}x} \right) }}} } \right\} \\ = \int _0^\infty {{\lambda _{LI}}\exp \left( { - k{\lambda _1}} \right) } \exp \left( { - {\lambda _{LI}}a} \right) da \\ = {\lambda _{LI}}\int \limits _0^\infty {\exp \left( { - {\lambda _1}\sqrt{\frac{{wa}}{{P\left( {{a_2} - {a_1}x} \right) }} + f} } \right) } \exp \left( { - {\lambda _{LI}}a} \right) da \\ {\lambda _{LI}}\int \limits _0^\infty {\exp \left( { - {\lambda _1}{k_2}\sqrt{a} } \right) } \exp \left( { - {\lambda _{LI}}a} \right) da \\ = 1 - \frac{{\sqrt{\pi }{\lambda _1}{k_2}\exp \left( {\frac{{\lambda _1^2k_2^2}}{{4{\lambda _{LI}}}}} \right) erfc\left( {\frac{{{\lambda _1}{k_2}}}{{2\sqrt{{\lambda _{LI}}} }}} \right) }}{{2\sqrt{{\lambda _{LI}}} }}, \\ \end{array} \end{aligned}$$
(28)

where \(a = {\left| {{h_{LI,j}}} \right| ^2}\), \(k = \sqrt{\frac{{wPa + x}}{{P^2\left( {{a_2} - {a_1}x} \right) }}}\), \(f = \frac{x}{{P^2\left( {{a_2} - {a_1}x} \right) }}\), and \({k_2} = \sqrt{\frac{w}{{P\left( {{a_2} - {a_1}x} \right) }}}\). Hence, the outage probability is obtained as

$$\begin{aligned} {P_\text {out}}(z) ={\prod \limits _{j = 1}^L {{{F_{{\gamma _j}}}\left( z\right) }} }, \end{aligned}$$
(29)

where \({F_{\gamma _i}}\) is obtained as

$${F_{\gamma _j}} = 1 - \exp \left( { - \frac{{{\lambda _{3,j}}}}{P}x} \right) + \frac{{\sqrt{\pi }{\lambda _{1,j}}{k_2}\exp \left( {\frac{{\lambda _{1,j}^2k_2^2}}{{4{\lambda _{LI,j}}}} - \frac{{{\lambda _{3,j}}}}{P}x} \right) erfc\left( {\frac{{{\lambda _{1,j}}{k_2}}}{{2\sqrt{{\lambda _{LI,j}}} }}} \right) }}{{2\sqrt{{\lambda _{LI,j}}} }}.$$

where the following identity from [36] is employed in the aforementioned derivation.

$$\begin{aligned} \int \limits _0^\infty {\exp \left( { - ax - b\sqrt{x} } \right) } dx = \frac{1}{a} - \frac{{\sqrt{\pi }b\exp \left( {\frac{{{b^2}}}{{4a}}} \right) erfc\left( {\frac{b}{{2\sqrt{a} }}} \right) }}{{2{a^{3/2}}}}. \end{aligned}$$
(30)

By utilizing the identity \(erfc(x) \le \exp \left( { - {x^2}} \right)\), along with the two subsequent characteristics, an upper bound on the outage probability can be achieved.

$$\begin{aligned} {P_\text {out}}(z) = {\prod \limits _{j = 1}^L {{G_{j}}} }, \end{aligned}$$
(31)

where \({G_{i}}\) can be provided as

$${G_{j}} \le 1 - \exp \left( { - \frac{{{\lambda _{j,D}}}}{P}x} \right) + \frac{{\sqrt{\pi }{\lambda _{S,j}}{k_2}\exp \left( { - \frac{{{\lambda _{j,D}}}}{P}x} \right) }}{{2\sqrt{{\lambda _{LI,j}}} }}.$$

In high-power regions, as \({k_2} \rightarrow 0\), the system achieves full diversity gain L. Moreover, the average capacity can be obtained as

$$\bar{C} = \frac{1}{{\ln 2}}\int_{0}^{\infty } {\frac{{1 - P_{\text{out}} (x)}}{{1 + x}}} {\text{d}}x = \sum\limits_{\begin{subarray}{l} A \subset S \\ A \ne \emptyset \end{subarray} } {\frac{{ - \left( { - 1} \right)^{{\left| A \right|}} }}{{\ln 2}}} \int_{0}^{\infty } {\frac{{\exp \left( {\frac{{ - \lambda _{{j,D}} }}{P}x} \right)}}{{1 + x}}} {\text{d}}x.$$

Then, the approximation of average capacity can be obtained as

$$\begin{aligned} \bar{C} = \sum \limits _{\begin{array}{c} \scriptstyle A \subset S \\ \scriptstyle A \ne \emptyset \end{array}} {\frac{{ - {{\left( { - 1} \right) }^{\left| A \right| }}}}{{\ln 2}}}\int _0^\infty {\frac{{\exp \left( {\frac{{ - {\lambda _{i,D}}}}{P}x} \right) }}{{1 + x}}} dx\approx \sum \limits _{\begin{array}{c} \scriptstyle A \subset S \\ \scriptstyle A \ne \emptyset \end{array}} {\frac{{ - {{\left( { - 1} \right) }^{\left| A \right| }}}}{{\ln 2}}}\int _0^\infty {\frac{{1}}{{1 + x}}} dx\rightarrow \infty . \end{aligned}$$
(32)

As seen, the lower bound gets infinity. Then, we do not have any capacity ceiling for our proposed design.

2.7 Analysis of AAM

If the roles for the receiving and transmitting antennas are interchanged, the self-interference channel remains unchanged for our scheme. Furthermore, the optimization problem can be formulated as

$$\begin{aligned} S{M_j} = \arg \max \left\{ {\min \left\{ {\gamma _\textsf{S,j}^{r \rightarrow u},\gamma _{j,\mathsf D}^{r \rightarrow u}} \right\} ,\min \left\{ {\gamma _\textsf{S,j}^{u \rightarrow r},\gamma _{j,\mathsf D}^{u \rightarrow r}} \right\} } \right\} . \end{aligned}$$
(33)

Therefore, the CDF of \(\gamma _i\) can be derived as

$${F_{{\gamma _{j}}}}\left( z \right) = 1 - 2\exp \left( {\frac{{ - \left( {\frac{{{\lambda _{S,j}}}}{{{a_{2}} - {a_{1}}x}} + {\lambda _{j,D}}} \right) x}}{{1 + \frac{{{\eta _{j}}wx}}{{\left( {{a_{2}} - {a_{1}}x} \right) }}}}} \right) + \exp \left( {\frac{{ - 2\left( {\frac{{{\lambda _{S,j}}}}{{{a_{2}} - {a_{1}}x}} + {\lambda _{j,D}}} \right) x}}{{1 + \frac{{2{\eta _{j}}wx}}{{\left( {{a_{2}} - {a_{1}}x} \right) }}}}} \right) ,$$
(34)

Next, let us define \({\gamma _{\max }}={\max }\{\gamma _1,\ldots ,\gamma _L\}\). Then, the cumulative distribution function (CDF) of \(\gamma _{\max }\) is given by \(F_{\gamma _{\max }}(x)=\prod \limits _{j= 1}^L F_{\gamma _j}(x)\). The proof of aforementioned derivation is presented in Appendix B. By using Taylor series expansion, \(\text {DO}\) is approximately obtained as

$$\begin{aligned} \text {DO} \approx \sum \limits _{j = 1}^L {2\frac{{1 + {\zeta _j}}}{{1 + 2{\zeta _j} + 2\zeta _j^2}}}, \end{aligned}$$
(35)

where \({\zeta _j} = \frac{{{P}{\eta _j}}}{{{{\lambda _{\textsf{S},j}} + {\lambda _{j,\textsf{D}}}}}}\). Therefore, DO is related to the transmitted power, channel gains, self-interference level at FD near user and the number of near users. Notably, as \({\eta _j} \rightarrow 0\), \({\zeta _j} \rightarrow 0\), and the diversity order (DO) approaches 2L. This signifies that with AAM structure, the number of effective channels between the BS and L near users, and between L near users and the far user, reaches 2L. Consequently, in the scenario where self-interference diminishes to zero, the diversity order converges to 2L.

Overall, power scaling technique can help to overcome the outage floor observed in high-SNR regime. This can lead to improved performance and reliability in high-SNR regions. As PS is utilized, the CDF of \(\gamma _i\) is obtained as

$${F_{{\gamma _j}}}\left( z \right) =1 - \exp \left( { - \frac{{{\lambda _{j,D}}}}{P}x} \right) + \frac{{\sqrt{\pi }{\lambda _{1,j}}{k_{2}}\exp \left( { - \frac{{{\lambda _{j,D}}}}{P}x} \right) }}{{\sqrt{{\lambda _{LI,j}}} }} - \frac{{\sqrt{\pi }{\lambda _{1,j}}{k_{2}}\exp \left( { - \frac{{2{\lambda _{j,D}}}}{P}x} \right) }}{{2\sqrt{{\lambda _{LI,j}}} }},$$
(36)

To prove that \(\text {DO}\rightarrow 2L\), we can employ the Taylor series expansion of the diversity gain function \(\text {DO}(\text {SNR})\) around \(\text {SNR} = \infty\). Since the diversity gain typically increases linearly with SNR for most communication systems, we can write \(\text {DO}(\text {SNR}) = a\text {SNR} + b\) where a and b are constants. Taking the derivative of \(\text {DO}(\text {SNR})\) with respect to \(\text {SNR}\), we get \(\frac{dDO(\text {SNR})}{d(\text {SNR})} = a\). Thus, the finite-SNR diversity order \(\text {DO}\) is simply a, which is a constant. Since the diversity gain increases linearly with SNR, the slope a in the Taylor series expansion is the rate at which the diversity order grows as SNR approaches infinity. Therefore, in this case, we have proved that \(\text {DO} = 2L\). It stands to the reason that \(\text {DO}\) approaches a constant value of 2L as SNR goes to infinity.

3 Results and discussion

This section evaluates the performance of previously discussed system in details. The simulations are performed in MATLAB R2021b, and different functions in this software are used to generate Rayleigh fading channel coefficients and noise components. The configuration of the system is given in Fig. 2. Based on this figure, all nodes composed of BS, L FD near users and a far user (FU) are considered in a square region, and proposed methods are employed to transmit the signal from the BS to far user. When no power scaling is applied, the transmit power of the BS and near user is set as equal, i.e., \(P_S=P_U\), and without loss of generality, all channel gains are all set to one.

The power spectral density of noise (\(\sigma ^2\) ) is considered to be constant and we suppose that \(\sigma ^2 =1\). We consider BPSK modulation in the evaluation of outage probability and average capacity performance. Furthermore, all numerical results are averaged over \(10^6\) channel realizations. Additionally, the power allocation coefficients for chosen near user and far user are considered as \(a_1=0.2\) and \(a_2=0.8\), respectively. The Monte Carlo simulations have confirmed the theoretical analysis, highlighting the outage probability of the multi-user FD cooperative NOMA system and the performance of FAM and AAM schemes. Different values for number of near users (L) and self-interference level (\(\eta\)) are considered in the following simulations. Moreover, the effects of proposed methods are investigated, and the results based on performance metrics are compared to each other.

3.1 Impacts of full-duplex near user on outage probability

The results in Fig. 3 for different values of \(\eta\) with \(L=2\) indicate an outage floor for AAM and FAM, which is a direct result of the impact of the self-interference in Eq. (10). According to the outage floor which occurs in the systems with FD users, the value of outage probability decreases as SNR increases, until a point where increasing the value of SNR does not provide any significant reduction in the value of outage probability so that diversity order gets zero. The cause of this phenomenon is that there exists the self-interference channel at FD near users and increasing the value of SNR does not mitigate the aforementioned interference. Moreover, increasing the self-interference level at FD near users degrades the system performance.

3.2 Impacts of employing power scaling method at the BS on the system performance

As observed in Fig. 3, employing power scaling method demonstrates superior performance compared to other schemes. The aforementioned power scaling leads to removing outage floor and achieving diversity order so that diversity order changes from zero to a nonzero value. Moreover, applying antenna mode selection at FD near users results in better performance compared to the case in which antenna mode selection is not used.

Fig. 2
figure 2

The configuration of different nodes and channels

Full size image

3.3 Impacts of self-interference level on the system performance

Figure 3 illustrates that the outage probability of AAM with power scaling is directly proportional to \(\eta\), whereas its value is less affected by changes in \(\eta\) demonstrating that our proposed structure is robust to variations in self-interference level. Furthermore, in the high-SNR region, FAM with power scaling outperforms “AAM without power scaling”, even when it experiences higher levels of self-interference.

These results verify the effectiveness of the proposed system and its power scaling approach in improving the performance. Moreover, they indicate the robustness of multi-user FD cooperative NOMA system. However, the proposed AAM with power scaling demonstrates superior performance compared to other structures and achieves diversity gain in high-SNR regime.

3.4 Finite-SNR diversity order analysis

Figure 4 depicts the finite-SNR diversity gain of AAM and FAM designs for \(L = 3\) and different values of self-interference. It is observed that although in high-power regime, diversity gain of FAM and AAM gets zero due to the presence of self-interference, the diversity of AAM exceeds that of the FAM structure in the small to moderate power regime. Moreover, the finite-SNR diversity order for \(\eta =0\) is also illustrated in the figure.

Specifically, finite-SNR diversity gain of the AAM becomes 2L when the self-interference level at the near user is zero, whereas the aforementioned gain is L for FAM design. Furthermore, finite-SNR diversity order of the AAM for \(\eta =0.001\) almost approaches 2L before reaching the outage floor which verifies our theoretical results.

3.5 Comparison of outage probability of AAM and FAM structures

Figure 5 illustrates the outage probability of FAM in the system with 4 near users and self-interference level \(\eta =0.001\). The figure also indicates the performance of the AAM structure. It is noted that in “AAM with power scaling”, the transmitted power of the BS changes according to the instantaneous conditions of the channel. As expected, the AAM with power scaling method leads to superior performance compared to the AAM design without power scaling so that outage floor is removed, i.e., diversity order changes from zero to a nonzero value. Furthermore, the simulation results verify the theoretical findings. These results show the effectiveness of the proposed AAM and its ability to outperform the FAM structure, as well as the verification of theoretical results through simulations.

3.6 Impacts of L and power scaling on the average capacity of the system

Figure 6 depicts the average capacity of “AAM” and “AAM with power scaling” structures for \(L = 3, 5\). The figure illustrates that there exists a capacity ceiling for AAM design, whereas there is no capacity ceiling for AAM with power scaling (AAM-PS). These observations indicate the potential advantages of AAM-PS over AAM. Additionally, it shows that the capacity of the system increases as L increases, highlighting the scalability of the AAM design in accommodating more near users.

3.7 Comparison of energy efficiency of FD NOMA and HD NOMA systems

Figure 7 compares the energy efficiency of proposed FD NOMA and half-duplex NOMA (HD NOMA) systems, where energy efficiency is considered as the ratio of the throughput to total energy consumption. The figure shows that in the high-SNR region, the increase in SNR has a minimal effect on the energy efficiency of FD NOMA systems, while for HD NOMA systems, energy efficiency improves as SNR approaches infinity.

Moreover, it provides insights into the energy efficiency differences between FD NOMA and HD NOMA systems so that the aforementioned metric for FD NOMA system is lower than that of HD NOMA system. Moreover, there exists a point where after that point, energy efficiency remains constant. The aforementioned point for FD NOMA system is smaller than that of HD NOMA systems.

Fig. 3
figure 3

The outage probability for \(L=2\) and different scenarios

Full size image
Fig. 4
figure 4

Finite-SNR diversity gain for the proposed structures

Full size image
Fig. 5
figure 5

The outage probability of the proposed structures against transmitted power for \(L=4\) and \(\eta =0.001\)

Full size image
Fig. 6
figure 6

The impacts of power scaling on average capacity for AAM

Full size image
Fig. 7
figure 7

The energy efficiency for HD and FD near users

Full size image

3.8 Discussion

The effectiveness of our proposed approach in 6G Massive MIMO systems is explored in this section. Our suggested methodology is particularly suitable for deployment in 6G Massive MIMO systems which the outage probability decreases to zero and indicates the optimum performance. Hence, our proposed framework emerges as a suitable option for massive MIMO systems. For large-scale MIMO near users equipped with a considerable number of receive and transmit antennas, based on the following steps, an optimization problem by machine learning algorithms can be formulated.

  1. (1)

    Identify all receive and transmit antennas at FD near users.

  2. (2)

    Segment the step 1 into some distinct subsets.

  3. (3)

    Employ max–min on the aforementioned subsets.

  4. (4)

    Obtain the maximum value of the results obtained from step 3.

4 Conclusion

The paper has presented a novel approach that combines adaptive antenna mode (AAM), power scaling, near-user selection and power allocation for multi-user FD cooperative NOMA systems. The near user helps the BS to transmit information to the far user, but an outage floor issue has been identified in high-power regions. Power scaling techniques have been developed to address this issue and enhance the system performance. The evaluation of the asymptotic structure shows that the use of PS technique increases achievable diversity. Specifically, the FAM scheme achieves full diversity order for low self-interference values, while the AAM scheme achieves twice of full diversity order. These results demonstrate significant improvements in diversity through the proposed techniques. Additionally, the research indicates that PS techniques can overcome the capacity ceiling and outage floor in multi-user FD cooperative NOMA systems, potentially enhancing the overall performance and reliability, especially in high-power regions.