1 Introduction

Internet of Things (IoT) has become one of the main-stream emerging communication paradigms, which connects all internet devices through information sensing equipment to achieve intelligent identification and management [1, 2]. As a Architectural pattern of IoT, edge cloud combines the advantages of edge computing and cloud computing, which has the functions of low latency data processing, data filtering and simplification, providing better choices for the design and deployment of IoT systems [3]. However, spectrum scarcity has become a major bottleneck for the employment of IoT devices in edge cloud environment [4]. Cognitive radio (CR) has great promise to improve the spectrum efficiency (SE) of the system. In particular, we consider the underlay cognitive radio scenario, which allows secondary users (SUs) to access the spectrum of the primary network while ensuring the quality of service (QoS) of the primary users (PUs) [5, 6]. Besides CR, non-orthogonal multiple access (NOMA) is another promising technique to enhance system SE and achieve massive connectivity of IoT devices in CR networks by employing the underlying strategy. Different from the traditional orthogonal multiple access (OMA), NOMA services multiple users with the same radio resource, in which users are distinguished by different power, which can be achieved by applying successive interference cancellation (SIC) at the receiver [7,8,9].

Integrating the NOMA technology into CR networks will have great potential to improve SE and satisfy the IoT requirements of massive connectivity as it can serve more SUs by sharing the licensed spectrum with PUs [8, 10, 11]. However, the interference constraint in cognitive radio non-orthogonal multiple access (CR-NOMA) can severely limit the transmission rate achievable of the SUs [12]. In addition, due to the effects of non-line-of-sight transmission, deep fading, and shadow effects between the transmitter and receiver of the SUs in the actual transmission process, direct link transmission may have poor channel conditions [13]. Two-way relay (TWR) is an attractive solution to circumvent this challenge, which can simultaneously send or receive user information from different directions within the same time slot. Compared to the traditional one-way relay, TWR not only reduces long-distance transmission path loss but also improves the utilization of spectrum resources [14].

However, with the development of intelligent IoT, a large number of sensing devices have flooded into the network access side, how to improve the network access capability is also an urgent and noteworthy issue. Latest, massive multiple-input multiple-output (MIMO) techniques have also been identified as a key enabling technology for improving the access performance of the next-generation wireless network, which adjusts multiple-antenna units by controlling the phase and signal amplitude of the signals emitted by each antenna unit, generating directional beams, and allowing the wireless signal energy to form an electromagnetic wave superposition at the mobile phone position. Massive MIMO not only increases the number of user access, but also ensures higher communication capabilities [15]. In addition to high SE and the QoS of the uses, in practice, due to the broadcast nature of NOMA as well as RF link itself, the potential eavesdropper may be presented in the wireless environment, stealing part of the information sent to the destination node [16]. Therefore, it is urgent to ensure the secure communication of the CR-NOMA systems and studies the physical layer security performance of the cognitive radio-inspired NOMA network, a user pairing method based on optimal channel gain is proposed. However, in the scheme, the NOMA users are limited, and large-scale access scenarios are not considered. To meet the requirements of high SEE, massive connectivity and low power consumption for IoT devices in CR-NOMA networks, a scheme is investigated to improve the SEE of the unlicensed SUs.

The main contributions are summarized as follows:

  • TWR cooperative CR-NOMA with massive MIMO scheme is proposed to improve the security energy efficiency of the cognitive network, where two groups of NOMA users exchange the message with the aid of the TWR. Considering the existence of physical layer security problems in practical systems, the joint optimization strategy based on power splitting (PS) ratios, transmission power and antenna selection is proposed, which can achieve a better tradeoff between security and energy consumption.

  • The problem of maximizing security energy efficiency is formulated under the constraints of ensuring the primary user’s quality of service and NOMA user de-coded successfully. Fractional programming is introduced to convert the original fraction optimization into tractable integral expression.

  • The objective function is a non-convex and complex multi-objective optimization problem. To tackle the problem, transformed it into three single-objective problem. The semi-determined relaxation algorithm based on the first-order Taylor series expansion function is proposed to convert the problem into convex function. The novel orthogonal antenna selection algorithm is proposed to optimize the number of antennas. In order to get the optimal solution of SEE, a multi-objective iterative algorithm (MOIA) is proposed to solve it.

  • The numerical results are presented to demonstrate under same parameters the proposed scheme of cognitive massive-MIMO-NOMA with two-way relay has more obvious advantages in improving system security energy efficiency. Moreover, due to the impact of correlation between the antennas, as the number of relay antennas increases to infinity, the SEE converges to a specific value, which exist the optimal number of antennas.

2 Related Works

Recently, two-way relay (TWR) as a promising relay technology has been widely used in the cooperative transmission of CR-NOMA systems. Different from one-way relay, two-way relay can send or receive user information from different directions in the same time slot, which has huge potential in improving user transmission performance and spectrum utilization [17]. The authors of [18] and [19] are dedicated to improving the signal-to-noise ratio of cooperative TWR in the cognitive radio network. Specifically, in [18], the two-way relay was applied in cognitive hybrid satellite-terrestrial networks and multi-antenna secondary users transmit, the content caching technology was designed to minimize network transmission delay. The authors of [19] study the two-way relay assistant cooperative network, the algorithm of joint optimal relay selection and spectrum allocation was proposed to maximize the receive signal-to-noise ratio of SUs. Recently, TWR technology has been widely used in NOMA systems to improve spectral efficiency [20, 21]. In [20], the achievable ergodic secrecy rates of NOMA users have been deduced, which has proved the effectiveness of TWR in improving the security rate of the main link and eavesdropping link. In [21], the asymptotic outage probabilities and ergodic rate of the TWR-NOMA system were analyzed under the conditions of perfect successive interference cancellation and imperfect serial interference cancellation.

Besides, relay cooperation CR-NOMA system, multiple-antenna technology has been testified as an efficient scheme to improve the performance of NOMA systems for IoT devices [22, 23]. Tang et al. [24] propose a hybrid cell-free large-scale antenna NOMA system model, and the system throughput is optimized under the constraints of intra-cluster interference and inter-cluster interference, which shows that the proposed scheme has obvious advantages in improving the utilization of spectrum resources and increasing the number of users access. Based on [24], the authors of [25] study the user bandwidth efficiency of cell-free large-scale antenna NOMA system, the maximum–minimum optimization strategy is proposed to optimize system performance. In [26], the impact of predefined power allocation and range power allocation schemes on the throughput of massive MIMO-NOMA systems was investigated. In [27], the outage probability performance of multi-antenna NOMA systems affected by the actual machine-type communications was studied, including receiving sequence and access quantity of the receivers, which demonstrated the performance gains of the proposed multi-antenna have an obvious advantage over the single-antenna. In order to reduce the complexity of the massive MIMO system and improve the energy efficiency of the system, [28] proposed the algorithm based on the dinkelback and alternating optimization framework.

Although increasing the number of antennas has a clear advantage in improving system performance, to a certain extent, results in power consumption and the waste of radio frequency (RF) chains. Transmission antenna selection (TAS) has been widely used in large-scale antenna systems to solve the problem [29]. In [30], the backhaul combination scheme based on a large-scale MIMO-NOMA system was investigated, in which the optimization algorithm is proposed to maximize the signal-to-interference-plus-noise ratio of the backhaul link. In [31], the outage probability for Cooperative MIMO-NOMA was studied, and in order to improve the system’s outage probability, a hybrid antenna scheme based on antenna selection and maximum ratio combining is proposed. In [32], in order to improve the performance of MISO-NOMA system for the Internet of Vehicles (IoV) field, a fusion antenna selection scheme for near and far users is proposed, which verifies the obvious advantages in improving the channel attenuation. Similar to [32], the authors of [33] consider the joint antenna selection algorithm to improve the outage probability. Considering cooperative cognitive NOMA systems, in [34], a novel joint user selection and antenna transmit of scheme is proposed. It is envisioned that TAS can significantly reduce the complexity of large-scale antenna CR-NOMA systems [30,31,32,33,34]. However, the interference of the secondary user to the primary user is ignored in the antenna selection process. In this paper, we propose a novel-based orthogonal selection algorithm, which is aimed to maximize the transmission performance of SUs with minimal interference to the primary users (PUs).

In addition, due to the broadcast nature of NOMA as well as CR and the dual function of RF signals [35], NOMA CRNs relying on TWR is vulnerable to be eavesdropped. Therefore, it is urgent to ensure the secure communication of the CR-NOMA system [36, 37]. In [38], the physical layer security performance of the cognitive radio-inspired NOMA network is studied, and a user pairing method based on optimal channel gain is proposed. In [39], the problem of security resource allocation of CR-NOMA system under the influence of interference with base stations to effectively improve the security rate of the system is studied. Tang et al. [40] propose a secure transmission mode of 3D UAV based on CR-NOMA network, and compared with the benchmark scheme, the method has obvious advantages in improving security. In order to improve the physical layer security performance of the CR-NOMA system, Zhao et al. [41] propose a secure beamforming scheme to improve the secure transmission of CR-NOMA system.

3 System Model

The system model as shown in Fig. 1, which mainly contains two groups of secondary users (SUs) \(G_{1} = \left\{ {C_{1} ,C_{2} ,...,C_{K} } \right\}\),\(G_{2} = \left\{ {C_{K + 1} ,C_{K + 2} ,...,C_{2K} } \right\}\) and a pair of primary users. The two-way relay is applied in the system to assist the secondary users transmission. It is assumed that TWR has M antennas, the cognitive users have N antennas, while other nodes are equipped with one antenna. Relay assists the exchange of information between two sets of secondary users, mainly including two time slots, multiple access and broadcast time slots.

Fig. 1
figure 1

The system model for a cognitive massive MIMO-NOMA network for edge-cloud environment

Full size image

Multiple-Access Slot: SUs transmits the information to TWR with the uplink NOMA. As the result, the TWR is equipped with two set of antennas, and the self-interference exists when receiving and transmitting signals. The signal received by the TWR is given by

$${\mathbf{y}}_{r} = \sum\limits_{i = 1}^{2K} {{\mathbf{h}}_{i} \sqrt {P_{s,i} x_{i} } } + {\mathbf{h}}_{PR} \sqrt {P_{t} s_{n} } + {\mathbf{n}}_{r} ,$$
(1)

where \({\mathbf{h}}_{i} \in {\mathbb{C}}^{M \times N}\) is the channel between the \(C_{i}\) and TWR, with \(\left| {{\mathbf{h}}_{1} } \right|^{2} \le \left| {{\mathbf{h}}_{2} } \right|^{2} \le ,..., \le \left| {{\mathbf{h}}_{K} } \right|^{2}\),\(\left| {{\mathbf{h}}_{K + 1} } \right|^{2} \le \left| {{\mathbf{h}}_{K + 2} } \right|^{2} \le ,..., \le \left| {{\mathbf{h}}_{2K} } \right|^{2}\). \({\mathbf{h}}_{PR}\) is the channel between the PT and TWR, \(x_{i} (i = 1,2,...2K)\) and \(s_{n}\) are the transmission signals of \(C_{i}\) and PU, respectively. \({\mathbf{n}}_{r} \sim CN(0,\delta^{2} {\mathbf{I}})\) is the Gaussian noise at TWR. \(P_{s,i}\) and \(P_{t}\) are the transmit power of \(C_{i}\) and PU, respectively. For the received signal, the received signal-to-interference-plus-noise ratio (SINR) at TWR to detect \(x_{m}\) of \(G_{1}\) is given by

$$\gamma_{D \to m} = \frac{{\left| {{\mathbf{h}}_{m} } \right|^{2} P_{s,m} }}{{\sum\nolimits_{i = 1}^{K} {\left| {{\mathbf{h}}_{i} } \right|^{2} } P_{s,i} + \sum\nolimits_{i = K + 1}^{2K} {\left| {{\mathbf{h}}_{i} } \right|^{2} } P_{s,i} + \delta^{2} }},$$
(2)

where \(\sum\nolimits_{i = K + 1}^{2K} {\left| {{\mathbf{h}}_{i} } \right|^{2} } P_{s,i}\) is the interference from \(G_{2}\) at the TWR. The rate at TWR to decode the n-th NOMA user in \(G_{2}\) is given by

$$\gamma_{D \to n} = \frac{{\left| {{\mathbf{h}}_{n} } \right|^{2} P_{s,n} }}{{\sum\nolimits_{i = 1}^{K} {\left| {{\mathbf{h}}_{i} } \right|^{2} } P_{s,i} + \sum\nolimits_{i = K + 1}^{2K} {\left| {{\mathbf{h}}_{i} } \right|^{2} } P_{s,i} + \delta^{2} }}.$$
(3)

Broadcast plot: The information is exchanged between the \(G_{1}\) and \(G_{2}\) by the virtue of TWR. Since the cognitive users in the two sets have the same transmission mechanism, the only one set users in \(G_{1}\) are considered. Therefore, in the broadcast plot, relay node transmits information to the users with downlink NOMA, and the signals received at k-th SUs in \(G_{1}\) can be written as

$$\begin{aligned} y_{k} & = {\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{k} x_{k} \sqrt {a_{k} P_{R} } + {\hat{\mathbf{h}}}_{Rk} \sum\limits_{i = 1,i \ne k}^{K} {\sqrt {a_{i} P_{R} } } {\mathbf{w}}_{i} x_{i} \hfill \\ \, & \quad + \sum\limits_{i = 1}^{K} {\sqrt {a_{i} P_{R} } } {\tilde{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} x_{i} + n_{k} , \hfill \\ \end{aligned}$$
(4)

where \(P_{R}\) denotes the transmit power of TWR. \(n_{k} \sim CN(0,\sigma_{k}^{2} )\) is the Gaussian noise, where \(a_{i} (i = 1,2,...,K)\) denotes the power allocation of TWR with \(\sum\nolimits_{i = 1}^{K} {a_{i} } = 1\). Moreover, we adopt a low complexity maximum ratio transmission (MRT) beamforming scheme. Such a beamforming scheme can be expressed as \({\mathbf{w}}_{i} = {\hat{\mathbf{h}}}_{Ri} /\left\| {{\hat{\mathbf{h}}}_{Ri} } \right\|\). Without loss of generality, the channel gains are sorted as \(\left| {{\mathbf{h}}_{R1} {\mathbf{w}}_{1} } \right|^{2} \le \left| {{\mathbf{h}}_{R2} {\mathbf{w}}_{2} } \right|^{2} \le ... \le \left| {{\mathbf{h}}_{Rk} {\mathbf{w}}_{k} } \right|^{2} \le ... \le \left| {{\mathbf{h}}_{RK} {\mathbf{w}}_{K} } \right|^{2}\). According to NOMA protocol, received SINR of k-th SU can be given by

$$\gamma_{k} = \frac{{P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{k} } \right|^{2} a_{k} }}{{\sum\nolimits_{i = k + 1}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} a_{i} + \left( {P_{R} \sigma_{e}^{2} \sum\nolimits_{i = 1}^{K} {\left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} a_{i} } + \sigma_{k}^{2} } \right)} }}.$$
(5)

Therefore, the rate of \(C_{k}\) can be written as \(R_{k} = \log_{2} (1 + \gamma_{k} )\). With the purpose of realizing user \(C_{k}\) to decode the signal of user \(C_{l}\), it should be satisfied \(l < k\), so the decoding rate can be given by

$$R_{k \to l} = \log_{2} \left( {1 + \frac{{P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{k} } \right|^{2} a_{l} }}{{\sum\nolimits_{i = k + 1}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} a_{i} + b + \sigma_{l}^{2} } }}} \right),$$
(6)

where \(b = P_{R} \sigma_{e}^{2} (\sum\nolimits_{i = k + 1}^{K} {a_{i} \left| {{\mathbf{w}}_{i} } \right|^{2} } )\). Due to the broadcast nature of the wireless channel, it is easy to be eavesdropped by illegal users during the actual transmission of information, so two eavesdroppers \(E_{1}\) and \(E_{2}\) are considered in the system model. Thus, the signal transmitted by TWR eavesdropped by \(E_{1}\) can be written as

$$y_{e} = \sum\limits_{i = k}^{K} {{\mathbf{h}}_{e} {\mathbf{w}}_{i} x_{i} } \sqrt {a_{i} P_{R} } + n_{e} ,$$
(7)

where he denotes the channel gain from TWR to \(E_{1}\). \(n_{e}\) is the additive white Gaussian noise due to the receiving antenna at the eavesdropper with the variance \(\sigma_{E}^{2}\). Therefore, the signal received by eavesdropper \(E_{1}\) to overhear \(C_{k}\) is obtained by

$$R_{e}^{k} = \log_{2} \left( {1 + \frac{{P_{R} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{k} } \right|^{2} a_{k} }}{{\sum\nolimits_{i = k + 1}^{K} {P_{R} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} a_{i} + \sigma_{E}^{2} } }}} \right).$$
(8)

Just like all cognitive radio networks, in the CR-NOMA system, cognitive users can communicate normally under the constraint of the interference to the primary users is less than the minimum threshold. The received rate of primary user can be obtained as

$$R_{P} = \log_{2} \left( {1 + \frac{{P_{t} \left| {{\mathbf{h}}_{PP} } \right|^{2} }}{{P_{R} \sum\nolimits_{i = 1}^{2K} {\left| {{\mathbf{h}}_{PP} {\mathbf{w}}_{i} } \right|^{2} + \sigma_{P}^{2} } }}} \right),$$
(9)

where \({\mathbf{h}}_{RP}\) and \({\mathbf{h}}_{PP}\) denote the channel from TWR to PR and PT to PR, respectively, \(n_{P}\) is the Gaussian noise of PR.

4 Problem Formulation

In this part, we evaluate the system performance with the security energy efficiency (SEE), because the SEE is defined as the ratio of security rate to the actual energy consumption, so we first give the expression of the system’s security sum rate. For the convenience, cognitive users on both sides of TWR are assumed to have transport mechanisms, so only the cognitive users in \(G_{1}\) is considered. For \(1 \le k \le K\), therefore, the security rate achievable by the k-th user and the sum rate can be, respectively, given by

$$R_{S}^{k} = \left[ {R_{k} - R_{e}^{k} } \right]^{ + } ,$$
(10)
$$R_{S} = \sum\limits_{k = 1}^{K} {R_{S}^{k} } ,$$
(11)

where \(\left[ x \right]^{ + } \triangleq \max (0,x)\). From above, it can be seen when \(R_{k} \le R_{e}^{k}\), \(R_{S}^{k}\) will be zero. Furthermore, it assume that when \(k = N\), we have \(R_{n} \le R_{e}^{N}\), In this case, there are \(N\) users whose secrecy rate is zero. The channels of the \(K\) cognitive users are sorted as \(0 \le \left| {{\mathbf{h}}_{R1} {\mathbf{w}}_{1} } \right|^{2} \le \left| {{\mathbf{h}}_{R2} {\mathbf{w}}_{2} } \right|^{2} \le ... \le \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{k} } \right|^{2} \le ... \le \left| {{\mathbf{h}}_{RK} {\mathbf{w}}_{W} } \right|^{2}\). Then, the expression of formula (15) can be rewritten as

$$R_{S} = \sum\nolimits_{k = N + 1}^{K} {(R_{k} - R_{e}^{k} )} .$$
(12)

The actual energy consumption of the system can be given by

$$P_{tot} = P_{R} /\xi + P_{C} ,$$
(13)

where \(\xi\) is energy efficiency, \(P_{R}\) is the TWR transmission power of TWR, and \(P_{C}\) denotes other energy consumption for system operation besides \(P_{R}\). Consequently, the objective function is set to maximize the security energy efficiency by adjusting power allocation and transmit power. To the end, the optimization problem p0 is formulated as

$$\mathop {\max }\limits_{{P_{R} ,a_{i} ,M}} \eta_{{{\text{SEE}}}} = \frac{{R_{S} }}{{P_{{{\text{tot}}}} }},$$
(14)
$$s.t. \, R_{S} \ge r_{{{\text{th}}}} ,$$
(14a)
$$R_{k} \ge R_{m} ,$$
(14b)
$$\sum\nolimits_{i = 1}^{K} {a_{i} } = 1,$$
(14c)
$$P_{R} \ge P_{\max } ,$$
(14d)

where \(r_{{{\text{th}}}}\) and \(R_{m}\) represents the minimum constraints for the transmission rate of the primary users and cognitive users, respectively; Eq. (18c) denotes the constraint of the TWR power allocation. Moreover, \(P_{\max }\) is the maximum transmit power of TWR.

5 Optimization Solution

In this section, we first decouple the problem into three subproblems, i.e., the optimization of transmission power, power allocation and antenna selection. Next, we propose a multi-objective alternative iterative algorithm to obtain global optimal of security energy efficiency.

5.1 Optimization of the Transmission Power

The objective function in (18) is non-convex. In order to avoid high complexity of the solution to this problem, we introduce the following parameter transformation. In this section, the fractional programming is introduced to reduce the complexity by transforming fractional problems into equation problems with parameter \(\lambda\), which can be formulated as \(F(\Xi ,\lambda ) = R_{S} - \lambda P_{{{\text{tot}}}}\), and \(\Xi\) is the collection of variable \(P_{R}\), \(M\) and \(a_{i}\), and only if it is the root that satisfies with the formula \(g(\lambda ) = 0\), which of the optimal value of \(g(\lambda )\). Theorem 3.1 is formulated to solve the optimal solution \(\lambda^{*}\) of \(g(\lambda )\).

Theorem 3.1

g(λ) is convex, continuous and decreasing function of \(\lambda\), and the optimal solution of problem \(a_{i}\) exists at \(F(\Xi ,\lambda ) = 0\).

It is assumed that optimize the transmit power \(P_{R}\) with the variables power allocation \(a_{i}\),\(\lambda^{*}\) and the number of transmission antennas M are given; therefore, the optimization problem is formulated as \(p_{1}\):

$$\mathop {\max }\limits_{{P_{R} }} \eta_{{{\text{SEE}}}} = R_{S} - \lambda P_{{{\text{tot}}}} ,$$
(15)
$$s.t. \, R_{P} \ge r_{{{\text{th}}}} ,$$
(15a)
$$R_{k} \ge R_{m} ,$$
(15b)
$$P_{R} \le P_{\max } .$$
(15c)

Obviously, it is very difficult to solve problem \(p_{1}\) directly because it is a non-convex problem. A convex approximation method based on Taylor expansion is proposed to transform the constraints (15a) and (15b) approximate them as convex constraints. According to Taylor formula, the secrecy sum rate \(R_{S}\) can be approximately formulated as

$$\begin{aligned} R_{SS} = \log_{2} \left( {\sum\limits_{i = k + 1}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + P_{R} \tilde{\sigma } + \sigma_{k}^{2} } \right) \hfill \\ \quad \quad - \log_{2} \left( {\sum\limits_{i = k}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + P_{R} \tilde{\sigma } + \sigma_{k}^{2} } \right) \hfill \\ \quad \quad + \log_{2} \left( {\sum\limits_{i = k + 1}^{K} {P_{R} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{E}^{2} } \right) \hfill \\ \quad \quad - \log_{2} \left( {\sum\limits_{i = k}^{K} {P_{R} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{k} + \sigma_{E}^{2} } \right), \hfill \\ \end{aligned}$$
(16)

where \(\tilde{\sigma } = \sigma_{e}^{2} \sum\nolimits_{i = 1}^{K} {a_{i} }\). The second and fourth terms in \(R_{{{\text{tot}}}}^{*}\) are extended to approximate affine functions with the first-order Taylor formula:

$$\begin{aligned} \log_{2} \left( {\sum\limits_{i = k + 1}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + P_{R} \tilde{\sigma } + \sigma_{k}^{2} } \right) \hfill \\ \quad \quad = \frac{{\left( {\sum\limits_{i = k}^{K} {\left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} a_{i} + } \tilde{\sigma }} \right)P_{R} }}{{P_{0} \left( {\sum\limits_{i = k}^{K} {\left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} a_{i} + } \tilde{\sigma } + P_{0} \sigma_{k}^{2} } \right)\ln 2}} + \Theta_{1} + \varepsilon_{1} , \hfill \\ \end{aligned}$$
(17)
$$\begin{aligned} \log_{2} \left( {\sum\nolimits_{i = k}^{K} {P_{R} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{E}^{2} } \right) \hfill \\ \quad \quad = \frac{{\log_{2} \sum\nolimits_{i = k}^{K} {P_{R} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} }}{{\ln 2\left( {P_{0} \sum\nolimits_{i = k}^{K} {\left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{E}^{2} } \right)}} + \Theta_{2} + \varepsilon_{2} , \hfill \\ \end{aligned}$$
(18)

where \(\varepsilon_{1}\) and \(\varepsilon_{2}\) denote the Taylor expansion error, and the specific expressions of \(\Theta_{1}\) and \(\Theta_{2}\) are, respectively, given by

$$\begin{aligned} \Theta_{1} = \log_{2} \left( {\sum\nolimits_{i = k + 1}^{K} {P_{0} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{E}^{2} } \right) \hfill \\ \quad \quad - \frac{{\left( {\sum\nolimits_{i = k}^{K} {\left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} a_{i} P_{0} + } \tilde{\sigma }} \right)}}{{\left( {P_{0} \sum\nolimits_{i = k}^{K} {\left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} a_{i} + } \tilde{\sigma }P_{0} + \sigma_{k}^{2} } \right)\ln 2}}, \hfill \\ \end{aligned}$$
(19)
$$\begin{aligned} \Theta_{2} = \log_{2} \left( {\sum\limits_{i = k + 1}^{K} {P_{0} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{E}^{2} } \right) \hfill \\ \quad \quad - \frac{{\left( {\sum\nolimits_{i = k}^{K} {\left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} a_{i} P_{0} } } \right)}}{{\left( {P_{0} \sum\nolimits_{i = k}^{K} {\left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} a_{i} + } \sigma_{E}^{2} } \right)\ln 2}}. \hfill \\ \end{aligned}$$
(20)

Based on (22) and (23), the sum rate of SUs is rewritten as

$$\begin{aligned} R_{S}^{*} = \log_{2} \left( {\sum\limits_{i = k}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + P_{R} \tilde{\sigma } + \sigma_{k}^{2} } \right) \hfill \\ \quad \quad - \frac{{\left( {\sum\limits_{i = k}^{K} {\left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \tilde{\sigma }} \right)P_{R} }}{{\left( {\sum\limits_{i = k}^{K} {P_{0} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + P_{0} \tilde{\sigma } + \sigma_{k}^{2} } \right)\ln 2}} \hfill \\ \quad \quad - \log_{2} \left( {\sum\limits_{i = k}^{K} {P_{R} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{E}^{2} } \right) \hfill \\ \quad \quad - \frac{{\sum\limits_{i = k}^{K} {\left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} P_{R} }}{{\left( {\sum\limits_{i = k}^{K} {P_{0} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{E}^{2} } \right)\ln 2 + \varsigma }}, \hfill \\ \end{aligned}$$
(21)

where \(\varsigma = \sum\nolimits_{i = 1}^{2} {\Theta_{i} } + \varepsilon_{i}\), \(\varsigma\) is a constant for the given \(P_{0}\). Constraints (15a) and (15b) are also converted by this way, which can be, respectively, given by

$$\begin{aligned} R_{P}^{*} = \log_{2} \left( {\sum\limits_{i = k}^{K} {P_{R} \left| {{\mathbf{h}}_{RP} {\mathbf{w}}_{i} } \right|^{2} } + P_{t} \left| {{\mathbf{h}}_{PP} } \right|^{2} + \sigma_{P}^{2} } \right) \hfill \\ \, \quad \quad + \frac{{\sum\nolimits_{i = 1}^{2K} {P_{R} \left| {{\mathbf{h}}_{RP} {\mathbf{w}}_{i} } \right|^{2} } }}{{\ln 2(\left| {{\mathbf{h}}_{RP} {\mathbf{w}}_{i} } \right|^{2} + \sigma_{P}^{2} )}} + \Theta_{3} + \varepsilon_{3} , \hfill \\ \end{aligned}$$
(22)
$$\begin{aligned} R_{k}^{*} = \log_{2} \left( {\sum\limits_{i = k}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + P_{R} \tilde{\sigma } + \sigma_{k}^{2} } \right) \hfill \\ \quad \quad - \frac{{P_{R} \left( {\sum\limits_{i = k}^{K} {\left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \tilde{\sigma }} \right)}}{{\left( {\sum\nolimits_{i = k}^{K} {P_{0} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + P_{0} \tilde{\sigma } + \sigma_{k}^{2} } \right)\ln 2}} + \Theta_{1} + \varepsilon_{1} , \hfill \\ \end{aligned}$$
(23)

where \(\varepsilon_{3}\) is the Taylor expansion error:

$$\begin{aligned} \Theta_{3} = \log_{2} \left( {\sum\limits_{i = 1}^{2K} {P_{0} \left| {{\mathbf{h}}_{RP} {\mathbf{w}}_{i} } \right|^{2} } + P_{t} \left| {{\mathbf{h}}_{PP} } \right|^{2} } \right) \hfill \\ \quad \quad - \sum\limits_{i = 1}^{2K} {P_{0} \left| {{\mathbf{h}}_{RP} {\mathbf{w}}_{i} } \right|^{2} } /\ln 2(P_{0} \left| {{\mathbf{h}}_{RP} {\mathbf{w}}_{i} } \right|^{2} + \sigma_{P}^{2} ). \hfill \\ \end{aligned}$$
(24)

By deriving from the above formula, problem \(p_{1}\) can be rewritten as

$$\mathop {\max }\limits_{{P_{R} }} \eta_{{{\text{SEE}}}} = R_{k}^{*} - \lambda (P_{R} /\xi + P_{C} ),$$
(25)
$$s.t. \, R_{P}^{*} \ge r_{{{\text{th}}}} ,$$
(25a)
$$R_{k}^{*} \ge R_{m} ,$$
(25b)
$$P_{R} \le P_{\max } .$$
(25c)

Note that the problem is convex and satisfies the Lagrangian dual optimization problem. The dual problem of \(P_{R}\) can be expressed as

$$\mathop {\max }\limits_{{P_{R} \ge 0}} \left\{ {\mathop {\min }\limits_{w \ge 0,\mu \ge 0,\gamma \ge 0} L(P_{R} ,w,\mu ,\gamma )} \right\}.$$
(26)

Assuming that inequality constraints (25a) and (25b) are strictly feasible, that is \(P_{R}\), they both exist, \(R_{P}^{*} (P_{R} ) \ge r_{th}\), \(R_{k}^{*} (P_{R} ) \ge R_{m}\),\(P_{R} \le P_{\max }\). Then, there exists \(P_{R}\) and \(w\),\(\mu\),\(\gamma\), so that \(P_{R}\) is the optimal solution of the original problem, \(w\),\(\mu\),\(\gamma\) is the optimal solution of the dual problem:

$$\begin{aligned} \mathop {\max }\limits_{{P_{R} \ge 0}} \left\{ {\mathop {\min }\limits_{w \ge 0,\mu \ge 0,\gamma \ge 0} L(P_{R} ,w,\mu ,\gamma )} \right\} \hfill \\ \quad = \mathop {\min }\limits_{w \ge 0,\mu \ge 0,\gamma \ge 0} \left\{ {\mathop {\max }\limits_{{P_{R} \ge 0}} L(P_{R} ,w,\mu ,\gamma )} \right\}. \hfill \\ \end{aligned}$$
(27)

With the dual variables, according to Lagrange’s dual algorithm, formula (25) can be expressed as

$$\begin{aligned} L(P_{R} ,w,\mu ,\gamma ) & = R_{S}^{*} - \lambda (P_{R} /\xi + P_{C} ) \hfill \\& \quad + w(R_{P}^{*} - r_{th} ){ + }\mu {(}R_{k}^{*} - R_{m} {) + }\gamma (P_{\max } - P_{R} ), \hfill \\ \end{aligned}$$
(28)

where \(w\), \(\mu\) and \(\gamma\) are dual variables associated with constraints (26). The extremum points \(P_{R}^{*}\), \(w*\), \(\mu^{*}\) and \(\gamma^{*}\) can be obtained by the first-order partial of (29):

$$\left\{ \begin{aligned} \nabla_{{P_{R} }} L(P_{R} ,w,\mu ,\gamma ) = 0 \hfill \\ \nabla_{w} L(P_{R} ,w,\mu ,\gamma ) = 0 \hfill \\ \nabla_{\mu } L(P_{R} ,w,\mu ,\gamma ) = 0 \hfill \\ \nabla_{\gamma } L(P_{R} ,w,\mu ,\gamma ) = 0 \hfill \\ \end{aligned} \right..$$
(29)

Based on the constraints of (25a), (25b) and (25c), the value range of \(P_{R}\) is obtained by

$$S_{u} = \{ \Xi_{1} \le P_{R} \le \min (\Xi_{1} ,P_{\max } )\} ,$$
(30)

where

$$\left\{ \begin{aligned} \Xi_{1} = \frac{{R_{M} \sigma_{k}^{2} }}{{\left[ {\left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{k} } \right|^{2} a_{k} - R_{M} \left( {\sum\nolimits_{i = k + 1}^{K} {P_{0} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{k} } \right|^{2} } a_{i} + \sigma_{e}^{2} \sum\limits_{i = 1}^{2K} {a_{i} } } \right)} \right]}} \hfill \\ \Xi_{2} = (P_{t} \left| {{\mathbf{h}}_{PP} } \right|^{2} - R_{t} \sigma_{P}^{2} )/R_{t} \sum\limits_{i = 1}^{2K} {\left| {{\mathbf{h}}_{RP} {\mathbf{w}}_{i} } \right|^{2} } \hfill \\ \end{aligned} \right.,$$

where \(R_{t} = 2^{{r_{th} }} - 1\), \(R_{M} = 2^{{R_{m} }} - 1\). According to the properties of solving the univariate optimal solution, the optimal value of \(P_{R}\) can be obtained by

$$P_{R} = \left\{ \begin{aligned} P_{R}^{*} \, \Xi_{1} \le P_{R} \le \min (\Xi_{2} ,P_{\max } ) \hfill \\ \Xi_{1} \, P_{R}^{*} \le \Xi_{1} \le \min (\Xi_{2} ,P_{\max } ) \hfill \\ \Xi_{2} \, \Xi_{1} \le \Xi_{2} \le \min (P_{R}^{*} ,P_{\max } ) \hfill \\ P_{\max } \, \Xi_{1} \le P_{\max } \le \min (\Xi_{2} ,P_{R}^{*} ) \hfill \\ \end{aligned} \right..$$
(31)

5.2 Optimization of the Power Allocation

In this section, the power allocation is proposed to maximize the security energy efficiency under the constraint of the user’s minimum transmission rate requirement. It is assumed that \(P_{R}\), and \(M\) are given, and the objective function \(p_{2}\) can be written as

$$\mathop {\max }\limits_{{a_{k} ,1 \le k \le K}} \eta_{{{\text{SEE}}}} = \frac{{R_{S} }}{{P_{{{\text{tot}}}} }},$$
(32)
$$a_{k} \ge \phi_{m} \left( {P_{R} \sum\limits_{i = k + 1}^{K} {\left| {{\mathbf{h}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{n}^{2} } \right),$$
(32a)
$$\sum\limits_{i = 1}^{K} {a_{i} } = 1.$$
(32b)

For the convenience, we considered under high SNR conditions to solve the optimal power allocation, so we assume that \(\sigma^{2} = \sigma_{E}^{2} = \sigma_{n}^{2}\), where \(\sigma_{n}^{2} = P_{R} \sigma_{e}^{2} \sum\limits_{i = 1}^{K} {a_{i} } + \sigma_{k}^{2}\). Since \(R_{N} = R_{e}^{N}\), we have \(\left| {{\hat{\mathbf{h}}}_{e} {\mathbf{w}}_{N} } \right|^{2} = \left| {{\hat{\mathbf{h}}}_{RN} {\mathbf{w}}_{N} } \right|^{2}\),\(\sum\limits_{i = N + 1}^{K} {\left| {{\hat{\mathbf{h}}}_{RN} {\mathbf{w}}_{i} } \right|^{2} } = \sum\limits_{i = N + 1}^{K} {\left| {{\hat{\mathbf{h}}}_{e} {\mathbf{w}}_{i} } \right|^{2} }\); therefore, the \(R_{S}\) can be reformulated as follows:

$$\begin{aligned} R_{S} = \log_{2} \left( {\sum\limits_{i = N + 1}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{N + 1} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{n}^{2} } \right) \hfill \\ \quad \quad - \log_{2} \left( {\sum\limits_{i = N + 1}^{K} {P_{R} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{n}^{2} } \right) \hfill \\ \quad \quad + \sum\limits_{k = N + 1}^{K - 1} {\left[ {\log_{2} \left( {\sum\limits_{i = k + 1}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{k + 1} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{n}^{2} } \right)} \right.} \hfill \\ \quad \quad - \sum\limits_{k = N + 1}^{K - 1} {\log_{2} } \left. {\left( {\sum\limits_{i = k + 1}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{n}^{2} } \right)} \right]. \hfill \\ \end{aligned}$$
(33)

For notations simplicity, we define

$$\Gamma_{m} \triangleq \left\{ \begin{aligned} P_{R} \left| {{\mathbf{h}}_{e} {\mathbf{w}}_{i} } \right|^{2} , \, k = N \, \hfill \\ P_{R} \left| {{\hat{\mathbf{h}}}_{k} {\mathbf{w}}_{i} } \right|^{2} , \, N \le k \le M - 1 \hfill \\ \end{aligned} \right.,$$
$$t_{m} \triangleq \sum\limits_{i = k + 1}^{M} {a_{i} } , \, \quad N \le m \le M - 1.$$

Based on above, let us denote a new function:

$$\begin{aligned} J_{m} (t_{m} ) \triangleq \log {}_{2}(\Gamma_{m + 1} t_{m} + \sigma_{n}^{2} ) \hfill \\ \, - \log {}_{2}(\Gamma_{m} t_{m} + \sigma_{n}^{2} ). \hfill \\ \end{aligned}$$
(34)

Then, \(R_{S}\) in (32) can be rewritten as

$$R_{S} = \sum\limits_{k = N}^{K - 1} {J_{m} (t_{m} )} .$$
(35)

Next, we have to solve this optimization problem. Here, we transform the function \(J_{m} (t_{m} )\) with the monotonicity. The first-order derivative of \(J_{m} (t_{m} )\) is given by

$$\frac{{dJ_{m} (t_{m} )}}{{dt_{m} }} = \frac{{(\Gamma_{m + 1} - \Gamma_{m} )\sigma_{n}^{2} }}{{\ln 2(\Gamma_{m + 1} t_{m} + \sigma_{n}^{2} )(\Gamma_{m} t_{m} + \sigma_{n}^{2} )}}.$$
(36)

The denominator of (36) is positive, since \(\Gamma_{m + 1} - \Gamma_{m} > 0\), and hence \(dJ_{m} (t_{m} )/dt_{m} > 0\), \(J_{m} (t_{m} )\) is the monotonically increasing function of \(t_{m}\). Consequently, the maximization of \(J_{m}\) is equivalent to finding the maximum of \(J_{m}\). The optimization problem of \(J_{m} (t_{m} )\) can be written as

$$\mathop {\max }\limits_{{a_{k} ,1 \le k \le K}} t_{m} ,$$
(37)
$$s.t. \, a_{k} \ge \Theta_{m} \left( {\sum\limits_{i = k + 1}^{K} {P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} } a_{i} + \sigma_{n}^{2} } \right),$$
(37a)
$$\sum\limits_{i = 1}^{K} {a_{i} } = 1,$$
(37b)

where\(\Theta = (2R_{m} - 1)/P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{k} } \right|^{2}\); it is observed that the problem (37) is convex. The conditions of Karush–Kuhn–Tucker (KKT) are met, so we get the optimal solution of problem (38) as

$$\lambda = \left\{ \begin{aligned} u_{m} - \sum\limits_{i = 1}^{m - 1} {u_{m} \Theta_{i} P_{R} \left| {\hat{\user2{h}}_{Ri} {\varvec{w}}_{i} } \right|^{2} ,{ 1} \le m \le k} \hfill \\ u_{m} - \sum\limits_{i = 1}^{m - 1} {u_{m} \Theta_{i} P\left| {\hat{\user2{h}}_{Ri} {\varvec{w}}_{i} } \right|^{2} + 1, \, k \le m \le M} \hfill \\ \end{aligned} \right.,$$
(38)
$$u_{i} \left[ {\Theta_{i} \left( {\sum\limits_{j = i + 1}^{K} {a_{j} P_{R} \left| {\hat{\user2{h}}_{Ri} {\varvec{w}}_{i} } \right|^{2} + \sigma_{n}^{2} } } \right) - a_{i} } \right] = 0,$$
(39)
$$u_{i} \ge 0,\quad 1 \le i \le K,$$
(40)
$$\lambda \left( {\sum\limits_{i = 1}^{K} { \, a_{i} - {1}} } \right){ = 0},$$
(41)
$$\lambda \ge 0,$$
(42)

where \(u_{i}\) and \(\lambda\) are the Lagrange multipliers for the inequality constraints (37a) and (37b), respectively. In order to derive the optimal solution of problem (37), constraint (37a) should be equal. Based on this, the closed-form expression for \(a_{k}\) can be obtained as

$$a_{k}^{*} = \left\{ \begin{aligned} (2^{{R_{m} - 1}} )\left[ {P_{R} \left| {\hat{\user2{h}}_{Ri} {\varvec{w}}_{i} } \right|^{2} \left( {1 - \sum\limits_{i = 1}^{k - 1} {a_{i}^{*} } } \right) + \sigma_{n}^{2} } \right],{ 1} \le k < K \hfill \\ 1 - \sum\limits_{i = 1}^{k - 1} {a_{i}^{*} } , \, \quad \quad \, k = K \, \hfill \\ \end{aligned} \right..$$
(43)

5.3 Orthogonal Antenna Selection

The ergodic secrecy rate of \(C_{k}\) is given by

$$R_{S}^{k} = E\left\{ {[R_{k} - R_{e}^{k} ]^{ + } } \right\} = [E\left\{ {R_{k} } \right\} - E\{ R_{e}^{k} \}^{ + } ],$$
(44)

where \(\{ x\}^{ + } = \max (x,0)\) [37]. Due to the channel hardening characteristics, this approximation is reasonable when the TWR is equipped with a large number of antennas [20]. The achievable rate of \(C_{k}\) is given by

$$\tilde{R}_{k} = E\left\{ {R_{k} } \right\} = \log_{2} (1 + \tilde{\gamma }_{k} ),$$
(45)

where\(E\left\{ \cdot \right\}\) denotes the expectation operator, and \(\tilde{\gamma }_{k}\) is the average SINR of \(C_{k}\):

$$\begin{aligned} E\left\{ {P_{R} \left| {\hat{\user2{h}}_{Rk} {\varvec{w}}_{k} } \right|^{2} a_{k} } \right\} \hfill \\ \quad = P_{R} a_{k} E\left( {\left| {\hat{\user2{h}}_{Rk} {\varvec{w}}_{k} } \right|^{2} } \right)\mathop = \limits^{(a)} P_{R} a_{k} \left| {E\left( {\left| {\hat{\user2{h}}_{Rk} } \right|} \right)} \right|^{2} \hfill \\ \quad \mathop = \limits^{(b)} P_{R} a_{k} \Gamma^{2} (M + 0.5)/\Gamma^{2} (M)\mathop = \limits^{(c)} P_{R} a_{k} M, \hfill \\ \end{aligned}$$
(46)

where \(\Gamma ( \cdot )\) is the gamma function, step (a) is based on the fact that \(\left| {\hat{\user2{h}}_{Rk} } \right|^{2}\) has a scaled Chi distribution with \(2T\) degrees of freedom by a factor of \(1/\sqrt 2\). Therefore, \(E\left(\left| {\hat{\user2{h}}_{Rk} } \right|^{2} \right) = \Gamma^{2} (M + 0.5)/\Gamma^{2} (M)\), and step (b) is obtained using the approximation \(\Gamma^{2} (M + 0.5)/\Gamma^{2} (M) \to (M \to \infty ) = M\), which we can obtain by

$$\begin{aligned} E\left\{ {P_{R} \sum\limits_{i = k + 1}^{K} {\left| {\hat{\user2{h}}_{Rk} {\varvec{w}}_{k} } \right|^{2} a_{i} + P_{R} \sigma_{e}^{2} \sum\limits_{i = 1}^{2K} {a_{i} + \sigma_{k}^{2} } } } \right\} \hfill \\ \quad \quad = P_{R} \sigma_{e}^{2} \sum\limits_{i = 1}^{2K} {a_{i} + \sigma_{k}^{2} } . \hfill \\ \end{aligned}$$
(47)

Therefore, the can be simplified as the follows:

$$\overline{R}_{k} = E\{ R_{k} \} = \log_{2} \left( {1 + \frac{{P_{R} a_{k} M}}{{P_{R} \sigma_{e}^{2} \sum\nolimits_{i = 1}^{2K} {a_{i} + \sigma_{k}^{2} } }}} \right).$$
(48)

Therefore, we have

$$E\{ R_{e}^{k} \} = \log_{2} (1 + 1/\sigma_{e}^{2} ).$$
(49)

Based on the constraints of (48) and (49), the average of \(\eta_{SEE}\) is given by

$$\overline{\eta }_{{{\text{SEE}}}} = \frac{{\sum\nolimits_{k = 1}^{K} {\log_{2} } \left( {1 + \frac{{P_{R} a_{k} M}}{{P_{R} \sigma_{e}^{2} \sum\limits_{i = 1}^{2K} {a_{i} + \sigma_{k}^{2} } - \log_{2} \left( {1 + \frac{1}{{\sigma_{E}^{2} }}} \right)}}} \right)}}{{(P_{R} /\xi + P_{C} )}}.$$
(50)

Taking the second derivative of \(\overline{\eta }_{{{\text{SEE}}}}\) with respect to \(M\), we achieve

$$\frac{{d\overline{\eta }_{{{\text{SEE}}}} }}{{d^{2} M}} = \frac{{ - \ln 2P_{R}^{2} a_{R}^{2} (P_{R} /\xi + P_{C} )^{ - 2} (P_{R} /\xi + P_{C} )^{2} }}{{\ln 2^{2} (P_{R} a_{e}^{2} \sum\nolimits_{i}^{2K} {a_{i} } + \sigma_{R}^{2} + P_{R} a_{k} M)^{2} }}.$$
(51)

It is observed that \(d\overline{\eta }_{{{\text{SEE}}}} /d^{2} M \le 0\); therefore, the security energy efficiency is an upward convex function on the number of antennas. In order to reduce the complexity and the interference to the primary user and improve the security performance of the system, the orthogonal antenna selection algorithm is proposed. A new function is defined, which can be formulated as follows:

$$\Delta ({\mathbf{h}}_{i} ,{\mathbf{h}}_{j} ) \triangleq \left| {{\mathbf{h}}_{i}^{H} {\mathbf{h}}_{j} } \right|/\left| {{\mathbf{h}}_{i} } \right|\left| {{\mathbf{h}}_{j} } \right|.$$
(52)

Antenna \(i\) and \(j\) are called the \(\delta\)-orthogonal if and only if

$$\Delta ({\mathbf{h}}_{i} ,{\mathbf{h}}_{j} ) \le \delta .$$
(53)

In this section, an orthogonal antenna selection algorithm based on the security energy efficiency is proposed, which can be described in Table 1.

Table 1 Algorithm orthogonal antenna selection (OAS)
Full size table

The whole orthogonal antenna selection algorithm is divided into two steps. The first step is the \(\delta_{p}\)-orthogonal antenna selection. This step mainly selects the number of orthogonal antennas less than the primary user’s transmission channel \(\Delta ({\mathbf{h}}_{pp} ,{\mathbf{h}}_{m} ) \le \delta_{p}\) from all antennas as the candidate antenna set \(Q_{1}\), with the goal of keeping orthogonal with the primary user to minimize the secondary user’s interference to the primary user in the cognitive radio environment. The second step is to select the antenna with the best channel conditions \(S(1) = \arg \max_{{k \in Q_{1} }} \left\| {{\hat{\mathbf{h}}}_{m} } \right\|\) from the set of candidate antennas with the first stage as the initial antenna in the second stage. In the second stage, the antenna selection is mainly based on the \(\delta_{c}\)-orthogonal algorithm on the basis of the set of candidate antennas \(Q_{1}\) in the first stage. This part is mainly based on the user’s convenience in selecting the number of antennas that meet the conditions based on the optimal channel gain in the set of candidate antennas. Simultaneously, select the antenna with better front \(M\) as the final candidate antenna set.

5.4 MOIT for Joint Optimization

This section proposed a multi-objective iterative algorithm for joint optimization, which is designed to maximize the security energy efficiency of cognitive Massive-MIMO-NOMA with two-way relay based on the single-objective optimal solution transmission power \(P_{R}\), power allocation \(a_{k}\) and the number of antennas \(M\). The main details of the multi-objective iterative algorithm are shown in Table 2

Table 2 Algorithm 2 multi-objective iterative algorithm (MOIA) 4. Simulation result
Full size table

6 Simulation Result

In this section, the simulation results are designed to evaluate the performance of the cognitive massive MIMO-NOMA proposed in this paper. The set of simulation parameters designed in this paper as shown in Table 3. The channel matrices are modeled as \({\mathbf{H}}_{R} = {\mathbf{G}}_{R} {\mathbf{D}}_{R}^{1/2}\), where \({\mathbf{G}}_{R}\) is small-scale fading matrices having i.i.d. elements associated with the pdf CN(0,1), while the large-scale fading matrices \({\mathbf{D}}_{R}\) similar to [2].

Table 3 Simulation parameter setting
Full size table

In Fig. 2, we evaluate the security energy efficiency of the system using the orthogonal antenna selection (OAS) algorithm and the max–min–max (AIA) algorithm with different relay antenna number M. As expected, in the region of a large number of antennas M, the AIA algorithm can achieve higher security energy efficiency values. This is because when the relay node is equipped with a small number of antennas, the influence of intra-cluster interference, imperfect CSI, and the noise between NOMA users is large. It is also very obvious from Fig. 2 that the existence of imperfect channel state information will affect the realization of the system’s security energy efficiency.

Fig. 2
figure 2

The security energy efficiency vs. the number of relay antennas

Full size image

Figure 3 illustrates the sum rate versus the transmit power of TWR with different number of cognitive users. From Fig. 3, we can observe that until the transmit power \(P_{R}\) reaches a certain value, as the TWR transmit power increases, the sum rate of the cognitive users increases. This is because although increasing the transmission power will improve the overall system sum rate, it will also increase the interference between NOMA users, which cause the values of system sum rate to stabilize. Note that in Fig. 3, the NOMA scheme is better than the OMA, the reason of which is NOMA users are less interfered by other users when decoding. Moreover, the proposed power allocation and antenna selection algorithm can obviously improve the spectral efficiency compared to the OMA.

Fig. 3
figure 3

Sum rate vs. the transmit power of TWR

Full size image

In order to verify the superiority of the large-scale antenna selection method based on TWR assistance proposed in this article in improving the performance of CR-NOMA systems, a comparative analysis was conducted with the relay selection CR-NOMA system (RS CR-NOMA) in [42]. The results are shown in Fig. 4. It can be seen that as the transmission power increases, the advantages of the proposed method in improving system throughput. Since our method not only considers the use of TWR to assist users in transmission, but also introduces large-scale antennas to enhance the overall throughput capacity of the system. In addition, it also considers the interference that large-scale antennas may cause to the system, and proposes an antenna selection mechanism. Therefore, the method proposed in this article has obvious application prospects in large-scale IOT access scenarios compared to other CR-NOMA system.

Fig. 4
figure 4

Security energy efficiency vs. the transmit power of TWR

Full size image

Figure 5 shows the security energy efficiency versus channel estimation error with different transmission power. One can be observe that large channel estimation error will reduce the security energy efficiency of the system. It can be explained that larger estimation error will reduce the achievable rate of \(C_{k}\), resulting in lower security performance. It also be seen the security energy efficiency of TWR is superior to the OWR.

Fig. 5
figure 5

Security energy efficiency vs. channel estimation error

Full size image

Figure 6 shows the relationship between security sum rate and minimum transmission rate requirements for cognitive NOMA users. It can be see that the SSR decreases with Rm increases, the reason of which is that in order to meet the increasing minimum transmission rate Rm requirements, the relay transmitter needs to allocate more transmission power to the NOMA users with poor channel conditions, which affects the realization of the higher security sum rate. Further, as \(R_{m}\) becomes very large, the security sum rate approaches zero, since limited by the transmission power \(P_{R}\), When \(R_{m}\) increases to a certain value, which cannot satisfy the QoS requirements for all the users.

Fig. 6
figure 6

Security sum rate vs. the Rm for different numbers of users

Full size image

In this paper, to analyze the robustness of the proposed method, we consider NOMA adopts an incomplete serial interference cancellation (iSIC) method during decoding, that is, user k is interfered by all other NOMA users during decoding, and received SINR of k-th SU can be rewritten as

$$\gamma_{k} = \frac{{P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{k} } \right|^{2} a_{k} }}{{\sum\nolimits_{i = k + 1}^{K} {\kappa_{i} } + \sum\nolimits_{i = k + 1}^{K} {\kappa_{i} } \iota_{i} + \left( {P_{R} \sigma_{e}^{2} \sum\nolimits_{i = k + 1}^{K} {\left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} a_{i} } + \sigma_{k}^{2} } \right)}},$$
(54)

where the first summation in (54) is the non-removable interference of the weaker users \(\kappa_{i} = P_{R} \left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} a_{i}\), while the second term is the residual interference caused by the SIC inefficiency, with influence factors for iSIC \(0 \le \iota_{i} \le 1\). The channel error plus hardware distortion noise power is denoted by \((P_{R} \sigma_{e}^{2} \sum\nolimits_{i = 1}^{K} {\left| {{\hat{\mathbf{h}}}_{Rk} {\mathbf{w}}_{i} } \right|^{2} a_{i} } + \sigma_{k}^{2} )\).

Based on the above imperfect serial interference cancellation method for system CR-NOMA system interference, the robustness of the proposed scheme in this paper is verified, and compared with the method in [42], and the results are shown in Fig. 7.

Fig. 7
figure 7

Security energy efficiency vs. influence factors for iSIC

Full size image

Figure 7 shows the relationship between security energy efficiency and influence factors for iSIC. It can be see that with the increase of interference factors, the security energy efficiency of the system continue to decrease. Obviously, because introducing the interference will cause a decrease in system throughput, thereby affecting system performance. At the same time, it can be seen that the system performance has indeed decreased after considering robustness, but the decrease is not significant. In addition, the method proposed in this article has better robustness compared to the method mentioned in reference [42]. This is because this article uses TWR method for user information exchange and improves the system transmission performance through large-scale antennas, thereby ensuring its robustness.

7 Conclusions

In this paper, in order to better meet the needs of IoT large-scale access, we focus on edge cloud environments to study a two-way relay CR-NOMA system with large-scale antennas, in which in order to complete the exchange of information between multiple pairs of cognitive users, TWR is equipped with massive antenna. The security energy efficiency is proposed to evaluate the performance of the CR-NOMA system. In order to reduce system complexity and power consumption, a new orthogonal antenna selection algorithm is proposed. We further proposed the MOIA algorithm to jointly optimize the transmission power, power distribution and the number of transmission antennas to maximize the security energy efficiency. According to the analysis results, the performance of massive MIMO CR-NOMA outperformed the MIMO. Next, the relay cooperative transmission system based on mobile edge computing is also the focus of future research, and at the same time, we will explore applying the proposed system to practical wireless environments.