1 Introduction

Multiuser communication is implemented for more efficient and higher data rate transmission in cellular communication systems. Cooperative communication is also used to provide quality of service (QoS) for multiple users in these systems through using multiple relays. Increasing transmitter power may lead to interference among users. Thus to overcome this problems, it is necessary to use cooperative communications to increase coverage and capacity and overcome the multipath fading effect without increasing of transmitter power.

Recently, cooperative communication has attracted much interest considering higher data rate in high access to the internet, a huge number of mobile applications, the high rate of watching videos, social media, 4G and 5G requirements and other massive factors [1]. Orthogonal frequency division multiple access (OFDMA) system using cooperative scheme has been presented in long term evolution (LTE) Advanced in [2,3,4,5]. To reduce the interference, transmission power must be limited at both base stations and relays [6, 7]. In [8, 9], linear precoders broadcasting to given multiple-input multiple-output (MIMO) receivers and using signal-to-noise plus interference (SINR) constraint are employed. For the uplink, linear minimum mean-squared error (LMMSE) precoding/decoding method is applied in [10] while for the downlink, the similar strategy is used in [11, 12]. The challenges in relay allocations have been mainly discussed in [13,14,15] in which outage probability is assumed as a constraint for single-user communication to choose best relay for transmission/reception or to improve QoS parameters. Multiuser cooperative wireless networks with single-carrier frequency division multiple access (SC-FDMA) at the source or destination is presented in [16]. Relay selection and power allocation have been studied for cooperative system in [17] while throughput analysis in energy harvesting systems is discussed in [18]. A parallel relay network is considered in [19] in which data rate maximization is analyzed. Using a pair of transmitter–receiver and single antenna as a relay, peer-to-peer communication is analyzed in [20, 21]. Joint iterative power allocation and interference cancellation in spread spectrum networks is presented in [22] in which multiple hops and the amplify-and-forward (AF) relaying strategy are applied for both uplink and downlink. A multiuser detection strategy using joint estimation of spatial signatures, channel gains and transmitted symbols of all users is suggested in [23]. The methodology is obtained from a tri-linear tensor model for the received signal that combines the source–destination and relay-destination links. In [24,25,26,27], a wireless communication scheme has been considered with multiple source–destination pairs communicating through several cooperative AF relay terminals equipped with multiple antennas. In [28,29,30,31,32,33,34,35,36,37,38], the effect of channel state information feedback on outage analysis in multiuser cooperative communication system is studied. Outage probability is the main QoS parameter for this communication system. For reducing the complexity of optimum ML (Maximum Likelihood) algorithm in multiuser detection problem, a GA-aided MMSE multiuser detection is proposed for employment in a SDMA-OFDM system in [39], which is capable of achieving a similar performance to that attained by optimum ML especially with large number of users. A joint swarm intelligence algorithm as sub-optimum algorithm called ant colony and particle swarm optimization (AC-PSO) has been stated in [40]. AC-PSO is the combination of PSO and ant colony optimization (ACO) proposed for multiuser detection in this paper and an acceptable improvement has been obtained. In [41], a sophisticated nonlinear multiuser detection is devised which exploits the power of genetic algorithm. Other sub-optimum multiuser detectors such as decision-level data fusion technologies [42] and opportunistic multiuser detection [43, 44] have been analyzed in recent years.

In this paper a novel method for multiuser detection in cooperative communication system has been suggested. As known, ML optimum method for multiuser detection is not applicable (because of high complexity) and thus, sub-optimum methods must be presented. In other side, the MUD (multiuser detection) problem is NP-complete hard problem with many variables due to many users of the system. Our method, modified dimension-reduction genetic algorithm (MDR-GA), divides the problem into multiple simpler problems with less parameters such that in first steps, more important variables should be optimized. It is noticeable that because of cooperative state, the transmitter power limitation and SINR condition must be considered in fitness function optimization that makes the problem more difficult. The constraint of SINR should be considered to guarantee the quality of the communication system and the cooperative strategy is assumed AF scheme in this paper. It is shown that the performance of the suggested method is similar to the optimum ML detection approach that is the benchmark for multiuser detection methods. In addition, the QoS of the cooperative communication system is preserved using the proposed technique. The effect of cooperative relay nodes and genetic algorithm parameters are discussed and analyzed in this paper.

The rest of this paper is organized as follows. Section 2 discusses the proposed multiuser cooperative system; Sect. 3 states the steps used to realize the suggested algorithm. Simulation results are stated in Sect. 4 and finally, some conclusions are drawn in Sect. 5.

2 System model

Consider a MIMO link with M transmitter and receiver antennas. Without loss of generality, the number of transmitter users is assumed to be equal to receiver users. K cooperative relay nodes are also the interface of this communication. The baseband model of this system is depicted in Fig. 1.

Fig. 1
figure 1

Cooperative multiuser detection model

Full size image

Considering the relays, their received signals are stated as below:

$$\varvec{y} = \varvec{H}_{{\varvec{SR}}}^{\varvec{T}} \varvec{s} + \varvec{n}_{\varvec{R}}$$
(1)

where \(\varvec{H}_{{\varvec{SR}}}^{\varvec{T}}\) is the transpose of the channel matrix between source nodes and relay ones with size M  \(\times\)  K; \(\varvec{y}\) is the received signal vector of relay nodes with size K  \(\times\)  1 and \(\varvec{n}_{\varvec{R}}\) is the Gaussian noise vector in relay nodes (K  \(\times\)  1 vector). The transmitted signal of source nodes is represented by the vector \(\varvec{s}\) (M  \(\times\)  1 vector) that is defined as below:

$$s = \left[ {\sqrt {p_{s1} } b_{1} , \sqrt {p_{s2} } b_{2} , \ldots , \sqrt {p_{sM} } b_{M} } \right]$$
(2)

where \(p_{si}\) and \(b_{i}\) are i-th user power and data bit (by assuming BPSK modulation) of transmitted signal, respectively.

The received signal in multiuser receivers is given as below:

$$\varvec{r} = \left( {AF} \right)\varvec{H}_{{\varvec{RD}}}^{\varvec{T}} \varvec{y} + \varvec{n}_{\varvec{D}} .$$
(3)

In this equation, r is the received signal vector of M receiver nodes with size M  \(\times\)  1 and \(\varvec{H}_{{\varvec{RD}}}\) is the channel matrix between relays and receivers by size K  \(\times\)  M. AF is the scalar amplification factor of Amplify and Forward cooperative strategy of relays assumed in this paper (equal for all relays) and \({\mathbf{y}}\) is the received signal vector in Eq. (1). \(\varvec{n}_{\varvec{D}}\) is also the Gaussian noise vector in receivers with size M  \(\times\)  1.

In multiuser communication systems, Based on [45], to select the maximum likelihood sequence of transmitted data bits b, the below log likelihood function (LLF) must be maximized:

$$f\left( \varvec{b} \right) = \varvec{b}^{\varvec{T}} \varvec{r} - \varvec{b}^{\varvec{T}} {\varvec{\Omega}}\varvec{b}\varvec{.}$$
(4)

And the transmitted optimized vector can be stated as below due to [45]:

$$\hat{\varvec{b}} = arg\left\{ {\mathop {\hbox{max} }\limits_{\varvec{b}} \left[ {f\left( \varvec{b} \right)} \right]} \right\}.$$
(5)

\(f\left( \varvec{b} \right)\) is the scalar that must be maximized by selecting the best data stream of transmitter vector, \(\hat{\varvec{b}}\). \(\varvec{r}\) is the received signal as in Eq. (3) and \({\varvec{\Omega}}\) is the cross correlation matrix which shows interaction or interference among different users. This optimization problem has no constraints in cooperative multiuser communication system. Due to the discussions in previous section, we have used two constraints on this problem. First constraint is the transmitter power of each user which must be limited to the predefined value as \(P_{th }\) and the second constraint is the satisfied SINR in each receiver which must be set higher than the outage threshold. Thus, the global optimization can be drawn as follow:

$$\begin{aligned} & f\left( \varvec{b} \right) = \varvec{b}^{\varvec{T}} \varvec{r} - \varvec{b}^{\varvec{T}} {\varvec{\Omega}}\varvec{b} \\ & s.t. P_{{s_{i} }} \le P_{th } and SINR_{i} \ge \gamma \\ & \quad \quad i = 1,2, \ldots ,M \\ \end{aligned}$$
(6)

where \(\gamma\) is the outage SINR threshold. Due to the model of transmitters, relays and receivers, the SINR can be defined as:

$$SINR = \frac{{p_{signal} }}{{p_{interfer} + p_{noise} }}.$$
(7)

Thus, substituting \(\varvec{y}\) from (1–3), we have:

$$\varvec{r} = \left( {AF} \right)\varvec{H}_{{\varvec{RD}}}^{\varvec{T}} \left( {\varvec{H}_{{\varvec{SR}}}^{\varvec{T}} \varvec{s} + \varvec{n}_{\varvec{R}} } \right) + \varvec{n}_{\varvec{D}} .$$
(8)

Before obtaining the SINR, below assumptions can be used for simplicity:

  1. (a)

    All relay’s power gains are equal

  2. (b)

    The matrixes \(\varvec{H}_{{\varvec{RD}}}\) and \(\varvec{H}_{{\varvec{SR}}}\) are symmetric i.e.:

    $$\begin{aligned} \varvec{H}_{RD,ij} & = \varvec{H}_{RD,ji} \\ \varvec{H}_{SR,ij} & = \varvec{H}_{SR,ji} . \\ \end{aligned}$$
    (9)

In consequence, without loss of generality, the SINR parameter is calculated for user 1 that is assumed as desired user and other users (user 2 to user M) are considered as interfering users. Equation (8) can be rewritten as below that the first term includes desired signal and interference signals while the second and third terms are related to relay and receiver noises as:

$${\varvec{r}} = (AF) {\varvec{H}}_{\varvec{RD}}^{\varvec{T}} {\varvec{H}}_{\varvec{SR}}^{\varvec{T}} {\varvec{s}} + (AF) {\varvec{H}}_{\varvec{RD}}^{\varvec{T}} {\varvec{n}}_{\varvec{R}} + {\varvec{n}}_{\varvec{D}} .$$
(10)

Extending the first term for user 1 as desired signal and other users as interfering signals, we have:

$$\begin{aligned} r_{1} & = \left( {AF} \right)H_{RD,11} \left( {\sqrt {P_{{R_{1} }} } H_{SR,11} \sqrt {P_{{s_{1} }} } b_{1} + \sqrt {P_{{R_{2} }} } H_{SR,12} \sqrt {P_{{s_{2} }} } b_{2} } \right. \\ & \left. {\quad + \ldots + \sqrt {P_{{R_{M} }} } H_{SR,1M} \sqrt {P_{{s_{M} }} } b_{M} } \right) \\ & \quad + \ldots + \left( {AF} \right)H_{RD,1K} \left( {\sqrt {P_{{R_{1} }} } H_{SR,K1} \sqrt {P_{{s_{1} }} } b_{1} + \sqrt {P_{{R_{2} }} } H_{SR,K2} \sqrt {P_{{s_{2} }} } b_{2} } \right. \\ & \quad \left. { + \ldots + \sqrt {P_{{R_{M} }} } H_{SR,KM} \sqrt {P_{{s_{M} }} } b_{M} } \right) + \left( {AF} \right)H_{RD,11} \sqrt {P_{{R_{1} }} } n_{R1} \\ & \quad + \ldots + \left( {AF} \right)H_{RD,1K} \sqrt {P_{{R_{1} }} } n_{RK} + n_{D1} . \\ \end{aligned}$$
(11)

Therefore due to above equation, the total power of desired signal of user 1 is as below:

$$P_{signal} = \left( {AF} \right)^{2} \left| {b_{1} } \right|^{2} P_{{s_{1} }} P_{{R_{1} }} \mathop \sum \limits_{j = 1}^{K} H_{RD,1j}^{2} H_{SR,j1}^{2} .$$
(12)

where \(| |\) is used for the norm of the matrix. The total power of interference signals for user 1 can be written as:

$$P_{interfer} = \left( {AF} \right)^{2} \mathop \sum \limits_{p = 2}^{M} P_{{s_{p} }} P_{{R_{p} }} \left| {b_{p} } \right|^{2} \mathop \sum \limits_{j = 1}^{K} H_{RD,1j}^{2} H_{SR,jp}^{2} .$$
(13)

And the total noise power of the receiver is obtained as:

$$P_{noise} = \left( {AF} \right)^{2} P_{R} \sigma_{nR}^{2} \mathop \sum \limits_{j = 1}^{K} H_{RD,1j}^{2} + \sigma_{nD}^{2} .$$
(14)

Finally due to (12–14) and replacing in (7), SINR for user 1 is obtained as below:

$$SINR_{1} = \frac{{\left( {\left| {b_{1} } \right|^{2} P_{{s_{1} }} P_{{R_{1} }} \mathop \sum \nolimits_{j = 1}^{K} H_{RD,1j}^{2} H_{SR,j1}^{2} } \right)}}{{\mathop \sum \nolimits_{p = 2}^{M} P_{{s_{p} }} P_{{R_{p} }} \left| {b_{p} } \right|^{2} \mathop \sum \nolimits_{j = 1}^{K} H_{RD,1j}^{2} H_{SR,jp}^{2} + P_{R} \sigma_{nR}^{2} \mathop \sum \nolimits_{j = 1}^{K} H_{RD,1j}^{2} + \sigma_{nD}^{2} /\left( {AF} \right)^{2} }}.$$
(15)

Now due to the conditional optimization problem and its constraints, Lagrange approach must be used to obtain the optimum point of the problem. Therefore, we define the Lagrange function J as below:

$$J = \varvec{b}^{\varvec{T}} r - \varvec{b}^{\varvec{T}} {\varvec{\Omega}}\varvec{b} + \lambda \left( {P_{{s_{i} }} - P_{th } } \right) + \mu \left( {SINR_{i} - \gamma } \right).$$
(16)

where \(\lambda ,\mu\) are Lagrange coefficients. Using previous equations and SINR obtained in (15), we have:

$$J = \varvec{b}^{\varvec{T}} r - \varvec{b}^{\varvec{T}} {\varvec{\Omega}}\varvec{b} + \lambda \left( {P_{{s_{1} }} - P_{th } } \right) + \mu \left( {\frac{{\left( {\left| {\varvec{b}_{1} } \right|^{2} P_{{s_{1} }} P_{{R_{1} }} \mathop \sum \nolimits_{j = 1}^{K} H_{RD,1j}^{2} H_{SR,j1}^{2} } \right)}}{{\mathop \sum \nolimits_{p = 2}^{M} P_{{s_{p} }} P_{{R_{p} }} \left| {\varvec{b}_{\varvec{p}} } \right|^{2} \mathop \sum \nolimits_{j = 1}^{K} H_{RD,1j}^{2} H_{SR,jp}^{2} + P_{R} \sigma_{nR}^{2} \mathop \sum \nolimits_{j = 1}^{K} H_{RD,1j}^{2} + \sigma_{nD}^{2} /\left( {AF} \right)^{2} }} - \gamma } \right)$$
(17)

Now the optimum point is obtained after derivation of (17) with respect to \(\varvec{b}\) that is NP-complete (Nondeterministic Polynomial time) hard problem. Because of this fact, evolutionary algorithm can be applied to obtain the optimum point. Thus in this paper, a novel modified genetic algorithm is chosen to gain this purpose considering the specified constraints.

3 Proposed modified dimension-reduction genetic algorithm (MDR-GA)

In the proposed strategy to provide the optimum detected data, the MDR-GA is applied. In the evolutionary computation literatures using basic genetic algorithms, the large number of parameters may cause the algorithm to be more complex and to fall in the local minimum/maximum instead of global ones. Thus, suggestion of dimension-reduction approach can be precious to use the evolutionary algorithms in optimizing the NP-complete hard problem. As we know, the number of users simultaneously communicate with each other grows increasingly in cellular systems and the dimension of this problem for multiuser detection becomes very large. Thus, it is important to use dimension-reduction algorithms to achieve the optimum point.

It is obvious that the performance of optimization algorithm is improved while reducing the number of parameters. The probability of achieving the optimum point reduces when the number of the parameters increases as elaborated more in [46] for the sample function \(F\left( {\vec{x}} \right) = 0.5 + \left( {sin^{2} \left| {\vec{x}} \right| - 0.5} \right)e^{{ - 0.2\left| {\vec{x}} \right|}} , \vec{x} = \left( {x_{1} ,x_{2} , \ldots ,x_{l} } \right)\) in two cases \(\vec{x} = x_{1}\) and \(\vec{x} = (x_{1} ,x_{2} )\). It is verified that by reducing the number of parameters, the chance of optimum point finding is reduced exponentially. Our proposed method is a modified dimension-reduction genetic algorithm (MDR-GA) in which complex multiuser optimization problem is converted to some simpler problems with smaller number of users.

The main idea of MDR-GA is that the optimization is done firstly for main parameters. This means that in multiuser problem, the users with larger interference signals are used for making initial population vector of genetic algorithm. Thus, the optimization process is firstly done for these main users as the first generations. In the next generations and iterations, the main parameters are set fixed and other parameters (users) are inserted in the optimization process. The main difference of the MDR-GA compared to conventional GA is that, the algorithm is applied on several number of the parameters while the other parameters are fixed in each iteration. The set of first optimization parameters are selected based on the mathematical criterion analyzed in the sequel.

Considering Eq. (6), the matrix \({\varvec{\Omega}}\) is the correlation matrix and the main diagonal represents the auto correlation of desired signal while non-diagonal elements describe the correlation values of other users on the desired user. This can be a criterion to select the more correlated parameters first to be optimized. The reason of this selection is that the users with more cross-correlation value show more interference on the desired user and consequently the maximization of fitness function is more critical with existence of these users. Due to this fact, the criterion is defined as:

$$id_{i} = \mathop \sum \limits_{j = 1,j \ne i}^{M} abs\left( {{\varvec{\Omega}}_{ij} } \right), i = 1,2, \ldots ,M$$
(18)

where \(M\) is the number of users and i is the desired user index. Any parameter (user) with higher \(id\) is a more substantial variable. As stated previously, a number of variables (assume \(h\)) are firstly designated for optimization in the suggested algorithm, and the rest (\(M - h\)) are fixed at the earlier value. In the next step, a number of chromosomes (population) are created using these \(h\) variables, and the GA is applied on this population once. Next, the best response is kept as the new values of these \(h\) variables, and this procedure continues again. The flowchart is provided in Fig. 2.

Fig. 2
figure 2

The flowchart of MDR-GA

Full size image

4 Simulation results

In this section the simulation results are discussed. Simulation parameters for investigation of the proposed algorithm in cooperative multiuser communication system is given in Table 1.

Table 1 Simulation parameters
Full size table

First, the cooperative advantage to reduce the transmitter power is analyzed. In Fig. 3, the transmitter power versus satisfied SINR for different number of relay antennas is depicted. As seen when the number of relay nodes is 4, the transmitted power is high i.e. the interference among the different users is substantially increased. While using 6 or 8 number of relays, the transmitted power is decreased noticeably. Thus due to using of cooperative diversity gains, the transmitted power is significantly decreased. In this figure, the number of transmitter and receiver users is fixed to 6 (the user number is fixed for fair condition of simulation to investigate the cooperative system).

Fig. 3
figure 3

The effect of increasing relay numbers on transmitted power

Full size image

In Fig. 4, the effect of increasing relay numbers on the convergence rate of MDR-GA is presented. It is observable that by using more number of relays, the iterations needed for MDR-GA convergence are decreased. This fact is because of cooperative advantage of the relay nodes and their diversity gains. When the relay number increases, the minimum qualified SINR is sooner achieved because more relays cause more SINR and better received signal. Thus the MDR-GA converges faster than the case we use fewer relay nodes.

Fig. 4
figure 4

The effect of relay antenna numbers on MDR-GA convergence speed

Full size image

In Fig. 5, the effect of using multi-antenna relay nodes on transmitted power is tested. It is interesting that we can obtain more advantage of cooperative diversity using more than one antenna in each relay. Figure 5 shows this work by using two and three antennas instead of one in each relay and it is seen with this extended diversity, the transmitted power is also decreased. Note that for better performance, MRC (Maximum Ratio Combining) algorithm is also used in two and three antennas case to select the best signal. It is noticeable that the difference between two-antenna case and three-antenna one is negligible and this shows that using antennas more than two or three numbers in the relay nodes is not effective on the system performance due to the complexities of this work, multiple antennas management and their costs.

Fig. 5
figure 5

The effect of using more than one antenna in relay nodes on transmitter power

Full size image

The proposed algorithm’s performance for data detection that is near the ML optimum multiuser detector is verified in Fig. 6. In this figure, BER parameter versus SINR is discussed and the proposed MDR-GA is compared with other efficient approaches in this field. As shown, the result of suggested algorithm is interesting related to other algorithms and shows the better performance compared to other methods.

Fig. 6
figure 6

The proposed algorithm’s performance compared to other efficient method and optimum ML detector

Full size image

The number of users surely influences the system performance in multiuser cooperative communication systems and it is important that the proposed method must be robust to user number increasing. This effect is studied in Fig. 7. It is obviously understood that increasing the user numbers, the performance is degraded due to interference yielded. Due to Fig. 7, the proposed method is more robust to this phenomenon relative to other algorithms compared because of using dimension-reduction characteristic of MDR-GA.

Fig. 7
figure 7

The proposed algorithm’s performance versus number of users compared to other efficient method and optimum ML detector

Full size image

In Fig. 8, the proposed method for various population sizes of proposed MDR-GA is analyzed. As seen from this figure, MDR-GA with more population size has more interesting results in comparison to other cases in which the flexibility and improvability of the proposed algorithm has been shown. Of course, it must be noted that with more population size, the complexity of the proposed method is increased that is not interesting. As mentioned above, the algorithm with proposed MDR-GA results in less BER near to optimum ML method.

Fig. 8
figure 8

The BER of proposed method with various population size of proposed GA

Full size image

5 Conclusion

In this paper, MDR-GA strategy is proposed for optimization of cooperative multiuser detection problem. Multiuser detection in cooperative communication is an NP-complete hard problem in which some constraints such as transmitter power’s limitation and SINR threshold must be satisfied. MDR-GA with the approach of Lagrange multipliers method is used for optimum detection of users’ data while satisfying the constraints of the problem. Using MDR-GA, the problem is divided into some simpler problems to be optimized more conveniently. MDR-GA is applied to reduce problem dimension and obtain near-optimum detection of users’ data with little BER. The proposed method’s performance is near the optimum ML detection method as shown in simulation results. In cooperative scheme, low transmitter power and satisfied SINR can be guaranteed and thus, the interference among users is desirably limited in cellular communication systems. Moreover, applying more relays in cooperative system can help the system to be improved in terms of lower required transmitter power and less number of iterations of proposed MDR-GA for convergence.