1 Introduction

For integrated sensing and communication (ISAC), one important research direction is to employ various beamforming techniques to avoid interference between the two functions [1]. Compared to beamforming, another technique which can form directional transmission is directional modulation (DM). DM is a physical layer security technique that scrambles the constellation mappings in undesired directions while keeping them in the desired direction(s) [2,3,4,5]. In [6], a four-dimensional array was used to transmit a digitally modulated signal to a predetermined direction of interest, while transmitting a time-modulated signal in other directions to distort the signal. In [7], the concept of directional modulation was adopted to enhance the security of multi-user multi-input multi-output (MIMO) communication systems in the presence of multi-antenna eavesdroppers through symbol-level precoding. The security of legitimate users in DM with random frequency diverse array (DM-RFDA) was studied in [8]. In [9], a multi-direction DM scheme was proposed according to channel conditions of each user. Interestingly, the two topics of DM and ISAC have converged recently by considering the implementation of radar target tracking using DM transmitters [10, 11], where the receive bit error rate (BER) was used as an indicator of the presence of targets in the angular domain.

However, existing research in DM is normally applied to a fully digital structure, with each antenna connected to a digital-to-analog converter (DAC), which incurs high-level power consumption and hardware cost, in particular in the case of massive MIMO and mmWave communication for the next-generation communication systems [12]. Therefore, a hybrid structure with a combination of digital beamforming and analog beamforming techniques has been proposed, which greatly reduces the number of required DACs. Various hybrid beamforming structures have been studied in the past, including fully connected structures and partially connected structures [13,14,15,16]. In [17], performance of these two structures in a multi-user multiple-input single-output (MU-MISO) system was studied under the same total transmit power constraint, and it is shown that the performance difference between them is small when the number of users is small. The partially connected structures require fewer phase shifters and are easier to implement than the fully connected structures, and in this work we choose this class of structures as the basis for our DM design. The localized architecture and the interleaved architecture are two main types of implementation for partially connected structures. For a uniform array, the antennas of the same subarray are adjacent in the localized structure, and the antennas of different subarrays are interleaved to form the interleaved subarray structure [18,19,20]. Since the antennas of each subarray of the interleaved structure are distributed over a larger aperture, a narrower beam can be formed.

Recently, the multi-beam multiplexing problem was studied based on the hybrid beamforming structure, where multiple separate user beams can be transmitted to different directions by the array through a common set of analog coefficients [21,22,23]. In this work, combining beamforming and DM for a more effective directional transmission with low sidelobes, the multi-beam multiplexing DM design problem is studied based on a uniform linear array (ULA), by employing an interleaved structure and a localized structure, respectively. Considering the limited resolution of the phase shift part of the analog beamforming stage in the hybrid structures, i.e., only a limited number of discrete phase shifters are available [24,25,26], an iterative optimization algorithm with lower complexity than the exhaustive search method is proposed. Through comparison of design examples, it is shown that the interleaved subarray structure can produce a narrower mainlobe and a lower sidelobe level than the localized subarray structure.

The remaining part of this paper is structured as follows. The DM designs incorporating the beamforming requirement based on an interleaved subarray structure and a localized subarray structure are formulated in Sect. 2. The iterative optimization algorithm for multi-beam multiplexing DM design is proposed in Sect. 3 considering finite solution in phase shift. Design examples are provided in Sect. 4, followed by conclusions in Sect. 5.

2 DM designs based on hybrid beamforming structures

A linear interleaved subarray structure with uniformly distributed antennas is shown in Fig. 1, which is divided into N subarrays, and each subarray is connected to a DAC. Assuming that each subarray has M antennas, the distance from the zeroth antenna to the l-th antenna is \(d_{l}\) \((l = 1, \ldots , MN-1)\). Then, the antenna array aperture of the structure is \(d_{MN-1}=(MN-1)d_1\). The steering vector of the n-th subarray can be expressed as

$$\begin{aligned} \begin{aligned} {\textbf{s}}_{n}(\omega ,\theta )&= \left[ e^{j\omega nd_{1}\sin \theta /c}, e^{j\omega (n+N)d_{1}\sin \theta /c}, \ldots , e^{j\omega (n+N(M-1))d_{1}\sin \theta /c}\right] ^{T}, \\ \text{ for }\quad \,\,\, n&= 0, \ldots , N-1\\ \end{aligned} \end{aligned}$$
(1)

where c is the speed of propagation, \(\omega\) is the angular frequency, \(\theta \in [0^{\circ }, 180^{\circ }]\) is the transmission angle, and \(\{ \cdot \}^{T}\) is the transpose operation. Suppose there are R transmission angles that are divided into the desired and undesired ones, where the number of desired angles is r. The corresponding steering vectors \({\textbf{s}}_{n,{h}}\) and \({\textbf{s}}_{n,{l}}\) can be expressed as

$$\begin{aligned} \begin{aligned} {\textbf{s}}_{n,{h}}(\omega ,\theta _{h})&= \left[ e^{j\omega nd_{1}\sin \theta _{h}/c}, e^{j\omega (n+N)d_{1}\sin \theta _{h}/c}, \ldots , e^{j\omega (n+N(M-1))d_{1}\sin \theta _{h}/c}\right] ^{T},\\ {\textbf{s}}_{n,{l}}(\omega ,\theta _{l})&= \left[ e^{j\omega nd_{1}\sin \theta _{l}/c}, e^{j\omega (n+N)d_{1}\sin \theta _{l}/c}, \ldots , e^{j\omega (n+N(M-1))d_{1}\sin \theta _{l}/c}\right] ^{T},\\ \text{ for }\quad \,\,\, h&= 0, \ldots , r-1,\ l = r, \ldots , R-1. \end{aligned} \end{aligned}$$
(2)

The localized structure obtained by rearranging the subarrays of the interleaved structure is shown in Fig. 2, and the corresponding steering vectors can be obtained by replacing the mathematical representation of the array position in (2).

The steering matrices \({\textbf{S}}_{n,\textrm{main}}\) and \({\textbf{S}}_{n,\textrm{side}}\) corresponding to the n-th subarray can be expressed as

$$\begin{aligned} \begin{aligned} {\textbf{S}}_{n,\textrm{main}}&= [{\textbf{s}}_{n,{0}}(\omega ,\theta _{0}), {\textbf{s}}_{n,{1}}(\omega ,\theta _{1}), \ldots , {\textbf{s}}_{n,{r-1}}(\omega ,\theta _{r-1})],\\ {\textbf{S}}_{n,\textrm{side}}&= [{\textbf{s}}_{n,{r}}(\omega ,\theta _{r}), {\textbf{s}}_{n,{r+1}}(\omega ,\theta _{r+1}), \ldots , {\textbf{s}}_{n,{R-1}}(\omega ,\theta _{R-1})], \end{aligned} \end{aligned}$$
(3)

The digital coefficient corresponding to the n-th subarray is a complex number \(w_{D,n}\). The digital beamforming vector can be expressed as

$$\begin{aligned} {\textbf{w}}_{D} = [w_{D,0}, w_{D,1}, \ldots , w_{D,N-1}]. \end{aligned}$$
(4)

The analog weighting factors corresponding to the n-th subarray can be written as

$$\begin{aligned} \begin{aligned} {\textbf{w}}_{A,n}&=[e^{j\beta _{A,n,0}},e^{j\beta _{A,n,1}},\ldots , e^{j\beta _{A,n,M-1}}], \end{aligned} \end{aligned}$$
(5)

where

$$\begin{aligned} \beta _{A,n,m} \in [0,2\pi ], \quad \text{ for }\ m=0,1,\ldots ,M-1. \end{aligned}$$
(6)

For B-ary signaling in DM, the b-th set of weight factors \({\textbf{w}}_{b,A,n}\) can be written as

$$\begin{aligned} \begin{aligned} {\textbf{W}}_{b,A}&=\left[ \begin{array}{c} {\textbf{w}}_{b,A,0}\\ {\textbf{w}}_{b,A,1}\\ \ldots \\ {\textbf{w}}_{b,A,N-1}\end{array}\right] ,\\ \text{ with }\quad {\textbf{w}}_{b,A,n}&=[e^{j\beta _{b,A,n,0}},e^{j\beta _{b,A,n,1}},\ldots , e^{j\beta _{b,A,n,M-1}}] \end{aligned} \end{aligned}$$
(7)

where \(b = 0, \ldots , B-1\) and \(\beta _{b,A,n,m} \in [0,2\pi ]\). The b-th set of digital coefficient vector is given by

$$\begin{aligned} \begin{aligned} {\textbf{w}}_{b,D}&=[w_{b,D,0}, w_{b,D,1}, \ldots , w_{b,D,N-1}]. \end{aligned} \end{aligned}$$
(8)

The design response for each symbol at R transmission angles can be expressed as

$$\begin{aligned} \begin{aligned} {\textbf{p}}_{b,\textrm{main}}&= [p_{b}(\omega ,\theta _0), p_{b}(\omega ,\theta _1), \ldots , p_{b}(\omega ,\theta _{r-1})],\\ {\textbf{p}}_{b,\textrm{side}}&= [p_{b}(\omega ,\theta _r), p_{b}(\omega ,\theta _{r+1}), \ldots , p_{b}(\omega ,\theta _{R-1})].\\ \end{aligned} \end{aligned}$$
(9)

To transmit the desired signal in the desired direction and the scrambled signal in the undesired direction, the optimization of the set of digital and analog coefficients for the DM design of the b-th symbol can be formulated as

$$\begin{aligned} \mathop {{\text{min}}}\limits_{{{\mathbf{w}}_{{b,D}} ,{\mathbf{W}}_{{b,A}} }} & \left\| {{\mathbf{p}}_{{b,{\text{side}}}} - \sum\limits_{{n = 0}}^{{N - 1}} {{\mathbf{w}}_{{b,D}} } (n){\mathbf{W}}_{{b,A}} (n,:){\mathbf{S}}_{{n,{\text{side}}}} } \right\|_{2} \\ {\text{subject to}} & \quad \sum\limits_{{n = 0}}^{{N - 1}} {{\mathbf{w}}_{{b,D}} } (n){\mathbf{W}}_{{b,A}} (n,:){\mathbf{S}}_{{n,{\text{main}}}} = {\mathbf{p}}_{{b,{\text{main}}}} , \\ \end{aligned}$$
(10)

where \(||\cdot ||_{2}\) represents the \(l_{2}\) norm. Note that the magnitude of the response \({\textbf {p}}_{b,\textrm{side}}\) in the sidelobe region is usually designed to be a low value, and the objective function means minimizing the difference with \({\textbf {p}}_{b,\textrm{side}}\). Note that there is no straightforward solution to the grating lobe problem; however, one observation is that although the individual subarrays have an adjacent spacing larger than half wavelength, the overall array after combining the subarrays together via the digital coefficients has a spacing smaller than half wavelength, which gives us some assurance that the grating lobes generated by individual subarrays could be suppressed to some degree by optimizing the digital part of the structure.

3 Proposed multi-beam multiplexing scheme based on DM design

With inter-subarray coding, N subarrays can be used for multiplexing N beams [21, 22], and DM is designed by optimizing the set of digital coefficients and the shared analog coefficients. The digital coefficient vector of the x-th beam can be expressed as

$$\begin{aligned} \begin{aligned} {\textbf{w}}_{b,D,x}&=\left[ w_{b,D,0,x}, w_{b,D,1,x}, \ldots , w_{b,D,N-1,x}\right] ,\\ \text{ for } \qquad x&=0, 1, \ldots , N-1. \end{aligned} \end{aligned}$$
(11)

Note that for B-ary signaling, there are B possible symbols transmitted by each beam in the symbol time, and for the case of N beam multiplexing, \(B^N\) groups of symbol combinations need to be considered. Combining the digital coefficient vectors of N beams into one matrix, given by

$$\begin{aligned} \begin{aligned} {\textbf{W}}_{b,D}&=[{\textbf{w}}_{b,D,0}^{T},{\textbf{w}}_{b,D,1}^{T},\ldots ,{\textbf{w}}_{b,D,N-1}^{T}]. \end{aligned} \end{aligned}$$
(12)

Without loss of generality, the mainlobe region \(\Theta _{\textrm{main},x}\) of the x-th beam includes r desired transmission angles, and the sidelobe region \(\Theta _{\textrm{side},x}\) includes \(R-r\) undesired transmission angles. The corresponding steering matrices \({\textbf{S}}_{n,\textrm{main},x}\) and \({\textbf{S}}_{n,\textrm{side},x}\) become

$$\begin{aligned} \begin{aligned} {\textbf{S}}_{n,\textrm{main},x}&= [{\textbf{s}}_{n,{0}}(\omega ,\theta _{0,x}), {\textbf{s}}_{n,{1}}(\omega ,\theta _{1,x}),\ldots , {\textbf{s}}_{n,{r-1}}(\omega ,\theta _{r-1,x})],\\ {\textbf{S}}_{n,\textrm{side},x}&= [{\textbf{s}}_{n,{r}}(\omega ,\theta _{r,x}), {\textbf{s}}_{n,{r+1}}(\omega ,\theta _{r+1,x}), \ldots ,{\textbf{s}}_{n,{R-1}}(\omega ,\theta _{R-1,x})], \end{aligned} \end{aligned}$$
(13)

where \(\theta _{h,x}\in \Theta _{\textrm{main},x}\) and \(\theta _{l,x}\in \Theta _{\textrm{side},x}\). Then, the response to the b-th symbol designed separately for the x-th beams can be written as

$$\begin{aligned} \begin{aligned} {\textbf{p}}_{b,\textrm{main},x}&= [p_{b,x}(\omega ,\theta _{0,x}), p_{b,x}(\omega ,\theta _{1,x}), \ldots , p_{b,x}(\omega ,\theta _{r-1,x})],\\ {\textbf{p}}_{b,\textrm{side},x}&= [p_{b,x}(\omega ,\theta _{r,x}), p_{b,x}(\omega ,\theta _{r+1,x}), \ldots , p_{b,x}(\omega ,\theta _{R-1,x})].\\ \end{aligned} \end{aligned}$$
(14)

Accordingly, the problem of optimizing \({\textbf{W}}_{b,D}\) and \({\textbf{w}}_{b,A}\) for multi-beam multiplexing can be formulated as

$$\begin{aligned} & \mathop {{\text{min}}}\limits_{{{\mathbf{W}}_{{b,D}} ,{\mathbf{W}}_{{b,A}} }} \sum\limits_{{x = 0}}^{{N - 1}} {\left\| {{\mathbf{p}}_{{b,{\text{side}},x}} - \sum\limits_{{n = 1}}^{{N - 1}} {{\mathbf{W}}_{{b,D}} } (n,x){\mathbf{W}}_{{b,A}} (n,:){\mathbf{S}}_{{n,{\text{side}},x}} } \right\|_{2} } \\ & \quad {\text{subject to}}\sum\limits_{{n = 0}}^{{N - 1}} {\left\| {\sum\limits_{{x = 0}}^{{N - 1}} {{\mathbf{W}}_{{b,D}} } (n,x){\mathbf{W}}_{{b,A}} (n,:)} \right\|_{2}^{2} } \le \eta , \\ & \quad \sum\limits_{{n = 1}}^{{N - 1}} {{\mathbf{W}}_{{b,D}} } (n,x){\mathbf{W}}_{{b,A}} (n,:){\mathbf{S}}_{{n,{\text{main}},x}} = {\mathbf{p}}_{{b,{\text{main}},x}} , \\ & \quad {\text{for }}\;x = 0,1, \ldots ,N - 1 \\ \end{aligned}$$
(15)

where \(\eta\) represents the total power consumption. However, the infinite precision phase shifters considered in (15) are impractical. Considering that \(\beta _{b,A,n,m}\) can only be selected from a discrete set \(F \in [0,2\pi /2^\alpha ,\ldots ,(2^\alpha -1)2\pi /2^\alpha ]\), where \(\alpha\) is an integer number. Therefore, the new optimization scheme becomes

$$\begin{aligned} & \mathop {{\text{min}}}\limits_{{{\mathbf{W}}_{{b,D}} ,{\mathbf{W}}_{{b,A}} }} \sum\limits_{{x = 0}}^{{N - 1}} {\left\| {{\mathbf{p}}_{{b,{\text{side}},x}} - \sum\limits_{{n = 1}}^{{N - 1}} {{\mathbf{W}}_{{b,D}} } (n,x){\mathbf{W}}_{{b,A}} (n,:){\mathbf{S}}_{{n,{\text{side}},x}} } \right\|_{2} } \\ & \quad {\text{subject to}} \\ & \quad \sum\limits_{{n = 0}}^{{N - 1}} {\left\| {\sum\limits_{{x = 0}}^{{N - 1}} {{\mathbf{W}}_{{b,D}} } (n,x){\mathbf{W}}_{{b,A}} (n,:)} \right\|_{2}^{2} } \le \eta , \\ & \quad \sum\limits_{{n = 1}}^{{N - 1}} {{\mathbf{W}}_{{b,D}} } (n,x){\mathbf{W}}_{{b,A}} (n,:){\mathbf{S}}_{{n,{\text{main}},x}} = {\mathbf{p}}_{{b,{\text{main}},x}} \\ & \quad \beta _{{b,A,n,m}} \in F{\text{ }} \\ & \quad {\text{for }}\begin{array}{*{20}c} {x = 0,1, \ldots ,N - 1;} & {n = 0,1, \ldots ,N - 1;} \\ {m = 0,1, \ldots ,M - 1.} & {} \\ \end{array} \\ \end{aligned}$$
(16)

The formulation (16) is non-convex due to the discrete phase value of \(\beta _{b,A,n,m}\). In order to solve this problem, we can employ an exhaustive searching method to find the objective functions of \(2^{\alpha MN}\) possible phase shift combinations of \({\textbf{W}}_{b,D}\) for the b-th symbol combination. However, when \(\alpha\) and the number of phase shifters are large, the computational complexity \(O(2^{\alpha MN})\) for each symbol combination is very high and becomes infeasible.

To solve this problem, the m-th element of the n-th subarray is optimized by fixing \(MN-1\) phase shifts, and an iterative optimization method for \({\textbf{W}}_{b,D}\) and \({\textbf{W}}_{b,A}\) is proposed. Specifically, at each iteration, the objective function value \(J_{b}(\beta _{b,A,n,m})\) of the optimized element of the corresponding b-th symbol is compared with the current minimum cost function value \(J_{b}(\min )\); if the objective function value \(J_{b}(\beta _{b,A,n,m})\) is not less than \(J_{b}(\min )\), the objective function value \(J_{b}(\min )\) will not be changed; otherwise, update \(J_{b}(\min )\) and the corresponding m-th phase shifter of n subarrays. Since \(J_{b}(\min )\) can only remain the same or be updated to a smaller value, the value of the cost function maintains a non-increasing trend and converges. Optimization can be achieved using the CVX toolbox in MATLAB [27]. The iteration process is described as follows:

  1. 1

    Initialize all elements in \({\textbf{W}}_{b,A}\) to \(\pi /2\) for the b-th symbol.

  2. 2

    Based on the given \({\textbf{W}}_{b,A}\), optimize \({\textbf{W}}_{b,D}\) according to (16), and obtain the corresponding current minimum cost function value \(J_{b}(\min )\). (When \({\textbf{W}}_{b,A}\) is given, the problem of optimizing \({\textbf{W}}_{b,D}\) is convex and can be described as

    $$\begin{aligned} & \mathop {{\text{min}}}\limits_{{{\mathbf{W}}_{{b,D}} }} \sum\limits_{{x = 0}}^{{N - 1}} {\left\| {{\mathbf{p}}_{{b,{\text{side}},x}} - \sum\limits_{{n = 1}}^{{N - 1}} {{\mathbf{W}}_{{b,D}} } (n,x){\mathbf{W}}_{{b,A}} (n,:){\mathbf{S}}_{{n,{\text{side}},x}} } \right\|_{2} } \\ & \quad {\text{subject to}}\sum\limits_{{n = 0}}^{{N - 1}} {\left\| {\sum\limits_{{x = 0}}^{{N - 1}} {{\mathbf{W}}_{{b,D}} } (n,x){\mathbf{W}}_{{b,A}} (n,:)} \right\|_{2}^{2} } \le \eta , \\ & \quad \sum\limits_{{n = 1}}^{{N - 1}} {{\mathbf{W}}_{{b,D}} } (n,x){\mathbf{W}}_{{b,A}} (n,:){\mathbf{S}}_{{n,{\text{main}},x}} = {\mathbf{p}}_{{b,{\text{main}},x}} \\ & \quad {\text{for }}\quad x = 0,1, \ldots ,N - 1. \\ \end{aligned}$$
    (17)

    Using the cvx toolbox to solve (17), the minimum cost function value \(J_{b}(\min )\) can be obtained.)

  3. 3

    Optimize each element in \({\textbf{W}}_{b,A}\) in turn. Take the elements in F in turn and assign them to phase value \(\beta _{b,A,n,m}\) of the m-th element of the n-th subarray and calculate the corresponding objective function value according to (16), i.e., if there are \(2^{\alpha }\) elements in F, the corresponding \(2^{\alpha }\) objective function values can be obtained. Select the smallest objective function value \(J_{b}(\beta _{b,A,n,m})\) and compare it with \(J_{b}(\min )\); if \(J_{b}(\beta _{b,A,n,m})\) is not less than \(J_{b}(\min )\), \(\beta _{b,A,n,m}\) is not updated; otherwise, it is updated to the phase corresponding to the smallest objective function value \(J_{b}(\beta _{b,A,n,m})\), and \(J_{b}(\min )\) is also updated.

  4. 4

    Go back to Step 3) until \(J_{b}(\min )\) for the b-th group of symbols converges. In this paper, if \(J_{b}(\min )\) is not updated for three consecutive times, the objective function is considered to have converged.

  5. 5

    Go back to Step 1) until \(B^N\) groups of beam multiplexing are designed.

Note that in each iteration, the value of the optimized phase shifter is fixed, and the phase shifter will not be optimized again due to changes in the values of other phase shifters, which is different from the exhaustive search method. Through the above method, the minimum cost function can be obtained by optimizing \({\textbf{W}}_{b,D}\) and \({\textbf{W}}_{b,A}\). Assuming that z represents the number of iterations for the b-th symbol combination, the complexity of the proposed method is \(O(zMN2^{\alpha })\), which is lower than the exhaustive search method.

Note that the limitation on beamforming would be the same for all other designs based on discrete phase shifts, i.e., a suboptimal beam response compared to the design with an infinite precision. Moreover, as it is a hybrid structure, its performance will not be as good as a fully digital implementation. Furthermore, since the optimization problem is not convex, a globally optimum solution may not be found.

4 Design examples

In this section, design examples based on the interleaved subarray and localized subarray structures are given. Assume that both subarrays are equipped with 26 antennas, i.e., \(N=2\), \(M=26\), and \(x = \{0,1\}\). The spacing between adjacent antennas is \(d_{1} = \lambda /3\). The total power consumption is set to \(\eta =1\). Supposing that there are 172 undesired transmission angles and one desired transmission angle for each beam, the sidelobe regions \(\Theta _{\textrm{side},0} \in [-90^{\circ }, -15^{\circ }]\cup [-5^{\circ }, 90^{\circ }]\) and \(\Theta _{\textrm{side},1} \in [-90^{\circ }, -5^{\circ }]\cup [5^{\circ }, 90^{\circ }]\) corresponding to the zeroth beam and the first beam formed at a sampling interval of \(1^{\circ }\), and the mainlobe angles are \(\Theta _{\textrm{main},0}=-10^{\circ }\) and \(\Theta _{\textrm{main},1}=0^{\circ }\), respectively. The designed 16 groups of dual-beam multiplexing results are shown in Table 1. The desired response amplitudes corresponding to the received symbols ‘00’, ‘01’, ‘11’, ‘10’ in the mainlobe directions are 1 (the gain is 0 dB), and the phases are \(45^{\circ }\), \(135^{\circ }\), \(-135^{\circ }\) and \(-45^{\circ }\), respectively, and random phases and magnitudes of 0.1 are set in the sidelobe region. Note that when choosing the parameters for our design, we have used some existing papers in this area as a reference, in particular those related to multi-beam multiplexing design [21, 22].

The optimal phase values of all phase shifters for the zeroth subarray of the interleaved subarray structure corresponding to the beam of the zeroth, fourth, eighth, and twelfth groups are shown in Tables 2, 3, 4, and 5. The resultant magnitude responses are shown in Fig. 3a–d, and we can see that the magnitudes of the beam responses at the sidelobe regions \(\Theta _{\textrm{side},0}\) and \(\Theta _{\textrm{side},1}\) are lower than 0 dB in the corresponding two mainlobe directions. The phase patterns corresponding to \(x=0\) are shown in Fig. 4a, b, which are of \(90^{\circ }\) intervals in the \(-10^\circ\) direction and random in other directions, and the phase patterns corresponding to \(x=1\) are shown in Fig. 4c and d, which satisfies the DM design requirement. Figure 5 shows the variation of the cost function value with the number of iterations for 16 groups of symbol combinations, reaching convergence around the fifth.

Assuming that all directions have the same level of additive white Gaussian noise, and by setting the signal-to-noise ratio to 12dB, BER is shown in Fig. 6, showing that the BER in the mainlobe direction is down to \(10^{-5}\), while the BER in other directions is around 0.5. By defining the target angle resolution as the beamwidth when the BER is less than \(10^{-3}\), the resolution is about \(1^{\circ }\) at two target angles of \(-10^{\circ }\) and \(0^{\circ }\).

Fig. 1
figure 1

Hybrid beamforming structure based on interleaved subarrays

Full size image
Fig. 2
figure 2

Hybrid beamforming structure based on localized subarrays

Full size image
Fig. 3
figure 3

Resultant magnitude responses for the DM beam multiplexing design in (16). a–d include all magnitude responses

Full size image
Fig. 4
figure 4

Resultant phase patterns for the DM beam multiplexing design in (16). a–d include all phase responses

Full size image
Fig. 5
figure 5

Cost function versus iterations in (16)

Full size image
Fig. 6
figure 6

Spatial distribution of BER for dual-beam multiplexing

Full size image
Fig. 7
figure 7

Comparisons of the magnitude responses based on the interleaved subarray structure and the localized subarray structure. a–d represent the magnitude responses of two multiplexed beams corresponding to beam \(b=0\), \(b=4\), \(b=8\), and \(b=12\), respectively

Full size image
Fig. 8
figure 8

Variation of the 3dB beamwidth of the symbol ‘00’ of beam 0 with the number of antennas for the interleaved and localized structures

Full size image
Table 1 Designed beam multiplexing group and corresponding symbols and beam numbers
Full size table
Table 2 Optimized phase value for the zeroth subarray corresponding to beam \(b=0\) in (16)
Full size table
Table 3 Optimized phase value for the zeroth subarray corresponding to beam \(b=4\) in (16)
Full size table
Table 4 Optimized phase value for the zeroth subarray corresponding to beam \(b=8\) in (16)
Full size table
Table 5 Optimized phase value for the zeroth subarray corresponding to beam \(b=12\) in (16)
Full size table

The comparison of magnitude responses based on the two interleaved and localized subarray structures under the same parameters is also provided. The magnitude responses corresponding to the beam of the zeroth, fourth, eighth, and twelfth groups are shown in Fig. 7a–d, which shows that the interleaved structure forms a narrower mainlobe, and compared with the sidelobe levels \(-\)13.69 dB, \(-\)13.94 dB, \(-\)13.96 dB, and \(-\)13.71 dB of the localized subarray structure of beam 1, beam 9, beam 17, and beam 25, the interleaved subarray structure has lower sidelobe levels, which are \(-13.34\) dB, \(-13.874\) dB, \(-13.62\) dB, and \(-14.1\) dB, respectively. For the case that beam 0 transmits the symbol ‘00’, the variation of the 3dB bandwidth of the two structures with the number of antennas is shown in Fig. 8. It can be seen that increasing the number of antennas can reduce the beamwidth, and when the number of antennas is 52, the interleaved subarray structure forms a \(3^\circ\) narrower beam than the localized subarray.

5 Conclusions

In this paper, a DM design for multi-beam multiplexing based on a hybrid beamforming structure with discrete phase shifters is investigated. The proposed alternating optimization algorithm has lower complexity than the exhaustive search method, and the objective function is guaranteed to converge. The obtained magnitude responses and phase patterns show that a given modulation pattern can only be received in multiple desired directions with beam multiplexing, but scrambled in the undesired directions due to random phase variations and low sidelobe levels by the DM design, and the BER further verifies the validity of the design. According to the magnitude responses of the interleaved subarray structure and the localized structure, the former produces a narrower beam and thus has a better performance in the beam-multiplexed DM design.