A prototype filter design based on channel estimation for OQAM/OFDM systems

A prototype filter design for offset quadrature amplitude–modulation based orthogonal frequency division multiplexing (OQAM/OFDM) systems is proposed in this study. The influence of channel estimation is considered, with the object to minimize stop-band energy. An efficient preamble structure is proposed to improve the channel estimation and save the frequency spectral efficiency. The reciprocal of the signal-to-interference-plus-noise ratio (RSINR) is derived to measure the influence of the prototype for channel estimation. After that, the problem of prototype filter design is formulated as an optimization problem with constraints on the RSINR. To accelerate the convergence and obtain a global optimal solution, the box-based branch and bound algorithm is utilized to solve the optimization problem. Simulation results demonstrate the validity and efficiency of the prototype filter designed in this study.


Introduction
Current communication networks have difficulty satisfying the transmission requirements of the internet of things, such as communication capacity, transmission quality, data rate, and secrecy. As the newest generation of mobile communication systems, fifth-generation (5G) mobile communication technology is attracting increased attention. As an alternative modulation scheme for 5G communication, offset quadrature amplitude modulation-based orthogonal frequency division multiplexing (OQAM/OFDM) outperforms OFDM with respect to spectral efficiency, side lobes, and sensitivity to asynchronous transmission [1,2]. As one of the important components of OQAM/OFDM systems, the prototype filter will affect the side lobes of the system and the power spectral density of the transmitted signal. Hence the prototype filter is a crucial component of OQAM/OFDM systems and plays a significant role in their overall performance.
Many effective prototype filter design methods have been discussed in the literature, and these can be classified as direct and indirect approaches [3]. Indirect approaches mainly include the windowing-based method and frequency sampling method. Direct approaches optimize all coefficients of the prototype filter. With many more degrees of design freedom, direct approaches have the potential to achieve better performance than indirect approaches [4].
However, direct approaches require the optimization of thousands of coefficients, and the complexity grows rapidly with the number of coefficients. To reduce the computational complexity, Androulakis et al. [5] utilized the α-based branch and bound (αBB) algorithm to design the prototype filter. However, the αBB algorithm can only marginally narrow the search scope of the optimal solution. Aiming to obtain superior spectrum features, Ricardo et al. [6] used convex optimization to design the prototype filter. Viholainen et al. [3] proposed an efficient iterative algorithm to design the prototype filter when the subcarrier number is large. The algorithm was obtained by deriving the gradient vector of the objective function [3]. The unknowns in these methods make it difficult to obtain the global optimum. The box-based branch and bound (Box-BB) method can solve this problem. Wu et al. proposed this method to decrease the number of iterations [7]. The method converged quickly and obtained the global optimum.
Accurate channel estimation (CE) at the receiver is indispensable for reliable signal detection. There have been some studies on CE for the OQAM/OFDM system. Two modified CE methods were proposed by placing guard symbols at different sides of a reference preamble symbol [8]. Both proposed preambles improved the CE performance, but the existence of guard symbols reduced spectrum efficiency. To overcome this defect and increase spectrum efficiency, coded auxiliary pilot symbols that also carry information were designed [9]. At the same time, some pilot symbols were used to deal with imaginary interference. However, at least two pilot symbols were needed to achieve acceptable performance in practice. An efficient preamble structure was proposed to achieve a better CE with less preamble consumption [10]. Because compressed sensing has no pilot, CE methods based on it were proposed to further increase the spectral efficiency. When prior sparse knowledge was unknown, an adaptive, regularized, compressive sampling matching pursuit was chosen as the support set to achieve better CE accuracy [11]. The complexity of this method was high, which limited its application. Inspired by Wang et al. [11], two improved CE methods were derived, based respectively on the interference approximation method (IAM) and a pair of real pilots [12]. Both methods based on compressed sensing achieved remarkable results compared with the conventional CE method. However, whether the system, like the scattering channel, is sparse is debatable. No matter what CE method is utilized, the intrinsic interference must be eliminated to guarantee CE performance.
Few studies have taken CE into account the prototype filter design. The prototype filter is designed by utilizing the mean square error (MSE) of the CE [13]. However, zero symbols inserted in the preambles will reduce the system's spectral efficiency. Inspired by Inching et al. [10], a preamble without guard symbols is considered in this paper to achieve higher transmission efficiency. The proposed structure would improve the accuracy of CE by increasing pseudo pilot power.
The influence of noise may also be ignored in prototype filter design. For example, the weight signal-to-interference ratio was treated as the utility function to design a prototype filter under a highly doubly dispersive channel without noise [14]. However, the noise power is generally equal to, or even larger than, the interference power in practical systems. Therefore, the noise influence should be given sufficient attention in prototype filter design. The signal-to-interferenceplus-noise ratio (SINR) is usually adopted to measure the noise influence. A bigger SINR means better system performance. Therefore, it will be difficult to set the threshold when the SINR is added to the constraints. Under this condition, in this study, the reciprocal of the SINR (RSINR) is adopted to measure the influence of noise.
The contributions of this paper are two-fold.
(1) A novel preamble structure is proposed to improve the channel spectral efficiency. Zero symbols are generally inserted between preambles and data symbols, but in this study, preambles without zero symbols are considered in order to acquire a higher transmission efficiency. This enables us to eliminate the influence of intrinsic interference on CE. Because of unknown items, a pre-decision method is adopted to estimate the channel impulse response (CIR) of the proposed preamble structure. (2) The influence of interference on CE and of noise is considered in the prototype filter design. The RSINR is utilized to measure these influences. In this study, the RSINR is expressed as functions of CIR and noise and is then added to the constraints. A lower out-of-band interference power means better system performance. Since the stop-band energy determines the total leaked out-of-band interference power, to minimize the stop-band energy is the optimization objective in this study.
The rest of the study is organized as follows. In section 2, the OQAM/OFDM model is described, the preamble is proposed, and the CE is obtained. In section 3, the prototype filter design is formulated by minimizing the stop-band energy with constraints on the RSINR, and a Box-BB algorithm is utilized to solve the optimization problem. The results of the simulations are discussed in section 4, and our conclusions are presented in section 5.

System model
Instead of transmitting complex symbols c m;n ¼ c R m;n þ jc I m;n , OQAM/OFDM transmits the real-valued symbols a m, n . a m, n is from the constellation mapping of C R m;n and C I m;n . According to Achaichia et al. [15], the continuous time baseband equivalent of the transmitting signal in OQAM/OFDM can be written as where M is the number of subcarriers, V 0 = 1/T 0 = 1/2τ 0 denotes the subcarrier spacing, T 0 is the constellation interval, and τ 0 is the offset between the real and imaginary parts of OQAM. g(t) is the prototype filter function. ϕ m, n is an additional phase term, where ϕ 0 can be chosen arbitrarily. For simplification, ϕ 0 = 0 in this study.
Considering the multi-path channel with additive white Gaussian noise η(t), when the transmitted OQAM/OFDM signal is passed through, the baseband version of the received signal is described as.
where h(t) is the CIR and Δ is the maximum delay spread of the channel. We assume that In this case, we rewrite (3) as where The demodulation signal at the (m 0, n 0 ) position can be written as where 〈x, y〉 is the inner product of x and y, η 1 where R Á f g is the real part of the argument, and b H m 0 is Dirac's delta function.
Thus the OQAM/OFDM system model has been built. To improve the preamble, CE based on the preamble is needed to analyze the relationship between the preamble and CE.

CE based on the preamble
We define m = m 0 + p, . n = n 0 + q, . p, q ∈ Z, and the inner product of the pulse shaping filter in Eq. (5) can be rewritten as g m;n t ð Þ; g m0;n0 where * denotes the conjugate operation. We make the can be rewritten as where A g (τ, v) is the ambiguity function of the filter, Since g(t) is an even function, its instantaneous autocorre- is the inverse Fourier transform of γ g (τ, t), and therefore, A g (τ, v) is a real-valued function [16].
From Eq. (5), we can obtain . With the increase in |p| and |q|, C p, q becomes very close to zero. For example, for an isotropic orthogonal transformation algorithm (IOTA) function, when (p, q) ∉ Ω 1, 1 , we have [17].
from which, we can conclude that only one column of zero symbols is needed to prevent adjacent unknown data symbols from interfering with the preamble. However, the inserted zero symbols will affect the CE. According to LeLe et al. [17], when only one column of zero symbols is inserted in the preamble, the CE performance of the preamble is worse than that of the IAM. The structure of the IAM is shown in Fig. 1, from which it is seen that there are two columns of zero symbols. Since a m 0 −1;n 0 and a m 0 þ1;n 0 are opposite in sign, Eq. (10) can be rewritten as where b m 0 ;n 0 ¼ a m 0 ; n 0 þ C 1;0 a m 0 þ1;n 0 þ C −1;0 a m 0 −1;n 0 . b m 0 ; n 0 can be treated as the pseudo pilot. Equation (13) implies that greater pseudo pilot power leads to a better CE. The power of pseudo pilot b m 0 ; n 0 for the IAM is where σ 2 a is the variance of the transmitted OQAM/OFDM symbols a m. n .
CE based on the IAM has two obvious drawbacks: (1) the preamble duration is 3τ 0 , which leads to low spectrum efficiency, and (2) the second item in Eq. (13) cannot be ignored when both the signal-to-noise ratio (SNR) and constellation mapping order are high.
To overcome these drawbacks and increase the pseudo pilot power, a new preamble with duration τ 0 is proposed in this paper. The structure of the preamble is shown in Fig. 2. For sufficient accuracy and concise description, only three-tap interference of neighborhood symbols is considered.
The pseudo pilot power of the proposed structure is from which, we can conclude that the proposed structure can improve system performance for its larger pseudo pilot power. This means that the proposed structure can decrease the interference caused by the adjacent unknown data symbol, promote CE performance, and increase the frequency spectrum efficiency.

RSINR
According to the previous discussion, the noise and influence of the prototype filter on CE should be given sufficient attention in the prototype filter design. To take these into account while designing the prototype filter, the RSINR is chosen to measure their influences. The expression of RSINR based on the CIR is derived in this section.
Because the a m 0 þp;n 0 þq in Eq. (18) belongs to a random signal, its value cannot be determined during transmission. As a result, the term a i2 ð Þ m 0 ;n 0 is unknown, in which case Eq. (19) will be invalid. To solve this problem, the pre-decision method [18] is adopted.
First, the initial channel frequency response H m 0 can be obtained from (16) as Next, a m 0 þp;n 0 þq in the term a i2 By inverse Fourier transform, the CIR of the OQAM/OFDM system is obtained as We then discretize the CIR b h t ð Þ into the form b h 1 , as follows: where ⌈x⌉ denotes the smallest integer greater than or equal to x, and T s ¼ T 0 2M is the sampling period. The desired power of the system is closely related to the transmitted symbol. The transmitted symbol at the position (m 0 , n 0 ) can be estimated as b a m 0 ;n 0 ¼R The QAM symbols are reconstructed by We then define where A g k; l ½ ¼ A g kT s ; l MT s , and L h is the number of channel taps at the sampling rate. Then the desired power expression of the system is where σ 2 c is the variance of the complex symbol c. Similarly, the interference power and noise power are expressed as [19].
where σ 2 n is the variance of the noise η. Thus the expression of RSINR is obtained by Eqs. (30), (31), and (32) as 3 Prototype filter design

Problem formulation
The prototype filter in the OQAM/OFDM system is assumed to be real-valued, symmetric, and of unit energy. According to the previous discussion, when there is intrinsic interference and noise in the system, the prototype filter will influence the CE. Therefore, the RSINR will be chosen as an important constraint in the problem formulation. At the same time, the prototype filter design should minimize the stop-band energy. Thus the prototype filter design problem can be formulated as G e jω À Á 2 dω s:t: where jG e jω ð Þj ¼ j ∑ k¼0 L g−1 g k ½ e −jωk j is the magnitude response of the prototype filter, TH is a constant threshold, and L g is the length of the prototype filter. According to Siohan et al. [20], There are many variables in P1. The objective and constraint functions have quadratic forms. It is difficult to directly solve the optimization problem P1. As everyone knows, orthogonal transformation does not change the nature of the original function. However, it can be utilized to change the form of (34).
The first constraint function in (34) implies that only half the filter coefficients are independent. To reduce the number of variables, we replace g[k] with a new variable, The objective function can be rewritten in vector form as f 0 (x) = x T Cx, where C is a real symmetric positive-definite matrix. λ 1 , λ 2 , …λ L denote positive eigenvalues in ascending order, and v 1 , v 2 , …, v L denote corresponding unit orthogonal eigenvectors.
Then the orthogonal transformation X = VZ is applied to the variable X, where Z = [z 1 z 2 …z L ] T is the new orthogonal variable.
Finally, the optimization problem after variable substitution and orthogonal transformation is expressed as

Problem solution
Variable substitution reduces the number of variables by half.
After orthogonal transformation, all functions in (36) take standard forms. According to Androulakis et al. [5], P2 can be directly solved by the αBB algorithm.
The αBB algorithm has disadvantages. If there is no intersection between the two constraint functions in (36) and a certain subset, then the αBB algorithm will be invalid. Furthermore, it never pays to determine the parameter ɑ and confirm the upper and lower bounds for the αBB algorithm because it is time consuming.
To avoid these disadvantages, the Box-BB algorithm [7] is proposed. The Box-BB algorithm to solve optimization problem P2 is as follows.

Simulation result
The validity of the proposed prototype filter design method is verified through a series of simulations. We set the number of subcarriers as M = 128, the subcarrier spacing as v 0 = 10.94 kHz, the filter length as L g = 4M + 1, and the power ratio threshold as TH = 10 −4 in simulations. The ITU-VehB channel model without mobility is adopted for the highly frequencyselective channels [22].

Performance of the proposed preamble
The truncation of an IOTA filter of length 4T 0 [23] is chosen as the prototype filter in this simulation. Figure 3 illustrates the CE performances of the proposed structure (A) and the IAM (B) for two kinds of constellation mapping. For 16-QAM modulation, when the SNR is low (SNR < 15 dB), the CE performances of A are slightly better than those of B. Along with the SNR increasing, the CE performances of A become better and better, and the performance advantages become increasingly obvious. This is because there are imaginary preamble symbols in the proposed structure to increase the pseudo pilot power. Additionally, the IAM performance curve tends to flatten due to the performance platform when SNR > 15 dB. Similarly, for 4-QAM modulation, the IAM performance curve tends to be gentle when SNR > 20 dB. From Fig. 3, we can conclude that the proposed structure in this study is better than the IAM structure when it comes to CE. Therefore, the proposed structure is able to increase the CE performance. Moreover, because the proposed structure is without zero symbols, the frequency spectrum of the system is improved. Figure 4 shows the normalized amplitude responses of different filters, from which it can be seen that the prototype filter designed in this paper has a much higher amplitude than the other three filters, which means that it has a greater energy spectral density and lower stop-band energy. To further reveal   Table 1 presents the stop-band energy of different filters. It shows that the stop-band energies of the IOTA filter, the filter designed by Yuan et al. [13], and that designed by Wu et al. [7] are − 32 dB, − 43 dB, and − 41 dB, respectively. The proposed prototype filter achieves the lowest stop-band energy (− 50 dB).

Performance of the designed prototype filter
Since the proposed filter is designed with a constraint on the RSINR, not only the influence of noise is considered to guarantee the effectiveness of the method. The desired power is also increased as the interference power decreases. Wu et al. [7] did not consider the influence of noise, but it should not be ignored in practice. It caused worse performance of the method proposed by Wu et al. in the simulation when compared with the method proposed in this study. Similarly, Yuan et al. [13] did not consider the desired symbol power in the prototype filter design problem, which may have caused the solution to the optimization problem to not be the minimum.

Conclusion
In this study, a new prototype filter was designed based on the CE for the OQAM/OFDM system in frequency-selective channels. The proposed method is able to minimize the stopband energy and increase the CE performance. RSINR, which is utilized to measure the influence of the prototype filter on the CE, is an important constraint for prototype filter design. A new preamble structure was proposed to save the frequency spectrum before the RSINR calculation. Then an optimization problem of the prototype filter design was formulated to minimize the stop-band energy with constraints on the RSINR. Finally, the Box-BB algorithm was employed to solve this optimization problem in order to obtain a low convergence time and the global optimal solution. Simulation results verified the superiority of the designed prototype filter over other filters.