On a Sparse Approximation of Compressible Signals


 Many physical phenomena can be modeled by compressible signals, i.e., the signals with rapidly declining sample amplitudes. Although all the samples are usually nonzero, due to practical reasons such signals are attempted to be approximated as sparse ones. Because sparsity of compressible signals cannot be unambiguously determined, a decision about a particular sparse representation is often a result of comparison between a residual error energy of a reconstruction algorithm and some quality measure. The paper explores a relation between mean square error (MSE) of the recovered signal and the residual error. A novel, practical solution that controls the sparse approximation quality using a target MSE value is the result of these considerations. The solution was tested in numerical experiments using orthogonal matching pursuit (OMP) algorithm as the signal reconstruction procedure. The obtained results show that the proposed quality metric provides fine control over the approximation process of the compressible signals in the mean sense even though it has not been directly designed for use in compressed sensing methods such as OMP.


Introduction
Consider an unknown compressible discrete signal described by a vector x in the C N space, where C denotes the complex number set. The support T x of x is defined to be set T x = supp(x) = {i : x i = 0}. Due to specific properties of x, all the signal samples usually have nonzero amplitudes so the cardinality of T x is equal to the signal dimensionality N , i.e., |T x | = N . The second relevant feature related to the signal compressibility is a rapid decrease in the signal sample amplitudes. For instance, when the power law is considered, the amplitude values of x, when sorted in decreasing order, conform to the following inequality [1]: where G, r are positive real constants and U (i) indexes the sorted signal amplitudes. The compressible signal is usually reconstructed from noisy measurement samples. The presence of noise has negative impact on the signal samples recovery, especially these ones with low amplitudes. Therefore, it is suggested to exclude them from an estimation process (use a sparse approximation instead) than to recover their amplitudes with high errors that could result in reduction in overall estimation quality. The quality of the sparse approximationx of the compressible signal depends on both the recovered support Tx and estimation precision of its corresponding sample amplitudes. For the given support Tx , the best estimate is provided by the least squares (LS) method [10]. The only problem that remains is how to determine the proper signal support. Orthogonal Greedy Algorithms (OGAs) that belong to the family of the Compressed Sensing (CS) methods try to solve this issue [14]. They restore the signal x step-by-step in an iterative manner. Details of the reconstruction process depend on a particular algorithm. The Orthogonal Matching Pursuit (OMP) recovers a single signal element per iteration [15]. Another known method -the Compressive Sampling Matching Pursuit (CoSaMP) estimates several signal samples at once in the single iteration [12]. In both the mentioned methods and their many modifications [16,21], the first stage of the OGA's operation is the support Tx update. A desirable update route of Tx assumes selection of the consecutive elements ofx according to the descending order of their amplitudes in x. Unfortunately, system noise and some particular OGA properties usually violate the support recovery due to an incorrect selection of elements to be estimated. Consequently, it can result in reduction in signal reconstruction accuracy with the LS procedure in the second stage of OGA. But it is worth to notice that a recent research report reveals that OMP can trace the proper order of sparse signal reconstruction with high probability if the signal coefficients exhibit decay [13].
Selection of an appropriate moment when the iterative process should be stopped is a challenging task due to lack of a priori knowledge about the signal sparsity which, in case of compressible signals, can be only approximated. Thus, any thresholding based on a predetermined number of iterations is rather ineffective. Therefore, many known stopping rules compare the energy of the OGA residual error to some reference level that is based on a scaled value of the system noise power. Their definitions can also ensure the support recovery, if some additional conditions are met [2,11,17,18], but they are developed on the basis of the strict sparsity assumption. The average properties of the residual error at the moment of perfect support recovery (i.e., the exact sparsity detection) in relation to the stopping rule design are analyzed in [6]. Yet another approach based on a hypothesis test on the presence of signal part in the residual error is proposed in [19].
According to the support definition, some part of the compressible signal must be left unrecognized when sparse signal approximation is applied. How to select the support for the sparse representation if the MSE level of the reconstructed signal can be presupposed in advance? Is it possible to implement such an idea of the support determination to manage the reconstruction task with some OGAs? Attempts to answer these questions are the contribution of this paper. To do so, the following issues are considered: -an analysis of the compressible signal substitution with a sparse one. Mathematical formulas that express, both the MSE and an average residual error of a LS-based reconstruction algorithm, in terms of a given sparse approximation are derived using second-order signal properties (see Sect. 2); -a validation of the proposed formulas with statistics obtained from numerical experiments; -a relation between the MSE and the residual error and its application for developing novel and practical stopping thresholds which control the approximation process of the compressible signal (see Sect. 3). The threshold definitions directly use the MSE properties of the reconstructed signal; -experimental tests of a compressible signal estimation with the proposed stopping thresholds (see Sect. 4); The presented theoretical analysis assumes correct support recovery, but the final result is also empirically tested with OGA even if this assumption can not be met perfectly. To evaluate a quality of the MSE-controlled sparse approximation, numerical experiments of the compressible signal estimation with a modified orthogonal matching pursuit (OMP) algorithm were performed. The classical implementation of the OMP was supplemented with the proposed definitions of the threshold levels. The presented simulation results depict the performance of compressible signal reconstruction using the MSE metric as a quality measure. Comparisons with a case of the correct support recovery and the classical stopping threshold were included.
Similar results but without a solid mathematical background were presented in the conference paper [8] in relation to a channel estimation problem in communication systems. The results are only focused on the stopping thresholds definitions but they did not provide detailed analyses of the MSE and the residual error. It is worth to notice that the problem of accurate sparse approximation concerns a vast spectrum of compressed sensing methods in many different application areas including communications systems, image processing, power systems [3,4,7].
In this article, a closer look at the MSE metric will explain the meaning of sparse approximation in practical implementation and will let to propose how to control the OMP method knowing a target MSE level. The link between the MSE and the residual error will be explored in case of use of greedy methods for compressible signal estimation. These methods are considered mathematically tractable approximations of the L0-norm solution (e.g., OMP method). A similar relationship was investigated in [20] but for other signal recovery conditions. In that reference work, a sparse signal recovery was analyzed (no mention of a compressible signal case). The estimation result, that was obtained using an optimization based on the L1-norm objective function, was linked to the residual error. Ultimately, there was suggested a good tradeoff between the signal sparsity and the residual error value. Also in this article, the sparsity of the final results depends on a threshold value of the residual error but this value can be additionally adjusted by the MSE settings.
The following main notation rules are used through the paper. Matrices and vectors are denoted by the calligraphic font (e.g., C) and boldface letters, respectively. Italic letter with a subscript represents the entry of the particular vector or matrix. Superscript in parentheses denotes the current iteration number. (·) H denotes the conjugate transpose. A vector (matrix) with |T notation as the subscript, e.g., c |T , denotes the vector (matrix) with the same size and values as c but only for those elements whose position corresponds to indices (columns) indicated by numbers belonging to the set T . All the other elements are zero. A notation c T denotes a specific vector (reduced in size) that is based on c |T , but the elements not indicated by T are excluded while the indexing of the remaining ones is unchanged. The similar relation applies to matrices but the columns are deleted.

Sparse Approximation of the Compressible Signal
The compressible signal x is estimated based on a measurement vector y ∈ C M , that is linearly related to x according to the following formula: where A with entries a m,n is a M × N complex measurement matrix and the vector e of size M × 1 with entries e i has a general meaning of the measurement error that is caused by an environmental noise, a quantization noise, etc. Assume that the respective elements of the vector e and the matrix A are modeled as a realization of the independent and identically distributed Gaussian zero mean stationary processes E and A with σ 2 e and σ 2 a variances, respectively. An unambiguous least squares solution of x can be provided only for the approximated signal with the support Tx of cardinality equal to or less than M. This result is as follows [10]:x The residual error of the sparse approximation is defined as follows:

MSE of the Sparse Approximation
Define a mean square error (MSE) ofx |Tx as follows: where the operator E[. . .] represents the expectation value over two random variables E and A. Here, it is necessary to emphasize that only elements of the submatrix A T x are considered random variables. They did not participate in signal estimation but are responsible for an additional noise-like distortion as it is stated later. Because only part of x, which is pointed out by the support Tx , is recovered, it is convenient to split the MSE into an estimation error E and an approximation error A . Next paragraphs provide their definitions in detail.
Using the disjoint sets Tx and T x , the vector x can be split into two separate vectors: of the same length N . The vector x |Tx relates to the part of the signal to be recovered. The second one, x |T x , which is left out of reconstruction (i.e.,x |T x ≡ 0), becomes the source of the approximation error. This error is defined as: and expresses how much the actual signal x differs from its sparse representation x |Tx . The estimation error E is defined as the variance of the LS signal estimatex Tx [10]. Due to the sparse approximation, the LS estimation can be affected not only by the measurement error e but also by the unrecovered signal elements x T x . It is noticeable in the rewritten form of Eq. (2): Substituting (8) into (3), the least squares estimatex Tx can be expressed as: where the last element of (9) is the additional part of the estimation error independent of the distortion produced by noise e. In general case of the measurement matrix, when only noise e is considered random the estimation error could be written as follows: Because, according to the assumption, the signal A T x x T x can be considered as another noise process independent of e then using the approximation error definition (7), the estimation error can be simplified to: Finally, using Eqs. (11) and (7) in (5), the MSE definition can be rewritten in closedform as follows: For a further analysis, the most important feature of the MSE error is its dependence on the approximation error A . Figure 1 depicts an example of comparison between the MSE calculated according to Eq. (12) and a reference value obtained as average: for only L = 10 independent trials with various measurement matrices A, noise samples e, SNR ratios and a constant instance of signal x in the single averaging. The small value of L proves a good convergence. An complex power-law decaying signal consistent with condition (1) was used as an example of the compressible signal (G = 1, r = 0.85). The x-axis represents the number of elements estimated in the recovered signal. These elements are arranged in such a way that they always correspond to the highest amplitude elements in the original signal. This order of the MSE presentation is also the proper sequence of the support recovery by iterative compressed sensing methods. The results presented in Fig. 1 show that it is possible to use (12) for MSE evaluation in case of the sparse approximation of compressible signal. Moreover, Eq. (12) is valid for all kind of signals.

Residual Error Analysis
Energy of the residual error in LS procedure for a given support Tx is [10]:  (14) can be rewritten after some mathematical manipulations as follows: where is the projection matrix with tr{R} = |Tx | [9]. Using the definition of the approximation error (7), the expectation value of (15) over the stationary processes E (noise) and A (entries of the measurement matrix) is: The result presented in (17) corresponds to the one obtained in [6] but here it is complemented by the influence of the approximation error. Figure 2 depicts that the expectation value of the residual error (17) corresponds to the averaged value of (15) (named as r ref ) obtained for the same settings as in the MSE analysis in the previous subsection. Hence, Eq. (17) is proposed to be used in the next mathematical analysis.

MSE-Based Decision Threshold
The residual error is the basic measure that provides instant information about estimation quality in the ith iteration step of OGAs. It is commonly used to stop the iterative procedure if its energy drops below a predefined threshold. The paper proposes a threshold definition that is based on the assumed MSE level (MSE S ) of the finally estimated signal. In every step of the iterative procedure, the current energy of the residual error is tested against its expected value Γ S for the given MSE S . The threshold Γ S can be determined by combining Eqs. (12) and (17) and substituting the approximation error there. Finally the threshold level can be described as follows: where SNR = σ 2 a /σ 2 e . If MSE S value is selected too optimistically, the assumed quality may not be attainable during estimation process (due to specific properties of the signal and noise) and OGA's iterations will not be interrupted. To prevent it, an additional stopping threshold is suggested. It is developed knowing, that two components of (12) are monotonic but in the opposite direction. If better and better support approximation of x is obtained, the approximation error decreases but the noise part of the estimation error increases. Thereby, there can be found a balance point: that allows, when applied in (17), to define the so-called balanced threshold: Consequently, using (20) and (21), the final stopping rule for any OGA can be determined as follows:

Numerical Experiments
The developed threshold definition was implemented in OMP algorithm (see Algorithm 1) and experimentally tested for power-law decaying signals with G = 1 and r = 0.85. The measurement matrix size was set to 120 × 160. The signals were recovered in noisy environment. The signal-to-noise ratio (SNR) varied in the range of 5[dB] to 25[dB] with 2[dB] step. MSE of the estimated signals was the main results of the performed simulations and the object of discussion thereafter. The simulations results were obtained averaging over K = 50 independent simulation runs for a single instance of the compressible signal and over P = 50 independent signals. Figure 3 depicts the MSE performance of the signal estimation as a function of SNR ratio. The results obtained for the proposed stopping rule (22) were compared to the ones for the classical approach when the residual error energy is tested against the noise energy-r (i) 2 2 < Mσ 2 e [5]. There were considered two values of the quality factor: MSE S = {−15[dB]; −20[dB]}. Additionally, the reference results (labeled as "ref" in the figure) are provided. They are obtained according to the assumption that signal samples are restored in the descending order of their actual amplitudes. The depicted results show that it is possible to control the estimation process using the MSE value as the predetermined quality factor. Performance degradation for OMP algorithm is caused because the desirable recovery order valid for the "ref" case is violated by noise presence and the reconstructed signal structure. But it is the inherent

Conclusions
The paper presents the combined analysis of the residual error and the MSE of the estimated signal with respect to sparse approximation of the compressible signal using LS estimation procedure. The obtained results were then used to propose the practical rule that allows a reconstruction algorithm to control the signal recovery process using a predetermined value of MSE. The solution, which was implemented in OMP method, reduced an overall reconstruction complexity because the given recovery process was interrupted when the target quality level was reached. The threshold definition can be especially suggested for use in the recovery methods with proper support detection. An interesting direction of a future work is to use the MSE-driven control mechanism for estimation of underlying sparse process using a structured (model-based) compressed sensing approach when quality metric is still evaluated in compressible domain. It could be useful in wireless transmission systems where channel estimation is often shifted from the channel impulse response domain to the propagation paths domain.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.