Abstract
In this chapter we will cover the basics of the celebrated method of Least Squares (LS). The approach to this method is different from the stochastic gradient approach from the previous chapter. As always, the idea will be to obtain an estimation of a given system using input-output measured pairs (and no statistical information), and assuming a model in which the input and output pairs are linearly related. We will also present the Recursive Least Squares (RLS) algorithm, which will be a recursive and a more computational efficient implementation of the LS method. One of its advantage is that it can be used in real time as the input-output pairs are received. In this sense, it will be very similar to the adaptive filters obtained in the previous chapter. Several important properties of LS and RLS will be discussed.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Notice that at this moment of the presentation we are not assuming that the vectors \(\mathbf x (i)\) are generated from a scalar signal \(x(i)\) through a tapped-delay line structure.
- 2.
The reader should notice that the factor \(\frac{1}{n}\) does not modify the solution in (5.4).
- 3.
Notice that \(\fancyscript{R}\left(\mathbf C ^T\mathbf C \right)=\fancyscript{R}\left(\mathbf C ^{T}\right)\) [3], where \(\fancyscript{R}\left(\mathbf A \right)\) denotes the range or column space of the matrix \(\mathbf A \). This implies that it should exist at least one \(\mathbf w \) which satisfies Eq. (5.7).
- 4.
In recent years there was a growing interest in underdetermined systems with sparseness conditions on the solution. This is commonly known as compressed sensing or compressed sampling. The interested reader can see [4] and the references therein.
- 5.
As \(\mathbf C ^T\mathbf C \) is singular \(\fancyscript{N}\left(\mathbf C ^T\mathbf C \right)\) contains others vectors besides the null vector.
- 6.
\(\mathbf U ^T\mathbf U =\mathbf U \mathbf U ^T=\mathbf I _n\) and \(\mathbf V ^T\mathbf V =\mathbf V \mathbf V ^T=\mathbf I _L\).
- 7.
Notice that as \(\mathbf M \) is a symmetric positive definite matrix, its square root is well defined [3].
- 8.
We will restrict ourselves to the regular LS. Similar properties for the more general WLS also exist.
- 9.
- 10.
The Cramér-Rao lower bound is a general lower bound on the variance of any estimator of a given parameter. It depends on the pdf of the random variables that influence a given observation linked to the parameter to be estimated. As an absolute lower bound, the obtention of an estimator that is able to attain it is very relevant. When such an estimator exists, it is said to be an efficient estimator. In the majority of the cases (specially when the random variables involved are not Gaussian), it is not possible to obtain an efficient estimator. For more details on the Cramér-Rao lower bound the interested reader could see [9] and [10].
- 11.
Notice that, in order to simplify the notation a little bit, we use \(\mathbf w \) instead of \(\hat{\mathbf w }\) which was used in the previous sections to denote the LS solution.
- 12.
In the following, and without loss of generality we will concentrate on the RLS algorithm known as Exponentially Weighted RLS (EWRLS), which generalizes the standard RLS algorithm (with \(\lambda =1\)) and it is by far the most used and popular version. Other important variant is the sliding window RLS algorithm [11].
- 13.
The fact that these implementation of LS algorithms have \(O(L)\) should not lead to the conclusion that they need the same computational resources than an LMS algorithm. Although their complexity can be put as \(ML\), the value of \(M\) can be large, whereas for the LMS \(M=2\).
- 14.
From the mathematical point of view, it is simpler for this problem to work with complex signals. The modifications required for the RLS to work with complex signals are straightforward. See Sect. 4.1.3.
- 15.
The restrictions reflect some a priori knowledge about the problem. It is assumed that \(\theta _i\), \(i=0,\dots , M\) are known, and therefore, they can be used. As explained above, other signals coming from different angles could be present in \(\mathbf v (n)\). But since they are not known, they cannot be used in the restrictions.
- 16.
The frequency and the velocity of any narrowband wave cannot be arbitrary and are linked by \(\lambda f=c\), where \(\lambda \) is the wavelength which has units of length. It is common in beamforming to express the separation between elements in numbers of wavelength. For the example in this section \(\lambda =20\) cm, so the separation between the array elements is \(\lambda /2\).
- 17.
Although the restrictions \(\mathbf w ^H\mathbf s (\theta _1)= \mathbf w ^H\mathbf s (\theta _2)=0\) cannot be imposed, it is clear that the influence of the signals coming from \(\theta _1\) and \(\theta _2\) will appear on \(\mathbf R _{\mathbf x }\).
References
C. Meyer, Matrix Analysis and Applied Linear Algebra (SIAM, Philadelphia, 2001)
A. Papoulis, Probability, Random Variables, and Stochastic Processes (McGraw-Hill, New York, 1965)
R.A. Horn, C.R. Johnson, Matrix Analysis (Cambridge University Press, New York, 1990)
E.J. Candes, M.B. Wakin, An introduction to compressive sampling. IEEE Signal Process. Mag. 25, 21–30 (2008)
G.H. Golub, C.F. van Loan, Matrix Computations (The John Hopkins University Press, Baltimore, 1996)
D. Serre, Matrices: Theory and Applications (Springer, New York, 2002)
S. Haykin, Adaptive Filter Theory, 4th edn. (Prentice-Hall, Upper Saddle River, 2002)
A.W. Naylor, G.R. Sell, Linear Operator Theory in Engineering and Science (Springer, New York, 1982)
S.M. Kay, Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory (Prentice-Hall, Upper Saddle River, 1993)
L. Scharf, Statistical Signal Processing: Detection, Estimation and Time Series Analysis (Prentice-Hall, Upper Saddle River, 1991)
D. Manolakis, F. Ling, J. Proakis, Efficient time-recursive least-squares algorithms for finite-memory adaptive filtering. IEEE Trans. Circuits Syst. 34, 400–408 (1987)
A.H. Sayed, T. Kailath, A state-space approach to adaptive RLS filtering. IEEE Signal Process. Mag. 11, 18–60 (1994)
N.E. Hubing, S.T. Alexander, Statistical analysis of the soft constrained initialization of recursive least squares algorithms. Proc. IEEE ICASSP 3, 1277–1280 (1990)
L. Guo, L. Ljung, Exponential stability of general tracking algorithms. IEEE Trans. Autom. Control 40, 1376–1387 (1995)
R.M. Johnstone, C.R. Johnson, R.R. Bitmead, B.D. Anderson, Exponential convergence of recursive least squares with exponential forgetting factor. Syst. Control Lett. 2, 77–82 (1982)
G.V. Moustakides, Study of the transient phase of the forgetting factor RLS. IEEE Trans. Signal Process. 45, 2468–2476 (1997)
L. Ljung, T. Söderström, Theory and Practice of Recursive Identification (MIT Press, Cambridge, 1983)
A.H. Sayed, Adaptive Filters (Wiley, Hoboken, 2008)
J.M. Cioffi, T. Kailath, Fast, recursive-least-squares transversal filters for adaptive filtering. IEEE Trans. Acoust. Speech Signal Process. ASSP-32, 304–337 (1984)
H.L. Van Trees, Optimum Array Processing (Detection, Estimation, and Modulation Theory, Part IV) (Wiley, New York, 2002)
J. Benesty, J. Chen, Y. Huang, Microphone Array Signal Processing (Springer, Berlin, 2008)
A. Goldsmith, Wireless Communications (Cambridge University Press, New York, 2005)
IEEE Standard for information technology—telecommunications and information exchange between systems—local and metropolitan area networks—specific requirements. Part 11: Wireless LAN Medium Access Control and Physical Layer Specifications. Amendment 5: Enhancements for Higher Throughput. LAN/MAN Standards Committee of IEEE Computer Society (IEEE, New York, 2009)
B.D. Van Veen, K.M. Buckley, Beamforming: a versatile approach to spatial filtering. IEEE Signal Process. Mag. 5, 4–24 (1988)
M. Ghavami, Wideband smart antenna theory using rectangular array structures. IEEE Trans. Signal Process. 50, 2143–2151 (2002)
D.M. Bates, D.G. Watts, Nonlinear Regression Analysis and Its Applications (Wiley, New York, 1988)
H.W. Engl, M. Hanke, Regularization of Inverse Problems (Kluwer Academic Publishers, Boston, 2000)
B.D.O. Anderson, J.B. Moore, Optimal Filtering (Prentice-Hall, Englewood Cliffs, 1979)
S. Haykin, A.H. Sayed, J.R. Zeidler, P. Wei, P. Yee, Adaptive tracking of linear time-invariant systems by extended RLS algorithms. IEEE Trans. Signal Process. 45, 1118–1128 (1997)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2013 The Author(s)
About this chapter
Cite this chapter
Rey Vega, L., Rey, H. (2013). Least Squares. In: A Rapid Introduction to Adaptive Filtering. SpringerBriefs in Electrical and Computer Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-30299-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-30299-2_5
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-30298-5
Online ISBN: 978-3-642-30299-2
eBook Packages: EngineeringEngineering (R0)