Estimation

Abstract

Suppose we have the model $\mathbf{y} =\boldsymbol{\theta } +\boldsymbol{\varepsilon }$, where $\mathrm{E}[\boldsymbol{\varepsilon }] = \mathbf{0}$, $\mathrm{Var}[\boldsymbol{\varepsilon }] =\sigma ^{2}\mathbf{I}_{n}$, and $\boldsymbol{\theta }\in \varOmega$, a p-dimensional vector space. One reasonable estimate of $\boldsymbol{\theta }$ would be the value $\hat{\boldsymbol{\theta }}$, called the least squares estimate, that minimizes the total “error” sum of squares

$$\displaystyle{SS =\sum _{ i=1}^{n}\varepsilon _{ i}^{2} =\parallel \mathbf{y}-\boldsymbol{\theta }\parallel ^{2}}$$

subject to $\boldsymbol{\theta }\in \varOmega$. A clue as to how we might calculate $\hat{\boldsymbol{\theta }}$ is by considering the simple case in which y is a point P in three dimensions and Ω is a plane through the origin O. We have to find the point Q ($=\hat{\boldsymbol{\theta }}$) in the plane so that PQ ² is a minimum; this is obviously the case when OQ is the orthogonal projection of OP onto the plane. This idea can now be generalized in the following theorem.