As Good as GOLD: Gram–Schmidt Orthogonalization by Another Name

Abstract

Generalized orthogonal linear derivative (GOLD) estimates were proposed to correct a problem of correlated estimation errors in generalized local linear approximation (GLLA). This paper shows that GOLD estimates are related to GLLA estimates by the Gram–Schmidt orthogonalization process. Analytical work suggests that GLLA estimates are derivatives of an approximating polynomial and GOLD estimates are linear combinations of these derivatives. A series of simulation studies then further investigates and tests the analytical properties derived. The first study shows that when approximating or smoothing noisy data, GLLA outperforms GOLD, but when interpolating noisy data GOLD outperforms GLLA. The second study shows that when data are not noisy, GLLA always outperforms GOLD in terms of derivative estimation. Thus, when data can be smoothed or are not noisy, GLLA is preferred whereas when they cannot then GOLD is preferred. The last studies show situations where GOLD can produce biased estimates. In spite of these possible shortcomings of GOLD to produce accurate and unbiased estimates, GOLD may still provide adequate or improved model estimation because of its orthogonal error structure. However, GOLD should not be used purely for derivative estimation because the error covariance structure is irrelevant in this case. Future research should attempt to find orthogonal polynomial derivative estimators that produce accurate and unbiased derivatives with an orthogonal error structure.

Appendix: Mathematical Background and a Lemma

1.1 Background

Three items of sometimes unfamiliar mathematics will be vitally important: the dot product, vector projection, and the Gram–Schmidt orthogonalization process. For vectors $\varvec{x} = \left( x_1 ,~ x_2 ,~ x_3 ,\ldots , x_n \right) ^\mathsf{T}$ and $\varvec{y} = \left( y_1 ,~ y_2 ,~ y_3 , \ldots , y_n \right) ^\mathsf{T}$, the dot product, or scalar product, of the two vectors is written

$$\begin{aligned} \underbrace{ \varvec{x} \bullet \varvec{y} }_\text {Dot} = \underbrace{ \varvec{x}^\mathsf{T} \varvec{y} }_\text {Matrix} = \underbrace{ x_1 y_1 + x_2 y_2 + x_3 y_3 + \cdots + x_n y_n }_\text {Concrete Summation} = \underbrace{ \sum _{i=1}^n x_i y_i }_\text {Full Summation} \end{aligned}$$

(55)

The projection of $\varvec{x}$ in the direction of $\varvec{y}$ is defined as

$$\begin{aligned} \text {proj}_{\varvec{y}} (\varvec{x}) = \dfrac{\varvec{x} \bullet \varvec{y}}{\varvec{y} \bullet \varvec{y}} \varvec{y} \end{aligned}$$

(56)

It can be derived without much difficulty, but the idea of vector projection is most easily shown graphically as in Figure 3.

Fig. 3

The projection of two vectors, $x_1$ and $x_2$, in the direction of a third vector, y.

In many situations, it can be useful to have an orthogonal set of vectors spanning a space of interest. The Gram–Schmidt orthogonalization process begins with an arbitrary set of basis vectors and produces an orthogonal set of basis vectors spanning the same space. Gram–Schmidt orthogonalization can be used to solve least squares problems (Björck 1967) and is often covered in introductory linear algebra course books (e.g., Lay, 2003; Leon, 2006).

If $\varvec{u}_1, \varvec{u}_2, \varvec{u}_3, \ldots , \varvec{u}_m$ are the original basis vectors, then $\varvec{u'}_1, \varvec{u'}_2, \varvec{u'}_3, \ldots , \varvec{u'}_m$ are the new orthogonal basis vectors defined by

$$\begin{aligned} \varvec{u'}_1&= \varvec{u}_1 \end{aligned}$$

(57)

$$\begin{aligned} \varvec{u'}_2&= \varvec{u}_2 - \text {proj}_{\varvec{u'}_1} (\varvec{u}_2) \end{aligned}$$

(58)

$$\begin{aligned} \varvec{u'}_3&= \varvec{u}_3 - \text {proj}_{\varvec{u'}_1} (\varvec{u}_3) - \text {proj}_{\varvec{u'}_2} (\varvec{u}_3) \end{aligned}$$

(59)

$$\begin{aligned} \vdots \nonumber \\ \varvec{u'}_m&= \varvec{u}_m - \sum _{i=1}^{m-1} \text {proj}_{\varvec{u'}_i} (\varvec{u}_m). \end{aligned}$$

(60)

So the $k^{th}$ new basis vector,

$$\begin{aligned} \varvec{u'}_k = \varvec{u}_k - \sum _{i=1}^{k-1} \text {proj}_{\varvec{u'}_i} (\varvec{u}_k) = \varvec{u}_k - \sum _{i=1}^{k-1} \dfrac{ \varvec{u}_k \bullet \varvec{u'}_i }{ \varvec{u'}_i \bullet \varvec{u'}_i} \varvec{u'}_i. \end{aligned}$$

(61)

Figure 4 illustrates a two-dimensional example of the Gram–Schmidt orthogonalization process. When beginning with a nonorthogonal set of basis vectors $\{\varvec{u_1}, \varvec{u_2}\}$ that spans the two-dimensional space, then the Gram–Schmidt procedure produces a new set of orthogonal basis vectors $\{\varvec{u'_1}, \varvec{u'_2}\}$ that spans the same space.

Fig. 4

The construction of a set of new orthogonal basis vectors, $u_1'$ and $u_2'$, from an original set of basis vectors, $u_1$ and $u_2$, by Gram–Schmidt Orthogonalization.

1.2 Lemma Regarding Orthogonal Vectors and Projections

Let $\varvec{u}_1, \ldots , \varvec{u}_m$ be any full rank set of m vectors, and let $\varvec{u'}_1, \ldots , \varvec{u'}_m$ be the set of orthogonalized vectors produced by applying the Gram–Schmidt orthogonalization process to $\varvec{u}_1, \ldots , \varvec{u}_m$. We want to show that for any $k \in \{ 1, 2, 3, \ldots , m \}, ~~ \varvec{u'}_k \bullet \varvec{u'}_k = \varvec{u'}_k \bullet \varvec{u}_k$. By substitution based on the Gram–Schmidt definition and that of vector projection

$$\begin{aligned} \varvec{u'}_k \bullet \varvec{u'}_k = \varvec{u'}_k \bullet \left( \varvec{u}_k - \sum _{i=1}^{k-1} \text {proj}_{\varvec{u'}_i} (\varvec{u}_k) \right) = \varvec{u'}_k \bullet \left( \varvec{u}_k - \sum _{i=1}^{k-1} \dfrac{ \varvec{u}_k \bullet \varvec{u'}_i }{ \varvec{u'}_i \bullet \varvec{u'}_i} \varvec{u'}_i \right) . \end{aligned}$$

(62)

And because the dot product follows a distributive law

$$\begin{aligned} = \varvec{u'}_k \bullet \varvec{u}_k - \varvec{u'}_k \bullet \left( \sum _{i=1}^{k-1} \dfrac{ \varvec{u}_k \bullet \varvec{u'}_i }{ \varvec{u'}_i \bullet \varvec{u'}_i} \varvec{u'}_i \right) . \end{aligned}$$

(63)

Again, applying the distributive law and expanding the summation,

$$\begin{aligned} = \varvec{u'}_k \bullet \varvec{u}_k - \underbrace{\varvec{u'}_k \bullet \left( \dfrac{ \varvec{u}_k \bullet \varvec{u'}_1 }{ \varvec{u'}_1 \bullet \varvec{u'}_1} \varvec{u'}_1 \right) - \varvec{u'}_k \bullet \left( \dfrac{ \varvec{u}_k \bullet \varvec{u'}_2 }{ \varvec{u'}_2 \bullet \varvec{u'}_2} \varvec{u'}_2 \right) - \ldots - \varvec{u'}_k \bullet \left( \dfrac{ \varvec{u}_k \bullet \varvec{u'}_{k-1} }{ \varvec{u'}_{k-1} \bullet \varvec{u'}_{k-1}} \varvec{u'}_{k-1} \right) }_{\text {Want to show this is } 0}. \end{aligned}$$

(64)

Now rearranging terms

$$\begin{aligned}&= \varvec{u'}_k \bullet \varvec{u}_k - \left( \dfrac{ \varvec{u}_k \bullet \varvec{u'}_1 }{ \varvec{u'}_1 \bullet \varvec{u'}_1} \right) \underbrace{ \left( \varvec{u'}_k \bullet \varvec{u'}_1 \right) }_{=0} - \left( \dfrac{ \varvec{u}_k \bullet \varvec{u'}_2 }{ \varvec{u'}_2 \bullet \varvec{u'}_2} \right) \underbrace{ \left( \varvec{u'}_k \bullet \varvec{u'}_2 \right) }_{=0}\nonumber \\&\quad - \ldots - \left( \dfrac{ \varvec{u}_k \bullet \varvec{u'}_{k-1} }{ \varvec{u'}_{k-1} \bullet \varvec{u'}_{k-1}} \right) \underbrace{ \left( \varvec{u'}_k \bullet \varvec{u'}_{k-1} \right) }_{=0}. \end{aligned}$$

(65)

The underbraced terms are all zero because we know that the new Gram–Schmidt basis vectors, $\varvec{u'}_i$, are orthogonal. So finally,

$$\begin{aligned} = \varvec{u'}_k \bullet \varvec{u}_k. \end{aligned}$$

(66)

We have thus shown that $\varvec{u'}_k \bullet \varvec{u'}_k = \varvec{u'}_k \bullet \varvec{u}_k$, and the proof is complete.

$\square$