Skip to main content
Log in

Further remarks on the connection between fixed linear model and mixed linear model

Statistical Papers Aims and scope Submit manuscript

Abstract

The linear mixed model \(\mathbf {y}= \mathbf {X}\varvec{\beta }+ \mathbf {Z}\mathbf {u}+ \varvec{\varepsilon }\), where \(\varvec{\beta }\) is the vector of fixed effects and \(\mathbf {u}\) the vector of random effects, has strong links with a particular augmented linear model including only fixed effects. This goes back to Henderson’s mixed model equations and was recently exploited by Haslett and Puntanen (Stat Papers 51:465–475, 2010), who restated Henderson’s result in a model with possibly singular covariance matrices. In this paper we point out that the connection between the two models is actually very straightforward: a mixed linear model can be obtained from the augmented model by a simple linear transformation. This, for example, immediately opens up a new viewpoint for studying the relationship between the BLUEs and BLUPs in the two models. In doing so we discuss a modification of the Frisch–Waugh–Lovell theorem as well as some not so well established theorems about the BLUPs of random errors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Baksalary JK, Rao CR, Markiewicz A (1992) A study of the influence of the “natural restrictions” on estimation problems in the singular Gauss–Markov model. J Stat Plan Inf 31:335–351

    Article  MATH  MathSciNet  Google Scholar 

  • Bhimasankaram P, Sengupta D (1996) The linear zero functions approach to linear models. Sankhyā Ser B 58:338–351

    MATH  MathSciNet  Google Scholar 

  • Christensen R (2011) Plane answers to complex questions: the theory of linear models, vol 4. Springer, New York

    Book  Google Scholar 

  • Davidson R, MacKinnon JG (1993) Estimation and inference in econometrics. Oxford University Press, New York

    MATH  Google Scholar 

  • Davidson R, MacKinnon JG (2004) Econometric theory and methods. Oxford University Press, New York

    Google Scholar 

  • Groß J (2004) The general Gauss–Markov model with possibly singular dispersion matrix. Stat Pap 45:311–336

  • Groß J, Puntanen S (2000) Estimation under a general partitioned linear model. Linear Algebra Appl 321:131–144

    Article  MATH  MathSciNet  Google Scholar 

  • Groß J, Puntanen S (2005) Extensions of the Frisch–Waugh–Lovell theorem. Discussiones Mathematicae—Probab Stat 25:39–49

    MATH  Google Scholar 

  • Haslett SJ, Puntanen S (2010) Equality of BLUEs or BLUPs under two linear models using stochastic restrictions. Stat Pap 51:465–475

    Article  MATH  MathSciNet  Google Scholar 

  • Haslett SJ, Puntanen S (2011) On the equality of the BLUPs under two linear mixed models. Metrika 74:381–395

    Article  MATH  MathSciNet  Google Scholar 

  • Henderson CR (1975) Best linear unbiased estimation and prediction under a selection model. Biometrics 31:423–447

    Article  MATH  Google Scholar 

  • Henderson CR, Kempthorne O, Searle SR, von Krosigh CN (1959) The estimation of environmental and genetic trends from records subject to culling. Biometrics 15:192–218

    Article  MATH  Google Scholar 

  • Hodges JS (1998) Some algebra and geometry for hierarchical models, applied to diagnostics. JRSS Ser B (Stat Methodol) 60:497–536

    Article  MATH  MathSciNet  Google Scholar 

  • LaMotte LR (2007) A direct derivation of the REML likelihood function. Stat Pap 48:321–327

    Article  MATH  MathSciNet  Google Scholar 

  • Lee Y, Nelder JA (2001) Hierarchical generalised linear models: a synthesis of generalised linear models, random-effect models and structured dispersions. Biometrika 88:987–1006

    Article  MATH  MathSciNet  Google Scholar 

  • Puntanen S, Styan GPH, Isotalo J (2011) Matrix tricks for linear statistical models: our personal top twenty. Springer, Heidelberg

    Book  Google Scholar 

  • Rao CR (1967) Least squares theory using an estimated dispersion matrix and its application to measurement of signals. In: Le Cam LM, Neyman J (eds) Proceedings of the fifth Berkeley symposium on mathematical statistics and probability: Berkeley, California, 1965/1966, vol 1. University of California Press, Berkeley, pp 355–372

  • Rao CR, Mitra SK (1971) Generalized inverse of matrices and its applications. Wiley, New York

    MATH  Google Scholar 

  • Robinson GK (1991) That BLUP is a good thing: the estimation of random effects. Stat Sci 6:15–51 (discussion 32–51)

    Article  MATH  Google Scholar 

  • Searle SR (1997) The matrix handling of BLUE and BLUP in the mixed linear model. Linear Algebra Appl 264:291–311

    Article  MATH  MathSciNet  Google Scholar 

  • Searle SR, Casella G, McCulloch CE (1992) Variance components. Wiley, New York

    Book  MATH  Google Scholar 

  • Sengupta D, Jammalamadaka SR (2003) Linear models: an integrated approach. World Scientific, River Edge

    Google Scholar 

  • Zyskind G (1967) On canonical forms, non-negative covariance matrices and best and simple least squares linear estimators in linear models. Ann Math Stat 38:1092–1109

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgments

We would like to thank Katarzyna Filipiak and Augustyn Markiewicz for organizing a research group meeting on Mixed and Multivariate Models in the Mathematical Research and Conference Center in Bȩdlewo, Poland, October 2013, supported by the Stefan Banach International Mathematical Center; that meeting triggered the work presented in this paper. We would also like to acknowledge Stephen J. Haslett’s involvement in bringing up the intriguing question about the relationship between a mixed linear model and an augmented linear model during his previous collaboration with the second author. We would like to thank two anonymous reviewers, whose comments led to a considerable improvement in the presentation of the manuscript. Thanks go also to Lynn Roy LaMotte for helpful remarks.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to B. Arendacká.

Appendix

Appendix

Here we show the proofs for Theorems 1, 3 and Lemma 1. Before doing so, we collect a few known facts to be used later. Let us recall that in a general linear model

$$\begin{aligned} \mathbf {y}= \mathbf {X}\varvec{\beta }+ \varvec{\varepsilon }, \quad \text {i.e.,} \quad \fancyscript{A}= \{ \mathbf {y}, \mathbf {X}\varvec{\beta }, \mathbf {V}\}, \end{aligned}$$
(4.1)

where \( \mathbf {X}\) is a known \(n\times p\) (\(n>p\)) model matrix (not necessarily of full column rank), the vector \( \mathbf {y}\) is an observable \(n\)-dimensional random vector, \(\varvec{\beta }\) is a \(p\times 1\) vector of unknown parameters, and \( \varvec{\varepsilon }\) is an unobservable vector of random errors with expectation \({{\mathrm{E}}}( \varvec{\varepsilon }) = \mathbf {0}\) and covariance matrix \({{\mathrm{cov}}}( \varvec{\varepsilon }) = \mathbf {V}\), where the \(n \times n\) nonnegative definite matrix \(\mathbf {V}\) is known. The linear estimator \(\mathbf {A}\mathbf {y}\), where \(\mathbf {A}\) is an \(n \times n\) matrix, is said to be the best linear unbiased estimator, \({{\mathrm{BLUE}}}\), of \(\mathbf {X}\varvec{\beta }\) if \(\mathbf {A}\mathbf {y}\) is unbiased and its covariance matrix is minimal in the Löwner sense among all unbiased linear estimators of \(\mathbf {X}\varvec{\beta }\). Similarly, if \(\mathbf {G}\mathbf {y}\) is an unbiased linear predictor for \(\mathbf {L}\varvec{\varepsilon }\), i.e., \({{\mathrm{E}}}(\mathbf {G}\mathbf {y}) = {{\mathrm{E}}}(\mathbf {L}\varvec{\varepsilon }) = \mathbf {0}_{n}\), so that \(\mathbf {G}\mathbf {X}= \mathbf {0}_{n \times p}\), then \(\mathbf {G}\mathbf {y}\) is the \({{\mathrm{BLUP}}}\) for \(\mathbf {L}\varvec{\varepsilon }\) if the covariance matrix of the prediction error \(\mathbf {L}\varvec{\varepsilon }- \mathbf {G}\mathbf {y}\) is minimal among all unbiased linear predictors of \(\mathbf {L}\varvec{\varepsilon }\).

Lemmas 2 and 3 characterize \({{\mathrm{BLUE}}}\)s and \({{\mathrm{BLUP}}}\)s, respectively. For \({{\mathrm{BLUE}}}\)s, see, e.g., Rao (1967) and Zyskind (1967); Eq. (4.2) is sometimes called the “fundamental” \({{\mathrm{BLUE}}}\) equation. Lemma 3 for the \({{\mathrm{BLUP}}}\) follows directly from what is known as the fundamental equation for the \({{\mathrm{BLUP}}}\) of vector of future observations \(\mathbf {y}_f=\mathbf {X}_{f}\varvec{\beta }+\varvec{\varepsilon }_{f}\); see (Puntanen et al. (2011), Proposition 10.6), since we want to predict a vector \(\mathbf {y}_{f}= \mathbf {0}\varvec{\beta }+\mathbf {L}\varvec{\varepsilon }=\mathbf {L}\varvec{\varepsilon }\). (The fact that our prediction does not really concern future, does not play a role.)

Lemma 2

Consider the general linear model \( \fancyscript{A}=\{\mathbf {y}, \mathbf {X}\varvec{\beta }, \mathbf {V}\}\). Then the estimator \(\mathbf {A}\mathbf {y}\) is the \({{\mathrm{BLUE}}}\) for \(\mathbf {X}\varvec{\beta }\) if and only if \(\mathbf {A}\) satisfies the equation

$$\begin{aligned} \mathbf {A}(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot } ) = (\mathbf {X}: \mathbf {0}). \end{aligned}$$
(4.2)

The corresponding condition for \(\mathbf {B}\mathbf {y}\) to be the \({{\mathrm{BLUE}}}\) of an estimable parametric function \(\mathbf {K}\varvec{\beta }\) (i.e., \(\mathbf {K}\varvec{\beta }\) has an unbiased linear estimator) is

$$\begin{aligned} \mathbf {B}(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot } ) = (\mathbf {K}: \mathbf {0}). \end{aligned}$$
(4.3)

Lemma 3

Consider the general linear model \( \fancyscript{A}=\{\mathbf {y}, \mathbf {X}\varvec{\beta }, \mathbf {V}\}\), with random errors denoted \(\varvec{\varepsilon }\). Then the linear estimator \(\mathbf {G}\mathbf {y}\) is the \({{\mathrm{BLUP}}}\) for \(\mathbf {L}\varvec{\varepsilon }\) if and only if \(\mathbf {G}\) satisfies the equation

$$\begin{aligned} \mathbf {G}(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot }) = (\mathbf {0}: \mathbf {L}\mathbf {V}\mathbf {X}^{\bot })\, , \end{aligned}$$
(4.4)

or equivalently

$$\begin{aligned} \mathbf {G}(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot }) = (\mathbf {0}: {{\mathrm{cov}}}(\mathbf {L}\varvec{\varepsilon },\mathbf {y})\mathbf {X}^{\bot })\,. \end{aligned}$$
(4.5)

Equations (4.2)–(4.4) have always solutions for \(\mathbf {A}\), \(\mathbf {B}\) and \(\mathbf {G}\) [see e.g. (Puntanen et al. (2011), p. 267)] but the solutions are unique if and only if \({{\mathrm{rank}}}(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot } ) = n\). Even though such matrices \(\mathbf {A}, \mathbf {B}\) and \(\mathbf {G}\) may not be unique, the numerical observed values of \(\mathbf {A}\mathbf {y}\), \(\mathbf {B}\mathbf {y}\) and \(\mathbf {G}\mathbf {y}\) are unique (with probability \(1\)) once the random vector \(\mathbf {y}\) has obtained its value in the space

$$\begin{aligned} {{\mathrm{\fancyscript{C}}}}(\mathbf {X}: \mathbf {V})={{\mathrm{\fancyscript{C}}}}(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot } ) \,. \end{aligned}$$
(4.6)

[For the last equality see e.g. (Puntanen et al. (2011), p. 123).]

1.1 Proof of Theorem 1

Proof

Because \(\mathbf {M}_{2} = \mathbf {F}\mathbf {U}= \mathbf {U}'\mathbf {F}' \) for some \(\mathbf {U}\), we conclude that (a) holds. Now, if \(\mathbf {G}\mathbf {y}\) is the \({{\mathrm{BLUE}}}\) for \(\mathbf {F}'\mathbf {X}_{1}\varvec{\beta }_{1}\) under \(\fancyscript{A}_{12}\), then the matrix \(\mathbf {G}\) is a solution to

$$\begin{aligned} \mathbf {G}\bigl [\mathbf {X}_{1}:\mathbf {X}_{2}:\mathbf {V}\mathbf {(}\mathbf {X}_{1}: \mathbf {X}_{2})^\bot \bigr ] =( \mathbf {F}'\mathbf {X}_{1} : \mathbf {0}: \mathbf {0}). \end{aligned}$$
(4.7)

Since \(\mathbf {G}\mathbf {X}_{2}=\mathbf {0}\), matrix \(\mathbf {G}\) can be expressed as

$$\begin{aligned} \mathbf {G}=\mathbf {A}\mathbf {M}_{2}=\mathbf {A}\mathbf {U}'\mathbf {F}' \quad \text { for some } \mathbf {A}. \end{aligned}$$
(4.8)

Moreover, in view of (Rao and Mitra (1971), Complement 7, p. 118), we have

$$\begin{aligned} \mathbf {F}(\mathbf {F}'\mathbf {X}_{1})^\bot \in \{ ( \mathbf {X}_{1}:\mathbf {X}_{2})^\bot \}. \end{aligned}$$
(4.9)

In light of (4.8) and (4.9), (4.7) can be written as

$$\begin{aligned} \mathbf {A}\mathbf {U}'\mathbf {F}' \bigl [\mathbf {X}_{1} : \mathbf {X}_{2}:\mathbf {V}\mathbf {F}(\mathbf {F}' \mathbf {X}_{1})^\bot \bigr ] =( \mathbf {F}'\mathbf {X}_{1} : \mathbf {0}: \mathbf {0}), \end{aligned}$$
(4.10)

from which we see that \(\mathbf {A}\mathbf {U}'\) fulfills the fundamental equation for the \({{\mathrm{BLUE}}}\) for \(\mathbf {F}'\mathbf {X}_{1}\varvec{\beta }_{1}\) under \(\fancyscript{A}_{\mathbf {F}}\) and so \(\mathbf {A}\mathbf {U}'(\mathbf {F}'\mathbf {y})=\mathbf {G}\mathbf {y}\) is the \({{\mathrm{BLUE}}}\) for \(\mathbf {F}'\mathbf {X}_{1}\varvec{\beta }_{1}\) under \(\fancyscript{A}_{\mathbf {F}}\).

To go the other way, taking a representation \(\mathbf {B}\mathbf {F}'\mathbf {y}\) of the \({{\mathrm{BLUE}}}\) of \(\mathbf {F}'\mathbf {X}_{1}\varvec{\beta }_{1}\) under \(\fancyscript{A}_{\mathbf {F}}\), i.e., \(\mathbf {B}\) is an arbitrary solution to

$$\begin{aligned} \mathbf {B}\bigl [\mathbf {F}' \mathbf {X}_{1}:\mathbf {F}'\mathbf {V}\mathbf {F}(\mathbf {F}'\mathbf {X}_{1})^\bot \bigr ]=(\mathbf {F}'\mathbf {X}_{1}: \mathbf {0}), \end{aligned}$$
(4.11)

it can be shown (using \(\mathbf {F}' \mathbf {X}_{2} = \mathbf {0}\)) that \(\mathbf {B}\mathbf {F}'\) fulfills the fundamental equation for the \({{\mathrm{BLUE}}}\) of \(\mathbf {F}'\mathbf {X}_{1}\varvec{\beta }_{1}\) under \(\fancyscript{A}_{12}\). \(\square \)

1.2 Proof of Theorem 3

Proof

Let \(\mathbf {G}\mathbf {y}\) be the \({{\mathrm{BLUP}}}\) for \(\mathbf {L}\varvec{\varepsilon }_{12}\) under \(\fancyscript{A}_{12}\). Then \(\mathbf {G}\) fulfills Eq. (4.4), i.e.,

$$\begin{aligned} \mathbf {G}(\mathbf {X}:\mathbf {V}\mathbf {X}^\bot )=(\mathbf {0}:\mathbf {L}\mathbf {V}\mathbf {X}^\bot ), \end{aligned}$$
(4.12)

where \(\mathbf {X}=(\mathbf {X}_1:\mathbf {X}_2)\). First we observe that \( \mathbf {M}_{2} = \mathbf {F}\mathbf {U}= \mathbf {U}'\mathbf {F}'\) for some \(\mathbf {U}\), and since \(\mathbf {G}\mathbf {X}_{2}=\mathbf {0}\), matrix \(\mathbf {G}\) can be expressed as

$$\begin{aligned} \mathbf {G}=\mathbf {A}\mathbf {M}_{2}=\mathbf {A}\mathbf {U}'\mathbf {F}' \quad \text { for some}\,\, \mathbf {A}. \end{aligned}$$
(4.13)

Thus, combining (4.9) and (4.13), from (4.12) we have

$$\begin{aligned} \mathbf {A}\mathbf {U}'\mathbf {F}'\bigl [\mathbf {X}:\mathbf {V}\mathbf {F}(\mathbf {F}'\mathbf {X}_{1})^\bot \bigr ] = \bigl [\mathbf {0}:\mathbf {L}\mathbf {V}\mathbf {F}(\mathbf {F}'\mathbf {X}_{1})^\bot \bigr ] \end{aligned}$$
(4.14)

and so \(\mathbf {A}\mathbf {U}'\) fulfills the Eq. (4.4) for the \({{\mathrm{BLUP}}}\) of \(\mathbf {L}\varvec{\varepsilon }_{12}\) under \(\fancyscript{A}_{\mathbf {F}}\). [Notice that \(\mathbf {L}\mathbf {V}\mathbf {F}={{\mathrm{cov}}}(\mathbf {L}\varvec{\varepsilon }_{12},\mathbf {F}'\mathbf {y})\), as required.] In other words, \(\mathbf {A}\mathbf {U}'\mathbf {F}'\mathbf {y}=\mathbf {G}\mathbf {y}\) is the \({{\mathrm{BLUP}}}\) for \(\mathbf {L}\varvec{\varepsilon }_{12}\) under \(\fancyscript{A}_{\mathbf {F}}\).

For the opposite direction, take a representation \(\mathbf {W}\mathbf {F}'\mathbf {y}\) of the \({{\mathrm{BLUP}}}\) for \(\mathbf {L}\varvec{\varepsilon }_{12}\) under \(\fancyscript{A}_{\mathbf {F}}\). Then

$$\begin{aligned} \mathbf {W}\bigl [ \mathbf {F}'\mathbf {X}_1:\mathbf {F}'\mathbf {V}\mathbf {F}(\mathbf {F}'\mathbf {X}_{1})^\bot \bigr ] = \bigl [ \mathbf {0}:\mathbf {L}\mathbf {V}\mathbf {F}(\mathbf {F}'\mathbf {X}_{1})^\bot \bigr ] \end{aligned}$$
(4.15)

and hence, using (4.9), we can conclude that \(\mathbf {W}\mathbf {F}'\mathbf {y}\) is the \({{\mathrm{BLUP}}}\) for \(\mathbf {L}\varvec{\varepsilon }_{12}\) under \(\fancyscript{A}_{12}\). \(\square \)

1.3 Proof of Lemma 1

Proof

Take an arbitrary representation \(\mathbf {G}\mathbf {y}\) of the \({{\mathrm{BLUP}}}\) for \(\mathbf {L}\varvec{\varepsilon }\). Then \(\mathbf {G}\) is a solution to

$$\begin{aligned} \mathbf {G}(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot }) = ( \mathbf {0}: \mathbf {L}\mathbf {V}\mathbf {X}^{\bot }) \,. \end{aligned}$$
(4.16)

Thus

$$\begin{aligned} \mathbf {G}=(\mathbf {0}: \mathbf {L}\mathbf {V}\mathbf {X}^{\bot })(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot })^{+} + \mathbf {N}_{1} \mathbf {Q}_{(\mathbf {X}: \mathbf {V})} \, , \end{aligned}$$
(4.17)

for some \(\mathbf {N}_1\). We observe that

$$\begin{aligned} \mathbf {G}=\mathbf {L}(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot })(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot })^{+} - \mathbf {B}_{1} = \mathbf {L}\mathbf {P}_{(\mathbf {X}:\mathbf {V})}-\mathbf {B}_{1} \,, \end{aligned}$$
(4.18)

where

$$\begin{aligned} \mathbf {B}_{1} = \mathbf {L}(\mathbf {X}: \mathbf {0})(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot })^{+} - \mathbf {N}_{1} \mathbf {Q}_{(\mathbf {X}: \mathbf {V})} \,, \end{aligned}$$
(4.19)

which satisfies the fundamental equation for the \({{\mathrm{BLUE}}}\) of \(\mathbf {L}\mathbf {X}\varvec{\beta }\),

$$\begin{aligned} \mathbf {B}_{1}(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot }) = (\mathbf {L}\mathbf {X}: \mathbf {0}) \, . \end{aligned}$$
(4.20)

In the other direction, take an arbitrary representation \(\mathbf {B}\mathbf {y}\) for the \({{\mathrm{BLUE}}}\) of \(\mathbf {L}\mathbf {X}\varvec{\beta }\). Then

$$\begin{aligned} \mathbf {B}=(\mathbf {L}\mathbf {X}: \mathbf {0})(\mathbf {X}: \mathbf {V}\mathbf {X}^{\bot })^{+} + \mathbf {N}_{2}\mathbf {Q}_{(\mathbf {X}: \mathbf {V})} \, , \end{aligned}$$
(4.21)

for some \(\mathbf {N}_2\). Because \(\mathbf {L}\mathbf {P}_{(\mathbf {X}:\mathbf {V})}-\mathbf {B}\) fulfills Eq. (4.16), the predictor \((\mathbf {L}\mathbf {P}_{(\mathbf {X}:\mathbf {V})}-\mathbf {B})\mathbf {y}\) is the \({{\mathrm{BLUP}}}\) for \(\mathbf {L}\varvec{\varepsilon }\). \(\square \)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Arendacká, B., Puntanen, S. Further remarks on the connection between fixed linear model and mixed linear model. Stat Papers 56, 1235–1247 (2015). https://doi.org/10.1007/s00362-014-0634-2

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00362-014-0634-2

Keywords

Mathematics Subject Classification

Navigation