Statistics and Computing

, Volume 29, Issue 3, pp 483–500 | Cite as

On the estimation of variance parameters in non-standard generalised linear mixed models: application to penalised smoothing

  • María Xosé Rodríguez-ÁlvarezEmail author
  • Maria Durban
  • Dae-Jin Lee
  • Paul H. C. Eilers


We present a novel method for the estimation of variance parameters in generalised linear mixed models. The method has its roots in Harville (J Am Stat Assoc 72(358):320–338, 1977)’s work, but it is able to deal with models that have a precision matrix for the random effect vector that is linear in the inverse of the variance parameters (i.e., the precision parameters). We call the method SOP (separation of overlapping precision matrices). SOP is based on applying the method of successive approximations to easy-to-compute estimate updates of the variance parameters. These estimate updates have an appealing form: they are the ratio of a (weighted) sum of squares to a quantity related to effective degrees of freedom. We provide the sufficient and necessary conditions for these estimates to be strictly positive. An important application field of SOP is penalised regression estimation of models where multiple quadratic penalties act on the same regression coefficients. We discuss in detail two of those models: penalised splines for locally adaptive smoothness and for hierarchical curve data. Several data examples in these settings are presented.


Generalised linear mixed models Generalised additive models Variance parameters Smoothing parameters REML Effective degrees of freedom 



This research was supported by the Basque Government through the BERC 2018-2021 program and by Spanish Ministry of Economy and Competitiveness MINECO through BCAM Severo Ochoa excellence accreditation SEV-2013-0323 and through projects MTM2017-82379-R funded by (AEI/FEDER, UE) and acronym “AFTERAM”, MTM2014-52184-P and MTM2014-55966-P. The MRI/DTI data were collected at Johns Hopkins University and the Kennedy-Krieger Institute. We are grateful to Pedro Caro and Iain Currie for useful discussions, to Martin Boer and Cajo ter Braak for the detailed reading of the paper and their many suggestions, and to Bas Engel for sharing with us his knowledge. We are also grateful to the two peer referees for their constructive comments of the paper.


  1. Breslow, N.E., Clayton, D.G.: Approximate inference in generalized linear mixed models. J. Am. Stat. Assoc. 88(421), 9–25 (1993)zbMATHGoogle Scholar
  2. Camarda, C.G., Eilers, P.H., Gampe, J.: Sums of smooth exponentials to decompose complex series of counts. Stat. Model. 16(4), 279–296 (2016)MathSciNetGoogle Scholar
  3. Crainiceanu, C.M., Ruppert, D., Carroll, R.J., Joshi, A., Goodner, B.: Spatially adaptive Bayesian penalized splines with heteroscedastic errors. J. Comput. Graph. Stat. 16(2), 265–288 (2007)MathSciNetGoogle Scholar
  4. Crump, S.L.: The present status of variance component analysis. Biometrics 7(1), 1–16 (1951)MathSciNetGoogle Scholar
  5. Cui, Y., Hodges, J.S., Kong, X., Carlin, B.P.: Partitioning degrees of freedom in hierarchical and other richly-parameterized models. Technometrics 52, 124–136 (2010)MathSciNetGoogle Scholar
  6. Currie, I.D., Durban, M.: Flexible smoothing with P-splines: a unified approach. Stat. Model. 2(4), 333–349 (2002)MathSciNetzbMATHGoogle Scholar
  7. Currie, I.D., Durban, M., Eilers, P.H.C.: Generalized linear array models with applications to multidimensional smoothing. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 68(2), 259–280 (2006)MathSciNetzbMATHGoogle Scholar
  8. Davies, P.L., Gather, U., Meise, M., Mergel, D., Mildenberger, T.: Residual-based localization and quantification of peaks in X-ray diffractograms. Ann. Appl. Stat. 2(3), 861–886 (2008)MathSciNetzbMATHGoogle Scholar
  9. Davies, P.L., Gather, U., Meise, M. Mergel, D., Mildenberger, T., Bernholt, T., Hofmeister, T.: diffractometry: baseline identification and peak decomposition for x-ray diffractograms. R package version 0.1-10 (2018)Google Scholar
  10. Djeundje, V.A., Currie, I.D.: Appropriate covariance-specification via penalties for penalized splines in mixed models for longitudinal data. Electron. J. Stat. 4, 1202–1224 (2010)MathSciNetzbMATHGoogle Scholar
  11. Durban, M., Aguilera-Morillo, M.C.: On the estimation of functional random effects. Stat. Model. 17(1–2), 50–58 (2017)MathSciNetGoogle Scholar
  12. Durban, M., Harezlak, J., Wand, M.P., Carroll, R.J.: Simple fitting of subject-specific curves for longitudinal data. Stat. Med. 24(8), 1153–1167 (2005)MathSciNetGoogle Scholar
  13. Eilers, P.H.C.: Discussion of Verbyla et al. J. R. Stat. Soc. Ser. C (Appl. Stat.) 48, 300–311 (1999)Google Scholar
  14. Eilers, P.H.C., Marx, B.D.: Flexible smoothing with B-splines and penalties. Stat. Sci. 11(2), 89–121 (1996)MathSciNetzbMATHGoogle Scholar
  15. Engel, B.: The analysis of unbalanced linear models with variance components. Stat. Neerl. 44, 195–219 (1990)MathSciNetGoogle Scholar
  16. Engel, B., Buist, W.: Analysis of a generalized linear mixed model: a case study and simulation results. Biom. J. 38(1), 61–80 (1996)zbMATHGoogle Scholar
  17. Engel, B., Keen, A.: A simple approach for the analysis of generalizea linear mixed models. Stat. Neerl. 48(1), 1–22 (1994)zbMATHGoogle Scholar
  18. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)MathSciNetzbMATHGoogle Scholar
  19. Gilmour, A.R., Thompson, R., Cullis, B.R.: Average information REML: an efficient algorithm for variance parameter estimation in linear mixed models. Biometrics 51(4), 1440–1450 (1995)zbMATHGoogle Scholar
  20. Goldsmith, J., Bobb, J., Crainiceanu, C.M., Caffo, B., Reich, D.: Penalized functional regression. J. Comput. Graph. Stat. 20(4), 830–851 (2011)MathSciNetGoogle Scholar
  21. Goldsmith, J., Crainiceanu, C.M., Caffo, B., Reich, D.: Longitudinal penalized functional regression for cognitive outcomes on neuronal tract measurements. J. R. Stat. Soc. Ser. C (Appl. Stat.) 61(3), 453–469 (2012)MathSciNetGoogle Scholar
  22. Goldsmith, J., Scheipl, F., Huang, L., Wrobel, J., Gellar, J., Harezlak, J., McLean, M.W., Swihart, B., Xiao, L., Crainiceanu, C., Reiss, P.T.: refund: Regression with Functional Data. R package version 0.1-16 (2016)Google Scholar
  23. Graser, H.-U., Smith, S.P., Tier, B.: A derivative-free approach for estimating variance components in animal models by restricted maximum likelihood. J. Anim. Sci. 2(64), 1362–1373 (1987)Google Scholar
  24. Green, P.J.: Penalized likelihood for general semi-parametric regression models. Int. Stat. Rev./Revue Internationale de Statistique 55(3), 245–259 (1987)MathSciNetzbMATHGoogle Scholar
  25. Greven, S., Scheipl, F.: A general framework for functional regression modelling. Stat. Model. 17(1–2), 1–35 (2017)MathSciNetGoogle Scholar
  26. Groll, A., Tutz, G.: Variable selection for generalized linear mixed models by L1-penalized estimation. Stat. Comput. 24(2), 137–154 (2014)MathSciNetzbMATHGoogle Scholar
  27. Harville, D.A.: Maximum likelihood approaches to variance component estimation and to related problems. J. Am. Stat. Assoc. 72(358), 320–338 (1977)MathSciNetzbMATHGoogle Scholar
  28. Harville, D.A.: Matrix Algebra from a Statistician’s Perspective. Springer, Berlin (1997)zbMATHGoogle Scholar
  29. Hastie, T.J., Tibshirani, R.J.: Generalized Additive Models. Chapman & Hall, London (1990)zbMATHGoogle Scholar
  30. Heckman, N., Lockhart, R., Nielsen, J.D.: Penalized regression, mixed effects models and appropriate modelling. Electron. J. Stat. 7, 1517–1552 (2013)MathSciNetzbMATHGoogle Scholar
  31. Henderson, C.R.: Selection index and expected genetic advance. Stat. Genet. Plant Breed. 982, 141–163 (1963)Google Scholar
  32. Hunter, D.R., Li, R.: Variable selection using MM algorithms. Ann. Stat. 33(4), 1617–1642 (2005)MathSciNetzbMATHGoogle Scholar
  33. Johnson, D.L., Thompson, R.: Restricted maximum likelihood estimation of variance components for univariate animal models using sparse matrix techniques and average information. J. Dairy Sci. 78, 449–456 (1995)Google Scholar
  34. Karas, M., Brzyski, D., Dzemidzic, M., Goñi, J., Kareken, D.A., Randolph, T.W., Harezlak, J.: Brain connectivity-informed regularization methods for regression. Stat. Biosci. (2017).
  35. Krivobokova, T.: Smoothing parameter selection in two frameworks for penalized splines. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 75(4), 725–741. (2009)
  36. Krivobokova, T., Crainiceanu, C.M., Kauermann, G.: Fast adaptive penalized splines. J. Comput. Graph. Stat. 17(1), 1–20 (2008)MathSciNetGoogle Scholar
  37. Lee, D.-J.: Smoothing mixed model for spatial and spatio-temporal data. PhD thesis. Department of Statistics, Universidad Carlos III de Madrid, Spain (2010)Google Scholar
  38. McCullagh, P., Nelder, J.: Generalized Linear Models. Chapman and Hall/CRC Monographs on Statistics and Applied Probability Series, 2nd edn. Chapman & Hall, London (1989)Google Scholar
  39. Patterson, H.D., Thompson, R.: Recovery of inter-block information when block sizes are unequal. Biometrika 58(3), 545–554 (1971)MathSciNetzbMATHGoogle Scholar
  40. R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2018)Google Scholar
  41. Reiss, P .T., Ogden, R .T.: Smoothing parameter selection for a class of semiparametric linear models. J. R. Stat. Soc. Ser. B Stat. Methodol. 71(2), 505–523 (2009)MathSciNetzbMATHGoogle Scholar
  42. Rodríguez-Álvarez, M.X., Durban, M., Lee, D.-J., Eilers, P.H.C.: Fast estimation of multidimensional adaptive p-spline models. In: Friedl, H., Wagner, H. (eds.) Proceedings of the 30th International Workshop on Statistical Modelling, pp 330 – 335. arXiv:1610.06861 (2015a)
  43. Rodríguez-Álvarez, M.X., Lee, D.-J., Kneib, T., Durban, M., Eilers, P.H.C.: Fast smoothing parameter separation in multidimensional generalized P-splines: the sap algorithm. Stat. Comput. 25, 941–957 (2015b)MathSciNetzbMATHGoogle Scholar
  44. Rodríguez-Álvarez, M.X., Durban, M., Lee, D.-J., Eilers, P.H.C., Gonzalez, F.: Spatio-temporal adaptive penalized splines with application to neuroscience. In: Dupuy, J.-F., Josse, J. (eds.) Proceedings of the 31th International Workshop on Statistical Modelling, pp. 267–272. arXiv:1610.06860 (2016)
  45. Rodríguez-Álvarez, M.X., Boer, M.P., van Eeuwijk, F.A., Eilers, P.H.: Correcting for spatial heterogeneity in plant breeding experiments with p-splines. Spat. Stat. 23, 52–71 (2018)MathSciNetGoogle Scholar
  46. Ruppert, D., Carroll, R.J.: Spatially-adaptive penalties for spline fitting. Aust. N. Z. J. Stat. 42(2), 205–223 (2000)Google Scholar
  47. Ruppert, D., Wand, M.P., Carroll, R.: Semiparametric Regression. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  48. Schall, R.: Estimation in generalized linear models with random effects. Biometrika 78(4), 719–727 (1991)MathSciNetzbMATHGoogle Scholar
  49. Simpkin, A., Newell, J.: An additive penalty p-spline approach to derivative estimation. Comput. Stat. Data Anal. 68, 30–43 (2013)MathSciNetzbMATHGoogle Scholar
  50. Smith S.P.: Estimation of genetic parameters in non-linear models. In: Gianola, D., Hammond, K. (eds.) Advances in Statistical Methods for Genetic Improvement of Livestock. Advanced Series in Agricultural Sciences, vol. 18. Springer, Berlin, Heidelberg (1990)Google Scholar
  51. Taylor, J.D., Verbyla, A.P., Cavanagh, C., Newberry, M.: Variable selection in linear mixed models using an extended class of penalties. Aust. N. Z. J. Stat. 54(4), 427–449 (2012)MathSciNetzbMATHGoogle Scholar
  52. Tibshirani, R.J.: Adaptive piecewise polynomial estimation via trend filtering. Ann. Stat. 42(1), 285–323 (2014)MathSciNetzbMATHGoogle Scholar
  53. Wand, M.P.: Smoothing and mixed models. Comput. Stat. 18(2), 223–249 (2003)zbMATHGoogle Scholar
  54. Wood, S.N.: Fast stable direct fitting and smoothness selection for generalized additive models. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 70(3), 495–518 (2008)MathSciNetzbMATHGoogle Scholar
  55. Wood, S .N.: Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 73(1), 2–36 (2011)MathSciNetGoogle Scholar
  56. Wood, S.N.: Generalized Additive Models: An Introduction with R, 2nd edn. Chapman & Hall CRC, London (2017)zbMATHGoogle Scholar
  57. Wood, S.N., Fasiolo, M.: A generalized Fellner-Schall method for smoothing parameter optimization with application to Tweedie location, scale and shape models. Biometrics 73, 1071–1081 (2017)MathSciNetzbMATHGoogle Scholar
  58. Wood, S.N., Pya, N., Säfken, B.: Smoothing parameter and model selection for general smooth models. J. Am. Stat. Assoc. 111(516), 1548–1563 (2016)MathSciNetGoogle Scholar
  59. Zou, H., Li, R.: One-step sparse estimates in nonconcave penalized likelihood models. Ann. Stat. 36(4), 1509–1533 (2008)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.BCAM - Basque Center for Applied MathematicsBilbaoSpain
  2. 2.IKERBASQUE, Basque Foundation for ScienceBilbaoSpain
  3. 3.Department of Statistics and EconometricsUniversidad Carlos III de MadridLeganésSpain
  4. 4.Erasmus University Medical CentreRotterdamThe Netherlands

Personalised recommendations