, Volume 79, Issue 8, pp 919–951 | Cite as

Nonparametric estimation in a mixed-effect Ornstein–Uhlenbeck model

  • Charlotte DionEmail author


Two adaptive nonparametric procedures are proposed to estimate the density of the random effects in a mixed-effect Ornstein–Uhlenbeck model. First a kernel estimator is introduced with a new bandwidth selection method developed recently by Goldenshluger and Lepski (Ann Stat 39:1608–1632, 2011). Then, we adapt an estimator from Comte et al. (Stoch Process Appl 7:2522–2551, 2013) in the framework of small time interval of observation. More precisely, we propose an estimator that uses deconvolution tools and depends on two tuning parameters to be chosen in a data-driven way. The selection of these two parameters is achieved through a two-dimensional penalized criterion. For both adaptive estimators, risk bounds are provided in terms of integrated \(\mathbb {L}^2\)-error. The estimators are evaluated on simulations and show good results. Finally, these nonparametric estimators are applied to neuronal data and are compared with previous parametric estimations.


Stochastic differential equations Ornstein–Uhlenbeck process Mixed-effect model Nonparametric estimation Deconvolution method Kernel estimator Neuronal data 

Mathematics Subject Classification

62G07 62M05 



The author would like to thank Fabienne Comte and Adeline Samson for very useful discussions and advice.


  1. Birgé L, Massart P (1997) From model selection to adaptive estimation. Springer, New YorkCrossRefzbMATHGoogle Scholar
  2. Birgé L, Massart P (1998) Minimum contrast estimators on sieves: exponential bounds and rates of convergence. Bernoulli 4:329–375MathSciNetCrossRefzbMATHGoogle Scholar
  3. Bissantz N, Dümbgen L, Holzmann H, Munk A (2007) Nonparametric confidence bands in deconvolution density estimation. J R Stat Soc Series B (Stat Methodol) 69:483–506CrossRefGoogle Scholar
  4. Briane M, Pagès G (2006) Théorie de l’intégration. Vuibert, ParisGoogle Scholar
  5. Butucea C, Tsybakov A (2007) Sharp optimality in density deconvolution with dominating bias II. Teor Veroyatnost i Primenen 52:336–349MathSciNetCrossRefzbMATHGoogle Scholar
  6. Carroll R, Hall P (1988) Optimal rates of convergence for deconvolving a density. J Am Stat Assoc 83:1184–1186 ISSN 01621459MathSciNetCrossRefzbMATHGoogle Scholar
  7. Chagny G (2013) Warped bases for conditional density estimation. Math Methods Stat 22:253–282MathSciNetCrossRefzbMATHGoogle Scholar
  8. Comte F, Genon-Catalot V, Rozenholc Y (2007) Penalized nonparametric mean square estimation of the coefficients of diffusion processes. Bernoulli 13:514–543MathSciNetCrossRefzbMATHGoogle Scholar
  9. Comte F, Genon-Catalot V, Samson A (2013) Nonparametric estimation for stochastic differential equation with random effects. Stoch Process Appl 7:2522–2551MathSciNetCrossRefzbMATHGoogle Scholar
  10. Comte F, Johannes J (2012) Adaptive functional linear regression. Ann Stat 40:2765–2797MathSciNetCrossRefzbMATHGoogle Scholar
  11. Comte F, Rozenholc Y, Taupin M-L (2006) Penalized contrast estimator for adaptive density deconvolution. Can J Stat 34:431–452MathSciNetCrossRefzbMATHGoogle Scholar
  12. Comte F, Samson A (2012) Nonparametric estimation of random-effects densities in linear mixed-effects model. J Nonparametr Stat 24:951–975MathSciNetCrossRefzbMATHGoogle Scholar
  13. Davidian M, Giltinan D (1995) Nonlinear models for repeated measurement data. CRC pressGoogle Scholar
  14. Delattre M, Genon-Catalot V, Samson A (2015) Estimation of population parameters in stochastic differential equations with random effects in the diffusion coefficient. ESAIM Probab Stat 19:671–688MathSciNetCrossRefzbMATHGoogle Scholar
  15. Delattre M, Genon-Catalot V, Samson A (2016) Mixtures of stochastic differential equations with random effects: application to data clustering. J Stat Plan Inference 173:109–124MathSciNetCrossRefzbMATHGoogle Scholar
  16. Delattre M, Lavielle M (2013) Coupling the SAEM algorithm and the extended Kalman filter for maximum likelihood estimation in mixed-effects diffusion models. Stat Interface 6:519–532MathSciNetCrossRefzbMATHGoogle Scholar
  17. Diggle P, Heagerty P, Liang K, Zeger S (2002) Analysis of longitudinal data. Oxford statistical science seriesGoogle Scholar
  18. Dion C, Genon-Catalot V (2015) Bidimensional random effect estimation in mixed stochastic differential model. Stoch Inference Stoch Process 18(3):1–28zbMATHGoogle Scholar
  19. Donnet S, Foulley J, Samson A (2010) Bayesian analysis of growth curves using mixed models defined by stochastic differential equations. Biometrics 66:733–741MathSciNetCrossRefzbMATHGoogle Scholar
  20. Donnet S, Samson A (2008) Parametric inference for mixed models defined by stochastic differential equations. ESAIM Prob Stat 12:196–218MathSciNetCrossRefzbMATHGoogle Scholar
  21. Donnet S, Samson A (2013) A review on estimation of stochastic differential equations for pharmacokinetic–pharmacodynamic models. Adv Drug Deliv Rev 65:929–939CrossRefGoogle Scholar
  22. Donnet S, Samson A (2014) Using PMCMC in EM algorithm for stochastic mixed models: theoretical and practical issues. J Soc Fr Stat 155:49–72MathSciNetzbMATHGoogle Scholar
  23. Fan J (1991) On the optimal rates of convergence for nonparametric deconvolution problems. Ann Statist 19:1257–1272MathSciNetCrossRefzbMATHGoogle Scholar
  24. Genon-Catalot V, Jacod J (1993) On the estimation of the diffusion coefficient for multi-dimensional diffusion processes. Ann Inst Henri Poincaré B Probab Stat 29:119–151MathSciNetzbMATHGoogle Scholar
  25. Genon-Catalot V, Larédo C (2016) Estimation for stochastic differential equations with mixed effects. Statistics. doi: 10.1080/02331888.2016.1141910 MathSciNetzbMATHGoogle Scholar
  26. Goldenshluger A, Lepski O (2011) Bandwidth selection in kernel density estimation: oracle inequalities and adaptive minimax optimality. Ann Stat 39:1608–1632MathSciNetCrossRefzbMATHGoogle Scholar
  27. Hoffmann M (1999) Adaptive estimation in diffusion processes. Stoch Process Appl 79:135–163MathSciNetCrossRefzbMATHGoogle Scholar
  28. Klein T, Rio E (2005) Concentration around the mean for maxima of empirical processes. Ann Probab 33:1060–1077MathSciNetCrossRefzbMATHGoogle Scholar
  29. Kutoyants Y (2004) Statistical inference for ergodic diffusion processes. Springer, LondonCrossRefzbMATHGoogle Scholar
  30. Lacour C (2006) Rates of convergence for nonparametric deconvolution. C R Math Acad Sci Paris 342:877–882MathSciNetCrossRefzbMATHGoogle Scholar
  31. Lacour C, Massart P (2016) Minimal penalty for Goldenshluger–Lepski method. Stoch Processes Appl. doi: 10.1016/
  32. Lansky P, Sanda P, He J (2006) The parameters of the stochastic leaky integrate-and-fire neuronal model. J Comput Neurosci 21:211–223MathSciNetCrossRefzbMATHGoogle Scholar
  33. Picchini U, De Gaetano A, Ditlevsen S (2010) Stochastic differential mixed-effects models. Scand J Stat 37:67–90MathSciNetCrossRefzbMATHGoogle Scholar
  34. Picchini U, Ditlevsen S (2011) Practicle estimation of high dimensional stochastic differential mixed-effects models. Comput Stat Data Anal 55:1426–1444MathSciNetCrossRefzbMATHGoogle Scholar
  35. Picchini U, Ditlevsen S, De Gaetano A, Lansky P (2008) Parameters of the diffusion leaky integrate-and-fire neuronal model for a slowly fluctuating signal. Neural Comput 20:2696–2714CrossRefzbMATHGoogle Scholar
  36. Pinheiro J, Bates D (2000) Mixed-effect models in S and Splus. Springer, New YorkCrossRefGoogle Scholar
  37. Talagrand M (1996) New concentration inequalities in product spaces. Invent Math 126:505–563MathSciNetCrossRefzbMATHGoogle Scholar
  38. Yu Y, Xiong Y, Chan Y, He J (2004) Corticofugal gating of auditory information in the thalamus: an in vivo intracellular recording study. J Neurosci 24:3060–3069CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.LJK, UMR CNRS 5224Université Grenoble AlpesGrenobleFrance
  2. 2.MAP5, UMR CNRS 8145Université Paris DescartesParisFrance

Personalised recommendations