, Volume 79, Issue 8, pp 919–951 | Cite as

Nonparametric estimation in a mixed-effect Ornstein–Uhlenbeck model



Two adaptive nonparametric procedures are proposed to estimate the density of the random effects in a mixed-effect Ornstein–Uhlenbeck model. First a kernel estimator is introduced with a new bandwidth selection method developed recently by Goldenshluger and Lepski (Ann Stat 39:1608–1632, 2011). Then, we adapt an estimator from Comte et al. (Stoch Process Appl 7:2522–2551, 2013) in the framework of small time interval of observation. More precisely, we propose an estimator that uses deconvolution tools and depends on two tuning parameters to be chosen in a data-driven way. The selection of these two parameters is achieved through a two-dimensional penalized criterion. For both adaptive estimators, risk bounds are provided in terms of integrated \(\mathbb {L}^2\)-error. The estimators are evaluated on simulations and show good results. Finally, these nonparametric estimators are applied to neuronal data and are compared with previous parametric estimations.


Stochastic differential equations Ornstein–Uhlenbeck process Mixed-effect model Nonparametric estimation Deconvolution method Kernel estimator Neuronal data 

Mathematics Subject Classification

62G07 62M05 


  1. Birgé L, Massart P (1997) From model selection to adaptive estimation. Springer, New YorkCrossRefMATHGoogle Scholar
  2. Birgé L, Massart P (1998) Minimum contrast estimators on sieves: exponential bounds and rates of convergence. Bernoulli 4:329–375MathSciNetCrossRefMATHGoogle Scholar
  3. Bissantz N, Dümbgen L, Holzmann H, Munk A (2007) Nonparametric confidence bands in deconvolution density estimation. J R Stat Soc Series B (Stat Methodol) 69:483–506CrossRefGoogle Scholar
  4. Briane M, Pagès G (2006) Théorie de l’intégration. Vuibert, ParisGoogle Scholar
  5. Butucea C, Tsybakov A (2007) Sharp optimality in density deconvolution with dominating bias II. Teor Veroyatnost i Primenen 52:336–349MathSciNetCrossRefMATHGoogle Scholar
  6. Carroll R, Hall P (1988) Optimal rates of convergence for deconvolving a density. J Am Stat Assoc 83:1184–1186 ISSN 01621459MathSciNetCrossRefMATHGoogle Scholar
  7. Chagny G (2013) Warped bases for conditional density estimation. Math Methods Stat 22:253–282MathSciNetCrossRefMATHGoogle Scholar
  8. Comte F, Genon-Catalot V, Rozenholc Y (2007) Penalized nonparametric mean square estimation of the coefficients of diffusion processes. Bernoulli 13:514–543MathSciNetCrossRefMATHGoogle Scholar
  9. Comte F, Genon-Catalot V, Samson A (2013) Nonparametric estimation for stochastic differential equation with random effects. Stoch Process Appl 7:2522–2551MathSciNetCrossRefMATHGoogle Scholar
  10. Comte F, Johannes J (2012) Adaptive functional linear regression. Ann Stat 40:2765–2797MathSciNetCrossRefMATHGoogle Scholar
  11. Comte F, Rozenholc Y, Taupin M-L (2006) Penalized contrast estimator for adaptive density deconvolution. Can J Stat 34:431–452MathSciNetCrossRefMATHGoogle Scholar
  12. Comte F, Samson A (2012) Nonparametric estimation of random-effects densities in linear mixed-effects model. J Nonparametr Stat 24:951–975MathSciNetCrossRefMATHGoogle Scholar
  13. Davidian M, Giltinan D (1995) Nonlinear models for repeated measurement data. CRC pressGoogle Scholar
  14. Delattre M, Genon-Catalot V, Samson A (2015) Estimation of population parameters in stochastic differential equations with random effects in the diffusion coefficient. ESAIM Probab Stat 19:671–688MathSciNetCrossRefMATHGoogle Scholar
  15. Delattre M, Genon-Catalot V, Samson A (2016) Mixtures of stochastic differential equations with random effects: application to data clustering. J Stat Plan Inference 173:109–124MathSciNetCrossRefMATHGoogle Scholar
  16. Delattre M, Lavielle M (2013) Coupling the SAEM algorithm and the extended Kalman filter for maximum likelihood estimation in mixed-effects diffusion models. Stat Interface 6:519–532MathSciNetCrossRefMATHGoogle Scholar
  17. Diggle P, Heagerty P, Liang K, Zeger S (2002) Analysis of longitudinal data. Oxford statistical science seriesGoogle Scholar
  18. Dion C, Genon-Catalot V (2015) Bidimensional random effect estimation in mixed stochastic differential model. Stoch Inference Stoch Process 18(3):1–28MATHGoogle Scholar
  19. Donnet S, Foulley J, Samson A (2010) Bayesian analysis of growth curves using mixed models defined by stochastic differential equations. Biometrics 66:733–741MathSciNetCrossRefMATHGoogle Scholar
  20. Donnet S, Samson A (2008) Parametric inference for mixed models defined by stochastic differential equations. ESAIM Prob Stat 12:196–218MathSciNetCrossRefMATHGoogle Scholar
  21. Donnet S, Samson A (2013) A review on estimation of stochastic differential equations for pharmacokinetic–pharmacodynamic models. Adv Drug Deliv Rev 65:929–939CrossRefGoogle Scholar
  22. Donnet S, Samson A (2014) Using PMCMC in EM algorithm for stochastic mixed models: theoretical and practical issues. J Soc Fr Stat 155:49–72MathSciNetMATHGoogle Scholar
  23. Fan J (1991) On the optimal rates of convergence for nonparametric deconvolution problems. Ann Statist 19:1257–1272MathSciNetCrossRefMATHGoogle Scholar
  24. Genon-Catalot V, Jacod J (1993) On the estimation of the diffusion coefficient for multi-dimensional diffusion processes. Ann Inst Henri Poincaré B Probab Stat 29:119–151MathSciNetMATHGoogle Scholar
  25. Genon-Catalot V, Larédo C (2016) Estimation for stochastic differential equations with mixed effects. Statistics. doi:10.1080/02331888.2016.1141910 MathSciNetMATHGoogle Scholar
  26. Goldenshluger A, Lepski O (2011) Bandwidth selection in kernel density estimation: oracle inequalities and adaptive minimax optimality. Ann Stat 39:1608–1632MathSciNetCrossRefMATHGoogle Scholar
  27. Hoffmann M (1999) Adaptive estimation in diffusion processes. Stoch Process Appl 79:135–163MathSciNetCrossRefMATHGoogle Scholar
  28. Klein T, Rio E (2005) Concentration around the mean for maxima of empirical processes. Ann Probab 33:1060–1077MathSciNetCrossRefMATHGoogle Scholar
  29. Kutoyants Y (2004) Statistical inference for ergodic diffusion processes. Springer, LondonCrossRefMATHGoogle Scholar
  30. Lacour C (2006) Rates of convergence for nonparametric deconvolution. C R Math Acad Sci Paris 342:877–882MathSciNetCrossRefMATHGoogle Scholar
  31. Lacour C, Massart P (2016) Minimal penalty for Goldenshluger–Lepski method. Stoch Processes Appl. doi:10.1016/
  32. Lansky P, Sanda P, He J (2006) The parameters of the stochastic leaky integrate-and-fire neuronal model. J Comput Neurosci 21:211–223MathSciNetCrossRefMATHGoogle Scholar
  33. Picchini U, De Gaetano A, Ditlevsen S (2010) Stochastic differential mixed-effects models. Scand J Stat 37:67–90MathSciNetCrossRefMATHGoogle Scholar
  34. Picchini U, Ditlevsen S (2011) Practicle estimation of high dimensional stochastic differential mixed-effects models. Comput Stat Data Anal 55:1426–1444MathSciNetCrossRefMATHGoogle Scholar
  35. Picchini U, Ditlevsen S, De Gaetano A, Lansky P (2008) Parameters of the diffusion leaky integrate-and-fire neuronal model for a slowly fluctuating signal. Neural Comput 20:2696–2714CrossRefMATHGoogle Scholar
  36. Pinheiro J, Bates D (2000) Mixed-effect models in S and Splus. Springer, New YorkCrossRefGoogle Scholar
  37. Talagrand M (1996) New concentration inequalities in product spaces. Invent Math 126:505–563MathSciNetCrossRefMATHGoogle Scholar
  38. Yu Y, Xiong Y, Chan Y, He J (2004) Corticofugal gating of auditory information in the thalamus: an in vivo intracellular recording study. J Neurosci 24:3060–3069CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.LJK, UMR CNRS 5224Université Grenoble AlpesGrenobleFrance
  2. 2.MAP5, UMR CNRS 8145Université Paris DescartesParisFrance

Personalised recommendations