Nonlinear nonparametric mixed-effects models for unsupervised classification

Abstract

In this work we propose a novel EM method for the estimation of nonlinear nonparametric mixed-effects models, aimed at unsupervised classification. We perform simulation studies in order to evaluate the algorithm performance and we apply this new procedure to a real dataset.

Appendices

Appendix A: Proof of increasing likelihood property of the EM algorithm

In Sect. 2.2 we propose an EM algorithm for the estimation of the parameters of model (1). The update of the parameter described by Eqs. (3) and (4) provides an increasing of the likelihood function (2), that is

$$\begin{aligned} L\left(\left.\varvec{\beta }^{(up)}, \sigma ^{2(up)}\right| \mathbf y \right) \ge L\left(\left.\varvec{\beta }, \sigma ^2\right| \mathbf y \right) \end{aligned}$$

where $\beta ^{(up)}$ and $\sigma ^{2(up)}$ are the updated fixed effects and error variance. The likelihood $L(\varvec{\beta }^{(up)}, \sigma ^{2(up)}| \mathbf y )$ is computed summing up the random effects with respect to the updated discrete distribution $(\mathbf c _l^{(up)},\omega _l^{(up)})$ for $l=1,\ldots ,M$.

Thanks to the definition of likelihood function (2) we have that

$$\begin{aligned} \log \left[\frac{L\left(\left.\varvec{\beta }^{(up)}, \sigma ^{2(up)}\right|\mathbf y \right)}{L\left(\left.\varvec{\beta }, \sigma ^2\right|\mathbf y \right)}\right]= \sum _{i=1}^N \log \left[\frac{p(\left.\mathbf y _i\right|\varvec{\beta }^{(up)},\sigma ^{2(up)})}{p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2})}\right] \end{aligned}$$

All these terms can be bounded above by the quantity

$$\begin{aligned} \log \left[\frac{p(\left.\mathbf y _i\right|\varvec{\beta }^{(up)},\sigma ^{2(up)})}{p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2})}\right]\ge Q_i(\theta ^{(up)},\theta )-Q_i(\theta ,\theta ) \end{aligned}$$

(9)

where

$$\begin{aligned} Q_i(\theta ^{(up)},\theta )=\sum _{l=1}^M \left(\frac{\omega _l p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2},\mathbf c _l)}{p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2})}\right) \log \left(\omega _l^{(up)} p(\left.\mathbf y _i\right|\varvec{\beta }^{(up)},\sigma ^{2(up)},\mathbf c _l^{(up)})\right). \end{aligned}$$

$Q_i(\theta ,\theta )$ is analogously defined, and $\theta =(\varvec{\beta }, \mathbf c _1,\ldots ,\mathbf c _M,\omega _1,\ldots , \omega _M, \sigma ^2)$.

This bound can be found thanks to the convexity of the logarithm since

$$\begin{aligned}&\log \left[\frac{p(\left.\mathbf y _i\right|\varvec{\beta }^{(up)},\sigma ^{2(up)})}{p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2})}\right]=\log \sum _{l=1}^M \frac{\omega _l^{(up)} p(\left.\mathbf y _i\right|\varvec{\beta }^{(up)},\sigma ^{2(up)},\mathbf c _l^{(up)})}{p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2})}\\&\quad =\log \sum _{l=1}^M \left(\frac{\omega _l p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2},\mathbf c _l)}{p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2})}\right) \left( \frac{\omega _l^{(up)} p(\left.\mathbf y _i\right|\varvec{\beta }^{(up)},\sigma ^{2(up)},\mathbf c _l^{(up)})}{\omega _l p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2},\mathbf c _l)}\right)\\&\quad \ge \sum _{l=1}^M \left(\frac{\omega _l p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2},\mathbf c _l)}{p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2})}\right) \log \left( \frac{\omega _l^{(up)} p(\left.\mathbf y _i\right|\varvec{\beta }^{(up)},\sigma ^{2(up)},\mathbf c _l^{(up)})}{\omega _l p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2},\mathbf c _l)}\right)\\&\quad =Q_i(\theta ^{(up)},\theta )-Q_i(\theta ,\theta ) \end{aligned}$$

Defining

$$\begin{aligned} Q(\theta ^{(up)},\theta )=\sum _{i=1}^N Q_i(\theta ^{(up)},\theta ) \qquad \text{ and} \qquad Q(\theta ,\theta )=\sum _{i=1}^N Q_i(\theta ,\theta ) \end{aligned}$$

we obtain, thanks to Eq. (9), an upper bound for the quantity of interest

$$\begin{aligned} \log \left[\frac{L\left(\left.\varvec{\beta }^{(up)}, \sigma ^{2(up)}\right|\mathbf y \right)}{L\left(\left.\varvec{\beta }, \sigma ^2\right|\mathbf y \right)}\right]\ge Q(\theta ^{(up)},\theta )-Q(\theta ,\theta ) \end{aligned}$$

We have now to show that $\forall \theta$

$$\begin{aligned} Q(\theta ^{(up)},\theta )\ge Q(\theta ,\theta ) \end{aligned}$$

In order to prove this result we can show that, $\forall \theta$ fixed, $\theta ^{(up)}$ is defined as

$$\begin{aligned} Q(\theta ^{(up)},\theta )=\text{ arg}\max _{\tilde{\theta }} Q(\tilde{\theta },\theta ) \end{aligned}$$

Defining $W_{il}$ as in Eq. (5) we obtain

$$\begin{aligned} Q({\tilde{\theta }},\theta )&= \sum _{i=1}^N\sum _{l=1}^M \left(\frac{\omega _l p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2},\mathbf c _l)}{p(\left.\mathbf y _i\right|\varvec{\beta },\sigma ^{2})}\right) \log \left({\tilde{\omega }}_l p(\left.\mathbf y _i\right| {\tilde{\varvec{\beta }}},{\tilde{\sigma }}^2,{\tilde{\mathbf{c }}}_l)\right)\\&= \sum _{i=1}^N\sum _{l=1}^M W_{il} \log \left({\tilde{\omega }}_l p(\left.\mathbf y _i\right|\varvec{\tilde{\beta }},{\tilde{\sigma }}^2,{\tilde{\mathbf{c }}}_l)\right)\\&= \sum _{i=1}^N\sum _{l=1}^M W_{il} \log {\tilde{\omega }}_l+ \sum _{i=1}^N\sum _{l=1}^M W_{il}\log p(\left.\mathbf y _i\right| {\tilde{\varvec{\beta }}},{\tilde{\sigma }}^2,{\tilde{\mathbf{c }}}_l)\\&= J_1({\tilde{\omega }}_1,\ldots ,{\tilde{\omega }}_M)+J_2({\tilde{\varvec{\beta }}}, {\tilde{\mathbf{c }}}_1,\ldots ,{\tilde{\mathbf{c }}}_M, {\tilde{\sigma }}^2) \end{aligned}$$

The functionals $J_1$ and $J_2$ can be maximized separately. The update (3) for the weights of the random effects distribution $\omega _1,\ldots ,\omega _M$ is obtained in closed form maximizing the functional $J_1$.

The functional $J_1$ can be written as

$$\begin{aligned} J_1({\tilde{\omega }}_1,\ldots ,{\tilde{\omega }}_M) = \sum _{l=1}^{M-1}\sum _{i=1}^N W_{il}\,\log {\tilde{\omega }}_l+ \sum _{i=1}^N W_{iM}\,\log \left(1-\sum _{l=1}^{M-1}{\tilde{\omega }}_l\right) \end{aligned}$$

Imposing the gradient of the functional $J_1$ equal to zero we obtain

$$\begin{aligned} \frac{\partial J_1}{\partial {\tilde{\omega }}_l}=\frac{\sum _{i=1}^N W_{il}}{{\tilde{\omega }}_l}-\frac{\sum _{i=1}^N W_{iM}}{{\tilde{\omega }}_M}=0 \qquad \forall l = 1,\dots ,M-1 \end{aligned}$$

that is equivalent to

$$\begin{aligned} \frac{\sum _{i=1}^N W_{il}}{{\tilde{\omega }}_l}=\frac{\sum _{i=1}^N W_{ik}}{{\tilde{\omega }}_k} \qquad \forall l,k = 1,\dots ,M \end{aligned}$$

Since $\sum _{l=1}^M W_{il}=1$, we obtain $\omega _l^{(up)}=\sum _{i=1}^N W_{il}/N$.

On the other hand the update (4) for the fixed effects $\varvec{\beta }$, the error variance $\sigma ^2$ and the support points of the random effects distribution $\mathbf c _1,\ldots ,\mathbf c _M$ is obtained maximizing the functional $J_2$ in an iterative way, described in Sect. 2.2 and in “Appendix B”.

Appendix B: Details on NLNPEM Algorithm

The NLNPEM is the following:

• 1. Define a starting discrete distribution for random effects with support on $M=N$ points $(\mathbf c _l^{(0)},\omega _l^{(0)})$ for $l=1,\ldots ,M$, a starting estimate for the fixed effects $\varvec{\beta }^{(0)}$ and for $\sigma ^{2(0)}$ and the tolerance parameters $D$ and ${\tilde{\omega }}$;

• 2. given $(\mathbf c _l^{(k-1)}, \omega _l^{(k-1)})$ for $l=1,\ldots ,M$, $\varvec{\beta }^{(k-1)}$ and $\sigma ^{2(k-1)}$, update the weights $\omega _1^{(k)},\ldots , \omega _M^{(k)}$ of the random effect distribution, according to Eq. (3);

• 3.

  • (a) initialize $\mathbf c _l^{(k,0)}=\mathbf c _l^{(k-1)}$ for $l=1,\ldots ,M, \varvec{\beta }^{(k,0)}=\varvec{\beta }^{(k-1)}$ and $\sigma ^{2(k,0)}=\sigma ^{2(k-1)}$;

  • (b) given $(\mathbf c _l^{(k,j-1)},\omega _l^{(k-1)})$ for $l=1,\ldots ,M, \varvec{\beta }^{(k,j-1)}$ and $\sigma ^{2(k,j-1)}$ update the $M$ support points $\mathbf c _1^{(k,j)},\ldots ,\mathbf c _M^{(k,j)}$ according to Eq. (6);

  • (c) given $(\mathbf c _l^{(k,j)},\omega _l^{(k-1)})$ for $l=1,\ldots ,M, \varvec{\beta }^{(k,j-1)}$ and $\sigma ^{2(k,j-1)}$ maximize Eq. (4) with respect to $\varvec{\beta }$ and $\sigma ^2$ obtaining $\varvec{\beta }^{(k,j)}$ and $\sigma ^{2(k,j)}$;

  • (d) iterate steps 3(b) and 3(c) until convergence and set $\mathbf c _l^{(k)}=\mathbf c _l^{(k,j)}$ for $l=1,\ldots ,M, \varvec{\beta }^{(k)}=\varvec{\beta }^{(k,j)}$ and $\sigma ^{2(k)}=\sigma ^{2(k,j)}$;

• 4. iterate steps 2 and 3 until convergence;

• 5. reduce the support of the discrete distribution, according with the tuning parameters $D$ and ${\tilde{\omega }}$;

• 6. iterate steps 2, 3 and 5 until convergence.

The algorithm reaches convergence when parameters and discrete distribution stop changing or when there is no variation in the log-likelihood function.

Appendix C: Details on simulation study

1.1 Appendix C.1: The linear case

We simulated 8 datasets of linear curves grouped in a number of clusters that vary form 2 to 10. Different values of the error variance $\sigma ^2$ have been chosen for each test set, in order to obtain noisy observations for each curve. Some examples of simulated data are shown in left panels of Fig. 1. Datasets addressed with the name “S” contain groups in which only slopes is random, “I” datasets contain groups where only intercept is random and “SI” datasets contain curves where both slope and intercept are random. The simulated datasets are then:

• lin2S: 2 balanced groups, each one composed by 25 curves, with the same intercept (equal to $4$), 2 different slopes ($\mathbf c = (c_1,c_2) = (1,2)$) and $\sigma = 1$;

• lin2I: 2 balanced groups, each one composed by 25 curves with the same slope (equal to $1$), 2 different intercept ($\mathbf c = (c_1,c_2) = (3,10)$) and $\sigma = 0.65$;

• lin4SI: 4 balanced groups, each one composed by 25 curves, where location points $\mathbf c = (\mathbf c _1,\mathbf c _2,\mathbf c _3,\mathbf c _4)$ are obtained from all possible combinations of 2 different slopes (equal to $1$ and $3$) and 2 different intercepts (equal to $40$ and $60$), i.e., $\mathbf c _1 = (1,40)$, $\mathbf c _2 = (1,60)$, $\mathbf c _3 = (3,40)$ and $\mathbf c _4 = (3,60)$ with $\sigma = 1$;

• lin3S: 3 unbalanced groups, composed by 24, 24 and 2 curves respectively, with the same intercept (equal to $4$), 3 different slopes ($\mathbf c = (c_1,c_2,c_3) = (1,2,3.5)$) and $\sigma = 1$;

• lin3I: 3 unbalanced groups, composed by 24, 24 and 2 curves respectively, with the same slope (equal to $1$), 3 different intercepts ($\mathbf c = (c_1,c_2,c_3) = (2,7,14)$) and $\sigma = 1$;

• lin9SI: 9 unbalanced groups, 6 of whom containing 24 curves and 3 containing 2 curves, where location points $\mathbf c = (c_1,c_2,c_3,c_4,c_5,c_6,c_7,c_8,c_9)$ are obtained from all possible combinations of 3 different slopes (equal to $1$, $4$ and $7$) and 3 different intercept (equal to $20$, $35$ and $60$) with $\sigma = 1.5$;

• lin10S: 10 balanced groups, each one composed by 50 curves with the same intercept (equal to $1$), 10 different slopes ($\mathbf c = (c_1,c_2,c_3,c_4,c_5,c_6,c_7,c_8,c_9,c_{10}) = (0.5,2,4,5.5,7.5,$ $10,12,13.5,16,20)$) and $\sigma = 1.5$;

• lin10I: 10 balanced groups, each one composed by 15 curves with the same slope (equal to $1$), 10 different intercepts ($\mathbf c = (c_1,c_2,c_3,c_4,c_5,c_6,c_7,c_8,c_9,c_{10}) = (1,5,10,15,20,25,30,$ $35,40,45)$) and $\sigma = 1$.

 

1.2 Appendix C.2: The exponential case

We simulated 3 datasets of exponential growth curves where only asymptote varies and is considered as random. All datasets are then addressed with the name “A”. They are:

• exp2A: 2 balanced groups, each one composed by 25 curves, with the same growth rate ($\lambda =0.5$), 2 different asymptotes ($\mathbf c = (c_1,c_2) = (1,1.5)$) and $\sigma =0.04$;

• exp3A: 3 unbalanced groups of 24, 24 and 2 curves respectively, with the same growth rate ($\lambda =0.5$), 3 different asymptotes ($\mathbf c = (c_1,c_2,c_3) = (1,1.5,2.3)$) and $\sigma =0.04$;

• exp10A: 10 balanced groups, each one composed by 5 curves, with the same growth rate ($\lambda =0.5$), 10 different asymptotes ($\mathbf c = (c_1,c_2,c_3,c_4,c_5,c_6,c_7,c_8,c_9,c_{10})$ $=(1,1.25,1.5,$ $1.75,2,2.25,2.5,2.75,3,3.25))$ and $\sigma =0.04$.

 

1.3 Appendix C.3: The logistic case

We simulated 8 datasets of logistic growth curves. Datasets addressed with the name “A” represent random asymptote cases, “I” datasets contain groups where only inflection point is random and “AI” ones contain curves where both asymptote and inflection point are random. We then have:

• logis2A: 2 balanced groups, each one composed by 25 curves, with $\delta =6$, $\gamma =1$, 2 different asymptotes ($\mathbf c = (c_1,c_2) = (1,2)$) and $\sigma =0.04$;

• logis2I: 2 balanced groups, each one composed by 25 curves, with $\alpha =1$, $\gamma =1$, 2 different inflection points ($\mathbf c = (c_1,c_2) = (6,8)$) and $\sigma =0.04$;

• logis4AI: 4 balanced groups, each one composed by 25 curves, where location points $\mathbf c = (\mathbf c _1,\mathbf c _2,\mathbf c _3,\mathbf c _4)$ are obtained from all possible combinations of 2 different asymptotes (equal to $1$ and $2$) and 2 different inflection points (equal to $6$ and $10$), i.e., $\mathbf c _1 = (1,6)$, $\mathbf c _1 = (1,10)$, $\mathbf c _1 = (2,6)$ and $\mathbf c _4 = (2,10)$ with $\gamma =1$ and $\sigma =0.04$;

• logis3A: 3 unbalanced groups of 24, 24 and 2 curves respectively, with $\delta =6$, $\gamma =1$, 3 different asymptotes ($\mathbf c = (c_1,c_2,c_3) = (1,2,3.5)$) and $\sigma =0.04$;

• logis3I: 3 unbalanced groups of 24, 24 and 2 curves respectively, with $\alpha =1$, $\gamma =1$, 3 different inflection points ($\mathbf c = (c_1,c_2,c_3) = (6,8,11.5)$) and $\sigma =0.04$;

• logis9AI: 9 unbalanced groups of curves (6 of whom containing 24 curves and 3 containing 2 curves), where location points $\mathbf c = (c_1,c_2,c_3,c_4,c_5,c_6,c_7, c_8,c_9)$ are obtained from all possible combinations of 3 different asymptotes (equal to $1$, $2$ and $4$) and 3 different inflection points (equal to $6$, $8$ and $11.5$) with $\gamma =1$ and $\sigma =0.04$;

• logis10A: 10 balanced groups, each one composed by 5 curves, with $\delta =6$, $\gamma =1$, 10 different asymptotes ($\mathbf c = (c_1,c_2,c_3,c_4,c_5,c_6,c_7, c_8,c_9,c_{10}) = (1,1.25,1.5,1.75,2,2.25,2.5, 2.75,3,3.25))$ and $\sigma =0.04$;

• logis10I: 10 balanced groups, each one composed by 5 curves, with $\alpha =1$, $\gamma =1$, 10 different inflection points ($\mathbf c = (c_1,c_2,c_3,c_4,c_5,c_6,c_7,c_8,c_9,c_{10}) = (4.5,5.5,7,8,9.5,10.5,12, 13,14.5,16))$ and $\sigma =0.04$.

Appendix D: Comparison of results

Comparison of estimates carried out by npmlreg and NLNPEM method are reported here, for some cases of interest mentioned in the paper (Tables 4, 5, 6).

• Linear case—Random-intercept case (lin2I)

• Linear case—Random-slope case (lin3S)

• Linear case—Random-intercept case (lin10I)

Table 4 Estimates carried out by npmlreg and NLNPEM method on lin2I dataset, where intercept is considered as random, with 2 balanced groups

Table 5 Estimates carried out by unpmlreg and NLNPEM method on lin3S dataset, where slope is considered as random, with 3 unbalanced groups

Table 6 Estimates carried out by npmlreg and NLNPEM method on lin10I dataset, where intercept is considered as random, with 10 balanced groups