Introduction

Sleep is a highly regulated and dynamic physiologic process. Simplistically, the regulation of sleep-wakefulness involves reciprocal interactions between two opposing control systems: one that promotes wakefulness and one that promotes sleep [1]. Normal vigilance is generally separated into three specific states: awake (WAKE), non-rapid eye movement (NREM) and rapid eye movement (REM) sleep [2]. Most mammals cycle or transition between the different vigilance states at regular intervals throughout their resting phase and have been shown to respond to drug in similar way (see Ivarsson et al. [3] for a review). It has been shown repeatedly that sleep fragmentation (i.e. increased number of transitions between sleep and wake states) in subjects suffering from sleep disorders or as a direct cause of pharmaceutically active drugs, is strongly linked to increased daytime sleepiness [4]. Therefore, the characterization of the time course of the transitions between different vigilance states could be useful not only to identify mechanisms of action and screen novel compounds for possible (side) effects, but also to reverse or restore the physiological processes that are degraded by sleep problems.

The different vigilance states can be identified using changes in electroencephalography (EEG) and electromyography (EMG) activity: WAKE is characterized by desynchronized, low amplitude EEG activity and relatively high EMG activity; NREM displays synchronized, high amplitude EEG activity accompanied by low EMG activity; whereas REM shows desynchronized low amplitude EEG activity with low or absent EMG activity [5]. Note that the circadian sleep pattern shows frequent transitions between the vigilance states (these frequencies are higher in rats compared to human). Besides, the likelihood of next vigilance state is function of the current vigilance state leading to multiple correlated states. As a result, the analysis of data from such studies is complex. For instance, proportional odds models are not suitable since they do not account for the within-individual dependency in the data while any model aimed at inferring from these data should at least account for the correlations between the different vigilance states. Markov models have proved to be suitable for this type of highly correlated longitudinal data [6]. Hence, most of the recent papers have adopted a Markovian approach to address this problem [7, 8]. More elaborated models have been developed by Karlsson et al. [9] and Kjellsson et al. [10, 11] who proposed Markov chain models using a mixed effects approach by modeling the transition probabilities as binary logistic functions of nighttime and stage time (duration of last contiguous time spent in a stage). More recently, Bizzotto et al. [12, 13] has extended this approach by modeling the transition probabilities using multinomial logistic functions.

The present manuscript differs in several respects from the above papers. Indeed, all previous works were performed in a clinical drug development setting while in this paper we describe the development of a continuous time Markov model to assess sleep fragmentation in a pre-clinical setting. On the other hand, the transition probabilities of the Markov model were directly derived from the so-called Kolmogorov (forward–backward) equations that describe their time-evolutions [6]. Moreover, the sleep states were modeled as hidden states of the Markov model. We applied this approach to compare the effects of methylphenidate, a powerful stimulant which can significantly disturb sleep, and of a new chemical entity (NCE) that potentially could affect sleep fragmentation. To the best of our knowledge, this is the first manuscript using a hidden Markov model to model sleep fragmentation and also the first to implement such a model in NONMEM.

Materials and methods

Study design

Methylphenidate (Ritalin®) is a psychomotor stimulant that is commonly used in the treatment of Attention deficit/hyperactivity disorder (ADHD). The NCE is a lead compound for an internal project that potentially could affect sleep fragmentation. The effects of methylphenidate and the NCE on sleep were determined on two cohorts of rats (n = 6–8 per group) in a placebo controlled cross-over design. Rats were orally dosed with compound or vehicle at light onset in a Latin square design and receiving a single dose of 3, 10 or 30 mg of methylphenidate or 2, 6.7, 20, 26.8 or 40 mg of the NCE. The doses of methylphenidate were based on previously published data showing effects on sleep and locomotor activity [14, 15, 16] and the doses of the NCE were chosen to cover the range from approximately 0.3 to 10 times the 12 h average free plasma multiples of the primary pharmacology (based on internal in vitro assays). Recording of EEG and EMG signals began immediately after dosing and continued for at least 12 h. The sleep stage analysis was performed with an automatic sleep stage discriminator based on similar logic described previously [5] to allocate every 12 s epoch to one of the three vigilance states, WAKE, NREM (non-REM) and REM. The scores were subsequently aggregated in 5 min epoch and the residence times in each of the three vigilance states were reported.

Animals

Male Sprague–Dawley rats (Charles River, Margate, UK, approximately 300 g at time of surgery) were singly housed under a 12 h light/dark cycle (lights on at 11:00 am) with food and water ad libitum. As described previously [5, 17], animals were implanted with radiotelemetric transmitters (Data Sciences International, St. Paul, MN, USA) intraperitoneally under isoflurane anaesthesia for the recording of the EEG and EMG and were allowed to recover from the implantation of the device for at least 2 weeks, and the experiment was started once the animals were certified fit to continue by a veterinary surgeon. All experiments were performed in accordance with the ethical guidelines for animal research and fully complied with Home Office legislation.

Hidden Markov model

In this analysis only two vigilances states were considered: the WAKE and the SLEEP state, with the latter obtained by merging the vigilance sleep stages REM and NREM sleep. In every 5 min epoch, the time spent in each of the vigilance states (residence time) was reported. A model that aims at inferring from these data should not discard the dependency between these observations. While the analysis of such dense and continuous data (in the rest of this paper by continuous data, we mean the length of sojourn in a vigilance state), when taking into account the dependency between observations, is theoretically feasible (using for example autoregressive models such as in time series analysis) it can be computationally prohibitive. Therefore, we decided to binarize the data (the length of sojourn is binarized) by defining a cut-off point. The cut-off point was arbitrary chosen to be 2.5 min. When the residence time in the WAKE state was less or equal to 2.5 min, the animal was assumed to be in the SLEEP state and the observation was reset to 0. On the other hand when the residence time in the WAKE state was greater than 2.5 min, the animal was assumed to be in the WAKE state and the observation was reset to 1. Under the additional assumption that at any given time the next vigilance state is fully and only dependent on the present vigilance state, sleep fragmentation would possess the Markov property (the Markov property is characterized by two main assumptions: the present depends on the past and given the present, the future is independent from the past). Thus a Markov model could be used to model such data.

However, due to the arbitrarily selected cut-off point, the binarization could be biased leading to misclassification of the observations. Determining the cut-off with the least misclassification errors could be a way to overcome this problem but our attempts to estimate this point were unsuccessful. Instead, we used a hidden Markov model to account for misclassification errors.

A hidden Markov model is a Markov model whose states are not observed (hidden). In a regular Markov model, there is a one-to-one relationship between the observations and the states. In our setting, for instance, the state SLEEP corresponds to observing 0. In a hidden Markov process, the states are not anymore identifiable from the observations. Hence, the observation might be set to 0, whereas the animal is actually awake. Therefore the sleep status was modelled as a continuous time two-state hidden Markov model (Fig. 1) which accounts for the misclassifications induced by the binarization.

Fig. 1
figure 1

Two-state hidden Markov process

Full size image

All inferences are conditional on the observation times. Let P01 and P10 (Fig. 1) denote respectively the transition probabilities from the SLEEP state to the WAKE state and vice versa. P00 and P11 (Fig. 1) are the probabilities of not moving from the present state; SLEEP and WAKE respectively. Identification of the hidden Markov model requires the present state, P01 and P10 and also the probability of misclassification to be known. The probability of observing 1 when the true state is SLEEP (i.e. misclassifying a 0 as a 1) and the probability of observing 1 when the true state is WAKE (i.e. correct classification in the WAKE state) are denoted by P0 and P1 respectively. Ideally, the probability of misclassification should be low, resulting in a P0 close to 0 and P1 close to 1. Hence P0 is conceptually analog to a type I error, whereas P1 to the power of wrong classification of the observations. This also provides an indication on whether the arbitrary selected cut-off point was appropriate or not.

A Markov model can, by definition, be parameterized by its intensities of transition. This can be achieved by solving the system of Kolmogorov’s differential equations [6]. Let u and v denote the intensities of transitioning from WAKE to SLEEP and from SLEEP to WAKE respectively. For a two-state Markov process, the transition probabilities over a time interval t are shown in Eqs. 1 and 2 for P01 and P10 respectively, in which u is the intensity of acquisition of sleep while v is the intensity of clearance of sleep.

$$ P_{01} \left( t \right) = \frac{v}{u + v}\left( {1 - e^{{ - \left( {u + v} \right)t}} } \right) $$
(1)
$$ P_{10} \left( t \right) = \frac{u}{u + v}\left( {1 - e^{{ - \left( {u + v} \right)t}} } \right) $$
(2)

As mentioned above t is the length of a time interval. Since each individual is observed at successive time points, the contribution of each subject to the likelihood is function of these transitions probabilities where t is the time between two consecutive observations.

Pharmacokinetic model

The pharmacokinetics of methylphenidate and the NCE were described with compartmental models with first order absorption and elimination. The PK parameters were estimated on internal Pfizer data from satellite PK experiments in rats, using a population PK modelling approach (data not shown). The estimated PK parameters for a typical rat were subsequently used to simulate the PK in the PK-PD model. The typical estimated values for the maximum plasma concentration and the time at which the peak concentration is reach are provided in Table 1 for both compounds.

Table 1 PK parameter estimates
Full size table

Placebo and drug effect models

The intensities are the rates of transitioning from one state to another. These rates are the inverse of the residence times, which are known to be exponentially distributed [6]. It was assumed that drugs can change these intensities. Several models (linear, power and Emax) for the effects of drugs on these intensities were explored. For sake of conciseness, only the results of the Emax model are reported in this manuscript as that was the best model according to the Bayesian Information Criterion (BIC). However the classical Emax model provided non-significant estimates of the EC50. Hence we decided to use the (generally) more flexible model, a version of the 4-parameter logistic model [18] to estimate the effect of the drug on both intensities (note that the 4-parameter logistic model is a specific parameterization of the sigmoidal model in which a baseline and a slope factor can also be specified: here these two parameters are fixed to zero and one respectively and only the remaining two parameters are estimated). The possible delay in the onset of the drug effect relatively to drug exposure was tested on both intensities via an effect-compartment. However, in the final model (selection based on the BIC and the parameter precisions); this delay was included for the acquisition of sleep only. In this placebo controlled experiment, it was assumed that the placebo treatment could also change the transition intensities.

More specifically, the intensities u and v are given by Eqs. 3 and 4, in which u 0 and v 0 are the baseline intensities, P max the maximum placebo effect, which will wash-out with an equilibrium half-life Teq,plac. The latter was assumed to be the same for both intensities. The drug effects on the intensities u and v (Drg u and Drg v ) are described by Eqs. 5 and 6 respectively, in which C p is the drug concentration in plasma and C e is the concentration in the hypothetical effect compartment, which is derived from Eq. 7 describing the delay or hysteresis between the plasma and effect concentration using the equilibrium constant k e0 and an equilibrium half-life T eq,drug of ln(2)/k e0. E max is the maximum drug effect while EC 50 is the plasma or effect compartment concentration corresponding to half of the maximum drug effect.

$$ u = \exp \left( {u_{0} + P_{{\max_{u} }} e^{{ - (\ln (2)/T_{eq,plac} )t}} + Drg_{u} } \right) $$
(3)
$$ v = \exp \left( {v_{0} + P_{{\max_{v} }} e^{{ - (\ln (2)/T_{eq,plac} )t}} + Drg_{v} } \right) $$
(4)
$$ {\text{Drg}}_{u} = E_{{\max_{u} }} - \frac{{E_{{\max_{u} }} }}{{1 + \frac{{C_{e} }}{{EC_{50u} }}\exp \left( {C_{e} - EC_{50u} } \right)}} $$
(5)
$$ Drg_{v} = E_{{\max_{v} }} - \frac{{E_{{\max_{v} }} }}{{1 + \frac{{C_{p} }}{{EC_{50v} }}\exp \left( {C_{p} - EC_{50v} } \right)}} $$
(6)
$$ \frac{{dC_{e} }}{dt} = k_{e0} \left( {C_{p} - C_{e} } \right) $$
(7)

Note that in Eqs. 3 and 4, the time t is TAD.

It was assumed that the misclassification errors were highest around the cut-off point and decrease exponentially as a function proportional to the distance from that point (2.5 min). More specifically:

$$ P_{0} = \frac{1}{{1 + \exp \left( { - \delta .\theta_{0} } \right)}} $$
(8)
$$ P_{1} = \frac{1}{{1 + \exp \left( { - \delta .\theta_{1} } \right)}} $$
(9)
$$ \delta = \exp \left( { - \left| {l - 2.5} \right|} \right)\quad l\;{\text{is}}\;{\text{length}}\;{\text{of}}\;{\text{time}}\;{\text{in}}\;{\text{WAKE}} $$
(10)

The parameters θ 0 and θ 1 are to be estimated.

We set an additional constraint on both P 0 and P 1; if l is greater than 4 P 0 is set to 0 and when l is less than 1 P 1 is set to 0. The meaning of this latter constraint is that the probability of misclassification is zero when and only when the length of consecutive time in WAKE is less than 1 min or greater than 4 min during the 5 min epoch.

Software and model evaluation

The model parameters were estimated by maximizing the likelihood of the model using the Laplacian method in the nonlinear mixed effect modelling package NONMEM (version 7.1.2; Icon Development Solutions, Ellicott City, Maryland, USA). At each time point the residual on frequency of the WAKE state (including all subjects and all scenarios) divided by their variances were calculated to provide for both compounds the time course of the weighted residuals in Fig. 3. The predictive performance of the model was assessed by simulations, using S-Plus® for Windows (version 8.0 Professional, release 1, Insightful Corp., Seattle, USA). This was achieved by simulating 100 replicate datasets using the final model (parameters and uncertainty). The hidden Markov property of the model was then applied to each sample of parameters. This means that for a given observation and a given sleep state at the present time, the next sleep status was predicted by sampling from a Bernoulli distribution with the appropriate transition probability. In this approach, all rats were assumed to be awake at the start of the experiment. Subsequently, the observation corresponding to that time point was predicted by sampling from a Bernoulli distribution with the appropriate misclassification probability. These simulations were used to reconstruct for each dose, the observed frequency over time of animals in the WAKE state (Fig. 4). For both compounds, the predicted transition probability from awake to sleep over time for a typical rat is depicted in Fig. 5. To evaluate the overall effect of methylphenidate and the NCE respectively on sleep fragmentation, the difference between log ratio of the intensity of acquisition of sleep over clearance of sleep under active treatment and the same quantity when only placebo is given was calculated. When this difference is zero the drug has no effect on sleep. A positive difference shows induction of sleep, whereas wakefulness is promoted when the difference is negative (Fig. 6). Further evaluations of the model were performed using the theory of receiver operating characteristic (ROC) curves [19]. Basically, the predictions are compared to the observations for each individual at each time point. The ratio of true 1’s (true positive TP: correctly predicted 1 divided by the total number of 1’s observed) as well as the ratio of false 1’s (false positive FP: number of observed 0 predicted as 1 divided by the total number of 0’s observed) are calculated for each time point. However, as the model is a probabilistic model, we actually need to sample such a contingency table to proceed. This was done as follows: a threshold is set between 0 and 1. At each time point, for each individual, the predicted probability of the observation being 1 is compared to the threshold. When the probability is greater than the threshold then the predicted observation is set to 1 whereas if it is less than the threshold the prediction is set to 0. The process is repeated for each threshold (note that this threshold has no relationship with the cut-off point which was used for the binarization) of the 100 equally spaced points of [0, 1]. The ROC curve is hence obtained (Fig. 7) and its AUC is an indicator of the predictive performance of the model. We want this AUC as close as possible to 1. It is generally accepted that an AUC over 0.8 corresponds to a good predictive model.

Results

The observed time course of transitions between SLEEP and WAKE for the two cohorts of rats receiving methylphenidate and the NCE is shown in Fig. 2. Both compounds show different changes in the sleep pattern compared to placebo, since methylphenidate increases the residence time, whereas the NCE increases the frequency of the transitions.

Fig. 2
figure 2

Time course of transitions from WAKE to SLEEP and vice versa for a placebo and b methylphenidate in the methylphenidate cohort and c the placebo versus d NCE in the NCE cohort. At the start of the experiment all rats are in the WAKE state. A value of 0 (horizontal line) means no transition, whereas a positive values indicates a change from WAKE to SLEEP while the reverse is indicated by a negative value

Full size image

All model parameters (Table 2) could be estimated precisely, since their relative standard errors were below 50% of the parameter estimates except the Emax for the effect of methylphenidate on the transition from WAKE to SLEEP, which appears not to be different from 0. The expected indicator of type I error was less than 0.05 while the expected indicator of power was greater than 0.92. This suggests that the binarization method was appropriate since it has induced only a low probability of misclassification.

Table 2 Model parameter estimates (mean and 95% confidence limits reported by NONMEM) of the hidden Markov model
Full size table

The weighted residuals in Fig. 3 are approximately normally distributed and were generally within the 95% confidence interval of the standard normal distribution. This suggests that the model was not misspecified. The observed and simulated number of animals in the WAKE state (Panel A vs. panel C and B vs. panel D in Fig. 4 for methylphenidate and the NCE respectively) closely resemble, indicating an adequate description of the sleep fragmentation by the two-state continuous hidden Markov model. Figure 5 shows different patterns for the transition probability of the NCE compared to methylphenidate. Both compounds tend to promote wakefulness when compare to placebo, as shown in Figs. 4 and 6.

Fig. 3
figure 3

Weighted residuals on frequency of the WAKE state for the fit of the hidden Markov model to the data of a the methylphenidate cohort and b the NCE cohort

Full size image
Fig. 4
figure 4

The observed and simulated frequency over time of animals in the WAKE state for the methylphenidate (a observations; c predictions) and the NCE cohort (b observations; d predictions). The frequency plots for each treatment group are stacked

Full size image
Fig. 5
figure 5

The transition probability from awake to sleep over time for a typical rat in a the methylphenidate cohort and b the NCE cohort

Full size image
Fig. 6
figure 6

The placebo-corrected log ratio of the intensity of acquisition of sleep over clearance of sleep: a methylphenidate and b the NCE

Full size image

Discussion

For both compounds, the estimated E max value was higher for the intensity v than for u (Table 2), suggesting a higher drug effect on the clearance of sleep compared to the acquisition of sleep. Methylphenidate appears to inhibit the transitions from one sleep status to another as shown by the negative E max values, whereas the NCE stimulates these transitions. As can be observed in Fig. 2b, inhibition of both transitions by methylphenidate results in longer residence times. In contrast, stimulating the intensities of the transitions by the NCE increases the number of transitions between SLEEP and WAKE (Fig. 2d).

Figure 4a, c for methylphenidate and the NCE respectively show that the onset of acquisition of sleep (i.e. transition from WAKE to SLEEP) is delayed whether the animal received a placebo or an active treatment. Figure 5a shows that methylphenidate delays the onset of sleep for low doses. There seems to be a dose dependence decreasing this delay with increasing dose so that the placebo onset of sleep is almost identical to the high dose onset. Figure 5b shows comparable onsets of sleep for the placebo and the active doses. The NCE acts as a stimulator of both intensities, which results in higher transition probabilities with respect to placebo. This pattern is clearly different from the effects of methylphenidate showing that the NCE and methylphenidate have clearly different mechanism of action on sleep fragmentation.

Figure 6 shows the overall effect of methylphenidate and the NCE respectively on sleep fragmentation. These were negative for both compounds; suggesting that both methylphenidate and the NCE tend to promote wakefulness when administrated to rats. However, the maximum effect of methylphenidate was −2.6 while that of the NCE was fivefold less (−0.55). This suggests that methylphenidate was much more potent than the NCE in inducing sleep fragmentation in rats.

The predictive performance of the model was evaluated using the ROC curves in Fig. 7. This shows a higher predictive performance for methylphenidate than for the NCE though the model performs well for both compounds since the AUC of these curves were estimated to be 0.86 and 0.93 for the NCE and methylphenidate respectively.

Fig. 7
figure 7

ROC curves for methylphenidate and the NCE

Full size image

It has been shown repeatedly that fragmented sleep is less restorative than consolidated sleep and causes reduced daytime functioning [4]. However, it is still unclear how to best quantify the fragmentation of sleep and also which specific changes in the sleep pattern are directly linked to impairments in daytime function. Currently, there are unconvincing correlations between sleepiness and the standard measures of arousals. Therefore, there is a clear need to identify more sensitive approaches to measure sleep fragmentation and the approach described here could address some of these questions, but more work and analysis is needed, especially on clinical data, before any clear conclusions on the applicability can be drawn.

In conclusion, this manuscript describes the development of a hidden Markov model to assess drug-induced sleep disturbance across multiple drug classes on the basis of EEG recordings in rats. Markov models can accommodate highly correlated observations. Unlike previous papers in this field [9–13], the transition probabilities were derived from the Kolmogorov (forward–backward) equations. This methodology can be used to explicitly model the ultradian regulation (REM vs. non-REM). Here, the REM and NREM were merged into one vigilance stage. Our methodology is applicable when all three vigilance states are considered as long as the observations are discrete (using for instance the 12 s epoch instead of the aggregated data in 5 min). However there may be computational issues with NONMEM because of the size of the data and the rapidly increasing number of parameters to estimate. Models such as those developed in [9–13] might then be more suitable. By binarizing the original observations, the approach adopted in this paper provides a far less computationally prohibitive method than a method based on autoregressive models for instance. It should be acknowledged, however, that binarization of data does, in principle, induce misclassification errors. However, the developed hidden Markov model offers an elegant way of accounting for these errors which were shown to be acceptable in this analysis. Indeed, it was shown that the complex sleep pattern in rats was well described by a two-state continuous time hidden Markov model, which was successfully applied to studies designed to assess and compare the effects of methylphenidate and an NCE on sleep disturbance in telemetered rats. This model can be used to quantify differences in sleep fragmentation and provides insight into the nature of the underlying mechanism of action of drug inducing sleep fragmentation. As a result, this approach can be applied to screen NCEs early in development for their possible effects on sleep fragmentation compared to an active control, such as methylphenidate. For the NCE evaluated in this paper it was found that it induces sleep fragmentation in a dose-dependent manner, but to a much less extent (i.e. five times less potent) and with a different mechanism of action compared to methylphenidate.