A Hamilton-Jacobi-Bellman approach for termination of seizure-like bursting

Abstract

We use Hamilton-Jacobi-Bellman methods to find minimum-time and energy-optimal control strategies to terminate seizure-like bursting behavior in a conductance-based neural model. Averaging is used to eliminate fast variables from the model, and a target set is defined through bifurcation analysis of the slow variables of the model. This method is illustrated for a single neuron model and for a network model to illustrate its efficacy in terminating bursting once it begins. This work represents a numerical proof-of-concept that a new class of control strategies can be employed to mitigate bursting, and could ultimately be adapted to treat medically intractible epilepsy in patient-specific models.

1 Introduction

Epilepsy affects as many as 3 million people in the United States alone and annually costs approximately 12.5 billion dollars (CDC 1994; Kobau et al. 2007). For many of these people, seizures remain poorly controlled despite the use of anti-convulsive medication. This has prompted researchers to search for other therapies to help mitigate seizure frequency and duration. Among these treatments, Deep Brain Stimulation (DBS), a method by which a high frequency, pulsatile stimulus is periodically injected into the anterior nucleus of the thalamus, has been successful in clinical trials in suppressing seizure frequency and severity (Fisher et al. 2010; Lee et al. 2006; Lim et al. 2007). Furthermore, brief pulses of electrical stimulation have been shown to suppress seizure-like cortical afterdischarges, raising the possibility that DBS could terminate seizures during the ictal phase (Motamedi et al. 2002; Kinoshita et al. 2004). While these medical interventions are undeniably effective in some patients, these methods are ad hoc, are administered in an open loop manner, do not take into account the underlying seizure dynamics, and are most likely far from optimal. This has led researchers to develop alternative control strategies to better control epileptic seizures. For instance, Gluckman et al. (2001) used a proportional feedback algorithm to control epiliptic activity in brain slices with an electric field. Methods for control of periodicity in chaotically bursting neural systems have been proposed and have proven successful in brain slices (Schiff et al. 1994; Slutzky et al. 2003). Also, Ching et al. (2012) investigated the feasibility of grid-based brain electrical stimulation to suppress seizure propagation with a proportional control strategy.

The exact mechanism by which seizures are created and sustained is unknown, but many studies are beginning to investigate the role of the extracellular microenvironment in pathological neural bursting behavior (Kager et al. 2000; Bazhenov et al. 2004; Park and Durand 2006; Fröhlich et al. 2010). In this work, we present a control strategy for terminating bursting behavior for the model presented in Cressman et al. (2009), which includes a conductance-based neural model as well as local intra- and extracellular ion concentration dynamics. For certain parameters, this model displays periodic, recurrent seizure-like activity, and our control strategy seeks to find both time-optimal and energy-optimal DBS stimuli which will terminate seizure-like bursting behavior by driving each pathological neuron to a sufficiently refractory target set. We illustrate this strategy for a single neuron and for a homogeneous network of pathologically bursting neurons with the control objective of terminating a seizure once it begins. This novel approach to DBS has the potential to significantly improve the management of seizures in patients with medically intractable epilepsy.

2 Single neuron model

We consider a six-dimensional conductance-based model for neural activity with intracellular and extracellular ion concentration dynamics (Cressman et al. 2009). We choose this model because it exhibits periodic, seizure-like bursting behavior. The equations for this model are as follows:

$$\begin{array}{@{}rcl@{}} C\dot{V} &=&f_{V}(V,n,h,[\text{K}]_{\text{o}},[\text{Na}]_{\text{i}},[\text{Ca}]_{\text{i}})\\ &=&I_{\text{Na}}(V,h,[\text{Na}]_{\text{i}}) + I_{\mathrm{K}}(V,n,[\text{K}]_{\text{o}},[\text{Na}]_{\text{i}},[\text{Ca}]_{\text{i}}) + I_{\text{Cl}}(V) + u(t), \end{array} $$

(1)

$$ \dot{n} =f_{n}(V,n) =\phi \left[ \alpha_{n}(V)(1-n)-\beta_{n}(V)n \right], $$

(2)

$$ \dot{h} =f_{h}(V,h) =\phi \left[ \alpha_{h}(V)(1-h)-\beta_{h}(V)h \right], $$

(3)

$$\begin{array}{@{}rcl@{}} \dot{ [\text{Ca}]_{\mathrm{i}}} =f_{\text{Ca}}(V)= -0.002g_{\text{Ca}}(V-V_{\text{Ca}})/ \notag\\\left[ 1+\exp(-(V+25)/2.5) \right] -[\text{Ca}]_{\mathrm{i}}/80, \end{array} $$

(4)

$$\begin{array}{@{}rcl@{}} \dot{[\text{K}]_{\text{o}}} &=& f_{\mathrm{K}}(V,n,[\text{K}]_{\text{o}},[\text{Na}]_{\text{i}},[\text{Ca}]_{\text{i}}) \\ &=& -0.33I_{K}(V,n,[\text{K}]_{\text{o}},[\text{Na}]_{\text{i}},[\text{Ca}]_{\text{i}})\notag\\ &&- 2 \beta I_{\text{pump}}([\text{K}]_{\text{o}},[\text{Na}]_{\text{i}}) - I_{\text{glia}}([\text{K}]_{\text{o}}) - I_{\text{diff}}([\mathrm{K}]_{\mathrm{o}}) , \end{array} $$

(5)

$$\begin{array}{@{}rcl@{}} \dot{[\text{Na}]_{\mathrm{i}}} &=& f_{\text{Na}}(V,h,[\text{K}]_{\text{o}},[\text{Na}]_{\text{i}})\notag\\ &=& \frac{0.33}{\beta} I_{\text{Na}}(V,h,[\text{Na}]_{\text{i}}) - 3I_{\text{pump}}([\mathrm{K}]_{\mathrm{o}},[\text{Na}]_{\mathrm{i}}). \end{array} $$

(6)

Here, V represents the transmembrane voltage of the neuron, C represents the cell membrane capacitance, n and h represent gating variables, and [Ca]_i, [Na]_i, and [K]_o represent intracellular calcium, intracellular sodium, and extracellular potassium concentrations, respectively. We have augmented the voltage equation by additively including DBS input, u(t)=I(t)/C. For a full explanation of all model functions and parameters we refer the reader to Appendix ; code for this model is available from ModelDB.¹ For the parameters used in this paper, the model exhibits periodic seizure-like bursting behavior due to the slow dynamics of [K]_o and [Na]_i. Figure 1 displays the periodic behavior of the slow ion dynamics. The extracellular potassium rises slowly, increasing the excitability of the neuron until it reaches a level where the neuron is depolarized beyond its spiking threshold, leading to bursting behavior which rapidly increases the extracellular potassium concentration. Finally, the intracellular sodium concentrations grow until bursting behavior terminates, allowing [K]_o and [Na]_i to recover.

Fig. 1

Periodic dynamics of the full model. Neural dynamics fluctuate between bursting and quiescence for different concentrations of extracellular potassium and intracellular sodium. Note that the line showing the time evolution of the slow variable [K]_o appears thicker at times because of small, rapid fluctuations associated with the fast bursting dynamics

3 Bifurcation analysis to determine a target set

In order to terminate seizure-like bursting in this model, we must first identify the regimes for which Eqs. (1)–(6) exhibit bursting behavior. To produce bursting, a model must have a mechanism to generate spiking behavior and a separate mechanism with slow dynamics (Ermentrout and Terman 2010). For the model presented in Section 2, Eqs. (1)–(4) (fast variables) describe the neural spiking behavior, with variables that change on a much shorter time scale than [K]_o and [Na]_i (slow variables). Treating the slow variables as constants, we perform a bifurcation analysis on the full model to quantitatively analyze the bursting and quiescent regimes of the model. Using MATCONT (Dhooge et al. 2003), we are able to follow a curve of Saddle-Node Infinite Periodic orbit (SNIPER) bifurcations (Guckenheimer and Holmes 2010), a codimension one bifurcation which gives rise to a stable limit cycle. The left panel of Fig. 2 shows the curve of SNIPER bifurcations for the fast spiking variables from Eqs. (1)–(4). For ion concentrations to the right of the dotted curve, the neuron is in a bursting regime, $\mathcal {B}$, and for concentrations to the left of the curve, the neuron is in a quiescent regime, $\mathcal {Q}$. The right panel of Fig. 2 shows the trajectory of the slow variables from Fig. 1 with the line of SNIPER bifurcations shown for reference.

Fig. 2

The left panel shows the line of SNIPER bifurcations for the full model (1)–(6). The regions denoted by $\mathcal {Q}$ and $\mathcal {B}$ denote regions of quiescence and bursting, respectively. The right panel shows the trajectory of the potassium and sodium concentrations for the full model in the absence of control. Note that the trajectory follows a counterclockwise path

A naive approach to terminating bursting behavior in this model is to drive a neuron from $\mathcal {B}$ to anywhere in $\mathcal {Q}$. This objective can be accomplished, for instance, by simply giving an inhibitory stimulus until the neuron reaches the quiescent regime. In Fig. 3 we employ this strategy to drive a bursting neuron to a quiescent regime, but because the ion concentrations have not been significantly altered, the neuron begins bursting soon after the inhibitory control is removed.

Fig. 3

A naive approach to terminating bursting behavior. Top-left and bottom-left panels show the voltage trace and applied control, respectively. Soon after the onset of bursting, an inhibitory stimulus is applied, terminating bursting. The right panel shows the corresponding ion concentrations and the line of SNIPER bifurcations as a solid and dashed line, respectively. The starting point and ending point are shown as a solid and hollow dot, respectively. Soon after the control is turned off, the neuron resumes bursting

In order to further refine a target set, we define a refractory index for quiescent neurons,

$$ \mathcal{R}([\mathrm{K}]_{\mathrm{o}}(0) , [\text{Na}]_{\mathrm{i}}(0)) = \min \{ t | ([\mathrm{K}]_{\mathrm{o}}(t) , [\text{Na}]_{\mathrm{i}}(t)) \in \mathcal{B} \}, $$

(7)

which measures the time it takes for a quiescent neuron to begin bursting in the absence of a control input. We note that the refractory index is largely independent of the initial conditions of the fast variables, provided they do not cause the neuron to spike. Figure 4 shows the results of a numerical calculation of the refractory index as well as the line which separates the boundary of the target set $\mathcal {T} = \{ x | \mathcal {R}(x) \geq 15 , \quad d(x,\mathcal {B}) \geq 0.2 \}$, where x=[[K]_o,[Na]_i]^T, and $d(x,\mathcal {B})$ is the minimum distance from point x to the bursting regime using the 2-norm. We recognize that in an experimental setting, it may be difficult to measure the ion concentrations precisely, and include the second condition, $d(x,\mathcal {B}) \geq 0.2$ as a margin of safety to ensure that the neuron has reached $\mathcal {Q}$.

Fig. 4

Refractory index, $\mathcal {R}$, plotted as a function of [K]_o and [Na]_i. The boundary of the target set, $\mathcal {T} = \{ x | \mathcal {R}(x) \geq 15 , \quad d(x,\mathcal {B}) \geq 0.2 \}$ is plotted as a solid line. The boundary of SNIPER bifurcations is plotted as a dashed line for reference

We note that for this particular model, only the intracellular and sodium extracellular potassium dynamics fluctuate on a slow time scale to give a bifurcation boundary that exists in a two dimensional space. For a different model with extra slow variables, for instance, if the glial buffering capacity also varied on a slow time scale, the bifurcation boundary and subsequent target set will exist in a higher dimensional space.

4 Optimal control of a bursting neuron

We consider a single neuron with DBS input u(t)

$$ \dot{z} = F(z) + Bu, $$

(8)

where z=[V,n,h,[Ca]_i,[K]_o,[Na]_i]^T, B=[1,0,0,0,0,0]^T, and

$$ F(z) = \left[\begin{array}{lllll} f_{V}(V,n,h,[\mathrm{K}]_{\mathrm{o}},[\text{Na}]_{\mathrm{i}},[\text{Ca}]_{\mathrm{i}}) \\ f_{n}(V,n) \\ f_{h}(V,h) \\ f_{\text{Ca}}(V)\\ f_{\mathrm{K}}(V,n,[\mathrm{K}]_{\mathrm{o}},[\text{Na}]_{\mathrm{i}},[\text{Ca}]_{\mathrm{i}}) \\ f_{\text{Na}}(V,h,[\mathrm{K}]_{\mathrm{o}},[\text{Na}]_{\mathrm{i}}) \\ \end{array}\right]. $$

(9)

Implementing an optimal control strategy for the full model would be difficult given the fast-slow dynamics of the system, and would likely require a control feedback system with knowledge of each of the state variables to be implemented effectively. In order to simplify the control problem, we first reduce the system by eliminating the fast-time-scale dynamics. Notice that f _K and f _Na only depend on the fast variables in the terms I _K and I _Na. This allows us to reduce (8) by averaging out the fast dynamics, resulting in

$$ \dot{x} = G(x) + I(x,u), $$

(10)

where x=[[K]_o,[Na]_i]^T, $I(x,u) = [ -0.33\overline {I_{\mathrm {K}}}(x,u) , \frac {0.33}{\beta } \overline {I_{\text {Na}}}(x,u)]^{T}$, and

$$\begin{array}{@{}rcl@{}} G(x) = \left[\begin{array}{lllll} - 2 \beta I_{\text{pump}}([\mathrm{K}]_{\mathrm{o}},[\text{Na}]_{\mathrm{i}}) - I_{\text{glia}}([\mathrm{K}]_{\mathrm{o}}) - I_{\text{diff}}([\mathrm{K}]_{\mathrm{o}}) \\ - 3I_{\text{pump}}([\mathrm{K}]_{\mathrm{o}},[\text{Na}]_{\mathrm{i}}). \end{array}\right]. \end{array} $$

(11)

Here, $\overline {I_{\mathrm {K}}}$ and $\overline {I_{\text {Na}}}$ are determined by fixing the external potassium concentration, internal sodium concentration, and control input, then allowing the model cell to reach either a steady state or a periodic orbit, and finally time-averaging I _K and I _Na over one second. The terms $\overline {I_{\mathrm {K}}}(x,u)$ and $\overline {I_{\text {Na}}}(x,u)$ are calculated by interpolating the numerically time-averaged data.

4.1 Minimum time control

The following optimal control strategies will use a Hamilton-Jacobi-Bellman (HJB) approach (Kirk 1998; Danzl et al. 2010). If the control objective is to find a control law that will take the neuron to the target set, $\mathcal {T}$, in the minimum possible time, we can begin by defining the time, t _{m i n} ∈[0,∞), to be the minimum time it takes for some initial state, x(0), to reach the target set under the influence of a control signal u(t), i.e.

$$ t_{min}(x,u(t)) = \min \{ t_{1} | x(t_{1}) \in \mathcal{T} , \quad x(0) = x \}. $$

(12)

For a given initial state, x, the time-optimal stimulus will minimize the cost functional

$$ J_{t}(x,u(t)) = {\int}_{0}^{t_{min}} 1\;dt =t_{min}(x,u(t)). $$

(13)

We note that t _{m i n} is not known a priori, and must be found through calculation of the optimal stimulus and optimal state trajectories. We also consider bounds on the maximum input for practical hardware limitations and tissue sensitivity: u _min≤u≤u _max. For this problem, we take u _min=−2 and u _max=10, with a more restrictive limit on the magnitude of hyperpolarizing (negative) current so that the transmembrane voltage does not drop too far below its normal resting levels. We define the minimum-time-to-reach value function

$$\begin{array}{@{}rcl@{}} \mathcal{V}_{t}(x) = \underset{{ u_{\min} \leq u(t) \leq u_{\max}}}{\inf} J_{t}(x,u(t)) = \underset{{ u_{\min} \leq u(t) \leq u_{\max}}}{\inf} t_{min}(x,u(t)). \end{array} $$

(14)

Using the minimum-time-to-reach, Hamilton-Jacobi-Bellman framework, the value function $\mathcal {V}_{t}(x)$ is the solution of the following equation (Bardi and Capuzzo-Dolcetta 2010):

$$\begin{array}{@{}rcl@{}} 0 = \underset{{ u_{\min} \leq u(t) \leq u_{\max}}}{\min} \{ 1 &+&\ \nabla \mathcal{V}_{t}^{T}(x) (G(x) + I(x,u)) \} = 1\notag\\ &+& \nabla \mathcal{V}_{t}^{T}(x)G(x) + \underset{{ u_{\min} \leq u(t) \leq u_{\max}}}{\min} \{ \nabla \mathcal{V}_{t}^{T}(x)I(x,u) \}, \end{array} $$

(15)

with boundary condition $\mathcal {V}_{t}(x) = 0 \quad \forall x \in \mathcal {T}$, where $\nabla \mathcal {V}_{t}$ is the gradient of the value function with respect to the state x. To find the optimal control, u ^∗(t), we must first solve (15) for $\mathcal {V}_{t}(x)$, and then minimize the term which depends on the control, resulting in the control policy

$$ u^{*}(x(t)) = \text{arg}\min \left(\nabla \mathcal{V}_{t}^{T}(x)I(x,u) \right). $$

(16)

We will obtain a numerical solution for Eq. (15) using Ian Mitchell’s “Level Set Methods Toolbox ^″ (Mitchell 2007), a computational tool to solve time dependent PDE’s. While Eq. (15) is not time dependent, following the methodology of Osher (1993) and Mitchell (2007) we can convert it to a time dependent PDE. We first define a function

$$\begin{array}{@{}rcl@{}} K(x,\nabla\mathcal{V}_{t}(x)) = 1 + \nabla \mathcal{V}_{t}^{T}(x)G(x) + \underset{{ u_{\min} \leq u(t) \leq u_{\max}}}{\min} \{ \nabla \mathcal{V}_{t}^{T}(x)I(x,u) \}, \end{array} $$

(17)

and rewrite (15) as

$$\begin{array}{*{20}l} K(x,\nabla\mathcal{V}_{t}(x)) &= 0 \quad \text{on} \quad \mathcal{D} \backslash \partial \mathcal{T}, \\ \mathcal{V}_{t}(x) &= 0 \quad \text{on} \quad \partial \mathcal{T} , \end{array} $$

(18)

where $\partial \mathcal {T}$ is the boundary of the target set $\mathcal {T}$ and $\mathcal {D}$ is the spatial domain. Provided that the boundary conditions are not characteristic, i.e.

$$ {\sum}_{i = 1}^{d}q_{i} \frac{ \partial K(x,q)}{\partial q_{i}} \neq 0 \quad \text{on} \quad \partial \mathcal{T}, $$

(19)

we can define an auxiliary function φ(x,s) and change variables

$$ \mathcal{V}_{t}(x) \leftarrow s \quad \text{and} \quad \nabla \mathcal{V}_{t}(x) \leftarrow \frac{-\nabla \varphi(x,s)}{ \varphi_{s}(x,s)}, $$

(20)

where φ _s =∂ φ/∂ s. Algebraic manipulation yields

$$\begin{array}{@{}rcl@{}} 0 = \varphi_{s}(x,s) - \nabla \varphi(x,s)G(x) - \underset{{ u_{\min} \leq u(t) \leq u_{\max}}}{\min} \{ \nabla \varphi(x,s)I(x,u) \}, \end{array} $$

(21)

with initial conditions

$$\begin{array}{*{20}l} \varphi(x,0) &= 0 \quad \in \quad \partial \mathcal{T}, \\ \varphi(x,0) &<0 \quad \in \quad \mathcal{T} \backslash \partial \mathcal{T}, \\ \varphi(x,0) &> 0 \quad \in \quad \mathcal{D} \backslash \mathcal{T}. \end{array} $$

(22)

Typically, Eq. (22) is achieved by using a signed distance function for ϕ(x,0). Equation (21) can be solved with the Level Set Methods Toolbox, from which we can extract

$$ \mathcal{V}_{t}(x) = \{ t | \varphi(x,t) = 0 \}. $$

(23)

The Matlab scripts for these calculations can be found at http://www.me.ucsb.edu/~moehlis/pubs.html.

4.2 Optimal energy control

Suppose we still want to reach the target set $\mathcal {T}$, using a stimulus which consumes a minimum amount of energy. We can solve this problem by defining a new cost function

$$ J_{e}(x,u(t)) = {\int}_{0}^{t_{end}} u^{2} dt + \gamma q(x(t_{end})), $$

(24)

where t _{e n d} is the duration of the stimulus, ${\int }_{0}^{t_{end}} u^{2} dt$ represents the amount of power consumed by the stimulus, q(x(t _{e n d} )) is an end point cost function, and γ is a penalizing scaler which sets the relative importance of each term. As with the optimal time control problem, we set bounds on the maximum input for hardware limitations and tissue sensitivity: u _{m i n} ≤u≤u _{m a x} . To maintain consistency with the previous section, we take u _{m i n} =−2 and u _{m a x} =10. We define the minimum-energy value function

$$ \mathcal{V}_{e}(x,\tau) = \underset{{ u_{\min} \leq u(t) \leq u_{\max} \\ \forall t \in [\tau,t_{end}] }}{\inf} J(x,u(t)). $$

(25)

Notice that the minimum-energy value function, $\mathcal {V}_{e}$, is a function of the time and state, whereas in the minimum-time-to-reach scenario, the value function, $\mathcal {V}_{t}$, is only a function of the state. We can find the optimal stimulus for Eq. (24) by solving the HJB equation (Kirk 1998)

$$ 0 = \frac{\partial \mathcal{V}_{e}}{\partial t}(x,\tau) + \min_{u_{\min} \leq u(t) \leq u_{\max} } \mathcal{H}(x,\nabla \mathcal{V}_{e},u), $$

(26)

where

$$\begin{array}{@{}rcl@{}} \mathcal{H}(x,\nabla \mathcal{V}_{e},u) = u(t)^{2} + [\nabla \mathcal{V}_{e}(x(t),t)]^{T} (G(x(t)) + I(x(t),u(t))), \end{array} $$

(27)

and with endpoint boundary condition

$$ \mathcal{V}_{e}(x(t_{end}),t_{end}) = \gamma q(x(t_{end})). $$

(28)

Here $\nabla \mathcal {V}_{e}$ is the gradient of the value function with respect to state x. The resulting optimal control policy is

$$ u^{*}(x,t) = \text{arg}\min (u^{2} + \nabla \mathcal{V}^{T}_{e}(x,t) I(x,u) ). $$

(29)

To calculate the optimal control u ^∗(x,t) from Eq. (29), we first solve (26) for $\mathcal {V}_{e}(x,t)$ with endpoint boundary condition (28). We use a sigmoid $q(x(t_{end})) = 1/(1+\exp (-5(d(x,\mathcal {T})-1.2)))$ as the endpoint cost, where $d(x,\mathcal {T})$ is the minimum distance from x to the target set using the 2-norm. This endpoint cost is chosen as an appropriate penalty for failing to reach $\mathcal {T}$. Using Ian Mitchell’s “Level Set Methods Toolbox ^″ (Mitchell 2007), we solve (26) with γ=1000.

5 Results and discussion

We first solve for $\mathcal {V}_{t}$ using the minimum-time methodology presented in Section 4.1. Using this information, we determine an optimal control policy based on Eq. (16). The cost function $\mathcal {V}_{t}(x)$ and optimal control policy are shown in Fig. 5. The numerics suggest that for our particular choice of parameters in Eq. (15), $\text {argmin} (\nabla \mathcal {V}_{t}^{T}(x)I(x,u))$ is unique for all x, thus Eq. (16) would imply that the resulting minimum-time control policy is unique. Notice that the control policy is not strictly of the bang-bang type as is common for minimum time problems. This happens because, unlike problems for which the control input is simply added to the right-hand-side of the dynamic equations, the influence of the applied control is a function of the time-averaged neural dynamics.

Fig. 5

Left and right panels show $\mathcal {V}_{t}(x)$ and u ^∗(x), respectively, for the target set defined in Section 3. The boundary of the target set is shown as a solid line and the line of SNIPER bifurcations is shown as a dashed line for reference

In order to test the validity of the reduction, we apply the control policy shown in Fig. 5 to both the reduced and full dynamics, Eqs. (10) and (8) respectively. The external control is set to zero until the neuron exhibits seizure-like behavior (i.e. bursting), and to calculate the control we use an initial condition that is just to the right of the SNIPER bifurcation along the orbit shown in the right panel of Fig. 2. Figure 6 shows the result of this simulation. The left panel shows the trajectory for the reduced model and full model as thin, grey and black lines, respectively. The top-right panel gives the control input applied to the full and reduced model as a black and grey line, respectively. Note that for the reduced model, the control and associated state trajectory are time optimal. The bottom right panel gives the voltage trace for the neuron from the full model. The control policy applied to the reduced model gives t _{m i n} =1.61 while the same control policy applied to the full model gives t _{m i n} =1.65. We see good agreement between the solutions obtained from full and reduced models, and conclude that the reduction by elimination of fast variables is a useful approach to solving this problem. Note that even though it takes more than one second to reach the target set, bursting activity terminates as soon as the stimulus switches from positive to negative. When we apply this control policy to the full model, bursting activity terminates after 0.53 seconds while, without any control input, bursting activity lasts for 6.36 seconds. The control policy applied to the full model yields a 92 % decrease in the duration of bursting activity. Note that the positive stimulus induces bursting that is more rapid than when the control is not applied, but there are approximately half as many spikes as compared to when control is not applied. Also, it is possible to further improve the reduction in bursting time by reducing the restriction on the size of the external input, particularly, by increasing the value of u _{m a x} .

Fig. 6

The left panel shows the trajectory of the slow potassium and sodium variables under the influence of the time-optimal control policy from Fig. 5 for the reduced system (grey line) and the full system (black line). The top-right panel shows the resulting optimal control waveforms for the full and reduced systems shown as black and grey lines, respectively. We see good agreement between the two models. The bottom-right panel shows the transmembrane voltage as a function of time for the full system

Next, we compare the minimum-time stimulus to other similar, non-optimal stimuli, with results shown in Fig. 7. The total time to reach the target set for u _{o p t} ,u ₂,u ₃, and u ₄ is 1.651, 1.664, 1.653, and 1.799 units of time, respectively. The stimuli u ₂ and u ₃ use signals with similar amplitude to u _{o p t} , but change the time at which the transition from positive and negative control occurs. We find that these two non-optimal stimuli only marginally increase the time to reach the target set. The stimulus u ₄ varies from the u _{o p t} in the magnitude of the positive control used. We find that this has a relatively larger effect on the overall time required to reach the target set. It is worth noting that neural spiking ends when each stimulus switches from an excitatory (positive) to an inhibitory (negative) stimulus, but spiking will return if the inhibitory stimulus is not applied for a sufficient amount of time. We also note that each stimulus reaches the target set in a different location and, if desired, the cost function in the optimal control problem could be reformulated to balance the trade off between the time to reach the target set and the refractory index at the end time, $\mathcal {R}(x(t_{end}))$.

Fig. 7

A comparison of the minimum-time control, u _{o p t} to other similar, but non-optimal controls. The left panel shows the trajectory of each neuron under the influence of their respective controls in the ([K]_o,[Na]_i) plane. The target set and line of SNIPER bifurcations are shown as thick, solid and dashed lines, respectively. The total time to reach the target set for u _{o p t} ,u ₂,u ₃, and u ₄ is 1.651, 1.664, 1.653, and 1.799 units of time, respectively

For simulations of multiple neurons, we consider a network model consisting of two layers of one-dimensional networks comprised of 60 excitatory pyramidal cells (PC’s) and 60 inhibitory interneurons (IC’s), which is inspired by Ullah et al. (2009) and Gutkin et al. (2001). Both neuron types are modeled as conductance-based cells with ionic concentration gradients. Neurons within the same layer are aligned in a ring and coupled through spatially dependent synapses as well as lateral diffusion of potassium through extracellular space. For further details of the equations and parameters used in the network model, we refer the reader to Appendices A and B.

We apply control to the network model assuming that each neuron receives an identical control input. The population average potassium, $\overline {[\mathrm {K}]}_{\mathrm {o}}$, and sodium, $\overline {[\text {Na}]}_{\mathrm {i}}$, levels of the excitatory neurons in the network are monitored, and the control is applied based on the strategy from Fig. 5. Network results are shown in Fig. 8. The average value of the ion concentrations reaches the target set approximately 1.6 seconds after onset of bursting activity, and the bursting lasts approximately one half-second, whereas in the absence of control, network bursting lasts for approximately 6.4 seconds. Note that the results for the noisy network simulation are similar to the results for the single neuron from Fig. 6.

Fig. 8

Network simulation using the minimum-time control strategy from Fig. 5. The top left panels show the path of average ion concentrations for excitatory cells from the network. The boundary of the target set is shown as a bold line for reference. The top right panel shows the applied control, and the bottom panel shows the transmembrane voltage of a characteristic excitatory neuron within the network

For most of the duration of the minimum-time stimulus, the controller is operating at or close to the positive and negative limits of the applied stimulus. Thus, we expect that if we relax the minimum time constraint, to give solutions that reach the target set at times that are larger but still close to the minimum time, we could save a significant amount of energy. With this in mind, we use the minimum-energy methodology presented in Section 4.2 to solve for $\mathcal {V}_{e}(x,t)$, and calculate the optimal stimuli for the full model for different choices of t _{e n d} . The resulting energy-optimal stimuli and the associated trajectories in the ([K]_o,[Na]_i) plane are shown in the right and left panels of Fig. 9, respectively. For the energy-optimal stimuli in Fig. 9, the overall energy used, ${\int }_{0}^{t_{end}}u^{2}dt$, in order of decreasing values of t _{e n d} is 5.65, 6.29, 7.42, and 55.3 units. We find that by slightly relaxing the minimum time constraint, it is possible to find stimuli which use an order of magnitude less energy than the minimum-time stimulus, which may be attractive from a clinical perspective.

Fig. 9

Optimal energy control to the target set. The left panel gives the trajectories in the ([K]_o,[Na]_i) plane. The target set and line of SNIPER bifurcations are plotted as grey solid and dashed lines, respectively. The right panel shows the minimum-energy stimulus for various choices of t _{e n d}

As with all control methods, this methodology is not without limitations. Safety concerns such as Faradaic charge injection (Merill et al. 2005) become important when DBS is implemented in the long term. Also, positive charge injection during the application of the optimal control serves to temporarily increase the bursting activity of the neuron, which may be harmful. These and other biologically relevant safety issues must be carefully considered before experiments can be performed on real neurons, but because of the generality of the Hamilton-Jacobi-Bellman approach, they can be addressed through modification to parameters in the calculation of the optimal control. This implementation of the time-optimal control strategy would require a model of real epileptic neurons that is accurate enough for control purposes, as well as a way to estimate the intra- and extracellular ion concentrations in real time. These challenges are beginning to be addressed through Kalman filtering, and may be feasible in the future (Ullah and Schiff 2010; Schiff 2010).

While this particular analysis was applied to a single-neuron model of seizure-like bursting activity, this methodology can be adapted to include different considerations for different models. For instance, this particular control strategy was applied to a model of periodic recurrent seizure-like activity, which does not accurately reflect all types of seizures. For a different model which exhibits seizure-like activity spontaneously or as the result of an external input, this methodology could still be implemented by defining an appropriate, non-pathological target set without consideration of the refractory period, as the seizure-like activity does not occur periodically. Furthermore, the single-neuron mechanism of bursting activity in this model is caused by pathological fluctuations of cellular ion concentrations, but at a network level, seizures are thought to occur as the result of an imbalance of synaptic excitation and inhibition (Cossart et al. 2001; Wendling et al. 2002; Gnatkovsky et al. 2008; Avoli and de Curtis 2011). This framework can still be handled for an appropriate model of seizure-like activity with fast-slow dynamics by averaging the fast, spiking dynamics for any general control input and applying the HJB control framework to the remaining slow dynamics.

6 Conclusion

We have described a method for driving a periodically bursting neuron to a sufficiently refractory target set in minimum time using electrical stimuli. Also, the total energy consumption for the minimum-time stimulus has been compared to the energy-optimal stimuli obtained for stimuli of slightly larger duration than the minimum time. We find when an entire network exhibits pathological, seizure-like bursting, this control methodology can reduce the amount of time the network spends in the bursting state by an order of magnitude. While the problem formulation is relatively complex, the resulting control strategy is quite simple.

The specific model used in this study examines the dynamic relationship between cellular sodium, potassium and calcium concentrations and their effect on the qualitative behavior of inhibitory and excitatory neurons. In this study, we find that once bursting begins, an initial excitatory stimulus increases the firing rate of bursting neurons, quickly increasing the extracellular sodium and extracellular potassium levels. Once a sufficiently large concentration of intracellular sodium has been reached, an inhibitory stimulus suppresses neural firing allowing ion exchange pumps, glial cells, and diffusive mechanisms to remove excess potassium, ending the bursting activity. Interestingly, when the intracellular sodium concentration is larger, ion exchange pumps work faster to remove excess potassium, causing the refractory index of x(t _{e n d} ) for the minimum time to reach control strategy to be much larger than required. For a different model, such as a network model for seizure-like behavior with large scale network dynamics such as network excitation and inhibition, different mechanisms of seizure termination could be exploited to find minimum-time, or optimal-energy control inputs.

We emphasize that the specific control strategy employed in this paper is not meant to be a definitive treatment for epileptic seizures. The model we have used is not perfect, as each neuron in this model spends much time close to a bifurcation point, and cannot spike more than a few times before reaching bursting state, which does not reflect the physiological need to propagate information through neurological spikes. The preceding method is meant as a proof of concept that more sophisticated methods than proportional feedback control have the potential to be successfully implemented given an accurate model of epilepsy and a clear control objective. Further investigation of models and mechanisms of seizure initiation and termination are needed, but this method shows promise in improving existing DBS strategies for termination of medically intractable epileptic seizures.

Appendices

Appendix A: Network model of epilepsy

The excitatory and inhibitory model for network simulations is as follows:

$$C \dot{V}^{e/i} = I_{\text{Na}}^{e/i}+ I_{\mathrm{K}}^{e/i} + I_{\text{Cl}}^{e/i} + I_{\text{syn}}^{e/i} + \eta^{e/i}(t)+u(t), $$

(30)

$$ \dot{n}^{e/i} =\phi \left[ \alpha_{n}(1-n^{e/i})-\beta_{n}n^{e/i} \right], $$

(31)

$$ \dot{h}^{e/i} =\phi \left[ \alpha_{h}(1-h^{e/i})-\beta_{h}h^{e/i} \right], $$

(32)

$$\begin{array}{@{}rcl@{}}&\dot{ [\text{Ca}]_{\mathrm{i}}}^{e/i} = -0.002g_{\text{Ca}}(V^{e/i}-V_{\text{Ca}})/\notag\\ &\left[ 1+\exp(-(V^{e/i}+25)/2.5) \right], \end{array} $$

(33)

$$\begin{array}{@{}rcl@{}} \dot{[\mathrm{K}]_{\mathrm{o}}}^{e/i} = -0.33I_{K}^{e/i} - 2 \beta I_{\text{pump}}^{e/i} - I_{\text{glia}}^{e/i} - I_{\text{diff}}^{e/i} \end{array} $$

(34)

$$ + \frac{D}{\Delta x^{2}}([\text{K}]_{\text{o}}^{i/e} + [\text{K}]_{\text{o(+)}}^{e/i} + [\text{K}]_{\text{o(-)}}^{e/i} -3[\text{K}]_{\text{o}}^{e/i}), $$

(35)

$$ \dot{[\text{Na}]_{\text{i}}}^{e/i} = \frac{0.33}{\beta} I_{\text{Na}}^{e/i}- 3I_{\text{pump}}^{e/i}. $$

(36)

Here, the equations from Eqs. (1)–(6) have been modified to include synaptic currents I _syn, and Gaussian white noise. The equations and parameters that describe the synaptic current to each neuron can be found in Ullah et al. (2009). Neurons in the same layer are positioned in a ring. In the potassium dynamic equations, $[\mathrm {K}]_{\mathrm {o}}^{i/e}$ refers to the ion concentration around the nearest neighbor in the adjacent layer, and $[\mathrm {K}]_{\mathrm {o}(+)}^{e/i}$ and $[\mathrm {K}]_{\mathrm {o}(-)}^{e/i}$ refer to ion concentrations of adjacent neurons within the same layer. Parameters Δx=20.0μm and D=2.5×10⁻⁶cm²/s represent distances between cells and the diffusion coefficient for potassium in water, respectively. The i.i.d. noise associated with each neuron, $\eta ^{e/i} = \sqrt {2B} \mathcal {N}(0,1)$, is assumed to be zero-mean Gaussian white noise with variance 2B=0.01. We use the algorithm presented in Honeycutt (1992) to simulate the noisy system. Supporting ionic functions are given in Appendix B.

Appendix B: Supporting ionic functions

Supporting ionic functions are as follows; note that the e/i superscripts for the network model from Appendix A have been dropped for convenience of notation:

$$\begin{array}{*{20}l} I_{\mathrm{Na}} &=& -g_{\mathrm{Na}} [ m_{\infty} (V)]^{3} h \left( V - V_{\text{Na}} \right) - g_{\text{Na}}(V - V_{\text{Na}}), \\ I_{\mathrm{K}} &=& - \left( g_{\mathrm{K}} n^{4} + \frac{g_{\mathrm{AHP}} [\mathrm{Ca}]_{i}}{1 + [\mathrm{Ca}]_{\mathrm{i}} } \right) (V-V_{ \mathrm K }) - g_{\mathrm{K}}(V-V_{ \mathrm K }), \\ I_{\mathrm{Cl}} &=& -g_{L}(V-V_{L}),\\ I_{\mathrm{diff}} &=& \epsilon([\mathrm{K}]_{\mathrm{o}} - k_{o,\infty}),\\ I_{\mathrm{pump}} &=& \left( \frac{ \rho}{1 + \exp((25 - [\mathrm{Na}]_{\mathrm{i}} )/3) } \right) \left( \frac{1}{1 + \exp (5.5- [\mathrm{K}]_{\mathrm{o}})} \right) , \\ I_{\mathrm{glia}}^{e/i} &=& \frac{G_{\mathrm{glia}}}{1 + \exp ((18 - [\mathrm{K}]_{\mathrm{o}})/2.5)} , \\ V_{\mathrm{Na}} &=& 26.64 \log \left( \frac{[\mathrm{Na}]_{\mathrm{o}}}{[\mathrm{Na}]_{\mathrm{i}}} \right),\\ V_{\mathrm{K}} &=& 26.64 \log \left( \frac{[{\mathrm{K}}]_{\mathrm{o}}}{[{\mathrm{K}}]_{\mathrm{i}}} \right), [\mathrm{Na}]_{\mathrm{o}} &=& 144 + \beta (18.0 - [\mathrm{Na}]_{\mathrm{i}}) \\ , [{\mathrm{K}}]_{\mathrm{i}} &=& 140 + (18.0 - [\mathrm{Na}]_{\mathrm{i}}).\end{array} $$

For inhibitory neurons, g _AHP=0, otherwise g _AHP=0.01mS/m ². Supporting rate equations are:

$$\begin{array}{*{20}l} m_{\infty}(V) &=& \alpha_{m}(V)/(\alpha_{m}(V) + \beta_{m}(V)), \\ \alpha_{m}(V) &=& 0.1(V+30)/(1-\exp(-0.1(V + 30))), \\ \beta_{m}(V) &=& 4 \exp(-(V + 55)/18), \\ \alpha_{n}(V) &=& 0.01(V+34)/(1-\exp(-0.1(V + 34))), \\ \beta_{n}(V) &=& 0.125 \exp(-(V + 44)/80), \\ \alpha_{h}(V) &=& 0.07 \exp(-(V + 44)/20), \\ \beta_{h}(V) &=& 1/ (1+ \exp(-0.1(V + 4)))), \\ \end{array} $$

Other constants are as follows: C=1μF/cm²,g _Na=100mS/m ²,g _K=40mS/m ²,g _L =0.05mS/m ²,g _K=0.05mS/m ²,g _Na=0.0175mS/m ²,ϕ=3s ⁻¹,V _L =81.93mV,g _Ca=0.1mS/m ²,V _Ca=120mV,β=7,ρ=1mM/s,k _o,∞ =8mM,𝜖=4/3s ⁻¹,G _glia=66.6mM/s.