Open Access
Part of the Lecture Notes in Computational Science and Engineering book series (LNCSE, volume 110)


This chapters targets generalizations of the simple decay model \(u^{\prime}=-au\) and also to look at additional numerical solution methods. We consider first variable coefficients, \(u^{\prime}=a(t)u+b(t)\), and later a completely general scalar ODE \(u^{\prime}=f(u,t)\) and its generalization to a system of such general ODEs. Among numerical methods, we treat implicit multi-step methods, and several families of explicit methods: Leapfrog schemes, Runge–Kutta methods, and Adams–Bashforth formulas.

It is time to consider generalizations of the simple decay model \(u^{\prime}=-au\) and also to look at additional numerical solution methods. We consider first variable coefficients, \(u^{\prime}=a(t)u+b(t)\), and later a completely general scalar ODE \(u^{\prime}=f(u,t)\) and its generalization to a system of such general ODEs. Among numerical methods, we treat implicit multi-step methods, and several families of explicit methods: Leapfrog schemes, Runge–Kutta methods, and Adams–Bashforth formulas.

3.1 Model Extensions

This section looks at the generalizations to \(u^{\prime}=-a(t)u\) and \(u^{\prime}=-a(t)u+b(t)\). We sketch the corresponding implementations of the θ-rule for such variable-coefficient ODEs. Verification can no longer make use of an exact solution of the numerical problem so we make use of manufactured solutions, for deriving an exact solution of the ODE problem, and then we can compute empirical convergence rates for the method and see if these coincide with the expected rates from theory. Finally, we see how our numerical methods can be applied to systems of ODEs.

The example programs associated with this chapter are found in the directory src/genz .

3.1.1 Generalization: Including a Variable Coefficient

In the ODE for decay, \(u^{\prime}=-au\), we now consider the case where a depends on time:
$$u^{\prime}(t)=-a(t)u(t),\quad t\in(0,T],\quad u(0)=I\thinspace.$$
A Forward Euler scheme consists of evaluating (Ch3.E1) at \(t=t_{n}\) and approximating the derivative with a forward difference \([D^{+}_{t}u]^{n}\):
$$\frac{u^{n+1}-u^{n}}{\Delta t}=-a(t_{n})u^{n}\thinspace.$$
The Backward Euler scheme becomes
$$\frac{u^{n}-u^{n-1}}{\Delta t}=-a(t_{n})u^{n}\thinspace.$$
The Crank–Nicolson method builds on sampling the ODE at \(t_{n+\frac{1}{2}}\). We can evaluate a at \(t_{n+\frac{1}{2}}\) and use an average for u at times t n and \(t_{n+1}\):
$$\frac{u^{n+1}-u^{n}}{\Delta t}=-a(t_{n+\frac{1}{2}})\frac{1}{2}(u^{n}+u^{n+1})\thinspace.$$
Alternatively, we can use an average for the product au:
$$\frac{u^{n+1}-u^{n}}{\Delta t}=-\frac{1}{2}(a(t_{n})u^{n}+a(t_{n+1})u^{n+1})\thinspace.$$
The θ-rule unifies the three mentioned schemes. One version is to have a evaluated at the weighted time point \((1-\theta)t_{n}+\theta t_{n+1}\),
$$\frac{u^{n+1}-u^{n}}{\Delta t}=-a((1-\theta)t_{n}+\theta t_{n+1})((1-\theta)u^{n}+\theta u^{n+1})\thinspace.$$
Another possibility is to apply a weighted average for the product au,
$$\frac{u^{n+1}-u^{n}}{\Delta t}=-(1-\theta)a(t_{n})u^{n}-\theta a(t_{n+1})u^{n+1}\thinspace.$$
With the finite difference operator notation the Forward Euler and Backward Euler schemes can be summarized as
$$[D^{+}_{t}u =-au]^{n},$$
$$[D^{-}_{t}u =-au]^{n}\thinspace.$$
The Crank–Nicolson and θ schemes depend on whether we evaluate a at the sample point for the ODE or if we use an average. The various versions are written as
$$[D_{t}u =-a\overline{u}^{t}]^{n+\frac{1}{2}},$$
$$[D_{t}u =-\overline{au}^{t}]^{n+\frac{1}{2}},$$
$$[D_{t}u =-a\overline{u}^{t,\theta}]^{n+\theta},$$
$$[D_{t}u =-\overline{au}^{t,\theta}]^{n+\theta}\thinspace.$$

3.1.2 Generalization: Including a Source Term

A further extension of the model ODE is to include a source term \(b(t)\):
$$u^{\prime}(t)=-a(t)u(t)+b(t),\quad t\in(0,T],\quad u(0)=I\thinspace.$$
The time point where we sample the ODE determines where \(b(t)\) is evaluated. For the Crank–Nicolson scheme and the θ-rule we have a choice of whether to evaluate \(a(t)\) and \(b(t)\) at the correct point or use an average. The chosen strategy becomes particularly clear if we write up the schemes in the operator notation:
$$[D^{+}_{t}u =-au+b]^{n},$$
$$[D^{-}_{t}u =-au+b]^{n},$$
$$[D_{t}u =-a\overline{u}^{t}+b]^{n+\frac{1}{2}},$$
$$[D_{t}u =\overline{-au+b}^{t}]^{n+\frac{1}{2}},$$
$$[D_{t}u =-a\overline{u}^{t,\theta}+b]^{n+\theta},$$
$$[D_{t}u =\overline{-au+b}^{t,\theta}]^{n+\theta}\thinspace.$$

3.1.3 Implementation of the Generalized Model Problem

Deriving the θ-rule formula

Writing out the θ-rule in (Ch3.E20), using ( 1.44) and ( 1.45), we get
$$\frac{u^{n+1}-u^{n}}{\Delta t}=\theta(-a^{n+1}u^{n+1}+b^{n+1}))+(1-\theta)(-a^{n}u^{n}+b^{n})),$$
where a n means evaluating a at \(t=t_{n}\) and similar for \(a^{n+1}\), b n , and \(b^{n+1}\). We solve for \(u^{n+1}\):
$$u^{n+1}=((1-\Delta t(1-\theta)a^{n})u^{n}+\Delta t(\theta b^{n+1}+(1-\theta)b^{n}))(1+\Delta t\theta a^{n+1})^{-1}\thinspace.$$

Python code

Here is a suitable implementation of (Ch3.E21) where \(a(t)\) and \(b(t)\) are given as Python functions:

This function is found in the file (vc stands for ‘‘variable coefficients’’).

Coding of variable coefficients

The solver function shown above demands the arguments a and b to be Python functions of time t, say

Here, a(t) has three parameters a0, tp, and k, which must be global variables.

A better implementation, which avoids global variables, is to represent a by a class where the parameters are attributes and where a special method __call__ evaluates \(a(t)\):

For quick tests it is cumbersome to write a complete function or a class. The lambda function construction in Python is then convenient. For example,

is equivalent to the def a(t) definition above. In general,

is equivalent to

One can use lambda functions directly in calls. Say we want to solve \(u^{\prime}=-u+1\), \(u(0)=2\):

Whether to use a plain function, a class, or a lambda function depends on the programmer’s taste. Lazy programmers prefer the lambda construct, while very safe programmers go for the class solution.

3.1.4 Verifying a Constant Solution

An extremely useful partial verification method is to construct a test problem with a very simple solution, usually \(u=\text{const}\). Especially the initial debugging of a program code can benefit greatly from such tests, because 1) all relevant numerical methods will exactly reproduce a constant solution, 2) many of the intermediate calculations are easy to control by hand for a constant u, and 3) even a constant u can uncover many bugs in an implementation.

The only constant solution for the problem \(u^{\prime}=-au\) is u = 0, but too many bugs can escape from that trivial solution. It is much better to search for a problem where \(u=C=\text{const}\neq 0\). Then \(u^{\prime}=-a(t)u+b(t)\) is more appropriate: with \(u=C\) we can choose any \(a(t)\) and set \(b=a(t)C\) and \(I=C\). An appropriate test function is

An interesting question is what type of bugs that will make the computed u n deviate from the exact solution C. Fortunately, the updating formula and the initial condition must be absolutely correct for the test to pass! Any attempt to make a wrong indexing in terms like a(t[n]) or any attempt to introduce an erroneous factor in the formula creates a solution that is different from C.

3.1.5 Verification via Manufactured Solutions

Following the idea of the previous section, we can choose any formula as the exact solution, insert the formula in the ODE problem and fit the data \(a(t)\), \(b(t)\), and I to make the chosen formula fulfill the equation. This powerful technique for generating exact solutions is very useful for verification purposes and known as the method of manufactured solutions, often abbreviated MMS.

One common choice of solution is a linear function in the independent variable(s). The rationale behind such a simple variation is that almost any relevant numerical solution method for differential equation problems is able to reproduce a linear function exactly to machine precision (if u is about unity in size; precision is lost if u takes on large values, see Exercise 3.1). The linear solution also makes some stronger demands to the numerical method and the implementation than the constant solution used in Sect. 3.1.4, at least in more complicated applications. Still, the constant solution is often ideal for initial debugging before proceeding with a linear solution.

We choose a linear solution \(u(t)=ct+d\). From the initial condition it follows that \(d=I\). Inserting this u in the left-hand side of (Ch3.E14), i.e., the ODE, we get
Any function \(u=ct+I\) is then a correct solution if we choose
With this \(b(t)\) there are no restrictions on \(a(t)\) and c.
Let us prove that such a linear solution obeys the numerical schemes. To this end, we must check that \(u^{n}=ca(t_{n})(ct_{n}+I)\) fulfills the discrete equations. For these calculations, and later calculations involving linear solutions inserted in finite difference schemes, it is convenient to compute the action of a difference operator on a linear function t:
$$[D_{t}^{+}t]^{n} =\frac{t_{n+1}-t_{n}}{\Delta t}=1,$$
$$[D_{t}^{-}t]^{n} =\frac{t_{n}-t_{n-1}}{\Delta t}=1,$$
$$[D_{t}t]^{n} =\frac{t_{n+\frac{1}{2}}-t_{n-\frac{1}{2}}}{\Delta t}=\frac{(n+\frac{1}{2})\Delta t-(n-\frac{1}{2})\Delta t}{\Delta t}=1\thinspace.$$
Clearly, all three finite difference approximations to the derivative are exact for \(u(t)=t\) or its mesh function counterpart \(u^{n}=t_{n}\).
The difference equation for the Forward Euler scheme
with \(a^{n}=a(t_{n})\), \(b^{n}=c+a(t_{n})(ct_{n}+I)\), and \(u^{n}=ct_{n}+I\) then results in
which is always fulfilled. Similar calculations can be done for the Backward Euler and Crank–Nicolson schemes, or the θ-rule for that matter. In all cases, \(u^{n}=ct_{n}+I\) is an exact solution of the discrete equations. That is why we should expect that \(u^{n}-u_{\mbox{\footnotesize e}}(t_{n})=0\) mathematically and \(|u^{n}-u_{\mbox{\footnotesize e}}(t_{n})|\) less than a small number about the machine precision for \(n=0,\ldots,N_{t}\).

The following function offers an implementation of this verification test based on a linear exact solution:

Any error in the updating formula makes this test fail!

Choosing more complicated formulas as the exact solution, say \(\cos(t)\), will not make the numerical and exact solution coincide to machine precision, because finite differencing of \(\cos(t)\) does not exactly yield the exact derivative \(-\sin(t)\). In such cases, the verification procedure must be based on measuring the convergence rates as exemplified in Sect. 3.1.6. Convergence rates can be computed as long as one has an exact solution of a problem that the solver can be tested on, but this can always be obtained by the method of manufactured solutions.

3.1.6 Computing Convergence Rates

We expect that the error E in the numerical solution is reduced if the mesh size \(\Delta t\) is decreased. More specifically, many numerical methods obey a power-law relation between E and \(\Delta t\):
$$E=C\Delta t^{r},$$
where C and r are (usually unknown) constants independent of \(\Delta t\). The formula (Ch3.E26) is viewed as an asymptotic model valid for sufficiently small \(\Delta t\). How small is normally hard to estimate without doing numerical estimations of r.

The parameter r is known as the convergence rate. For example, if the convergence rate is 2, halving \(\Delta t\) reduces the error by a factor of 4. Diminishing \(\Delta t\) then has a greater impact on the error compared with methods that have r = 1. For a given value of r, we refer to the method as of r-th order. First- and second-order methods are most common in scientific computing.

Estimating r

There are two alternative ways of estimating C and r based on a set of m simulations with corresponding pairs \((\Delta t_{i},E_{i})\), \(i=0,\ldots,m-1\), and \(\Delta t_{i}<\Delta t_{i-1}\) (i.e., decreasing cell size).
  1. 1.

    Take the logarithm of (Ch3.E26), \(\ln E=r\ln\Delta t+\ln C\), and fit a straight line to the data points \((\Delta t_{i},E_{i})\), \(i=0,\ldots,m-1\).

  2. 2.
    Consider two consecutive experiments, \((\Delta t_{i},E_{i})\) and \((\Delta t_{i-1},E_{i-1})\). Dividing the equation \(E_{i-1}=C\Delta t_{i-1}^{r}\) by \(E_{i}=C\Delta t_{i}^{r}\) and solving for r yields
    $$r_{i-1}=\frac{\ln(E_{i-1}/E_{i})}{\ln(\Delta t_{i-1}/\Delta t_{i})}$$
    for \(i=1,\ldots,m-1\). Note that we have introduced a subindex \(i-1\) on r in (Ch3.E27) because r estimated from a pair of experiments must be expected to change with i.

The disadvantage of method 1 is that (Ch3.E26) might not be valid for the coarsest meshes (largest \(\Delta t\) values). Fitting a line to all the data points is then misleading. Method 2 computes convergence rates for pairs of experiments and allows us to see if the sequence r i converges to some value as \(i\rightarrow m-2\). The final \(r_{m-2}\) can then be taken as the convergence rate. If the coarsest meshes have a differing rate, the corresponding time steps are probably too large for (Ch3.E26) to be valid. That is, those time steps lie outside the asymptotic range of \(\Delta t\) values where the error behaves like (Ch3.E26).


We can compute \(r_{0},r_{1},\ldots,r_{m-2}\) from E i and \(\Delta t_{i}\) by the following function

Experiments with a series of time step values and \(\theta=0,1,0.5\) can be set up as follows, here embedded in a real test function:

The manufactured solution is conveniently computed by sympy. Let us choose \(u_{\mbox{\footnotesize e}}(t)=\sin(t)e^{-2t}\) and \(a(t)=t^{2}\). This implies that we must fit b as \(b(t)=u^{\prime}(t)-a(t)\). We first compute with sympy expressions and then we convert the exact solution, a, and b to Python functions that we can use in the subsequent numerical computing:

The complete code is found in the function test_convergence_rates in the file .

Running this code gives the output

We clearly see how the convergence rates approach the expected values.

Why convergence rates are important

The strong practical application of computing convergence rates is for verification: wrong convergence rates point to errors in the code, and correct convergence rates bring strong support for a correct implementation. Experience shows that bugs in the code easily destroy the expected convergence rate.

3.1.7 Extension to Systems of ODEs

Many ODE models involve more than one unknown function and more than one equation. Here is an example of two unknown functions \(u(t)\) and \(v(t)\):
$$u^{\prime} =au+bv,$$
$$v^{\prime} =cu+dv,$$
for constants \(a,b,c,d\). Applying the Forward Euler method to each equation results in a simple updating formula:
$$u^{n+1} =u^{n}+\Delta t(au^{n}+bv^{n}),$$
$$v^{n+1} =u^{n}+\Delta t(cu^{n}+dv^{n})\thinspace.$$
On the other hand, the Crank–Nicolson or Backward Euler schemes result in a \(2\times 2\) linear system for the new unknowns. The latter scheme becomes
$$u^{n+1} =u^{n}+\Delta t(au^{n+1}+bv^{n+1}),$$
$$v^{n+1} =v^{n}+\Delta t(cu^{n+1}+dv^{n+1})\thinspace.$$
Collecting \(u^{n+1}\) as well as \(v^{n+1}\) on the left-hand side results in
$$(1-\Delta ta)u^{n+1}+bv^{n+1} =u^{n},$$
$$cu^{n+1}+(1-\Delta td)v^{n+1} =v^{n},$$
which is a system of two coupled, linear, algebraic equations in two unknowns. These equations can be solved by hand (using standard techniques for two algebraic equations with two unknowns x and y), resulting in explicit formulas for \(u^{n+1}\) and \(v^{n+1}\) that can be directly implemented. For systems of ODEs with many equations and unknowns, one will express the coupled equations at each time level in matrix form and call software for numerical solution of linear systems of equations.

3.2 General First-Order ODEs

We now turn the attention to general, nonlinear ODEs and systems of such ODEs. Our focus is on numerical methods that can be readily reused for time-discretization of PDEs, and diffusion PDEs in particular. The methods are just briefly listed, and we refer to the rich literature for more detailed descriptions and analysis – the books [12, 2, 3, 4] are all excellent resources on numerical methods for ODEs. We also demonstrate the Odespy Python interface to a range of different software for general first-order ODE systems.

3.2.1 Generic Form of First-Order ODEs

ODEs are commonly written in the generic form
$$u^{\prime}=f(u,t),\quad u(0)=I,$$
where \(f(u,t)\) is some prescribed function. As an example, our most general exponential decay model (Ch3.E14) has \(f(u,t)=-a(t)u(t)+b(t)\).
The unknown u in (Ch3.E36) may either be a scalar function of time t, or a vector valued function of t in case of a system of ODEs with m unknown components:
In that case, the right-hand side is a vector-valued function with m components,
$$\begin{aligned}\displaystyle f(u,t)=(&\displaystyle f^{(0)}(u^{(0)}(t),\ldots,u^{(m-1)}(t)),\\ \displaystyle&\displaystyle f^{(1)}(u^{(0)}(t),\ldots,u^{(m-1)}(t)),\\ \displaystyle&\displaystyle\vdots\\ \displaystyle&\displaystyle f^{(m-1)}(u^{(0)}(t),\ldots,u^{(m-1)}(t)))\thinspace.\end{aligned}$$

Actually, any system of ODEs can be written in the form (Ch3.E36), but higher-order ODEs then need auxiliary unknown functions to enable conversion to a first-order system.

Next we list some well-known methods for \(u^{\prime}=f(u,t)\), valid both for a single ODE (scalar u) and systems of ODEs (vector u).

3.2.2 The θ-Rule

The θ-rule scheme applied to \(u^{\prime}=f(u,t)\) becomes
$$\frac{u^{n+1}-u^{n}}{\Delta t}=\theta f(u^{n+1},t_{n+1})+(1-\theta)f(u^{n},t_{n})\thinspace.$$
Bringing the unknown \(u^{n+1}\) to the left-hand side and the known terms on the right-hand side gives
$$u^{n+1}-\Delta t\theta f(u^{n+1},t_{n+1})=u^{n}+\Delta t(1-\theta)f(u^{n},t_{n})\thinspace.$$
For a general f (not linear in u), this equation is nonlinear in the unknown \(u^{n+1}\) unless \(\theta=0\). For a scalar ODE (m = 1), we have to solve a single nonlinear algebraic equation for \(u^{n+1}\), while for a system of ODEs, we get a system of coupled, nonlinear algebraic equations. Newton’s method is a popular solution approach in both cases. Note that with the Forward Euler scheme (\(\theta=0\)) we do not have to deal with nonlinear equations, because in that case we have an explicit updating formula for \(u^{n+1}\). This is known as an explicit scheme. With \(\theta\neq 1\) we have to solve (systems of) algebraic equations, and the scheme is said to be implicit.

3.2.3 An Implicit 2-Step Backward Scheme

The implicit backward method with 2 steps applies a three-level backward difference as approximation to \(u^{\prime}(t)\),
$$u^{\prime}(t_{n+1})\approx\frac{3u^{n+1}-4u^{n}+u^{n-1}}{2\Delta t},$$
which is an approximation of order \(\Delta t^{2}\) to the first derivative. The resulting scheme for \(u^{\prime}=f(u,t)\) reads
$$u^{n+1}=\frac{4}{3}u^{n}-\frac{1}{3}u^{n-1}+\frac{2}{3}\Delta tf(u^{n+1},t_{n+1})\thinspace.$$
Higher-order versions of the scheme (Ch3.E39) can be constructed by including more time levels. These schemes are known as the Backward Differentiation Formulas (BDF), and the particular version (Ch3.E39) is often referred to as BDF2.

Note that the scheme (Ch3.E39) is implicit and requires solution of nonlinear equations when f is nonlinear in u. The standard 1st-order Backward Euler method or the Crank–Nicolson scheme can be used for the first step.

3.2.4 Leapfrog Schemes

The ordinary Leapfrog scheme

The derivative of u at some point t n can be approximated by a central difference over two time steps,
$$u^{\prime}(t_{n})\approx\frac{u^{n+1}-u^{n-1}}{2\Delta t}=[D_{2t}u]^{n}$$
which is an approximation of second order in \(\Delta t\). The scheme can then be written as
in operator notation. Solving for \(u^{n+1}\) gives
$$u^{n+1}=u^{n-1}+2\Delta tf(u^{n},t_{n})\thinspace.$$
Observe that (Ch3.E41 ) is an explicit scheme, and that a nonlinear f (in u) is trivial to handle since it only involves the known u n value. Some other scheme must be used as starter to compute u 1, preferably the Forward Euler scheme since it is also explicit.

The filtered Leapfrog scheme

Unfortunately, the Leapfrog scheme (Ch3.E41) will develop growing oscillations with time (see Problem 3.6). A remedy for such undesired oscillations is to introduce a filtering technique. First, a standard Leapfrog step is taken, according to (Ch3.E41), and then the previous u n value is adjusted according to
$$u^{n}\ \leftarrow\ u^{n}+\gamma(u^{n-1}-2u^{n}+u^{n+1})\thinspace.$$
The γ-terms will effectively damp oscillations in the solution, especially those with short wavelength (like point-to-point oscillations). A common choice of γ is 0.6 (a value used in the famous NCAR Climate Model).

3.2.5 The 2nd-Order Runge–Kutta Method

The two-step scheme
$$u^{*} =u^{n}+\Delta tf(u^{n},t_{n}),$$
$$u^{n+1} =u^{n}+\Delta t\frac{1}{2}\left(f(u^{n},t_{n})+f(u^{*},t_{n+1})\right),$$
essentially applies a Crank–Nicolson method (Ch3.E44) to the ODE, but replaces the term \(f(u^{n+1},t_{n+1})\) by a prediction \(f(u^{*},t_{n+1})\) based on a Forward Euler step (Ch3.E43). The scheme (Ch3.E43)–(Ch3.E44) is known as Huen’s method, but is also a 2nd-order Runge–Kutta method. The scheme is explicit, and the error is expected to behave as \(\Delta t^{2}\).

3.2.6 A 2nd-Order Taylor-Series Method

One way to compute \(u^{n+1}\) given u n is to use a Taylor polynomial. We may write up a polynomial of 2nd degree:
$$u^{n+1}=u^{n}+u^{\prime}(t_{n})\Delta t+{\frac{1}{2}}u^{\prime\prime}(t_{n})\Delta t^{2}\thinspace.$$
From the equation \(u^{\prime}=f(u,t)\) it follows that the derivatives of u can be expressed in terms of f and its derivatives:
$$\begin{aligned}\displaystyle u^{\prime}(t_{n})&\displaystyle=f(u^{n},t_{n}),\\ \displaystyle u^{\prime\prime}(t_{n})&\displaystyle=\frac{\partial f}{\partial u}(u^{n},t_{n})u^{\prime}(t_{n})+\frac{\partial f}{\partial t}\\ \displaystyle&\displaystyle=f(u^{n},t_{n})\frac{\partial f}{\partial u}(u^{n},t_{n})+\frac{\partial f}{\partial t},\end{aligned}$$
resulting in the scheme
$$u^{n+1}=u^{n}+f(u^{n},t_{n})\Delta t+\frac{1}{2}\left(f(u^{n},t_{n})\frac{\partial f}{\partial u}(u^{n},t_{n})+\frac{\partial f}{\partial t}\right)\Delta t^{2}\thinspace.$$
More terms in the series could be included in the Taylor polynomial to obtain methods of higher order than 2.

3.2.7 The 2nd- and 3rd-Order Adams–Bashforth Schemes

The following method is known as the 2nd-order Adams–Bashforth scheme:
$$u^{n+1}=u^{n}+\frac{1}{2}\Delta t\left(3f(u^{n},t_{n})-f(u^{n-1},t_{n-1})\right)\thinspace.$$
The scheme is explicit and requires another one-step scheme to compute u 1 (the Forward Euler scheme or Heun’s method, for instance). As the name implies, the error behaves like \(\Delta t^{2}\).
Another explicit scheme, involving four time levels, is the 3rd-order Adams–Bashforth scheme
The numerical error is of order \(\Delta t^{3}\), and the scheme needs some method for computing u 1 and u 2.
More general, higher-order Adams–Bashforth schemes (also called explicit Adams methods) compute \(u^{n+1}\) as a linear combination of f at \(k+1\) previous time steps:
where \(\beta_{j}\) are known coefficients.

3.2.8 The 4th-Order Runge–Kutta Method

The perhaps most widely used method to solve ODEs is the 4th-order Runge–Kutta method, often called RK4. Its derivation is a nice illustration of common numerical approximation strategies, so let us go through the steps in detail to learn about algorithmic development.

The starting point is to integrate the ODE \(u^{\prime}=f(u,t)\) from t n to \(t_{n+1}\):
We want to compute \(u(t_{n+1})\) and regard \(u(t_{n})\) as known. The task is to find good approximations for the integral, since the integrand involves the unknown u between t n and \(t_{n+1}\).
The integral can be approximated by the famous Simpson’s rule:
$$\int\limits_{t_{n}}^{t_{n+1}}f(u(t),t)dt\approx\frac{\Delta t}{6}\left(f^{n}+4f^{n+\frac{1}{2}}+f^{n+1}\right)\thinspace.$$
The problem now is that we do not know \(f^{n+\frac{1}{2}}=f(u^{n+\frac{1}{2}},t_{n+\frac{1}{2}})\) and \(f^{n+1}=(u^{n+1},t_{n+1})\) as we know only u n and hence f n . The idea is to use various approximations for \(f^{n+\frac{1}{2}}\) and \(f^{n+1}\) based on well-known schemes for the ODE in the intervals \([t_{n},t_{n+\frac{1}{2}}]\) and \([t_{n},t_{n+1}]\). We split the integral approximation into four terms:
$$\int\limits_{t_{n}}^{t_{n+1}}f(u(t),t)dt\approx\frac{\Delta t}{6}\left(f^{n}+2\hat{f}^{n+\frac{1}{2}}+2\tilde{f}^{n+\frac{1}{2}}+\bar{f}^{n+1}\right),$$
where \(\hat{f}^{n+\frac{1}{2}}\), \(\tilde{f}^{n+\frac{1}{2}}\), and \(\bar{f}^{n+1}\) are approximations to \(f^{n+\frac{1}{2}}\) and \(f^{n+1}\), respectively, that can be based on already computed quantities. For \(\hat{f}^{n+\frac{1}{2}}\) we can apply an approximation to \(u^{n+\frac{1}{2}}\) using the Forward Euler method with step \(\frac{1}{2}\Delta t\):
$$\hat{f}^{n+\frac{1}{2}}=f(u^{n}+\frac{1}{2}{\Delta t}f^{n},t_{n+\frac{1}{2}})$$
Since this gives us a prediction of \(f^{n+\frac{1}{2}}\), we can for \(\tilde{f}^{n+\frac{1}{2}}\) try a Backward Euler method to approximate \(u^{n+\frac{1}{2}}\):
$$\tilde{f}^{n+\frac{1}{2}}=f(u^{n}+\frac{1}{2}\Delta t\hat{f}^{n+\frac{1}{2}},t_{n+\frac{1}{2}})\thinspace.$$
With \(\tilde{f}^{n+\frac{1}{2}}\) as a hopefully good approximation to \(f^{n+\frac{1}{2}}\), we can for the final term \(\bar{f}^{n+1}\) use a Crank–Nicolson method on \([t_{n},t_{n+1}]\) to approximate \(u^{n+1}\):
$$\bar{f}^{n+1}=f(u^{n}+\Delta t\hat{f}^{n+\frac{1}{2}},t_{n+1})\thinspace.$$
We have now used the Forward and Backward Euler methods as well as the Crank–Nicolson method in the context of Simpson’s rule. The hope is that the combination of these methods yields an overall time-stepping scheme from t n to \(t_{n}{+1}\) that is much more accurate than the \(\mathcal{O}(\Delta t)\) and \(\mathcal{O}(\Delta t^{2})\) of the individual steps. This is indeed true: the overall accuracy is \(\mathcal{O}(\Delta t^{4})\)!
To summarize, the 4th-order Runge–Kutta method becomes
$$u^{n+1}=u^{n}+\frac{\Delta t}{6}\left(f^{n}+2\hat{f}^{n+\frac{1}{2}}+2\tilde{f}^{n+\frac{1}{2}}+\bar{f}^{n+1}\right),$$
where the quantities on the right-hand side are computed from (Ch3.E48)–(Ch3.E50). Note that the scheme is fully explicit so there is never any need to solve linear or nonlinear algebraic equations. However, the stability is conditional and depends on f. There is a whole range of implicit Runge–Kutta methods that are unconditionally stable, but require solution of algebraic equations involving f at each time step.

The simplest way to explore more sophisticated methods for ODEs is to apply one of the many high-quality software packages that exist, as the next section explains.

3.2.9 The Odespy Software

A wide range of methods and software exist for solving (Ch3.E36). Many of the methods are accessible through a unified Python interface offered by the Odespy [10] package. Odespy features simple Python implementations of the most fundamental schemes as well as Python interfaces to several famous packages for solving ODEs: ODEPACK, Vode, rkc.f, rkf45.f, as well as the ODE solvers in SciPy, SymPy, and odelab.

The code below illustrates the usage of Odespy the solving \(u^{\prime}=-au\), \(u(0)=I\), \(t\in(0,T]\), by the famous 4th-order Runge–Kutta method, using \(\Delta t=1\) and \(N_{t}=6\) steps:

The previously listed methods for ODEs are all accessible in Odespy:
  • the θ-rule: ThetaRule

  • special cases of the θ-rule: ForwardEuler, BackwardEuler, CrankNicolson

  • the 2nd- and 4th-order Runge–Kutta methods: RK2 and RK4

  • The BDF methods and the Adam-Bashforth methods: Vode, Lsode, Lsoda, lsoda_scipy

  • The Leapfrog schemes: Leapfrog and LeapfrogFiltered

3.2.10 Example: Runge–Kutta Methods

Since all solvers have the same interface in Odespy, except for a potentially different set of parameters in the solvers’ constructors, one can easily make a list of solver objects and run a loop for comparing a lot of solvers. The code below, found in complete form in , compares the famous Runge–Kutta methods of orders 2, 3, and 4 with the exact solution of the decay equation \(u^{\prime}=-au\). Since we have quite long time steps, we have included the only relevant θ-rule for large time steps, the Backward Euler scheme (\(\theta=1\)), as well. Figure 3.1 shows the results.
Fig. 3.1

Behavior of different schemes for the decay equation

With the odespy.BackwardEuler method we must either tell that the problem is linear and provide the Jacobian of \(f(u,t)\), i.e., \(\partial f/\partial u\), as the jac argument, or we have to assume that f is nonlinear, but then specify Newton’s method as solver for the nonlinear equations (since the equations are linear, Newton’s method will converge in one iteration). By default, odespy.BackwardEuler assumes a nonlinear problem to be solved by Picard iteration, but that leads to divergence in the present problem.

Visualization tip

We use Matplotlib for plotting here, but one could alternatively import scitools.std as plt instead. Plain use of Matplotlib as done here results in curves with different colors, which may be hard to distinguish on black-and-white paper. Using scitools.std, curves are automatically given colors and markers, thus making curves easy to distinguish on screen with colors and on black-and-white paper. The automatic adding of markers is normally a bad idea for a very fine mesh since all the markers get cluttered, but scitools.std limits the number of markers in such cases. For the exact solution we use a very fine mesh, but in the code above we specify the line type as a solid line (-), which means no markers and just a color to be automatically determined by the backend used for plotting (Matplotlib by default, but scitools.std gives the opportunity to use other backends to produce the plot, e.g., Gnuplot or Grace).

Also note the that the legends are based on the class names of the solvers, and in Python the name of the class type (as a string) of an object obj is obtained by obj.__class__.__name__.

The runs in Fig. 3.1 and other experiments reveal that the 2nd-order Runge–Kutta method (RK2) is unstable for \(\Delta t> 1\) and decays slower than the Backward Euler scheme for large and moderate \(\Delta t\) (see Exercise 3.5 for an analysis). However, for fine \(\Delta t=0.25\) the 2nd-order Runge–Kutta method approaches the exact solution faster than the Backward Euler scheme. That is, the latter scheme does a better job for larger \(\Delta t\), while the higher order scheme is superior for smaller \(\Delta t\). This is a typical trend also for most schemes for ordinary and partial differential equations.

The 3rd-order Runge–Kutta method (RK3) also has artifacts in the form of oscillatory behavior for the larger \(\Delta t\) values, much like that of the Crank–Nicolson scheme. For finer \(\Delta t\), the 3rd-order Runge–Kutta method converges quickly to the exact solution.

The 4th-order Runge–Kutta method (RK4) is slightly inferior to the Backward Euler scheme on the coarsest mesh, but is then clearly superior to all the other schemes. It is definitely the method of choice for all the tested schemes.

Remark about using the θ-rule in Odespy

The Odespy package assumes that the ODE is written as \(u^{\prime}=f(u,t)\) with an f that is possibly nonlinear in u. The θ-rule for \(u^{\prime}=f(u,t)\) leads to
$$u^{n+1}=u^{n}+\Delta t\left(\theta f(u^{n+1},t_{n+1})+(1-\theta)f(u^{n},t_{n})\right),$$
which is a nonlinear equation in \(u^{n+1}\). Odespy’s implementation of the θ-rule (ThetaRule) and the specialized Backward Euler (BackwardEuler) and Crank–Nicolson (CrankNicolson) schemes must invoke iterative methods for solving the nonlinear equation in \(u^{n+1}\). This is done even when f is linear in u, as in the model problem \(u^{\prime}=-au\), where we can easily solve for \(u^{n+1}\) by hand. Therefore, we need to specify use of Newton’s method to solve the equations. (Odespy allows other methods than Newton’s to be used, for instance Picard iteration, but that method is not suitable. The reason is that it applies the Forward Euler scheme to generate a start value for the iterations. Forward Euler may give very wrong solutions for large \(\Delta t\) values. Newton’s method, on the other hand, is insensitive to the start value in linear problems.)

3.2.11 Example: Adaptive Runge–Kutta Methods

Odespy also offers solution methods that can adapt the size of \(\Delta t\) with time to match a desired accuracy in the solution. Intuitively, small time steps will be chosen in areas where the solution is changing rapidly, while larger time steps can be used where the solution is slowly varying. Some kind of error estimator is used to adjust the next time step at each time level.

A very popular adaptive method for solving ODEs is the Dormand-Prince Runge–Kutta method of order 4 and 5. The 5th-order method is used as a reference solution and the difference between the 4th- and 5th-order methods is used as an indicator of the error in the numerical solution. The Dormand-Prince method is the default choice in MATLAB’s widely used ode45 routine.

We can easily set up Odespy to use the Dormand-Prince method and see how it selects the optimal time steps. To this end, we request only one time step from t = 0 to \(t=T\) and ask the method to compute the necessary non-uniform time mesh to meet a certain error tolerance. The code goes like

Running four cases with tolerances \(10^{-1}\), \(10^{-3}\), \(10^{-5}\), and \(10^{-7}\), gives the results in Fig. 3.2. Intuitively, one would expect denser points in the beginning of the decay and larger time steps when the solution flattens out.
Fig. 3.2

Choice of adaptive time mesh by the Dormand-Prince method for different tolerances

3.3 Exercises

Exercise 3.1 (Experiment with precision in tests and the size of u)

It is claimed in Sect. 3.1.5 that most numerical methods will reproduce a linear exact solution to machine precision. Test this assertion using the test function test_linear_solution in the program. Vary the parameter c from very small, via c=1 to many larger values, and print out the maximum difference between the numerical solution and the exact solution. What is the relevant value of the tolerance in the float comparison in each case?

Filename: test_precision.

Exercise 3.2 (Implement the 2-step backward scheme)

Implement the 2-step backward method (Ch3.E39) for the model \(u^{\prime}(t)=-a(t)u(t)+b(t)\), \(u(0)=I\). Allow the first step to be computed by either the Backward Euler scheme or the Crank–Nicolson scheme. Verify the implementation by choosing \(a(t)\) and \(b(t)\) such that the exact solution is linear in t (see Sect. 3.1.5). Show mathematically that a linear solution is indeed a solution of the discrete equations.

Compute convergence rates (see Sect. 3.1.6) in a test case using \(a=\text{const}\) and b = 0, where we easily have an exact solution, and determine if the choice of a first-order scheme (Backward Euler) for the first step has any impact on the overall accuracy of this scheme. The expected error goes like \(\mathcal{O}(\Delta t^{2})\).

Filename: decay_backward2step.

Exercise 3.3 (Implement the 2nd-order Adams–Bashforth scheme)

Implement the 2nd-order Adams–Bashforth method (Ch3.E46) for the decay problem \(u^{\prime}=-a(t)u+b(t)\), \(u(0)=I\), \(t\in(0,T]\). Use the Forward Euler method for the first step such that the overall scheme is explicit. Verify the implementation using an exact solution that is linear in time. Analyze the scheme by searching for solutions \(u^{n}=A^{n}\) when \(a=\text{const}\) and b = 0. Compare this second-order scheme to the Crank–Nicolson scheme.

Filename: decay_AdamsBashforth2.

Exercise 3.4 (Implement the 3rd-order Adams–Bashforth scheme)

Implement the 3rd-order Adams–Bashforth method (Ch3.E47) for the decay problem \(u^{\prime}=-a(t)u+b(t)\), \(u(0)=I\), \(t\in(0,T]\). Since the scheme is explicit, allow it to be started by two steps with the Forward Euler method. Investigate experimentally the case where b = 0 and a is a constant: Can we have oscillatory solutions for large \(\Delta t\)?

Filename: decay_AdamsBashforth3.

Exercise 3.5 (Analyze explicit 2nd-order methods)

Show that the schemes (Ch3.E44) and (Ch3.E45) are identical in the case \(f(u,t)=-a\), where a > 0 is a constant. Assume that the numerical solution reads \(u^{n}=A^{n}\) for some unknown amplification factor A to be determined. Find A and derive stability criteria. Can the scheme produce oscillatory solutions of \(u^{\prime}=-au\)? Plot the numerical and exact amplification factor.

Filename: decay_RK2_Taylor2.

Project 3.6 (Implement and investigate the Leapfrog scheme)

A Leapfrog scheme for the ODE \(u^{\prime}(t)=-a(t)u(t)+b(t)\) is defined by
A separate method is needed to compute u 1. The Forward Euler scheme is a possible candidate.
  1. a)

    Implement the Leapfrog scheme for the model equation. Plot the solution in the case a = 1, b = 0, I = 1, \(\Delta t=0.01\), \(t\in[0,4]\). Compare with the exact solution \(u_{\mbox{\footnotesize e}}(t)=e^{-t}\).

  2. b)

    Show mathematically that a linear solution in t fulfills the Forward Euler scheme for the first step and the Leapfrog scheme for the subsequent steps. Use this linear solution to verify the implementation, and automate the verification through a test function.



It can be wise to automate the calculations such that it is easy to redo the calculations for other types of solutions. Here is a possible sympy function that takes a symbolic expression u (implemented as a Python function of t), fits the b term, and checks if u fulfills the discrete equations:

  1. a)

    Show that a second-order polynomial in t cannot be a solution of the discrete equations. However, if a Crank–Nicolson scheme is used for the first step, a second-order polynomial solves the equations exactly.

  2. b)

    Create a manufactured solution \(u(t)=\sin(t)\) for the ODE \(u^{\prime}=-au+b\). Compute the convergence rate of the Leapfrog scheme using this manufactured solution. The expected convergence rate of the Leapfrog scheme is \(\mathcal{O}(\Delta t^{2})\). Does the use of a 1st-order method for the first step impact the convergence rate?

  3. c)

    Set up a set of experiments to demonstrate that the Leapfrog scheme (Ch3.E52) is associated with numerical artifacts (instabilities). Document the main results from this investigation.

  4. d)
    Analyze and explain the instabilities of the Leapfrog scheme (Ch3.E52):
    1. 1.

      Choose \(a=\text{const}\) and b = 0. Assume that an exact solution of the discrete equations has the form \(u^{n}=A^{n}\), where A is an amplification factor to be determined. Derive an equation for A by inserting \(u^{n}=A^{n}\) in the Leapfrog scheme.

    2. 2.

      Compute A either by hand and/or with the aid of sympy. The polynomial for A has two roots, A 1 and A 2. Let u n be a linear combination \(u^{n}=C_{1}A_{1}^{n}+C_{2}A_{2}^{n}\).

    3. 3.

      Show that one of the roots is the reason for instability.

    4. 4.

      Compare A with the exact expression, using a Taylor series approximation.

    5. 5.

      How can C 1 and C 2 be determined?

  5. e)

    Since the original Leapfrog scheme is unconditionally unstable as time grows, it demands some stabilization. This can be done by filtering, where we first find \(u^{n+1}\) from the original Leapfrog scheme and then replace u n by \(u^{n}+\gamma(u^{n-1}-2u^{n}+u^{n+1})\), where γ can be taken as 0.6. Implement the filtered Leapfrog scheme and check that it can handle tests where the original Leapfrog scheme is unstable.

Filename: decay_leapfrog.

Problem 3.7 (Make a unified implementation of many schemes)

Consider the linear ODE problem \(u^{\prime}(t)=-a(t)u(t)+b(t)\), \(u(0)=I\). Explicit schemes for this problem can be written in the general form
for some choice of \(c_{0},\ldots,c_{m}\). Find expressions for the c j coefficients in case of the θ-rule, the three-level backward scheme, the Leapfrog scheme, the 2nd-order Runge–Kutta method, and the 3rd-order Adams–Bashforth scheme.

Make a class ExpDecay that implements the general updating formula (Ch3.E53). The formula cannot be applied for \(n<m\), and for those n values, other schemes must be used. Assume for simplicity that we just repeat Crank–Nicolson steps until (Ch3.E53) can be used. Use a subclass to specify the list \(c_{0},\ldots,c_{m}\) for a particular method, and implement subclasses for all the mentioned schemes. Verify the implementation by testing with a linear solution, which should be exactly reproduced by all methods.

Filename: decay_schemes_unified.

Copyright information

© The Author(s) 2016

Open Access This chapter is distributed under the terms of the Creative Commons Attribution‐NonCommercial 4.0 International License (, which permits any noncommercial use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, a link is provided to the Creative Commons license and any changes made are indicated.

The images or other third party material in this chapter are included in the work’s Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work’s Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.

Authors and Affiliations

  1. 1.Simula Research LaboratoryLysakerNorway

Personalised recommendations