Generalizations
Abstract
This chapters targets generalizations of the simple decay model \(u^{\prime}=au\) and also to look at additional numerical solution methods. We consider first variable coefficients, \(u^{\prime}=a(t)u+b(t)\), and later a completely general scalar ODE \(u^{\prime}=f(u,t)\) and its generalization to a system of such general ODEs. Among numerical methods, we treat implicit multistep methods, and several families of explicit methods: Leapfrog schemes, Runge–Kutta methods, and Adams–Bashforth formulas.
It is time to consider generalizations of the simple decay model \(u^{\prime}=au\) and also to look at additional numerical solution methods. We consider first variable coefficients, \(u^{\prime}=a(t)u+b(t)\), and later a completely general scalar ODE \(u^{\prime}=f(u,t)\) and its generalization to a system of such general ODEs. Among numerical methods, we treat implicit multistep methods, and several families of explicit methods: Leapfrog schemes, Runge–Kutta methods, and Adams–Bashforth formulas.
3.1 Model Extensions
This section looks at the generalizations to \(u^{\prime}=a(t)u\) and \(u^{\prime}=a(t)u+b(t)\). We sketch the corresponding implementations of the θrule for such variablecoefficient ODEs. Verification can no longer make use of an exact solution of the numerical problem so we make use of manufactured solutions, for deriving an exact solution of the ODE problem, and then we can compute empirical convergence rates for the method and see if these coincide with the expected rates from theory. Finally, we see how our numerical methods can be applied to systems of ODEs.
The example programs associated with this chapter are found in the directory src/genz .
3.1.1 Generalization: Including a Variable Coefficient
3.1.2 Generalization: Including a Source Term
3.1.3 Implementation of the Generalized Model Problem
Deriving the θrule formula
Python code
Here is a suitable implementation of (Ch3.E21) where \(a(t)\) and \(b(t)\) are given as Python functions:
This function is found in the file decay_vc.py (vc stands for ‘‘variable coefficients’’).
Coding of variable coefficients
The solver function shown above demands the arguments a and b to be Python functions of time t, say
Here, a(t) has three parameters a0, tp, and k, which must be global variables.
A better implementation, which avoids global variables, is to represent a by a class where the parameters are attributes and where a special method __call__ evaluates \(a(t)\):
For quick tests it is cumbersome to write a complete function or a class. The lambda function construction in Python is then convenient. For example,
is equivalent to the def a(t) definition above. In general,
is equivalent to
One can use lambda functions directly in calls. Say we want to solve \(u^{\prime}=u+1\), \(u(0)=2\):
Whether to use a plain function, a class, or a lambda function depends on the programmer’s taste. Lazy programmers prefer the lambda construct, while very safe programmers go for the class solution.
3.1.4 Verifying a Constant Solution
An extremely useful partial verification method is to construct a test problem with a very simple solution, usually \(u=\text{const}\). Especially the initial debugging of a program code can benefit greatly from such tests, because 1) all relevant numerical methods will exactly reproduce a constant solution, 2) many of the intermediate calculations are easy to control by hand for a constant u, and 3) even a constant u can uncover many bugs in an implementation.
The only constant solution for the problem \(u^{\prime}=au\) is u = 0, but too many bugs can escape from that trivial solution. It is much better to search for a problem where \(u=C=\text{const}\neq 0\). Then \(u^{\prime}=a(t)u+b(t)\) is more appropriate: with \(u=C\) we can choose any \(a(t)\) and set \(b=a(t)C\) and \(I=C\). An appropriate test function is
An interesting question is what type of bugs that will make the computed u ^{ n } deviate from the exact solution C. Fortunately, the updating formula and the initial condition must be absolutely correct for the test to pass! Any attempt to make a wrong indexing in terms like a(t[n]) or any attempt to introduce an erroneous factor in the formula creates a solution that is different from C.
3.1.5 Verification via Manufactured Solutions
Following the idea of the previous section, we can choose any formula as the exact solution, insert the formula in the ODE problem and fit the data \(a(t)\), \(b(t)\), and I to make the chosen formula fulfill the equation. This powerful technique for generating exact solutions is very useful for verification purposes and known as the method of manufactured solutions, often abbreviated MMS.
One common choice of solution is a linear function in the independent variable(s). The rationale behind such a simple variation is that almost any relevant numerical solution method for differential equation problems is able to reproduce a linear function exactly to machine precision (if u is about unity in size; precision is lost if u takes on large values, see Exercise 3.1). The linear solution also makes some stronger demands to the numerical method and the implementation than the constant solution used in Sect. 3.1.4, at least in more complicated applications. Still, the constant solution is often ideal for initial debugging before proceeding with a linear solution.
The following function offers an implementation of this verification test based on a linear exact solution:
Any error in the updating formula makes this test fail!
Choosing more complicated formulas as the exact solution, say \(\cos(t)\), will not make the numerical and exact solution coincide to machine precision, because finite differencing of \(\cos(t)\) does not exactly yield the exact derivative \(\sin(t)\). In such cases, the verification procedure must be based on measuring the convergence rates as exemplified in Sect. 3.1.6. Convergence rates can be computed as long as one has an exact solution of a problem that the solver can be tested on, but this can always be obtained by the method of manufactured solutions.
3.1.6 Computing Convergence Rates
The parameter r is known as the convergence rate. For example, if the convergence rate is 2, halving \(\Delta t\) reduces the error by a factor of 4. Diminishing \(\Delta t\) then has a greater impact on the error compared with methods that have r = 1. For a given value of r, we refer to the method as of rth order. First and secondorder methods are most common in scientific computing.
Estimating r
 1.
Take the logarithm of (Ch3.E26), \(\ln E=r\ln\Delta t+\ln C\), and fit a straight line to the data points \((\Delta t_{i},E_{i})\), \(i=0,\ldots,m1\).
 2.Consider two consecutive experiments, \((\Delta t_{i},E_{i})\) and \((\Delta t_{i1},E_{i1})\). Dividing the equation \(E_{i1}=C\Delta t_{i1}^{r}\) by \(E_{i}=C\Delta t_{i}^{r}\) and solving for r yieldsfor \(i=1,\ldots,m1\). Note that we have introduced a subindex \(i1\) on r in (Ch3.E27) because r estimated from a pair of experiments must be expected to change with i.$$r_{i1}=\frac{\ln(E_{i1}/E_{i})}{\ln(\Delta t_{i1}/\Delta t_{i})}$$(3.27)
The disadvantage of method 1 is that (Ch3.E26) might not be valid for the coarsest meshes (largest \(\Delta t\) values). Fitting a line to all the data points is then misleading. Method 2 computes convergence rates for pairs of experiments and allows us to see if the sequence r _{ i } converges to some value as \(i\rightarrow m2\). The final \(r_{m2}\) can then be taken as the convergence rate. If the coarsest meshes have a differing rate, the corresponding time steps are probably too large for (Ch3.E26) to be valid. That is, those time steps lie outside the asymptotic range of \(\Delta t\) values where the error behaves like (Ch3.E26).
Implementation
We can compute \(r_{0},r_{1},\ldots,r_{m2}\) from E _{ i } and \(\Delta t_{i}\) by the following function
Experiments with a series of time step values and \(\theta=0,1,0.5\) can be set up as follows, here embedded in a real test function:
The manufactured solution is conveniently computed by sympy. Let us choose \(u_{\mbox{\footnotesize e}}(t)=\sin(t)e^{2t}\) and \(a(t)=t^{2}\). This implies that we must fit b as \(b(t)=u^{\prime}(t)a(t)\). We first compute with sympy expressions and then we convert the exact solution, a, and b to Python functions that we can use in the subsequent numerical computing:
The complete code is found in the function test_convergence_rates in the file decay_vc.py .
Running this code gives the output
We clearly see how the convergence rates approach the expected values.
Why convergence rates are important
The strong practical application of computing convergence rates is for verification: wrong convergence rates point to errors in the code, and correct convergence rates bring strong support for a correct implementation. Experience shows that bugs in the code easily destroy the expected convergence rate.
3.1.7 Extension to Systems of ODEs
3.2 General FirstOrder ODEs
We now turn the attention to general, nonlinear ODEs and systems of such ODEs. Our focus is on numerical methods that can be readily reused for timediscretization of PDEs, and diffusion PDEs in particular. The methods are just briefly listed, and we refer to the rich literature for more detailed descriptions and analysis – the books [12, 2, 3, 4] are all excellent resources on numerical methods for ODEs. We also demonstrate the Odespy Python interface to a range of different software for general firstorder ODE systems.
3.2.1 Generic Form of FirstOrder ODEs
Actually, any system of ODEs can be written in the form (Ch3.E36), but higherorder ODEs then need auxiliary unknown functions to enable conversion to a firstorder system.
Next we list some wellknown methods for \(u^{\prime}=f(u,t)\), valid both for a single ODE (scalar u) and systems of ODEs (vector u).
3.2.2 The θRule
3.2.3 An Implicit 2Step Backward Scheme
Note that the scheme (Ch3.E39) is implicit and requires solution of nonlinear equations when f is nonlinear in u. The standard 1storder Backward Euler method or the Crank–Nicolson scheme can be used for the first step.
3.2.4 Leapfrog Schemes
The ordinary Leapfrog scheme
The filtered Leapfrog scheme
3.2.5 The 2ndOrder Runge–Kutta Method
3.2.6 A 2ndOrder TaylorSeries Method
3.2.7 The 2nd and 3rdOrder Adams–Bashforth Schemes
3.2.8 The 4thOrder Runge–Kutta Method
The perhaps most widely used method to solve ODEs is the 4thorder Runge–Kutta method, often called RK4. Its derivation is a nice illustration of common numerical approximation strategies, so let us go through the steps in detail to learn about algorithmic development.
The simplest way to explore more sophisticated methods for ODEs is to apply one of the many highquality software packages that exist, as the next section explains.
3.2.9 The Odespy Software
A wide range of methods and software exist for solving (Ch3.E36). Many of the methods are accessible through a unified Python interface offered by the Odespy [10] package. Odespy features simple Python implementations of the most fundamental schemes as well as Python interfaces to several famous packages for solving ODEs: ODEPACK, Vode, rkc.f, rkf45.f, as well as the ODE solvers in SciPy, SymPy, and odelab.
The code below illustrates the usage of Odespy the solving \(u^{\prime}=au\), \(u(0)=I\), \(t\in(0,T]\), by the famous 4thorder Runge–Kutta method, using \(\Delta t=1\) and \(N_{t}=6\) steps:

the θrule: ThetaRule

special cases of the θrule: ForwardEuler, BackwardEuler, CrankNicolson

the 2nd and 4thorder Runge–Kutta methods: RK2 and RK4

The BDF methods and the AdamBashforth methods: Vode, Lsode, Lsoda, lsoda_scipy

The Leapfrog schemes: Leapfrog and LeapfrogFiltered
3.2.10 Example: Runge–Kutta Methods
With the odespy.BackwardEuler method we must either tell that the problem is linear and provide the Jacobian of \(f(u,t)\), i.e., \(\partial f/\partial u\), as the jac argument, or we have to assume that f is nonlinear, but then specify Newton’s method as solver for the nonlinear equations (since the equations are linear, Newton’s method will converge in one iteration). By default, odespy.BackwardEuler assumes a nonlinear problem to be solved by Picard iteration, but that leads to divergence in the present problem.
Visualization tip
We use Matplotlib for plotting here, but one could alternatively import scitools.std as plt instead. Plain use of Matplotlib as done here results in curves with different colors, which may be hard to distinguish on blackandwhite paper. Using scitools.std, curves are automatically given colors and markers, thus making curves easy to distinguish on screen with colors and on blackandwhite paper. The automatic adding of markers is normally a bad idea for a very fine mesh since all the markers get cluttered, but scitools.std limits the number of markers in such cases. For the exact solution we use a very fine mesh, but in the code above we specify the line type as a solid line (), which means no markers and just a color to be automatically determined by the backend used for plotting (Matplotlib by default, but scitools.std gives the opportunity to use other backends to produce the plot, e.g., Gnuplot or Grace).
Also note the that the legends are based on the class names of the solvers, and in Python the name of the class type (as a string) of an object obj is obtained by obj.__class__.__name__.
The runs in Fig. 3.1 and other experiments reveal that the 2ndorder Runge–Kutta method (RK2) is unstable for \(\Delta t> 1\) and decays slower than the Backward Euler scheme for large and moderate \(\Delta t\) (see Exercise 3.5 for an analysis). However, for fine \(\Delta t=0.25\) the 2ndorder Runge–Kutta method approaches the exact solution faster than the Backward Euler scheme. That is, the latter scheme does a better job for larger \(\Delta t\), while the higher order scheme is superior for smaller \(\Delta t\). This is a typical trend also for most schemes for ordinary and partial differential equations.
The 3rdorder Runge–Kutta method (RK3) also has artifacts in the form of oscillatory behavior for the larger \(\Delta t\) values, much like that of the Crank–Nicolson scheme. For finer \(\Delta t\), the 3rdorder Runge–Kutta method converges quickly to the exact solution.
The 4thorder Runge–Kutta method (RK4) is slightly inferior to the Backward Euler scheme on the coarsest mesh, but is then clearly superior to all the other schemes. It is definitely the method of choice for all the tested schemes.
Remark about using the θrule in Odespy
3.2.11 Example: Adaptive Runge–Kutta Methods
Odespy also offers solution methods that can adapt the size of \(\Delta t\) with time to match a desired accuracy in the solution. Intuitively, small time steps will be chosen in areas where the solution is changing rapidly, while larger time steps can be used where the solution is slowly varying. Some kind of error estimator is used to adjust the next time step at each time level.
A very popular adaptive method for solving ODEs is the DormandPrince Runge–Kutta method of order 4 and 5. The 5thorder method is used as a reference solution and the difference between the 4th and 5thorder methods is used as an indicator of the error in the numerical solution. The DormandPrince method is the default choice in MATLAB’s widely used ode45 routine.
We can easily set up Odespy to use the DormandPrince method and see how it selects the optimal time steps. To this end, we request only one time step from t = 0 to \(t=T\) and ask the method to compute the necessary nonuniform time mesh to meet a certain error tolerance. The code goes like
3.3 Exercises
Exercise 3.1 (Experiment with precision in tests and the size of u)
It is claimed in Sect. 3.1.5 that most numerical methods will reproduce a linear exact solution to machine precision. Test this assertion using the test function test_linear_solution in the decay_vc.py program. Vary the parameter c from very small, via c=1 to many larger values, and print out the maximum difference between the numerical solution and the exact solution. What is the relevant value of the tolerance in the float comparison in each case?
Filename: test_precision.
Exercise 3.2 (Implement the 2step backward scheme)
Implement the 2step backward method (Ch3.E39) for the model \(u^{\prime}(t)=a(t)u(t)+b(t)\), \(u(0)=I\). Allow the first step to be computed by either the Backward Euler scheme or the Crank–Nicolson scheme. Verify the implementation by choosing \(a(t)\) and \(b(t)\) such that the exact solution is linear in t (see Sect. 3.1.5). Show mathematically that a linear solution is indeed a solution of the discrete equations.
Compute convergence rates (see Sect. 3.1.6) in a test case using \(a=\text{const}\) and b = 0, where we easily have an exact solution, and determine if the choice of a firstorder scheme (Backward Euler) for the first step has any impact on the overall accuracy of this scheme. The expected error goes like \(\mathcal{O}(\Delta t^{2})\).
Filename: decay_backward2step.
Exercise 3.3 (Implement the 2ndorder Adams–Bashforth scheme)
Implement the 2ndorder Adams–Bashforth method (Ch3.E46) for the decay problem \(u^{\prime}=a(t)u+b(t)\), \(u(0)=I\), \(t\in(0,T]\). Use the Forward Euler method for the first step such that the overall scheme is explicit. Verify the implementation using an exact solution that is linear in time. Analyze the scheme by searching for solutions \(u^{n}=A^{n}\) when \(a=\text{const}\) and b = 0. Compare this secondorder scheme to the Crank–Nicolson scheme.
Filename: decay_AdamsBashforth2.
Exercise 3.4 (Implement the 3rdorder Adams–Bashforth scheme)
Implement the 3rdorder Adams–Bashforth method (Ch3.E47) for the decay problem \(u^{\prime}=a(t)u+b(t)\), \(u(0)=I\), \(t\in(0,T]\). Since the scheme is explicit, allow it to be started by two steps with the Forward Euler method. Investigate experimentally the case where b = 0 and a is a constant: Can we have oscillatory solutions for large \(\Delta t\)?
Filename: decay_AdamsBashforth3.
Exercise 3.5 (Analyze explicit 2ndorder methods)
Show that the schemes (Ch3.E44) and (Ch3.E45) are identical in the case \(f(u,t)=a\), where a > 0 is a constant. Assume that the numerical solution reads \(u^{n}=A^{n}\) for some unknown amplification factor A to be determined. Find A and derive stability criteria. Can the scheme produce oscillatory solutions of \(u^{\prime}=au\)? Plot the numerical and exact amplification factor.
Filename: decay_RK2_Taylor2.
Project 3.6 (Implement and investigate the Leapfrog scheme)
 a)
Implement the Leapfrog scheme for the model equation. Plot the solution in the case a = 1, b = 0, I = 1, \(\Delta t=0.01\), \(t\in[0,4]\). Compare with the exact solution \(u_{\mbox{\footnotesize e}}(t)=e^{t}\).
 b)
Show mathematically that a linear solution in t fulfills the Forward Euler scheme for the first step and the Leapfrog scheme for the subsequent steps. Use this linear solution to verify the implementation, and automate the verification through a test function.
Hint
It can be wise to automate the calculations such that it is easy to redo the calculations for other types of solutions. Here is a possible sympy function that takes a symbolic expression u (implemented as a Python function of t), fits the b term, and checks if u fulfills the discrete equations:
 a)
Show that a secondorder polynomial in t cannot be a solution of the discrete equations. However, if a Crank–Nicolson scheme is used for the first step, a secondorder polynomial solves the equations exactly.
 b)
Create a manufactured solution \(u(t)=\sin(t)\) for the ODE \(u^{\prime}=au+b\). Compute the convergence rate of the Leapfrog scheme using this manufactured solution. The expected convergence rate of the Leapfrog scheme is \(\mathcal{O}(\Delta t^{2})\). Does the use of a 1storder method for the first step impact the convergence rate?
 c)
Set up a set of experiments to demonstrate that the Leapfrog scheme (Ch3.E52) is associated with numerical artifacts (instabilities). Document the main results from this investigation.
 d)Analyze and explain the instabilities of the Leapfrog scheme (Ch3.E52):
 1.
Choose \(a=\text{const}\) and b = 0. Assume that an exact solution of the discrete equations has the form \(u^{n}=A^{n}\), where A is an amplification factor to be determined. Derive an equation for A by inserting \(u^{n}=A^{n}\) in the Leapfrog scheme.
 2.
Compute A either by hand and/or with the aid of sympy. The polynomial for A has two roots, A _{1} and A _{2}. Let u ^{ n } be a linear combination \(u^{n}=C_{1}A_{1}^{n}+C_{2}A_{2}^{n}\).
 3.
Show that one of the roots is the reason for instability.
 4.
Compare A with the exact expression, using a Taylor series approximation.
 5.
How can C _{1} and C _{2} be determined?
 1.
 e)
Since the original Leapfrog scheme is unconditionally unstable as time grows, it demands some stabilization. This can be done by filtering, where we first find \(u^{n+1}\) from the original Leapfrog scheme and then replace u ^{ n } by \(u^{n}+\gamma(u^{n1}2u^{n}+u^{n+1})\), where γ can be taken as 0.6. Implement the filtered Leapfrog scheme and check that it can handle tests where the original Leapfrog scheme is unstable.
Problem 3.7 (Make a unified implementation of many schemes)
Make a class ExpDecay that implements the general updating formula (Ch3.E53). The formula cannot be applied for \(n<m\), and for those n values, other schemes must be used. Assume for simplicity that we just repeat Crank–Nicolson steps until (Ch3.E53) can be used. Use a subclass to specify the list \(c_{0},\ldots,c_{m}\) for a particular method, and implement subclasses for all the mentioned schemes. Verify the implementation by testing with a linear solution, which should be exactly reproduced by all methods.
Filename: decay_schemes_unified.
Copyright information
Open Access This chapter is distributed under the terms of the Creative Commons Attribution‐NonCommercial 4.0 International License (http://creativecommons.org/licenses/bync/4.0/), which permits any noncommercial use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, a link is provided to the Creative Commons license and any changes made are indicated.
The images or other third party material in this chapter are included in the work’s Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work’s Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.