Automatic numerical error analysis with interval arithmetic: an example
The goal of this example is to demonstrate that interval arithmetic performs automatic numerical error analysis. The reader can think of interval arithmetic as a computationally cheap way to get guaranteed lower and upper bounds on the range of a function over a given domain of the variables. The obtained bounds are not necessarily sharp, but they are guaranteed to enclose the true range of the function despite the intermediate computations being carried out in floating-point arithmetic, and potentially suffer catastrophic loss of precision. Interval arithmetic can safely work with infinity, division by zero, etc., and automatically keeps track of the numerical error propagation throughout the intermediate floating-point computations.
We examine the numerical behavior of the following two functions:
$$\begin{aligned} f(n) = \left( 1 + \frac{1}{n}\right) ^n, \quad g(n) = \left( 1 - \frac{1}{n}\right) ^{-n}, \quad n \ge 2. \end{aligned}$$
(1)
It is known, see e.g. Li and Yeh (2013), that f(n) is monotone increasing, g(n) is monotone decreasing, \({\lim \nolimits _{n\rightarrow \infty }}f(n)= {\lim \nolimits _{n\rightarrow \infty }}g(n)=e\), and as a consequence \(f(n)<e<g(n)\). If we carried out the computations in exact arithmetic, we would get tighter and tighter enclosures for e as n increases. However, we get the erratic results shown in Table 1 when the computations are carried out with 64 bit floating-point numbers on a computer. The source code of the example is available on GitHub at Baharev (2016) if the reader wishes to reproduce the numerical results, or analyze the implementation. For \(k=9,10,11,12,15\), the supposed lower bounding f(n) values exceed the true value of e, where \(n=10^k\); these are the rows with negative entries in the column for \(e-f(n)\). For \(k=9,12,14,15,17\), the supposed upper bounding g(n) values fall below the true value of e; these are the rows with negative entries in the column for \(g(n)-e\). For \(k=9,12,15\), the supposed lower bounding values exceed even the supposed upper bounding values, meaning that we did not even get an enclosure. The f(n) values are supposed to be increasing, but for \(k=13,14,16,17\) it is clearly not the case. Similarly, g(n) is supposed to be monotone decreasing, but it is violated, e.g., at \(k=13,16\).
Table 1 The numerical values of f(n) and g(n) defined in (1) when evaluated with 64 bit floating-point numbers on a computer where \(n = 10^{k}\)
Since the f(n) and g(n) functions are fairly simple, one could carry out a rigorous error analysis of these functions with pen and paper, and figure out the accuracy of the table entries. We will do this automatically with interval arithmetic.
Interval arithmetic takes the function \(f(10^k)\), and the interval for k as input. (The interval for k is just the point interval [k, k] for each row in Table 1.) The output of the interval function evaluation is a rigorous enclosure of the range of \(f(10^k)\) over [k, k]. In exact arithmetic, the range of \(f(10^k)\) over the point interval [k, k] is obviously just a single real number, the value \(f(10^k)\). However, we carry out the computations with 64 bit floating-point numbers on a computer. Interval arithmetic automatically keeps track of the numerical errors that occur during these computations, and we do not get a single real number but a pair of floating-point numbers as a result, that is, an interval enclosing the value of \(f(10^k)\), the interval \([\underline{f}(10^k),\overline{f}(10^k)]\). These intervals are given in Table 2 for \(k=1,2,\dots ,17\), together with the similarly computed \([\underline{g}(10^k),\overline{g}(10^k)]\).
Table 2 Rigorous enclosures of f(n) and g(n) defined in (1) where \(n=10^k\)
The wide intervals in Table 2, for example the rows for \(k\ge 15\), are a clear sign of the intermediate computations suffering catastrophic loss of precision; it is guaranteed that with interval arithmetic we always get informed (through wide intervals) when this happens. We consider this one of the biggest advantages of this approach.
Despite the serious numerical difficulties for \(k\ge 9\), the above discussed properties of f(n) and g(n) are still preserved in some form: (i) \(\overline{f}(n)\) is monotone increasing, and \(\underline{g}(n)\) is monotone decreasing, (ii) \(\underline{f}(n)<e<\overline{g}(n)\) holds (unlike in Table 1, there are no negative entries in the last two columns of Table 2). The fact that these properties are preserved is not a coincidence either but the guaranteed behavior of interval arithmetic. However, note that we did not get tighter and tighter enclosures for e as k increased: The enclosure \([\underline{f}(10^k),\overline{g}(10^k)]\) reaches its minimum width at \(k=8\), then the width starts increasing. We cannot blame interval arithmetic for this: Interval arithmetic is implemented on the top of 64 bit floating-point numbers, and unless one uses some extended precision arithmetic, e cannot be enclosed better with this simple approach. The tightest verified enclosure we got is [2.71828179, 2.71828186] for \(k=8\); indeed, the correct value is \(2.718281828\ldots \), and it is enclosed.
Let us emphasize again that for this simple example one could have derived bounds on the numerical errors of the entries in Table 1 with pen and paper. The advantage of interval arithmetic is that the numerical error analysis of the computations happens fully automatically, and therefore certain kinds of human errors are completely eliminated.
A formal overview of interval arithmetic
Interval arithmetic is an extension of real arithmetic defined on the set of real intervals, rather than the set of real numbers. According to a survey paper by Kearfott (1996), a form of interval arithmetic perhaps first appeared in Burkill (1924). Modern interval arithmetic was originally invented independently in the late 1950s by several researchers; including Warmus (1956), Sunaga (1958) and finally Moore (1959), who set firm foundations for the field in his many publications, including the foundational book Moore (1966). Since then, interval arithmetic is being used to rigorously solve numerical problems.
Let \(\mathbb {IR}\) be the set of all real intervals, and take \(\mathbf {x}, \mathbf {y}\in \mathbb {IR}\). We set \(\underline{x} := \inf {\mathbf {x}}\) and \(\overline{x} := \sup {\mathbf {x}}\), such that \(\mathbf {x}=[\underline{x},\overline{x}]\). Furthermore, the width of \(\mathbf {x}\) is defined as \({\text {wid}}(\mathbf {x}) := \overline{x}-\underline{x}\), the radius of \(\mathbf {x}\) as \({\text {rad}}(\mathbf {x}) := \tfrac{1}{2}{\text {wid}}(\mathbf {x})\), the magnitude of \(\mathbf {x}\) as \(|\mathbf {x}| := \max (\underline{x},\overline{x})\), and the mignitude of \(\mathbf {x}\) as \(\langle \mathbf {x}\rangle := \min \{|x|\mid x\in \mathbf {y}\}\). For \(\mathbf {x}\) bounded we set the midpoint of \(\mathbf {x}\) as \(\check{x} := \tfrac{1}{2}(\underline{x}+\overline{x})\). We define the elementary operations for interval arithmetic by the rule \(\mathbf {x}\diamond \mathbf {y}= \Box \{x \diamond y \mid x \in \mathbf {x}, y \in \mathbf {y}\}, \forall \diamond \in \{+, -, \times , \div , \hat{}~\}\), where \(\Box S\) denotes the smallest interval containing the set S. (The symbol ‘\(\Box \)’ is a box, and it refers to the tightest interval hull, also called as box hull.) Thus, the ranges of the five elementary interval arithmetic operations are exactly the ranges of their real-valued counterparts. Although this rule characterizes these operations mathematically, the usefulness of interval arithmetic is due to the operational definitions based on interval bounds Hickey et al. (2001). For example, let \(\mathbf {x}= [\underline{x}, \overline{x}]\) and \(\mathbf {y}= [\underline{y}, \overline{y}]\), it can be easily proved that
$$\begin{aligned} \mathbf {x}+ \mathbf {y}= & {} [\underline{x}+\underline{y},\overline{x}+\overline{y}],\\ \mathbf {x}- \mathbf {y}= & {} [\underline{x}-\overline{y},\overline{x}-\underline{y}],\\ \mathbf {x}\times \mathbf {y}= & {} [\min \{\underline{x}\underline{y}, \underline{x}\overline{y}, \overline{x}\underline{y}, \overline{x}\overline{y} \}, \max \{\underline{x}\underline{y}, \underline{x}\overline{y}, \overline{x}\underline{y}, \overline{x}\overline{y} \}],\\ \mathbf {x}\div \mathbf {y}= & {} \mathbf {x}\times 1/\mathbf {y} if 0 \notin \mathbf {y}, where 1/\mathbf {y}= [1/\overline{y}, 1/\underline{y}],\\ \mathbf {x}^{\mathbf {y}}= & {} [\min \{\underline{x}^{\underline{y}}, \underline{x}^{\overline{y}}, \overline{x}^{\underline{y}}, \overline{x}^{\overline{y}} \}, \max \{\underline{x}^{\underline{y}}, \underline{x}^{\overline{y}}, \overline{x}^{\underline{y}}, \overline{x}^{\overline{y}} \}],~~\underline{y}>0, ~~\underline{x}\ge 0 . \end{aligned}$$
In addition, for a function \(\varphi :\mathbb {R}\rightarrow \mathbb {R}\) and an interval \(\mathbf {x}\) we define
$$\begin{aligned} \varphi (\mathbf {x}) := \Box \{\varphi (x)\mid x\in \mathbf {x}\}. \end{aligned}$$
Moreover, if a function f composed of these elementary arithmetic operations and elementary functions \(\varphi \in \{\sin ,\cos ,\exp ,\log ,\ldots \}\), i.e., factorable function, is given, bounds on the range of f can be obtained by replacing the real arithmetic operations and the real functions by their corresponding interval arithmetic counterparts.
The finite nature of computers precludes an exact representation of the reals. In practice, the real set, \(\mathbb {R}\), is approximated by a finite set \(\bar{\mathbb {F}} = \mathbb {F}\cup \{-\infty , +\infty \}\), where \(\mathbb {F}\) is the set of floating-point numbers. The set of real intervals is then approximated by the set \(\mathbb {I}\) of intervals with bounds in \(\bar{\mathbb {F}}\). The power of interval arithmetic lies in its implementation on computers. In particular, outwardly rounded interval arithmetic allows rigorous enclosures for the ranges of operations and functions. This makes a qualitative difference in scientific computations, since the results are now intervals in which the exact result is guaranteed to lie. Interval arithmetic can be carried out for virtually every expression that can be evaluated with floating-point arithmetic. However, two important points have to be considered: Interval arithmetic is only subdistributive, so expressions that are equivalent in real arithmetic differ in interval arithmetic, giving different amounts of overestimation (the amount by which the real range of the function over an interval and the result computed by interval arithmetic differ). Therefore, computations should be arranged so that overestimation of ranges is minimized. Readers are referred to Alefeld and Herzberger (1983), Neumaier (1990), Hickey et al. (2001), Jaulin et al. (2001) for more details on basic interval methods.
Let \(f:\mathbf {x}\rightarrow \mathbb {R}\) be continuously differentiable, and assume the existence of \(\hat{x}\in \mathbf {x}\) with \(f(\hat{x}) = 0\), and let \(\tilde{x}\in \mathbf {x}\). Then by the mean value theorem we get
$$\begin{aligned} f(\hat{x}) = 0 = f(\tilde{x}) + f'(\xi )(\hat{x}-\tilde{x}), \end{aligned}$$
for some \(\xi \in \mathbf {x}\). Therefore,
$$\begin{aligned} \hat{x} = \tilde{x} - \frac{f(\tilde{x})}{f'(\xi )}. \end{aligned}$$
Now let \(\mathfrak f'\) be an interval extension of \(f'\) and \(\mathfrak f\) be an interval extension of f, i.e. a specific expression for computing an enclosure for the range of f over a given input interval \(\mathbf {x}\). Then by the properties of interval arithmetic we get
$$\begin{aligned} \hat{x} \in \tilde{x} - \frac{\mathfrak f(\tilde{x})}{\mathfrak f'(\mathbf {x})} =: N(\mathfrak f,\mathfrak f';\mathbf {x},\tilde{x}). \end{aligned}$$
The operator N is called univariate interval Newton operator. Using this operator we can define the interval Newton iteration as
$$\begin{aligned} \mathbf {x}^{(k+1)} = \mathbf {x}^{(k)}\cap N(\mathfrak f,\mathfrak f';\mathbf {x}^{(k)},\check{x}^{(k)}), \end{aligned}$$
(2)
starting with \(\mathbf {x}^{(0)} = \mathbf {x}\). This iteration has the properties that whenever \(\mathbf {x}^{(k)} = \emptyset \) for some k, then \(\mathbf {x}\) does not contain a zero of f. Otherwise \(\hat{x}\in \mathbf {x}^{(k)}\) for all k, and \({\text {wid}}(\mathbf {x}^{(k+1)}) = O({\text {wid}}(\mathbf {x}^{(k)})^2)\) locally under mild assumptions on f and \(\mathbf {x}^{(0)}\). Furthermore, if for any k we find that \(\mathbf {x}^{(k+1)} \subseteq \text {int}(\mathbf {x}^{(k)})\), i.e., that the interval Newton operator maps the box \(\mathbf {x}^{(k)}\) into its interior, then \(\mathbf {x}^{(k)}\) contains a unique zero of f.
The interval Newton operator requires an interval extension \(\mathfrak {f}'\) of the derivative of f. For every factorable function f such an extension can be constructed using algorithmic differentiation techniques, e.g., see Berz et al. (1996), Griewank and Corliss (1991), Griewank and Walther (2008). For univariate functions the most efficient approach is via the algebra of differential numbers \(\mathcal {D}_1:=\mathbb {R}\times \mathbb {R}\), equipped with the following basic operations: Let \(df:=(f,f'), dg:=(g,g')\in \mathcal {D}_1\), and \(\varphi :\mathbb {R}\rightarrow \mathbb {R}\). Define
$$\begin{aligned} \begin{aligned} df\pm dg&:= (f\pm g,f'\pm g'),\\ df\cdot dg&:= (f\cdot g, f'\cdot g + f\cdot g'),\\ df/dg&:= (f/g, (f'\cdot g - f\cdot g')/g^2),\quad g\ne 0,\\ df^{dg}&:= (f^g, f^g\cdot (g'\cdot \log (f)+g\cdot f'/f)),\quad f>0,\\ \varphi (df)&:= (\varphi (f),\varphi '(f)\cdot f'). \end{aligned} \end{aligned}$$
(3)
The set of real numbers is embedded in \(\mathcal {D}_1\) by \(r\mapsto (r,0)\).
If \(\tilde{f}(x)\) is an expression representing f using arithmetic operations and elementary functions, we can use \(\tilde{f}\) to calculate \((y,y') = \tilde{f}((x,1))\) on \(\mathcal {D}_1\) by replacing the operations and elementary functions in \(\tilde{f}\) by their counterparts on \(\mathcal {D}_1\), and then \(y' = f'(x)\).
This approach can be generalized to compute an interval extension \(\mathfrak {f}'\) of \(f'\) by defining the algebra of interval differential numbers \(\mathbb {I}\mathcal {D}_1:=\mathbb {IR}\times \mathbb {IR}\) and introducing again the operations (3) on \(\mathbb {I}\mathcal {D}_1\) now using interval arithmetic operations in the components of the interval differential numbers. Using this algebra and an expression \(\tilde{f}\) for f, we get by computing \((\mathbf {y},\mathbf {y}') = \tilde{f}((\mathbf {x},1))\) an enclosure \(\mathbf {y}'\supseteq f'(\mathbf {x})\) and thereby an interval extension \(\mathfrak {f}'\) of \(f'\).