Solving Systems of Linear Equations

Abstract

In this chapter we present algorithms and error analysis of numerical methods for solving linear systems $Ax=b$ with nonsingular square matrices. Here we present only direct methods. They are called direct because in the absence of rounding errors they would give the exact solution of $Ax = b$ after a finite number of steps.

The original version of this chapter was revised. An Erratum to this book can be found at https://doi.org/10.1007/978-3-319-57304-5_8

Questions

8.1

(Programming)

Solve in MATLAB$^\circledR$ Poisson’s equation (8.11) on the unit square $\{(x_1, x_2):0 \le x_1, x_2 \le 1 \}$ with boundary conditions $u=0$ on the boundary of this square and with function $f(x_1,x_2)$ given by (8.81) using the lu function and programs of Section 1.1. Compare the obtained results with those of Section 8.4.4. Optional: extend these results to three-dimensional case.

8.2

(Programming)

Improve Algorithm 8.1, overwriting L and U on A. Write your own MATLAB$^\circledR$ program and test it on your own examples.

8.3

(Programming)

Apply the bisection algorithm (see Algorithm 8.11 below) to find the roots of the polynomial $p(x)=(x-2)^9$ and of some of your own polynomials, where p(x) is evaluated using Horner’s rule (Algorithm 8.4). Write your own MATLAB$^\circledR$ program. Confirm that changing the input interval for $x=[x_{left}, x_{right}]$ slightly changes the computed root drastically. Modify the algorithm to use the relative condition number for polynomial evaluation (8.26) to stop bisecting when the rounding error in the computed value of p(x) gets so large that its sign cannot be determined. Present your results similarly to the results of Fig. 8.2.

Hint: use the MATLAB$^\circledR$ function coeffs to compute the coefficients of a polynomial p(x).

Here we present the bisection algorithm to find roots of a polynomial p(x). Suppose that the input interval for x where we want to find roots of $p(x)=0$ is $x \in [x_{left}, x_{right}]$. At every iteration, this algorithm divides the input interval in two by computing the midpoint $x_{middle} = ( x_{left}+ x_{right}) / 2$ of the input interval as well as the value of the polynomial $p(x_{middle})$ at that point. We will compute the value of the polynomial $p(x_{middle})$ using Horner’s rule (Algorithm 8.4). Then if $p_{left}$ and $p_{mid}$ have opposite signs, the bisection algorithm sets $x_{middle}$ as the new value for $x_{right}$, and if $p_{right}$ and $p_{mid}$ have opposite signs, then the method sets $x_{middle}$ as the new $x_{left}$. If $p(x_{middle})=0$, then $x_{middle}$ may be taken as the root of the polynomial and the algorithm stops.

Algorithm 8.11

Bisection algorithm to find zeros of the polynomial p(x).

• 0. Initialization: set the left $x_{left}$ and the right $x_{right}$ bounds for the input interval for $x \in [x_{left}, x_{right}]$ where we will seek roots of the polynomial. Set the computational tolerance tol.

• 1. Evaluate the polynomial p(x) at the points $x_{left}$ and $x_{right}$ to get $p_{left}=p(x_{left})$ and $p_{right}=p(x_{right})$ using Algorithm 8.4.

  Perform steps 2–3 while $x_{right} - x_{left} > 2 tol$

• 2. Compute point $x_{mid} = {(x_{left} + x_{right})}/{2}$ and then $p_{mid}=p(x_{mid})$, using Algorithm 8.4.

• 3. Check:

  If $p_{left} p_{mid} < 0$, then we have a root in the interval $[x_{left}, x_{mid}]$. Assign $x_{right} = x_{mid}$ and $p_{right} = p_{mid}$.

  Else if $p_{right} p_{mid} < 0$, then we have a root in the interval $[x_{mid}, x_{right}]$. Assign $x_{left} = x_{mid}$ and $p_{left} = p_{mid}$.

  Else we have found a root at $x_{mid}$ and assign $x_{left} = x_{mid}, x_{right} = x_{mid}$.

• 4. Compute the root as ${(x_{left} + x_{right})}/{2}$.

8.4

Write Algorithm 8.1 for the case $n=3$. Using Algorithm 8.1, perform LU factorization of the matrix

$$\begin{aligned} A= \left( \begin{array}{ccc} 4 &{} 1 &{} 1\\ 1 &{} 8 &{} 1 \\ 1 &{} 1 &{} 16 \end{array} \right) . \end{aligned}$$

8.5

Using Cholesky’s algorithm (Algorithm 8.10), perform the factorization $A = L L^T$ of the matrix A in Question 8.4.

8.6

(Programming)

Implement Hager’s algorithm, Algorithm 8.7, in MATLAB$^\circledR$. Test it on different matrices. Take, for example, $A=\text {hilb}(N)$ or $A= \text {rand}(N, N)$ for different N.

8.7

Let us consider the solution of the linear system $A X=B$, where A is an $n\times n$ matrix, B is an $n\times m$ matrix, and X is an unknown $n\times m$ matrix. We have two methods to solve it:

• 1. Factorization of $A=PLU$ and then using the algorithms of forward and backward substitution, Algorithms 8.2 and 8.3 to find every column of X.

• 2. Computation of $A^{-1}$ by Gaussian elimination and then finding of $X = A^{-1} B$.

Count the number of FLOPS required for every algorithm. Show that the first algorithm requires fewer FLOPS than the second one.

Hint: use material of Section 8.2.2.

8.8

Derive formula (8.58) for the operation count in Cholesky decomposition. Hint: use the formula

$$\sum _{i=1}^{n} i^2 = \frac{(n+1)n(2n+1)}{6}$$

and the formula for the sum of an arithmetic progression,

$$\sum _{k=1}^{n} a_i = \frac{n(a_1 + a_n)}{2}.$$

8.9

Let A be an s.p.d. matrix. Show that $|a_{ij}| < \sqrt{a_{ii} a_{jj}}$.

8.10

Suppose A is an invertible nonsingular square matrix of order n and that u, v are vectors. Suppose furthermore that $1 + v^T A^{-1}u \ne 0$. Prove the Sherman–Morrison formula

$$\begin{aligned} (A+uv^T)^{-1} = A^{-1} - {A^{-1}uv^T A^{-1} \over 1 + v^T A^{-1}u}. \end{aligned}$$

Here, $uv^T$ is the outer product of two vectors u and v.

8.11

Suppose A is an invertible square matrix of order n and that U, V are $n\times k$ rectangular matrices with $k \le n$. Prove the Sherman–Morrison–Woodburg formula, which states that $T= I + V^T A^{-1} U$ is nonsingular if and only if $A + UV^T$ is nonsingular and

$$\begin{aligned} (A+ U V^T)^{-1} = A^{-1} - A^{-1} U T^{-1} V^T A^{-1}. \end{aligned}$$

8.12

(Programming)

Similarly, with MATLAB$^\circledR$ programs of Sections  1.1, 1.2, solve the three-dimensional problem¹⁴

$$\begin{aligned} \begin{aligned} - \triangle u(x)&= f(x)~ ~ \text{ in }~~ \varOmega , \\ u&= 0 ~\text{ on }~ \partial \varOmega \end{aligned} \end{aligned}$$

(8.82)

on the unit cube $\varOmega =[0,1] \times [0,1]\times [0,1]$. Choose an appropriate function f(x).

Hint. We discretize the unit cube $\varOmega$ with ${x_1}_i = i h_1, {x_2}_j = j h_2, {x_3}_k = k h_3$, where

$$ h_1 = \frac{1}{n_i -1},\quad h_2 = \frac{1}{n_j -1},\quad h_3 = \frac{1}{n_k -1} $$

are the steps of the discrete finite difference mesh and $n_i, n_j, n_k$ are the numbers of discretization points in the directions $x_1, x_2, x_3$, respectively. The indices (i, j, k) are such that $0 \le i < n_i$, $0 \le j < n_j$, $0 \le j < n_k$. The global node numbers $n_{glob}$ in the three-dimensional case can be computed as

$$\begin{aligned} \begin{aligned} n_{glob}&= j + n_j \left( (i-1) + n_i (k-1) \right) . \end{aligned} \end{aligned}$$

(8.83)

We take $n_i = n_j = n_k = n = N+2$, $h_1 = h_2 = h_3 = 1/(n -1) = 1/(N+1)$ and obtain the following scheme for the solution of Poisson’s equation (8.11) in three dimensions:

$$\begin{aligned} \begin{aligned} - \frac{u_{i+1,j, k} - 2 u_{i,j, k} + u_{i-1,j, k}}{ h_1^2 }&- \frac{u_{i, j+1,k} - 2 u_{i,j,k} + u_{i, j-1,k}}{ h_2^2} \\&- \frac{u_{i,j, k+1} - 2 u_{i,j,k} + u_{i,j, k-1}}{ h_3^2} = \frac{f_{i,j,k}}{ a_{i,j, k}}, \end{aligned} \end{aligned}$$

(8.84)

where $u_{i,j,k}, f_{i,j,k}, a_{i,j, k}$ are values of u, f, a, respectively, at the discrete point $n_{glob}$ with indices (i, j, k). We rewrite Eq. (8.84) with $h= h_1=h_2 = h_3$ as

$$\begin{aligned} 6 u_{i,j, k} - u_{i+1,j, k} - u_{i-1,j,k}- u_{i, j+1,k} - u_{i, j-1,k} - u_{i,j, k+1} - u_{i,j, k-1} = h^2 \frac{f_{i,j,k}}{a_{i,j, k}}. \end{aligned}$$

(8.85)

Again, we recognize that the scheme (8.85) is a system of linear equations $A u = b$. The matrix A is of size $(n_i-2)(n_j -2)(n_k-2)=N^3$, and on the unit cube it is given by the block matrix

$$ A =\left( \begin{array}{c|ccc|c} A_N &{} -I_N &{} O_N &{} -I_N &{} \ddots \\ \hline -I_N &{} A_N &{} -I_N &{} \ddots &{} \ddots \\ \ddots &{} \ddots &{} \ddots &{} \ddots &{} \ddots \\ -I_N &{} \ddots &{} -I_N &{} A_N &{} -I_N \\ \hline \ddots &{} -I_N &{} O_N &{} -I_N &{} A_N \end{array} \right) $$

with zero blocks $O_N$ of order N. The blocks $A_N$ of size $N\times N$ on the main diagonal of this matrix are given by

$$ A_N =\left( \begin{array}{cccccc} 6 &{} -1 &{} 0 &{} \cdots &{} \cdots &{} 0 \\ -1 &{} 6 &{} -1 &{} 0 &{} \cdots &{} 0 \\ 0 &{} -1 &{} 6 &{} 0 &{} \cdots &{} 0 \\ \cdots &{} \cdots &{} \cdots &{} \cdots &{} \cdots &{} \cdots \\ 0 &{} \cdots &{} \cdots &{} 0 &{} -1 &{} 6 \end{array} \right) . $$

8.13

(Programming)

Use the MATLAB$^\circledR$ programs of Sections  1.1, 1.2 and solve the problem in Example 8.2 on an L-shaped 2D domain.¹⁵

Fig. 8.5

Eigenvalues in Question 8.14 computed and estimated by the Gershgorin disk theorem. The computed eigenvalues of A are $\lambda _1 =9.8218, \lambda _2 = 8.1478, \lambda _3= 1.8995, \lambda _4 = -10.86.$

8.14

(Programming)

Use formula (8.80) to estimate the eigenvalues of the matrix

$$\begin{aligned} A = \left( \begin{array}{cccc} 10 &{} -1 &{} 0 &{} 1\\ 0.2 &{} 8 &{} 0.2 &{} 0.2\\ 1 &{} 1 &{} 2 &{} 1\\ -1 &{} -1 &{} -1 &{} -11\\ \end{array} \right) . \end{aligned}$$

(8.86)

Write your own MATLAB$^\circledR$ program to present results similar to those in Fig. 8.5, which shows Gershgorin disks together with their centers and the computed eigenvalues $\lambda _i$. Hint: use the MATLAB$^\circledR$ function eigs(A) to compute eigenvalues.

8.15

(Programming)

Use formula (8.80) to estimate the eigenvalues of the matrix

$$\begin{aligned} A = \left( \begin{array}{cccc} 7 &{} 5 &{} 2 &{} 1 \\ 2 &{} 8 &{} 3 &{} 2\\ 1 &{} 1 &{} 5 &{} 1 \\ 1 &{} 1 &{} 1 &{} 6\\ \end{array} \right) . \end{aligned}$$

(8.87)

Write your own MATLAB$^\circledR$ program to present results similar to those in Fig. 8.6, which shows Gershgorin disks together with their centers and the computed eigenvalues $\lambda _i$. Hint: use the MATLAB$^\circledR$ function eigs(A) to compute eigenvalues.

Fig. 8.6

Eigenvalues in Question 8.15 computed and estimated by the Gershgorin disk theorem. The computed eigenvalues are $\lambda _1 = 12.2249 + 0.0000i$, $\lambda _2 = 4.4977 + 0.6132i$, $\lambda _3 = 4.4977 - 0.6132i$, $\lambda _4 = 4.7797 +0.0000i.$

8.16

Prove that the matrix A in the numerical examples of Sections 8.1.3 and 8.4.4 is s.p.d. Hint: prove that for all $u,\, v\in {\mathbb {R}}^{(N+2)^2}$ that equal zero on the boundary of the mesh domain, we can write

$$ (Au,v)=\sum \limits _{i, j=1}^{N-1}(u_{i+1,j}-u_{i, j})(v_{i+1,j}-v_{i,j})+ \sum \limits _{i, j=1}^{N-1}(u_{i, j+1}-u_{i,j})(v_{i, j+1}-v_{i, j}). $$

8.17

(Programming)

Write your own MATLAB$^\circledR$ program and solve numerically the problem of Example 8.5 via the solution of the system of linear equations $Ay=b$ with b, A given in (8.78), (8.79), respectively. Use the recurrence formulas (8.66), (8.71) to obtain the solution of this system.