Abstract
This paper offers a review of numerical methods for computation of the eigenvalues of Hermitian matrices and the singular values of general and some classes of structured matrices. The focus is on the main principles behind the methods that guarantee high accuracy even in the cases that are ill-conditioned for the conventional methods. First, it is shown that a particular structure of the errors in a finite precision implementation of an algorithm allows for a much better measure of sensitivity and that computation with high accuracy is possible despite a large classical condition number. Such structured errors incurred by finite precision computation are in some algorithms e.g. entry-wise or column-wise small, which is much better than the usually considered errors that are in general small only when measured in the Frobenius matrix norm. Specially tailored perturbation theory for such structured perturbations of Hermitian matrices guarantees much better bounds for the relative errors in the computed eigenvalues. Secondly, we review an unconventional approach to accurate computation of the singular values and eigenvalues of some notoriously ill-conditioned structured matrices, such as e.g. Cauchy, Vandermonde and Hankel matrices. The distinctive feature of accurate algorithms is using the intrinsic parameters that define such matrices to obtain a non-orthogonal factorization, such as the LDU factorization, and then computing the singular values of the product of thus computed factors. The state of the art software is discussed as well.
Similar content being viewed by others
Notes
Here the real case is cited for the sake of simplicity. An analogous analysis applies to the complex Hermitian case.
Real matrices are used only for the sake of simplicity of the presentation.
Here the matrix absolute value is defined element-wise.
It is this mixing of large and small columns by orthogonal transformations oblivious of the difference in length that destroys the accuracy of the bidiagonalizaton. For illustrating examples see [44].
We can also consider column-wise backward errors as in Sect. 4.2, but this involves the behavior of scaled condition numbers of the iterates.
Alternatively, we may assume that X and Y are already well conditioned (thus properly scaled) and that the computed matrices satisfy \( \Vert \delta X\Vert _2 \le \epsilon _1 \Vert X\Vert _2,\quad \Vert \delta Y\Vert _2 \le \epsilon _2 \Vert Y\Vert _2,\quad |\delta D_{ii}| \le \epsilon _3 |D_{ii}|,\quad i=1,\ldots , p. \)
The theory in [107] has been developed for Hermitian pencils \(H-\lambda M\) with positive definite M. Here we take \(M=I\) for the sake of simplicity.
This term is used for typical errors occurring in the finite precision (computer) floating point arithmetic.
Recall the example of the SVD of Vandermonde matrices in Sect. 4.6.3.
References
Alfa, A.S., Xue, J., Ye, Q.: Accurate computation of the smallest eigenvalue of a diagonally dominant \(M\)-matrix. Math. Comput. 71(237), 217–236 (2002)
Anderson, E., Bai, Z., Bischof, C., Blackford, L.S., Demmel, J., Dongarra, Jack J., Du Croz, J., Hammarling, S., Greenbaum, A., McKenney, A., Sorensen, D.: LAPACK Users’ Guide, 3rd edn. Society for Industrial and Applied Mathematics, Philadelphia (1999)
Argyris, J.H.: The natural factor formulation of the stiffnesses for the matrix displacement method. Comput. Methods Appl. Mech. Eng. 5, 97–119 (1975)
Barlow, J.: More accurate bidiagonal reduction for computing the singular value decomposition. SIAM J. Matrix Anal. Appl. 23, 761–798 (2002)
Barlow, J., Demmel, J.: Computing accurate eigensystems of scaled diagonally dominant matrices. SIAM J. Numer. Anal. 27(3), 762–791 (1990)
Bhatia, R.: Matrix Analysis. Graduate Texts in Mathematics, 169. Springer, Berlin (1997)
Björck, Å.: Numerical Methods in Matrix Computations. Springer, Berlin (2015)
Bujanović, Z., Drmač, Z.: How a numerical rank revealing instability affects computer aided control system design. Technical report, SLICOT Working Note 2010-1 (WGS/NICONET Reports) (2010)
Bujanović, Z., Drmač, Z.: New robust ScaLAPACK routine for computing the QR factorization with column pivoting. arXiv e-prints. arXiv:1910.05623 (2019)
Bunch, J.R., Parlett, B.N.: Direct methods for solving symmetric indefinite systems of linear equations. SIAM J. Numer. Anal. 8(4), 639–655 (1971)
Bürgisser, P., Cucker, F.: Condition: The Geometry of Numerical Algorithms. Springer Publishing Company, Incorporated, New York (2013)
Businger, P.A., Golub, G.H.: Linear least squares solutions by Householder transformations. Numer. Math. 7, 269–276 (1965)
Chu, M.T.: The generalized Toda flow, the QR algorithm and the center manifold theory. SIAM J. Algebraic Discret. Methods 5(2), 187–201 (1984)
de Rijk, P.P.M.: A one-sided Jacobi algorithm for computing the singular value decomposition on a vector computer. SIAM J. Sci. Stat. Comput. 10(2), 359–371 (1989)
Demmel, J.: On floating point errors in Cholesky. LAPACK working note 14, Computer Science Department, University of Tennessee (1989)
Demmel, J.: The inherent inaccuracy of implicit tridiagonal QR. Preprint series 963, IMA, University of Minnesota, Minneapolis (1992)
Demmel, J.: Accurate singular value decompositions of structured matrices. SIAM J. Matrix Anal. Appl. 21(2), 562–580 (1999)
Demmel, J., Dumitriu, I., Holtz, O.: Toward accurate polynomial evaluation in rounded arithmetic. In: Pardo, L., et al. (eds.) Foundations of Computational Mathematics, pp. 36–105. Cambridge University Press, Cambridge (2006)
Demmel, J., Dumitriu, I., Holtz, O., Koev, P.: Accurate and efficient expression evaluation and linear algebra. Acta Numer. 17, 87–145 (2008)
Demmel, J., Gragg, W.: On computing accurate singular values and eigenvalues of acyclic matrices. Linear Algebra Appl. 185, 203–218 (1993)
Demmel, J., Gu, M., Eisenstat, S., Slapničar, I., Veselić, K., Drmač, Z.: Computing the singular value decomposition with high relative accuracy. Linear Algebra Appl. 299, 21–80 (1999)
Demmel, J., Kahan, W.: Accurate singular values of bidiagonal matrices. SIAM J. Sci. Stat. Comput. 11(5), 873–912 (1990)
Demmel, J., Koev, P.: Accurate SVDs of weakly diagonally dominant M-matrices. Numer. Math. 98, 99–104 (2004)
Demmel, J., Koev, P.: Accurate SVDs of polynomial Vandermonde matrices involving orthonormal polynomials. Linear Algebra Appl. 417(2), 382–396 (2006)
Demmel, J., Veselić, K.: Jacobi’s method is more accurate than QR. SIAM J. Matrix Anal. Appl. 13(4), 1204–1245 (1992)
Dhillon, I.S., Parlett, B.N.: Multiple representations to compute orthogonal eigenvectors of symmetric tridiagonal matrices. Linear Algebra Appl. 387, 1–28 (2004)
Dhillon, I.S., Parlett, B.N.: Orthogonal eigenvectors and relative gaps. SIAM J. Matrix Anal. Appl. 25(3), 858–899 (2004)
Dhillon, I.S., Parlett, B.N., Vömel, C.: The design and implementation of the MRRR algorithm. ACM Trans. Math. Softw. 32(4), 533–560 (2006)
Dongarra, J., Moler, C., Bunch, J., Stewart, G.: LINPACK Users’ Guide. Society for Industrial and Applied Mathematics, Philadelphia (1979)
Dopico, F.M., Koev, P.: Accurate symmetric rank revealing and eigendecompositions of symmetric structured matrices. SIAM J. Matrix Anal. Appl. 28(4), 1126–1156 (2006)
Dopico, F.M., Koev, P.: Perturbation theory for the LDU factorization and accurate computations for diagonally dominant matrices. Numer. Math. 119, 337 (2011)
Dopico, F.M., Koev, P., Molera, J.M.: Implicit standard Jacobi gives high relative accuracy. Numer. Math. 113(4), 519–553 (2009)
Dopico, F.M., Molera, J.M., Moro, J.: An orthogonal high relative accuracy algorithm for the symmetric eigenproblem. SIAM J. Matrix Anal. Appl. 25(2), 301–351 (2003)
Dopico, F.M., Moro, J.: A note on multiplicative backward errors of accurate SVD algorithms. SIAM J. Matrix Anal. Appl. 25(4), 1021–1031 (2004)
Dopico, F.M., Moro, J., Molera, J.M.: Weyl-type relative perturbation bounds for eigensystems of Hermitian matrices. Linear Algebra Appl. 309, 3–18 (2000)
Drmač, Z.: Computing the singular and the generalized singular values. Ph.D. thesis, Lehrgebiet Mathematische Physik, Fernuniversität Hagen, Germany (1994)
Drmač, Z.: On the condition behaviour in the Jacobi method. SIAM J. Matrix Anal. Appl. 17(3), 509–514 (1996)
Drmač, Z.: Implementation of Jacobi rotations for accurate singular value computation in floating point arithmetic. SIAM J. Sci. Comput. 18, 1200–1222 (1997)
Drmač, Z.: Accurate computation of the product induced singular value decomposition with applications. SIAM J. Numer. Anal. 35(5), 1969–1994 (1998)
Drmač, Z.: A posteriori computation of the singular vectors in a preconditioned Jacobi SVD algorithm. IMA J. Numer. Anal. 19, 191–213 (1999)
Drmač, Z.: New accurate algorithms for singular value decomposition of matrix triplets. SIAM J. Matrix Anal. Appl. 21(3), 1026–1050 (2000)
Drmač, Z.: A global convergence proof for cyclic Jacobi methods with block rotations. SIAM J. Matrix Anal. Appl. 31(3), 1329–1350 (2010)
Drmač, Z.: SVD of Hankel matrices in Vandermonde–Cauchy product form. Electron. Trans. Numer. Anal. 44, 593–623 (2015)
Drmač, Z.: Algorithm 977: a QR-preconditioned QR SVD method for computing the SVD with high accuracy. ACM Trans. Math. Softw. 44(1), 11:1–11:30 (2017)
Drmač, Z., Bujanović, Z.: On the failure of rank-revealing QR factorization software—a case study. ACM Trans. Math. Softw. 35(2), 1–28 (2008)
Drmač, Z., Hari, V.: On the quadratic convergence of the \(J\)-symmetric Jacobi method. Numer. Math. 64, 147–180 (1993)
Drmač, Z., Veselić, K.: New fast and accurate Jacobi SVD algorithm: I. SIAM J. Matrix Anal. Appl. 29(4), 1322–1342 (2008)
Drmač, Z., Veselić, K.: New fast and accurate Jacobi SVD algorithm: II. SIAM J. Matrix Anal. Appl. 29(4), 1343–1362 (2008)
Eisenstat, S., Ipsen, I.: Relative perturbation techniques for singular value problems. SIAM J. Numer. Anal. 32(6), 1972–1988 (1995)
Fernando, K.V., Parlett, B.N.: Implicit Cholesky algorithms for singular values and vectors of triangular matrices. Numer. Linear Algebra Appl. 2(6), 507–531 (1995)
Forsythe, G.E., Henrici, P.: The cyclic Jacobi method for computing the principal values of a complex matrix. Trans. Am. Math. Soc. 94(1), 1–23 (1960)
Gasca, M., Pena, J.M.: On Factorizations of Totally Positive Matrices, pp. 109–130. Springer Netherlands, Dordrecht (1996)
Goldstine, H.H., Murray, H.H., von Neumann, J.: The Jacobi method for real symmetric matrices. J. Assoc. Comp. Mach.6, 59–96 (1959) [also in J. von Neumann, Collected Works, vol. V, pages 573-610, Pergamon Press, New York, 1973]
Golub, G., Klema, V., Stewart, G.W.: Rank degeneracy and least squares problems. Technical report STAN-CS-76-559, Computer Science Department, Stanford University (1976)
Golub, G.H., Kahan, W.: Calculating the singular values and pseudo-inverse of a matrix. SIAM J. Numer. Anal. 2(2), 205–224 (1965)
Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. The Johns Hopkins University Press, Baltimore (2013)
Gonnet, P., Pachón, R., Trefethen, L.N.: Robust rational interpolation and least-squares. Electron. Trans. Numer. Anal. 38, 146–167 (2011)
Großer, B., Lang, B.: An \(O(n^2)\) algorithm for the bidiagonal SVD. Linear Algebra Appl. 358(1), 45–70 (2003)
Gu, M., Eisenstat, S.: A divide-and-conquer algorithm for the bidiagonal SVD. SIAM J. Matrix Anal. Appl. 16, 79–92 (1995)
Gu, M., Eisenstat, S.: An efficient algorithm for computing a strong rank-revealing QR factorization. SIAM J. Sci. Comput. 17(4), 848–869 (1996)
Gu, M., Eisenstat, S.C.: A divide-and-conquer algorithm for the symmetric tridiagonal eigenproblem. SIAM J. Matrix Anal. Appl. 16(1), 172–191 (1995)
Gutknecht, M.H., Trefethen, L.N.: Real polynomial Chebyshev approximation by the Carathéodory–Fejér method. SIAM J. Numer. Anal. 19(2), 358–371 (1982)
Hammarling, S.: Numerical solution of the stable, non-negative definite Lyapunov equation. IMA J. Numer. Anal. 2, 303–323 (1982)
Hari, V.: On sharp quadratic convergence bounds for the serial Jacobi methods. Numer. Math. 60, 375–406 (1991)
Haut, T., Beylkin, G., Monzón, L.: Solving Burgers’ equation using optimal rational approximations. Appl. Comput. Harmon. Anal. 34(1), 83–95 (2013)
Haut, T.S., Beylkin, G.: Fast and accurate con-eigenvalue algorithm for optimal rational approximations. SIAM J. Matrix Anal. Appl. 33(4), 1101–1125 (2012)
Heath, M.T., Laub, A.J., Paige, C.C., Ward, R.C.: Computing the singular value decomposition of a product of two matrices. SIAM J. Sci. Stat. Comput. 7, 1147–1159 (1986)
Hestenes, M.R.: Inversion of matrices by biorthogonalization and related results. J. SIAM 6(1), 51–90 (1958)
Higham, N.J.: Accuracy and Stability of Numerical Algorithms, 2nd edn. Society for Industrial and Applied Mathematics, Philadelphia (2002)
Higham, N.J.: \(J\)-orthogonal matrices: properties and generation. SIAM Rev. 45(3), 504–519 (2003)
Hogben, L.: Handbook of Linear Algebra, 2nd edn. CRC Press, Taylor & Francis Group, Boca Raton, FL (2014)
Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (1990)
Ipsen, I.C.F.: Relative perturbation results for matrix eigenvalues and singular values. Acta Numer. 7, 151–201 (1998)
Jacobi, C.G.J.: Über ein leichtes Verfahren die in der Theorie der Säcularstörungen vorkommenden Gleichungen numerisch aufzulösen. Crelle’s Journal für reine und angew Math. 30, 51–95 (1846)
Kahan, W.: How futile are mindless assessments of roundoff in floating-point computation? Technical report, Department of Mathematics, University of California at Berkeley (2006)
Kleinsteuber, M.: A sort-Jacobi algorithm for semisimple Lie algebras. Linear Algebra Appl. 430(1), 155–173 (2009)
Kleinsteuber, M., Helmke, U., Huper, K.: Jacobi’s algorithm on compact Lie algebras. SIAM J. Matrix Anal. Appl. 26(1), 42–69 (2004)
Koev, P.: Accurate eigenvalues and SVDs of totally nonnegative matrices. SIAM J. Matrix Anal. Appl. 27(1), 1–23 (2005)
Koev, P.: Accurate computations with totally nonnegative matrices. SIAM J. Matrix Ana. Appl. 29(3), 731–751 (2007)
Li, Ren-Cang: Relative perturbation theory: I. Eigenvalue and singular value variations. SIAM J. Matrix Anal. Appl. 19(4), 956–982 (1998)
Li, Ren-Cang: Relative perturbation theory: II. Eigenspace and singular subspace variations. SIAM J. Matrix Anal. Appl. 20(2), 471–492 (1998)
Van Loan, ChF: Generalizing the singular value decomposition. SIAM J. Numer. Anal. 13(1), 76–83 (1976)
Mascarenhas, W.F.: A note on Jacobi being more accurate than QR. SIAM J. Matrix Anal. Appl. 15(1), 215–218 (1993)
Mathias, R., Stewart, G.W.: A block QR algorithm for singular value decomposition. Linear Algebra Appl. 182, 91–100 (1993)
Ostrowski, A.M.: A quantitative formulation of Sylvester’s law of inertia. Proc. Natl. Acad. Sci. (USA) 45, 740–744 (1959)
Paige, C.C.: Properties of numerical algorithms related to computing controllability. IEEE Trans. Autom. Control 26(1), 130–138 (1981)
Parlett, B.N.: The Symmetric Eigenvalue Problem, Classics In Applied Mathematics 20. SIAM, Philadelphia (1998)
Parlett, B.N., Dhillon, I.S.: Relatively robust representations of symmetric tridiagonals. Linear Algebra Appl. 309(1), 121–151 (2000)
Peláez, M., Moro, J.: Accurate factorization and eigenvalue algorithms for symmetric DSTU and TSC matrices. SIAM J. Matrix Anal. Appl. 28(4), 1173–1198 (2006)
Pietzsch, E.: Genaue Eigenwertberechnung nichtsingulärer schiefsymmetrischer Matrizen mit einem Jacobi–änlichen Verfahren. Ph.D. thesis, Lehrgebiet Mathematische Physik, Fernuniversität Hagen, Germany (1993)
Rice, J.H.: A theory of condition. SIAM J. Numer. Anal. 3(2), 287–310 (1966)
Rutishauser, H.: The Jacobi method for real symmetric matrices. Numer. Math. 9, 1–10 (1966)
Slapničar, I.: Accurate symmetric eigenreduction by a Jacobi method. Ph.D. thesis, Lehrgebiet Mathematische Physik, Fernuniversität Hagen, Germany (1992)
Slapničar, I.: Componentwise analysis of direct factorization of real symmetric and Hermitian matrices. Linear Algebra Appl. 272, 227–275 (1998)
Slapničar, I.: Highly accurate symmetric eigenvalue decomposition and hyperbolic SVD. Linear Algebra Appl. 358, 387–424 (2002)
Slapničar, I., Truhar, N.: Relative perturbation theory for hyperbolic eigenvalue problem. Linear Algebra Appl. 309(1), 57–72 (2000)
Smoktunowicz, Alicja: A note on the strong componentwise stability of algorithms for solving symmetric linear systems. Demonstr. Math. 28(2), 443–448 (1995)
Sorensen, D., Zhou, Y.: Direct methods for matrix Sylvester and Lyapunov equations. J. Appl. Math. 2003(6), 277–303 (2003)
Stewart, G.W.: QR sometimes beats Jacobi. Technical report TR-95-32, Department of Computer Science and Institute for Advanced Computer Studies, University of Maryland, College Park, MD 20742 (1995)
Stewart, G.W.: The QLP approximation to the singular value decomposition. Technical report TR-97-75, Department of Computer Science and Institute for Advanced Computer Studies, University of Maryland, College Park, MD 20742 (1997)
Stewart, G.W., Sun, J.-G.: Matrix Perturbation Theory. Academic Press, New York (1990)
van der Sluis, A.: Condition numbers and equilibration of matrices. Numer. Math. 14, 14–23 (1969)
Vavasis, S.A.: Stable finite elements for problems with wild coefficients. SIAM J. Numer. Anal. 33(3), 890–916 (1996)
Veselić, K.: A Jacobi eigenreduction algorithm for definite matrix pairs. Numer. Math. 64, 241–269 (1993)
Veselić, K.: Perturbation theory for the eigenvalues of factorised symmetric matrices. Linear Algebra Appl. 309, 85–102 (2000)
Veselić, K., Hari, V.: A note on a one-sided Jacobi algorithm. Numer. Math. 56, 627–633 (1989)
Veselić, K., Slapničar, I.: Floating-point perturbations of Hermitian matrices. Linear Algebra Appl. 195, 81–116 (1993)
Watkins, D.S.: Isospectral flows. SIAM Rev. 26(3), 379–391 (1984)
Wilkinson, J.H.: Rounding Errors in Algebraic Processes. Prentice-Hall, Inc., New York (1963)
Wilkinson, J.H.: The Algebraic Eigenvalue Problem. Springer, Berlin (1965)
Ye, Q.: Computing singular values of diagonally dominant matrices to high relative accuracy. Math. Comput. 77, 2195–2230 (2008)
Acknowledgements
The author wishes to thank Jesse Barlow, Jim Demmel, Froilán Martínez Dopico, Vjeran Hari, Plamen Koev, Juan Manuel Molera Molera, Eberhard Pietzsch, Ivan Slapničar, Ninoslav Truhar, Krešimir Veselić, for numerous exciting discussions on accurate matrix computations, and in particular to Julio Moro Carreño for encouragement to write this paper and for many useful comments that improved the presentation. Funding was provided by Hrvatska Zaklada za Znanost (IP-11-2013-9345, IP-2019-04-6268).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Drmač, Z. Numerical methods for accurate computation of the eigenvalues of Hermitian matrices and the singular values of general matrices. SeMA 78, 53–92 (2021). https://doi.org/10.1007/s40324-020-00229-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40324-020-00229-8
Keywords
- Backward error
- Condition number
- Eigenvalues
- Hermitian matrices
- Jacobi method
- LAPACK
- Perturbation theory
- Rank revealing decomposition
- Singular value decomposition