Numerical Thinking in Algorithm Design and Analysis

  • Shang-Hua TengEmail author


To me, numerical analysis is one of the most fascinating fields in computing. It is at the intersection of computer science and mathematics; it concerns subjects that can be either continuous or discrete; it involves algorithm design as well as software implementation; and it has success throughout engineering, business, medicine, social sciences, natural sciences, and digital animation fields. Its community has both theorists and practitioners, who often respect and admire each other’s work – in fact, in this area there are many practitioners who are also great theoreticians. Its objectives to solve larger and larger problems have pushed the envelope of computer science, particularly in the advancement of computer architectures, compiler technologies, programming languages, and software tools. Its collaborative culture and genuine need to share data and information among scientists and engineers have led a physicist to make a connection between the hypertext idea and the Internet protocols to create the world wide web. Many pioneers in computing including John von Neumann, Alan Turing, Claude Shannon, Richard Hamming, James Wilkinson, Velvel Kahan, and Gene Golub contributed to this field, not to mention the foot prints left by many great minds before them, e.g., Newton’s method, Lagrange interpolation, Gaussian elimination, Euler’s method, Jacobi iteration, and Chebyshev polynomials.


Delaunay Triangulation Gaussian Elimination Laplacian Matrix Simplex Algorithm Weighted Point 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. I. Abraham, Y. Bartal, O. Neiman. Nearly Tight Low Stretch Spanning Trees. FOCS 2008, Pages 781-790.Google Scholar
  2. N. Alon, R. M. Karp, D. Peleg, and D. West. A graph-theoretic game and its application to the k-server problem. SIAM Journal on Computing, 24(1):78–100, February 1995.CrossRefzbMATHMathSciNetGoogle Scholar
  3. C. J. Alpert and S.-Z. Yao. Spectral partitioning: the more eigenvectors, the better. In DAC ’95: Proceedings of the 32nd ACM/IEEE conference on Design automation, pages 195–200. ACM, 1995.Google Scholar
  4. R. Andersen, C. Borgs, J. T. Chayes, J. E. Hopcroft, K. Jain, and V. S. Mirrokni, and S.-H. Teng. Robust PageRank and locally computable spam detection features. In Fourth International Workshop on Adversarial Information Retrieval on the Web, ACM International Conference Proceeding Series, 2008 69–76.Google Scholar
  5. R. Andersen, C. Borgs, J. T. Chayes, J. E. Hopcraft, V. S. Mirrokni, and S.-H. Teng. Local computation of pagerank contributions. In Anthony Bonato and Fan R. K. Chung, editors, WAW, volume 4863 of Lecture Notes in Computer Science, pages 150–165. Springer, 2007.Google Scholar
  6. R. Andersen, F. Chung, and K. Lang. Local graph partitioning using pagerank vectors. Proceedings: 47th Annual Symposium on Foundations of Computer Science, pages 475–486, 2006.Google Scholar
  7. R. Andersen, Y. Peres. Finding sparse cuts locally using evolving sets. STOC, 2009: 235–244.Google Scholar
  8. S. Arora. Polynomial time approximation schemes for euclidean traveling salesman and other geometric problems. J. ACM, 45(5):753–782, 1998.CrossRefzbMATHMathSciNetGoogle Scholar
  9. S. Arora, E. Hazan, and S. Kale. 0(sqrt (log n)) approximation to Sparsest Cut in õ(n2) time. In IEEE FOCS’ 04, pages 238–247, 2004.Google Scholar
  10. S. Arora and S. Kale. A combinatorial, primal-dual approach to semidefinite programs. In ACM STOC ’07, pages 227–236, 2007.Google Scholar
  11. S. Arora, S. Rao, and U. Vazirani. Expander flows, geometric embeddings and graph partitioning. In ACM STOC ’04, pages 222–231, 2004.Google Scholar
  12. D. Arthur and S. Vassilvitskii. Worst-case and smoothed analysis of the icp algorithm, with an application to the k-means method. In IEEE FOCS ’06, pages 153–164, 2006.Google Scholar
  13. D. Arthur and B. Manthey and H. Röglin. k-Means Has Polynomial Smoothed Complexity. In IEEE FOCS, pages 405–414, 2009.Google Scholar
  14. Nina Balcan and Avrim Blum and Anupam Gupta Approximate clustering without the approximation. SIAM/ACM SODA 2009: 1068–1077.Google Scholar
  15. J. Batson, D. A. Spielman, and N. Srivastava. Twice-Ramanujan sparsifiers. Available at, 2008.
  16. M. Bebendorf and W. Hackbusch. Existence of H-matrix approximants to the inverse FE-matrix of elliptic operators with L -coefficients. Numerische Mathematik, 95(1):1–28, July 2003.CrossRefzbMATHMathSciNetGoogle Scholar
  17. A. A. Benczúr and D. R. Karger. Approximating s-t minimum cuts in O(n2) time. In ACM STOC ’96,Google Scholar
  18. M. Bern, J.Gilbert, B. Hendrickson, N. Nguyen, and S. Toledo. Support-graph preconditioners. In SIAM. J. Matrix Anal. and Appl. 27(4), 930–951, 2006.Google Scholar
  19. Avrim Blum and John Dunagan. Smoothed analysis of the perceptron algorithm for linear programming. In SIAM/ACM SODA ’02, pages 905–914, 2002.Google Scholar
  20. E. G. Boman and B. Hendrickson. Support theory for preconditioning. SIAM Journal on Matrix Analysis and Applications, 25(3):694–717, 2003.CrossRefzbMATHMathSciNetGoogle Scholar
  21. E. G. Boman, B. Hendrickson, S. Vavasis. Solving epplitic finite element systems in nearly-linear time with support preconditioners.Google Scholar
  22. W. L. Briggs, V. E. Henson, and S. F. McCormick. A Multigrid Tutorial, 2nd Edition. SIAM, 2001.Google Scholar
  23. S. Brin and L. Page, The anatomy of a large-scale hypertextual Web search engine, Proceedings of the seventh international conference on World Wide Web 7, 107–117, 1998.Google Scholar
  24. J. Cheeger. A lower bound for smallest eigenvalue of laplacian. In Problems in Analysis, pages 195–199, In R.C. Gunning editor,, Princeton University Press, 1970.Google Scholar
  25. X. Chen, X. Deng, and S.-H. Teng. Settling the complexity of computing two-player Nash equilibria. In J. ACM, 56(3): (2009a)Google Scholar
  26. X. Chen, D. Dai, Y. Du, and S.-H. Teng. Settling the Complexity of Arrow-Debreu Equilibria in Markets with Additively Separable Utilities. In IEEE FOCS, 273–282, 2009b.Google Scholar
  27. Siu-Wing Cheng, Tamal K. Dey, Herbert Edelsbrunner, Michael A. Facello, and Shang-Hua Teng. Sliver exudation. Journal of ACM, 47:883–904, 2000.CrossRefMathSciNetGoogle Scholar
  28. P Chew. There is a planar graph almost as good as the complete graph. In SCG ’86: Proceedings of the second annual symposium on Computational geometry, pages 169–177. ACM, 1986.Google Scholar
  29. P. Christiano, J. A. Kelner, A. Madry, D. A. Spielman, and S.-H. Teng. Electrical Flows, Laplacian Systems, and Faster Approximation of Maximum Flow in Undirected Graphs
  30. T. Cormen. C. Leiserson, R. Rivest, and C. Stein. Introduction to Algorithms, 3rd edition.Google Scholar
  31. S. I. Daitch and D. A. Spielman. Support-graph preconditioners for 2-dimensional trusses.Google Scholar
  32. S. I. Daitch and D. A. Spielman. Faster approximate lossy generalized flow via interior point algorithms. In ACM STOC’08, pages 451–460, 2008.Google Scholar
  33. Valentina Damerow, Friedhelm Meyer auf der Heide, Harald Räcke, Christian Scheideler, and Christian Sohler. Smoothed motion complexity. In Proc. 11th Annual European Symposium on Algorithms (ESA’03), pages 161–171, 2003.Google Scholar
  34. I. Daubechies, Ten Lectures on Wavelets, Society for Industrial and Applied Mathematics, 1992Google Scholar
  35. P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin, and M. Sagar Acquiring the Reflectance Field of a Human Face SIGGRAPH Conference Proceedings, 2000Google Scholar
  36. S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman. Indexing by latent semantic analysis Journal of the American Society for Information Science, 41 (6), 391407, 1990.Google Scholar
  37. James Demmel. Applied Numerical Linear Algebra. SIAM, 1997.Google Scholar
  38. Jian Ding, James R. Lee, and Yuval Peres. Cover times, blanket times, and majorizing measures. ACM STOC’ 11, 2011.Google Scholar
  39. W. E. Donath and A. J. Hoffman. Algorithms for partitioning of graphs and computer logic based on eigenvectors of connection matrices. IBM Technical Disclosure Bulletin, 15:938–944, 1972.Google Scholar
  40. D. L. Donoho, Compressed Sensing, IEEE Transactions on Information Theory, 52(4), 12891306, 2006.Google Scholar
  41. S. Dughmi and T. Roughgarden Black-Box Randomized Reductions in Algorithmic Mechanism Design. IEEE FOCS 2010: 775–784Google Scholar
  42. Herbert Edelsbrunner, Xiang-Yang Li, Gary Miller, Andreas Stathopoulos, Dafna Talmor, Shang-Hua Teng, Alper Üngör, and Noel Walkington. Smoothing and cleaning up slivers. In STOC ’00, pages 273–277, 2000.Google Scholar
  43. M. Elkin, Y. Emek, D. A. Spielman, and S.-H. Teng. Lower-stretch spanning trees. SIAM Journal on Computing, 32(2):608–628, 2008.CrossRefMathSciNetGoogle Scholar
  44. Matthias Englert, Heiko Röglin, and Berthold Vöcking. Worst case and probabilistic analysis of the 2-opt algorithm for the tsp: extended abstract. In SIAM/ACM SODA ’07, pages 1295–1304, 2007.Google Scholar
  45. S. Even and R. E. Tarjan. Network flow and testing graph connectivity. SIAM Journal on Computing, 4(4):507–518, Dec. 1975.CrossRefzbMATHMathSciNetGoogle Scholar
  46. M. Fiedler. Algebraic connectibity of graphs. Czechoslovak Mathematical Journal, 23(98):298–305, 1973.MathSciNetGoogle Scholar
  47. A. M. Frieze, G. L. Miller, and S.-H. Teng. Separator Based Parallel Divide and Conquer in Computational Geometry. ACM SPAA 1992: 420-429Google Scholar
  48. J. A. George. Nested dissection of a regular finite element mesh. SIAM J. Numer. Anal., 10:345–363, 1973.CrossRefzbMATHMathSciNetGoogle Scholar
  49. J. R. Gilbert and R. E. Tarjan. The analysis of a nested dissection algorithm. Numerische Mathematik, 50(4):377–404, February 1987.CrossRefzbMATHMathSciNetGoogle Scholar
  50. A. V. Goldberg and S. Rao. Beyond the ow decomposition barrier. J. ACM, 45(5):783797, 1998.Google Scholar
  51. G. H. Golub and M. Overton. The convergence of inexact Chebychev and Richardson iterative methods for solving linear systems. Numerische Mathematik, 53:571–594, 1988.CrossRefzbMATHMathSciNetGoogle Scholar
  52. G. H. Golub and C. F. Van Loan. Matrix Computations. second edition, 1989.Google Scholar
  53. K. Gremban. Combinatorial Preconditioners for Sparse, Symmetric, Diagonall y Dominant Linear Systems. PhD thesis, Carnegie Mellon University, CMU-CS-96-123, 1996.Google Scholar
  54. A. Gulli and A. Signorini. The indexable web is more than 11.5 billion pages. In WWW ’05: Special interest tracks and posters of the 14th international conference on World Wide Web, pages 902–903. ACM, 2005.Google Scholar
  55. Li-Sha Huang and Shang-Hua Teng. On the approximation and smoothed complexity of leontief market equilibria. In FAW: Frontiers of Algorithms Workshop, pages 96–107, 2007.Google Scholar
  56. A. Joshi. Topics in Optimization and Sparse Linear Systems, Ph.D. thesis, UIUC, 1997.Google Scholar
  57. A. T. Kalai and A. Samorodnitsky and S.-H. Teng Learning and Smoothed Analysis. In IEEE FOCS’09, 395–404, 2009.Google Scholar
  58. J. A. Kelner and E. Nikolova. On the hardness and smoothed complexity of quasi-concave minimization. In IEEE FOCS’07, pages 552–563, 2007.Google Scholar
  59. J. A. Kelner and D A. Spielman. A randomized polynomial-time simplex algorithm for linear programming. In ACM STOC ’06, pages 51–60, 2006.Google Scholar
  60. R. Khandekar, S. Rao, and U. Vazirani. Graph partitioning using single commodity flows. In ACM STOC ’06, pages 385–390, 2006.Google Scholar
  61. J. Kleinberg. Authoritative sources in a hyperlinked environment. J. ACM, 46 (5), 1999, 604–632.CrossRefzbMATHMathSciNetGoogle Scholar
  62. A. Kolla, Y. Makarychev, A. Saberi, and S.-H. Teng Subgraph Sparsification. ACM STOC’10 2010.Google Scholar
  63. I. Koutis. G. Miller, and R. Peng. Solving SDD linear systems in time Õ(mlognlog(1 / ε)). arXiv:1102.4842.Google Scholar
  64. I. Koutis. G. Miller, and D. Tolliver. Combinatorial preconditioners and multilevel solvers for problems in computer vision and image prcessing. International Symp. of Visual Computing, 1067–1078, 2009.Google Scholar
  65. C.E. Lemke. Bimatrix equilibrium points and mathematical programming. Management Science, 11:681–689, 1965.CrossRefMathSciNetGoogle Scholar
  66. C.E. Lemke and JR. J.T. Howson. Equilibrium points of bimatrix games. J. Soc. Indust. Appl. Math., 12:413–423, 1964.Google Scholar
  67. X.-Y. Li and S.-H. Teng. Generate sliver free three dimensional meshes. In ACM-SIAM SODA’01, pages 28–37, 2001.Google Scholar
  68. R. J. Lipton, D. J. Rose, and R. E. Tarjan. Generalized nested dissection. SIAM Journal on Numerical Analysis, 16(2):346–358, April 1979.CrossRefzbMATHMathSciNetGoogle Scholar
  69. T. Leighton and S. Rao. Multicommodity max-flow min-cut theorems and their use in designing approximation algorithms. Journal of the ACM, 46(6):787–832, November 1999.CrossRefzbMATHMathSciNetGoogle Scholar
  70. L. Lovasz and M. Simonovits. Random walks in a convex body and an improved volume algorithm. RSA: Random Structures & Algorithms, 4:359–412, 1993.CrossRefzbMATHMathSciNetGoogle Scholar
  71. B. Maggs, G. Miller, O. Parekh, R. Ravi, and S. M. Woo. Finding effective support-tree preconditioners. ACM SPAA 176–185, 2005.Google Scholar
  72. A. Madry and J. Kelner Faster generation of random spanning trees. IEEE FOCS’09, 2009.Google Scholar
  73. Gary L. Miller, Dafna Talmor, Shang-Hua Teng, and Noel Walkington. A delaunay based numerical method for three dimensions: generation, formulation, and partition. In ACM STOC ’95, pages 683–692, 1995.Google Scholar
  74. G. L. Miller, S.-H. Teng, W. P. Thurston, and S. A. Vavasis. Separators for sphere-packings and nearest neighbor graphs. J. ACM, 44(1): 1–29 (1997)CrossRefzbMATHMathSciNetGoogle Scholar
  75. G. L. Miller, S.-H. Teng, W. P. Thurston, and S. A. Vavasis. Geometric Separators for Finite-Element Meshes. SIAM J. Scientific Computing, 19(2): 364–386 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  76. Joseph S. B. Mitchell. Guillotine subdivisions approximate polygonal subdivisions: A simple polynomial-time approximation scheme for geometric tsp, k-mst, and related problems. SIAM J. Comput., 28(4):1298–1309, 1999.Google Scholar
  77. Michael Mitzenmacher and Salil P. Vadhan. Why simple hash functions work: exploiting the entropy in a data stream. SIAM-ACM SODA 2008:746–755.Google Scholar
  78. J. Nash. Noncooperative games. Annals of Mathematics, 54:289–295, 1951,.CrossRefMathSciNetGoogle Scholar
  79. E. Nikolova, J. A. Kelner, M. Brand, and M. Mitzenmacher. Stochastic shortest paths via quasi-convex maximization. In ESA’06: Proceedings of the 14th conference on Annual European Symposium, pages 552–563, 2006.Google Scholar
  80. L. Orecchia, L. J. Schulman, U. V. Vazirani, and N. K. Vishnoi. On partitioning graphs via single commodity flows. In ACM STOC ’08, pages 461–470, 2008.Google Scholar
  81. J. Rief. Efficient approximate solution of sparse linear systems. Computer and Mathematics with Applications, 36 (9): 37–58, 1998.CrossRefGoogle Scholar
  82. H. Röglin and S.-H. Teng. Smoothed analysis of multiobjective optimization. IEEE FOCS, 681–690, 2009.Google Scholar
  83. H. Röglin and B. Vöcking. Smoothed analysis of integer programming. Math. Program., 110(1):21–56, 2007.CrossRefMathSciNetGoogle Scholar
  84. M. Rudelson and R. Vershynin. The littlewood-offord problem and invertibility of random matrices. UC Davis, 2006.Google Scholar
  85. Arvind Sankar, Daniel A. Spielman, and Shang-Hua Teng. Smoothed analysis of the condition numbers and growth factors of matrices. SIAM Journal on Matrix Analysis and Applications, to appear, 2005.Google Scholar
  86. Guido Schäfer, Luca Becchetti, Stefano Leonardi, Alberto Marchetti-Spaccamela, and Tjark Vredeveld. Average case and smoothed competitive analysis of the multi-level feedback algorithm. In IEEE FOCS ’03, page 462, 2003.Google Scholar
  87. A. Sharma, X. Liu, P. Miller, A. Nakano, R. K. Kalia, P. Vashishta, W. Zhao, T. J. Campbell, and A. Haas. Immersive and interactive exploration of billion-atom systems. In VR ’02: Proceedings of the IEEE Virtual Reality Conference 2002, page 217. IEEE Computer Society, 2002.Google Scholar
  88. J. R. Shewchuk. Tetrahedral mesh generation by Delaunay refinement. In Proc. 14th Annual ACM Symposium on Computational Geometry, pages 86–95, 1998.Google Scholar
  89. G. Shklarski and S. Toledo. Rigidity in finite-element matrices: Sufficient conditions for the rigidity of structures and substructures. SIAM J. Matrix Anal. and Appl. 30(1): 7–40, 2008.CrossRefzbMATHMathSciNetGoogle Scholar
  90. D. Spielman. Graphs and networks: Random walks and spectral graph drawing. Computer Science, Yale,, Sept. 18, 2007.
  91. D. A. Spielman and N. Srivastava. Graph sparsification by effective resistances. In ACM STOC, pages 563–568, 2008.Google Scholar
  92. D. Spielman and S.-H. Teng. Spectral partitioning works: planar graphs and finite element meshes. In IEEE FOCS ’96, pages 96–105, 1996.Google Scholar
  93. Daniel A. Spielman and Shang-Hua Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. J. ACM, 51(3):385–463, 2004.Google Scholar
  94. Daniel A. Spielman and Shang-Hua Teng. Smoothed Analysis: An attempt to explain the behavior of algorithms in practice. CACM, 52(10):77–84, 2009.Google Scholar
  95. D. A. Spielman and S.-H. Teng. Nearly-linear time algorithms for graph partitioning, graph sparsification, and solving linear systems. ACM STOC, 81–90, 2003.Google Scholar
  96. D. A. Spielman and S.-H. Teng. A local clustering algorithm for massive graphs and its application to nearly-linear time graph partitioning. CoRR, abs/0809.3232, 2008. Available at
  97. D. A. Spielman and S.-H. Teng. Nearly-linear time algorithms for preconditioning and solving symmetric, diagonally dominant linear systems. CoRR, abs/cs/0607105, 2008. Available at
  98. D. A. Spielman and S.-H. Teng. Spectral sparsification of graphs. CoRR, abs/0808.4134, 2008. Available at
  99. G. Strang. Linear Algebra and its Application, 2nd. Ed. Academic Press, New York, San Francisco, London, 1980.Google Scholar
  100. D. A. Tolliver. Spectral rounding and image segmentation. PhD thesis, Pittsburgh, PA, USA, 2006. Adviser-Miller,, Gary L. and Adviser-Collins,, Robert T.Google Scholar
  101. D. A. Tolliver and G. L. Miller. Graph partitioning by spectral rounding: Applications in image segmentation and clustering. In CVPR ’06: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1053–1060. IEEE Computer Society, 2006.Google Scholar
  102. L. N. Trefethen and D. Bau. Numerical Linear Algebra. SIAM, Philadelphia, PA, 1997.CrossRefzbMATHGoogle Scholar
  103. P. Vaidya. Solving linear equations with symmetric diagonally dominant matrices by constructing good preconditioners. An invited at IMA, U. Minnesota, Oct. 1991.Google Scholar
  104. R. Vershynin. Beyond hirsch conjecture: Walks on random polytopes and smoothed complexity of the simplex method. In IEEE FOCS ’06, pages 133–142, 2006.Google Scholar
  105. Van H. Vu and Terence Tao. The condition number of a randomly perturbed matrix. In ACM STOC ’07, pages 248–255, 2007.Google Scholar
  106. J. H. Wilkinson. Error analysis of direct methods of matrix inversion. J. Assoc. Comput. Mach., 8:261–330, 1961.MathSciNetGoogle Scholar
  107. D. Zhou, J. Huang and B. Schlkopf. Learning from Labeled and Unlabeled Data on a Directed Graph. the 22nd International Conference on Machine Learning, 1041–1048. 2005.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Department of Computer Science, Viterbi School of EngineeringUniversity of Southern CaliforniaLos AngelesUSA

Personalised recommendations