Algorithms: A Survey

  • Luc Pronzato
  • Andrej Pázman
Part of the Lecture Notes in Statistics book series (LNS, volume 212)


We consider the maximization of a design criterion ϕ( ⋅) with respect to ξΞ, the set of probability measures on \(\mathcal{X}\) compact.


Support Point Bundle Method Exact Design Ellipsoid Method Discrete Probability Measure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Ahipasaoglu, S., P. Sun, and M. Todd (2008). Linear convergence of a modified Frank-Wolfe algorithm for computing minimum volume enclosing ellipsoids. Optim. Methods Softw. 23, 5–19.MathSciNetzbMATHCrossRefGoogle Scholar
  2. Atwood, C. (1973). Sequences converging to D-optimal designs of experiments. Ann. Statist. 1(2), 342–352.MathSciNetzbMATHCrossRefGoogle Scholar
  3. Atwood, C. (1976). Convergent design sequences for sufficiently regular optimality criteria. Ann. Statist. 4(6), 1124–1138.MathSciNetzbMATHCrossRefGoogle Scholar
  4. Atwood, C. (1980). Convergent design sequences for sufficiently regular optimality criteria, II singular case. Ann. Statist. 8(4), 894–912.MathSciNetzbMATHCrossRefGoogle Scholar
  5. Avriel, M. (2003). Nonlinear Programming. Analysis and Methods. New York: Dover. [Originally published by Prentice Hall, 1976].Google Scholar
  6. Ben-Tal, A. and A. Nemirovskii (2001). Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications. Philadelphia: MPS/SIAM Series on Optim. 2.Google Scholar
  7. Birgin, E., J. Martínez, and M. Raydan (2000). Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim. 10(4), 1196–1211.MathSciNetzbMATHCrossRefGoogle Scholar
  8. Bland, R., D. Goldfarb, and M. Todd (1981). The ellipsoid method: a survey. Oper. Res. 29(6), 1039–1091.MathSciNetzbMATHCrossRefGoogle Scholar
  9. Bohachevsky, I., M. Johnson, and M. Stein (1986). Generalized simulated annealing for function optimization. Technometrics 28(3), 209–217.zbMATHCrossRefGoogle Scholar
  10. Böhning, D. (1985). Numerical estimation of a probability measure. J. Stat. Plann. Inference 11, 57–69.zbMATHCrossRefGoogle Scholar
  11. Böhning, D. (1986). A vertex-exchange-method in D-optimal design theory. Metrika 33, 337–347.MathSciNetzbMATHCrossRefGoogle Scholar
  12. Boissonnat, J.-D. and M. Yvinec (1998). Algorithmic Geometry. Cambridge Univ. Press.zbMATHCrossRefGoogle Scholar
  13. Bonnans, J., J. Gilbert, C. Lemaréchal, and C. Sagastizábal (2006). Numerical Optimization. Theoretical and Practical Aspects. Heidelberg: Springer. [2nd ed.].Google Scholar
  14. Boyd, S. and L. Vandenberghe (2004). Convex Optimization. Cambridge: Cambridge Univ. Press.zbMATHGoogle Scholar
  15. Calamai, P. and J. Moré (1987). Projected gradient methods for linearly constrained problems. Math. Programming 39, 93–116.MathSciNetzbMATHCrossRefGoogle Scholar
  16. Cook, R. and C. Nachtsheim (1980). A comparison of algorithms for constructing exact D-optimal designs. Technometrics 22(3), 315–324.zbMATHCrossRefGoogle Scholar
  17. Cormen, T., C. Leiserson, R. Rivest, and C. Stein (2001). Introduction to Algorithms. MIT Press and McGraw-Hill. [2nd ed.].Google Scholar
  18. Correa, R. and C. Lemaréchal (1993). Convergence of some algorithms for convex minimization. Math. Programming 62, 261–275.MathSciNetzbMATHCrossRefGoogle Scholar
  19. Dai, Y.-H. and R. Fletcher (2006). New algorithms for singly linearly constrained quadratic programs subject to lower and upper bounds. Math. Programming A 106, 403–421.MathSciNetzbMATHCrossRefGoogle Scholar
  20. Dem’yanov, V. and V. Malozemov (1974). Introduction to Minimax. New York: Dover.Google Scholar
  21. den Boeff, E. and D. den Hertog (2007). Efficient line search methods for convex functions. SIAM J. Optim. 18(1), 338–363.MathSciNetCrossRefGoogle Scholar
  22. den Hertog, D. (1994). Interior Point Approach to Linear, Quadratic and Convex Programming. Dordrecht: Kluwer.zbMATHCrossRefGoogle Scholar
  23. Dette, H., A. Pepelyshev, and A. Zhigljavsky (2008). Improving updating rules in multiplicative algorithms for computing D-optimal designs. J. Stat. Plann. Inference 53, 312–320.MathSciNetzbMATHGoogle Scholar
  24. Ecker, J. and M. Kupferschmid (1983). An ellipsoid algorithm for nonlinear programming. Math. Programming 27, 83–106.MathSciNetzbMATHCrossRefGoogle Scholar
  25. Ecker, J. and M. Kupferschmid (1985). A computational comparison of the ellipsoidal algorithm with several nonlinear programming algorithms. SIAM J. Control Optim. 23(5), 657–674.MathSciNetzbMATHCrossRefGoogle Scholar
  26. Ermoliev, Y. and R.-B. Wets (Eds.) (1988). Numerical Techniques for Stochastic Optimization Problems. Berlin: Springer.Google Scholar
  27. Fedorov, V. (1971). The design of experiments in the multiresponse case. Theory Probab. Appl. 16(2), 323–332.zbMATHCrossRefGoogle Scholar
  28. Fedorov, V. (1972). Theory of Optimal Experiments. New York: Academic Press.Google Scholar
  29. Fellman, J. (1974). On the allocation of linear observations. Comment. Phys. Math. 44, 27–78.MathSciNetGoogle Scholar
  30. Fellman, J. (1989). An empirical study of a class of iterative searches for optimal designs. J. Stat. Plann. Inference 21, 85–92.CrossRefGoogle Scholar
  31. Frank, M. and P. Wolfe (1956). An algorithm for quadratic programming. Naval Res. Logist. Quart. 3, 95–110.MathSciNetCrossRefGoogle Scholar
  32. Galil, Z. and J. Kiefer (1980). Time- and space-saving computer methods, related to Mitchell’s DETMAX, for finding D-optimum designs. Technometrics 22(3), 301–313.MathSciNetzbMATHCrossRefGoogle Scholar
  33. Gauchi, J.-P. and A. Pázman (2006). Designs in nonlinear regression by stochastic minimization of functionals of the mean square error matrix. J. Stat. Plann. Inference 136, 1135–1152.zbMATHCrossRefGoogle Scholar
  34. Goffin, J.-L. and K. Kiwiel (1999). Convergence of a simple subgradient level method. Math. Programming 85, 207–211.MathSciNetzbMATHCrossRefGoogle Scholar
  35. Goodwin, G. and R. Payne (1977). Dynamic System Identification: Experiment Design and Data Analysis. New York: Academic Press.zbMATHGoogle Scholar
  36. Grötschel, M., L. Lovász, and A. Schrijver (1980). Geometric Algorithms and Combinatorial Optimization. Berlin: Springer.Google Scholar
  37. Harman, R. and L. Pronzato (2007). Improvements on removing non-optimal support points in D-optimum design algorithms. Statist. Probab. Lett. 77, 90–94.MathSciNetzbMATHCrossRefGoogle Scholar
  38. Hearn, D. and S. Lawphongpanich (1989). Lagrangian dual ascent by generalized linear programming. Oper. Research Lett. 8, 189–196.MathSciNetzbMATHCrossRefGoogle Scholar
  39. Hiriart-Urruty, J. and C. Lemaréchal (1993). Convex Analysis and Minimization Algorithms, part 1 and 2. Berlin: Springer.Google Scholar
  40. Johnson, M. and C. Nachtsheim (1983). Some guidelines for constructing exact D-optimal designs on convex design spaces. Technometrics 25, 271–277.MathSciNetzbMATHGoogle Scholar
  41. Kelley, J. (1960). The cutting plane method for solving convex programs. SIAM J. 8, 703–712.MathSciNetGoogle Scholar
  42. Khachiyan, L. (1979). A polynomial algorithm in linear programming. Doklady Akademïa Nauk SSSR 244, 1093–1096. [English transl. Soviet Math. Doklady, 20, 191–194].Google Scholar
  43. Khachiyan, L. (1996). Rounding of polytopes in the real number model of computation. Math. Oper. Res. 21(2), 307–320.MathSciNetzbMATHCrossRefGoogle Scholar
  44. Khachiyan, L. and M. Todd (1993). On the complexity of approximating the maximal inscribed ellipsoid for a polytope. Math. Programming A61(2), 137–159.Google Scholar
  45. Kushner, H. and D. Clark (1978). Stochastic Approximation Methods for Constrained and Unconstrained Systems. Heidelberg: Springer.CrossRefGoogle Scholar
  46. Kushner, H. and J. Yang (1993). Stochastic approximation with averaging of the iterates: optimal asymptotic rate of convergence for general processes. SIAM J. Control Optim. 31(4), 1045–1062.MathSciNetzbMATHCrossRefGoogle Scholar
  47. Kushner, H. and G. Yin (1997). Stochastic Approximation Algorithms and Applications. Heidelberg: Springer.zbMATHGoogle Scholar
  48. Lemaréchal, C., A. Nemirovskii, and Y. Nesterov (1995). New variants of bundle methods. Math. Programming 69(1), 111–147.MathSciNetzbMATHCrossRefGoogle Scholar
  49. Levin, A. (1965). On an algorithm for the minimization of convex functions. Soviet Math. Dokl. 6, 286–290.Google Scholar
  50. Ljung, L. (1987). System Identification, Theory for the User. Englewood Cliffs: Prentice Hall.zbMATHGoogle Scholar
  51. McCormick, G. and R. Tapia (1972). The gradient projection method under mild differentiability conditions. SIAM J. Control 10(1), 93–98.MathSciNetzbMATHCrossRefGoogle Scholar
  52. Minoux, M. (1983). Programmation Mathématique, Théorie et Algorithmes, vol. 1 & 2. Paris: Dunod.Google Scholar
  53. Molchanov, I. and S. Zuyev (2001). Variational calculus in the space of measures and optimal design. In A. Atkinson, B. Bogacka, and A. Zhigljavsky (Eds.), Optimum Design 2000, Chapter 8, pp. 79–90. Dordrecht: Kluwer.Google Scholar
  54. Molchanov, I. and S. Zuyev (2002). Steepest descent algorithm in a space of measures. Stat. Comput. 12, 115–123.MathSciNetCrossRefGoogle Scholar
  55. Nesterov, Y. (1995). Complexity estimates of some cutting plane methods based on the analytic center. Math. Programming 69, 149–176.MathSciNetzbMATHGoogle Scholar
  56. Nesterov, Y. (2004). Introductory Lectures to Convex Optimization: A Basic Course. Dordrecht: Kluwer.Google Scholar
  57. Nesterov, Y. and A. Nemirovskii (1994). Interior-Point Polynomial Algorithms in Convex Programming. Philadelphia: SIAM.zbMATHCrossRefGoogle Scholar
  58. Pázman, A. (1986). Foundations of Optimum Experimental Design. Dordrecht (co-pub. VEDA, Bratislava): Reidel (Kluwer group).Google Scholar
  59. Pázman, A. and L. Pronzato (1992). Nonlinear experimental design based on the distribution of estimators. J. Stat. Plann. Inference 33, 385–402.zbMATHCrossRefGoogle Scholar
  60. Polak, E. (1971). Computational Methods in Optimization, a Unified Approach. New York: Academic Press.Google Scholar
  61. Polyak, B. (1987). Introduction to Optimization. New York: Optimization Software.Google Scholar
  62. Polyak, B. (1990). New stochastic approximation type procedures. Automat. i Telemekh. 7, 98–107.MathSciNetGoogle Scholar
  63. Polyak, B. and A. Juditsky (1992). Acceleration of stochastic approximation by averaging. SIAM J. Control Optim. 30, 838–855.MathSciNetzbMATHCrossRefGoogle Scholar
  64. Pronzato, L. and E. Walter (1985). Robust experiment design via stochastic approximation. Math. Biosci. 75, 103–120.MathSciNetzbMATHCrossRefGoogle Scholar
  65. Pronzato, L. and E. Walter (1988). Robust experiment design via maximin optimization. Math. Biosci. 89, 161–176.MathSciNetzbMATHCrossRefGoogle Scholar
  66. Pronzato, L., H. Wynn, and A. Zhigljavsky (2000). Dynamical Search. Boca Raton: Chapman & Hall/CRC.zbMATHGoogle Scholar
  67. Pukelsheim, F. and S. Reider (1992). Efficient rounding of approximate designs. Biometrika 79(4), 763–770.MathSciNetCrossRefGoogle Scholar
  68. Robertazzi, T. and S. Schwartz (1989). An accelerated sequential algorithm for producing D-optimal designs. SIAM J. Sci. Statist. Comput. 10(2), 341–358.MathSciNetCrossRefGoogle Scholar
  69. Shah, S., J. Mitchell, and M. Kupferschmid (2000). An ellipsoid algorithm for equality-constrained nonlinear programs. Comput. Oper. Res. 28(1), 85–92.MathSciNetCrossRefGoogle Scholar
  70. Shimizu, K. and E. Aiyoshi (1980). Necessary conditions for min-max problems and algorithm by a relaxation procedure. IEEE Trans. Automatic Control 25, 62–66.MathSciNetzbMATHCrossRefGoogle Scholar
  71. Shor, N. (1977). Cut-off method with space extension in convex programming problems. Kibernetika 13(1), 94–95. [English Trans. Cybernetics Syst. Anal., 13(1), 94–96].Google Scholar
  72. Shor, N. (1985). Minimization Methods for Non-Differentiable Functions. Berlin: Springer.zbMATHCrossRefGoogle Scholar
  73. Shor, N. and O. Berezovski (1992). New algorithms for constructing optimal circumscribed and inscribed ellipsoids. Optim. Methods Softw. 1, 283–299.CrossRefGoogle Scholar
  74. Silvey, S., D. Titterington, and B. Torsney (1978). An algorithm for optimal designs on a finite design space. Comm. Statist. – Theory Methods A7(14), 1379–1389.Google Scholar
  75. St. John, R. and N. Draper (1975). D-optimality for regression designs: a review. Technometrics 17(1), 15–23.Google Scholar
  76. Tarasov, S., L. Khachiyan, and I. Erlich (1988). The method of inscribed ellipsoids. Soviet Math. Dokl. 37(1), 226–230.MathSciNetzbMATHGoogle Scholar
  77. Titterington, D. (1976). Algorithms for computing D-optimal designs on a finite design space. In Proc. 1976 Conf. on Information Science and Systems, Baltimore, pp. 213–216. Dept. of Electronic Engineering, John Hopkins Univ.Google Scholar
  78. Titterington, D. (1978). Estimation of correlation coefficients by ellipsoidal trimming. J. Roy. Statist. Soc. C27(3), 227–234.Google Scholar
  79. Todd, M. (1982). On minimum volume ellipsoids containing part of a given ellipsoid. Math. Oper. Res. 7(2), 253–261.MathSciNetzbMATHCrossRefGoogle Scholar
  80. Todd, M. and E. Yildirim (2007). On Khachiyan’s algorithm for the computation of minimum volume enclosing ellipsoids. Discrete Appl. Math. 155, 1731–1744.MathSciNetzbMATHCrossRefGoogle Scholar
  81. Torsney, B. (1983). A moment inequality and monotonicity of an algorithm. In K. Kortanek and A. Fiacco (Eds.), Proc. Int. Symp. on Semi-infinite Programming and Applications, Heidelberg, pp. 249–260. Springer.Google Scholar
  82. Torsney, B. (2009). W-iterations and ripples therefrom. In L. Pronzato and A. Zhigljavsky (Eds.), Optimal Design and Related Areas in Optimization and Statistics, Chapter 1, pp. 1–12. Springer.Google Scholar
  83. Uciński, D. and M. Patan (1982). D-optimal design of a monitoring network for parameter estimation of distributed systems. J. Global Optim. 39, 291–322.CrossRefGoogle Scholar
  84. Veinott, A. (1967). The supporting hyperplane method for unimodal programming. Oper. Research 15, 147–152.MathSciNetzbMATHCrossRefGoogle Scholar
  85. Walter, E. and L. Pronzato (1997). Identification of Parametric Models from Experimental Data. Heidelberg: Springer.zbMATHGoogle Scholar
  86. Welch, W. (1982). Branch-and-bound search for experimental designs based on D-optimality and other criteria. Technometrics 24(1), 41–28.MathSciNetzbMATHGoogle Scholar
  87. Wolfe, P. (1970). Convergence theory in nonlinear programming. In J. Abadie (Ed.), Integer and Nonlinear Programming, pp. 1–36. Amsterdam: North Holland.Google Scholar
  88. Wright, M. (1998). The interior-point revolution in constrained optimization. Technical Report 98–4–09, Computing Sciences Research Center, Bell Laboratories, Murray Hill, New Jersey 07974.Google Scholar
  89. Wu, C. (1978a). Some algorithmic aspects of the theory of optimal designs. Ann. Statist. 6(6), 1286–1301.MathSciNetzbMATHCrossRefGoogle Scholar
  90. Wu, C. (1978b). Some iterative procedures for generating nonsingular optimal designs. Comm. Statist. – Theory Methods A7(14), 1399–1412.Google Scholar
  91. Wu, C. and H. Wynn (1978). The convergence of general step–length algorithms for regular optimum design criteria. Ann. Statist. 6(6), 1273–1285.MathSciNetzbMATHCrossRefGoogle Scholar
  92. Wynn, H. (1970). The sequential generation of D-optimum experimental designs. Ann. Math. Statist. 41, 1655–1664.MathSciNetzbMATHCrossRefGoogle Scholar
  93. Wynn, H. (1972). Results in the theory and construction of D-optimum experimental designs. J. Roy. Statist. Soc. B34, 133–147.Google Scholar
  94. Ye, Y. (1997). Interior-Point Algorithms: Theory and Analysis. Chichester: Wiley.zbMATHCrossRefGoogle Scholar
  95. Yu, Y. (2010a). Monotonic convergence of a general algorithm for computing optimal designs. Ann. Statist. 38(3), 1593–1606.MathSciNetzbMATHCrossRefGoogle Scholar
  96. Yu, Y. (2010b). Strict monotonicity and convergence rate of Titterington’s algorithm for computing D-optimal designs. Comput. Statist. Data Anal. 54, 1419–1425.MathSciNetCrossRefGoogle Scholar
  97. Yu, Y. (2011). D-optimal designs via a cocktail algorithm. Stat. Comput. 21, 475–481.MathSciNetzbMATHCrossRefGoogle Scholar
  98. Zarrop, M. (1979). Optimal Experiment Design for Dynamic System Identification. Heidelberg: Springer.zbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Luc Pronzato
    • 1
  • Andrej Pázman
    • 2
  1. 1.French National Center for Scientific Research (CNRS)University of NiceSophia AntipolisFrance
  2. 2.Department of Applied Mathematics and StatisticsComenius UniversityBratislavaSlovakia

Personalised recommendations