Structure in Optimization: Factorable Programming and Functions

  • Trond Steihaug
  • Shahadat Hossain
  • Laurent Hascoët
Conference paper


It is frequently observed that effective exploitation of problem structure plays a significant role in computational procedures for solving large-scale nonlinear optimization problems. A necessary step in this regard is to express the computation in a manner that exposes the exploitable structure. The formulation of large-scale problems in many scientific applications naturally give rise to “structured” representation. Examples of computationally useful structures arising in large-scale optimization problems include unary functions, partially separable functions, and factorable functions. These structures were developed from 1967 through 1990. In this paper we closely examine commonly occurring structures in optimization with regard to efficient and automatic calculation of first- and higher-order derivatives. Further, we explore the relationship between source code transformation as in algorithmic differentiation (AD) and factorable programming. As an illustration, we consider some classical examples.


Algorithmic differentiation Source code transformation Factorable programming 


  1. 1.
    Griewank, A., Toint, Ph.L.: On the unconstrained optimization of partially separable functions. In: Powell, M.J.D. (ed.) Nonlinear Optimization 1981, pp. 301–312. Academic Press, New York (1982)Google Scholar
  2. 2.
    Conn, A.R., Gould, N.I.M., Toint, Ph.L.: An introduction to the structure of large scale nonlinear optimization problems and the LANCELOT project. In: Glowinski, R., Lichnewsky, A. (eds.) Computing Methods in Applied Sciences and Engineering, pp. 42–51. SIAM, Philadelphia (1990)Google Scholar
  3. 3.
    Conn, A.R., Gould, N.I.M., Toint, Ph.L.: LANCELOT: A Fortran Package for Large-Scale Nonlinear Optimization (Release A), 1st edn. Springer, Berlin (1992)Google Scholar
  4. 4.
    Conn, A.R., Gould, N.I.M., Toint, Ph.L.: Improving the decomposition of partially separable functions in the context of large-scale optimization: a first approach. In: Hager, W.W., Hearn, D.W., Pardalos, P.M. (eds.) Large Scale Optimization: State of the Art, pp. 82–94. Kluwer Academic Publishers, Amsterdam (1994)Google Scholar
  5. 5.
    Bouaricha, A., Morè, J.J.: Impact of partial separability on large-scale optimization. Comput. Optim. Appl. 7, 27–40 (1997)MathSciNetMATHCrossRefGoogle Scholar
  6. 6.
    Gay, D.M.: More AD of nonlinear AMPL models: computing Hessian information and exploiting partial separability. In: Berz, M., Bischof, C., Corliss, G., Griewank, A. (eds.) Computational Differentiation: Techniques, Applications, and Tools, pp. 173–184. SIAM, Philadelphia (1996)Google Scholar
  7. 7.
    Conforti, D., De Luca, L., Grandinetti, L., Musmanno, R.: A parallel implementation of automatic differentiation for partially separable functions using PVM. Parallel Comput. 22, 643–656 (1996)MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    McCormick, G.P., Sofer, A.: Optimization with unary functions. Math. Program. 52(1), 167–178 (1991)MathSciNetMATHCrossRefGoogle Scholar
  9. 9.
    Steihaug, T., Suleiman, S.: Global convergence and the Powell singular function. J. Glob. Optim. 1–9 (2012). doi: 10.1007/s10898-012-9898-z.
  10. 10.
    Hascoët, L., Hossain, S., Steihaug, T.: Structured computation in optimization and algorithmic differentiation. ACM Commun. Comput. Algebra 46(3) (2012)Google Scholar
  11. 11.
    Ghaemi, A., McCormick, G.P.: Symbolic factorable SUMT: What is it? How is it used? Technical Report T-402. Institute for Management Science and Engineering, The George Washington University, Washington DC (May 1979)Google Scholar
  12. 12.
    Kedem, G.: Automatic differentiation of computer programs. ACM Trans. Math. Softw. 6(2), 150–165 (1980)MathSciNetMATHCrossRefGoogle Scholar
  13. 13.
    Jackson, R.H.F., McCormick, G.P.: The polyadic structure of factorable function tensors with application to high-order minimization techniques. J. Optim. Theory Appl. 51(1), 63–94 (1986)MathSciNetMATHCrossRefGoogle Scholar
  14. 14.
    Jackson, R.H.F., McCormick, G.P.: Second-order sensitivity analysis in factorable programming: theory and applications. Math. Program. 41(1–3), 1–27 (1988)MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Rall, L.B.: Automatic Differentiation: Techniques and Applications. Lecture Notes in Computer Science, vol. 120. Springer, Berlin (1981)Google Scholar
  16. 16.
    Smith, E.M., Pantelides, C.C.: Global optimisation of nonconvex minlps. Comput. Chem. Eng. 21(Suppl.), S791–S796 (1997)Google Scholar
  17. 17.
    Goldfarb, D., Wang, S.Y.: Partial-update Newton methods for unary, factorable, and partially separable optimization. SIAM J. Optim. 3(2), 382–397 (1993)MathSciNetMATHCrossRefGoogle Scholar
  18. 18.
    McCormick, G.P.: Nonlinear Programming: Theory, Algorithms and Applications. Wiley, New York (1983)Google Scholar
  19. 19.
    McCormick, G.P.: Computability of global solutions to factorable nonconvex programs: part I convex underestimating problems. Math. Program. 10(1), 147–175 (1976)MathSciNetMATHCrossRefGoogle Scholar
  20. 20.
    Griewank, A., Walther, A.: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. 2nd edn, Number 105 in Other Titles in Applied Mathematics. SIAM, Philadelphia (2008)Google Scholar
  21. 21.
    McCormick, G.P.: A mini-manual for use of the SUMT computer program and the factorable programming language. Technical Report SOL 74–15. Department of Operations Research, Stanford University, Stanford (August 1974)Google Scholar
  22. 22.
    Mylander, W.C., Holmes, R., McCormick, G.P.: A Guide to SUMT-Version 4: The Computer Program Implementing the Sequential Unconstrained Minimization Technique for Nonlinear Programming. RAC-P-63, Research Analysis Corporation, McLean (1971)Google Scholar
  23. 23.
    Pugh, R.E.: A language for nonlinear programming problems. Math. Program. 2, 176–206 (1972)MathSciNetMATHCrossRefGoogle Scholar
  24. 24.
    Fiacco, A.V., McCormick, G.P.: Nonlinear Programming: Sequential Unconstrained Minimization Techniques. Wiley, New York (1968)Google Scholar
  25. 25.
    McCormick, G.P.: Minimizing structured unconstrained functions. Technical Paper RAC-TP-277. Research Analysis Corporation, McLean, Virginia (October 1967)Google Scholar
  26. 26.
    Hascoët, L., Pascual, V.: Tapenade 2.1 user’s guide. Technical Report 0300, INRIA (2004)Google Scholar
  27. 27.
    Hascoët, L.: Reversal strategies for adjoint algorithms. In: Bertot, Y., Huet, G., Lévy, J.-J., Plotkin, G. (eds.) From Semantics to Computer Science. Essays in Memory of Gilles Kahn, pp. 487–503. Cambridge University Press, New York (2009)Google Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  • Trond Steihaug
    • 1
  • Shahadat Hossain
    • 2
  • Laurent Hascoët
    • 3
  1. 1.Department of InformaticsUniversity of BergenBergenNorway
  2. 2.Department of Mathematics and Computer ScienceUniversity of LethbridgeLethbridgeCanada
  3. 3.INRIASophia-AntipolisFrance

Personalised recommendations