Entropic Regularization of the 0 Function

  • Jonathan M. BorweinEmail author
  • D. Russell Luke
Part of the Springer Optimization and Its Applications book series (SOIA, volume 49)


Many problems of interest where more than one solution is possible seek, among these, the one that is sparsest. The objective that most directly accounts for sparsity, the 0 metric, is usually avoided since this leads to a combinatorial optimization problem. The function \(\|{x\|}_{0}\) is often viewed as the limit of the p metrics. Naturally, there have been some attempts to use this as an objective for p small, though this is a nonconvex function for p < 1. We propose instead a scaled and shifted Fermi-Dirac entropy with two parameters, one controlling the smoothness of the approximation and the other the steepness of the metric. Our proposed metric is a convex relaxation for which a strong duality theory holds, yielding dual methods for metrics approaching the desired \(\|{\cdot \|}_{0}\) function. Without smoothing, we propose a dynamically reweighted subdifferential descent method with “exact” line search that is finitely terminating for constraints that are well-separated. This algorithm is shown to recapture in a special case certain well-known “greedy” algorithms. Consequently we are able to provide an explicit algorithm whose fixed point, under the appropriate assumptions, is the sparsest possible solution. The variational perspective yields general strategies to make the algorithm more robust.


Convex optimization Fenchel duality Entropy Regularization Sparsity Signal processing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



Jonathan M. Borwein, Research was supported by the Australian Research Council. D. Russell Luke, Research was supported by the US National Science Foundation grant DMS-0712796.


  1. 1.
    Borwein, J., Vanderwerff, J.: Convex Functions: Constructions, Characterizations and Counterexamples, Encyclopedias in Mathematics, vol. 109. Cambridge University Press, New York (2010)Google Scholar
  2. 2.
    Borwein, J.M., Lewis, A.S.: Convex analysis and nonlinear optimization: theory and examples, 2nd edn. Springer, New York (2006)zbMATHCrossRefGoogle Scholar
  3. 3.
    Borwein, J.M., Luke, D.R.: Duality and convex programming. In: O. Scherzer (ed.) Handbook of Mathematical Methods in Imaging, pp. 229–270. Springer-Verlag (2011)Google Scholar
  4. 4.
    Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34–81 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Candès, E., Recht, B.: Exact matrix completion via comvex optimization. Found. of Comput. Math. 9, 717–772 (2009)zbMATHCrossRefGoogle Scholar
  6. 6.
    Candès, E., Tao, T.: Near-optimal signal recovery from random projections: Universal encoding strategies? IEEE Trans. Inform. Theory 52, 5406–5425 (2006)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20, 33–61 (1998)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Donoho, D.L., Elad, M.: Optimally sparse representation in general (non-orthogonal) dictionaries via l1 minimization,. Proc. Natl. Acad. Sci. 100, 2197–2202 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Gribonval, R., Nielsen, M.: Sparse decompositions in unions of bases. IEEETrans. Inform. Theory 49, 3320–3325 (2003)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Luke, D.R.: Relaxed averaged alternating reflections for diffraction imaging. Inverse Problems 21, 37–50 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Luke, D.R.: Finding best approximation pairs relative to a convex and a prox-regular set in Hilbert space. SIAM J. Optim. 19, 714–739 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227–234 (1995)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Recht, B., Fazel, M., Parrilo, P.: Guaranteed minimum rank solutions of matrix equations via nuclear norm minimization. SIAM Rev. 52, 471–501 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Rockafellar, R.T., Wets, R.J.: Variational Analysis. Grundlehren der mathematischen Wissenschaften. Springer, Berlin (1998)zbMATHCrossRefGoogle Scholar
  15. 15.
    Santosa, F., Symes, W.W.: Linear inversion of band-limited reflection seismograms. SIAM J. Sci. Statist Comput. 7, 1307–1330 (1986)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Institute for Numerical and Applied MathematicsUniversity of GoettingenGoettingenGermany

Personalised recommendations