Constructive Approximation

, Volume 13, Issue 1, pp 57–98 | Cite as

Adaptive greedy approximations

  • G. Davis
  • S. Mallat
  • M. Avellaneda


The problem of optimally approximating a function with a linear expansion over a redundant dictionary of waveforms is NP-hard. The greedy matching pursuit algorithm and its orthogonalized variant produce suboptimal function expansions by iteratively choosing dictionary waveforms that best match the function’s structures. A matching pursuit provides a means of quickly computing compact, adaptive function approximations.

Numerical experiments show that the approximation errors from matching pursuits initially decrease rapidly, but the asymptotic decay rate of the errors is slow. We explain this behavior by showing that matching pursuits are chaotic, ergodic maps. The statistical properties of the approximation errors of a pursuit can be obtained from the invariant measure of the pursuit. We characterize these measures using group symmetries of dictionaries and by constructing a stochastic differential equation model.

We derive a notion of the coherence of a signal with respect to a dictionary from our characterization of the approximation errors of a pursuit. The dictionary elements slected during the initial iterations of a pursuit correspond to a function’s coherent structures. The tail of the expansion, on the other hand, corresponds to a noise which is characterized by the invariant measure of the pursuit map.

When using a suitable dictionary, the expansion of a function into its coherent structures yields a compact approximation. We demonstrate a denoising algorithm based on coherent function expansions.

AMS classification


Key words and phrases

Matching pursuit Adaptive approximations Greedy algorithms Denoising Overcomplete signal representation Time frequency analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    S. Chen, S. A. Billings, W. Luo (1989):Orthogonal least squares methods and their application to non-linear system identification. Internat. J. Control,50:1873–1896.zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    L. Cohen (1989):Time-frequency distributions: A review. Proc. IEEE,77:941–979.CrossRefGoogle Scholar
  3. 3.
    P. Collet, J. P. Eckmann (1980): Iterated Maps on the Interval as Dynamical Systems. Boston: Birkhauser.zbMATHGoogle Scholar
  4. 4.
    T. H. Cormen, C. E. Leiserson, R. L. Rivest (1991): Introduction to Algorithms. New York: McGraw-Hill.Google Scholar
  5. 5.
    I. Daubechies (1991): Ten Lectures on Wavelets. CBMS-NSF Series in Appl. Math. Philadelphia: SIAM.Google Scholar
  6. 6.
    G. Davis (1994): Adaptive Nonlinear Approximations. PhD dissertation, Department of Mathematics, New York University.Google Scholar
  7. 7.
    G. Davis, S. Mallat, Z. Zhang (1994):Adaptive time-frequency decompositions. Optical Engrg.,33:2183–2191.CrossRefGoogle Scholar
  8. 8.
    R. L. Devaney (1989): An Introduction to Chaotic Dynamical Systems. New York: Addison-Wesley.zbMATHGoogle Scholar
  9. 9.
    R. A. DeVore, B. Jawerth, B. J. Lucier (1992):Image compression through wavelet transform coding. IEEE Trans. Inform. Theory,38:719–746.CrossRefMathSciNetGoogle Scholar
  10. 10.
    R. A. DeVore, B. Jawerth, V. Popov (1992):Compression of wavelet decompositions. Amer. J. Math.,114:737–785.zbMATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    D. L. Donoho (1993):Wavelet shrinkage and W.V.D.: A 10-minute tour. In: Progress in Wavelet Analysis and Applications (Y. Meyer, S. Roques, eds.). Gif-sur-Yvette, France: Editions Frontieres, pp. 109–128.Google Scholar
  12. 12.
    C. W. Gardiner (1985): Handbook of Stochastic Methods. New York: Springer-Verlag.Google Scholar
  13. 13.
    M. R. Garey, D. S. Johnson (1979): Computers and Intractability: A Guide to the Theory of NP-Completeness. New York: W. H. Freeman.zbMATHGoogle Scholar
  14. 14.
    P. R. Halmos (1956): Lectures on Ergodic Theory. Tokyo: Mathematical Society of Japan.zbMATHGoogle Scholar
  15. 15.
    L. K. Jones (1987):On a conjecture of Huber concerning the convergence of projection pursuit regression. Ann. Statist.,15:880–882.zbMATHMathSciNetGoogle Scholar
  16. 16.
    A. Lasota, M. Mackey (1985): Probabilistic Properties of Deterministic Systems. New York: Cambridge University Press.zbMATHGoogle Scholar
  17. 17.
    S. Mallat, Z. Zhang (1993):Matching pursuit with time-frequency dictionaries. IEEE Trans. Signal Processing,41:3397–3415.zbMATHCrossRefGoogle Scholar
  18. 18.
    Y. C. Pati, R. Rezaiifar, P. S. Krishnaprasad (1993):Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. Proceedings of the 27th Annual Asilomar Conference on Signals, Systems, and Computers, pp. 40–44. IEEE Computer Society Press.Google Scholar
  19. 19.
    S. Qian, D. Chen (1994):Signal representation via adaptive normalized gaussian functions. IEEE Trans. Signal Processing,36:1–11.zbMATHMathSciNetGoogle Scholar
  20. 20.
    M. Reed, B. Simon (1972): Methods of Modern Mathematical Statistics, Vol. 1. New York: Academic Press.Google Scholar
  21. 21.
    N. Saito (1994):Simultaneous noise suppression and signal compression using a library of orthonormal bases and the minimum description length criterion. Wavelets in Geophysics (Efi Foufoula-Georgiou, Praveen Kumar, eds.). Academic Press, San Diego.Google Scholar
  22. 22.
    L. Blum, M. Shub, S. Smale (1989):On a theory of computation and complexity over the real numbers: NP-completeness, recursive functions, and universal machines. Bull. Amer. Math. Soc.,21:1–46.zbMATHCrossRefMathSciNetGoogle Scholar
  23. 23.
    P. Walters (1975): Ergodic theory—Introductory Lectures. New York: Springer-Verlag.zbMATHGoogle Scholar
  24. 24.
    Z. Zhang (1993): Matching Pursuit. PhD dissertation, New York University.Google Scholar

Copyright information

© Springer-Verlag New York Inc 1997

Authors and Affiliations

  • G. Davis
    • 1
  • S. Mallat
    • 2
  • M. Avellaneda
    • 3
  1. 1.Mathematics DepartmentDartmouth CollegeHanoverU.S.A.
  2. 2.Ecole PolytechniqueCMAPPalaiseau CedexFrance
  3. 3.Courant InstituteNew York UniversityNew YorkU.S.A.

Personalised recommendations