On the Global-Local Dichotomy in Sparsity Modeling

  • Dmitry BatenkovEmail author
  • Yaniv Romano
  • Michael Elad
Part of the Applied and Numerical Harmonic Analysis book series (ANHA)


The traditional sparse modeling approach, when applied to inverse problems with large data such as images, essentially assumes a sparse model for small overlapping data patches and processes these patches as if they were independent from each other. While producing state-of-the-art results, this methodology is suboptimal, as it does not attempt to model the entire global signal in any meaningful way—a nontrivial task by itself.

In this paper we propose a way to bridge this theoretical gap by constructing a global model from the bottom-up. Given local sparsity assumptions in a dictionary, we show that the global signal representation must satisfy a constrained underdetermined system of linear equations, which forces the patches to agree on the overlaps. Furthermore, we show that the corresponding global pursuit can be solved via local operations. We investigate conditions for unique and stable recovery and provide numerical evidence corroborating the theory.


Sparse representations Inverse problems Convolutional sparse coding 



The research leading to these results has received funding from the European Research Council under European Union’s Seventh Framework Programme, ERC Grant agreement no. 320649. The authors would also like to thank Jeremias Sulam, Vardan Papyan, Raja Giryes, and Gitta Kutinyok for inspiring discussions.


  1. 1.
    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G.S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, TensorFlow: large-scale machine learning on heterogeneous systems (2015). Software available from
  2. 2.
    R. Aceska, J.L. Bouchot, S. Li, Local sparsity and recovery of fusion frames structured signals. preprint (2015).
  3. 3.
    M. Aharon, M. Elad, Sparse and redundant modeling of image content using an image-signature-dictionary. SIAM J. Imag. Sci. 1(3), 228–247 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    U. Ayaz, S. Dirksen, H. Rauhut, Uniform recovery of fusion frame structured sparse signals. Appl. Comput. Harmon. Anal. 41(2), 341–361 (2016).
  5. 5.
    S. Basu, R. Pollack, M.F. Roy, Algorithms in Real Algebraic Geometry. Algorithms and Computation in Mathematics, 2nd edn., vol. 10 (Springer, Berlin, 2006)Google Scholar
  6. 6.
    T. Blumensath, M. Davies, Sparse and shift-invariant representations of music. IEEE Trans. Audio Speech Lang. Process. 14(1), 50–57 (2006). CrossRefGoogle Scholar
  7. 7.
    T. Blumensath, M.E. Davies, Sampling theorems for signals from the union of finite-dimensional linear subspaces. IEEE Trans. Inf. Theory 55(4), 1872–1882 (2009). MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    P. Boufounos, G. Kutyniok, H. Rauhut, Sparse recovery from combined fusion frame measurements. IEEE Trans. Inf. Theory 57(6), 3864–3876 (2011). MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011). CrossRefzbMATHGoogle Scholar
  10. 10.
    H. Bristow, A. Eriksson, S. Lucey, Fast convolutional sparse coding. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2013), pp. 391–398Google Scholar
  11. 11.
    A.M. Bruckstein, D.L. Donoho, M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009). MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    E.J. Candes, Modern statistical estimation via oracle inequalities. Acta Numer. 15, 257–325 (2006). MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    S. Chen, S.A. Billings, W. Luo, Orthogonal least squares methods and their application to non-linear system identification. Int. J. Control. 50(5), 1873–1896 (1989)CrossRefzbMATHGoogle Scholar
  14. 14.
    W. Dong, L. Zhang, G. Shi, X. Li, Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22(4), 1620–1630 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    D.L. Donoho, M. Elad, Optimally sparse representation in general (nonorthogonal) dictionaries via l1 minimization. Proc. Natl. Acad. Sci. 100(5), 2197–2202 (2003). doi:
  16. 16.
    C. Ekanadham, D. Tranchina, E.P. Simoncelli, A unified framework and method for automatic neural spike identification. J. Neurosci. Methods 222, 47–55 (2014). doi: 10.1016/j.jneumeth.2013.10.001.
  17. 17.
    M. Elad, Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing (Springer, New York, 2010)CrossRefzbMATHGoogle Scholar
  18. 18.
    M. Elad, M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Y.C. Eldar, M. Mishali, Block sparsity and sampling over a union of subspaces, in 2009 16th International Conference on Digital Signal Processing (IEEE, New York, 2009), pp. 1–8. Google Scholar
  20. 20.
    Y.C. Eldar, M. Mishali, Robust recovery of signals from a structured union of subspaces. IEEE Trans. Inf. Theory 55(11), 5302–5316 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
  22. 22.
    S. Foucart, H. Rauhut, A Mathematical Introduction to Compressive Sensing (Springer, New York, 2013). CrossRefzbMATHGoogle Scholar
  23. 23.
    D. Gabay, B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2(1), 17–40 (1976)CrossRefzbMATHGoogle Scholar
  24. 24.
    R. Glowinski, On alternating direction methods of multipliers: a historical perspective, in Modeling, Simulation and Optimization for Science and Technology (Springer, Dordrecht, 2014), pp. 59–82zbMATHGoogle Scholar
  25. 25.
    R. Grosse, R. Raina, H. Kwong, A.Y. Ng, Shift-invariance sparse coding for audio classification (2012). arXiv preprint arXiv: 1206.5241Google Scholar
  26. 26.
    R. Grosse, R. Raina, H. Kwong, A.Y. Ng, Shift-invariance sparse coding for audio classification. arXiv: 1206.5241 [cs, stat] (2012). arXiv: 1206.5241
  27. 27.
    S. Gu, W. Zuo, Q. Xie, D. Meng, X. Feng, L. Zhang, Convolutional sparse coding for image super-resolution, in Proceedings of the IEEE International Conference on Computer Vision (2015), pp. 1823–1831Google Scholar
  28. 28.
    F. Heide, W. Heidrich, G. Wetzstein, Fast and flexible convolutional sparse coding, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, New York, 2015), pp. 5135–5143Google Scholar
  29. 29.
    J. Huang, T. Zhang, D. Metaxas, Learning with structured sparsity. J. Mach. Learn. Res. 12, 3371–3412 (2011)MathSciNetzbMATHGoogle Scholar
  30. 30.
    J. Huang, T. Zhang, et al., The benefit of group sparsity. Ann. Stat. 38(4), 1978–2004 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    K. Kavukcuoglu, P. Sermanet, Y.l. Boureau, K. Gregor, M. Mathieu, Y.L. Cun, Learning convolutional feature hierarchies for visual recognition, in Advances in Neural Information Processing Systems, ed. by J.D. Lafferty, C.K.I. Williams, J. Shawe-Taylor, R.S. Zemel, A. Culotta, vol. 23 (Curran Associates, Red Hook, 2010), pp. 1090–1098.
  32. 32.
    A. Kyrillidis, L. Baldassarre, M.E. Halabi, Q. Tran-Dinh, V. Cevher, Structured sparsity: discrete and convex approaches, in Compressed Sensing and Its Applications. Applied and Numerical Harmonic Analysis, ed. by H. Boche, R. Calderbank, G. Kutyniok, J. Vybíral (Springer, Cham, 2015), pp. 341–387.
  33. 33.
    P.L. Lions, B. Mercier, Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    M.A. Little, N.S. Jones, Generalized methods and solvers for noise removal from piecewise constant signals. II. New methods. Proc. R. Soc. Lond. A: Math. Phys. Eng. Sci. rspa20100674 (2011). doi:
  35. 35.
    Y.M. Lu, M.N. Do, A theory for sampling signals from a union of subspaces. IEEE Trans. Signal Process. 56, 2334–2345 (2007)MathSciNetCrossRefGoogle Scholar
  36. 36.
    J. Mairal, G. Sapiro, M. Elad, Learning multiscale sparse representations for image and video restoration. Multiscale Model. Simul. 7(1), 214–241 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    J. Mairal, F. Bach, J. Ponce, G. Sapiro, A. Zisserman, Non-local sparse models for image restoration. in 2009 IEEE 12th International Conference on Computer Vision (IEEE, New York, 2009), pp. 2272–2279Google Scholar
  38. 38.
    J. Mairal, F. Bach, J. Ponce, Sparse modeling for image and vision processing. Found. Trends Comput. Graph. Vis. 8(2–3), 85–283 (2014).
  39. 39.
    Maplesoft, a division of Waterloo Maple Inc.
  40. 40.
    V. Papyan, M. Elad, Multi-scale patch-based image restoration. IEEE Trans. Image Process. 25(1), 249–261 (2016). MathSciNetCrossRefGoogle Scholar
  41. 41.
    V. Papyan, Y. Romano, M. Elad, Convolutional neural networks analyzed via convolutional sparse coding. J. Mach. Learn. Res. 18(83), 1–52 (2017)MathSciNetGoogle Scholar
  42. 42.
    V. Papyan, J. Sulam, M. Elad, Working locally thinking globally: theoretical guarantees for convolutional sparse coding. IEEE Trans. Signal Process. 65(21), 5687–5701 (2017)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Y.C. Pati, R. Rezaiifar, P. Krishnaprasad, Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition, in Asilomar Conference on Signals, Systems and Computers (IEEE, New York, 1993), pp. 40–44Google Scholar
  44. 44.
    R. Quiroga, Spike sorting. Scholarpedia 2(12), 3583 (2007).
  45. 45.
    Y. Romano, M. Elad, Boosting of image denoising algorithms. SIAM J. Imag. Sci. 8(2), 1187–1219 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    Y. Romano, M. Elad, Patch-disagreement as away to improve K-SVD denoising, in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, New York, 2015), pp. 1280–1284Google Scholar
  47. 47.
    Y. Romano, M. Protter, M. Elad, Single image interpolation via adaptive nonlocal sparsity-based modeling. IEEE Trans. Image Process. 23(7), 3085–3098 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    L.I. Rudin, S. Osher, E. Fatemi, Nonlinear total variation based noise removal algorithms. Physica D 60(1), 259–268 (1992). MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    C. Rusu, B. Dumitrescu, S. Tsaftaris, Explicit shift-invariant dictionary learning. IEEE Signal Process. Lett. 21, 6–9 (2014). CrossRefGoogle Scholar
  50. 50.
    E. Smith, M.S. Lewicki, Efficient coding of time-relative structure using spikes. Neural Comput. 17(1), 19–45 (2005). CrossRefzbMATHGoogle Scholar
  51. 51.
    A.M. Snijders, N. Nowak, R. Segraves, S. Blackwood, N. Brown, J. Conroy, G. Hamilton, A.K. Hindle, B. Huey, K. Kimura, S. Law, K. Myambo, J. Palmer, B. Ylstra, J.P. Yue, J.W. Gray, A.N. Jain, D. Pinkel, D.G. Albertson, Assembly of microarrays for genome-wide measurement of DNA copy number. Nat. Genet. 29(3), 263–264 (2001).
  52. 52.
    J. Sulam, M. Elad, Expected patch log likelihood with a sparse prior, in International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition (Springer, New York, 2015), pp. 99–111Google Scholar
  53. 53.
    J. Sulam, B. Ophir, M. Elad, Image denoising through multi-scale learnt dictionaries, in 2014 IEEE International Conference on Image Processing (ICIP) (IEEE, New York, 2014), pp. 808–812CrossRefGoogle Scholar
  54. 54.
    J.J. Thiagarajan, K.N. Ramamurthy, A. Spanias, Shift-invariant sparse representation of images using learned dictionaries, in IEEE Workshop on Machine Learning for Signal Processing, 2008, MLSP 2008 (2008), pp. 145–150
  55. 55.
    J.A. Tropp, A.C. Gilbert, M.J. Strauss, Algorithms for simultaneous sparse approximation. Part i: greedy pursuit. Signal Process. 86(3), 572–588 (2006)zbMATHGoogle Scholar
  56. 56.
    J. Yang, J. Wright, T.S. Huang, Y. Ma, Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  57. 57.
    G. Yu, G. Sapiro, S. Mallat, Solving inverse problems with piecewise linear estimators: From gaussian mixture models to structured sparsity. IEEE Trans. Image Process. 21(5), 2481–2499 (2012). MathSciNetCrossRefzbMATHGoogle Scholar
  58. 58.
    M.D. Zeiler, D. Krishnan, G.W. Taylor, R. Fergus, Deconvolutional networks, in 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (IEEE, New York, 2010), pp. 2528–2535. CrossRefGoogle Scholar
  59. 59.
    M. Zeiler, G. Taylor, R. Fergus, Adaptive deconvolutional networks for mid and high level feature learning, in 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2018–2025 (2011). doi:
  60. 60.
    D. Zoran, Y. Weiss, From learning models of natural image patches to whole image restoration, in 2011 IEEE International Conference on Computer Vision (ICCV) (IEEE, New York, 2011), pp. 479–486. Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of MathematicsMassachusetts Institute of TechnologyCambridgeUSA
  2. 2.Department of Electrical EngineeringTechnion - Israel Institute of TechnologyHaifaIsrael
  3. 3.Department of Computer ScienceTechnion - Israel Institute of TechnologyHaifaIsrael

Personalised recommendations