Advertisement

On the Global-Local Dichotomy in Sparsity Modeling

  • Dmitry Batenkov
  • Yaniv Romano
  • Michael Elad
Chapter
Part of the Applied and Numerical Harmonic Analysis book series (ANHA)

Abstract

The traditional sparse modeling approach, when applied to inverse problems with large data such as images, essentially assumes a sparse model for small overlapping data patches and processes these patches as if they were independent from each other. While producing state-of-the-art results, this methodology is suboptimal, as it does not attempt to model the entire global signal in any meaningful way—a nontrivial task by itself.

In this paper we propose a way to bridge this theoretical gap by constructing a global model from the bottom-up. Given local sparsity assumptions in a dictionary, we show that the global signal representation must satisfy a constrained underdetermined system of linear equations, which forces the patches to agree on the overlaps. Furthermore, we show that the corresponding global pursuit can be solved via local operations. We investigate conditions for unique and stable recovery and provide numerical evidence corroborating the theory.

Keywords

Sparse representations Inverse problems Convolutional sparse coding 

Notes

Acknowledgments

The research leading to these results has received funding from the European Research Council under European Union’s Seventh Framework Programme, ERC Grant agreement no. 320649. The authors would also like to thank Jeremias Sulam, Vardan Papyan, Raja Giryes, and Gitta Kutinyok for inspiring discussions.

References

  1. 1.
    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G.S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, TensorFlow: large-scale machine learning on heterogeneous systems (2015). http://tensorflow.org/. Software available from tensorflow.org
  2. 2.
    R. Aceska, J.L. Bouchot, S. Li, Local sparsity and recovery of fusion frames structured signals. preprint (2015). http://www.mathc.rwth-aachen.de/~bouchot/files/pubs/FusionCSfinal.pdf
  3. 3.
    M. Aharon, M. Elad, Sparse and redundant modeling of image content using an image-signature-dictionary. SIAM J. Imag. Sci. 1(3), 228–247 (2008)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    U. Ayaz, S. Dirksen, H. Rauhut, Uniform recovery of fusion frame structured sparse signals. Appl. Comput. Harmon. Anal. 41(2), 341–361 (2016). https://doi.org/10.1016/j.acha.2016.03.006. http://www.sciencedirect.com/science/article/pii/S1063520316000294
  5. 5.
    S. Basu, R. Pollack, M.F. Roy, Algorithms in Real Algebraic Geometry. Algorithms and Computation in Mathematics, 2nd edn., vol. 10 (Springer, Berlin, 2006)Google Scholar
  6. 6.
    T. Blumensath, M. Davies, Sparse and shift-invariant representations of music. IEEE Trans. Audio Speech Lang. Process. 14(1), 50–57 (2006). http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1561263 CrossRefGoogle Scholar
  7. 7.
    T. Blumensath, M.E. Davies, Sampling theorems for signals from the union of finite-dimensional linear subspaces. IEEE Trans. Inf. Theory 55(4), 1872–1882 (2009). http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4802322 MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    P. Boufounos, G. Kutyniok, H. Rauhut, Sparse recovery from combined fusion frame measurements. IEEE Trans. Inf. Theory 57(6), 3864–3876 (2011).  https://doi.org/https://doi.org/10.1109/TIT.2011.2143890 MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011). http://dx.doi.org/10.1561/2200000016 CrossRefMATHGoogle Scholar
  10. 10.
    H. Bristow, A. Eriksson, S. Lucey, Fast convolutional sparse coding. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2013), pp. 391–398Google Scholar
  11. 11.
    A.M. Bruckstein, D.L. Donoho, M. Elad, From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009). http://epubs.siam.org/doi/abs/10.1137/060657704 MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    E.J. Candes, Modern statistical estimation via oracle inequalities. Acta Numer. 15, 257–325 (2006). http://journals.cambridge.org/abstract_S0962492906230010 MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    S. Chen, S.A. Billings, W. Luo, Orthogonal least squares methods and their application to non-linear system identification. Int. J. Control. 50(5), 1873–1896 (1989)CrossRefMATHGoogle Scholar
  14. 14.
    W. Dong, L. Zhang, G. Shi, X. Li, Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22(4), 1620–1630 (2013)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    D.L. Donoho, M. Elad, Optimally sparse representation in general (nonorthogonal) dictionaries via l1 minimization. Proc. Natl. Acad. Sci. 100(5), 2197–2202 (2003). doi:  https://doi.org/10.1073/pnas.0437847100. http://www.pnas.org/content/100/5/2197
  16. 16.
    C. Ekanadham, D. Tranchina, E.P. Simoncelli, A unified framework and method for automatic neural spike identification. J. Neurosci. Methods 222, 47–55 (2014). doi: 10.1016/j.jneumeth.2013.10.001. http://www.sciencedirect.com/science/article/pii/S0165027013003415
  17. 17.
    M. Elad, Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing (Springer, New York, 2010)CrossRefMATHGoogle Scholar
  18. 18.
    M. Elad, M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Y.C. Eldar, M. Mishali, Block sparsity and sampling over a union of subspaces, in 2009 16th International Conference on Digital Signal Processing (IEEE, New York, 2009), pp. 1–8. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5201211 Google Scholar
  20. 20.
    Y.C. Eldar, M. Mishali, Robust recovery of signals from a structured union of subspaces. IEEE Trans. Inf. Theory 55(11), 5302–5316 (2009)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
  22. 22.
    S. Foucart, H. Rauhut, A Mathematical Introduction to Compressive Sensing (Springer, New York, 2013). http://link.springer.com/content/pdf/10.1007/978-0-8176-4948-7.pdf CrossRefMATHGoogle Scholar
  23. 23.
    D. Gabay, B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2(1), 17–40 (1976)CrossRefMATHGoogle Scholar
  24. 24.
    R. Glowinski, On alternating direction methods of multipliers: a historical perspective, in Modeling, Simulation and Optimization for Science and Technology (Springer, Dordrecht, 2014), pp. 59–82MATHGoogle Scholar
  25. 25.
    R. Grosse, R. Raina, H. Kwong, A.Y. Ng, Shift-invariance sparse coding for audio classification (2012). arXiv preprint arXiv: 1206.5241Google Scholar
  26. 26.
    R. Grosse, R. Raina, H. Kwong, A.Y. Ng, Shift-invariance sparse coding for audio classification. arXiv: 1206.5241 [cs, stat] (2012). http://arxiv.org/abs/1206.5241. arXiv: 1206.5241
  27. 27.
    S. Gu, W. Zuo, Q. Xie, D. Meng, X. Feng, L. Zhang, Convolutional sparse coding for image super-resolution, in Proceedings of the IEEE International Conference on Computer Vision (2015), pp. 1823–1831Google Scholar
  28. 28.
    F. Heide, W. Heidrich, G. Wetzstein, Fast and flexible convolutional sparse coding, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, New York, 2015), pp. 5135–5143Google Scholar
  29. 29.
    J. Huang, T. Zhang, D. Metaxas, Learning with structured sparsity. J. Mach. Learn. Res. 12, 3371–3412 (2011)MathSciNetMATHGoogle Scholar
  30. 30.
    J. Huang, T. Zhang, et al., The benefit of group sparsity. Ann. Stat. 38(4), 1978–2004 (2010)MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    K. Kavukcuoglu, P. Sermanet, Y.l. Boureau, K. Gregor, M. Mathieu, Y.L. Cun, Learning convolutional feature hierarchies for visual recognition, in Advances in Neural Information Processing Systems, ed. by J.D. Lafferty, C.K.I. Williams, J. Shawe-Taylor, R.S. Zemel, A. Culotta, vol. 23 (Curran Associates, Red Hook, 2010), pp. 1090–1098. http://papers.nips.cc/paper/4133-learning-convolutional-feature-hierarchies-for-visual-recognition.pdf
  32. 32.
    A. Kyrillidis, L. Baldassarre, M.E. Halabi, Q. Tran-Dinh, V. Cevher, Structured sparsity: discrete and convex approaches, in Compressed Sensing and Its Applications. Applied and Numerical Harmonic Analysis, ed. by H. Boche, R. Calderbank, G. Kutyniok, J. Vybíral (Springer, Cham, 2015), pp. 341–387. http://link.springer.com/chapter/10.1007/978-3-319-16042-9_12. https://doi.org/10.1007/978-3-319-16042-9_12
  33. 33.
    P.L. Lions, B. Mercier, Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    M.A. Little, N.S. Jones, Generalized methods and solvers for noise removal from piecewise constant signals. II. New methods. Proc. R. Soc. Lond. A: Math. Phys. Eng. Sci. rspa20100674 (2011). doi:  https://doi.org/https://doi.org/10.1098/rspa.2010.0674. http://rspa.royalsocietypublishing.org/content/early/2011/06/07/rspa.2010.0674
  35. 35.
    Y.M. Lu, M.N. Do, A theory for sampling signals from a union of subspaces. IEEE Trans. Signal Process. 56, 2334–2345 (2007)MathSciNetCrossRefGoogle Scholar
  36. 36.
    J. Mairal, G. Sapiro, M. Elad, Learning multiscale sparse representations for image and video restoration. Multiscale Model. Simul. 7(1), 214–241 (2008)MathSciNetCrossRefMATHGoogle Scholar
  37. 37.
    J. Mairal, F. Bach, J. Ponce, G. Sapiro, A. Zisserman, Non-local sparse models for image restoration. in 2009 IEEE 12th International Conference on Computer Vision (IEEE, New York, 2009), pp. 2272–2279Google Scholar
  38. 38.
    J. Mairal, F. Bach, J. Ponce, Sparse modeling for image and vision processing. Found. Trends Comput. Graph. Vis. 8(2–3), 85–283 (2014). https://doi.org/10.1561/0600000058. http://www.nowpublishers.com/article/Details/CGV-058
  39. 39.
    Maplesoft, a division of Waterloo Maple Inc. http://www.maplesoft.com
  40. 40.
    V. Papyan, M. Elad, Multi-scale patch-based image restoration. IEEE Trans. Image Process. 25(1), 249–261 (2016).  https://doi.org/https://doi.org/10.1109/TIP.2015.2499698 MathSciNetCrossRefGoogle Scholar
  41. 41.
    V. Papyan, Y. Romano, M. Elad, Convolutional neural networks analyzed via convolutional sparse coding. J. Mach. Learn. Res. 18(83), 1–52 (2017)MathSciNetGoogle Scholar
  42. 42.
    V. Papyan, J. Sulam, M. Elad, Working locally thinking globally: theoretical guarantees for convolutional sparse coding. IEEE Trans. Signal Process. 65(21), 5687–5701 (2017)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Y.C. Pati, R. Rezaiifar, P. Krishnaprasad, Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition, in Asilomar Conference on Signals, Systems and Computers (IEEE, New York, 1993), pp. 40–44Google Scholar
  44. 44.
    R. Quiroga, Spike sorting. Scholarpedia 2(12), 3583 (2007).  https://doi.org/https://doi.org/10.4249/scholarpedia.3583
  45. 45.
    Y. Romano, M. Elad, Boosting of image denoising algorithms. SIAM J. Imag. Sci. 8(2), 1187–1219 (2015)MathSciNetCrossRefMATHGoogle Scholar
  46. 46.
    Y. Romano, M. Elad, Patch-disagreement as away to improve K-SVD denoising, in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, New York, 2015), pp. 1280–1284Google Scholar
  47. 47.
    Y. Romano, M. Protter, M. Elad, Single image interpolation via adaptive nonlocal sparsity-based modeling. IEEE Trans. Image Process. 23(7), 3085–3098 (2014)MathSciNetCrossRefMATHGoogle Scholar
  48. 48.
    L.I. Rudin, S. Osher, E. Fatemi, Nonlinear total variation based noise removal algorithms. Physica D 60(1), 259–268 (1992). http://www.sciencedirect.com/science/article/pii/016727899290242F MathSciNetCrossRefMATHGoogle Scholar
  49. 49.
    C. Rusu, B. Dumitrescu, S. Tsaftaris, Explicit shift-invariant dictionary learning. IEEE Signal Process. Lett. 21, 6–9 (2014). http://www.schur.pub.ro/Idei2011/Articole/SPL_2014_shifts.pdf CrossRefGoogle Scholar
  50. 50.
    E. Smith, M.S. Lewicki, Efficient coding of time-relative structure using spikes. Neural Comput. 17(1), 19–45 (2005). http://dl.acm.org/citation.cfm?id=1119614 CrossRefMATHGoogle Scholar
  51. 51.
    A.M. Snijders, N. Nowak, R. Segraves, S. Blackwood, N. Brown, J. Conroy, G. Hamilton, A.K. Hindle, B. Huey, K. Kimura, S. Law, K. Myambo, J. Palmer, B. Ylstra, J.P. Yue, J.W. Gray, A.N. Jain, D. Pinkel, D.G. Albertson, Assembly of microarrays for genome-wide measurement of DNA copy number. Nat. Genet. 29(3), 263–264 (2001). https://doi.org/10.1038/ng754. https://www.nature.com/ng/journal/v29/n3/full/ng754.html
  52. 52.
    J. Sulam, M. Elad, Expected patch log likelihood with a sparse prior, in International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition (Springer, New York, 2015), pp. 99–111Google Scholar
  53. 53.
    J. Sulam, B. Ophir, M. Elad, Image denoising through multi-scale learnt dictionaries, in 2014 IEEE International Conference on Image Processing (ICIP) (IEEE, New York, 2014), pp. 808–812CrossRefGoogle Scholar
  54. 54.
    J.J. Thiagarajan, K.N. Ramamurthy, A. Spanias, Shift-invariant sparse representation of images using learned dictionaries, in IEEE Workshop on Machine Learning for Signal Processing, 2008, MLSP 2008 (2008), pp. 145–150  https://doi.org/https://doi.org/10.1109/MLSP.2008.4685470
  55. 55.
    J.A. Tropp, A.C. Gilbert, M.J. Strauss, Algorithms for simultaneous sparse approximation. Part i: greedy pursuit. Signal Process. 86(3), 572–588 (2006)MATHGoogle Scholar
  56. 56.
    J. Yang, J. Wright, T.S. Huang, Y. Ma, Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefMATHGoogle Scholar
  57. 57.
    G. Yu, G. Sapiro, S. Mallat, Solving inverse problems with piecewise linear estimators: From gaussian mixture models to structured sparsity. IEEE Trans. Image Process. 21(5), 2481–2499 (2012).  https://doi.org/https://doi.org/10.1109/TIP.2011.2176743 MathSciNetCrossRefMATHGoogle Scholar
  58. 58.
    M.D. Zeiler, D. Krishnan, G.W. Taylor, R. Fergus, Deconvolutional networks, in 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (IEEE, New York, 2010), pp. 2528–2535. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5539957 CrossRefGoogle Scholar
  59. 59.
    M. Zeiler, G. Taylor, R. Fergus, Adaptive deconvolutional networks for mid and high level feature learning, in 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2018–2025 (2011). doi:  https://doi.org/10.1109/ICCV.2011.6126474
  60. 60.
    D. Zoran, Y. Weiss, From learning models of natural image patches to whole image restoration, in 2011 IEEE International Conference on Computer Vision (ICCV) (IEEE, New York, 2011), pp. 479–486. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6126278 Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of MathematicsMassachusetts Institute of TechnologyCambridgeUSA
  2. 2.Department of Electrical EngineeringTechnion - Israel Institute of TechnologyHaifaIsrael
  3. 3.Department of Computer ScienceTechnion - Israel Institute of TechnologyHaifaIsrael

Personalised recommendations