Abstract
Neural networks that are embedded with prior knowledge of the distribution of the original signal in applications of compressed sensing have attracted increasing attention. However, the maximal probability of the desired output by a neural network cannot guarantee that the statistical distribution of the recovered signal is consistent with the statistical distribution of the original signal. In this paper, we combine neural networks with information geometry to study the recovery of sparse signals that satisfy a certain distribution. We construct the geodesic distance between the distribution of the original signal and distribution of the recovered signal as the input for the neural network. Experiments show that the proposed method has a better reconstruction quality compared with existing algorithms.
Similar content being viewed by others
References
S. Amari, Natural gradient works efficiently in learning. Neural Comput. 10(2), 251–276 (1998)
S. Amari, Information geometry on hierarchy of probability distributions. IEEE Trans. Inf. Theory 47(5), 1701–1711 (2001)
S.I. Amari, Information geometry and its applications. J. Math. Psychol. 49(1), 101–102 (2005)
S. Amari, Information geometry in optimization, machine learning and statistical inference. Front. Electr. Electron. Eng. China 5(3), 241–260 (2010)
S. Amari, M. Kawanabe, Information geometry of estimating functions in semi-parametric statistical models. Bernoulli 3(1), 29–54 (1997)
T. Ardeshiri, K. Granstrom, E. Ozkan, U. Orguner, Greedy reduction algorithms for mixtures of exponential family. IEEE Signal Process. Lett. 22(6), 676–680 (2015)
K.A. Arwini, C.T.J. Dodson, A.J. Doig, W.W. Sampson, J. Scharcanski, S. Felipussi, J.M. Morel, F. Takens, B. Teissier, Information Geometry: Near Randomness and Near Independence (Springer, Berlin, 2008)
R.G. Baraniuk, V. Cevher, M.F. Duarte, C. Hegde, Model-based compressive sensing. IEEE Trans. Inf. Theory 56(4), 1982–2001 (2010)
D. Baron, S. Sarvotham, R.G. Baraniuk, Bayesian compressive sensing via belief propagation. IEEE Signal Process. 58(3), 269–280 (2010)
T. Blumensath, M.E. Davies, Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27(3), 265–274 (2009)
A. Buecher, J. Segers, Maximum likelihood estimation for the Frechet distribution based on block maxima extracted from a time series. Bernoulli 24(2), 1427–1462 (2018)
H.Q. Bui, C.N.H. La, M.N. Da, A fast tree-based algorithm for compressed sensing with sparse-tree prior. Signal Process. 108(10), 628–641 (2015)
L.L. Campbell, The relation between information theory and the differential, geometry approach to statistics. Inf. Sci. 35(3), 199–210 (1985)
E.J. Candes, T. Tao, Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)
E.J. Candes, T. Tao, Near-optimal signal recovery from random projections: universal encoding strategies. IEEE Trans. Inf. Theory 52(12), 5406–5425 (2006)
E.J. Candes, M.B. Wakin, An introduction to compressive sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008)
E.J. Candes, J. Romberg, T. Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)
E.J. Candès, J.K. Romberg, T. Tao, Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006)
A.Y. Carmi, L. Mihaylova, S.J. Godsill, Compressed Sensing and Sparse Filtering (Springer, Berlin, 2014)
S.S. Chen, D.L. Donoho, M.A. Saunders, Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)
I. Daubechies, M. Defrise, D. Mol, An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(2), 1413–1457 (2004)
S. Dineen, A.M. Society, Probability theory in finance: a mathematical guide to the Black–Scholes formula. Am. Math. Soc. 10(8), 164–187 (2011)
D.L. Donoho, Compressed sensing. IEEE Trans. Inf. Theory 52(12), 1289–1306 (2006)
D.L. Donoho, Y. Tsaig, I. Drori, J.L. Starck, Sparse solution of underdetermined systems of linear equations by stagewise orthogonal matching pursuit. IEEE Trans. Inf. Theory 58(22), 1094–1121 (2012)
T.V. Duong, D.Q. Phung, H.H. Bui, S. Venkatesh, Human behavior recognition with generic exponential family duration modeling in the hidden semi-Markov model, in International Conference on Pattern Recognition (2006), pp. 202–208
O.D. Escoda, L. Granai, P. Vandergheynst, On the use of a priori information for sparse signal approximations. IEEE Trans. Signal Process. 54(9), 3468–3482 (2006)
M.A.T. Figueiredo, R.D. Nowak, S.J. Wright, Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1(4), 586–597 (2007)
A. Flinth, Optimal choice of weights for sparse recovery with prior information. IEEE Trans. Inf. Theory 62(7), 4276–4284 (2016)
B.R. Frieden, Science from Fisher Information: A Unification (Cambridge University Press, Cambridge, 2004)
M.P. Friedlander, H. Mansour, R. Saab, O. Yilmaz, Recovering compressively sampled signals using partial support information. IEEE Trans. Inf. Theory 58(2), 1122–1134 (2012)
R. Garg, R. Khandekar, Gradient descent with sparsification: an iterative algorithm for sparse recovery with restricted isometry property, in Proceedings of the 26th International Conference On Machine Learning (2009), pp. 337–344
A. Gilbert, S. Guha, P. Indyk, S. Muthukrishnan, M. Strauss, Near-optimal sparse Fourier representations via sampling, in Proceedings of the 34th Annual ACM Symposium on Theory of Computing (2000), pp. 152–161
G. Gormode, S. Muthukrishan, Combinatorial algorithms for Compressed sensing, in Proceedings of the 40th Annual Conference on Information Sciences and Systems (2006), pp. 280–294
Q. Huynh-Thu, M. Ghanbari, Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 44(13), 800 (2008)
R.E. Kass, P.W. Vos, Geometrical Foundations of Asymptotic Inference (Wiley, New York, 2011)
M.A. Khajehnejad, W. Xu, A.S. Avestimehr, B. Hassibi, Analyzing weighted l(1) minimization for sparse recovery with nonuniform sparse models. IEEE Trans. Signal Process. 59(5), 1985–2001 (2011)
C. La, M.N. Do, Signal reconstruction using sparse tree representation, in Proceedings of Wavelets XI at SPIE Optics and Photonics (2005), pp. 120–125
C. La, M.N. Do, Tree-based orthogonal matching pursuit algorithm for signal reconstruction, in IEEE International Conference on Image Processing (2006), pp. 1277–1285
S.G. Mallat, Z. Zhang, Matching pursuits with time-frequency dictionaries. IEEE Trans. Inf. Theory 41(12), 3397–3415 (1993)
M.L. Menendez, D. Morales, L. Pardo, M. Salicrij, Statistical tests based on geodesic distances. Appl. Math. Lett. 8(1), 65–69 (1995)
D. Merhej, C. Diab, M. Khalil, R. Prost, Embedding prior knowledge within compressed sensing by neural networks. IEEE Trans. Neural Netw. 22(10), 1638–1649 (2011)
S. Muthukrishnan, Data streams: algorithms and applications. Found. Trends Theor. Comput. Sci. 1(2), 413-413 (2003)
D. Needell, J.A. Tropp, CoSaMP: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal. 26(3), 301–321 (2009)
D. Needell, R. Vershynin, Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit. Found. Comput. Math. 9(3), 317–334 (2009)
F. Nielsen, Pattern learning and recognition on statistical manifolds. Int. Workshop Similarity Based Pattern Recognit. 79(53), 1–25 (2013)
F. Nielsen, V. Garcia. Statistical exponential families: a digest with flash cards (2009). arXiv:0911.4863
B.J. Oommen, A. Thomas, “Anti-Bayesian” parametric pattern classification using order statistics criteria for some members of the exponential family. Pattern Recogn. 47(1SI), 40–55 (2014)
H. Palangi, R. Ward, L. Deng, Distributed compressive sensing: a deep learning approach. IEEE Trans. Signal Process. 64(17), 4504–4518 (2016)
H. Palangi, R. Ward, L. Deng, Convolutional deep stacking networks for distributed compressive sensing. Signal Process. 131, 181–189 (2017)
I. Rish, G. Grabarnik, Sparse signal recovery with exponential-family noise, in 47th Annual Allerton Conference on Communication, Control, and Computing (2009), pp. 60–66
M.W. Seeger, Bayesian inference and optimal design for the sparse linear model. J. Mach. Learn. Res. 9, 759–813 (2008)
J.A. Tropp, A.C. Gilbert, Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Inf. Theory 53(12), 4655–4666 (2007)
N. Vaswani, W. Lu, Modified-CS: modifying compressive sensing for problems with partially known support. IEEE Trans. Signal Process. 58(9), 4595–4607 (2010)
J. Wang, S. Kwon, B. Shim, Generalized orthogonal matching pursuit. IEEE Signal Process. 60(1212), 6202–6216 (2012)
Y. Xiang, J. Donley, E. Seletskaia, S. Shingare, J. Kamerud, B. Gorovits, A simple approach to determine a curve fitting model with a correct weighting function for calibration curves in quantitative ligand binding assays. AAPS J. 20(3), 45 (2018)
Y. Yang, Polynomial curve fitting and lagrange interpolation. Math. Comput. Educ. 47(3), 224 (2013)
H. Zayyani, M. Babaie-Zadeh, C. Jutten, Bayesian Pursuit algorithm for sparse representation, in 2009 IEEE International Conference on Acoustics, Speech and Signal Processing (2009), pp. 1549–1552
Y. Zhang, On Theory of Compressive Sensing via l 1-Minimization Simple Derivations and Extensions. Rice CAAM Department Technical Report (2008)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, M., Xiao, CB., Ning, ZH. et al. Neural Networks for Compressed Sensing Based on Information Geometry. Circuits Syst Signal Process 38, 569–589 (2019). https://doi.org/10.1007/s00034-018-0869-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00034-018-0869-6