Compressive System Identification

  • Avishy Y. CarmiEmail author
Part of the Signals and Communication Technology book series (SCT)


The first part of this chapter presents a novel Kalman filtering-based method for estimating the coefficients of sparse, or more broadly, compressible autoregressive models using fewer observations than normally required. By virtue of its (unscented) Kalman filter mechanism, the derived method essentially addresses the main difficulties attributed to the underlying estimation problem. In particular, it facilitates sequential processing of observations and is shown to attain a good recovery performance, particularly under substantial deviations from ideal conditions, those which are assumed to hold true by the theory of compressive sensing. In the remaining part of this chapter we derive a few information-theoretic bounds pertaining to the problem at hand. The obtained bounds establish the relation between the complexity of the autoregressive process and the attainable estimation accuracy through the use of a novel measure of complexity. This measure is suggested herein as a substitute to the generally incomputable restricted isometric property.


  1. 1.
    Alan P (1983) Forecasting with univariate Box-Jenkins models: concepts and cases. Wiley, New YorkGoogle Scholar
  2. 2.
    Angelosante D, Bazerque JA, Giannakis GB (2010) Online adaptive estimation of sparse signals: where RLS meets the \(l_1\)-norm. IEEE Trans Signal Process 58:3436–3447MathSciNetCrossRefGoogle Scholar
  3. 3.
    Angelosante D, Giannakis GB, Grossi E (2009) Compressed sensing of time-varying signals. Proceedings of the 16th international conference on digital signal processingGoogle Scholar
  4. 4.
    Asif MS, Charles A, Romberg J, Rozell C (2011)Estimation and dynamic updating of time-varying signals with sparse variations. In: International conference on acoustics, speech and signal processing (ICASSP), pp 3908–3911Google Scholar
  5. 5.
    Asif MS, Romberg J (2009) Dynamic updating for sparse time varying signals. In: Proceedings of the conference on information sciences and systems, pp 3–8Google Scholar
  6. 6.
    Baraniuk RG, Davenport MA, Ronald D, Wakin MB (2008) A simple proof of the restricted isometry property for random matrices. Constr Approx 28:253–263MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Benveniste A, Basseville M, Moustakides GV (1987) The asymptotic local approach to change detection and model validation. IEEE Trans Autom Control 32:583–592MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Blumensath T, Davies M (2009) Iterative hard thresholding for compressed sensing. Appl Comput Harmon Anal 27:265–274MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Bosch-Bayard J. et al (2005) Estimating brain functional connectivity with sparse multivariate autoregression. Philos Trans R Soc 360:969–981Google Scholar
  10. 10.
    Brockwell PJ, Davis RA (2009) Time Series: theory and methods, Springer, New YorkGoogle Scholar
  11. 11.
    Candes E, Tao T (2007) The Dantzig selector: statistical estimation when p is much larger than n. Ann Stat 35:2313–2351MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Candes EJ (2008) The restricted isometry property and its implications for compressed sensing. C R Math 346:589–592MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Candes EJ, Eldar YC, Needell D, Randall P (2011) Compressed sensing with coherent and redundant dictionaries. Appl Comput Harmon Anal 31:59–73MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Candes EJ, Romberg J, Tao T (2006) Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans Inf Theory 52:489–509MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Candes EJ, Tao T (2005) Decoding by linear programming. IEEE Trans Inf Theory 51:4203–4215MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Candes EJ, Tao T (2006) Near-optimal signal recovery from random projections: universal encoding strategies? IEEE Trans Inf Theory 52:5406–5425MathSciNetCrossRefGoogle Scholar
  17. 17.
    Candes EJ, Wakin MB (2008) An introduction to compressive sampling. IEEE Signal Process Mag 25:21–31CrossRefGoogle Scholar
  18. 18.
    Carmi A, Gurfil P, Kanevsky D (2008) A simple method for sparse signal recovery from noisy observations using Kalman filtering. Technical Report RC24709, Human Language Technologies, IBMGoogle Scholar
  19. 19.
    Carmi A, Gurfil P, Kanevsky D (2010) Methods for sparse signal recovery using Kalman filtering with embedded pseudo-measurement norms and quasi-norms. IEEE Trans Signal Process 58:2405–2409MathSciNetCrossRefGoogle Scholar
  20. 20.
    Carmi A, Mihaylova L, Kanevsky D (2012) Unscented compressed sensing. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing (ICASSP)Google Scholar
  21. 21.
    Charles A, Asif MS, Romberg J, Rozell C (2011) Sparsity penalties in dynamical system estimation. In: Proceedings of the conference on information sciences and systems, pp 1–6Google Scholar
  22. 22.
    Chen S, Billings SA, Luo W (1989) Orthogonal least squares methods and their application to non-linear system identification. Int J Contro 50:1873–1896MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Chen SS, Donoho DL, Saunders MA (1998) Atomic decomposition by basis pursuit. SIAM J Sci Comput 20:33–61MathSciNetCrossRefGoogle Scholar
  24. 24.
    Davis G, Mallat S, Avellaneda M (1997) Greedy adaptive approximation. Constr Approx 13:57–98MathSciNetzbMATHGoogle Scholar
  25. 25.
    Deurschmann J, Bar-Itzhack I, Ken G (1992) Quaternion normalization in spacecraft attitude determination. In: Proceedings of the AIAA/AAS astrodynamics conference, pp 27–37Google Scholar
  26. 26.
    Donoho DL (2006) Compressed sensing. IEEE Trans Inf Theory 52:1289–1306MathSciNetCrossRefGoogle Scholar
  27. 27.
    Durate MF, Davenport MA, Takhar D, Laska JN, Sun T, Kelly KF, Baraniuk RG (2008) Single pixel imaging via compressive sampling, IEEE Signal Process MagGoogle Scholar
  28. 28.
    Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32:407–499MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Friedman N, Nachman I, Peer D (1999) Learning Bayesian network structure from massive datasets: The sparse candidate algorithm. In: Proceedings of the fifteenth conference annual conference on uncertainty in, artificial intelligence (UAI-99), pp 206–215.Google Scholar
  30. 30.
    Granger CWJ (1969) Investigating causal relations by econometric models and cross-spectral methods. Econometrica 37:424–438CrossRefGoogle Scholar
  31. 31.
    Haufe S, Muller K, Nolte G, Kramer N (2008) Sparse causal discovery in multivariate time series, NIPS Workshop on causalityGoogle Scholar
  32. 32.
    Hirotugu A (1974) A new look at the statistical model identification. IEEE Trans Autom Control 19:716–723CrossRefzbMATHGoogle Scholar
  33. 33.
    James GM, Radchenko P, Lv J (2009) DASSO: connections between the Dantzig selector and LASSO. J Roy Stat Soc 71:127–142MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Ji S, Xue Y, Carin L (June 2008) Bayesian compressive sensing. IEEE Trans Signal Process 56:2346–2356MathSciNetCrossRefGoogle Scholar
  35. 35.
    Julier SJ, LaViola JJ (2007) On Kalman filtering with nonlinear equality constraints. IEEE Trans Signal Process 55:2774–2784MathSciNetCrossRefGoogle Scholar
  36. 36.
    Julier SJ, Uhlmann JK (1997) A new extension of the Kalman filter to nonlinear systems. In: Proceedings of the international symposium on aerospace/defense sensing, simulation and controls, pp 182–193Google Scholar
  37. 37.
    Kailath T (1980) Linear Systems. Prentice Hall, Englewood CliffsGoogle Scholar
  38. 38.
    Kalouptsidis N, Mileounis G, Babadi B, Tarokh V (2011) Adaptive algorithms for sparse system identification. Signal Proc 91:1910–1919CrossRefzbMATHGoogle Scholar
  39. 39.
    Laska JN, Boufounos PT, Davenport MA, Baraniuk RG (2011) Democracy in action: quantization, saturation, and compressive sensing. Appl Comput Harmon Anal 31:429–443MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Mallat S, Zhang Z (1993) Matching pursuits with time-frequency dictionaries. IEEE Trans Signal Process 4:3397–3415CrossRefGoogle Scholar
  41. 41.
    Mendel JM (1995) Lessons in estimation theory for signal processing, communications, and control. Prentice Hall, Englewood-CliffsGoogle Scholar
  42. 42.
    Pati YC, Rezifar R, Krishnaprasad PS (1993) Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In: Proceedings of the 27th asilomar conf. on signals, systems and comput., pp 40–44Google Scholar
  43. 43.
    Rudelson M (1999) Random vectors in the isotropic position. J Funct Anal 164:60–72MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Rudelson M, Vershynin R (2005) Geometric approach to error correcting codes and reconstruction of signals. Int Math Res Not 64:4019–4041MathSciNetCrossRefGoogle Scholar
  45. 45.
    Sanadaji BM, Vincent TL, Wakin MB, Toth, Poola K (2011) Compressive System Identification of LTI and LTV ARX models. In: Proceedings of the IEEE conference on decision and control and european control conference (CDC-ECC), pp 791–798Google Scholar
  46. 46.
    Schwarz GE (1978) Estimating the dimension of a model. Ann Stat 6:461–464CrossRefzbMATHGoogle Scholar
  47. 47.
    Sokal AD (1989) Monte carlo methods in statistical mechanics: foundations and new algorithms. Cours de Troisieme Cycle de la Physique en Suisse Romande, LaussaneGoogle Scholar
  48. 48.
    Tibshirani R (1996) Regression shrinkage and selection via the LASSO. J Roy Stat Soc B Method, 58:267–288Google Scholar
  49. 49.
    Tipping ME (2001) Sparse Bayesian learning and the relevance vector machine. Int J Mach Learn Res 1:211–244MathSciNetzbMATHGoogle Scholar
  50. 50.
    Tropp JA (2004) Greed is good: Algorithmic results for sparse approximation. IEEE Trans Inf Theory 50:2231–2242MathSciNetCrossRefGoogle Scholar
  51. 51.
    Tropp JA, Gilbert AC (2007) Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans Inf Theory 53:4655–4666MathSciNetCrossRefGoogle Scholar
  52. 52.
    Vaswani N (2008) Kalman filtered compressed sensing. In: Proceedings of the IEEE international conference on image processing (ICIP) pp 893–896Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.School of Mechanical and Aerospace EngineeringNanyang Technological UniversitySingaporeSingapore

Personalised recommendations