Advertisement

Compressible Reparametrization of Time-Variant Linear Dynamical Systems

  • Nico PiatkowskiEmail author
  • François Schnitzler
Chapter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9580)

Abstract

Linear dynamical systems (LDS) are applied to model data from various domains—including physics, smart cities, medicine, biology, chemistry and social science—as stochastic dynamic process. Whenever the model dynamics are allowed to change over time, the number of parameters can easily exceed millions. Hence, an estimation of such time-variant dynamics on a relatively small—compared to the number of variables—training sample typically results in dense, overfitted models. Existing regularization techniques are not able to exploit the temporal structure in the model parameters. We investigate a combined reparametrization and regularization approach which is designed to detect redundancies in the dynamics in order to leverage a new level of sparsity. On the basis of ordinary linear dynamical systems, the new model, called ST-LDS, is derived and a proximal parameter optimization procedure is presented. Differences to \(l_1\)-regularization-based approaches are discussed and an evaluation on synthetic data is conducted. The results show, that the larger the considered system, the more sparsity can be achieved, compared to plain \(l_1\)-regularization.

Keywords

Transition Matrice Exponential Family Dynamic Bayesian Network Linear Dynamical System Proximal Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgement

This work has been supported by Deutsche Forschungsgemeinschaft (DFG) within the Collaborative Research Center SFB 876 “Providing Information by Resource-Constrained Data Analysis”, project A1.

References

  1. 1.
    Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146(1–2), 459–494 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Cai, T., Liu, W., Luo, X.: A constrained \(\ell _1\) minimization approach to sparse precision matrix estimation. J. Am. Stat. Assoc. 106(494), 594–607 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Dagum, P., Galper, A., Horvitz, E.: Dynamic network models for forecasting. In: Proceedings of the 8th Annual Conference on Uncertainty in Artificial Intelligence, pp. 41–48 (1992)Google Scholar
  4. 4.
    Fearnhead, P.: Exact Bayesian curve fitting and signal segmentation. IEEE Trans. Signal Process. 53(6), 2160–2166 (2005)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Friedman, J., Hastie, T., Tibshirani, R.: Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9(3), 432–441 (2008)CrossRefzbMATHGoogle Scholar
  6. 6.
    Han, F., Liu, H.: Transition matrix estimation in high dimensional time series. In: Proceedings of the 30th International Conference on Machine Learning, pp. 172–180 (2013)Google Scholar
  7. 7.
    Kolar, M., Song, L., Ahmed, A., Xing, E.P.: Estimating time-varying networks. Ann. Appl. Stat. 4(1), 94–123 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Lauritzen, S.L.: Graphical Models. Oxford University Press, Oxford (1996)zbMATHGoogle Scholar
  9. 9.
    Magnus, J.R., Neudecker, H.: Matrix Differential Calculus with Applications in Statistics and Econometrics, 2nd edn. Wiley, Chichester (1999)zbMATHGoogle Scholar
  10. 10.
    Rodrigues de Morais, S., Aussem, A.: A novel scalable and data efficient feature subset selection algorithm. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008, Part II. LNCS (LNAI), vol. 5212, pp. 298–312. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  11. 11.
    Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Opt. 1(3), 127–239 (2014)CrossRefGoogle Scholar
  12. 12.
    Piatkowski, N., Lee, S., Morik, K.: Spatio-temporal random fields: compressible representation and distributed estimation. Mach. Learn. 93(1), 115–139 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Ravikumar, P., Wainwright, M.J., Lafferty, J.D.: High-dimensional ising model selection using \(\ell _1\)-regularized logistic regression. Ann. Appl. Stat. 38(3), 1287–1319 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Song, L., Kolar, M., Xing, E.P.: Time-varying dynamic Bayesian networks. Adv. Neural Inf. Process. Syst. 22, 1732–1740 (2009)Google Scholar
  15. 15.
    Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., Knight, K.: Sparsity and smoothness via the fused lasso. J. Royal Stat. Soc. Ser. B 67(1), 91–108 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Trabelsi, G., Leray, P., Ben Ayed, M., Alimi, A.M.: Dynamic MMHC: a local search algorithm for dynamic bayesian network structure learning. In: Tucker, A., Höppner, F., Siebes, A., Swift, S. (eds.) IDA 2013. LNCS, vol. 8207, pp. 392–403. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  17. 17.
    Wainwright, M.J., Jordan, M.I.: Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn. 1(1–2), 1–305 (2008)zbMATHGoogle Scholar
  18. 18.
    Wong, E., Awate, S., Fletcher, T.: Adaptive sparsity in gaussian graphical models. JMLR W&CP 28, 311–319 (2013)Google Scholar
  19. 19.
    Xuan, X., Murphy, K.: Modeling changing dependency structure in multivariate time series. In: Proceedings of the 24th International Conference on Machine Learning, pp. 1055–1062. ACM (2007)Google Scholar
  20. 20.
    Zhou, S., Lafferty, J.D., Wasserman, L.A.: Time varying undirected graphs. Mach. Learn. 80(2–3), 295–319 (2010)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Zhou, S., Rütimann, P., Xu, M., Bühlmann, P.: High-dimensional covariance estimation based on gaussian graphical models. J. Mach. Learn. Res. 12, 2975–3026 (2011)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Artificial Intelligence GroupTU DortmundDortmundGermany
  2. 2.TechnicolorCesson-sévignéFrance

Personalised recommendations