Abstract
Method of moments (MoM) has recently become an appealing alternative to standard iterative approaches like Expectation Maximization (EM) to learn latent variable models. In addition, MoM-based algorithms come with global convergence guarantees in the form of finite sample bounds. However, given enough computation time, by using restarts and heuristics to avoid local optima, iterative approaches often achieve better performance. We believe that this performance gap is in part due to the fact that MoM-based algorithms can output negative probabilities. By constraining the search space, we propose a non-negative spectral algorithm (NNSpectral) avoiding computing negative probabilities by design. NNSpectral is compared to other MoM-based algorithms and EM on synthetic problems of the PAutomaC challenge. Not only, NNSpectral outperforms other MoM-based algorithms, but also, achieves very competitive results in comparison to EM.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anandkumar, A., Ge, R., Hsu, D., Kakade, S.M., Telgarsky, M.: Tensor decompositions for learning latent variable models (2012). arXiv preprint arXiv:1210.7559
Bailly, R., Denis, F.: Absolute convergence of rational series is semi-decidable. Inf. Comput. 209(3), 280–295 (2011)
Bailly, R., Habrard, A., Denis, F.: A spectral approach for probabilistic grammatical inference on trees. In: Hutter, M., Stephan, F., Vovk, V., Zeugmann, T. (eds.) Algorithmic Learning Theory. LNCS, vol. 6331, pp. 74–88. Springer, Heidelberg (2010)
Balle, B.: Learning finite-state machines: algorithmic and statistical aspects. Ph.D. thesis (2013)
Balle, B., Hamilton, W., Pineau, J.: Methods of moments for learning stochastic languages: unified presentation and empirical comparison. In: Proceedings of ICML-14, pp. 1386–1394 (2014)
Balle, B., Quattoni, A., Carreras, X.: Local loss optimization in operator models: a new insight into spectral learning. In: Proceedings of ICML-12 (2012)
Carlyle, J.W., Paz, A.: Realizations by stochastic finite automata. J. Comput. Syst. Sci. 5(1), 26–40 (1971)
Cohen, S.B., Stratos, K., Collins, M., Foster, D.P., Ungar, L.H.: Experiments with spectral learning of latent-variable PCFGs. In: Proceedings of HLT-NAACL-13, pp. 148–157 (2013)
Denis, F., Esposito, Y.: On rational stochastic languages. Fundamenta Informaticae 86(1), 41–77 (2008)
Gillis, N.: The Why and How of nonnegative matrix factorization. ArXiv e-prints, January 2014
Glaude, H., Pietquin, O., Enderli, C.: Subspace identification for predictive state representation by nuclear norm minimization. In: Proceedings of ADPRL-14 (2014)
Guterman, A.E.: Rank and determinant functions for matrices over semirings. In: Young, N., Choi, Y. (eds.) Surveys in Contemporary Mathematics, pp. 1–33. Cambridge University Press, Cambridge (2007)
Gybels, M., Denis, F., Habrard, A.: Some improvements of the spectral learning approach for probabilistic grammatical inference. In: Proceedings of ICGI-12, vol. 34, pp. 64–78 (2014)
Lin, C.J.: Projected gradient methods for nonnegative matrix factorization. Neural Comput. 19(10), 2756–2779 (2007)
Thon, M., Jaeger, H.: Links between multiplicity automata, observable operator models and predictive state representations – a unified learning framework learning framework. J. Mach. Learn. Res. (2015, to appear)
Vavasis, S.A.: On the complexity of nonnegative matrix factorization. SIAM J. Optim. 20(3), 1364–1377 (2009)
Verwer, S., Eyraud, R., de la Higuera, C.: Results of the pautomac probabilistic automaton learning competition. J. Mach. Learn. Res. - Proc. Track 21, 243–248 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Glaude, H., Enderli, C., Pietquin, O. (2015). Non-negative Spectral Learning for Linear Sequential Systems. In: Arik, S., Huang, T., Lai, W., Liu, Q. (eds) Neural Information Processing. ICONIP 2015. Lecture Notes in Computer Science(), vol 9490. Springer, Cham. https://doi.org/10.1007/978-3-319-26535-3_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-26535-3_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-26534-6
Online ISBN: 978-3-319-26535-3
eBook Packages: Computer ScienceComputer Science (R0)