Abstract
Clustering count vectors is a challenging task given their sparsity and high-dimensionality. An efficient generative model called EMSD has been recently proposed, as an exponential-family approximation to the Multinomial Scaled Dirichlet distribution, and has shown to offer excellent modeling capabilities in the case of sparse count data and to overcome some limitations of the frameworks based on the Dirichlet distribution. In this work, we develop an approximate Bayesian learning framework for the parameters of a finite mixture of EMSD using the Stochastic Expectation Propagation approach. In this approach, we maintain a global posterior approximation that is being updated in a local way, which reduces the memory consumption, important when making inference in large datasets. Experiments on both synthetic and real count data have been conducted to validate the effectiveness of the proposed algorithm in comparison to other traditional learning approaches. Results show that SEP produces comparable estimates with traditional approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)
Bouguila, N., Ziou, D.: Unsupervised learning of a finite discrete mixture model based on the multinomial dirichlet distribution: Application to texture modeling. In: Fred, A.L.N. (ed.) Pattern Recognition in Information Systems, Proceedings of the 4th International Workshop on Pattern Recognition in Information Systems, PRIS 2004, in conjunction with ICEIS 2004, Porto, Portugal, April 2004, pp. 118–127. INSTICC Press (2004)
Boyd-Graber, J., Hu, Y., Mimno, D., et al.: Applications of topic models. Found. Trends® Inf. Retrieval 11(2–3), 143–296 (2017)
Bui, T.D., Hernández-Lobato, J.M., Li, Y., Hernández-Lobato, D., Turner, R.E.: Training deep gaussian processes using stochastic expectation propagation and probabilistic backpropagation. arXiv preprint arXiv:1511.03405 (2015)
Elkan, C.: Clustering documents with an exponential-family approximation of the dirichlet compound multinomial distribution. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 289–296. ACM (2006)
Fan, W., Bouguila, N.: Expectation propagation learning of a dirichlet process mixture of beta-liouville distributions for proportional data clustering. Eng. Appl. Artif. Intell. 43, 1–14 (2015)
Gelman, A., et al.: Expectation propagation as a way of life: a framework for bayesian inference on partitioned data. arXiv preprint arXiv:1412.4869 (2017)
Hoffman, M.D., Blei, D.M., Wang, C., Paisley, J.: Stochastic variational inference. J. Mach. Learn. Res. 14(1), 1303–1347 (2013)
Lafferty, J., McCallum, A., Pereira, F.C.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data (2001)
Li, Y., Hernández-Lobato, J.M., Turner, R.E.: Stochastic expectation propagation. In: Advances in Neural Information Processing Systems, pp. 2323–2331 (2015)
Lochner, R.H.: A generalized dirichlet distribution in bayesian life testing. J. Roy. Stat. Soc. Ser. B (Methodol.) 37(1), 103–113 (1975)
Ma, Z., Leijon, A.: Expectation propagation for estimating the parameters of the beta distribution. In: 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2082–2085. IEEE (2010)
Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 142–150. Association for Computational Linguistics (2011)
Madsen, R.E., Kauchak, D., Elkan, C.: Modeling word burstiness using the dirichlet distribution. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 545–552. ACM (2005)
Margaritis, D., Thrun, S.: A bayesian multiresolution independence test for continuous variables. arXiv preprint arXiv:1301.2292 (2013)
Minka, T.: Estimating a dirichlet distribution (2000)
Minka, T.: Power ep. Technical report, Microsoft Research, Cambridge (2004)
Minka, T., Lafferty, J.: Expectation-propagation for the generative aspect model. In: Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence, pp. 352–359. Morgan Kaufmann Publishers Inc. (2002)
Minka, T.P.: Expectation propagation for approximate bayesian inference. In: Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, pp. 362–369. Morgan Kaufmann Publishers Inc. (2001)
Minka, T.P.: A family of algorithms for approximate Bayesian inference. Ph.D. thesis, Massachusetts Institute of Technology (2001)
Neal, R.M.: Probabilistic inference using markov chain monte carlo methods (1993)
Opper, M., Winther, O.: A bayesian approach to on-line learning. In: On-line Learning in Neural Networks, pp. 363–378 (1998)
Sumba, X., Zamzami, N., Bouguila, B.: Improving the edcm mixture model with expectation propagation. In: 2020 Association for the Advancement of Artificial Intelligence AAAI. FLAIRS 33 (2020)
Wong, T.T.: Alternative prior assumptions for improving the performance of naïve bayesian classifiers. Data Min. Knowl. Disc. 18(2), 183–213 (2009)
Zamzami, N., Bouguila, N.: Text modeling using multinomial scaled dirichlet distributions. In: Mouhoub, M., Sadaoui, S., Ait Mohamed, O., Ali, M. (eds.) IEA/AIE 2018. LNCS (LNAI), vol. 10868, pp. 69–80. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-92058-0_7
Zamzami, N., Bouguila, N.: An accurate evaluation of msd log-likelihood and its application in human action recognition. In: 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1–5. IEEE (2019)
Zamzami, N., Bouguila, N.: Hybrid generative discriminative approaches based on multinomial scaled dirichlet mixture models. Appl. Intell 49(11), 3783–3800 (2019)
Zamzami, N., Bouguila, N.: A novel scaled dirichlet-based statistical framework for count data modeling: unsupervised learning and exponential approximation. Pattern Recogn. 95, 36–47 (2019)
Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Advances in Neural Information Processing Systems, pp. 649–657 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Sumba, X., Zamzami, N., Bouguila, N. (2021). Clustering Count Data with Stochastic Expectation Propagation. In: Nguyen, N.T., Chittayasothorn, S., Niyato, D., Trawiński, B. (eds) Intelligent Information and Database Systems. ACIIDS 2021. Lecture Notes in Computer Science(), vol 12672. Springer, Cham. https://doi.org/10.1007/978-3-030-73280-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-73280-6_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-73279-0
Online ISBN: 978-3-030-73280-6
eBook Packages: Computer ScienceComputer Science (R0)