Linear Time Model Selection for Mixture of Heterogeneous Components
Our main contribution is to propose a novel model selection methodology, expectation minimization of description length (EMDL), based on the minimum description length (MDL) principle. EMDL makes a significant impact on the combinatorial scalability issue pertaining to the model selection for mixture models having types of components. A goal of such problems is to optimize types of components as well as the number of components. One key idea in EMDL is to iterate calculations of the posterior of latent variables and minimization of expected description length of both observed data and latent variables. This enables EMDL to compute the optimal model in linear time with respect to both the number of components and the number of available types of components despite the fact that the number of model candidates exponentially increases with the numbers. We prove that EMDL is compliant with the MDL principle and enjoys its statistical benefits.
Keywordsheterogeneous mixture model model selection minimum description length
Unable to display preview. Download preview PDF.
- 2.Akaike, H.: Information theory and an extension of the maximum likelihood principle. In: Petrov, B.N., Caski, F. (eds.) Proceedings of the 2nd International Symposium on Information Theory, pp. 267–281 (1973)Google Scholar
- 3.Wallace, C.S., Dowe, D.L.: Intrinsic classification by mml - the snob program. In: Proceedings of the 7th Australian Joint Conf. on Artificial Intelligence, pp. 37–44 (1994)Google Scholar
- 5.Edwards, R.T., Dowe, D.L.: Single factor analysis in mml mixture modelling. In: Wu, X., Kotagiri, R., Korb, K.B. (eds.) PAKDD 1998. LNCS, vol. 1394, pp. 96–109. Springer, Heidelberg (1998)Google Scholar
- 7.Dennis, J.E., Torczon, V.J.: Derivative-free pattern search methods for multidisciplinary design problems. In: Proceedings of the 5th AIAA/ USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization (1994)Google Scholar
- 8.Momma, M., Bennett, K.P.: A pattern search method for model selection of support vector regression. In: Proceedings of the SIAM International Conference on Data Mining. SIAM, Philadelphia (2002)Google Scholar
- 9.Ghahramani, Z., Beal, M.J.: Variational inference for bayesian mixtures of factor analysers. In: Advances in Neural Information Processing Systems 12, pp. 449–455. MIT Press, Cambridge (2000)Google Scholar
- 11.Grunwald, P.D., Myung, I.J., Pitt, M.A.: Advances in minimum description length. MIT Press, Cambridge (2005)Google Scholar
- 18.Ueda, N., Nakano, R.: EM algorithm with split and merge operations for mixture models. IEICE transactions on information and systems E83-D(12), 2047–2055 (2000)Google Scholar
- 19.Asuncion, A., Newman, D.: UCI machine learning repository (2007)Google Scholar