Nonparametric Localized Feature Selection via a Dirichlet Process Mixture of Generalized Dirichlet Distributions

  • Wentao Fan
  • Nizar Bouguila
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7665)


In this paper, we propose a novel Bayesian nonparametric statistical approach of simultaneous clustering and localized feature selection for unsupervised learning. The proposed model is based on a mixture of Dirichlet processes with generalized Dirichlet (GD) distributions, which can also be seen as an infinite GD mixture model. Due to the nature of Bayesian nonparametric approach, the problems of overfitting and underfitting are prevented. Moreover, the determination of the number of clusters is sidestepped by assuming an infinite number of clusters. In our approach, the model parameters and the local feature saliency are estimated simultaneously by variational inference. We report experimental results of applying our model to two challenging clustering problems involving web pages and tissue samples which contain gene expressions.


Mixture Models Clustering Dirichlet Process Nonparametric Bayesian Generalized Dirichlet Localized Feature Selection Variational Inference 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alizadeh, A.A., Eisen, M.B., Davis, R.E., et al.: Distinct Types of Diffuse Large B-cell Lymphoma Identified by Gene Expression Profiling. Nature 403, 503–511 (2000)CrossRefGoogle Scholar
  2. 2.
    Attias, H.: A Variational Bayes Framework for Graphical Models. In: Proc. of Neural Information Processing Systems (NIPS), pp. 209–215 (1999)Google Scholar
  3. 3.
    Bishop, C.M.: Variational Learning in Graphical Models and Neural Networks. In: Proc. of ICANN, pp. 13–22. Springer (1998)Google Scholar
  4. 4.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet Allocation. Journal of Machine Learning Research 3, 993–1022 (2003)zbMATHGoogle Scholar
  5. 5.
    Blei, D.M., Jordan, M.I.: Variational Inference for Dirichlet Process Mixtures. Bayesian Analysis 1, 121–144 (2005)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Bouguila, N., Ziou, D.: A Hybrid SEM Algorithm for High-Dimensional Unsupervised Learning Using a Finite Generalized Dirichlet Mixture. IEEE Transactions on Image Processing 15(9), 2657–2668 (2006)CrossRefGoogle Scholar
  7. 7.
    Bouguila, N., Ziou, D.: High-Dimensional Unsupervised Selection and Estimation of a Finite Generalized Dirichlet Mixture Model Based on Minimum Message Length. IEEE Transactions on PAMI 29(10), 1716–1731 (2007)CrossRefGoogle Scholar
  8. 8.
    Boutemedjet, S., Bouguila, N., Ziou, D.: A Hybrid Feature Extraction Selection Approach for High-Dimensional Non-Gaussian Data Clustering. IEEE Transactions on PAMI 31(8), 1429–1443 (2009)CrossRefGoogle Scholar
  9. 9.
    Constantinopoulos, C., Titsias, M., Likas, A.: Bayesian Feature and Model Selection for Gaussian Mixture Models. IEEE Trans. on PAMI 28(6), 1013–1018 (2006)CrossRefGoogle Scholar
  10. 10.
    Fan, W., Bouguila, N., Ziou, D.: Unsupervised Anomaly Intrusion Detection via Localized BayesianFeature Selection. In: Proc. of ICDM, pp. 1032–1037 (2011)Google Scholar
  11. 11.
    Fan, W., Bouguila, N., Ziou, D.: Variational Learning for Finite Dirichlet Mixture Models and Applications. IEEE Trans. Neural Netw. Learning Syst. 23(5), 762–774 (2012)CrossRefGoogle Scholar
  12. 12.
    Ferguson, T.S.: Bayesian Density Estimation by Mixtures of Normal Distributions. Recent Advances in Statistics 24, 287–302 (1983)MathSciNetGoogle Scholar
  13. 13.
    Figueiredo, M., Jain, A.: Unsupervised Learning of Finite Mixture Models. IEEE Transactions on PAMI 24(3), 381–396 (2002)CrossRefGoogle Scholar
  14. 14.
    Ji, Y., Wu, C., Liu, P., Wang, J., Coombes, K.R.: Applications of Beta-mixture Models in Bioinformatics. Bioinformatics 21(9), 2118–2122 (2005)CrossRefGoogle Scholar
  15. 15.
    Jordan, M.I., Ghahramani, Z., Jaakkola, T.S., Saul, L.K.: An Introduction to Variational Methods for Graphical Models. Machine Learning 37(2), 183–233 (1999)zbMATHCrossRefGoogle Scholar
  16. 16.
    Law, M.H.C., Figueiredo, M.A.T., Jain, A.K.: Simultaneous Feature Selection and Clustering Using Mixture Models. IEEE Trans. on PAMI 26(9), 1154–1166 (2004)CrossRefGoogle Scholar
  17. 17.
    Li, Y., Dong, M., Hua, J.: Simultaneous Localized Feature Selection and Model Detection for Gaussian Mixtures. IEEE Transactions on PAMI 31, 953–960 (2009)CrossRefGoogle Scholar
  18. 18.
    Ma, Z., Leijon, A.: Bayesian Estimation of Beta Mixture Models with Variational Inference. IEEE Transactions on PAMI 33(11), 2160–2173 (2011)CrossRefGoogle Scholar
  19. 19.
    McLachlan, G.J., Khan, N.: On a Resampling Approach for Tests on the Number of Clusters with Mixture Model-based Clustering of Tissue Samples. J. Multivar. Anal. 90(1), 90–105 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Neal, R.M.: Markov Chain Sampling Methods for Dirichlet Process Mixture Models. Journal of Computational and Graphical Statistics 9(2), 249–265 (2000)MathSciNetGoogle Scholar
  21. 21.
    Sethuraman, J.: A Constructive Definition of Dirichlet Priors. Statistica Sinica 4, 639–650 (1994)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Teh, Y.W., Jordan, M.I., Beal, M.J., Blei, D.M.: Hierarchical Dirichlet Processes. Journal of the American Statistical Association 101, 705–711 (2004)MathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Wentao Fan
    • 1
  • Nizar Bouguila
    • 1
  1. 1.Concordia Institute for Information Systems EngineeringConcordia UniversityCanada

Personalised recommendations