Type-2 Fuzzy Topic Models

  • Jia Zeng
  • Zhi-Qiang Liu
Part of the Studies in Computational Intelligence book series (SCI, volume 591)


Latent Dirichlet allocation (LDA) is an important hierarchical Bayesian model for probabilistic topic modeling, which attracts worldwide interests and touches on many important applications in text mining, computer vision and computational biology. We first introduce a novel inference algorithm, called belief propagation (BP), for learning LDA, and then introduce how to speed up BP for fast topic modeling tasks. Following the “bag-of-words” (BOW) representation for video sequences, this chapter also introduces novel type-2 fuzzy topic models (T2 FTM) to recognize human actions. In traditional topic models (TM) for visual recognition, each video sequence is modeled as a “document” composed of spatial–temporal interest points called visual “word”. Topic models automatically assign a “topic” label to explain the action category of each word, so that each video sequence becomes a mixture of action topics for recognition. The T2 FTM differs from previous TMs in that it uses type-2 fuzzy sets (T2 FS) to encode the semantic uncertainty of each topic. We ca use the primary membership function (MF) to measure the degree of uncertainty that a document or a visual word belongs to a specific action topic, and use the secondary MF to evaluate the fuzziness of the primary MF itself. In this chapter, we implement two T2 FTMs: (1) interval T2 FTM (IT2 FTM) with all secondary grades equal one, and (2) vertical-slice T2 FTM (VT2 FTM) with unequal secondary grades based on our prior knowledge. To estimate parameters in T2 FTMs, we derive the efficient message-passing algorithm. Experiments on KTH and Weizmann human action data sets demonstrate that T2 FTMs are better than TMs to encode visual word uncertainties for human action recognition.


  1. 1.
    AlSumait, L., Barbará, D., Domeniconi, C.: On-line LDA: adaptive topic models for mining text streams with applications to topic detection and tracking. In: ICDM, pp. 3–12 (2008)Google Scholar
  2. 2.
    Asuncion, A., Welling, M., Smyth, P., Teh, Y.W.: On smoothing and inference for topic models. In: UAI, pp. 27–34 (2009)Google Scholar
  3. 3.
    Asuncion, A.: Approximate Mean Field for Dirichlet-Based Models. In: ICML Workshop on Topic Models (2010)Google Scholar
  4. 4.
    Banerjee, A., Basu, S.: Topic models over text streams: a study of batch and online unsupervised learning. In: SIAM Data Mining, pp. 437–442 (2007)Google Scholar
  5. 5.
    Berge, C.: Hypergraphs. North-Holland, Amsterdam (1989)MATHGoogle Scholar
  6. 6.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)MATHGoogle Scholar
  7. 7.
    Blank, M., Gorelick, L., Shechtman, E., Irani, M., Basri, R.: Actions as space-time shapes. In: ICCV, pp. 1395–1402 (2005)Google Scholar
  8. 8.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)MATHGoogle Scholar
  9. 9.
    Blei, D.M.: Introduction to probabilistic topic models. Commun. ACM 55, 77–84 (2012)CrossRefGoogle Scholar
  10. 10.
    Bottou, L., Cun, Y.L.: Large scale online learning. In: NIPS, pp. 217–224 (2004)Google Scholar
  11. 11.
    Bottou, L.: Online Learning and Stochastic Approximations. Cambridge University Press, Cambridge (1998)Google Scholar
  12. 12.
    Buntine, W.L.: Variational extensions to EM and multinomial PCA. In: ECML, pp. 23–34 (2002)Google Scholar
  13. 13.
    Canini, K.R., Shi, L., Griffths, T.L.: Online inference of topics with latent Dirichlet allocation. In: AISTATS, pp. 65–72 (2009)Google Scholar
  14. 14.
    Chang, J., Boyd-Graber, J., Gerris, S., Wang, C., Blei, D.: Reading tea leaves: How humans interpret topic models. In: NIPS, pp. 288–296 (2009)Google Scholar
  15. 15.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 27:1–27:27 (2011)Google Scholar
  16. 16.
    Chang, J., Blei, D.M.: Hierarchical relational models for document networks. Ann. Appl. Stat. 4(1), 124–150 (2010)CrossRefMathSciNetMATHGoogle Scholar
  17. 17.
    Cutler, R., Davis, L.S.: Robust real-time periodic motion detection, analysis, and applications. IEEE Trans. Pattern Anal. Mach. Intell. 22, 781–796 (2000)CrossRefGoogle Scholar
  18. 18.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Stat. Soc. B 39, 1–38 (1977)MathSciNetMATHGoogle Scholar
  19. 19.
    Dollár, P., Rabaud, V., Cottrell, G., Belongie, S.: Behavior recognition via sparse spatio-temporal features. Vis. Surveill. Perform. Eval. Tracking Surveill 2, 65–72 (2005)CrossRefGoogle Scholar
  20. 20.
    Eisenstein, J., Xing, E.: The CMU 2008 political blog corpus. Technical report, Carnegie Mellon University (2010)Google Scholar
  21. 21.
    Elidan, G., McGraw, I., Koller, D.: Residual belief propagation: informed scheduling for asynchronous message passing. In: UAI, pp. 165–173 (2006)Google Scholar
  22. 22.
    Fei-Fei, L., Perona, P.: A Bayesian hierarchical model for learning natural scene categories. In: CVPR, pp. 524–531 (2005)Google Scholar
  23. 23.
    Foundation, A.S., et al.: Apache Mahout (2010).
  24. 24.
    Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315(5814), 972–976 (2007)CrossRefMathSciNetMATHGoogle Scholar
  25. 25.
    Globerson, A., Chechik, G., Pereira, F., Tishby, N.: Euclidean embedding of co-occurrence data. J. Mach. Learn. Res. 8, 2265–2295 (2007)MathSciNetMATHGoogle Scholar
  26. 26.
    Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proc. Natl. Acad. Sci. 101, 5228–5235 (2004)CrossRefGoogle Scholar
  27. 27.
    Heinrich, G.: Parameter estimation for text analysis. Technical report, University of Leipzig (2008)Google Scholar
  28. 28.
    Hoffman, M., Blei, D., Bach, F.: Online learning for latent Dirichlet allocation. In: NIPS, pp. 856–864 (2010)Google Scholar
  29. 29.
    Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn. 42, 177–196 (2001)CrossRefMATHGoogle Scholar
  30. 30.
    Hoyer, P.O.: Non-negative matrix factorization with sparseness constraints. J. Mach. Learn. Res. 5, 1457–1469 (2004)MathSciNetMATHGoogle Scholar
  31. 31.
    Karnik, N.N., Mendel, J.M.: Centroid of a type-2 fuzzy set. Inf. Sci. 132, 195–220 (2001)CrossRefMathSciNetMATHGoogle Scholar
  32. 32.
    Karnik, N.N., Mendel, J.M.: Operations on type-2 fuzzy sets. Fuzzy Sets Syst. 122, 327–348 (2001)CrossRefMathSciNetMATHGoogle Scholar
  33. 33.
    Kschischang, F.R., Frey, B.J., Loeliger, H.A.: Factor graphs and the sum-product algorithm. IEEE Trans. Inf. Theory 47(2), 498–519 (2001)CrossRefMathSciNetMATHGoogle Scholar
  34. 34.
    Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999)CrossRefGoogle Scholar
  35. 35.
    Liang, Q., Mendel, J.M.: MPEG VBR video traffic modeling and classification using fuzzy technique. IEEE Trans. Fuzzy Syst. 9(1), 183–193 (2001)CrossRefGoogle Scholar
  36. 36.
    Liu, Z., Zhang, Y., Chang, E., Sun, M.: Plda+: parallel latent Dirichlet allocation with data placement and pipeline processing. ACM Trans. Intell. Syst. Technol. 2(3), 26 (2011)CrossRefGoogle Scholar
  37. 37.
    McCallum, A.K., Nigam, K., Rennie, J., Seymore, K.: Automating the construction of internet portals with machine learning. Inf. Retrieval 3(2), 127–163 (2000)CrossRefGoogle Scholar
  38. 38.
    Mendel, J.M., John, R.I.B., Liu, F.: Interval type-2 fuzzy logic systems made simple. IEEE Trans. Fuzzy Syst. p. accepted for publication (2005)Google Scholar
  39. 39.
    Mendel, J.M.: Uncertainty, fuzzy logic, and signal processing. Signal Process. 80(6), 913–933 (2000)CrossRefMATHGoogle Scholar
  40. 40.
    Mendel, J.M.: Uncertain Rule-based Fuzzy Logic Systems: Introduction and New Directions. Prentice-Hall, Upper Saddle River (2001)Google Scholar
  41. 41.
    Mendel, J.M., John, R.I.B.: Type-2 fuzzy sets made simple. IEEE Trans. Fuzzy Syst. 10(2), 117–127 (2002)CrossRefGoogle Scholar
  42. 42.
    Mimno, D., Hoffman, M.D., Blei, D.M.: Sparse stochastic inference for latent Dirichlet allocation. In: ICML (2012)Google Scholar
  43. 43.
    Minka, T., Lafferty, J.: Expectation-propagation for the generative aspect model. In: UAI, pp. 352–359 (2002)Google Scholar
  44. 44.
    Mitchell, H.: Pattern recognition using type-II fuzzy sets. Inf Sci 170, 409–418 (2005)CrossRefGoogle Scholar
  45. 45.
    Newman, D., Asuncion, A., Smyth, P., Welling, M.: Distributed algorithms for topic models. J. Mach. Learn. Res. 10, 1801–1828 (2009)MathSciNetMATHGoogle Scholar
  46. 46.
    Niebles, J., Wang, H., Fei-Fei, L.: Unsupervised learning of human action categories using spatial-temporal words. Int. J. Comput. Vis. 79, 299–318 (2008)CrossRefGoogle Scholar
  47. 47.
    Ousterhout, J., et al.: The case for RAMClouds: scalable high-performance storage entirely in DRAM. SIGOPS Oper. Syst. Rev. 43(4), 92–105 (2012)CrossRefGoogle Scholar
  48. 48.
    Porteous, I., Newman, D., Ihler, A., Asuncion, A., Smyth, P., Welling, M.: Fast collapsed Gibbs sampling for latent Dirichlet allocation. In: KDD, pp. 569–577 (2008)Google Scholar
  49. 49.
    Pruteanu-Malinici, I., Ren, L., Paisley, J., Wang, E., Carin, L.: Hierarchical Bayesian modeling of topics in time-stamped documents. IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 996–1011 (2010)CrossRefGoogle Scholar
  50. 50.
    Ramage, D., Hall, D., Nallapati, R., Manning, C.D.: Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora. In: Empirical Methods in Natural Language Processing, pp. 248–256 (2009)Google Scholar
  51. 51.
    Rosen-Zvi, M., Griffiths, T., Steyvers, M., Smyth, P.: The author-topic model for authors and documents. In: UAI, pp. 487–494 (2004)Google Scholar
  52. 52.
    Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: a local SVM approach. IN: ICPR, vol. 3, 32–36 (2004)Google Scholar
  53. 53.
    Sevastjanov, P., Bartosiewicz, P., Tkacz, K.: A new method for normalization of interval weights. Parallel Process. Appl. Math. 6068, 466–474 (2010)CrossRefGoogle Scholar
  54. 54.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Machine Intell. 22(8), 888–905 (2000)CrossRefGoogle Scholar
  55. 55.
    Smola, A., Narayanamurthy, S.: An architecture for parallel topic models. In: PVLDB, pp. 703–710 (2010)Google Scholar
  56. 56.
    Sullivan, J., Carlsson, S.: Recognizing and tracking human actions. In: ECCV, pp. 629–644 (2002)Google Scholar
  57. 57.
    Tappen, M.F., Freeman, W.T.: Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters. In: ICCV, pp. 900–907 (2003)Google Scholar
  58. 58.
    Teh, Y.W., Newman, D., Welling, M.: A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In: NIPS, pp. 1353–1360 (2007)Google Scholar
  59. 59.
    Thiesson, B., Meek, C., Heckerman, D.: Accelerating EM for large databases. Mach. Learn. 45, 279–299 (2001)CrossRefMATHGoogle Scholar
  60. 60.
    Wahabzada, M., Kersting, K., Pilz, A., Bauckhage, C.: More influence means less work: fast latent Dirichlet allocation by influence scheduling. In: CIKM, pp. 2273–2276 (2011)Google Scholar
  61. 61.
    Wahabzada, M., Kersting, K.: Larger residuals, less work: active document scheduling for latent Dirichlet allocation. In: ECML/PKDD, pp. 475–490 (2011)Google Scholar
  62. 62.
    Wang, Y., Mori, G.: Human action recognition by semilatent topic models. IEEE Trans. Pattern Anal. Mach. Intell. 31(10), 1762–1774 (2009)CrossRefGoogle Scholar
  63. 63.
    Wang, X.G., Ma, X.X., Grimson, W.E.L.: Unsupervised activity perception in crowded and complicated scenes using hierarchical Bayesian models. IEEE Trans. Pattern Anal. Mach. Intell. 31(3), 539–555 (2009)CrossRefGoogle Scholar
  64. 64.
    Welling, M., Rosen-Zvi, M., Hinton, G.: Exponential family harmoniums with an application to information retrieval. In: NIPS, pp. 1481–1488 (2004)Google Scholar
  65. 65.
    Winn, J., Bishop, C.M.: Variational message passing. J. Mach. Learn. Res. 6, 661–694 (2005)MathSciNetMATHGoogle Scholar
  66. 66.
    Wu, D., Nie, M.: Comparison and practical implementation of type-reduction algorithms for type-2 fuzzy sets and systems. In: FUZZ-IEEE, pp. 2131–2138 (2011)Google Scholar
  67. 67.
    Wu, H., Mendel, J.M.: Classification of battlefield ground vehicles using acoustic features and fuzzy logic rule-based classifiers. IEEE Trans. Fuzzy Syst. 15(1), 56–72 (2007)CrossRefGoogle Scholar
  68. 68.
    Yan, J., Liu, Z.Q., Gao, Y., Zeng, J.: Communication-efficient parallel belief propagation for latent Dirichlet allocation (2012). arXiv:1206.2190v1 [cs.LG]
  69. 69.
    Yan, F., Xu, N., Qi, Y.: Parallel inference for latent Dirichlet allocation on graphics processing units. In: NIPS, pp. 2134–2142 (2009)Google Scholar
  70. 70.
    Yao, L., Mimno, D., McCallum, A.: Efficient methods for topic model inference on streaming document collections. In: KDD, pp. 937–946 (2009)Google Scholar
  71. 71.
    Yu, H.F., Hsieh, C.J., Chang, K.W., Lin, C.J.: Large linear classification when data cannot fit in memory. In: KDD, pp. 833–842 (2010)Google Scholar
  72. 72.
    Zeng, J., Cao, X.Q., Liu, Z.Q.: Memory-efficient topic modeling (2012). arXiv:1206.1147 [cs.LG]
  73. 73.
    Zeng, J., Cao, X.Q., Liu, Z.Q.: Residual belief propagation for topic modeling (2012). arXiv:1201.0838v2 [cs.LG]
  74. 74.
    Zeng, J., Cheung, W.K., Li, C.H., Liu, J.: Coauthor network topic models with application to expert finding. In: IEEE/WIC/ACM WI-IAT, pp. 366–373 (2010)Google Scholar
  75. 75.
    Zeng, J., Cheung, W.K.W., Li, C.H., Liu, J.: Multirelational topic models. In: ICDM, pp. 1070–1075 (2009)Google Scholar
  76. 76.
    Zeng, J., Cheung, W.K., Liu, J.: Learning topic models by belief propagation. IEEE Trans. Pattern Anal. Mach. Intell. (2012). arXiv:1109.3437v4 [cs.LG]
  77. 77.
    Zeng, J., Liu, Z.Q., Cao, X.Q.: Online belief propagation for topic modeling (2012). arXiv:1206.1147 [cs.LG]
  78. 78.
    Zeng, J., Liu, Z.Q.: Interval type-2 fuzzy hidden Markov models. In: Proceedings of FUZZ-IEEE pp. 1123–1128 (2004)Google Scholar
  79. 79.
    Zeng, J., Liu, Z.Q.: Type-2 fuzzy hidden Markov models to phoneme recognition. In: 17th International Conference on Pattern Recognition 1, 192–195 (2004)Google Scholar
  80. 80.
    Zeng, J., Liu, Z.Q.: Type-2 fuzzy Markov random fields and their application to handwritten Chinese character recognition. IEEE Trans. Fuzzy Syst. 16(3), 747-760 (2008)Google Scholar
  81. 81.
    Zeng, J., Liu, Z.Q.: Type-2 fuzzy Markov random fields to handwritten character recognition. In: 18th International Conference on Pattern Recognition pp. 1162–1165 (2006)Google Scholar
  82. 82.
    Zeng, J., Liu, Z.Q.: Type-2 fuzzy sets for handling uncertainty in pattern recognition. In: Proceedings of FUZZ-IEEE pp. 6597–6602 (2006)Google Scholar
  83. 83.
    Zeng, J., Liu, Z.Q.: Type-2 fuzzy hidden Markov models and their application to speech recognition. IEEE Trans. Fuzzy Syst. 14(3), 454–467 (2006)CrossRefMathSciNetGoogle Scholar
  84. 84.
    Zeng, J., Xie, L., Liu, Z.Q.: Type-2 fuzzy Gaussian mixture models. Pattern Recognit 41(12), 3636–3643 (2008)CrossRefMATHGoogle Scholar
  85. 85.
    Zeng, J., Liu, Z.Q.: Type-2 fuzzy Markov random fields and their application to handwritten Chinese character recognition. IEEE Trans. Fuzzy Syst. 16(3), 747–760 (2008)CrossRefGoogle Scholar
  86. 86.
    Zeng, J.: A topic modeling toolbox using belief propagation. J. Mach. Learn. Res. 13, 2233–2236 (2012)MathSciNetMATHGoogle Scholar
  87. 87.
    Zhai, K., Boyd-Graber, J., Asadi, N., Alkhouja, M.: Mr. LDA: a flexible large scale topic modeling package using variational inference in mapreduce. In: WWW (2012)Google Scholar
  88. 88.
    Zhu, S., Zeng, J., Mamitsuka, H.: Enhancing MEDLINE document clustering by incorporating MeSH semantic similarity. Bioinformatics 25(15), 1944–1951 (2009)CrossRefGoogle Scholar
  89. 89.
    Zhu, S., Takigawa, I., Zeng, J., Mamitsuka, H.: Field independent probabilistic model for clustering multi-field documents. Inf. Process. Manage. 45(5), 555–570 (2009)CrossRefGoogle Scholar
  90. 90.
    Zipf, G.K.: Human behavior and the principle of least effort. Addison-Wesley, Cambridge (1949)Google Scholar

Copyright information

© Tsinghua University Press, Beijing and Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.School of Computer Science and TechnologySoochow UniversitySuzhouChina
  2. 2.School of Creative MediaCity University of Hong KongHong KongChina

Personalised recommendations