On Discriminative Joint Density Modeling

  • Jarkko Salojärvi
  • Kai Puolamäki
  • Samuel Kaski
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3720)


We study discriminative joint density models, that is, generative models for the joint density p(c,x) learned by maximizing a discriminative cost function, the conditional likelihood. We use the framework to derive generative models for generalized linear models, including logistic regression, linear discriminant analysis, and discriminative mixture of unigrams. The benefits of deriving the discriminative models from joint density models are that it is easy to extend the models and interpret the results, and missing data can be treated using justified standard methods.


Generalize Linear Model Linear Discriminant Analysis Exponential Family Joint Density Model Family 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    McCullagh, P., Nelder, J.A.: Generalized Linear Models, 2nd edn. CRC Press, Boca Raton (1990)Google Scholar
  2. 2.
    Rubinstein, Y.D., Hastie, T.: Discriminative vs informative learning. In: Heckerman, D., Mannila, H., Pregibon, D., Uthurusamy, R. (eds.) Proc. ACM KDD, pp. 49–53. AAAI Press, Menlo Park (1997)Google Scholar
  3. 3.
    Kontkanen, P., Myllymäki, P., Tirri, H.: Classifier learning with supervised marginal likelihood. In: Breese, J., Koller, D. (eds.) Proc. UAI 2001, pp. 277–284. Morgan Kaufmann Publishers, San Francisco (2001)Google Scholar
  4. 4.
    Ng, A.Y., Jordan, M.I.: On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. In: Dietterich, T.G., Becker, S., Ghahramani, Z. (eds.) Advances in NIPS 14, pp. 841–848. MIT Press, Cambridge (2002)Google Scholar
  5. 5.
    Nádas, A., Nahamoo, D., Picheny, M.A.: On a model-robust training method for speech recognition. IEEE Tr. on Acoustics, Speech, and Signal Processing 39, 1432–1436 (1988)CrossRefGoogle Scholar
  6. 6.
    Povey, D., Woodland, P., Gales, M.: Discriminative MAP for acoustic model adaptation. In: Proc. IEEE ICASSP 2003, vol. 1, pp. 312–315 (2003)Google Scholar
  7. 7.
    Buntine, W.: Variational extensions to EM and multinomial PCA. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 23–34. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  8. 8.
    Efron, B.: The geometry of exponential families. The Annals of Statistics 6, 362–376 (1978)zbMATHCrossRefMathSciNetGoogle Scholar
  9. 9.
    Schall, R.: Estimation in generalized linear models with random effects. Biometrika 78, 719–727 (1991)zbMATHCrossRefGoogle Scholar
  10. 10.
    Salojärvi, J., Puolamäki, K., Kaski, S.: Expectation maximization algorithms for conditional likelihoods. In: Proc. ICML 2005 (2005) (in press)Google Scholar
  11. 11.
    Sharma, S.: Applied Multivariate Techniques. John Wiley & Sons, Inc., Chichester (1996)Google Scholar
  12. 12.
    Puolamäki, K., Salojärvi, J., Savia, E., Simola, J., Kaski, S.: Combining eye movements and collaborative filtering for proactive information retrieval. In: Proc. SIGIR 2005 (2005) (in press)Google Scholar
  13. 13.
    Jaakkola, T.S., Meila, M., Jebara, T.: Maximum entropy discrimination. In: Solla, S.A., Leen, T.K., Müller, K.R. (eds.) Advances in NIPS 12, pp. 470–476. MIT Press, Cambridge (2000)Google Scholar
  14. 14.
    Nigam, K., McCallum, A.K., Thrun, S., Mitchell, T.M.: Text classification from labeled and unlabeled documents using EM. Machine Learning 39, 103–134 (2000)zbMATHCrossRefGoogle Scholar
  15. 15.
    Lewis, D.D., Yang, Y., Rose, T., Li, F.: Rcv1: A new benchmark collection for text categorization research. Journal of Machine Learning Research 5, 361–397 (2004)Google Scholar
  16. 16.
    Troyanskaya, O., Cantor, M., Sherlock, G., Brown, P., Hastie, T., Tibshirani, R., Botstein, D., Altman, R.B.: Missing value estimation methods for DNA microarrays. Bioinformatics 17, 520–525 (2001)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Jarkko Salojärvi
    • 1
  • Kai Puolamäki
    • 1
  • Samuel Kaski
    • 1
    • 2
  1. 1.Laboratory of Computer and Information ScienceHelsinki University of TechnologyFinland
  2. 2.Department of Computer ScienceUniversity of HelsinkiFinland

Personalised recommendations