Discriminative Learning Can Succeed Where Generative Learning Fails

  • Philip M. Long
  • Rocco A. Servedio
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4005)


Generative algorithms for learning classifiers use training data to separately estimate a probability model for each class. New items are then classified by comparing their probabilities under these models. In contrast, discriminative learning algorithms try to find classifiers that perform well on all the training data.

We show that there is a learning problem that can be solved by a discriminative learning algorithm, but not by any generative learning algorithm (given minimal cryptographic assumptions). This statement is formalized using a framework inspired by previous work of Goldberg [3].


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. Wiley, Chichester (2000)Google Scholar
  2. 2.
    Fischer, P., Pölt, S., Simon, H.U.: Probably almost Bayes decisions. In: Proceedings of the Fourth Annual COLT, pp. 88–94 (1991)Google Scholar
  3. 3.
    Goldberg, P.W.: When can two unsupervised learners achieve PAC separation? In: Helmbold, D.P., Williamson, B. (eds.) COLT 2001 and EuroCOLT 2001. LNCS, vol. 2111, p. 303. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  4. 4.
    Goldreich, O., Goldwasser, S., Micali, S.: How to construct random functions. Journal of the Association for Computing Machinery 33(4), 792–807 (1986)MathSciNetGoogle Scholar
  5. 5.
    Jaakkola, T., Haussler, D.: Exploiting generative models in discriminative classifiers. In: Advances in NIPS 11, pp. 487–493. Morgan Kaufmann, San Francisco (1998)Google Scholar
  6. 6.
    Jebara, T.: Machine learning: discriminative and generative. Kluwer, Dordrecht (2003)Google Scholar
  7. 7.
    Long, P., Servedio, R.: Discriminative Learning can Succeed where Generative Learning Fails (full version), available at: http://www.cs.columbia.edu/~rocco/papers/discgen.html
  8. 8.
    Ng, A.Y., Jordan, M.I.: On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In: NIPS (2001)Google Scholar
  9. 9.
    Raina, R., Shen, Y., Ng, A.Y., McCallum, A.: Classification with hybrid generative/discriminative models. In: NIPS (2004)Google Scholar
  10. 10.
    Håstad, J., Impagliazzo, R., Levin, L., Luby, M.: A pseudorandom generator from any one-way function. SIAM Journal on Computing 28(4), 1364–1396 (1999)CrossRefMathSciNetMATHGoogle Scholar
  11. 11.
    Valiant, L.G.: A theory of the learnable. In: Proc. 16th Annual ACM Symposium on Theory of Computing (STOC), pp. 436–445. ACM Press, New York (1984)Google Scholar
  12. 12.
    Vapnik, V.: Estimations of dependences based on statistical data. Springer, Heidelberg (1982)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Philip M. Long
    • 1
  • Rocco A. Servedio
    • 2
  1. 1.GoogleMountain ViewUSA
  2. 2.Columbia UniversityNew YorkUSA

Personalised recommendations