Annealed Discriminant Analysis

  • Gang Wang
  • Zhihua Zhang
  • Frederick H. Lochovsky
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3720)


Motivated by the analogies to statistical physics, the deterministic annealing (DA) method has successfully been demonstrated in a variety of applications. In this paper, we explore a new methodology to devise the classifier under the DA method. The differential cost function is derived subject to a constraint on the randomness of the solution, which is governed by the temperature T. While gradually lowering the temperature, we can always find a good solution which can both solve the overfitting problem and avoid poor local optima. Our approach is called annealed discriminant analysis (ADA). It is a general approach, where we elaborate two classifiers, i.e., distance-based and inner product-based, in this paper. The distance-based classifier is an annealed version of linear discriminant analysis (LDA) while the inner product-based classifier is a generalization of penalized logistic regression (PLR). As such, ADA provides new insights into the workings of these two classification algorithms. The experimental results show substantial performance gains over standard learning methods.


Cost Function Conditional Probability Linear Discriminant Analysis Discriminant Function Regularization Parameter 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Hofmann, T., Buhmann, J.: Pairwise data clustering by deterministic annealing. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 1–14 (1997)CrossRefGoogle Scholar
  2. 2.
    Miller, D., Rao, A.V., Rose, K., Gersho, A.: A global optimization technique for statistical classifier design. IEEE Transaction on Signal Processing 44, 3108–3122 (1996)CrossRefGoogle Scholar
  3. 3.
    Rao, A., Miller, D., Rose, K., Gersho, A.: A deterministic annealing approach for parsimonious design of piecewise regression models. IEEE Transactions on Pattern Analysis and Machine Intelligence 21, 159–173 (1999)CrossRefGoogle Scholar
  4. 4.
    Rose, K.: Deterministic annealing for clustering, compression, classification, regression, and related optimization problem. Proceedings of the IEEE 86, 2210–2239 (1998)CrossRefGoogle Scholar
  5. 5.
    Yuille, A.L., Stolortz, P., Utans, J.: Statistical physics, mixtures of distributions, and the EM algorithm. Neural Computation 6, 334–340 (1994)CrossRefGoogle Scholar
  6. 6.
    Rose, K., Gurewitz, E., Fox, G.C.: Statistical mechanics and phase transitions in clustering. Physics Review Letter 65, 945–948 (1990)CrossRefGoogle Scholar
  7. 7.
    Zhang, T.: Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research 5, 1225–1251 (2004)Google Scholar
  8. 8.
    Wahba, G.: Soft and hard classification by reproducing kernel Hilbert space methods. Proceedings of the National Academy of Sciences 99, 16524–16530 (2002)zbMATHCrossRefMathSciNetGoogle Scholar
  9. 9.
    Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions Pattern Analysis and Machine Intelligence 6, 721–741 (1984)zbMATHCrossRefGoogle Scholar
  10. 10.
    Nabney, I.: Netlab: algorithms for pattern recognition. Springer, Heidelberg (2001)Google Scholar
  11. 11.
    Hastie, T., Tishiran, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, Heidelberg (2001)zbMATHGoogle Scholar
  12. 12.
    McLachlan, G.J.: Discriminant analysis and statistical pattern recognition. John Wiley & Sons, Chichester (1992)CrossRefGoogle Scholar
  13. 13.
    Zhu, J., Hastie, T.: Classification of gene microarrays by penalized logistic regression. Biostatistics 5, 427–443 (2004)zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Gang Wang
    • 1
  • Zhihua Zhang
    • 1
  • Frederick H. Lochovsky
    • 1
  1. 1.Department of Computer ScienceHong Kong University of Science and TechnologyKowloon, Hong Kong

Personalised recommendations