Multiclass Penalized Likelihood Pattern Classification Algorithm

  • Amira Samy Talaat
  • Amir F. Atiya
  • Sahar A. Mokhtar
  • Ahmed Al-Ani
  • Magda Fayek
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7665)


Penalized likelihood is a general approach whereby an objective function is defined, consisting of the log likelihood of the data minus some term penalizing non-smooth solutions. Subsequently, this objective function is maximized, yielding a solution that achieves some sort of trade-off between the faithfulness and the smoothness of the fit.

In this paper we extend the penalized likelihood classification that we proposed in earlier work to the multi class case. The algorithms are based on using a penalty term based on the K-nearest neighbors and the likelihood of the training patterns’ classifications. The algorithms are simple to implement, and result in a performance competitive with leading classifiers.


Multiclass K-nearest neighbor Penalized likelihood Pattern classification Posterior probability 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Green, P.: Penalized Likelihood. In: Encyclopedia of Statistical Sciences Update, vol. 3. John Wiley Publishing, New Jersey (1999)Google Scholar
  2. 2.
    Gu, C., Kim, Y.J.: Penalized Likelihood Regression: General Formulation and Efficient Approximation. Can. J. Stat. 30, 619–628 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Green, P.J., Silverman, B.W.: Nonparametric Regression and Generalized Linear Models: A Roughness Penalty Approach. Chapman and Hall, London (1994)zbMATHGoogle Scholar
  4. 4.
    Wahba, G.: Spline Models for Observational Data. SIAM, Philadelphia (1990)zbMATHCrossRefGoogle Scholar
  5. 5.
    O’Sullivan, F., Yandell, B., Raynor, W.: Automatic Smoothing of Regression Functions in Generalized Linear Models. J. Am. Stat. Assoc. 81, 96–103 (1986)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Gu, C.: Cross-validating Non-Gaussian Data. J. Comput. Graph. Stat. 1, 169–179 (1992)Google Scholar
  7. 7.
    Lu, F., Hill, G.C., Wahba, G., Desiati, P.: Signal Probability Estimation with Penalized Likelihood Method on Weighted Data. Department of Statistics, University of Wisconsin, Technical Report No. 1106 (2005)Google Scholar
  8. 8.
    Wahba, G.: Soft and Hard Classification by Reproducing Kernel Hilbert Space Methods. Proceedings of the National Academy of Sciences 99, 16524–16530 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Wahba, G., Gu, C., Wang, Y., Chappell, R.: Soft Classification, a.k.a. Risk Estimation, via Penalized Log Likelihood and Smoothing Spline Analysis of Variance. Department of Statistics, University of Wisconsin, Technical Report No. 899 (1993)Google Scholar
  10. 10.
    Cawley, G., Talbot, N.L., Girolami, M.: Sparse Multinomial Logistic Regression via Bayesian L1 Regularisation. Adv. Neural Inf. Process. Syst. 19, 209–216 (2007)Google Scholar
  11. 11.
    Atiya, A.F., Al-Ani, A.: A Penalized Likelihood Based Pattern Classification Algorithm. Pattern Recogn. 42, 2684–2694 (2009)zbMATHCrossRefGoogle Scholar
  12. 12.
    UCI Machine Learning Repository (2012),
  13. 13.
    Chang, C.C., Lin, C.J.: LIBSVM toolbox (2012),
  14. 14.
    Vincent, P., Bengio, Y.: Manifold Parzen Windows. Adv. Neural Inf. Process. Syst. 15, 825–832 (2003)Google Scholar
  15. 15.
    Paris, S.: Parzen Windows Estimator Classifier (2008),
  16. 16.
    Goldberger, J., Roweis, S., Hinton, G., Salakhutdinov, R.: Neighbourhood Components Analysis. Adv. Neural Inf. Process. Syst. 17, 513–520 (2004)Google Scholar
  17. 17.
    Maaten, L.V.D.: NCA Toolbox for Dimensionality Reduction (2010),

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Amira Samy Talaat
    • 1
  • Amir F. Atiya
    • 2
  • Sahar A. Mokhtar
    • 1
  • Ahmed Al-Ani
    • 3
  • Magda Fayek
    • 2
  1. 1.Computers and Systems DepartmentElectronic Research InstituteEgypt
  2. 2.Department of Computer EngineeringCairo UniversityGizaEgypt
  3. 3.Faculty of Engineering and Information TechnologyUniv. of TechnologySydneyAustralia

Personalised recommendations