Multiclass Penalized Likelihood Pattern Classification Algorithm
Penalized likelihood is a general approach whereby an objective function is defined, consisting of the log likelihood of the data minus some term penalizing non-smooth solutions. Subsequently, this objective function is maximized, yielding a solution that achieves some sort of trade-off between the faithfulness and the smoothness of the fit.
In this paper we extend the penalized likelihood classification that we proposed in earlier work to the multi class case. The algorithms are based on using a penalty term based on the K-nearest neighbors and the likelihood of the training patterns’ classifications. The algorithms are simple to implement, and result in a performance competitive with leading classifiers.
KeywordsMulticlass K-nearest neighbor Penalized likelihood Pattern classification Posterior probability
Unable to display preview. Download preview PDF.
- 1.Green, P.: Penalized Likelihood. In: Encyclopedia of Statistical Sciences Update, vol. 3. John Wiley Publishing, New Jersey (1999)Google Scholar
- 6.Gu, C.: Cross-validating Non-Gaussian Data. J. Comput. Graph. Stat. 1, 169–179 (1992)Google Scholar
- 7.Lu, F., Hill, G.C., Wahba, G., Desiati, P.: Signal Probability Estimation with Penalized Likelihood Method on Weighted Data. Department of Statistics, University of Wisconsin, Technical Report No. 1106 (2005)Google Scholar
- 9.Wahba, G., Gu, C., Wang, Y., Chappell, R.: Soft Classification, a.k.a. Risk Estimation, via Penalized Log Likelihood and Smoothing Spline Analysis of Variance. Department of Statistics, University of Wisconsin, Technical Report No. 899 (1993)Google Scholar
- 10.Cawley, G., Talbot, N.L., Girolami, M.: Sparse Multinomial Logistic Regression via Bayesian L1 Regularisation. Adv. Neural Inf. Process. Syst. 19, 209–216 (2007)Google Scholar
- 12.UCI Machine Learning Repository (2012), http://archive.ics.uci.edu/ml/
- 13.Chang, C.C., Lin, C.J.: LIBSVM toolbox (2012), http://www.csie.ntu.edu.tw/~cjlin/libsvm/
- 14.Vincent, P., Bengio, Y.: Manifold Parzen Windows. Adv. Neural Inf. Process. Syst. 15, 825–832 (2003)Google Scholar
- 15.Paris, S.: Parzen Windows Estimator Classifier (2008), http://www.mathworks.com/matlabcentral/fileexchange/17450-parzen-classifier
- 16.Goldberger, J., Roweis, S., Hinton, G., Salakhutdinov, R.: Neighbourhood Components Analysis. Adv. Neural Inf. Process. Syst. 17, 513–520 (2004)Google Scholar
- 17.Maaten, L.V.D.: NCA Toolbox for Dimensionality Reduction (2010), http://homepage.tudelft.nl/19j49/Matlab_Toolbox_for_Dimensionality_Reduction.html