A Comparison of Validation Methods for Learning Vector Quantization and for Support Vector Machines on Two Biomedical Data Sets
We compare two comprehensive classification algorithms, support vector machines (SVM) and several variants of learning vector quantization (LVQ), with respect to different validation methods. The generalization ability is estimated by “multiple-hold-out” (MHO) and by “leave-one-out” (LOO) cross v method. The ξα-method, a further estimation method, which is only applicable for SVM and is computationally more efficient, is also used.
Calculations on two different biomedical data sets generated of experimental data measured in our own laboratory are presented. The first data set contains 748 feature vectors extracted of posturographic signals which were obtained in investigations of balance control in upright standing of 48 young adults. Two different classes are labelled as “without alcoholic impairment” and “with alcoholic impairment”. This classification task aims the detection of small unknown changes in a relative complex signal with high inter-individual variability.
The second data set contains 6432 feature vectors extracted of electroencephalographic and electroocculographic signals recorded during overnight driving simulations of 22 young adults. Short intrusions of sleep during driving, so-called microsleep events, were observed. They form examples of the first class. The second class contains examples of fatigue states, whereas driving is still possible. If microsleep events happen in typical states of brain activity, the recorded signals should contain typical alterations, and therefore discrimination from signals of the second class, which do not refer to such states, should be possible.
Optimal kernel parameters of SVM are found by searching minimal test errors with all three validation methods. Results obtained on both different biomedical data sets show different optimal kernel parameters depending on the validation method. It is shown, that the ξα-method seems to be biased and therefore LOO or MHO method should be preferred.
A comparison of eight different variants of LVQ and six other classification methods using MHO validation yields that SVM performs best for the second and more complex data set and SVM, GRLVQ and OLVQ1 show nearly the same performance for the first data set.
Unable to display preview. Download preview PDF.
- CORTES, C. and VAPNIK, V. (1995): Support Vector Networks. Machine Learning, 20, 273–297.Google Scholar
- DEVROYE, L.; GYORFI, L.; LUGOSI, G. (1996): A probabilistic theory of pattern recognition. Springer; New York.Google Scholar
- GOLZ, M.; SOMMER, D.; LEMBCKE, T.; KURELLA, B. (1998): Classification of the pre-stimulus-EEG of k-complexes using competitive learning networks. EUFIT’ 98. Aachen, 1767–1771.Google Scholar
- GOLZ, M.; SOMMER, D.; WALTHER, L.; EURICH, C. (2004): Discriminance Analysis of Postural Sway Trajectories with Neural Networks SCI2004, VII. Orlando, USA, 151–155.Google Scholar
- JOACHIMS, T. (2002): Learning to Classify Text Using Support Vector Machines. Kluwer; Boston.Google Scholar
- KEARNS, M. (1996): A Bound on the Error of Cross Validation Using the Approximation and Estimation Rates, with Consequences for the Training-Test Split. Advances in Neural Information Processing Systems, 8, 183–189.Google Scholar
- KOHONEN, T. (2001): Self-Organizing Maps (third edition). Springer, New York.Google Scholar
- MÜLLER, K.-R.; S. MIKA; RÄTSCH, G.; TSUDA, K.; SCHÖLKOPF, B. (2001): An Introduction to Kernel-Based Learning Algorithms. IEEE Transactions on Neural Networks, 12(2), 181–201.Google Scholar
- OSUNA, E.; FREUND, R.; GIROSI, F.; (1997): Training Support Vector Machines: an Application to Face Detection. Proceedings of CVPR’ 97. Puerto Rico.Google Scholar
- SATO, A. (1999): An Analysis of Initial State Dependence in Generalized LVQ. In: D. Willshaw et al. (Eds.): (ICANN’ 99). IEEE Press;, 928–933.Google Scholar
- SOMMER, D. and GOLZ, M. (2003): Short-Time Prognosis of Microsleep Events by Artificial Neural Networks. Proc. Medizin und Mobilität. Berlin, 149–151.Google Scholar
- VAN GESTEL, T.; SUYKENS, J.; BAESENS, B.; VIAENE, S.; VANTHIENEN, J.; DEDENE, G.; DE MOOR, B.; VANDEWALLE, J. (2002): Benchmarking least squares support vector machine classifiers. Machine Learning.Google Scholar
- VAPNIK, V. (1995): The Nature of Statistical Learning Theory. Springer, New York.Google Scholar