Sensitivity Analysis with Cross-Validation for Feature Selection and Manifold Learning

  • Cuixian Chen
  • Yishi Wang
  • Yaw Chang
  • Karl Ricanek
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7367)


The performance of a learning algorithm is usually measured in terms of prediction error. It is important to choose an appropriate estimator of the prediction error. This paper analyzes the statistical properties of the K-fold cross-validation prediction error estimator. It investigates how to compare two algorithms statistically. It also analyzes the sensitivity to the changes in the training/test set. Our main contribution is to experimentally study the statistical property of repeated cross-validation to stabilize the prediction error estimation, and thus to reduce the variance of the prediction error estimator. Our simulation results provide an empirical evidence to this conclusion. The experimental study has been performed on PAL dataset for age estimation task.


Feature Selection Prediction Error Support Vector Regression Mean Absolute Error Locality Preserve Projection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2001),
  3. 3.
    Chen, C., Chang, Y., Ricanek, K., Wang, Y.: Face age estimation using model selection. In: CVPRW, pp. 93–99 (2010)Google Scholar
  4. 4.
    Chen, C., Yang, W., Wang, Y., Ricanek, K., Luu, K.: Facial feature fusion and model selection for age estimation. In: 9th International Conference on Automatic Face and Gesture Recognition (2011)Google Scholar
  5. 5.
    Dietterich, T.G.: Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation 10, 1895–1923 (1998)CrossRefGoogle Scholar
  6. 6.
    Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Annal of Statistics 32, 407–499 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, New York (2009)CrossRefGoogle Scholar
  8. 8.
    He, X., Niyogi, P.: Locality preserving projections. In: Proceedings of Advances in Neural Information Processing Systems 16 (2003)Google Scholar
  9. 9.
    Minear, M., Park, D.C.: A lifespan database of adult facial stimuli. Behavior Research Methods, Instruments, & Computers 36, 630–633 (2004)CrossRefGoogle Scholar
  10. 10.
    Refaeilzadeh, L.T.P., Liu, H.: On comaprison of feature selection algorithms. In: Proceedings of AAAI Workshop on Evaluation Methods for Machine Learning II, pp. 34–39 (2007)Google Scholar
  11. 11.
    Patterson, E., Sethuram, A., Albert, M., Ricanek, K.: Comparison of synthetic face aging to age progression by forensic sketch artist. In: IASTED International Conference on Visualization, Imaging, and Image Processing, Palma de Mallorca, Spain (2007)Google Scholar
  12. 12.
    Refaeilzadeh, P., Tang, L., Liu, H.: Cross-validation. In: Encyclopedia of Database Systems, pp. 532–538 (2009)Google Scholar
  13. 13.
    Ricanek, K., Wang, Y., Chen, C., Simmons, S.J.: Generalized multi-ethnic face age-estimation. In: BTAS (2009)Google Scholar
  14. 14.
    Rodriguez, J., Perez, A., Lozano, J.: Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(3), 569–575 (2010)CrossRefGoogle Scholar
  15. 15.
    Salzberg, S.: On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery 1, 317–327 (1997)CrossRefGoogle Scholar
  16. 16.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Vapnik, V.: Statistical learning theory. Wiley Interscience, New York (1998)zbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Cuixian Chen
    • 1
  • Yishi Wang
    • 1
  • Yaw Chang
    • 1
  • Karl Ricanek
    • 1
  1. 1.University of North Carolina WilmingtonUSA

Personalised recommendations