Applied Intelligence

, Volume 37, Issue 4, pp 463–474 | Cite as

Evolutionary response surfaces for classification: an interpretable model

  • Rafael del Castillo-Gomariz
  • Nicolás García-Pedrajas


Response surfaces are powerful tools for both classification and regression because they are able to model many different phenomena and construct complex boundaries between classes. With very simple expressions, response surfaces are able to accurately solve difficult problems. Thus, the interpretability of the results is very interesting from the point of view of the expert, which is provided by a classification model from which useful information may be inferred.

However, response surfaces suffer from a major problem that limits their applicability. Even with a low degree and a moderate number of features, the number of terms in the surfaces is extremely large. Thus, standard learning algorithms find many problems to efficiently obtain the coefficients of the terms, and the risk of overfitting is high.

To overcome this problem we present evolutionary response surfaces for the classification of two-class problems. The use of a fitness function that combines accuracy and interpretability obtains accurate classifiers that are simple and interpretable by the expert. The results obtained for 20 problems from the UCI Machine Learning Repository are comparable with well-known classification algorithms with a more interpretable polynomial function.


Classification Genetic algorithms Response surfaces 


  1. 1.
    Adami H-O, Malker B, Holmberg L, Persson I, Stone B (1986) The relation between survival and age at diagnosis in breast cancer. N Engl J Med 315:559–563 CrossRefGoogle Scholar
  2. 2.
    An S-Y, Kang J-G, mChoi W-S, Oh S-Y (2011) A neural network based retrainable framework for robust object recognition with application to mobile robotics. Appl Intell 35:190–210 CrossRefGoogle Scholar
  3. 3.
    Assaleh K, Shanableh T (2010) Robust polynomial classifier using l 1-norm minimization. Appl Intell 33:330–339 CrossRefGoogle Scholar
  4. 4.
    Bambha NK, Bhattacharyya SS, Teich J, Zitzler E (2004) Systematic integration of parameterized local search into evolutionary algorithms. IEEE Trans Evol Comput 8:137–155 CrossRefGoogle Scholar
  5. 5.
    Brent M (2008) Steady progress and recent breakthroughs in the accuracy of automated gene annotation. Nat Rev Genet 9:62–73 CrossRefGoogle Scholar
  6. 6.
    Campbell WT, Assaleh KT, Broun CC (2002) Speaker recognition with polynomial classifiers. IEEE Trans Speech Audio Process 10:205–212 CrossRefGoogle Scholar
  7. 7.
    Cheng J, Li QS (2009) Application of the response surface methods to solve inverse reliability problems with implicit response functions. Comput Mech 43(4):451–459 CrossRefGoogle Scholar
  8. 8.
    del Castillo-Gomariz R, García-Pedrajas N (2006) Classification by means of evolutionary response surfaces. In: Proceedings of the 14th European symposium on artificial neural networks, Bruges, Belgium, April Google Scholar
  9. 9.
    Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30 MathSciNetzbMATHGoogle Scholar
  10. 10.
    Doi K (2007) Computer-aided diagnosis in medical imaging: Historical review, current status and future potential. Comput Med Imaging Graph 31:198–211 CrossRefGoogle Scholar
  11. 11.
    Eshelman LJ, Schaffer JD (1993) Real-coded genetic algorithms and interval-schemata. In: Whitley LD (ed) Foundations of genetic algorithms, vol 2. Morgan Kaufmann, San Mateo, pp 187–202 Google Scholar
  12. 12.
    Ferreira SLC, Bruns RE, da Silva EGP, dos Santos WNL, Quintella CM, David JM, de Andrade JB, Breitkreitz MC, Jardim ICSF, Neto BB (2007) Statistical designs and response surface techniques for the optimization of chromatographic systems. J Chromatogr A 1158(1–2):2–14 Google Scholar
  13. 13.
    Francis F, Sabu A, Nampoothiri KM, Ramachandran S, Ghosh S, Szakacs G, Pandey A (2003) Use of response surface methodology for optimizing process parameters for the production of α-amylase by aspergillus oryzae. Biochem Eng J 15(2):107–115 CrossRefGoogle Scholar
  14. 14.
    Frank A, Asuncion A (2010) UCI machine learning repository Google Scholar
  15. 15.
    García-Pedrajas N, Ortiz-Boyer D (2011) An empirical study of binary classifier fusion methods for multiclass classification. Inf Fusion 12:111–130 CrossRefGoogle Scholar
  16. 16.
    Goethals PL, Cho BR (2011) Solving the optimal process target problem using response surface designs in heteroscedastic conditions. Int J Prod Res 49(12):3455–3478 zbMATHCrossRefGoogle Scholar
  17. 17.
    Guyon I, Weston J, Barnhill S, Vapnik V (2002) Gene selection for cancer classification using support vector machines. Mach Learn 46:389–422 zbMATHCrossRefGoogle Scholar
  18. 18.
    Gönen M, Alpaydin E (2011) Regularizing multiple kernel learning using response surface methodology. Pattern Recognit 44:159–171 zbMATHCrossRefGoogle Scholar
  19. 19.
    Haykin S (1999) Neural networks—a comprehensive foundation, 2nd edn. Prentice-Hall, Upper Saddle River zbMATHGoogle Scholar
  20. 20.
    Herrera F, Lozano M, Verdegay JL (1998) Tackling real-coded genetic algorithms: operators and tools for behavioural analysis. Artif Intell Rev 12:265–319 zbMATHCrossRefGoogle Scholar
  21. 21.
    Janev M, Pekar D, Jakovljevic N, Delic V (2010) Eigenvalues driven Gaussian selection in continuous speech recognition using hmms with full covariance matrices. Appl Intell 33:107–116 CrossRefGoogle Scholar
  22. 22.
    Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. J Glob Optim 21:345–383 zbMATHCrossRefGoogle Scholar
  23. 23.
    Kalil SJ, Maugeri F, Rodrigues MI (2000) Response surface analysis and simulation as a tool for bioprocess design and optimization. Process Biochem 35(6):539–550 CrossRefGoogle Scholar
  24. 24.
    Khuri AI, Mukhopadhyay S (2010) Response surface methodology. Comput Stat 2:128–149 Google Scholar
  25. 25.
    Klein EJ, Rivera SL (2000) A review of criteria functions and response surface methodology for the optimization of analytical scale hplc separations. J Liq Chromatogr Relat Technol 23(14):2097–2121 CrossRefGoogle Scholar
  26. 26.
    Kuncheva L, Jain LC (1999) Nearest neighbor classifier: simultaneous editing and descriptor selection. Pattern Recognit Lett 20:1149–1156 CrossRefGoogle Scholar
  27. 27.
    Lew TL, Spencera AB, Scarpaa F, Wordena K, Rutherford A, Hemez F (2006) Identification of response surface models using genetic programming. Mech Syst Signal Process 20:1819–1831 CrossRefGoogle Scholar
  28. 28.
    Liu Ch-L, Sako H (2006) Class-specific feature polynomial classifier for pattern classification and its application to handwritten numeral recognition. Pattern Recognit 39:669–681 zbMATHCrossRefGoogle Scholar
  29. 29.
    Marquardt DW (1963) An algorithm for least-squares estimation of non-linear parameters. J Soc Ind Appl Math 11(2):431–441 MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Miller GF, Todd PM, Hedge SU (1991) Designing neural networks. Neural Netw 4:53–60 CrossRefGoogle Scholar
  31. 31.
    Minto CF, Schnider TW, Short TG, Gregg KM, Gentilini A, Shafer SL (2000) Response surface model for anesthetic drug interactions. Anesthesiology 92(6):1603–1616 CrossRefGoogle Scholar
  32. 32.
    Myers RH, Montgomery DC (2002) Response surface methodology: process and product optimization using designed experiments, 2nd edn. Wiley, New York zbMATHGoogle Scholar
  33. 33.
    Myers RH, Montgomery DC, Vining G, Borror CM, Kowalski SM (2004) Response surface methodology: a retrospective and literature survey. J Qual Technol 36(1):53–78 Google Scholar
  34. 34.
    Nock R (2002) Inducing interpretable voting classifiers without trading accuracy for simplicity: theoretical results, approximation algorithms, and experiments. J Artif Intell Res 17:137–170 MathSciNetzbMATHGoogle Scholar
  35. 35.
    Ong YS, Keane AJ (2004) Meta-Lamarckian learning in memetic algorithms. IEEE Trans Evol Comput 8:99–110 CrossRefGoogle Scholar
  36. 36.
    Ortiz-Boyer D, Hervás-Martínez C, García-Pedrajas N (2007) Improving crossover operator for real-coded genetic algorithms using virtual parents. J Heuristics 13:265–314 CrossRefGoogle Scholar
  37. 37.
    Park B-J, Pedrycz W, Oh S-K (2010) Polynomial-based radial basis function neural networks (p-rbf nns) and their application to pattern classification. Appl Intell 32:27–46 CrossRefGoogle Scholar
  38. 38.
    Puri S, Beg QK, Gupta R (2002) Optimization of alkaline protease production from bacillus sp. by response surface methodology. Curr Microbiol 44(4):286–290 CrossRefGoogle Scholar
  39. 39.
    Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann, San Mateo Google Scholar
  40. 40.
    Rawlings JO, Pantula SG, Dickey D (1998) Applied regression analysis: a research tool. Springer, New York zbMATHCrossRefGoogle Scholar
  41. 41.
    Satapathy SCh, Chittinemi S, Krishna SM, Murthy JVR, Reddy PVGDP (2012) Kalman particle swarm optimized polynomials for data classification. Appl Math Model 36:115–126 MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    Shaffer RE, Small GW (1996) Genetic algorithms for the optimization of piecewise linear discriminants. Chemom Intell Lab Syst 35:87–104 CrossRefGoogle Scholar
  43. 43.
    Shanableh T, Assaleh K (2010) Feature modeling using polynomial classifiers and stepwise regression. Neurocomputing 73:1752–1759 CrossRefGoogle Scholar
  44. 44.
    Sutton GC (1989) Computer-aided diagnosis: a review. Br J Surg 76:82–85 CrossRefGoogle Scholar
  45. 45.
    Tan KCh, Khor EF, Lee TH (2003) An evolutionary algorithm with advanced goal and priority specification for multi-objective optimization. J Artif Intell Res 18:183–215 MathSciNetzbMATHGoogle Scholar
  46. 46.
    Ting Ch-Y, Phon-Amnuaisuk S (2010) Optimal dynamic decision network model for scientific inquiry learning environment. Appl Intell 33:387–406 CrossRefGoogle Scholar
  47. 47.
    Toh K-A, Tran Q-L, Srinivasan D (2004) Benchmarking a reduced multivariate polynomial pattern classifier. IEEE Trans Pattern Anal Mach Intell 26:740–755 CrossRefGoogle Scholar
  48. 48.
    Tong L-I, Chang Y-Ch, Lin Sh-H (2011) Determining the optimal re-sampling strategy for a classification model with imbalanced data using design of experiments and response surface methodologies. Expert Syst Appl 28:4222–4227 CrossRefGoogle Scholar
  49. 49.
    Toropov VV, Alvarez LF (1998) Application of genetic programming and response surface methodology to optimization and inverse problems. In: Tanaka M, Dulikravich GS (eds) Inverse problems in engineering mechanics. Elsevier, Oxford, pp 551–560 CrossRefGoogle Scholar
  50. 50.
    Tran Q-L, Toh K-A, Srinivasan D, Wong K-L, Low ShQ-C (2005) An empirical comparison of nine pattern classifiers. IEEE Trans Syst Man Cybern, Part B, Cybern 35:1079–1091 CrossRefGoogle Scholar
  51. 51.
    Valova I, Milano G, Bowen K, Gueorguieva N (2011) Bridging the fuzzy, neural and evolutionary paradigms for automatic target recognition. Appl Intell 35:211–225 CrossRefGoogle Scholar
  52. 52.
    Vapnik V (1999) The nature of statistical learning theory. Springer, New York Google Scholar
  53. 53.
    Vladislavleva EJ, Smits GF, den Hertog D (2009) Order of nonlinearity as a complexity measure for models generated by symbolic regression via Pareto genetic programming. IEEE Trans Evol Comput 13:333–349 CrossRefGoogle Scholar
  54. 54.
    Yeun YS, Yang YS, Ruy WS, Kim BJ (2005) Polynomial genetic programming for response surface modeling part 1: a methodology. Struct Multidiscip Optim 29:19–34 MathSciNetzbMATHCrossRefGoogle Scholar
  55. 55.
    Yeun YS, Kim BJ, Yang YS, Ruy WS (2005) Polynomial genetic programming for response surface modeling part 2: adaptive approximate models with probabilistic optimization problems. Struct Multidiscip Optim 29:35–494 MathSciNetzbMATHCrossRefGoogle Scholar
  56. 56.
    Zhao X, Gao X-Sh, Hu Z-Ch (2007) Evolutionary programming based on non-uniform mutation. Appl Math Comput 192:1–11 MathSciNetzbMATHCrossRefGoogle Scholar
  57. 57.
    Zitzler E, Thiele L, Laumanns M, Fonseca CM, Grunert V (2003) Performance assessment of multiobjective optimizers: an analysis and review. IEEE Trans Evol Comput 7(2):117–132 CrossRefGoogle Scholar
  58. 58.
    Özcan E, Başaran C (2009) A case study of memetic algorithms for constraint optimization. Soft Comput 13(8–9):871–882 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Rafael del Castillo-Gomariz
    • 1
  • Nicolás García-Pedrajas
    • 1
  1. 1.Department of Computing and Numerical AnalysisUniversity of CordobaCórdobaSpain

Personalised recommendations