Automated Software Engineering

, Volume 21, Issue 1, pp 41–63 | Cite as

A method for evaluation of learning components

  • Niklas Lavesson
  • Veselka Boeva
  • Elena Tsiporkova
  • Paul Davidsson


Today, it is common to include machine learning components in software products. These components offer specific functionalities such as image recognition, time series analysis, and forecasting but may not satisfy the non-functional constraints of the software products. It is difficult to identify suitable learning algorithms for a particular task and software product because the non-functional requirements of the product affect algorithm suitability. A particular suitability evaluation may thus require the assessment of multiple criteria to analyse trade-offs between functional and non-functional requirements. For this purpose, we present a method for APPlication-Oriented Validation and Evaluation (APPrOVE). This method comprises four sequential steps that address the stated evaluation problem. The method provides a common ground for different stakeholders and enables a multi-expert and multi-criteria evaluation of machine learning algorithms prior to inclusion in software products. Essentially, the problem addressed in this article concerns how to choose the appropriate machine learning component for a particular software product.


Data mining Evaluation Machine learning 


  1. Alizadeh, A., et al.: Distinct types of diffuse large b-cell lymphoma identified by gene expression profiling. Nature 403, 503–511 (2000) CrossRefGoogle Scholar
  2. Allahyari, H., Lavesson, N.: User-oriented assessment of classification model understandability. In: 11th Scandinavian Conference on Artificial Intelligence. IOS Press, Amsterdam (2011) Google Scholar
  3. Alon, U., et al.: Broad patterns of gene expression revealed by clustering analysis of tumour and normal colon tissues probed by oligonucleotide arrays. Proc. Natl. Acad. Sci. USA 96, 6745–6750 (1999) CrossRefGoogle Scholar
  4. Bucila, C., Caruana, R., Niculescu-Mizil, A.: Model compression. In: 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York (2006) Google Scholar
  5. Dinesh Singh, E.A.: Gene expression correlates of clinical prostate cancer behavior. Cancer Cell 1, 203–209 (2002) CrossRefGoogle Scholar
  6. Fodor, J., Roubens, M.: Fuzzy Preference Modelling and Multicriteria Decision Support. Kluwer, Dordrecht (1994) CrossRefMATHGoogle Scholar
  7. Freitas, A.: On objective measures of rule interestingness. In: Second European Symposium on Principles of Data Mining & Knowledge Discovery. Springer, Berlin (1998) Google Scholar
  8. Gaines, B.: Transforming rules and trees into comprehensible knowledge structures. In: Fayyad, U.M., et al. (eds.) Advances in Knowledge Discovery and Data Mining, pp. 205–226. MIT Press, Cambridge (1996) Google Scholar
  9. Japkowicz, N., Shah, M.: Evaluating Learning Algorithms – a Classification Perspective. Cambridge University Press, Cambridge (2011) CrossRefMATHGoogle Scholar
  10. Lavesson, N., Davidsson, P.: Evaluating learning algorithms and classifiers. Int. J. Intell. Inf. Database Syst. 1(1), 37–52 (2007) Google Scholar
  11. Lavesson, N., Davidsson, P.: Generic methods for multi-criteria evaluation. In: Eighth SIAM International Conference on Data Mining. SIAM, Philadelphia (2008) Google Scholar
  12. Lavesson, N., Davidsson, P.: Approve: application-oriented validation and evaluation of supervised learners. In: The Fifth IEEE International Conference on Intelligent Systems, IS’2010. IEEE Press, New York (2010) Google Scholar
  13. Lavesson, N., Boldt, M., Davidsson, P., Jacobsson, A.: Learning to detect spyware using end user license agreements. Knowl. Inf. Syst. (2010) Google Scholar
  14. Menzies, T., Shepperd, M.: Special issue on repeatable results in software engineering prediction. Empirical Software Engineering 17(1–2) (2012) Google Scholar
  15. Nakhaeizadeh, G., Schnabl, A.: Development of multi-criteria metrics for evaluation of data mining algorithms. In: Third International Conference on Knowledge Discovery and Data Mining, pp. 37–42. AAAI Press, Melno Park (1997) Google Scholar
  16. Nakhaeizadeh, G., Schnabl, A.: Towards the personalization of algorithms evaluation in data mining. In: Fourth International Conference on Knowledge Discovery and Data Mining, pp. 289–293. AAAI Press, Melno Park (1998) Google Scholar
  17. Provost, F., Fawcett, T., Kohavi, R.: The case against accuracy estimation for comparing induction algorithms. In: 15th International Conference on Machine Learning, pp. 445–453. Morgan Kaufmann, San Francisco (1998) Google Scholar
  18. Roubens, M., Vincke, P.: Preference Modelling. Springer, Berlin (1985) CrossRefMATHGoogle Scholar
  19. Saaty, T.L.: The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation. McGraw-Hill, New York (1980) MATHGoogle Scholar
  20. Tsiporkova, E., Boeva, V.: Nonparametric recursive aggregation process. Kybernetika. J. Czech Soc. Cybern. Inf. Sci. 40(1) (2004) Google Scholar
  21. Tsiporkova, E., Boeva, V.: Multi-step ranking of alternatives in a multi-criteria and multi-expert decision making environment. Inf. Sci. 176(18) (2006) Google Scholar
  22. Tsiporkova, E., Tourwe, T., Boeva, V.: A collaborative decision support platform for product release definition. In: The Fifth International Conference on Internet and Web Applications and Services, ICIW 2010. IEEE Press, New York (2010) Google Scholar
  23. Vaidya, O.S., Kumar, S.: Analytic hierarchy process: an overview of applications. Eur. J. Oper. Res. 169(1), 1–29 (2004) CrossRefMathSciNetGoogle Scholar
  24. Wang, R.W., Strong, D.M.: Beyond accuracy: what data quality means to data consumers. J. Manag. Inf. Syst. 12(4), 5–34 (1996) MATHGoogle Scholar
  25. Wohlin, C., Andrews, A.A.: Prioritizing and assessing software project success factors and project characteristics using subjective data. Empir. Softw. Eng. 8(3), 285–308 (2003) CrossRefGoogle Scholar
  26. Yin, R.K.: Case Study Research: Design and Methods, 4th edn. Sage, London (2009) Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Niklas Lavesson
    • 1
  • Veselka Boeva
    • 2
  • Elena Tsiporkova
    • 3
  • Paul Davidsson
    • 4
  1. 1.Blekinge Institute of TechnologyKarlskronaSweden
  2. 2.Computer Systems and Technologies DepartmentTechnical University of Sofia, branch PlovdivPlovdivBulgaria
  3. 3.Software Engineering and ICT GroupSirris, The Collective Center for the Belgian Technological IndustryBrusselsBelgium
  4. 4.Malmö UniversityMalmöSweden

Personalised recommendations