Abstract
We have revisited the vivid discussion in the QSAR-related literature concerning the use of external versus cross-validation , and have presented a thorough statistical comparison of model performance parameters with the recently published SRD (sum of (absolute) ranking differences) method and analysis of variance (ANOVA) . Two case studies were investigated, one of which has exclusively used external performance merits. The SRD methodology coupled with ANOVA shows unambiguously for both case studies that the performance merits are significantly different, independently from data preprocessing . While external merits are generally less consistent (farther from the reference) than training and cross-validation based merits, a clear ordering and a grouping pattern of them could be acquired. The results presented here corroborate our earlier, recently published findings (SAR QSAR Environ. Res., 2015, 26, 683–700) that external validation is not necessarily a wise choice, and is frequently comparable to a random evaluation of the models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Andrić, F., Bajusz, D., Rácz, A., et al. (2016). Multivariate assessment of lipophilicity scales—Computational and reversed phase thin-layer chromatographic indices. Journal of Pharmaceutical and Biomedical Analysis, 127, 81–93. doi:10.1016/j.jpba.2016.04.001.
Bajusz, D., Rácz, A., & Héberger, K. (2015). Why is Tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of Cheminformatics, 7, 20. doi:10.1186/s13321-015-0069-3.
Chirico, N., & Gramatica, P. (2011). Real external predictivity of QSAR models: How to evaluate it? Comparison of different validation criteria and proposal of using the concordance correlation coefficient. Journal of Chemical Information and Modeling, 51, 2320–2335. doi:10.1021/ci200211n.
Consonni, V., Ballabio, D., & Todeschini, R. (2010). Evaluation of model predictive ability by external validation techniques. Journal of Chemometrics, 24, 194–201. doi:10.1002/cem.1290.
Esbensen, K. H., & Geladi, P. (2010). Principles of proper validation: Use and abuse of re-sampling for validation. Journal of Chemometrics, 24, 168–187. doi:10.1002/cem.1310.
Gramatica, P. (2014). External evaluation of QSAR models, in addition to cross-validation: Verification of predictive capability on totally new chemicals. Molecular Informatics, 33, 311–314. doi:10.1002/minf.201400030.
Gramatica, P., Cassani, S., Roy, P. P., et al. (2012). QSAR Modeling is not “push a button and find a correlation”: A case study of toxicity of (Benzo-)triazoles on Algae. Molecular Informatics, 31, 817–835. doi:10.1002/minf.201200075.
Gramatica, P., Chirico, N., Papa, E., et al. (2013). QSARINS: A new software for the development, analysis, and validation of QSAR MLR models. Journal of Computational Chemistry, 34, 2121–2132. doi:10.1002/jcc.23361.
Gütlein, M., Helma, C., Karwath, A., & Kramer, S. (2013). A large-scale empirical evaluation of cross-validation and external test set validation in (Q)SAR. Molecular Informatics, 32, 516–528. doi:10.1002/minf.201200134.
Hastie, T., Tibshirani, R., & Friedman, J. H. (2009). Cross-Validation. The elements of statistical learning: Data mining, inference, and prediction (2nd ed., pp. 241–249). New York: Springer.
Hawkins, D. M. (2004). The problem of overfitting. Journal of Chemical Information and Computer Sciences, 44, 1–12. doi:10.1021/ci0342472.
Hawkins, D. M., Basak, S. C., & Mills, D. (2003). Assessing model fit by cross-validation. Journal of Chemical Information and Computer Sciences, 43, 579–586. doi:10.1021/ci025626i.
Héberger, K. (2010). Sum of ranking differences compares methods or models fairly. TrAC Trends in Analytical Chemistry, 29, 101–109.
Héberger, K., Kolarević, S., Kračun-Kolarević, M., et al. (2014). Evaluation of single-cell gel electrophoresis data: Combination of variance analysis with sum of ranking differences. Mutation Research, Genetic Toxicology and Environmental Mutagenesis, 771, 15–22. doi:10.1016/j.mrgentox.2014.04.028.
Kollár-Hunek, K., & Héberger, K. (2013). Method and model comparison by sum of ranking differences in cases of repeated observations (ties). Chemometrics and Intelligent Laboratory Systems, 127, 139–146. doi:10.1016/j.chemolab.2013.06.007.
Lin, L. I.-K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45, 255–268.
Lin, L. I.-K. (1992). Assay validation using the concordance correlation coefficient. Biometrics, 48, 599. doi:10.2307/2532314.
Lindman, H. R. (1991). Analysis of variance in experimental design. New York: Springer.
Miller, A. (1990). Subset selection in regression. London: Chapman and Hall.
Rácz, A., Bajusz, D., & Héberger, K. (2015). Consistency of QSAR models: Correct split of training and test sets, ranking of models and performance parameters. SAR and QSAR in Environmental Research, 26, 683–700. doi:10.1080/1062936X.2015.1084647.
Roy, K., Das, R. N., Ambure, P., & Aher, R. B. (2016). Be aware of error measures. Further studies on validation of predictive QSAR models. Chemometrics and Intelligent Laboratory Systems, 152, 18–33. doi:10.1016/j.chemolab.2016.01.008.
Schüürmann, G., Ebert, R.-U., Chen, J., et al. (2008). External validation and prediction employing the predictive squared correlation coefficient test set activity mean vs training set activity mean. Journal of Chemical Information and Modeling, 48, 2140–2145. doi:10.1021/ci800253u.
Shi, L. M., Fang, H., Tong, W., et al. (2001). QSAR models using a large diverse set of estrogens. Journal of Chemical Information and Modeling, 41, 186–195. doi:10.1021/ci000066d.
Silla, J. M., Nunes, C. A., Cormanich, R. A., et al. (2011). MIA-QSPR and effect of variable selection on the modeling of kinetic parameters related to activities of modified peptides against dengue type 2. Chemometrics and Intelligent Laboratory Systems, 108, 146–149. doi:10.1016/j.chemolab.2011.06.009.
Acknowledgement
The work was supported by the Hungarian Scientific Research Fund (OTKA, grant number K 119269).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Héberger, K., Rácz, A., Bajusz, D. (2017). Which Performance Parameters Are Best Suited to Assess the Predictive Ability of Models?. In: Roy, K. (eds) Advances in QSAR Modeling. Challenges and Advances in Computational Chemistry and Physics, vol 24. Springer, Cham. https://doi.org/10.1007/978-3-319-56850-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-56850-8_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-56849-2
Online ISBN: 978-3-319-56850-8
eBook Packages: Chemistry and Materials ScienceChemistry and Material Science (R0)