Abstract
The model evaluation assesses the performance of models on new data. The procedure helps to select the optimal predictive model based on appropriate evaluation scores that reflect the model's quality. Evaluation scores differ for classification and regression. The former commonly uses area under the ROC curve and F1 scores, while the latter uses the mean average error and the mean squared error. In both cases, evaluation requires the use of k-fold cross-validation, or a similar sampling technique, to ensure the model is tested on a different set of data than it was trained on.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Antonio, N., de Almeida, A., & Nunes, L. (2019a). Hotel booking demand datasets. Data in Brief, 22, 41–49. https://doi.org/10.1016/j.dib.2018.11.126
Antonio, N., de Almeida, A., & Nunes, L. (2019b). Big data in hotel revenue management: Exploring cancellation drivers to gain insights into booking cancellation behavior. Cornell Hospitality Quarterly, 60(4), 298–319. https://doi.org/10.1177/1938965519851466
Breunig, M. M., Kriegel, H.-P., & Ng, R. (2000). LOF: Identifying density-based local outliers. Proceedings of ACM SIGMOD Record, 29(2), 93–104. https://doi.org/10.1145/335191.335388
Carreño, A., Inza, I., & Lozano, J. A. (2020). Analyzing rare event, anomaly, novelty and outlier detection terms under the supervised classification framework. Artificial Intelligence Review, 53, 3575–3594. https://link.springer.com/article/10.1007/s10462-019-09771-y
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Demšar, J., Curk, T., Erjavec, A., et al. (2013). Orange: Data mining toolbox in python. Journal of Machine Learning Research, 14(Aug), 2349–2353. http://jmlr.org/papers/v14/demsar13a.html
Drummond, C., & Holte, R. C. (2006). Cost curves: An improved method for visualizing classifier performance. Machine Learning, 65, 95–130. https://link.springer.com/article/10.1007/s10994-006-8199-5
Elkan, C. (2001). The foundations of cost-sensitive learning. Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, 2001, 973–978. http://web.cs.iastate.edu/~honavar/elkan.pdf
Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. https://doi.org/10.1016/j.patrec.2005.10.010
Hernandez-Orallo, J., Flach, P., & Ferri, C. (2012). A unified view of performance metrics: Translating threshold choice into expected classification loss. Journal of Machine Learning Research, 13, 2813–2869. https://www.jmlr.org/papers/volume13/hernandez-orallo12a/hernandez-orallo12a.pdf
Kawaguchi, K., Pack Kaelbling, L., & Bengio, Y. (2017). Generalization in deep learning. arXiv preprint. https://arxiv.org/abs/1710.05468
Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008). Isolation forest. In Eighth IEEE International Conference on Data Mining 2008. https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf
Možina, M., Demšar, J., Kattan, M., & Zupan, B. (2004). Nomograms for visualization of naive Bayesian classifier. In European conference on principles of data mining and knowledge discovery (pp. 337–348). Springer. https://doi.org/10.1007/978-3-540-30116-5_32
Platt, J. (2000). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in large margin classifiers (pp. 61–74). MIT Press.
Precision and recall. (n.d.). In Wikipedia. Retrieved 5 February, 2021, from https://en.wikipedia.org/wiki/Precision_and_recall.
Vapnik, V. (2000). The nature of statistical learning theory. Springer Science & Business Media. https://doi.org/10.1007/978-1-4757-3264-1
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Further Readings and Other Sources
Further Readings and Other Sources
-
Flach, P. A., & Kull, M. (2015). Precision-recall-gain curves: PR analysis done right. Advances in Neural Information Processing Systems, 838–846. Available at: https://papers.nips.cc/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf
-
Hastie, T., Tibshirani, R., & Friedman, J. (2017). Elements of statistical learning. Chap. 7, pp. 219–260. Available at: https://web.stanford.edu/~hastie/ElemStatLearn/printings/ESLII_print12_toc.pdf.
-
Orange video tutorials.: https://www.youtube.com/c/OrangeDataMining
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Pretnar Žagar, A., Demšar, J. (2022). Model Evaluation. In: Egger, R. (eds) Applied Data Science in Tourism. Tourism on the Verge. Springer, Cham. https://doi.org/10.1007/978-3-030-88389-8_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-88389-8_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88388-1
Online ISBN: 978-3-030-88389-8
eBook Packages: Business and ManagementBusiness and Management (R0)