Binary Classification Models Comparison: On the Similarity of Datasets and Confusion Matrix for Predictive Toxicology Applications

Abstract

Nowadays generating predictive models by applying machine learning and model ensembles techniques is a faster task facilitated by development of more user-friendly data mining tools. However, such progress raises the issues related to model management: once developed, many classifiers for example become accessible in collections of models. Choosing the relevant model from the collection can reduce costs of generating new predictive models: calculating the similarity of predictive models is the key to rank them, which may improve model selection or combination. For this aim we introduce a methodology to measure the similarity of classifiers by comparing their datasets, transfer functions and confusion matrices. We propose the Dataset Similarity Coefficient to calculate the similarity of datasets, and the Similarity of Models measure to calculate the similarity between such predictive models. In this paper we focus on toxicology applications of binary classification models. The results show that our methodology performs well in measuring models similarity from a collection of classifiers.