Relative Confusion Matrix: An Efficient Visualization for the Comparison of Classification Models

Abstract

Recent machine learning and deep learning algorithms have made important breakthroughs in tasks such as classification tasks. Multiple efficient methods and model architectures have been proposed to continuously improve the performance on these tasks. To determine the best classifier for a given task, machine learning experts generally rely on metrics such as the accuracy that measure both the number of correct predictions and the number of prediction errors. These metrics can be computed at a global scale, which means one value represents the performance of the whole model. They can also be computed at a class scale, which means the performance of the model is considered for each class individually. The most common way to efficiently represent the class-wise performance of such models is the standard Confusion Matrix. Yet, if this representation can efficiently highlight both well discriminated classes and problematic classes of a single classifier, the very few works leverage the matrix structure of this visualization to compare several models at a class scale. In this paper, we present the Relative Confusion Matrix (RCM), a matrix-based visualization leveraging a color encoding and a set of symbols to highlight class-wise differences of performance between two models. We conducted an evaluation with users to compare RCM with two confusion matrix design variants. Our results show that RCM’s encoding leads to a more efficient comparison of two models than existing approaches. This extended version develops the design choices with more details. It also includes an application scenario of RCM to compare various models generated through a Deep Neural Network visual pruning approach.