A New Supervised Clustering Framework Using Multi Discriminative Parts and Expectation–Maximization Approach for a Fine-Grained Animal Breed Classification (SC-MPEM)

Abstract

Fine-grained image classification is active research in the field of computer vision. Specifically, animal breed classification is an arduous task due to the challenges in camera traps images like occlusion, camouflage, poor illumination, pose variation, etc. In this paper, we propose a fine-grained animal breed classification model using supervised clustering based on Multi Part-Convolutional Neural Network (MP-CNN) and Expectation–Maximization (EM) clustering. The proposed model follows a straightforward pipeline that combines the deep feature extraction using the CNN pre-trained on ImageNet and classifies unsupervised data using EM clustering. Further, we also propose a multi discriminative part selection and detection for the precise classification of animal breeds without using bounding box and annotations on both training and testing phases. The model is tested on several benchmark datasets for animals, including the largest camera trap Snapshot Serengeti dataset and has achieved a cumulative accuracy of 98.4%. The results from the proposed model strengthen the belief that supervised training of deep CNN on a large and versatile dataset, extracts better features than most of the traditional approaches, even for the unsupervised tasks.

Appendix

The two important factors in deep learning models are the dataset split and dataset balancing. The proportion of train and test data plays an important role in computational time and complexity, besides influencing the performance. Similarly, dataset balancing is an essential step in training. Hence, the first set of preliminary results with SC-MPEM is analysing the effects of dataset splitting and data balancing.

The effects of different proportions of training data on the accuracy of the proposed system are evaluated on the benchmark animal datasets and the result for the same is presented in Table 10. The proposed model is validated against a different proportion of training data like 10%, 20%, 30%, 40% and 60%. SC-MPEM has achieved good accuracy even with 10% training data. Nevertheless, the accuracy increases with an increase in the proportion of training data. However, the accuracy starts saturating at around 40% of training data and further on, it does not have much impact on the performance. But, increasing the size of training data increases the computational cost and complexity. Hence, the proposed system is trained with 40% training data. Thus, the following experiments are trialed with a 40:40:20 proportions, where 40% is for training and testing and the remaining 20% is for validation. The effect of a balanced vs. imbalanced dataset is studied and the result is presented in Fig. 22. Following the previous experiment, the dataset proportion is maintained at 40:40:20 on all the benchmark datasets.

Table 10 Effects of different proportion of training data on benchmark datasets

Fig. 22

Performance of balanced versus imbalanced dataset

It is inferred that an imbalanced dataset affects performance. The difference between the balanced and imbalanced dataset is hard to neglect. Generally, it is good to have a balanced dataset for any classification problem, as an imbalanced dataset will have a bias favouring the majority class. Usually, the problem of an imbalanced dataset is predominant in a classification problem, specifically in a multi-class classification problem, where more than one class may have minimal data. Hence, we synthetically balanced the dataset using SMOTE. However, it is not always the case that a balanced dataset performs well. If the classes are balanced, then we will miss some valuable patterns in the dataset. Hence, it is preferred to have a large dataset though unbalanced. Balancing a large dataset by data augmentation techniques doesn’t make much difference in the overall results. Among the datasets, the Snapshot Serengeti is the largest dataset and so data augmentation had minimal effect on the performance when compared to the other three datasets. Hence, the Snapshot Serengeti dataset was not balanced, however, the remaining datasets were balanced. There exists a trade-off in every application and so is the performance metrics. For a balanced dataset, accuracy is the best metric. On the other hand, for an imbalanced dataset, precision and recall are the appropriate metrics to measure the performance.

The second set of results discusses the details of transfer learning (see Table 11). Specifically, we discuss various details like layer from which the features are extracted, NMI (Normalized Mutual Information) score and the time complexity. For estimating the NMI, we ensured that no image contains more than one label, as it makes it difficult to analyze if the clustering classified the images accurately or not. In order to understand the above three factors, we tried clustering the dataset using 5 different CNN architectures, 8 different clustering algorithms, and various choices of layers. For assessing the results of these, we utilized the NMI score and time complexity. For simplicity, we used the default value for all the hyper-parameters. Both KM and MKM were randomly initialized. Every experiment was run for a total of 10 times and an average value is taken over these 10 runs. In Table 11, the layer names are unchanged from the architecture and the NMI score is given in bold. The time complexity of each clustering for the various layers is also given below the NMI score. The best results are highlighted in bold italic values.

Table 11 Details of transfer learning—architectures, feature extraction layer, NMI score and clustering

From the table, it is inferred that features extracted from the penultimate layers of Inception v3 network and clustered using EM clustering produced the state-of-the-art result. Although AHC had the least time complexity, however, we do not consider it due to its poor NMI score. Besides, the tenfold cross-validation also stands in favor of the Inception v3 network. The result of each of the model is depicted in Fig. 23.

Fig. 23

Cross-validation results of the networks

The final sets of results discuss the misclassification and how they are tackled by MHKC. Figure 24 represents the training accuracy for the class pre-trained using MHKC. The highest score for the chosen images is 6.60 and the least one is 1.96. The precision for the training dataset is found to be 100%, which is a good indicator for better testing accuracy. We have improved the accuracy of misclassified horse images to 100% with MHKC. With 100% PR on training data, we achieved 99.97% on testing data. This is far better than the accuracy of TensorFlow and the same is depicted in Fig. 25. The feature vector is pre-computed and so the accuracy of the classifier corresponds merely to its kernel functions.

Fig. 24

Training accuracy and Precision-recall curve of misclassified class retrained using MHKC

Fig. 25

Testing accuracy and Precision-recall curve of MHKC re-trained class

As additional information regarding the performance of our proposed model on the Snapshot Serengeti dataset, we present the confusion matrix of the dataset in Fig. 26. Despite being the largest camera trap dataset, it has not been predominantly used for testing. Some of the reasons are the size of the dataset and the lack of resources to train and test such a huge dataset [5]. To counter this problem, Gomez et al. [5] have trained only 26 classes out of the 40 mammalian classes. In fact, the original Snapshot Serengeti has 48 classes, of which 40 classes are mammalian species. The remaining 8 classes include humans, birds, rodents, etc. We have intentionally excluded these 8 classes as they do not belong to animal species and by adding them; we needlessly increase the computational burden. Thus, we have trained and tested only 40 mammalian species as Norouzzadeh et al. [6].

Fig. 26

Confusion matrix of Snapshot Serengeti dataset