Evaluating pre-trained models for user feedback analysis in software engineering: a study on classification of app-reviews

Abstract

Context

Automatic classification of mobile applications users’ feedback is studied for different areas of software engineering. However, supervised classification requires a lot of manually labeled data, and with introducing new classes or new platforms, new labeled data and models are required. Employing Pre-trained neural Language Models (PLMs) have found success in the Natural Language Processing field. However, their applicability has not been explored for app review classification.

Objective

We evaluate using PLMs for issue classification from app reviews in multiple settings and compare them with the existing models.

Method

We set up different studies to evaluate the performance and time efficiency of PLMs compared to Prior approaches on six datasets: binary vs. multi-class, zero-shot, multi-task, and multi-resource settings. In addition, we train and study domain-specific (Custom) PLMs by incorporating app reviews in the pre-training. We report Micro and Macro Precision, Recall, and F1 scores and the time required for training and predicting with the models.

Results

Our results show that PLMs can classify the app issues with higher scores, except in multi-resource setting. On the largest dataset, results are improved by 13 and 8 micro- and macro-average F1-scores, respectively, compared to the Prior approaches. Domain-specific PLMs achieve the highest scores in all settings with less prediction time, and they benefit from pre-training with a larger number of app reviews. On the largest dataset, we obtain 98 and 92 micro- and macro-average F1-score (from 4.5 to 8.3 more F1-score compared to general pre-trained models), 71 F1-score in zero-shot setting, and 93 and 92 F1-score in multi-task and multi-resource settings, respectively, using the large domain-specific PLMs.

Conclusion

Although prior approaches achieve high scores in some settings, PLMs are the only models that can work well in the zero-shot setting. When trained on the app review dataset, the Custom PLMs have higher performance and lower prediction times.

Appendix

Table 3 Performance and efficiency of the Prior approaches (RQ1)

Table 4 Performance and efficiency of the pre-trained models (RQ1)

Table 5 RQ2 results: Performance of Custom pre-trained models (i.e., trained on app reviews data), shown by ‘C’ in front of the model name

Table 6 RQ2 results: Performance of Custom pre-trained models (i.e., trained on app reviews data), shown by ‘C’ in front of the model name

Table 7 RQ3-1 results for datasets $D_1$–$D_4$: Comparison among the PLMs, Custom PLMs (i.e., trained on app reviews data), and the Prior approaches for binary classification

Table 8 RQ3-1 results for datasets $D_5$–$D_7$: Comparison among the PLMs, Custom PLMs (i.e., trained on app reviews data), and the Prior approaches for binary classification

Table 9 RQ3-2 results: Comparison among the best PLMs (from RQ1 results), Custom PLMs (i.e., trained on app reviews data), and AR-Miner in zero-shot classification setting

Table 10 RQ3-3 results for Category classification task: Comparing the performance of PLMs and Custom PLMs (i.e., trained on app reviews data) in multi-task setting

Table 11 RQ3-3 results for Sentiment classification task: Comparing the performance of PLMs and Custom PLMs (i.e., trained on app reviews data) in multi-task setting

Table 12 RQ3-4 results for multi-resource setting: Comparing the performance of Prior approaches, PLMs, and Custom PLMs (i.e., trained on app reviews data) when the models are trained on app reviews and tested on Twitter data

Table 13 RQ3-4 results for multi-resource setting: Comparing the performance of Prior approaches, PLMs, and Custom PLMs (i.e., trained on app reviews data) when the models are trained on Twitter data and are tested on app reviews

Fig. 16

Prediction times of the Prior and PLMs for app issue classification (RQ1). Each line represents a dataset. The prediction times of $D_7$ are the highest, which can be related to the size of this dataset

Fig. 17

Prediction times of the Prior and PLMs for app issue classification (RQ1). Each model is shown in a colored line. The trend of the prediction time for the datasets is similar for all models, which can be related to the characteristics and size of each dataset. Among all PLMs, XLNET has the highest and ALBERT has the lowest prediction time

Fig. 18

Prediction times of the models in binary classification setting (RQ3). The XLNET is not shown in the plot as its prediction times are higher than other models with a large gap. The Custom ALBERT-based models have the lowest scores for $D_4$, $D_5$, and $D_7$ datasets. RoBERTa-based models have higher prediction scores than ALBERT-based ones

Fig. 19

Prediction times of the models in zero-shot setting (RQ3). The ALBERT-based models have lower prediction times compared to the RoBERTa-based models

Fig. 20

Prediction counts by each model divided by the class labels in dataset $D_1$. The counts are not normalized to show the differences among the classes

Fig. 21

Prediction counts by each model divided by the class labels in dataset $D_2$. The counts are not normalized to show the differences among the classes

Fig. 22

Prediction counts by each model divided by the class labels in dataset $D_3$. The counts are not normalized to show the differences among the classes

Fig. 23

Prediction counts by each model divided by the class labels in dataset $D_4$. The counts are not normalized to show the differences among the classes

Fig. 24

Prediction counts by each model divided by the class labels in dataset $D_5$. The counts are not normalized to show the differences among the classes

Fig. 25

Prediction counts by each model divided by the class labels in dataset $D_6$. The counts are not normalized to show the differences among the classes

Fig. 26

Prediction counts by each model divided by the class labels in dataset $D_7$. The counts are not normalized to show the differences among the classes