Abstract
The dialect of a machine learning model is comprised of the features encountered during training. Nonetheless, as time passes, a deployed machine learning model may encounter certain features for the first time. In conventional machine learning approaches, newly observed features are typically discarded during testing data sample preprocessing. In lifelong machine learning, newly observed features may have appeared in the feature space of previously learned tasks; consequently, the knowledge associated with those features present in the knowledge base is incorporated to handle these features. However, there may be some features that have yet to appear in the knowledge base; lifelong machine learning also discards such features. Features that were not seen before are called unseen features. In this paper, we propose an enhanced lifelong machine learning framework for handling unseen features during the testing phase that incorporates relative knowledge. To extract relative knowledge, we retrieve semantically similar features using a language model. In addition, semantically similar features are examined in the knowledge base, and the knowledge of those present in the knowledge base is incorporated in order to deal with unseen features. Experiments conducted on the Amazon review dataset indicate that the proposed method outperforms three baselines and is competitive with state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bayes, T.: An essay towards solving a problem in the doctrine of chances. Philos. Trans. R. Soc. Lond. (53), 370–418 (1763)
Bentley, J.L.: Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975). https://doi.org/10.1145/361002.361007
Chen, Z., Liu, B.: Lifelong Machine Learning, 2nd edn. Morgan & Claypool Publishers, San Rafael (2018)
Chen, Z., Ma, N., Liu, B.: Lifelong learning for sentiment classification. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers, Beijing, China), pp. 750–756. Association for Computational Linguistics (2015). https://doi.org/10.3115/v1/P15-2123
Cover, T.: Estimation by the nearest neighbor rule. IEEE Trans. Inf. Theory 14(1), 50–55 (1968). https://doi.org/10.1109/TIT.1968.1054098
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, Minneapolis, MN, USA, 2–7 June, pp. 4171–4186. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/n19-1423
Geng, B., Yang, M., Yuan, F., Wang, S., Ao, X., Xu, R.: Iterative network pruning with uncertainty regularization for lifelong sentiment classification. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2021, pp. 1229–1238. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3404835.3462902
Hoi, S.C., Sahoo, D., Lu, J., Zhao, P.: Online learning: a comprehensive survey. Neurocomputing 459, 249–289 (2021). https://doi.org/10.1016/j.neucom.2021.04.112
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: Fasttext.zip: compressing text classification models. CoRR abs/1612.03651 (2016). https://doi.org/10.48550/arXiv.1612.03651
Ke, Z., Liu, B., Wang, H., Shu, L.: Continual learning with knowledge transfer for sentiment classification. In: Hutter, F., Kersting, K., Lijffijt, J., Valera, I. (eds.) ECML PKDD 2020. LNCS (LNAI), vol. 12459, pp. 683–698. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67664-3_41
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of the 1st International Conference on Learning Representations, ICLR, Scottsdale, Arizona, USA, 2–4 May, Workshop Track Proceedings (2013). https://doi.org/10.48550/arXiv.1301.3781
Mosteller, F., Wallace, D.L.: The Federalist: Inference and Disputed Authorship. Addison-Wesley, Boston (1964)
Nigam, K., McCallum, A., Thrun, S., Mitchell, T.: Learning to classify text from labeled and unlabeled documents. In: Proceedings of the 15th AAAI Conference on Artificial Intelligence, pp. 792–799. AAAI Press (1998). https://doi.org/10.5555/295240.295806
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010). https://doi.org/10.1109/TKDE.2009.191
Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up? Sentiment classification using machine learning techniques. In: Proceedings of the 7th Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 79–86. Association for Computational Linguistics (2002). https://doi.org/10.3115/1118693.1118704
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 25–29 October 2014, Doha, Qatar, pp. 1532–1543. ACL (2014). https://doi.org/10.3115/v1/D14-1162
Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 2227–2237. Association for Computational Linguistics (2018). https://doi.org/10.18653/v1/N18-1202
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)
Ruvolo, P., Eaton, E.: Ella: an efficient lifelong learning algorithm. In: Proceedings of the 30th International Conference on Machine Learning, pp. 507–515. JMLR.org (2013)
Thrun, S.: Is learning the n-th thing any easier than learning the first? In: Proceedings of the Conference on Advances in Neural Information Processing Systems, pp. 640–646. The MIT Press (1996)
Thrun, S., Mitchell, T.M.: Lifelong robot learning. Robot. Auton. Syst. 15(1), 25–46 (1995)
Wang, H., Liu, B., Wang, S., Ma, N., Yang, Y.: Forward and backward knowledge transfer for sentiment classification. In: Proceedings of The 11th Asian Conference on Machine Learning, ACML, pp. 457–472. PMLR (2019). https://doi.org/10.48550/arXiv.1906.0350
Wang, H., Wang, S., Mazumder, S., Liu, B., Yang, Y., Li, T.: Bayes-enhanced lifelong attention networks for sentiment classification. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 580–591. International Committee on Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.coling-main.50
Wang, S., Zhou, M., Mazumder, S., Liu, B., Chang, Y.: Disentangling aspect and opinion words in target-based sentiment analysis using lifelong learning. CoRR abs/1802.05818, pp. 1–7 (2018). https://doi.org/10.48550/arXiv.1802.05818
Wasi, N.A., Abulaish, M.: An unseen features enhanced text classification approach. In: The Procedings of the International Joint Conference on Neural Networks (IJCNN), Queensland, Australia, p. 8 (2023)
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: generalized autoregressive pretraining for language understanding. In: Proceedings of the 32th Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019). https://doi.org/10.5555/3454287.3454804
Zhang, Y., Yang, Q.: A survey on multi-task learning. IEEE Trans. Knowl. Data Eng. 34(12), 5586–5609 (2022). https://doi.org/10.1109/TKDE.2021.3070203
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wasi, N.A., Abulaish, M. (2023). An Unseen Features-Enriched Lifelong Machine Learning Framework. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2023. ICCSA 2023. Lecture Notes in Computer Science, vol 13957. Springer, Cham. https://doi.org/10.1007/978-3-031-36808-0_34
Download citation
DOI: https://doi.org/10.1007/978-3-031-36808-0_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36807-3
Online ISBN: 978-3-031-36808-0
eBook Packages: Computer ScienceComputer Science (R0)