Abstract
Plagiarism is one of the leading problems in academic and industrial environments, which its goal is to find the similar items in a typical document or source code. This paper proposes an architecture based on a Long Short-Term Memory (LSTM) and attention mechanism called LSTM-AM-ABC boosted by a population-based approach for parameter initialization. Gradient-based optimization algorithms such as back-propagation (BP) are widely used in the literature for learning process in LSTM, attention mechanism, and feed-forward neural network, while they suffer from some problems such as getting stuck in local optima. To tackle this problem, population-based metaheuristic (PBMH) algorithms can be used. To this end, this paper employs a PBMH algorithm, artificial bee colony (ABC), to moderate the problem. Our proposed algorithm can find the initial values for model learning in all LSTM, attention mechanism, and feed-forward neural network, simultaneously. In other words, ABC algorithm finds a promising point for starting BP algorithm. For evaluation, we compare our proposed algorithm with both conventional and population-based methods. The results clearly show that the proposed method can provide competitive performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
El Moatez Billah Nagoudi, A.K., Cherroun, H., Schwab, D.: 2L-APD: a two-level plagiarism detection system for Arabic documents. Cybern. Inf. Technol. 18(1), 124–138 (2018)
He, H., Gimpel, K., Lin, J.: Multi-perspective sentence similarity modeling with convolutional neural networks. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (2015)
Joodaki, M., Dowlatshahi, M.B., Joodaki, N.Z.: An ensemble feature selection algorithm based on PageRank centrality and fuzzy logic. Knowl. Based Syst. 223, 107538 (2021)
Joodaki, M., Ghadiri, N., Maleki, Z., Shahreza, M.L.: A scalable random walk with restart on heterogeneous networks with Apache Spark for ranking disease-related genes through type-II fuzzy data fusion. J. Biomed. Inf. 115, 103688 (2021)
Pontes, E.L., Huet, S., Linhares, A.C., Torres-Moreno, J.-M.: Predicting the semantic textual similarity with siamese CNN and LSTM. arXiv preprint arXiv:1810.10641 (2018)
Sanborn, A., Skryzalin, J.: Deep learning for semantic similarity. In: CS224d: Deep Learning for Natural Language Processing. Stanford University, Stanford (2015)
Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)
Moravvej, S.V., Joodaki, M., Kahaki, M.J.M., Sartakhti, M.S.: A method based on an attention mechanism to measure the similarity of two sentences. In: 2021 7th International Conference on Web Research (ICWR). IEEE (2021)
Laskar, M.T.R., Huang, X., Hoque, E.: Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task. In: Proceedings of The 12th Language Resources and Evaluation Conference (2020)
Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018)
Chen, Q., Hu, Q., Huang, J.X., He, L.: CA-RNN: using context-aligned recurrent neural networks for modeling sentence similarity. In: Proceedings of the AAAI Conference on Artificial Intelligence (2018)
Bao, W., Bao, W., Du, J., Yang, Y., Zhao, X.: Attentive Siamese LSTM network for semantic textual similarity measure. In: 2018 International Conference on Asian Language Processing (IALP). IEEE (2018)
Chi, Z., Zhang, B.: A sentence similarity estimation method based on improved siamese network. J. Intell. Learn. Syst. Appl. 10(4), 121–134 (2018)
Ashkoofaraz, S.Y., Izadi, S.N.H., Tajmirriahi, M., Roshanzamir, M., Soureshjani, M.A., Moravvej, S.V., Palhang, M.: AIUT3D 2018 Soccer Simulation 3D League Team Description Paper
Vakilian, S., Moravvej, S.V., Fanian, A.: Using the cuckoo algorithm to optimizing the response time and energy consumption cost of fog nodes by considering collaboration in the fog layer. In: 2021 5th International Conference on Internet of Things and Applications (IoT). IEEE (2021)
Vakilian, S., Moravvej, S.V., Fanian, A.: Using the artificial bee colony (ABC) algorithm in collaboration with the fog nodes in the Internet of Things three-layer architecture. In: 2021 29th Iranian Conference on Electrical Engineering (ICEE) (2021)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Sartakhti, M.S., Kahaki, M.J.M., Moravvej, S.V., Javadi Joortani, M., Bagheri, A.: Persian language model based on BiLSTM model on COVID-19 Corpus. In: 2021 5th International Conference on Pattern Recognition and Image Analysis (IPRIA). IEEE (2021)
Graves, A.: Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013)
Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18(5–6), 602–610 (2005)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Moravvej, S.V., Kahaki, M.J.M., Sartakhti, M.S., Mirzaei, A.: A method based on attention mechanism using bidirectional long-short term memory (BLSTM) for question answering. In: 2021 29th Iranian Conference on Electrical Engineering (ICEE) (2021)
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., Zamparelli, R.: A SICK cure for the evaluation of compositional distributional semantic models. In: Lrec. Reykjavik (2014)
Phansalkar, V., Sastry, P.: Analysis of the back-propagation algorithm with momentum. IEEE Trans. Neural Netw. 5(3), 505–506 (1994)
Hagan, M., Demuth, H., Beale, M.: Neural Network Design (PWS, Boston, MA). Google Scholar Google Scholar Digital Library Digital Library (1996)
Yu, C.-C., Liu, B.-D.: A backpropagation algorithm with adaptive learning rate and momentum coefficient. In: Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No. 02CH37290). IEEE (2002)
Battiti, R.: First-and second-order methods for learning: between steepest descent and Newton’s method. Neural Comput. 4(2), 141–166 (1992)
Foresee, F.D., Hagan, M.T.: Gauss-Newton approximation to Bayesian learning. In: Proceedings of international conference on neural networks (ICNN'97). IEEE (1997)
Mirjalili, S., Mirjalili, S.M., Lewis, A.: Grey wolf optimizer. Adv. Eng. Softw. 69, 46–61 (2014)
Yang, X.-S.: A new metaheuristic bat-inspired algorithm. In: Nature Inspired Cooperative Strategies for Optimization (NICSO 2010), pp. 65–74. Springer (2010)
Yang, X.-S., Deb, S.: Cuckoo search via Lévy flights. In: 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC). IEEE (2009)
Mirjalili, S., Lewis, A.: The whale optimization algorithm. Adv. Eng. Softw. 95, 51–67 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Moravvej, S.V., Mousavirad, S.J., Moghadam, M.H., Saadatmand, M. (2021). An LSTM-Based Plagiarism Detection via Attention Mechanism and a Population-Based Approach for Pre-training Parameters with Imbalanced Classes. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Lecture Notes in Computer Science(), vol 13110. Springer, Cham. https://doi.org/10.1007/978-3-030-92238-2_57
Download citation
DOI: https://doi.org/10.1007/978-3-030-92238-2_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92237-5
Online ISBN: 978-3-030-92238-2
eBook Packages: Computer ScienceComputer Science (R0)