Skip to main content
SpringerLink
Log in

A joint attention enhancement network for text classification applied to citizen complaint reporting

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Citizen complaint classification plays an important role in the construction of the smart city. For text data, the most expressive semantic information is reflected in the keyword of the text. With the proposed Transformer structure and further expansion of the model structure, natural language processing has embarked on a path of fine-tuning the pre-trained model based on the multi-headed attention mechanism. Although the above method works well, it further deepens the black box model of the network. To verify whether the multi-headed attention mechanism adds enough attention to the keyword information, this paper proposes a joint attention enhancement network that places the attention mechanism outside the main network model. This paper uses the idea of lexical frequency statistics to obtain keyword information through the macroscopic use of corpus contents and improves the attention through knowledge incorporation based on soft attention. In this paper, a comparison experiment is performed by the current hot open-source network models on Hugging Face. Experiments show that the proposed model improves about 10%-20% in accuracy compared with the different original models, while the network training time only increases about 5%. The joint enhancement network can identify the key region of input data more accurately and converge quickly.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data Availability

The datasets analysed during the current study are not publicly available due citizen privacy but are available from us on reasonable request.

References

  1. Elattar EE, Sabiha NA, Alsharef M, Metwaly MK, Abd-Elhady AM, Taha IBM (2020) Short term electric load forecasting using hybrid algorithm for smart cities. Appl Intell 50:3379–3399. https://doi.org/10.1007/s10489-020-01728-x

    Article  Google Scholar 

  2. Bhanu M, Priya S, Moreira JM, Chandra J (2022) ST-AGP: spatio-temporal aggregator predictor model for multi-step taxi-demand prediction in cities. Appl Intell:1573–7497. https://doi.org/10.1007/s10489-022-03475-7

  3. Luo J, Qiu Z, Xie G, Feng J, Hu J, Zhang X (2018) Research on civic hotline complaint text classification model based on word2vec. In: 2018 International conference on cyber-enabled distributed computing and knowledge discovery (cyberc), pp 180–1803. https://doi.org/10.1109/CyberC.2018.00044

  4. Madyatmadja ED, Yahya BN, Wijaya C (2022) Contextual text analytics framework for citizen report classification: a case study using the indonesian language. IEEE Access 10:31432–31444. https://doi.org/10.1109/ACCESS.2022.3158940

    Article  Google Scholar 

  5. Wu L, Noels L (2022) Recurrent neural networks (rnns) with dimensionality reduction and break down in computational mechanics; application to multi-scale localization step. Comput Methods Appl Mech Eng 390:114476. https://doi.org/10.1016/j.cma.2021.114476

    Article  MathSciNet  MATH  Google Scholar 

  6. Arbane M, Benlamri R, Brik Y, Alahmar AD (2023) Social media-based COVID-19 sentiment classification model using Bi-LSTM. Expert Syst Appl 212:118710. https://doi.org/10.1016/j.eswa.2022.118710

    Article  Google Scholar 

  7. Dai Y, Zhou Q, Leng M, Yang X, Wang Y (2022) Improving the bi-LSTM model with XGBoost and attention mechanism: a combined approach for short-term power load prediction. Appl Soft Comput 130:109632. https://doi.org/10.1016/j.asoc.2022.109632

    Article  Google Scholar 

  8. Ma M, Xu Y, Song L, Liu G (2022) Symmetric transformer-based network for unsupervised image registration. Knowl-Based Syst 257:109959. https://doi.org/10.1016/j.knosys.2022.109959

    Article  Google Scholar 

  9. Maron ME (1961) Automatic indexing: an experimental inquiry. J ACM 8(3):404–417. https://doi.org/10.1145/321075.321084

    Article  MATH  Google Scholar 

  10. Kim Y (2014) Convolutional neural networks for sentence classification. CoRR arXiv:1408.5882

  11. Xie J, Hou Y, Wang Y, Wang Q, Vorotnitsky YI (2020) Chinese text classification based on attention mechanism and feature-enhanced fusion neural network. Computing, vol 102(6). https://doi.org/10.1007/s00607-019-00766-9

  12. Li YY, Xu LB (2020) Improving user attribute classification with text and social network attention. Cognit Comput 11:459–468. https://doi.org/10.1007/s12559-019-9624-y

    Article  Google Scholar 

  13. Yan Y, Liu FA, Zhuang X, Ju J (2022) An R-transformer_BiLSTM model based on attention for multi-label text classification. Neural Process Lett:1–24. https://doi.org/10.1007/s11063-022-10938-y

  14. Liu YJ, Lv YS (2019) Attention-based biGRU-CNN for Chinese question classification. J Ambient Intell Human Comput:1868–5145. https://doi.org/10.1007/s12652-019-01344-9

  15. Xin W, Yi C, Li Q, Xu J, Leung H (2018) Combining contextual information by self-attention mechanism in convolutional neural networks for text classification. In: International conference on web information systems engineering. https://doi.org/10.1007/978-3-030-02922-7_31

  16. Li J, Jin K, Zhou D, Kubota N, Ju Z (2020) Attention mechanism-based cnn for facial expression recognition. Neurocomputing 411:340–350. https://doi.org/10.1016/j.neucom.2020.06.014

    Article  Google Scholar 

  17. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: 3rd International conference on learning representations, ICLR 2015 ; Conference date: 07 May 2015 through 09 May 2015

  18. Wu Y, Li W (2022) Aspect-level sentiment classification based on location and hybrid multi attention mechanism. Appl Intell 52(10):11539–11554. https://doi.org/10.1007/s10489-021-02966-3

    Article  Google Scholar 

  19. Qiao X, Peng C, Liu Z (2019) Word-character attention model for chinese text classification. Int J Mach Learn Cybern 10:3521–3537. https://doi.org/10.1007/s13042-019-00942-5

    Article  Google Scholar 

  20. Catelli R, Casola V, De Pietro G, Fujita H, Esposito M (2021) Combining contextualized word representation and sub-document level analysis through bi-LSTM+CRF architecture for clinical de-identification. Knowl-Based Syst 213:106649. https://doi.org/10.1016/j.knosys.2020.106649

    Article  Google Scholar 

  21. Wu X, Gao C, Lin M, Zang L, Hu S (2022) Text smoothing: enhance various data augmentation methods on text classification tasks. In: Proceedings of the 60th annual meeting of the association for computational linguistics (volume 2: short papers). Association for computational linguistics, pp 871–875. https://doi.org/10.18653/v1/2022.acl-short.97

  22. Zhu X, Zhu Y, Zhang L, Chen Y (2022) A BERT-based multi-semantic learning model with aspect-aware enhancement for aspect polarity classification. Appl Intell:1–15. https://doi.org/10.1007/s10489-022-03702-1

  23. Li M, Chen L, Zhao J (2021) Sentiment analysis of Chinese stock reviews based on BERT model. Appl Intell 51:5016–5024. https://doi.org/10.1007/s10489-020-02101-8

    Article  Google Scholar 

  24. Catelli R, Bevilacqua L, Mariniello N, Scotto Di Carlo V, Magaldi M, Fujita H, De Pietro G, Esposito M (2022) Cross lingual transfer learning for sentiment analysis of Italian TripAdvisor reviews. Expert Syst Appl 209:118246. https://doi.org/10.1016/j.eswa.2022.118246

    Article  Google Scholar 

  25. Pota M, Ventura M, Fujita H, Esposito M (2021) Multilingual evaluation of pre-processing for BERT-based sentiment analysis of tweets. Expert Syst Appl 181:115119. https://doi.org/10.1016/j.eswa.2021.115119

    Article  Google Scholar 

  26. Guarasci R, Silvestri S, De Pietro G, Fujita H, Esposito M (2022) BERT syntactic transfer: a computational experiment on Italian, French and English languages. Comput Speech Lang 71:101261. https://doi.org/10.1016/j.csl.2021.101261

    Article  Google Scholar 

  27. Catelli R, Fujita H, De Pietro G, Esposito M (2022) Deceptive reviews and sentiment polarity: effective link by exploiting BERT. Expert Syst Appl 209:118290. https://doi.org/10.1016/j.eswa.2022.118290

    Article  Google Scholar 

  28. Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies. Association for computational linguistics, pp 4171–4186. https://doi.org/10.18653/v1/N19-1423

  29. Jia K (2021) Chinese sentiment classification based on word2vec and vector arithmetic in human–robot conversation. Comput Electr Eng 95:107423. https://doi.org/10.1016/j.compeleceng.2021.107423

    Article  Google Scholar 

  30. Kim D, Seo D, Cho S, Kang P (2019) Multi-co-training for document classification using various document representations: Tf–idf, lda, and doc2vec. Inf Sci 477:15–29. https://doi.org/10.1016/j.ins.2018.10.006

    Article  Google Scholar 

  31. Cui Y, Che W, Liu T, Qin B, Yang Z (2021) Pre-training with whole word masking for chinese BERT. IEEE/ACM Trans Audio Speech Lang Process 29:3504–3514. https://doi.org/10.1109/TASLP.2021.3124365

    Article  Google Scholar 

  32. Cui Y, Che W, Liu T, Qin B, Wang S, Hu G (2020) Revisiting pre-trained models for chinese natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: findings. Association for computational linguistics, pp 657–668. https://www.aclweb.org/anthology/2020.findings-emnlp.58

  33. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) RoBERTa: a robustly optimized BERT pretraining approach. CoRR arXiv:1907.11692

  34. Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2019) ALBERT: a lite BERT for self-supervised learning of language representations. CoRR arXiv:1909.11942

Download references

Funding

This work was supported in part by the Beijing Natural Science Foundation under Grant L191017, and in part by the National Natural Science Foundation of China under Grant 61673049.

Author information

Authors and Affiliations

  1. School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing, 100044, China

    Yuanhang Wang & Yonghua Zhou

  2. Zhongguancun Smart City Co., Ltd, Beijing, 100071, China

    Yiduo Mei

Authors
  1. Yuanhang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yonghua Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yiduo Mei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yonghua Zhou.

Ethics declarations

Conflict of Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Zhou, Y. & Mei, Y. A joint attention enhancement network for text classification applied to citizen complaint reporting. Appl Intell 53, 19255–19265 (2023). https://doi.org/10.1007/s10489-023-04490-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04490-y

Keywords

Search

Navigation