Financial Numeral Classification Model Based on BERT

  • Wei Wang
  • Maofu LiuEmail author
  • Yukun Zhang
  • Junyi Xiang
  • Ruibin Mao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11966)


Numerals contain rich semantic information in financial documents, and they play significant roles in financial data analysis and financial decision making. This paper proposes a model based on the Bidirectional Encoder Representations from Transformers (BERT) to identify the category and subcategory of a numeral in financial documents. Our model holds the obvious advantages in the fine-grained numeral understanding and achieves good performance in the FinNum task at NTCIR-14. The FinNum task is to classify the numerals in financial tweets into seven categories, and further extend these categories into seventeen subcategories. In our proposed model, we first analyze the obtained financial data from the FinNum task and enhance data for some subcategories by entity replacement. And then, we adopt our fine-tuning BERT model to finish the task. As a supplement, some popular traditional and deep learning models have been selected for comparative experiments, and the experimental results show that our model has achieved the state-of-the-art performances.


Financial numeral classification Financial data processing FinNum task BERT model 



The work presented in this paper is partially supported by the Major Projects of National Social Foundation of China under Grant No. 11&ZD189.


  1. 1.
    Dhar, V., Stein, R.M.: FinTech platforms and strategy. Commun. ACM 60(10), 32–35 (2017)CrossRefGoogle Scholar
  2. 2.
    Zhou, Y., Ni, B., Yan, S., Moulin, P., Tian, Q.: Pipelining localized semantic features for fine-grained action recognition. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 481–496. Springer, Cham (2014). Scholar
  3. 3.
    Karaoglu, S., et al.: Con-text: text detection for fine-grained object classification. IEEE Trans. Image Process. 26, 3965–3980 (2017)MathSciNetCrossRefGoogle Scholar
  4. 4.
    McCallum, A., Nigam, K.: A comparison of event models for Naive Bayes text classification. In: AI-98 Workshop on Learning for Text Categorization, vol. 752, no. 1, pp. 41–48 (1998)Google Scholar
  5. 5.
    Kim, Y.: Convolutional neural networks for sentence classification. Eprint Arxiv (2014)Google Scholar
  6. 6.
    Hüsken, M., Stagge, P.: Recurrent neural networks for time series classification. Neurocomputing 50, 223–235 (2003)CrossRefGoogle Scholar
  7. 7.
    Zhou, C., Sun, C., et al.: A C-LSTM neural network for text classification. Comput. Sci. 1(4), 39–44 (2015)Google Scholar
  8. 8.
    Devlin, J., Chang, M.W., Lee, K., et al.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  9. 9.
    Chen, C.C., Huang, H.H., Takamura, et al.: Overview of the NTCIR-14 FinNum task: fine-grained numeral understanding in financial social media data. In: Proceedings of the 14th NTCIR Conference on Evaluation of Information Access Technologies (2019)Google Scholar
  10. 10.
    Zhu, Y., Ryan, K., Richard, S., et al.: Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In: 2015 IEEE International Conference on Computer Vision, pp. 19–27 (2015)Google Scholar
  11. 11.
    Lee, et al.: BioBERT: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746 (2019)
  12. 12.
    Armand, J., Edouard, G., Piotr, B., Douze, M., et al.: compressing text classification models. arXiv preprint arXiv:1612.03651 (2016)
  13. 13.
    Moraes, R., Valiati, J.F., Neto, W.P.G.: Document-level sentiment classification: an empirical comparison between SVM and ANN. Expert Syst. Appl. 40(2), 621–633 (2013)CrossRefGoogle Scholar
  14. 14.
    Zhang, X., Zhao, J.B., et al.: Character-level convolutional networks for text classification. In: International Conference on Neural Information Processing Systems (2015)Google Scholar
  15. 15.
    Lai, S., Xu, L., Liu, K., et al.: Recurrent convolutional neural networks for text classification. In: Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)Google Scholar
  16. 16.
    Liu, Y., et al.: An attention-gated convolutional neural network for sentence classification. CoRR (2018)Google Scholar
  17. 17.
    Pappas, N., Popescu, B.A.: Multilingual hierarchical attention networks for document classification. arXiv preprint arXiv:1707.00896 (2017)
  18. 18.
    Zhang, Y., et al.: A text sentiment classification modeling method based on coordinated CNN-LSTM-attention model. Chin. J. Electron. 28(01), 124–130 (2019)Google Scholar
  19. 19.
    Liu, P.F., Qiu, X., Huang, X.: Adversarial multi-task learning for text classification. arXiv preprint arXiv:1704.05742 (2017)
  20. 20.
    Peters, M.E., et al.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018)
  21. 21.
    Alec, R., Karthik, N., Tim, S., et al.: Improving language understanding with unsupervised learning. Technical report. OpenAI (2018)Google Scholar
  22. 22.
    Chen, Q., Zhuo, Z., Wang, W.: BERT for joint intent classification and slot filling. arXiv preprint arXiv:1902.10909 (2019)
  23. 23.
    Schumaker, R.P., Chen, H.: Textual analysis of stock market prediction using breaking financial news. The AZFin text system (2009)Google Scholar
  24. 24.
    Suykens, J.A., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett. 9(3), 293–300 (1999)CrossRefGoogle Scholar
  25. 25.
    Vaswani, A., et al.: Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)Google Scholar
  26. 26.
    Yang, Y.: An evaluation of statistical approaches to text categorization. Inf. Retrieval 1(1–2), 69–90 (1999)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Wei Wang
    • 1
    • 2
  • Maofu Liu
    • 1
    • 2
    Email author
  • Yukun Zhang
    • 1
    • 2
  • Junyi Xiang
    • 1
    • 2
  • Ruibin Mao
    • 3
  1. 1.School of Computer Science and TechnologyWuhan University of Science and TechnologyWuhanChina
  2. 2.Hubei Province Key Laboratory of Intelligent Information Processing and Real-Time Industrial SystemWuhanChina
  3. 3.Center for Studies of Information ResourcesWuhan UniversityWuhanChina

Personalised recommendations