Skip to main content
SpringerLink
Log in

Dataset for Evaluation of Mathematical Reasoning Abilities in Russian

  • Conference paper
  • First Online:
Artificial Intelligence and Natural Language (AINL 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1292))

Included in the following conference series:

  • 404 Accesses

Abstract

We present a Russian version of DeepMind Mathematics Dataset. The original dataset is synthetically generated using inference rules and a set of linguistic templates. We translate the linguistic templates to Russian leaving the inference part without changes. So as a result we get a mathematically parallel dataset where the same mathematical problems are explored but in another language. We reproduce the experiment from the original paper to check whether the performance of a Transformer model is impacted by the differences of the languages in which math problems are expressed. Though our contribution is small compared to the original work, we think it is valuable given the fact that languages other than English (and Russian in particular) are underrepresented.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://ai2-website.s3.amazonaws.com/data/arithmeticquestions.pdf.

  2. 2.

    https://github.com/mannefedov/mathematics_dataset_russian.

References

  1. Bender, E.M.: On achieving and evaluating language-independence in NLP. Linguist. Issues Lang. Technol. 6, 1–26 (2011)

    Google Scholar 

  2. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., Gardner, M.: Drop: a reading comprehension benchmark requiring discrete reasoning over paragraphs. In: NAACL-HLT (2019)

    Google Scholar 

  3. Geva, M., Gupta, A., Berant, J.: Injecting numerical reasoning skills into language models. In: ACL (2020)

    Google Scholar 

  4. Hopkins, M., Bras, R.L., Petrescu-Prahova, C., Stanovsky, G., Hajishirzi, H., Koncel-Kedziorski, R.: Semeval-2019 task 10: math question answering. In: SemEval@NAACL-HLT (2019)

    Google Scholar 

  5. Koncel-Kedziorski, R., Hajishirzi, H., Sabharwal, A., Etzioni, O., Ang, S.D.: Parsing algebraic word problems into equations. Trans. Assoc. Comput. Linguist. 3, 585–597 (2015)

    Article  Google Scholar 

  6. Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., Hajishirzi, H.: Mawps: a math word problem repository. In: HLT-NAACL (2016)

    Google Scholar 

  7. Mielke, S.J.: Language diversity in ACL 2004–2016 (December 2016). https://sjmielke.com/acl-language-diversity.htm

  8. Munro, R.: Languages at ACL 2015 (July 2015). http://www.junglelightspeed.com/languages-at-acl-this-year/

  9. Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: EMNLP (2016)

    Google Scholar 

  10. Ravichander, A., Naik, A., Rosé, C.P., Hovy, E.H.: EQUATE: a benchmark evaluation framework for quantitative reasoning in natural language inference. In: CoNLL (2019)

    Google Scholar 

  11. Roy, S., Roth, D.: Solving general arithmetic word problems. In: EMNLP (2015)

    Google Scholar 

  12. Roy, S., Vieira, T., Roth, D.: Reasoning about quantities in natural language. Trans. Assoc. Comput. Linguist. 3, 1–13 (2015)

    Article  Google Scholar 

  13. Saxton, D., Grefenstette, E., Hill, F., Kohli, P.: Analysing mathematical reasoning abilities of neural models. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=H1gR5iR5FX

  14. Siyam, B., Saa, A.A., Alqaryouti, O., Shaalan, K.: Arabic arithmetic word problems solver. In: ACLING (2017)

    Google Scholar 

  15. Trask, A., Hill, F., Reed, S.E., Rae, J.W., Dyer, C., Blunsom, P.: Neural arithmetic logic units. In: NeurIPS (2018)

    Google Scholar 

  16. Vaswani, A., et al.: Attention is all you need. In: NIPS (2017)

    Google Scholar 

  17. Wallace, E., Wang, Y., Li, S., Singh, S., Gardner, M.: Do NLP models know numbers? Probing numeracy in embeddings. In: EMNLP/IJCNLP (2019)

    Google Scholar 

  18. Wang, Y., Liu, X., Shi, S.: Deep neural solver for math word problems. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 845–854. Association for Computational Linguistics, Copenhagen, Denmark (September 2017). https://doi.org/10.18653/v1/D17-1088, https://www.aclweb.org/anthology/D17-1088

Download references

Author information

Authors and Affiliations

  1. Higher School of Economics, Moscow, Russia

    Mikhail Nefedov

Authors
  1. Mikhail Nefedov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mikhail Nefedov .

Editor information

Editors and Affiliations

  1. ITMO University, St. Petersburg, Russia

    Andrey Filchenkov

  2. Haaga-Helia University of Applied Sciences, Helsinki, Finland

    Janne Kauttonen

  3. University of Helsinki, Helsinki, Finland

    Lidia Pivovarova

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nefedov, M. (2020). Dataset for Evaluation of Mathematical Reasoning Abilities in Russian. In: Filchenkov, A., Kauttonen, J., Pivovarova, L. (eds) Artificial Intelligence and Natural Language. AINL 2020. Communications in Computer and Information Science, vol 1292. Springer, Cham. https://doi.org/10.1007/978-3-030-59082-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59082-6_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59081-9

  • Online ISBN: 978-3-030-59082-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation