Skip to main content
SpringerLink
Log in

Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solving

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Math Word Problem (MWP) solving presents a challenging task in Natural Language Processing (NLP). This study aims to provide MWP solvers with a more diverse training set, ultimately improving their ability to solve various math problems. We propose several methods for data augmentation by modifying the problem texts and equations, such as synonym replacement, rule-based: question replacement, and rule based: reversing question methodologies over two English MWP datasets. This study extends by introducing a new in-context learning augmentation method, employing the Llama-7b language model. This approach involves instruction-based prompting for rephrasing the math problem texts. Performance evaluations are conducted on 9 baseline models, revealing that augmentation methods outperform baseline models. Moreover, concatenating examples generated by various augmentation methods further improves performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Yigit, G., Amasyali, M.F.: Enhancing multiple-choice question answering through sequential fine-tuning and curriculum learning strategies. Knowledge and Information Systems, 1–18 (2023)

  2. Wu, L., Wu, P., Zhang, X.: A seq2seq-based approach to question answering over knowledge bases. In: Semantic Technology: 9th Joint International Conference, JIST 2019, Hangzhou, China, November 25–27, 2019, Revised Selected Papers 9, pp. 170–181 (2020). Springer

  3. Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., Auli, M.: Eli5: Long form question answering. arXiv preprint arXiv:1907.09190 (2019)

  4. Jin, S., Lian, X., Jung, H., Park, J., Suh, J.: Building a deep learning-based qa system from a cqa dataset. Decision Support Systems, 114038 (2023)

  5. Abdel-Nabi H, Awajan A, Ali MZ. Deep learning-based question answering: a survey. Knowl Inf Syst. 2023;65(4):1399–485.

    Article  Google Scholar 

  6. Rogers A, Gardner M, Augenstein I. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. ACM Comput Surv. 2023;55(10):1–45.

    Article  Google Scholar 

  7. Yigit, G., Amasyali, M.F.: Ask me: A question answering system via dynamic memory networks. In: 2019 Innovations in Intelligent Systems and Applications Conference (ASYU), pp. 1–5 (2019). IEEE

  8. Xie, Z., Sun, S.: A goal-driven tree-structured neural model for math word problems. In: IJCAI, pp. 5299–5305 (2019)

  9. Zhang, J., Lee, R.K.-W., Lim, E.-P., Qin, W., Wang, L., Shao, J., Sun, Q.: Teacher-student networks with multiple decoders for solving math word problem. (2020). IJCAI

  10. Liang, Z., Zhang, J., Wang, L., Qin, W., Lan, Y., Shao, J., Zhang, X.: Mwp-bert: Numeracy-augmented pre-training for math word problem solving. In: Findings of the Association for Computational Linguistics: NAACL 2022, pp. 997–1009 (2022)

  11. Wang, L., Zhang, D., Zhang, J., Xu, X., Gao, L., Dai, B.T., Shen, H.T.: Template-based math word problem solvers with recursive neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 7144–7151 (2019)

  12. Zhang, J., Wang, L., Lee, R.K.-W., Bin, Y., Wang, Y., Shao, J., Lim, E.-P.: Graph-to-tree learning for solving math word problems. (2020). Association for Computational Linguistics

  13. Shen, Y., Jin, C.: Solving math word problems with multi-encoders and multi-decoders. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 2924–2934 (2020)

  14. Upadhyay, S., Chang, M.-W.: Annotating derivations: A new evaluation strategy and dataset for algebra word problems. arXiv preprint arXiv:1609.07197 (2016)

  15. Qin, J., Lin, L., Liang, X., Zhang, R., Lin, L.: Semantically-aligned universal tree-structured solver for math word problems. arXiv preprint arXiv:2010.06823 (2020)

  16. Kushman, N., Artzi, Y., Zettlemoyer, L., Barzilay, R.: Learning to automatically solve algebra word problems. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 271–281 (2014)

  17. Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., Hajishirzi, H.: Mawps: A math word problem repository. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1152–1157 (2016)

  18. Patel, A., Bhattamishra, S., Goyal, N.: Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 (2021)

  19. Miao, S.-Y., Liang, C.-C., Su, K.-Y.: A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15772 (2021)

  20. Wang, Y., Liu, X., Shi, S.: Deep neural solver for math word problems. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 845–854 (2017)

  21. Roy S, Roth D. Mapping to declarative knowledge for word problem solving. Transactions of the Association for Computational Linguistics. 2018;6:159–72.

    Article  Google Scholar 

  22. Mitra, A., Baral, C.: Learning to use formulas to solve simple arithmetic problems. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2144–2153 (2016)

  23. Fletcher CR. Understanding and solving arithmetic word problems: A computer simulation. Behavior Research Methods, Instruments, & Computers. 1985;17(5):565–71.

    Article  Google Scholar 

  24. Bakman, Y.: Robust understanding of word problems with extraneous information. arXiv preprint math/0701393 (2007)

  25. Yuhui, M., Ying, Z., Guangzuo, C., Yun, R., Ronghuai, H.: Frame-based calculus of solving arithmetic multi-step addition and subtraction word problems. In: 2010 Second International Workshop on Education Technology and Computer Science, vol. 2, pp. 476–479 (2010). IEEE

  26. Wang, L., Wang, Y., Cai, D., Zhang, D., Liu, X.: Translating a math word problem to an expression tree. arXiv preprint arXiv:1811.05632 (2018)

  27. Yigit, G., Amasyali, M.F.: Exploring the benefits of data augmentation in math word problem solving. In: 2023 International Conference on Innovations in Intelligent Systems and Applications (INISTA), pp. 1–6 (2023). IEEE

  28. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

  29. Lan, Y., Wang, L., Zhang, Q., Lan, Y., Dai, B.T., Wang, Y., Zhang, D., Lim, E.-P.: Mwptoolkit: an open-source framework for deep learning-based math word problem solvers. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 13188–13190 (2022)

  30. Hosseini, M.J., Hajishirzi, H., Etzioni, O., Kushman, N.: Learning to solve arithmetic word problems with verb categorization. In: EMNLP, pp. 523–533 (2014)

  31. Zhou, L., Dai, S., Chen, L.: Learn to solve algebra word problems using quadratic programming. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 817–822 (2015)

  32. Koncel-Kedziorski R, Hajishirzi H, Sabharwal A, Etzioni O, Ang SD. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics. 2015;3:585–97.

    Article  Google Scholar 

  33. Huang, D., Shi, S., Lin, C.-Y., Yin, J.: Learning fine-grained expressions to solve math word problems. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 805–814 (2017)

  34. Zhang, B., Xiong, D., Su, J., Duan, H., Zhang, M.: Variational neural machine translation. arXiv preprint arXiv:1605.07869 (2016)

  35. Huang, D., Liu, J., Lin, C.-Y., Yin, J.: Neural math word problem solver with reinforcement learning. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 213–223 (2018)

  36. Chiang, T.-R., Chen, Y.-N.: Semantically-aligned equation generation for solving and reasoning math word problems. arXiv preprint arXiv:1811.00720 (2018)

  37. Li, J., Wang, L., Zhang, J., Wang, Y., Dai, B.T., Zhang, D.: Modeling intra-relation in math word problems with different functional multi-head attentions. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6162–6167 (2019)

  38. Meng, Y., Rumshisky, A.: Solving math word problems with double-decoder transformer. arXiv preprint arXiv:1908.10924 (2019)

  39. Li, S., Wu, L., Feng, S., Xu, F., Xu, F., Zhong, S.: Graph-to-tree neural networks for learning structured input-output translation with applications to semantic parsing and math word problem. arXiv preprint arXiv:2004.13781 (2020)

  40. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  41. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach (2019)

  42. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI blog. 2019;1(8):9.

    Google Scholar 

  43. Shao, Z., Huang, F., Huang, M.: Chaining simultaneous thoughts for numerical reasoning. arXiv preprint arXiv:2211.16482 (2022)

  44. Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., Chen, W.: On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336 (2022)

  45. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

  46. Pi, X., Liu, Q., Chen, B., Ziyadi, M., Lin, Z., Fu, Q., Gao, Y., Lou, J.-G., Chen, W.: Reasoning like program executors. arXiv preprint arXiv:2201.11473 (2022)

  47. Chen, W., Ma, X., Wang, X., Cohen, W.W.: Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 (2022)

  48. Liang, Z., Yu, W., Rajpurohit, T., Clark, P., Zhang, X., Kaylan, A.: Let gpt be a math tutor: Teaching math word problem solvers with customized exercise generation. arXiv preprint arXiv:2305.14386 (2023)

  49. Lazaridou, A., Gribovskaya, E., Stokowiec, W., Grigorev, N.: Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115 (2022)

  50. Wei J, Wang X, Schuurmans D, Bosma M, Xia F, Chi E, Le QV, Zhou D, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst. 2022;35:24824–37.

    Google Scholar 

  51. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

  52. Liu Q, Guan W, Li S, Cheng F, Kawahara D, Kurohashi S. Roda: reverse operation based data augmentation for solving math word problems. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2021;30:1–11.

    Article  Google Scholar 

  53. Raiyan, S.R., Faiyaz, M.N., Kabir, S.M.J., Kabir, M., Mahmud, H., Hasan, M.K.: Math word problem solving by generating linguistic variants of problem statements. arXiv preprint arXiv:2306.13899 (2023)

  54. Liang, Z., Zhang, J., Wang, L., Qin, W., Lan, Y., Shao, J., Zhang, X.: Mwp-bert: Numeracy-augmented pre-training for math word problem solving. arXiv preprint arXiv:2107.13435 (2021)

Download references

Acknowledgements

This research is supported by The Scientific and Technological Research Council of Turkey (TÜBİTAK) in part of the project with 120E100 grant number. G. Yigit is supported by TUBİTAK - BÍDEB 2211/A national fellowship program for Ph.D. studies.

Author information

Author notes

  1. Gulsum Yigit and Mehmet Fatih Amasyali have contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Engineering, Yildiz Technical University, Istanbul, Turkey

    Gulsum Yigit & Mehmet Fatih Amasyali

  2. Department of Computer Engineering, Kadir Has University, Istanbul, Turkey

    Gulsum Yigit

Authors
  1. Gulsum Yigit
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehmet Fatih Amasyali
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gulsum Yigit.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Digital Ecosystems” guest edited by Richard Chbeir and Yannis Manolopoulos.

Appendix

Appendix

See Tables 8, 9.

Table 8 Prompts employed to generate rephrased versions of problem texts within the MAWPs-Single dataset
Full size table
Table 9 Prompts employed to generate rephrased versions of problem texts within the SVAMP dataset
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yigit, G., Amasyali, M.F. Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solving. SN COMPUT. SCI. 5, 506 (2024). https://doi.org/10.1007/s42979-024-02853-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-024-02853-x

Keywords

Search

Navigation