Abstract
This work aims to develop a Natural Language Processing (NLP) tool for benchmarking and assessing Arabic syntactic parsers. This tool is integrated within the Software Architecture For Arabic language pRocessing (SAFAR). Indeed, SAFAR contains several ANLP tools from simple preprocessing up to the semantic level. The benchmarking tool will take advantage of the available basic tools in addition to the flexibility and reusability of SAFAR. The benchmark process takes as input an evaluation corpus and one/several syntactic parsers implementations. As a result, it outputs the most common metrics used for evaluation namely: precision, recall, accuracy and F-measure. We introduced also a new metric called Gp-score which takes into account the execution time besides the accuracy. The execution time is very crucial for some tasks such as real-time automatic translators or in the context of processing huge data. This benchmarking solution will help researchers in comparing their parsers against each other; it will help as well other researchers in selecting the appropriate parser to use within their high level projects. Two Arabic syntactic parsers are evaluated to give a concrete example of this tool: The Stanford parser and the ATKS parser.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
References
Miniwatts Marketing Group (2001). http://www.internetworldstats.com
Hattab, M., Haddad, B., Yaseen, M., Duraidi, A., Shmais, A.A.: Addaall Arabic search engine: improving search based on combination of morphological analysis and generation considering semantic patterns. In: The 2nd International Conference on Arabic Language Resources & Tools (2009)
Ittycheriah, A., Roukos, S.: A maximum entropy word aligner for Arabic-English machine translation. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing (2005)
Jaafar, Y., Namly, D., Bouzoubaa, K., Yousfi, A.: Enhancing Arabic stemming process using resources and benchmarking tools. J. King Saud Univ. Comput. Inf. Sci. 29(2), 164–170 (2016)
Jaafar, Y., Bouzoubaa, K.: Benchmark of Arabic morphological analyzers: challenges and solutions. In: 9th International Conference on Intelligent Systems: Theories and Applications (SITA 2014), Rabat, Morocco (2014)
Jaafar, Y., Bouzoubaa, K., Yousfi, A., Tajmout, R., Khamar, H.: Improving Arabic morphological analyzers benchmark. Int. J. Speech Technol. 19(2), 259–267 (2016)
Green, S., Manning, C.D.: Better Arabic parsing: baselines, evaluations, and analysis. In: The 23rd International Conference on Computational Linguistics (COLING 2010), Beijing (2010)
Microsoft: Arabic Toolkit Service (ATKS). https://www.microsoft.com/en-us/research/project/arabic-toolkit-service-atks/. Accessed 01 Mar 2017
Pawlik, M., Augsten, N.: Tree edit distance. Inf. Syst. 56(C), 157–173 (2016)
Pawlik, M., Augsten, N.: RTED: a robust algorithm for the tree edit distance. In: Proceedings of the VLDB Endowment (2011)
Atwell, E.: Comparative evaluation of grammatical annotation models. Ind. Parsing Software Manuals 17, 25–46 (1996)
Tsarfaty, R., Nivre, J., Andersson, E.: Cross-framework evaluation for statistical parsing. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (2012)
Black, E., Abney, S., Flickenger, D., Gdaniec, C., Grishman, R., Harrison, P., Hindle, D., Ingria, R., Jelinek, F., Klavans, J., Liberman, M., Marcus, M., Roukos, S., Santorini, B., Strzalkowski, T.: A procedure for quantitatively comparing the syntactic coverage of English grammars. In: Proceedings of the Fourth DARPA Speech and Natural Language Workshop, Pacific Grove (1991)
Harrison, P., Abney, S., Black, E., Flickinger, D., Gdaniec, C., Grishman, R., Hindle, D., Ingria, R., Marcus, M., Santorini, B., Strzalkowski, T.: Evaluating syntax performance of parser/grammars. In: Proceedings of the Natural Language Processing Systems Evaluation Workshop, Berekely (1991)
Lin, D.: A dependency-based method for evaluating broad-coverage parsers. Nat. Lang. Eng. 4(02), 97–114 (1998)
Kummerfeld, J.K., Hall, D., Curran, J.R., Klein, D.: Parser showdown at the wall street corral: an empirical investigation of error types in parser output. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Jeju Island (2012)
Seddah, D., Candito, M., Crabbé, B.: Cross parser evaluation and tagset variation: a French Treebank study. In: Proceedings of the 11th International Conference on Parsing Technologies (2009)
Hall, D., Berg-Kirkpatrick, T., Klein, D.: Sparser, Better, Faster GPU Parsing. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore (2014)
Bikel, D.: Design of a multi-lingual, parallel-processing statistical parsing engine. In: Proceedings of the Second International Conference on Human Language Technology Research (2002)
Charniak, E.: A maximum-entropy-inspired parser. In: Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference (2000)
Collins, M.J.: A new statistical parser based on bigram lexical dependencies. In: Proceedings of the 34th Annual Meeting on Association for Computational Linguistics (1996)
Maamouri, M., Bies, A., Buckwalter, T., Mekki, W.: The Penn Arabic Treebank: building a large-scale annotated Arabic corpus. In: NEMLAR Conference on Arabic Language Resources and Tools, vol. 27, pp. 466–467 (2004)
Smrž, O., Bielický, V., Jakub, I.K.: Prague Arabic dependency treebank: a word on the million words. In: Proceedings of the Workshop on Arabic and Local Languages (LREC 2008), Marrakech (2008)
Habash, N., Roth, R.M.: CATiB: The Columbia Arabic Treebank. In: Proceedings of the ACL-IJCNLP 2009 Conference Short Papers (2009)
Dukes, K., Buckwalter, T.: A dependency treebank of the Quran using traditional Arabic grammar. In: 2010 The 7th International Conference on Informatics and Systems (INFOS), Cairo (2010)
Pradhan, S.S., Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., Weischedel, R.: Ontonotes: a unified relational semantic representation. Int. J. Semant. Comput. 1(04), 405–419 (2007)
Bies, A.: catalog.ldc.upenn.edu, 28.1.2003. https://catalog.ldc.upenn.edu/docs/LDC2003T06/arabic-POStags-collapse-to-PennPOStags.txt. Accessed 6 June 2017
Jaafar, Y., Bouzoubaa, K.: SAFAR: Software Architecture For Arabic language pRocessing. http://arabic.emi.ac.ma/safar/. Accessed 6 June 2017
Souteh, Y., Bouzoubaa, K.: SAFAR platform and its morphological layer. In: Eleventh Conference on Language Engineering, ESOLEC 2011, Cairo (2011)
Jaafar, Y., Bouzoubaa, K.: Arabic natural language processing from software engineering to complex pipeline. In: First International Conference on Arabic Computational Linguistics (ACLing), Egypt, Cairo (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Jaafar, Y., Bouzoubaa, K. (2018). A New Tool for Benchmarking and Assessing Arabic Syntactic Parsers. In: Lachkar, A., Bouzoubaa, K., Mazroui, A., Hamdani, A., Lekhouaja, A. (eds) Arabic Language Processing: From Theory to Practice. ICALP 2017. Communications in Computer and Information Science, vol 782. Springer, Cham. https://doi.org/10.1007/978-3-319-73500-9_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-73500-9_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-73499-6
Online ISBN: 978-3-319-73500-9
eBook Packages: Computer ScienceComputer Science (R0)