Artificial Intelligence Review

, Volume 49, Issue 3, pp 339–373 | Cite as

A survey on Urdu and Urdu like language stemmers and stemming techniques

  • Abdul Jabbar
  • Sajid Iqbal
  • Muhammad Usman Ghani Khan
  • Shafiq Hussain
Article
  • 172 Downloads

Abstract

Stemming is one of the basic steps in natural language processing applications such as information retrieval, parts of speech tagging, syntactic parsing and machine translation, etc. It is a morphological process that intends to convert the inflected forms of a word into its root form. Urdu is a morphologically rich language, emerged from different languages, that includes prefix, suffix, infix, co-suffix and circumfixes in inflected and multi-gram words that need to be edited in order to convert them into their stems. This editing (insertion, deletion and substitution) makes the stemming process difficult due to language morphological richness and inclusion of words of foreign languages like Persian and Arabic. In this paper, we present a comprehensive review of different algorithms and techniques of stemming Urdu text and also considering the syntax, morphological similarity and other common features and stemming approaches used in Urdu like languages, i.e. Arabic and Persian analyzed, extract main features, merits and shortcomings of the used stemming approaches. In this paper, we also discuss stemming errors, basic difference between stemming and lemmatization and coin a metric for classification of stemming algorithms. In the final phase, we have presented the future work directions.

Keywords

Stemming Natural Language Processing Information Retrieval Urdu Suffixes Stemming Techniques 

References

  1. Ababneh M, Al-Shalabi R, Kanaan G, Al-Nobani A (2012) Building an effective rule-based light stemmer for Arabic language to improve search effectiveness. Int Arab J Inf Technol (IAJIT) 9(4):368–372Google Scholar
  2. Abbas Q (2012) Building a hierarchical annotated corpus of urdu: the URDU. KON-TB treebank. In: International conference on intelligent text processing and computational linguistics. Berlin, pp 66–79Google Scholar
  3. Abu-Errub A, Odeh A, Shambour Q, Hassan OA-H (2014) Arabic roots extraction using morphological analysis. Int J Comput Sci 11:2Google Scholar
  4. Akram QA, Naseer A, Hussain S (2009) Assas-Band, an affix-exception-list based Urdu stemmer. In: Proceedings of the 7th workshop on Asian language resources. Association for Computational Linguistics, pp 40–46Google Scholar
  5. Aljlayl M, Frieder O (2002) On Arabic search: improving the retrieval effectiveness via a light stemming approach. In: Proceedings of the eleventh international conference on Information and knowledge management. ACM, pp 340–347Google Scholar
  6. Al-Kabi M, Al-Mustafa R (2006) Arabic root based stemmer. In: Proceedings of the international Arab conference on information technologyGoogle Scholar
  7. Al-Kabi M, Al-Shawakfa E, Alsmadi I (2013) The effect of stemming on Arabic text classification: an empirical study. Inf Retr Methods Multidiscip Appl 207–225Google Scholar
  8. Al-Kabi MN, Kazakzeh SA, Ata BMA, Al-Rababah SA, Alsmadi IM (2015) A novel root based Arabic stemmer. J King Saud Univ Comput Inf Sci 27(2):94–103Google Scholar
  9. Al-Omari A, Abuata B (2014) Arabic light stemmer (ARS). J Eng Sci Technol 9(6):702–717Google Scholar
  10. Al-Shammari ET (2013) Lemmatizing, stemming, and query expansion method and system. U.S. Patent No. 8,473,279. 25 Jun 2013Google Scholar
  11. Al-Shammari ET, Lin J (2008) Towards an error-free Arabic stemming. In: Proceedings of the 2nd ACM workshop on improving non English web searching. ACM, pp 9–16Google Scholar
  12. Balakrishnan V, Lloyd-Yemoh E (2014) Stemming and lemmatization: a comparison of retrieval performances. Lect Notes Softw Eng 2(3):262–267CrossRefGoogle Scholar
  13. Cambria E, White B (2014) Jumping NLP curves: a review of natural language processing research. IEEE Comput Intell Mag 9(2):48–57CrossRefGoogle Scholar
  14. Carpineto C, Romano G (2012) A survey of automatic query expansion in information retrieval. ACM Comput Surv (CSUR) 44(1):1CrossRefMATHGoogle Scholar
  15. Chen A, Gey FC (2002) Building an Arabic stemmer for information retrieval. In: TREC, pp 631–639Google Scholar
  16. Chris DP (1990) Another stemmer. ACM. SIGIR Forum 24(3):56–61CrossRefGoogle Scholar
  17. Dahab MY, Al-Mutawa R (2015) A comparative study on Arabic stemmers. Change 125(8):Google Scholar
  18. Dianati MH, Hadi SM, Rasekh AH, Fakhrahmad SM, Taghi-Zadeh H (2014) Words stemming based on structural and semantic similarity. Comput Eng Appl J 3(2):89–99Google Scholar
  19. Ebrahim S, Hegazy D, Mostafa MG, El-Beltagy SR (2015) English–Arabic statistical machine translation: state of the art. In: International conference on intelligent text processing and computational linguistics. Springer International Publishing, pp 520–533Google Scholar
  20. El-Beltagy Samhaa R, Rafea Ahmed (2011) An accuracy-enhanced light stemmer for arabic text. ACM Trans Speech Lang Process (TSLP) 7(2):2Google Scholar
  21. El-Defrawy M, El-Sonbaty Y, Belal NA (2015) Cbas: context based arabic stemmer. Int J Nat Lang Comput (IJNLC) 4(3):1–12CrossRefGoogle Scholar
  22. El Kholy A et al (2013) Selective combination of pivot and direct statistical machine translation models. In: Proceedings of the 6th international joint conference on natural language processingGoogle Scholar
  23. Estahbanati A, Javidan R, Dezfooli MA (2011) Implementation of a new method for stemming in Persian language. In: Proceedings of the international conference on web intelligence, mining and semantics. ACM, p 63Google Scholar
  24. Frakes WB (1992) Information retrieval: data structures and algorithms, Chapter 8. http://orion.lcg.ufrj.br/Dr.Dobbs/books/book5/chap08.htm. Retrieved 1 Oct 2015
  25. Ghwanmeh S, Kanaan G, Al-Shalabi R, Rabab’ah S (2009) Enhanced algorithm for extracting the root of Arabic words. In: Sixth international conference on computer graphics, imaging and visualization, 2009. CGIV’09. IEEE, pp 388–391Google Scholar
  26. Goweder A, Alhami H, Rashed T, Al-Musrati A (2008) A hybrid method for stemming Arabic text. J Comput Sci. http://eref.uqu.edu.sa/files/eref2/folder6/f181.pdf
  27. Gupta V, Joshi N, Mathur I (2013) Rule based stemmer in Urdu. In: 2013 4th international conference on computer and communication technology (ICCCT). IEEE, pp 129–132Google Scholar
  28. Gupta V, Joshi N, Mathur I (2015) Design and development of rule based inflectional and derivational Urdu stemmer ‘Usal’. In: 2015 international conference on futuristic trends on computational analysis and knowledge management (ABLAZE). IEEE, pp 7–12Google Scholar
  29. Habash N (2007) Arabic morphological representations for machine translation. Arabic computational morphology. Springer, Netherlands, pp 263–285CrossRefGoogle Scholar
  30. Hadni M, Lachkar A, Alaoui OS (2012) A new and efficient stemming technique for Arabic Text Categorization. In: 2012 international conference on multimedia computing and systems (ICMCS). IEEEGoogle Scholar
  31. Hadni M, Ouatik SA, Lachkar A (2013) Effective Arabic stemmer based hybrid approach for Arabic text categorization. Int J Data Min Knowl Manag Process (IJDKP) 3(4):1–14CrossRefGoogle Scholar
  32. Husain MS, Ahamad F, Khalid S (2013) A language independent approach to develop Urdu stemmer. Advances in computing and information technology. Springer, Berlin, pp 45–53CrossRefGoogle Scholar
  33. Hussain S (2008) Resources for Urdu language processing. In: IJCNLP, pp 99–100Google Scholar
  34. Hussain S, Afzal M, (2001) Urdu computing standards: Urdu zabta takhti (uzt) 1.01. In: Multi topic conference, (2001) IEEE INMIC 2001, Technology for the 21st century. Proceedings, IEEE International, IEEEGoogle Scholar
  35. Khan S, Anwar W, Bajwa U, Wang X (2015) Template based affix stemmer for a morphologically rich language. Int Arab J Inf Technol 12(2):146–154Google Scholar
  36. Khan SA, Anwar W, Ijaz BU, Wang X (2012) A light weight stemmer for Urdu language: a scarce resourced language. In: 24th international conference on computational linguistics, p 69Google Scholar
  37. Khansir AA, Mozafari N (2014) The impact of Persian language on Indian languages. Theory Pract Lang Stud 4(11):2360–2365Google Scholar
  38. Khoja S, Garside R (1999) Stemming Arabic text 1999. http://zeus.cs.pacificu.edu/shereen/research.htm#stemming. Accessed 27 Dec 2015
  39. Korenius T et al (2004) Stemming and lemmatization in the clustering of finnish text documents. In: Proceedings of the thirteenth ACM international conference on Information and knowledge management. ACMGoogle Scholar
  40. Lakshmi RV, Kumar SBR (2014) Literature review: stemming algorithms for Indian and Non-Indian languages. Int J Adv Res Comput Sci Technol 2(3):349–352Google Scholar
  41. Larkey LS, Ballesteros L, Connell ME (2002) Improving stemming for Arabic information retrieval: light stemming and co-occurrence analysis. In: Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 275–282Google Scholar
  42. Lehal RKVGGS (2012) Rule based Urdu stemmer. In: 24th international conference on computational linguistics, p 267Google Scholar
  43. Lovins JB (1968) Development of a stemming algorithm. Electronic Systems Laboratory, MIT Information Processing Group, CambridgeGoogle Scholar
  44. Madnani N, Tetreault J, Chodorow M (2012) Re-examining machine translation metrics for paraphrase identification. In: Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: human language technologies. Association for Computational LinguisticsGoogle Scholar
  45. Mahmoodi M, Varnamkhasti MM (2014) Design a Persian automated plagiarism detector (AMZPPD). arXiv preprint arXiv:1403.1618
  46. Majumder P, Mandar M, Swapan KP, Kole G, Mitra P, Datta K (2007) YASS: yet another suffix stripper. ACM Trans Inf Syst (TOIS) 25(4):18CrossRefGoogle Scholar
  47. Melucci M, Orio N (2003) A novel method for stemmer generation based on hidden Markov models. In: Proceedings of the twelfth international conference on information and knowledge management. ACM, pp 131–138Google Scholar
  48. Moghadam FM, Keyvanpour M (2015) Comparative study of various Persian stemmers in the field of information retrieval. J Inf Process Syst 11(3):450–464Google Scholar
  49. Mokhtaripour A, Jahanpour S (2006) Introduction to a new Farsi stemmer. In: Proceedings of the 15th ACM international conference on information and knowledge management. ACM, pp 826–827Google Scholar
  50. Mubashir Ali SK, Saleemi MH (2014) A novel stemming approach for Urdu language. J Appl Environ Biol Sci 4(7S)436–443. ISSN: 2090–4274. www.textroad.com
  51. Nwesri AFA, Tahaghoghi SMM, Scholer F (2005) Stemming Arabic conjunctions and prepositions. International symposium on string processing and information retrieval. Springer, Berlin, pp 206–217CrossRefGoogle Scholar
  52. Paice CD (1994) An evaluation method for stemming algorithms. Proceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval. Springer, New York, pp 42–50Google Scholar
  53. Piotrowski M (2012) Natural language processing for historical texts. Synth Lect Hum Lang Technol 5(2):1–157CrossRefGoogle Scholar
  54. Porter MF (1980) An algorithm for suffix stripping. Program 14(3):130–137Google Scholar
  55. Rahimi A (2015) A new hybrid stemming algorithm for Persian. arXiv preprint arXiv:1507.03077
  56. Rahimtoroghi E, Faili H, Shakery A (2010) A structural rule-based stemmer for Persian. In: 2010 5th international symposium on telecommunications (IST). IEEE, pp 574–578Google Scholar
  57. Rashidi A, Lighvan MZ (2014) HPS: a hierarchical Persian stemming method. arXiv preprint arXiv:1403.2837
  58. Sarabi Z, Hamidreza M, Mojgan F (2013) Parsi Pardaz: Persian Language Processing Toolkit. In: 2013 3rd international conference on computer and knowledge engineering (ICCKE). IEEEGoogle Scholar
  59. Saraee M, Bagheri A (2013) Feature selection methods in Persian sentiment analysis. International conference on application of natural language to information systems. Springer, Berlin, pp 303–308Google Scholar
  60. Seo Y-W, Ankolekar A, Sycara K (2004) Feature selection for extracting semantically rich words. No. CMU-RI-TR-04–18. Robotics Inst., Carnegie-Mellon Univ., PittsburghGoogle Scholar
  61. Sharifloo AA, Shamsfard M (2008) A bottom up approach to Persian stemming. In: IJCNLP, pp 583–588Google Scholar
  62. Sirsat SR, Chavan V, Mahalle HS (2013) Strength and accuracy analysis of affix removal stemming algorithms. Int J Comput Sci Inf Technol 4(2):265–269Google Scholar
  63. Taghi-Zadeh H, Hadi SM, Diyanati MH, Rasekh AH (2015) A new hybrid stemming method for Persian language. Digital Scholarship in the Humanities: fqv053Google Scholar
  64. Taghva K, Beckley R, Sadeh M (2005a) A stemming algorithm for the farsi language. In: Null. IEEE, pp 158–162Google Scholar
  65. Taghva K, Elkhoury R, Coombs J (2005b) Arabic stemming without a root dictionary. In: Innull. IEEE, pp 152–157Google Scholar
  66. Tahir N (2014) Impact of Arabic language on Urdu language. VFAST Trans Islam Res 5(1):1–13MathSciNetGoogle Scholar
  67. Tashakori M, Meybodi M, Oroumchian F (2002) Bon: the Persian stemmer. EurAsia-ICT 2002: information and communication technology. Springer, Berlin, pp 487–494CrossRefGoogle Scholar
  68. Zughoul M, Abu-Alshaar A (2005) English/Arabic/English machine translation: a historical perspective. Transl J 50(3):1022–1041Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  1. 1.Department of Computer ScienceInstitute of Southern PunjabMultanPakistan
  2. 2.Department of Computer ScienceBahauddin Zakariya UniversityMultanPakistan
  3. 3.Al-Khwarzmi Institute of Computer ScienceUniversity of Engineering and TechnologyLahorePakistan
  4. 4.Department of Computer ScienceBahauddin Zakariya University (Sahiwal Sub-campus)MultanPakistan

Personalised recommendations