Abstract
Tokenization is the mechanism of splitting or fragmenting the sentences and words to its possible smallest morpheme called as token. Morpheme is smallest possible word after which it cannot be broken further. As the tokenization is initial phase and as well very crucial phase of Part-Of-Speech (POS) tagging in Natural Language Processing (NLP). Tokenization could be sentence level and word level. This paper analyzes the possible tokenization methods that can be applied to tokenize the word efficiently.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Accessed on 6 Nov 2019. https://www.tutorialspoint.com/lucene/lucene_analysis.htm
Sennrich, R., et al.: Carried research on “Neural Machine Translation of Rare Words with Subword Units”. arXiv: 1508.07909v5 [cs, CL] (10 June 2016)
Accessed on 6 Sept 2019. https://openai.com/blog/better-language-models/
Kudo, T.: Subword regularization: improving neural network translation models with multiple subword candidates. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), pp. 66–75. Melbourne, Australia (July 15–20, 2018)
Guo, J., et al.: Critical Tokenization and its properties. Comput. Linguist. 23 (1997)
Prathibha, R.J., Padma, M.C.: Kannada text normalization in source analysis phase of machine translation system. Int. J. Eng. Technol. (IJET) 9(3S) (2017). ISSN (Print): 2319–8613 ISSN (Online): 0975-4024. https://doi.org/10.21817/ijet/2017/v9i3/170903s088
Vijayarani, S., Janani, R.: Text mining: open source tokenization tools—an analysis. Adv. Comput. Intell. Int. J. (ACII), 3(1) (2016)
Joseph, J., Jeba, J.R.: Information extraction using tokenization and clustering methods. Int. J. Recent Technol. Eng. (IJRTE) 8(4) (2019). ISSN: 2277–3878
Bulusu, A., Sucharita, V.: Research on machine learning techniques for POS tagging in NLP. Int. J. Recent Technol. Eng. (IJRTE), 8(1S4) (June 2019) ISSN: 2277–3878
Attia, M.A.: Arabic tokenization system. In: Proceedings of the 5th Workshop on Important Unresolved Matters, Prague, Czech Republic, pp. 65–72 (2007)
Barrett, Neil, Weber-Jahnke, Jens: Building a biomedical tokenizer using the token lattice design pattern and the adapted Viterbi algorithm. BMC Bioinform. 12(Suppl 3), S1 (2011)
Kolak, O.: A generative probabilistic OCR model for NLP applications. In: Proceedings of HLT-NAACL Main Papers, p. 55. Edmonton (May–June 2003)
Schütze, H., Padó, S.: Multi-word tokenization for natural language processing. 14 (2013)
Kudo, T., Richardson, J.: SentencePiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. In: Proceedings Conference on EM in NLP (System Demonstrations), pp. 66–71. Brussels, Belgium (October 31–November 4, 2018)
Tomanek, K., et al. Sentence and token splitting based on conditional random fields. CPACL (2007)
Megerdoomian, K., Zajac, R.: Processing Persian Text: Tokenization in the Shiraz Project. Memoranda in Computer and Cognitive Science MCCS-00-322 (April 2000)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Rai, A., Borah, S. (2021). Study of Various Methods for Tokenization. In: Mandal, J., Mukhopadhyay, S., Roy, A. (eds) Applications of Internet of Things. Lecture Notes in Networks and Systems, vol 137. Springer, Singapore. https://doi.org/10.1007/978-981-15-6198-6_18
Download citation
DOI: https://doi.org/10.1007/978-981-15-6198-6_18
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-6197-9
Online ISBN: 978-981-15-6198-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)