Skip to main content

Fast and Memory-Efficient TFIDF Calculation for Text Analysis of Large Datasets

  • 858 Accesses

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12798)

Abstract

Term frequency – Inverse Document Frequency (TFIDF) is a vital first step in text analytics for information retrieval and machine learning applications. It is a memory-intensive and complex task due to the need to create and process a large sparse matrix of term frequencies, with the documents as rows and the term as columns and populate it with the term frequency of each word in each document.

The standard method of storing the sparse array is the “Compressed Sparse Row” (CSR), which stores the sparse array as three one-dimensional arrays for the row id, column id, and term frequencies. We propose an alternate representation to the CSR: a list of lists (LIL) where each document is represented as its own list of tuples and each tuple storing the column id and the term frequency value. We implemented both techniques to compare their memory efficiency and speed. The new LIL representation increase the memory capacity by 52% and is only 12% slower in processing time. This enables researchers with limited processing power to be able to work on bigger text analysis datasets.

Keywords

  • Text analysis
  • Information retrieval
  • TFIDF
  • Term frequency
  • Data structures
  • Memory allocation

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-79457-6_47
  • Chapter length: 7 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   109.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-79457-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   139.99
Price excludes VAT (USA)
Fig. 1.

References

  1. Buluç, A., Fineman, J., Frigo, M., Gilbert, J., Leiserson, C.: Parallel sparse matrix-vector and matrix-transpose-vector multiplication using compressed sparse blocks. In Proceedings of the 21st Annual Symposium on Parallelism in Algorithms and Architectures (SPAA 2009), pp. 233–244. ACM, New York (2009). https://doi.org/10.1145/1583991.1584053

  2. Kim, S.-W., Gil, J.-M.: Research paper classification systems based on TF-IDF and LDA schemes. Hum. Cent. Comput. Inf. Sci. 9(1), 1–21 (2019). https://doi.org/10.1186/s13673-019-0192-7

    CrossRef  Google Scholar 

  3. Koushik, G., Rajeswari, K., Muthusamy, S.: Automated hate speech detection on Twitter. In: 2019 5th International Conference On Computing, Communication, Control And Automation (ICCUBEA), Pune, India, pp. 1–4 (2019) https://doi.org/10.1109/ICCUBEA47591.2019.9128428

  4. Michailidis, M.: Sentiment140 dataset with 1.6 million tweets. http://kaggle.com/kazanova/sentiment140. Accessed 19 Jan 2021

  5. Romadon, A., Lhaksmana, K., Kurniawan, I., Richasdy, D.: Analyzing TF-IDF and word embedding for implementing automation in job interview grading. In: 2020 8th International Conference on Information and Communication Technology (ICoICT), Yogyakarta, Indonesia, pp. 1–4 (2020). https://doi.org/10.1109/ICoICT49345.2020.9166364

  6. Stoer, J., Bulirsch, R.: Introduction to Numerical Analysis. Springer, New York (2002). https://doi.org/10.1007/978-0-387-21738-3ISBN 978-0-387-95452-3

    CrossRef  MATH  Google Scholar 

  7. Varol, C., Abdulhadi, H.: Comparision of string matching algorithms on spam email detection. In: 2018 International Congress on Big Data, Deep Learning and Fighting Cyber Terrorism (IBIGDELFT), Ankara, Turkey, pp. 6–11 (2018). https://doi.org/10.1109/IBIGDELFT.2018.8625317

  8. Yamout, F., Lakkis, R.: Improved TFIDF weighting techniques in document Retrieval. In: 2018 Thirteenth International Conference on Digital Information Management (ICDIM), Berlin, Germany, pp. 69–73 (2018). https://doi.org/10.1109/ICDIM.2018.8847156

  9. Yuan, H., Wang, Y., Feng, X., Sun, S.: Sentiment analysis based on weighted Word2vec and Att-LSTM. In: Proceedings of the 2018 2nd International Conference on Computer Science and Artificial Intelligence (CSAI 2018), pp. 420–424. ACM, New York (2018) https://doi.org/10.1145/3297156.3297228

  10. Zhao, G., Liu, Y., Zhang, W., Wang, Y.: TFIDF based feature words extraction and topic modeling for short text. In: Proceedings 2018. TFIDF based Feature Words Extraction and Topic Modeling for Short Text. Proceedings of the 2018 2nd International Conference on Management Engineering, Software Engineering and Service Sciences (ICMSS 2018), pp. 188–191. ACM, New York (2018). https://doi.org/10.1145/3180374.3181354

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Samah Senbel .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Senbel, S. (2021). Fast and Memory-Efficient TFIDF Calculation for Text Analysis of Large Datasets. In: Fujita, H., Selamat, A., Lin, J.CW., Ali, M. (eds) Advances and Trends in Artificial Intelligence. Artificial Intelligence Practices. IEA/AIE 2021. Lecture Notes in Computer Science(), vol 12798. Springer, Cham. https://doi.org/10.1007/978-3-030-79457-6_47

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-79457-6_47

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-79456-9

  • Online ISBN: 978-3-030-79457-6

  • eBook Packages: Computer ScienceComputer Science (R0)