An Effective Duplicate Removal Algorithm for Text Documents

  • Amit Jha
  • Devendra SomwanshiEmail author
  • Mahesh Bundele
Conference paper
Part of the Algorithms for Intelligent Systems book series (AIS)


Maintaining the data quality of any database is always a critical aspect. To maintain the data quality, first of all, there is always a need to remove the redundant data significantly, when multiple data sources are integrated. In this paper, a new and effective data cleaning algorithm is presented aiming to remove duplicate data in text documents having different data types such as numbers, characters, words, and special symbols. The algorithm takes text data as input and removes the duplicate data through checking. Experiment results depict that the algorithm is simple and effective to deal with duplicate data and works fairly well.


Data cleaning Duplicate removal Redundant data 


  1. 1.
    He L, Zhang Z, Tan Y, Liao M (2011) An efficient data cleaning algorithm based on attributes selection. In: 2011 IEEE 6th International Conference on Computer Sciences and Convergence Information Technology (ICCIT), pp 375–379Google Scholar
  2. 2.
    Beskales G, Soliman MA, Ilyas IF, Ben-David S, Kim Y (2010) ProbClean: a probabilistic duplicate detection system. In: 2010 IEEE 26th International Conference on Data Engineering (ICDE), pp 1193–1196Google Scholar
  3. 3.
    Maddodi S, Attigeri GV, Karunakar AK (2010) Data deduplication techniques and analysis. In: 2010 3rd International Conference on Emerging Trends in Engineering and Technology (ICETET), pp 664–668Google Scholar
  4. 4.
    Ali K, Warraich MA (2010) A framework to implement data cleaning in enterprise data warehouse for robust data quality. In: 2010 IEEE International Conference on Information and Emerging Technologies (ICIET), pp 1–6Google Scholar
  5. 5.
    Higazy A, El Tobely T, Yousef AH, Sarhan A (2013) Web-based Arabic/English duplicate record detection with nested blocking technique. In: 2013 8th International Conference on Computer Engineering & Systems (ICCES), pp 313–318Google Scholar
  6. 6.
    Zhang J (2010) An efficient and effective duplication detection method in large database applications. In: 2010 IEEE 4th International Conference on Network and System Security (NSS), pp 494–501Google Scholar
  7. 7.
    Lwin T, Nyunt TTS (2010) An efficient duplicate detection system for XML documents. In: 2010 second IEEE International Conference on Computer Engineering and Applications (ICCEA), vol 2, pp 178–182Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  1. 1.Poornima College of EngineeringJaipurIndia

Personalised recommendations