Gazetteer Compression Technique Based on Substructure Recognition

  • Jan Daciuk
  • Jakub Piskorski
Part of the Advances in Soft Computing book series (AINSC, volume 35)


Finite-state automata are state-of-the-art representation of dictionaries in natural language processing. We present a novel compression technique that is especially useful for gazetteers – a particular sort of dictionaries. We replace common substructures in the automaton by unique copies. To find them, we treat a transition vector as a string, and we apply a Ziv-Lempel-style text compression technique that uses suffix tree to find repetitions in lineaqr time. Empirical evaluation on real-world data reveals space savings of up to 18,6%, which makes this method highly attractive.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    1. Daciuk J. 2000. Experiments with Automata Compression. Proceedings of CIAA - Implementation and Application of Automata, London, Ontario, Canada, 105–112Google Scholar
  2. 2.
    2. Daciuk J., Mihov S., Watson B., Watson R. 2000. Incremental Construction of Minimal Acyclic Finite State Automata. Computational Linguistics, 26(1), pages 3–16CrossRefMathSciNetGoogle Scholar
  3. 3.
    3. Drozdzyński, W., Krieger H-U., Piskorski, J., Schäfer, U., Xu, F. Shallow Processing with Uni.cation and Typed Feature Structures — Foundations and Applications. In Künstliche Intelligenz, 2004(1), pages 17–23Google Scholar
  4. 4.
    4. Dan Gus.eld 1997). Algorithms on Strings, Trees, and Sequences. Cambridge University Press.Google Scholar
  5. 5.
    5. Hopcroft J. 1971. An nlogn Algorithm for Minimizing the states in a Finite Automaton. The Theory of Machines and Computations, Academic Press, 189–196.Google Scholar
  6. 6.
    6. Nederhof, M.-J. 2000. Practical experiments with regular approximation of context-free languages. Journal of Computational Linguistics, 26(1), pages 17–44CrossRefMathSciNetGoogle Scholar
  7. 7.
    7. Kowaltowski T, Lucchesi C. and Stol. J. 1993. Minimization of Binary Automata. Proceedings of the First South American String Processing Workshop, Belo Horizonte, Brasil.Google Scholar
  8. 8.
    8. Kowaltowski T., Lucchesi C., Stol. J. 1998. Finite Automata and E.cient Lexicon Implementation. Technical Report IC-98–02, University of Campinas, Brazil.Google Scholar
  9. 9.
    9. Piskorski J. 2005). On Compact Storage Models for Gazetteers. Proceedings of the 5th InternationalWorkshop on Finite-State Methods and Natural Language Processing, Helsinki, Finland, Springer LNAI.Google Scholar
  10. 10.
    10. Revuz D. 1991. Dictionnaires et Lexiques, Méthodes et Algorithmes. PhD Thesis, Université Paris 7.Google Scholar
  11. 11.
    11. Tarjan R, and Andrew Chi-Chih Yao. (1979) Storing a sparse table. Commun. ACM. 22(11), ACM PressGoogle Scholar

Copyright information

© Springer 2006

Authors and Affiliations

  • Jan Daciuk
    • 1
  • Jakub Piskorski
    • 2
  1. 1.Technical University of GdańskGdańskPoland
  2. 2.DFKI GmbHSaarbrückenGermany

Personalised recommendations