Automatic Indexing: The Assignment of Controlled Language Index Terms

Conclusions

Controlled language index terms are valuable index terms. When automating their assignment to texts, the knowledge about the words and phrases that imply the term concepts is needed. This knowledge is implemented in thesauri and knowledge bases for text categorization. Building thesauri automatically remains a very difficult task. Learning the classification patterns of broad text classes is somewhat easier. Constructing a text classifier that generalizes from example texts can build upon a long tradition of research in pattern recognition and of experiments in relevance feedback in retrieval. The problem is to correctly find the patterns in example texts that are associated with the subject or classification codes. Statistical techniques of pattern recognition, leaning of rules and trees, and training of neural nets are all based upon the principle that when a large number of examples or a limited number of good instances are available, the desired patterns will be identified based upon re-occurring features, and noise will be neglected. However, in text classification the number of features is enormous and many features have no relevance. In addition, the number of positive examples of each text class is often limited due to changing document collections and classification systems.