The Difficulty of Search and an Interesting Solution

Summary

Fundamentally, the current searching paradigm does not work well. Current searching assigns keywords to documents; users think of keywords of their own and these keywords are then matched against those of the body of documents. When the same word appears there is a match and the document is returned to the user. This relies on a number of assigned keywords on the document side, and usually a single search term from the user. Matching is a very hit or miss affair, involving problems of terminology, matching, indexing, relevance ranking and others which mean users are as often frustrated as happy with the results. We will discuss a solution which enables a search system to create and maintain a richer, deeper context for both the document and the user, and employs novel matching and ranking technology to provide better answers. Part of the technology requires a broad enough universe for searching which requires the use of federated search of disparate resources. This brings its own problems of syntactic and semantic normalisation as well as the intelligent utilisation of the results to provide better answers for the user.