Design and implementation of Metta, a metasearch engine for biomedical literature retrieval intended for systematic reviewers

The front-end

In general, our aim was to simplify the process of submitting queries so that users did not need to concern themselves with complex queries, or the use of search tags or other specialized commands. On the other hand, we wanted to retain flexibility as much as possible, so that users could adjust the pre-set options for each database if they so desired. As shown in Figure 2, the Metta homepage represents a concise interface that achieved both of these aims. Users choose one of 4 different search tracks (indicated at left) to carry out either a search for human-related studies (i.e. articles which are indexed under Humans in each database); a search for case reports and similar articles; a search for systematic reviews and meta-analyses; or a general search (i.e. utilizing only the default settings of each search engine). Each search track has built-in settings designed to optimize comprehensive retrieval of the relevant types of articles, while minimizing irrelevant articles. The tailored search strategies underlying these settings are listed within the Help page (Additional file 1). Although users cannot alter the search track settings directly, this does not limit the scope of possible searches since queries can still be built up freely on the general search track. Furthermore, by clicking on “show options” for any search engine, users can manually over-ride or add options and restrictions at will, though in practice, few, if any users chose to manually adjust the search settings within individual search engine options.

Open image in new window

In general, Metta employed the default query expansion and processing strategies of each database, with some exceptions (e.g. full-text searching was turned off for PsycINFO). Because all five databases employed three common search tags (title [ti], author [au] and abstract [ab]) these were permitted in user-entered queries. The permitted tags are shown as examples on the right side of the homepage, and in detail on the Help page (Additional file 1).

After a user inputs a query on the human track (e.g., olanzapine AND schizophrenia AND relapse) and clicks “search”, he or she will be directed to the search result page shown in Figure 3. The user inputted query and selected search track are indicated within the light yellow panel shown at the top. The query result summary is shown in the middle table, with columns showing each database with its number of results returned, connection status, and other parameter and status columns. The database name column offers the link to search result page (shown in Figure 3) of individual databases. Note that users may always switch between the result page and individual database result pages using the blue panel located at the top of the page. Figure 4 shows one of the individual database result pages, i.e. for PubMed. The snippet of each result record is in a Google-like format, showing the article title in a hyperlink to the article’s original link in the PubMed (or other source database) website, followed by journal title, publication year, issue & pages; then the authors & their affiliations; and finally abstract and original URL. Twenty records are displayed on a page.

Open image in new window

Open image in new window

On the result page (Figure 3), when the user clicks the “Export (with Deduplication)” link, this triggers Metta to pull all retrieved records from all databases, perform de-duplication for records from different databases (see below), and offer users the option to download the de-duplicated records as text files for offline use (Figures 5 and 6). The amount of time for export to be completed varies according to the number of records and their allocation among databases. Retrieval of full records was slowest for the PsycINFO database in which each record had to be re-queried and downloaded individually; although a thousand records could be exported in about a minute from the other databases, PsycINFO was at least 10 times slower. Users are offered two file download links for all the de-duplicated results, one for export in XML format (ideal for further computer processing) or BibTex format (this is designed to be compatible with with a wide range of commercial and open source reference manager software) (Figure 6).

Open image in new window

Open image in new window

Design and user issues

The front-end of Metta was intended to satisfy a wide range of different types of users, as well as a wide range of different search strategies. Certainly Metta was aligned to the needs of the majority of people searching the biomedical literature, who tend to carry out only one or two queries at a time, employ only one or a few search terms, and do not routinely use search tags [7, 8]. To accommodate the needs of users who employ an iterative approach (in which initial results are examined, and the initial query modified and resubmitted), we cached queries so that all previous queries from the same session were visible as users began to type in the query box.

A much more difficult decision was how to reconcile a unified query interface with the prevailing practice of many systematic reviewers to carry out long, complex queries that involve search tags and advanced commands which are specific to individual databases. The individual databases all have optional advanced search interfaces that allow users to build up queries consisting of separate query terms (linked to field tags) concatenated with AND, OR or NOT as well as restricting search term to specific text and meta-data fields. However, entering a search tag into Metta that is inappropriate for one or more of the databases will result in query errors. We did initially implement an Advanced Search page in Metta (patterned after PubMed’s Advanced Search Builder), but finally decided to remove it and only the basic query interface is currently active.

This decision was reinforced by an analysis of the reasons that reviewers exclude initially-retrieved clinical trial articles from final inclusion in systematic reviews (discussed in detail in [9, 10]). Briefly, we found that reviewers did not trust that metadata indexing of articles was adequately reliable, particularly with regard to study design aspects (such as randomization or use of placebos), and so a) did not employ these restrictions while carrying out initial searches, and b) utilized many different word and phrase combinations in the query, in an attempt to capture all possible relevant articles. This results in an initial search that may retrieve 10–100 times more records than are finally deemed to be relevant for inclusion in the systematic review.

The resources required to conduct systematic reviews are too great for several reasons. Construction of the query is a complex process carried out by specialists, typical searches return too many articles that will be excluded from the systematic review, and doing a comprehensive search across multiple databases is time consuming and incurs a lot of manual work in deduplication and other tasks necessary to clean up the search results. Metta is only the first step in a pipeline project that creates a series of computer-assisted tools to assist systematic reviewers [11] (another step of which is to re-tag clinical trial articles with study design labels). The pipeline is intended to present an alternate manner in which systematic review literature search and initial review could be performed which reduces resource utilization in several time and resource consuming areas. We felt that it was more important to keep Metta simple and predictable in its output, rather than designing it to handle extremely large sets of retrieved records (~5,000 or more) or to facilitate the entry of extremely complex queries by users via an advanced query builder.

Thus, for systematic reviewers to be expected to adopt Metta on a routine basis, it will be necessary to re-engineer multiple steps in the process by which articles are queried, retrieved, filtered and examined for inclusion. For example, better study design annotation and filtering of articles retrieved by Metta searches should give overall high-recall retrieval with better precision than is currently obtained by conventional searches, and thus largely obviate the need of reviewers to formulate highly complex queries. In any case, one of us (NRS) has found that it is easier simply to build up complex queries on a blank Word document and then cut-and-paste it into the Metta query window, than to build up queries step by step within an advanced query page.

The back-end

This includes export of full bibliographic records in XML format intended for use by automated informatics processing tools residing later in the pipeline project. We modified the XML format used by PubMed so that it was applicable across all of the databases. In addition, we created a separate link containing a text file of abbreviated bibliographic information in BibTex format, which can be readily imported into commercial reference manager software. A significant issue in this module is to identify articles that are retrieved from multiple databases. This allows duplicate results to be removed (only one copy is kept), which saves time for systematic reviewers [14]. Identification of multiple records that correspond to the same real-world entity is known as entity identification and record-linkage, and the problem has received much attention in the database community (e.g., [15, 16]). For Metta, the problem is solved in three steps. First, the semantic data units within each record are identified. For example, a citation record may have semantic data units Author, Title, Journal Name, etc. Second, a set of distance/similarity functions is used to compute how well two values of the same semantic from different records are matched. For example, an edit-distance function can be used to compute how similar two titles are. Third, for each pair of records, based on how similar their corresponding data units are, a decision is made on whether the two records are matched. Once citation records corresponding to the same article are identified, de-duplication in Metta was carried out by first retaining records that were indexed in PubMed (because of its better and more elaborate indexing scheme) and then following a priority order: EMBASE, Cochrane, CINAHL and lastly, PsycINFO. The details about the algorithms and code for de-duplication in Metta are described in a separate publication [17].

Challenges

Metta is a working demonstration site that continues to undergo field testing and modification to serve systematic reviewers. However, a number of important challenges will be important to tackle if Metta is to become deployed as a stable production service:

a) Perhaps the biggest challenge is the fact that unannounced changes to individual search engines (and their bibliographic databases) occurred regularly, if unpredictably, every few weeks or so. Some of the programming changes were major (e.g. a website based on pages rendering relatively simple HTML changed over to Javascript). These changes could result in very subtle errors. Once detected, manual re-programming and testing were required to ensure that Metta connected and exported from each search engine accurately. To allow automatic detection of changes to the query interfaces, databases or export functions, it will be necessary to deploy a series of test scripts which run pre-specified queries (with fixed publication dates) for which the correct number of retrieved and exported records are known. These need to be run daily through each search track, and any discrepancies will be flagged for manual inspection. b) At present, errors within the system (e.g. failures of connectivity) are flagged to users. However, it will be desirable to detect and deliver error messages that arise from one or more of the search engines, e.g., when they have inputted an inappropriate query. c) It remains an open question how much query processing should be performed by Metta. Some of the individual search engines already carry out more or less extensive types of automatic query processing -- for example, PubMed employs automatic Medical Subject Heading detection and expansion, phrase lists and spelling correction. CINAHL automatically expands Boolean searches to include partial matches if no exact matches are found. One could certainly add query processing modules that automatically recognize synonyms, abbreviations and their long forms, singular vs. plural word forms, drug names, American vs. British spelling variants, and so forth. Certain standardized search strategies could be built into these query modules. For example, research building upon the Cochrane Consortium guidelines has recommended incorporating a set of query terms for retrieving randomized controlled trials [18, 19, 20], though none of these identifies 100% of the relevant articles [21, 22]. Optimized strategies for clinical queries have also long been formulated e.g., [23, 24, 25]. Such prepackaged queries could be pre-set to be selected by users with a single click. As we study further how users interact with Metta, and learn whether it is likely to gain wide acceptance from systematic reviewers (or from the general community of biomedical searchers), additional query processing modules will be added. d) At present, Metta retrieves the bibliographic records but does not show links to full text articles or pdfs that may be present. Such a feature should assist reviewers in examining the full text of potentially relevant articles.