Search strategy development

What search strategy development is

A literature search for a systematic review is not an ordinary search. Its aim is to find as many as possible of the studies that meet the eligibility criteria, in a way that is documented and can be repeated, and to do so with a manageable number of irrelevant records. Ordinary searching accepts a few good results. A systematic search must be sensitive, which means it catches the relevant studies, while being specific enough that screening the results is feasible. Balancing the two is the craft of the work.

The importance of the search is easy to underestimate. Every later step depends on the set of records that the search produces. Studies that are not found are not screened, appraised or synthesized, and if they differ systematically from those that were found, the review's conclusions are biased. This is why the search is planned, tested, reviewed and reported with as much care as the analysis. The systematic review service places the search within the whole project.

What the service includes

From question to search concepts

The search begins with the question. For an intervention review, the main concepts are usually the population or condition and the intervention, and often the study design. The outcome and the comparator are generally left out of the search, because they are rarely indexed well and adding them lowers sensitivity. Each concept is expanded into the terms that authors and indexers might use: synonyms, spelling variants, abbreviations, older names, and related terms. Terms within a concept are combined with OR, and the concepts are combined with AND.

Capturing terms systematically helps. The searcher examines the index terms and the wording of a handful of known relevant studies, reads the thesaurus entries in each database and tests candidate terms to see what they retrieve. The search is built iteratively: a first version is run, the results are scanned, missing known studies are traced to the terms that would have found them, and the strategy is refined. The guide to building a search strategy explains the operators and the syntax.

Controlled vocabulary, free text and syntax

Two kinds of terms are combined in a strong search. Controlled vocabulary, such as MeSH in MEDLINE and PubMed or Emtree in Embase, is assigned by indexers and catches records regardless of the words the authors used, but it is applied unevenly and is absent from the newest records. Free-text terms search the words in the title, abstract and other fields and catch what indexing misses, but they depend on the authors' wording. Using both is standard practice because each covers the weaknesses of the other.

The syntax differs among platforms. Truncation and wildcards, proximity operators, phrase searching and field codes work differently in each interface, and a strategy cannot be copied between databases without translation. The translation is done and checked for each source, because the same words can retrieve very different sets. Where a database offers a filter for a study design, such as randomized trials, the filter is chosen from an evaluated and published one and not improvised, and its sensitivity and the risk of missing relevant studies are stated.

Choosing databases and sources

No single database covers the literature. The choice depends on the field. A health review would normally search MEDLINE, Embase and a register of controlled trials as a core, and may add databases for nursing, allied health, psychology or the regional literature. Psychology reviews draw on PsycINFO, education reviews on ERIC, management and economics reviews on databases such as Business Source Complete, EconLit or Scopus, and engineering and computing reviews on specialized sources. Multidisciplinary indexes such as Scopus and Web of Science help in topics that cross fields. Research on database combinations suggests that a small set of well-chosen sources can retrieve most relevant records, but that relying on one is risky. The guide to databases compares the main options.

Google Scholar can find items that other sources miss, but its limited functionality, lack of reproducibility and ranking-based retrieval mean it is used as a supplement and not as a primary source. Where access to a subscription database is needed, who provides it is agreed at the start.

Registries, grey literature and citation searching

Studies that are not published in journals are an important source of bias if missed. Trial registries such as ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform show completed and ongoing trials, including those never published, and allow comparison of registered with reported outcomes. Conference proceedings, theses, regulatory documents, preprint servers and the websites of relevant organizations are searched when the topic and the field make unpublished literature likely, with the documentation proportionate to the yield. The guide to grey literature explains the sources and the risks.

Citation searching complements database searching. The reference lists of included studies and relevant reviews are checked, and the papers that cite key studies are examined, forward and backward. Experts in the field can be asked about studies that may have been missed. These additional methods often find studies that the strategy did not, and they provide a check on its performance.

Peer review of the strategy

An error in a search strategy is easy to make and hard to notice: a misplaced bracket can change the logic, a missing synonym can drop a set of studies, and a mistranslated syntax can retrieve nothing. For that reason the strategy is reviewed by a second person. A published guideline, PRESS, offers a checklist covering the translation of the question, Boolean and proximity operators, subject headings, text word searching, spelling and syntax, limits and filters. Peer review has been found to catch errors that would otherwise go unnoticed, and it is recommended for any search that a review depends on. The result of the review and the changes that followed are recorded.

Managing the records

The searches return records from several sources, and many of them are duplicates. Before screening, the records are exported in a consistent format, merged and de-duplicated, using a combination of automated matching and manual checks, and the number removed is recorded for the flow diagram. De-duplication must be careful, because wrongly removing a record loses a study, and leaving duplicates in wastes screening time and risks counting a study twice. The search results and the de-duplicated set are saved so that they can be audited. The guide on de-duplication and screening describes the process.

Documentation and reporting

A search that cannot be reported cannot be repeated or judged. The record kept for each source includes the name of the database and the platform or interface, the date the search was run, the full strategy exactly as run, any limits or filters, and the number of records retrieved. PRISMA-S is the extension of PRISMA that lists what to report about searches, from the databases and registries to the supplementary methods and the peer review of the strategy. Search strategies are usually provided in full as a supplement, since journals rarely have room for them in the main text. Because the literature keeps growing, searches are updated before submission if a long time has passed since they were run, and the update is documented in the same way.

Deliverables

Optional extension: from search to published review

Limitations

No search finds everything. Indexing is imperfect, some studies are not indexed under any of the terms a searcher can think of, publication bias means that some results are never published, and the grey literature is largely out of reach. A search balances sensitivity against the burden of screening, and a more sensitive strategy retrieves more irrelevant records. The recall of a strategy can be estimated against a set of known studies, but it cannot be known exactly. The honest course is to describe the search fully, to show how it was tested and to state what could not be searched, so that readers can judge how much weight to give a review's conclusions. Search skills are also not a substitute for subject knowledge, and the question and the key terms should be checked by someone who knows the field.

How long does it take?

Developing and testing strategies, translating them across databases, and having them reviewed takes longer than running them. The time depends on the number of concepts, the number of databases and registries, how clearly the question is defined, and how many iterations are needed to retrieve the known studies. The size of the result set affects the screening that follows. After the question is clear, a schedule is agreed with its dependencies.

Frequently asked questions