Systematic review service

What a systematic review is, and what it is not

A systematic review is a structured way of finding, selecting, appraising and summarizing all the studies that address a defined question. Its defining features are that the question and methods are specified in advance, the search is comprehensive and documented so that it can be repeated, studies are chosen by explicit criteria, each study is assessed for risk of bias, and the synthesis and its limits are reported openly. These features protect against the selective use of evidence that makes an ordinary literature review unreliable.

A systematic review is not the same as a meta-analysis. The review is the process; the meta-analysis is an optional statistical step that combines the results when the studies are similar enough. Many systematic reviews include one, and many do not, either because only a few studies exist or because they differ too much in population, intervention or outcome. The distinction is explained in meta-analysis versus systematic review.

It is also not a quick summary. A well-conducted review is a research project in its own right, with a protocol, a team, a search, thousands of records to screen and a reproducible record of every decision. The value of the result comes from that discipline.

When a systematic review is the right design

The choice of review design should follow from the question, not from habit. A systematic review is appropriate when there is a focused question, a reasonable body of primary studies, and a need for a result that others can trust and repeat. The table compares the main alternatives.

The scoping stage of a project is where this choice is made. It is better to reach the right design before the search than to discover afterwards that the review answers a different question from the one intended.

What the service includes

Framing the question and the eligibility criteria

Everything in a review follows from the question, so the question deserves time. For intervention questions the usual frame is population, intervention, comparator and outcome (PICO), and variants such as PICOS add the study design. For other kinds of question, such as prognosis, exposure or qualitative experience, other frames fit better. The guide to PICO and related frameworks compares them.

The eligibility criteria turn the question into rules for deciding which studies count. They specify the population and setting, the interventions or exposures and comparators, the outcomes of interest and how they must have been measured, the study designs that will be accepted, and any limits on date, language or publication status. Each criterion should be justified, because each one can be challenged. Restricting to English-language publications, for example, is common and defensible in some topics and a source of bias in others.

Two problems recur. A question that is too broad produces an unmanageable number of studies that cannot be combined meaningfully, and a question that is too narrow may find nothing. The usual remedy is to refine the question against a scan of the literature before committing to a protocol, and to check whether a recent review already answers it.

Protocol and registration

A protocol records the review's question, criteria, search sources, screening and extraction procedures, risk-of-bias approach and synthesis plan before the results are known. Its purpose is to limit the freedom to change decisions in light of what the studies show, which is the main safeguard against selective reporting. The PRISMA-P guideline lists the items a protocol should contain.

Where a suitable registry exists, the protocol is registered. PROSPERO accepts systematic reviews with a health-related outcome, and other platforms are available for reviews outside its scope. Registration makes the plan public, allows readers to compare the published review with it, and helps to avoid duplication of effort. Changes after registration are recorded as amendments with the date and reason, and reported in the final paper.

Searching

The search is where many reviews are won or lost. A review that misses relevant studies has a problem that no later step can repair. A comprehensive search combines several bibliographic databases suited to the field, uses both controlled vocabulary and free-text terms for each concept in the question, and is supplemented by searching trial registries, reference lists of included studies and, where appropriate, grey literature such as theses, conference abstracts and reports. The strategy is tested against a set of studies already known to be relevant, and it is peer reviewed, ideally by a second searcher, using a checklist such as PRESS.

The search is documented in enough detail to be repeated: the databases and platforms, the dates searched, the full strategy for each, and the number of records retrieved. PRISMA-S sets out what to report. Because evidence accumulates, searches are updated before submission when a substantial time has passed. The topic is covered in the guide to building a search strategy and the guide to databases.

Screening and selection

Retrieved records are de-duplicated and then screened against the eligibility criteria, first by title and abstract and then by full text. Because screening involves judgment, it is normally done independently by two reviewers who resolve disagreements by discussion or by a third reviewer. Dual screening reduces the chance of excluding a relevant study by error, and where resources are limited a documented partial approach, such as a second reviewer checking a sample, can be used with its limits stated.

Every full-text exclusion is recorded with its reason, and the numbers at each stage feed the PRISMA flow diagram. One study can be reported in several papers, and several reports must be linked to a single study so that its data are counted once. The guide on de-duplication and screening describes the process in more detail.

Extraction and risk-of-bias assessment

Data are extracted onto a form that is piloted on a few studies before use. The form captures study characteristics, participants, interventions or exposures, outcome definitions, results and funding. Independent duplicate extraction, or at least independent verification of key data, is recommended because errors in extraction are common and can change conclusions. Where results are missing, they are sought from the full text, supplements and registry records and, where appropriate, from the authors.

Each included study is also assessed for risk of bias with a tool matched to its design, such as RoB 2 for randomized trials, ROBINS-I for non-randomized studies of interventions or QUADAS-2 for diagnostic accuracy. The judgments are used in the interpretation of the findings, in sensitivity analyses and in the rating of the certainty of the evidence. See risk-of-bias tools.

Synthesis

Synthesis combines the findings of the included studies. When the studies are sufficiently similar and report compatible data, a meta-analysis estimates an overall effect and examines differences between studies. When they are not, a structured narrative synthesis is the honest alternative. Narrative does not mean unstructured: the SWiM guideline describes how to group studies, to choose and report an effect measure, to present the results and to say how certain they are, without resorting to vote counting, which ignores the size and precision of each study.

In either case, the synthesis explores why results differ, using prespecified subgroups or study characteristics, and assesses the risk that the available evidence is distorted by publication bias or selective reporting. The certainty of the conclusions for each important outcome is then rated, for health questions usually with GRADE, and the conclusions are worded to match that certainty.

Reporting

PRISMA 2020 is the standard reporting guideline for systematic reviews. It consists of a checklist of 27 items covering the title, abstract, introduction, methods, results, discussion and other information, with a flow diagram of the study selection and a separate checklist for abstracts. Extensions exist for scoping reviews, diagnostic accuracy, network meta-analysis, individual participant data and literature searches. Journals increasingly ask for the completed checklist at submission.

Good reporting includes the full search strategies, the list of excluded full-text studies with reasons, the extracted data, the risk-of-bias judgments for each study, the code and the data used in any meta-analysis, and an account of every deviation from the protocol. The aim is that a reader could judge the review and, in principle, repeat it. See PRISMA 2020.

Deliverables

Optional extension: full manuscript and submission

Quality assurance

Limitations of systematic reviews

A systematic review is only as reliable as the studies it summarizes and the completeness of the search that found them. No search is complete, and studies with null or unfavorable results are less likely to be published and found. Differences between studies in populations, interventions and outcome definitions can limit what can be combined, and the quality of the primary evidence caps the certainty of the conclusions however rigorous the review. Reviews also take time, so their conclusions may lag behind new studies, which is the motivation for living systematic reviews in fast-moving fields.

Many topics already have several reviews, some redundant or of low quality, so a new review needs a clear reason to exist. Finally, a review summarizes evidence and does not replace judgment: it does not tell an individual patient or a particular institution what to do.

How long does a systematic review take?

The duration depends on the breadth of the question, the number of records to screen, the number of included studies, the number of outcomes, whether a meta-analysis is planned, and how quickly decisions are made. A narrow question with a few dozen included studies is a different project from a broad one with several hundred. After the scoping stage, a schedule is proposed with its dependencies, and any fixed deadline, such as a thesis submission or a grant report, is taken into account in the scope.

Frequently asked questions