Systematic review: the method

Why the method exists

Studies on the same question often disagree, and the studies that are easiest to find are not a random sample of those that exist. A traditional narrative review draws on whatever the author knows or finds, selects and weighs it by judgment that is not disclosed, and reaches conclusions that another author with other sources might not. Research in the 1980s and after showed that expert reviews could lag behind evidence, and that the conclusions of narrative reviews were influenced by the choices of the reviewers. The response was to apply to the review itself the principles of research: a defined question, a protocol, a systematic search, explicit inclusion criteria, assessment of the quality of the evidence, and an open account of the methods.

The result is a method that can be repeated and checked, whose conclusions are linked to evidence in a transparent way, and that is designed to minimize bias in the identification and selection of studies. Organizations such as Cochrane and, more widely, those that produce guidelines and health technology assessments rely on it, and the method has been adopted in education, social policy, management, the environment and other fields. The service is described on the page for the systematic review service.

Core principles

• A focused question. The question is specific enough to guide the search and the selection, and is stated in a structured form such as population, intervention, comparator and outcome.
• A protocol. The methods are planned in advance and, where possible, registered, so that changes can be seen and justified.
• A comprehensive search. Several sources are searched with documented strategies, and the search aims to find all eligible studies, including unpublished ones.
• Explicit selection. Studies are included or excluded according to criteria set in advance, applied by more than one person, with reasons recorded.
• Critical appraisal. The risk of bias in each study is assessed and taken into account.
• Transparent synthesis. Results are combined, with or without statistical pooling, by methods that are stated, and the certainty of the conclusions is assessed.
• Complete reporting. The report gives enough detail for readers to judge the review and, in principle, repeat it.

The stages

1. Define the question and criteria

   The question, the eligibility criteria for populations, interventions or exposures, comparators, outcomes and study designs, and the plan for the review are developed and written as a protocol.

2. Search

   Databases, registries and other sources are searched with strategies that combine controlled vocabulary and free text, and the strategies are peer reviewed and recorded. See search strategy development.

3. Select

   Records are de-duplicated and screened against the criteria, usually by two reviewers independently, first on title and abstract and then on full text. Reasons for exclusion are recorded, and the flow of studies is reported.

4. Extract and appraise

   Data are extracted on a piloted form, usually in duplicate or with independent verification, and the risk of bias in each study is assessed with a tool matched to the design. See data extraction and risk-of-bias assessment.

5. Synthesize

   Findings are combined by meta-analysis where studies are similar enough, or by structured narrative synthesis where they are not, with exploration of differences and assessment of reporting bias.

6. Interpret and report

   The certainty of the evidence for each outcome is rated, conclusions are drawn that fit that certainty, and the review is reported according to PRISMA 2020.

How the method limits bias

Each feature of the method guards against a specific source of bias. A protocol prevents the selective modification of criteria and analyses after results are seen. A comprehensive search reduces the chance that studies are missed because of their results, as published studies with positive findings are easier to find than unpublished ones, and trial registries and grey literature are searched for this reason. Independent screening and extraction reduce errors and individual bias in selecting and recording. Appraisal of risk of bias prevents weak studies from being treated as equal to strong ones. Prespecification of outcomes and analyses reduces selective reporting. And the full reporting allows others to detect the biases that remain.

The method does not eliminate bias. A review inherits the biases in the primary studies, such as flawed randomization or selective outcome reporting, and the safeguards reduce those added by the review process but do not repair the evidence. The assessment of certainty exists to describe how much those biases and other limits matter.

Synthesis with and without meta-analysis

The synthesis can take several forms. Where studies are similar enough in questions, populations and outcome measures, a meta-analysis estimates a pooled effect and examines differences between studies. Where they are not, a narrative synthesis is used. A good narrative synthesis is structured: studies are grouped in a way that makes sense for the question, a common effect measure or standard metric is chosen to allow comparison, results are presented in tables and plots, and the synthesis explains patterns. The SWiM guideline describes the elements of reporting such a synthesis. Vote counting, which tallies how many studies found an effect, should be avoided, because it ignores the size and precision of each study and can mislead. Qualitative evidence is synthesized with methods designed for it, and mixed-methods reviews integrate both kinds.

Whatever the form, the synthesis explores why results differ, assesses the risk that the available evidence is distorted by publication bias or selective reporting, and draws conclusions that match the certainty of the evidence. See meta-analysis.

Reporting standards

PRISMA 2020 is the main reporting guideline. It gives a 27-item checklist and a flow diagram, with a checklist for abstracts. It requires that the report state the objectives, the eligibility criteria, the information sources and full search strategies, the selection and data collection processes, the data items, the methods for assessing risk of bias and certainty, the synthesis methods, the results of each stage, and the limitations, together with the registration and protocol and the availability of data and code. Extensions cover searches, protocols, scoping reviews, diagnostic accuracy, network meta-analyses and individual participant data. Journals increasingly require the completed checklist. See PRISMA 2020.

Reading the flow of studies

The flow diagram is the visual summary of a systematic review's selection process and a quick test of its credibility. It shows the number of records identified from each source, the number removed as duplicates, the number screened, the number excluded at each stage with the reasons for full-text exclusions, and the number of studies and reports finally included. The numbers should add up at each step. A reader can see from it whether the search was broad, how many records were screened, how selective the criteria were and whether the final set is a small fraction of what was found, which is normal. A diagram with arithmetic that does not agree, or with no reasons for exclusion, signals problems in the conduct or the reporting of the review.

A study may be reported in several papers, so the diagram counts both reports and studies. The review needs rules for linking reports to studies, and it should say how many of the included studies came from sources other than the main databases, since this indicates how much the supplementary methods added.

Judging the quality of a review

Readers and editors judge a systematic review by a handful of features. Was there a protocol, and is it available? Was the question clear and the eligibility criteria stated? Was the search comprehensive, with several databases, strategies provided and sources beyond the databases? Was selection done in duplicate? Was risk of bias assessed with a suitable tool and used in the synthesis? Were the methods of synthesis appropriate, and was heterogeneity explored and not ignored? Was reporting bias considered? Were conclusions in line with the evidence and its certainty? Was the review reported against PRISMA 2020? Tools exist to appraise reviews systematically, such as AMSTAR 2 and ROBIS, and they are used when reviews are themselves synthesized in umbrella reviews, and by editors and guideline groups assessing which reviews to trust.

Variants of the method

Scoping review

Maps the extent and nature of the evidence on a broad topic and identifies gaps, usually without assessing risk of bias or pooling results.

Rapid review

Streamlines the methods, for example by limiting databases or using a single screener, to meet a deadline, with the limitations stated.

Umbrella review

Synthesizes the findings of existing systematic reviews, taking account of overlap among their primary studies.

Living systematic review

Updated continually as new evidence appears, in fields where evidence moves quickly.

Qualitative evidence synthesis

Integrates findings of qualitative studies with methods designed for them.

Mixed-methods review

Combines quantitative and qualitative evidence to answer a question that needs both.

The method across fields

Systematic review began in health and remains most developed there, where the Cochrane Handbook and the associated tools set the standard. Related methods have been developed in other fields, such as the Campbell approach in social policy and education and the guidance of the Collaboration for Environmental Evidence in conservation science. Conventions differ. Health reviews are usually of randomized trials and use risk-of-bias tools and GRADE. Reviews in other disciplines often include a wider range of designs and may synthesize in different ways, and the appraisal tools are less standardized. The research-area pages explain how the method is applied in each field, and the common principles of a protocol, a systematic search, explicit criteria and transparent reporting apply everywhere. See research areas.

Strengths and limitations

Strengths include transparency, reproducibility and the minimization of the biases of selection. A review can show where evidence is strong and where it is weak, highlight the gaps that new research should address, and provide a basis for guidelines and decisions. Limitations are equally real. Reviews are resource-intensive and slow. No search is complete. Publication bias and selective reporting affect the available evidence. The quality of a review is capped by that of the studies it includes. Many topics already have multiple reviews, some redundant or of poor quality, so a new review needs a reason to exist. And a review summarizes evidence and does not make decisions: it does not tell an individual patient or an institution what to do.

How we can help

Frequently asked questions