Umbrella review

What an umbrella review is

Systematic reviews have multiplied, and on many topics there are now several, sometimes dozens, which address overlapping questions, differ in what they include, and sometimes reach different conclusions. A decision maker faced with this wants a single, coherent account. An umbrella review responds by treating the systematic reviews themselves as the evidence. It identifies the reviews on a topic, appraises their quality, extracts and compares their findings, and synthesizes them, so that a reader can see what the body of reviews says as a whole and where it conflicts. The approach has several names, including overview of reviews, review of reviews and meta-review, and the terms are used with varying meanings.

Umbrella reviews are most common in health and are growing elsewhere. They typically address a broad question, for example the effects of all interventions for a condition, or the associations between a range of exposures and a range of outcomes. They sit above systematic reviews in the hierarchy of synthesis, and they are not a way to avoid the work of a systematic review: they rely on the quality of the reviews they summarize. See systematic review.

When an umbrella review is the right design

An umbrella review is appropriate when several good-quality systematic reviews already exist on the topic, when the question is broad enough that no single review covers it, when reviews conflict and the reasons need to be established, or when decision makers want a summary of what the evidence says across interventions or outcomes. It is efficient when the primary literature is so large that a new review would be impractical, or when reviews already exist for each component. It is not suitable when there are few reviews, when the existing reviews are of poor quality, when they are out of date and new studies are important, or when the question is about a new intervention with no reviews. In those cases a new systematic review, an update or a scoping review is better. A decision based on a review of reviews will be only as good as the reviews, so the first step is a check on whether the reviews found are good enough to build on.

The problem of overlap

The central methodological problem is that reviews on related questions often include the same primary studies. If an umbrella review counts the findings of five reviews that all include the same large trial, that trial is counted five times, which inflates the apparent weight of the evidence and gives a false impression of consistency. Overlap can be substantial. It is not always a problem, since reviews can legitimately share studies, but ignoring it misleads.

The recommended approach is to measure and report overlap. A citation matrix lists the primary studies in rows and the reviews in columns, showing which includes which. The corrected covered area, an index derived from the matrix, summarizes the degree of overlap as a percentage, with rough categories from slight to very high, and the index can be computed for the whole set and for pairs of reviews. The response to high overlap can be to select one review for each question, using explicit criteria such as quality, recency and comprehensiveness, to analyze reviews separately and avoid combining their results, to re-analyze the primary studies, or to present the findings with the overlap noted. The decision is made in the protocol, and the effect of overlap is discussed in the report. See the guidance in the references.

Conducting an umbrella review

1. Define the question and the eligibility

   The question and the criteria for reviews are stated, including what counts as a systematic review, the minimum methodological standard, the date range and the language.

2. Search for reviews

   Databases are searched with strategies that include a filter for systematic reviews, supplemented by sources of reviews such as the Cochrane Library and registries of reviews. The search is documented as in any systematic review.

3. Select and appraise

   Reviews are screened in duplicate, and each included review is appraised with a tool for reviews.

4. Extract data

   Characteristics of the reviews, the included studies, the methods, and the results, including effect estimates, heterogeneity and the certainty of evidence as assessed by the review authors, are extracted.

5. Assess overlap and synthesize

   Overlap is measured, the findings are summarized by question and outcome, and conflicts are explained.

6. Grade and report

   The strength or certainty of the evidence for each association or comparison is rated and the review is reported according to PRIOR.

A worked idea: reading a citation matrix

Suppose an umbrella review includes four reviews that together cite ten distinct primary studies. In a citation matrix the studies are rows and the reviews are columns, with a mark wherever a review includes a study. If every review includes the same six studies and each adds one or two of its own, the total number of marks is large relative to ten distinct studies, and the corrected covered area, which compares the number of inclusions beyond the first with the maximum possible overlap, will be high. The calculation is the total number of marks, minus the number of distinct studies, divided by the number of distinct studies times the number of reviews minus one. With 28 marks, 10 studies and 4 reviews, this is (28 minus 10) divided by (10 times 3), or 0.6, which is 60 percent and would be classed as very high overlap. The numbers are invented to show the arithmetic. The practical meaning is that the four reviews are largely re-reporting the same evidence, and counting their agreement as independent confirmation would be wrong.

Searching for reviews

The search for an umbrella review looks for systematic reviews and not primary studies. Filters for systematic reviews and meta-analyses are available for the main databases and are used with the subject terms. Specialist sources add coverage: the Cochrane Library for Cochrane reviews, registries of reviews, such as PROSPERO, to identify reviews in progress, collections of overviews, and the websites of guideline and assessment bodies. Because the definition of a systematic review differs, eligibility criteria set the minimum standard, such as a stated search of at least two databases with a reproducible strategy and some form of appraisal. Reviews that fall short are listed as excluded with reasons. Searching also looks for recent primary studies published after the latest review, since an umbrella review with a lag can miss important new evidence, and a decision is made about whether to update.

Appraising the reviews

The reviews must themselves be assessed, because quality varies enormously. Two tools are widely used. AMSTAR 2 is a critical appraisal tool with 16 items covering the protocol, search, selection, data extraction, risk of bias assessment, synthesis methods and the handling of bias in interpretation, with seven critical domains whose weaknesses lower the confidence in the review's results, which is rated from high to critically low. ROBIS assesses the risk of bias in a review in three phases, assessing the relevance of the question, the identification of concerns in four domains covering eligibility, identification and selection, data collection and appraisal, and synthesis, and an overall judgment of the risk of bias in the review. The two differ in their approach, and the umbrella review chooses one in advance, with at least two assessors.

Appraisal should shape the synthesis. Reviews of low or critically low confidence can be excluded or analyzed separately, and conclusions should rely on the reviews with the strongest methods. A review that did not assess the risk of bias of its included studies, for example, cannot support strong statements about the certainty of its findings, whatever its size.

Grading the evidence

Umbrella reviews use different ways of summarizing the strength of evidence. In the health sciences, many reuse the GRADE ratings given by the review authors, where available, and report them. For umbrella reviews of associations between exposures and outcomes, with many meta-analyses of observational studies, classification systems based on statistical criteria have been used. They typically combine the number of cases, the statistical significance of the pooled estimate, the heterogeneity, the evidence of small-study effects and excess significance, and the presence of a prediction interval that excludes the null, to classify associations as convincing, highly suggestive, suggestive or weak. These criteria are useful but should be viewed with care, since they depend on thresholds and on the quality of the underlying studies, and they do not replace the assessment of the risk of bias.

Synthesis and reporting conflicts

The synthesis can take several forms. Tables arrange the reviews by question, outcome and population, showing for each the estimate, the certainty and the quality of the review. Narrative text explains the patterns. Where reviews on the same question conflict, the umbrella review looks for the reasons: differences in the included studies, in eligibility criteria, in dates, in the handling of bias and heterogeneity, and in the statistical methods. Often the conflict is explained by differences in scope, or in the inclusion of a late study, and sometimes by errors or by flawed methods in one review. Explaining discordance is one of the most useful things an umbrella review can do. Re-analysis of the primary data is sometimes needed and is a substantial extension.

The conclusions are worded at the level of the reviews' quality and the overlap, and they identify where the evidence is strong, where it is weak or conflicting, and where there are no reviews.

Reporting

The PRIOR statement, Preferred Reporting Items for Overviews of Reviews, provides a reporting guideline specific to overviews of reviews of healthcare interventions. It adds items to PRISMA about the handling of overlap, the appraisal of included reviews, and the use of the reviews' own data and risk-of-bias assessments, and about how discordant results were handled. A protocol should be written and registered where a registry accepts it. The report includes a flow diagram of the selection of reviews, a table of characteristics, the appraisal results, the overlap analysis and the evidence summary. See PRISMA extensions.

Limitations

An umbrella review inherits the limitations of the reviews it summarizes and of their primary studies, and adds several of its own. Overlap can distort the weight of evidence. Reviews vary in quality and in the way they assess bias, so results are not directly comparable. The lag between primary studies and reviews, and between reviews and the umbrella review, means that recent studies may be missed. Reviews report data incompletely, and extracting from reviews can introduce errors, particularly when reviews differ in their handling of the same study. The method has been used on very broad questions that yield summaries that are too general to guide decisions. And a review of reviews can create an impression of authority that the underlying evidence does not justify. The results are not advice about the care of individual patients.

How we can help

Frequently asked questions