How to read a funnel plot

What a funnel plot is

A funnel plot displays each study in a meta-analysis as a point. The horizontal position is the study's effect estimate and the vertical position is a measure of its precision, most often the standard error, which is plotted with zero at the top so that the largest and most precise studies appear at the top and the smallest and least precise at the bottom. A vertical line marks the pooled estimate, and many plots add dashed lines showing the range within which 95 percent of studies would be expected to fall if they all estimated the same effect and differed only by chance. Those lines form an inverted funnel, which gives the plot its name.

The reasoning is simple. Large studies estimate the effect precisely and cluster near the true value. Small studies are noisy and scatter widely around it, in both directions. If the evidence is complete and the studies estimate the same effect, the points form a symmetrical funnel centered on the pooled estimate. If small studies with unfavorable or null results are missing, a part of the bottom of the funnel is empty and the plot looks lopsided. The method is a first look at the question of missing evidence, described in the guide to publication bias.

An example

In this example, invented for illustration, the largest studies at the top lie near the pooled estimate of 0.33, and the smaller studies lower down lie increasingly to the right of it. Studies with large standard errors report larger effects, and there is a conspicuous gap at the lower left, where small studies with small or null effects would be expected. The correlation between the standard error and the effect is 0.91. 0 of the 14 studies fall outside the dashed limits. The picture is a classic one for small-study effects. It does not say why the pattern arises, and the sections below set out the possible explanations.

Choosing the vertical axis

The vertical axis is a measure of study size or precision, and the choice matters. The standard error is generally recommended, because it is directly related to the scatter of estimates and gives the funnel its triangular shape. Alternatives include the inverse of the standard error, the variance, the sample size and the inverse variance, and each produces a different shape. Plotting against the sample size alone can mislead, as it ignores differences in the precision of studies of the same size. For ratio measures, the horizontal axis is on the log scale, so that the funnel is symmetric. For some measures, such as the log odds ratio, the standard error is mathematically related to the effect size even without bias, which can create apparent asymmetry, and for these the choice of axis and the choice of test needs care.

What can cause asymmetry

Publication bias

Small studies with non-significant or unfavorable results are less likely to be published, leaving a gap at the bottom of the funnel on the side of the unfavorable results.

Selective outcome reporting

Small studies report only favorable outcomes, so that the effect for the outcome analyzed is inflated in them.

Poor methodological quality of small studies

Small studies are more often at high risk of bias, such as lack of concealment or blinding, which exaggerates effects.

True heterogeneity

Small and large studies may differ in population or intervention. A treatment given more intensively in small trials may have a larger effect there, and the plot shows asymmetry that is real.

Artifact of the effect measure

For some measures, the standard error is correlated with the effect size by construction.

Chance

With few studies, asymmetry can arise by chance alone.

For these reasons the pattern is called a small-study effect, and the more neutral term is used in the statements of the review. The guide on small-study effects discusses them in more detail.

Contour-enhanced funnel plots

A contour-enhanced funnel plot adds shaded bands that show the regions of statistical significance, for example the areas where the study p-value is below 0.05 and 0.01. It helps to distinguish between the explanations. If the empty region of the funnel is in the area where studies would be non-significant, publication bias, in which non-significant studies are suppressed, becomes a more plausible explanation. If the missing studies would have been in an area of high significance, then other explanations, such as heterogeneity or quality, become more likely, since publication bias would not have removed significant results. The device does not prove the cause, but it makes the plot more informative and is recommended when assessing asymmetry. The method was proposed by Peters and colleagues.

Tests of asymmetry

Visual impressions of funnel plots are subjective, and studies have found that readers disagree on whether a plot is symmetric. Statistical tests offer an objective supplement. The best known is Egger's regression test, described in Egger's test, which regresses the standardized effect on precision. The Begg rank correlation test is an older alternative with lower power. For binary outcomes with odds ratios, modified tests, such as those of Harbord and Peters, avoid the false positives that arise from the link between the effect and its standard error. Tests have low power with few studies, and a non-significant test does not exclude bias. They are recommended only when there are at least about ten studies and when the studies are not all of similar size, since a plot with studies of similar precision has no funnel to examine. A significant test, like a lopsided plot, is a signal to investigate.

What to do about asymmetry

1. Check the data

   Extraction errors and unit mistakes in the small studies are an easy explanation to rule out.

2. Look for explanations

   Compare the small and large studies for differences in population, intervention, risk of bias and outcome definition, using subgroup analysis or meta-regression.

3. Search for missing studies

   Look in trial registries, regulatory documents and other unpublished sources for studies that were not found.

4. Run sensitivity analyses

   Compare the pooled estimate with and without the smaller studies, and with methods such as trim-and-fill or selection models, as sensitivity analyses. See trim-and-fill.

5. Reflect it in the conclusions

   If the possibility of bias remains, lower the certainty of evidence for publication bias, and word the conclusion with caution. See GRADE.

When a funnel plot is not appropriate

A funnel plot is uninformative when there are too few studies, since a handful of points cannot show a pattern, and the usual recommendation is a minimum of about ten. It is also of little use when the studies are of similar size, because the plot then has no vertical spread, or when heterogeneity is large, because the scatter may reflect real differences and not bias. In reviews of diagnostic accuracy, the standard funnel plot is misleading because the sample size is related to accuracy for reasons unconnected with bias, and a modified plot is used instead. For proportions and prevalence, the standard error is mathematically linked to the proportion, which produces apparent asymmetry. In each case, the review should say that the plot was not appropriate, and use other approaches to consider reporting bias.

Funnel plots in other settings

The plain funnel plot assumes one pooled estimate and one comparison. In a network meta-analysis, where many comparisons are combined, a comparison-adjusted funnel plot is used: each study's estimate is centered on the pooled estimate of its own comparison, and the comparisons are oriented in a common direction, for example by ordering treatments so that the newer one is always first, which allows asymmetry to be assessed across the network. In meta-analyses with a continuous moderator or several outcomes, plots can be drawn for each. For meta-analyses of proportions, the standard plot is misleading, as noted above. And for studies of diagnostic accuracy, a plot of the diagnostic odds ratio against a function of the effective sample size, with the associated test, is the recommended form. In each case the principle is the same: the plot shows how study results relate to their precision, and the task is to read the pattern with the right explanations in mind. See network meta-analysis.

Reporting the assessment

A review that assesses reporting bias should say which plot and test were used and how many studies they were based on, show the plot, report the test result with its estimate and interval, and interpret it, saying what explanations were considered. If the plot or test could not be done because of too few studies, the review states that and describes any other methods used to consider bias, such as searches of registries. The conclusion about the likelihood of bias feeds into the certainty of the evidence. PRISMA 2020 includes items for the methods and results of the assessment of reporting biases. Authors who treat the funnel plot as a formality, and do not interpret it, miss its main benefit, which is to prompt the right questions about the evidence. See PRISMA 2020.

Common mistakes

• Reading asymmetry as proof of publication bias.
• Drawing a funnel plot from a handful of studies and interpreting it.
• Using the sample size instead of the standard error for the vertical axis without noting the consequences.
• Relying on visual inspection alone, without a test or a second reader.
• Ignoring true heterogeneity as an explanation.
• Applying the standard funnel plot to diagnostic accuracy or prevalence data.
• Showing the plot but not discussing it in the interpretation.

Support

Frequently asked questions