Small-study effects in meta-analysis

What small-study effects are

In a meta-analysis, studies of different sizes estimate the same effect with different precision. In a world without bias and with no differences in what the studies did, the large and small studies would scatter around the same value, the small ones more widely. Small-study effects exist when the estimates of the small studies are systematically different from those of the large studies. Most often the small studies show larger benefits.

The term was introduced to name the observed pattern without presuming its cause. That matters because the same funnel plot asymmetry can come from a selection process that suppresses unwelcome small studies, or from a real difference in what small trials do, or from a feature of the measure. Calling it publication bias at the point of detection claims more than the data show.

Why small studies can differ

Several mechanisms produce the pattern, and more than one can operate at the same time.

• Publication bias. Small studies with unimpressive or unfavorable results are less likely to be written up, accepted and indexed, so those that remain are the more favorable ones. Large studies are harder to leave unpublished.
• Selective reporting within studies. Small studies may report the outcomes and analyses that came out best, and omit others. This leaves the study visible but biases what it contributes.
• Differences in methodological quality. Small trials have on average more risk of bias, such as inadequate concealment of allocation or lack of blinding, and these flaws tend to inflate effects.
• True heterogeneity. Small studies may be run in different settings, for example single specialized centers, with more selected patients or more intensively delivered interventions, so the true effect there may really be larger.
• Chance. Especially with few studies, an association between size and effect can appear without any systematic cause.
• Artifacts of the effect measure. For odds ratios and some other measures, the standard error is mathematically related to the effect estimate, which can produce asymmetry in a funnel plot with no bias at all.

Telling these apart from the data alone is hard. What helps is knowledge of the field, the risk-of-bias assessments, the registry records and the characteristics of the studies.

An example of the shift

Ten simulated studies, invented for this page, show larger effects among the less precise studies. The table compares pooled estimates calculated in different ways.

The fixed-effect estimate is 0.25, but the random-effects estimate, 0.34, is higher. The reason is that random-effects weights are flatter: adding the between-study variance (here 0.035) to every study's own variance reduces the dominance of the large studies. In the presence of small-study effects, this moves the pooled estimate toward the small studies, which are the ones that differ. The five most precise studies give 0.21, and the five least precise give 0.51. These data are simulated, but the pattern, in which the random-effects result exceeds the fixed-effect result and both exceed the large-study result, is a common signal in real data.

A difference between fixed-effect and random-effects estimates is therefore worth a look. It does not prove a problem. It does show that the choice of model matters, and that the small studies are driving part of the result.

How to examine them

No single check is enough, and the checks should be planned in the protocol.

• Funnel plot. A visual check for asymmetry, useful when there are about ten or more studies, ideally with contours showing significance.
• Asymmetry tests. Egger's regression and its variants, chosen to suit the effect measure. They have low power with few studies.
• Large versus small comparison. Pool the larger studies alone and compare with the small ones, or with the whole. Some authors look at the studies above a prespecified size or at the largest few. A consistent result across the size groups is reassuring.
• Cumulative meta-analysis by precision. Add studies from the most to the least precise and watch how the estimate moves.
• Meta-regression on standard error or sample size. A formal test of the association, which is closely related to Egger's test.
• Risk-of-bias subgroups. Compare studies at high and low risk of bias, since poor quality in small trials is a common explanation.
• Registry and protocol comparison. Check whether registered outcomes appear in the published reports, which addresses selective reporting directly.
• Search review. Ask what unpublished or non-indexed sources could exist, such as registries, regulatory documents and conference abstracts.

What to do about them

When small-study effects are present, the aim is to understand and describe them, and not to produce a corrected figure that looks authoritative. There are a few reasonable steps.

• Prefer an analysis that is less sensitive. If the pooled result changes materially between the fixed-effect and random-effects models, say so and present both, with a clear primary analysis chosen in advance.
• Run sensitivity analyses. Restrict to larger or lower-risk-of-bias studies, and apply methods such as trim-and-fill, selection models or regression-based adjustments. Describe them as sensitivity analyses with their assumptions.
• Explore explanations. Meta-regression or subgroup analyses on characteristics that differ between small and large studies, such as setting, dose and population, may show that the difference has a clinical explanation.
• Rate certainty honestly. In GRADE, publication bias is one of the five domains that can lower certainty, and a consistent pattern of small-study effects supported by a limited search is a reasonable reason to do so. The rating should rest on all the evidence, not on a test alone.
• Look further. A large, well-conducted trial that would settle the question is more informative than any adjustment of the small ones.

Why large trials sometimes disagree with meta-analyses of small ones

Empirical work has compared meta-analyses of small trials with later large trials on the same question. In a number of well-known cases, the large trial gave a smaller or null effect, and the earlier pooled small trials had overestimated the benefit. Small-study effects are one of the explanations proposed, together with differences in populations and in the quality of delivery. Not all such disagreements are due to bias, and it would be wrong to conclude that small trials always mislead. The lesson is narrower: when the evidence consists only of small studies and shows signs of asymmetry, confidence in the pooled estimate should be limited, and the conclusion should be framed as provisional. This is also an argument for preferring prospectively registered, adequately powered trials, and for reading their results beside the pooled estimates of earlier ones.

How to report them

Describe what was checked and what was found, in neutral language. For example: "The funnel plot and Egger's test suggested small-study effects. The pooled estimate was lower when restricted to the largest studies. This pattern may reflect publication bias, differences in risk of bias or heterogeneity between small and large trials." Present the funnel plot, the test, the sensitivity analyses and the effect on the certainty of evidence. Avoid stating that publication bias was detected or excluded. PRISMA 2020 asks reviews to report methods and results for assessing the risk of bias due to missing results in a synthesis.

Designing a review to limit the problem

Several choices made at the protocol stage reduce how much small-study effects can distort a review. A comprehensive search that goes beyond the main bibliographic databases to trial registries, regulatory submissions, conference proceedings and contact with investigators lowers the chance that unfavorable small studies are missing. Searching registries is especially useful because registered trials can be identified whether or not they were ever published, and the number of registered but unpublished trials can be counted and reported.

Prespecifying the primary model, the asymmetry tests and the sensitivity analyses prevents the temptation to select whichever gives the preferred result. Assessing risk of bias for each study makes it possible to examine whether the small studies are also the weaker ones. Extracting the prespecified outcome from every study, and comparing it with the registry entry, allows selective reporting to be spotted. When the evidence base is dominated by small studies, a clear statement in the protocol of how this will affect the certainty rating helps keep the conclusions consistent.

Reading the pattern with care

Three points of interpretation come up repeatedly. First, a pattern that is visible only in the smallest few studies can be driven by a single outlying trial, so check whether removing one study makes it disappear. Second, the direction matters: small studies reporting smaller benefits than large ones are unusual and point toward heterogeneity or a different mechanism, not toward suppression of unwelcome results. Third, the size of the difference should be set against the width of the interval. A modest difference with a wide interval is compatible with chance, and one that is large and consistent across several checks deserves more concern. None of these judgments can be reduced to a threshold, and they belong in a narrative that a reader can follow and challenge. It also helps to ask what a reader who wanted to act on the result would need: if the conclusion would be the same using only the large studies, the matter is of limited practical importance, and if it would reverse, that fact should appear in the abstract and not only in a supplementary table. A short sensitivity table near the main results, listing each analysis and its estimate, makes the dependence plain at a glance.

How we can help

We can examine small-study effects with funnel plots, asymmetry tests, size-restricted analyses and sensitivity methods, link them to risk-of-bias assessments and certainty ratings, and write the findings in neutral terms. [OWNER VERIFICATION REQUIRED] The relevant services are meta-analysis and risk of bias and certainty of evidence.

Frequently asked questions