The trim-and-fill method

What trim-and-fill does

Trim-and-fill was proposed by Duval and Tweedie in 2000 as a method for dealing with funnel plot asymmetry. It starts from a simple picture. If the funnel plot would be symmetric in the absence of publication bias, then asymmetry means that some studies on one side are missing. The method estimates how many are missing and where they would lie, adds them, and recalculates the pooled estimate.

It has two products. One is an estimate of the number of missing studies. The other is an adjusted pooled effect that allows for those studies. Both depend on assumptions that cannot be tested from the data, which is why the method has been widely used and widely criticized in the same breath.

The algorithm in plain terms

The procedure works in iterations.

1. Estimate the center. Calculate the pooled effect from all studies. This is the starting guess for the point around which the funnel should be symmetric.
2. Rank the studies. Rank the studies by the distance of their effect from the center, and identify those on the side where the plot is overpopulated. These are the studies without a counterpart on the other side.
3. Estimate the number of missing studies. Use a rank-based estimator to decide how many of the extreme studies have no mirror image. The estimators called L0 and R0 are the ones usually offered.
4. Trim. Temporarily remove that number of the most extreme studies and recalculate the center from the remaining ones.
5. Repeat. Rank again around the new center and update the number to trim, until the number stops changing.
6. Fill. Add back the trimmed studies together with a mirror image of each, reflected around the final center, with the same standard error. Recalculate the pooled effect from the complete set.

The software reports the number of studies filled in, and usually the adjusted estimate with its interval. The funnel plot is redrawn with the imputed studies marked in a different style, so that readers can see which points are real and which are imputed.

A worked example

The ten simulated studies used in the guide to Egger's test have a fixed-effect pooled estimate of 0.27. The funnel plot is lopsided, because the small studies report larger effects. Trim-and-fill with the L0 estimator and fixed-effect pooling suggests that 4 studies are missing.

After trimming, the center of the remaining studies is 0.24. Each trimmed study is reflected around this value, which produces the imputed effects above. With the 4 imputed studies added, the pooled estimate is 0.24, lower than the unadjusted 0.27. The data are simulated. The numbers show how the method works and do not describe real evidence.

Notice what the adjusted estimate depends on: the choice of estimator, the pooling model, and the assumption that the missing studies are exact mirror images of the extreme ones. Change any of these, and the numbers change.

What the method assumes

Trim-and-fill rests on several assumptions that need to be stated when it is used.

• The funnel would be symmetric without bias. If the true effects vary with study size for reasons other than publication, such as differences in populations or interventions, the method will find missing studies that do not exist.
• The mechanism is suppression of small studies with unfavorable results. Other mechanisms, such as selective reporting of outcomes within published studies, do not produce missing studies in the funnel in the same way.
• Missing studies mirror the observed ones. The imputed studies take the same standard errors as the trimmed ones, and their effects are reflections. Real missing studies may look different.
• Studies are independent and the pooled model is appropriate. The choice between fixed-effect and random-effects pooling changes the center and the result.

When these hold approximately, the method gives a useful indication. When they do not, it can mislead in either direction.

Criticisms and performance

Simulation studies have examined how well trim-and-fill performs, and the findings are mixed. The method can detect and partly correct for bias when its assumptions hold, but it can also adjust data that are not biased and, in the presence of substantial heterogeneity, it may add studies where none are missing. It tends to under-correct when the true bias is strong, since the number of missing studies is estimated conservatively. Its results are sensitive to the choice of estimator and to whether a fixed-effect or random-effects model is used for the centering.

Another criticism is conceptual. The method presents an imputed number as though it were an estimate of a real quantity. Real publication bias depends on the selection process, which is not observed. The adjusted estimate is conditional on a particular assumed process. Because of this, the Cochrane Handbook and many methodologists describe trim-and-fill as a sensitivity analysis. It shows how the pooled result would change if the funnel were made symmetric. It does not recover the true effect.

For these reasons, trim-and-fill is best used with few expectations. A small change in the estimate after adjustment is reassuring about the robustness of the result. A large change is a reason for concern, not a corrected answer.

Alternatives and complements

Several methods are used alongside or instead of trim-and-fill. Egger's regression and its variants test for asymmetry. Contour-enhanced funnel plots show whether the region where missing studies would lie is one of low statistical significance. Selection models estimate the probability of publication as a function of the p value. Regression-based adjustments, such as the PET-PEESE family and limit meta-analysis, project the effect to a study of infinite precision. Each makes different assumptions, and each can disagree with the others. When several methods agree, the conclusion is more secure. When they disagree, that disagreement is itself informative.

The most effective protection against publication bias is a thorough search that includes trial registries, conference abstracts, regulatory documents and contact with investigators. No statistical method restores studies that were never found.

How to report it

Report the trim-and-fill analysis as a sensitivity analysis and name every choice: the estimator (L0 or R0), the side on which studies were trimmed, the pooling model, the number of studies imputed, and the adjusted estimate with its confidence interval beside the unadjusted one. Show the funnel plot with imputed studies visibly marked. State the interpretation with caution, for example: "Trim-and-fill suggested that four studies might be missing, and the adjusted estimate was lower than the unadjusted estimate. This is a sensitivity analysis and depends on the assumption that asymmetry is caused by missing studies."

Use the result in the assessment of the certainty of evidence. If adjustment changes the conclusion, consider downgrading for publication bias. If it does not, state that the result was robust to this analysis, without claiming that publication bias is absent. Do not replace the primary estimate with the adjusted one without a prespecified reason.

Common mistakes

• Presenting the adjusted estimate as the true effect.
• Running the method with fewer than about ten studies, where asymmetry is hard to judge.
• Ignoring heterogeneity, which can produce asymmetry that is not caused by missing studies.
• Reporting only the number of imputed studies without the adjusted estimate and interval.
• Choosing the estimator or the model after seeing which gives the preferred result.
• Treating a non-significant change in the estimate as proof that there is no publication bias.

Fixed-effect or random-effects pooling

The pooling model is one of the choices that can change the answer. With a fixed-effect model, large studies dominate the center, so the center sits close to the most precise studies. A random-effects model gives small studies relatively more weight, which moves the center toward the small studies, and so toward the side where the asymmetry lies. The estimated number of missing studies and the adjusted estimate can both differ as a result.

Some authors recommend the fixed-effect version because it anchors the center on the large studies, which are the least likely to be affected by selective publication. Others use the random-effects version because it matches the model of the primary analysis. There is no agreed rule. The practical advice is to follow the model of the primary analysis unless there is a reason not to, to report the other as a check, and to say plainly that the two can differ.

Reading the filled funnel plot

A redrawn funnel plot is easy to over-interpret. The imputed points are not data. They sit exactly opposite the trimmed studies, at the same height, because the algorithm placed them there. They will always make the picture look more symmetric, which is what they were created to do. A reader should look at three things. The first is how many points were added relative to the number of real studies, since one or two added points are a different matter from a third of the total. The second is whether the added points fall in a region where non-significant results would be expected, which a contour-enhanced plot makes visible. The third is how far the pooled estimate moves, with its interval, and whether the conclusion would change.

If the added points fall in an area of high statistical significance, publication bias is a less convincing explanation, since studies with significant results are not usually the ones that are suppressed. In that case, the asymmetry points toward other causes, such as heterogeneity, and trim-and-fill is likely to be inappropriate.

A short decision guide

1. Check that there are enough studies, about ten or more, and that they differ in size.
2. Look at the funnel plot and consider causes of asymmetry other than publication bias.
3. Run an asymmetry test suited to the effect measure.
4. If asymmetry is present, run trim-and-fill as one of several sensitivity analyses, with the settings prespecified.
5. Compare the adjusted and unadjusted estimates and decide whether the conclusion is robust.
6. Describe the result in the discussion and in the certainty-of-evidence assessment, with its limitations.

How we can help

We can run trim-and-fill alongside asymmetry tests, contour-enhanced plots and selection models, report each with its assumptions, and describe the effect on the certainty of evidence. [OWNER VERIFICATION REQUIRED] The relevant services are meta-analysis and risk of bias and certainty of evidence.

Frequently asked questions