Handling zero-event studies in meta-analysis

Why zero events cause trouble

Most meta-analyses of binary outcomes work on the log scale. The log odds ratio is the log of the odds in one group divided by the odds in the other, and its variance is the sum of the reciprocals of the four cell counts. If any cell is zero, the odds ratio is zero or undefined, the log is minus infinity or undefined, and the variance is infinite. The study cannot enter an ordinary inverse-variance analysis without some change.

A study with zero events in one arm and some in the other is called a single-zero study. A study with zero events in both arms is a double-zero study. They are different problems. The single-zero study contains information, because it shows that the event was seen in one arm and not the other, but the point estimate is extreme and the standard error is not reliable. The double-zero study shows that the event is uncommon in that population over that follow-up, but it says nothing about the direction of the relative effect on its own.

These situations are not exotic. They arise whenever events are rare, trials are small, follow-up is short, or outcomes are adverse events with low incidence. In safety reviews, where the rarity of the event is the reason the question is hard, they can be the majority of the included studies.

The continuity correction and its problems

The traditional remedy is a continuity correction: add a small constant, usually 0.5, to every cell of a study that has a zero. The odds ratio can then be calculated, and the study joins the analysis. It is built into many software packages as a default, which is part of the difficulty, because users may not realize that a choice has been made on their behalf.

The correction is arbitrary. It treats a study with zero events in each of 50 and 50 patients identically to one with zero events in 500 and 500, apart from the weights. It tends to bias the odds ratio toward 1, because adding the same amount to each cell moves the estimate toward no effect. It can also make a double-zero study appear to show no difference and so pull the pooled estimate toward the null. In sparse data, the choice of constant can change the pooled result noticeably, and studies that have been shown to give different answers with different constants offer no reason to prefer one answer.

Some software applies the correction to all studies when any study has a zero, others only to the study with the zero, and some allow a treatment-arm correction proportional to the size of the arm. Each choice gives a different answer. If a correction is used at all, report the constant and the rule.

Alternatives to the continuity correction

Mantel-Haenszel. The Mantel-Haenszel odds ratio pools the cell counts directly, with weights that do not depend on the variance of the log odds ratio. Single-zero studies contribute without any correction, and double-zero studies contribute nothing to either the numerator or denominator, so they are excluded automatically. It performs well with sparse data in comparisons of meta-analytic methods, and it is a sensible default for rare events when the groups are of similar size. Its limitations are that variance estimates in the presence of heterogeneity need care and that it provides a fixed-effect estimate unless extended.

Peto odds ratio. The Peto method uses the difference between observed and expected events in one arm, and it also handles single-zero studies without correction. It works well when events are rare, the treatment effect is small to moderate, and the arms are of similar size. It can be biased when the effect is large or the groups are very unbalanced, so it is not a general solution.

Generalized linear mixed models. A binomial model with a logit link, fitted to the original counts with random effects, uses the exact likelihood and avoids the correction. These models handle zeros naturally, can allow heterogeneity, and give estimates of relative and absolute effects. They are computationally heavier and need assumptions about the distribution of random effects, and they can be unstable with very few studies.

Bayesian models. A binomial likelihood with weakly informative priors handles zeros and double-zero studies in the same framework, and it reports uncertainty for the between-study variance. The prior for the variance matters when there are few studies, and the sensitivity of the result to that prior should be examined.

Exact and beta-binomial approaches. Exact methods and beta-binomial models avoid large-sample approximations. They are used less frequently and can be useful as a check.

An example

Five simulated studies compare an intervention with a control for a rare adverse event. Events and group sizes are shown for the intervention and control groups.

Study C is a double-zero study, and studies A and E have zero events in the intervention arm. Four methods give four pooled odds ratios:

• Mantel-Haenszel with no correction: 0.27.
• Mantel-Haenszel with 0.5 added to every cell of every study: 0.40.
• Peto odds ratio: 0.31.
• Inverse-variance with 0.5 added to studies containing a zero: 0.36.

All four point to fewer events with the intervention, but they range from 0.27 to 0.40. That is the difference between a large and a moderate reduction. The data are simulated and tiny, so the example is a demonstration of sensitivity and not a recommendation of any method. The lesson is that a conclusion that depends on which correction was used is not a robust conclusion.

What to do with double-zero studies

There is a continuing debate about double-zero studies. Methods that exclude them, such as Mantel-Haenszel without correction, assume that they carry no information about the relative effect. In a strict sense this is correct, since the odds ratio is undefined. But some authors point out that they do carry information, because a study with zero events in large arms suggests a low baseline risk, and the contribution to the pooled estimate depends on the model. Models that use the exact likelihood treat them in a principled way, and some simulation studies suggest that excluding them can bias the result when many studies are double-zero. Others find little practical difference.

A defensible approach is to state the policy in the protocol, to run the main analysis with a method that is robust to sparse data, and to show a sensitivity analysis that includes double-zero studies through a model-based method. Report how many studies were double-zero. For absolute risk, double-zero studies are relevant even when the relative effect is not estimable, and a summary of absolute risks can include them.

Choosing a method

No single method is best in every case. The following considerations help in practice.

• How rare are the events? For very rare events (well under one percent), Peto and Mantel-Haenszel are often reasonable defaults when the arms are balanced and the effect is not large.
• Are the arms balanced? If group sizes differ greatly, avoid Peto and consider Mantel-Haenszel or a model-based method.
• How large is the effect? Peto is less reliable for large effects.
• Is heterogeneity expected? If so, a mixed model or a Bayesian model can estimate between-study variance, which fixed-effect methods ignore.
• How many studies? With very few, even model-based methods are fragile, and a clear statement of uncertainty matters more than the choice.
• Are absolute risks needed? Models that estimate baseline risk can report absolute effects directly.

Whatever is chosen, the same principles apply: say what was done and why, and show that the conclusion survives an alternative.

Reporting

Reports should state the number of studies with zero events in one or both arms, the method used to handle them, any continuity correction and its constant, the effect measure, and the sensitivity analysis. Present the data so that readers can reconstruct the analysis, with event counts and group sizes in a table, since the pooled estimate cannot be judged without them. In a summary of findings table, state the number of events in each group where this is meaningful, and note that the certainty of evidence may be lowered for imprecision when the confidence interval is wide or the total number of events is small.

Avoid the claim that an intervention is safe because no events occurred. A finding of zero events in a small study is compatible with a risk that is not negligible. A useful rule of thumb, sometimes called the rule of three, says that if no events are seen in n participants, the upper limit of a 95 percent confidence interval for the risk is about 3 divided by n. Zero events in 50 participants is compatible with a true risk of about 6 percent.

Planning for sparse data

The best time to deal with zero events is at the protocol stage. Before any data are extracted, decide whether the review is likely to contain rare outcomes, which can often be judged from the question itself, and write down the primary method, the sensitivity methods and the handling of double-zero studies. Consider whether a risk difference or an absolute-risk summary will be reported alongside the relative effect, since relative effects can be unstable or undefined when events are few.

During extraction, record zero counts as zero and not as missing. A blank cell and a true zero mean different things, and confusing them removes a study from the analysis for the wrong reason. Check also that the denominators are the numbers at risk for the outcome and not the numbers randomized, since the choice changes both the event rate and the weights. Finally, when a trial report says that no events occurred without giving group sizes, contact the authors or use the registry entry before assuming the sizes.

How we can help

We can choose and justify a method for sparse binary data, run Mantel-Haenszel, Peto, mixed-model and Bayesian analyses side by side, handle double-zero studies explicitly, and report the sensitivity of the result. [OWNER VERIFICATION REQUIRED] The relevant services are meta-analysis and statistical analysis.

Frequently asked questions