Network meta-analysis service

What a network meta-analysis is

A conventional pairwise meta-analysis compares two interventions, for example a drug against a placebo. In practice, patients and clinicians usually choose among several options, and the trials that exist rarely compare all of them directly. A network meta-analysis combines the whole set of trials into one analysis. It uses direct comparisons, where the trials compared two treatments head to head, and indirect comparisons, where two treatments are linked through a common comparator. If A was compared with C in some trials and B with C in others, the difference between A and B can be estimated indirectly through C.

The method gives an estimate of every treatment against every other, with confidence or credible intervals, and a way of summarizing which treatments tend to perform better. It can use evidence that pairwise analysis would discard, and it can make comparisons for which no head-to-head trial exists. Its strength is also its risk, because indirect evidence is only valid under assumptions that need to be examined and that cannot be proved. The method page on network meta-analysis sets out the general approach.

When a network meta-analysis is appropriate

A network meta-analysis suits a question of the form: among these treatments, which works best for this outcome in this population. Four conditions make it worth doing.

• Three or more interventions are relevant to the same clinical question, and decision makers need to compare them.
• The trials form a connected network, so that every treatment is linked to every other through some chain of comparisons.
• The trials are similar enough in populations, outcome definitions and treatment settings that the indirect comparisons are credible.
• Enough trials exist. A network with few trials per comparison gives imprecise estimates and weak tests of inconsistency.

It is the wrong choice when there are only two options, when the trials do not connect, when the patients in the different comparisons are very different, or when a few small trials would be asked to carry a large network. In those cases, a pairwise meta-analysis, a narrative synthesis or a more cautious subset analysis is better. A feasibility check on the included trials, before committing to the method, establishes which situation applies.

What the service includes

The network and its geometry

The first step is to draw the network. Each treatment is a node, and each direct comparison is an edge, with its thickness showing the number of trials or participants. The picture shows at a glance which comparisons are well supported, which rest on a single trial, and which are only indirect. A network dominated by comparisons against placebo, with few head-to-head trials, will give most of its information about active treatments through placebo, which makes the transitivity assumption especially important.

Defining the nodes needs thought. Lumping different doses, formulations or co-interventions into one node can hide real differences, and splitting them into many nodes spreads the evidence thin. Whether treatments are grouped is a clinical decision that is made, and justified, before the analysis. Multi-arm trials, which contribute several comparisons that are correlated because they share a control group, must be handled correctly in the model, and ignoring this dependence gives intervals that are too narrow.

Transitivity and consistency

Two related assumptions underlie the method. Transitivity is the clinical and methodological assumption that the trials making different comparisons are similar enough, in the characteristics that modify treatment effects, that an indirect comparison is valid. Put simply, the patients in the A versus C trials could in principle have been randomized to B. If the trials of A and of B enrolled very different patients, the indirect estimate for A versus B will be biased by the difference, however well the model fits.

Consistency is the statistical counterpart: the direct and indirect evidence for a comparison agree. It can be tested where a comparison has both kinds of evidence, using approaches such as comparing direct and indirect estimates in each loop of the network, node-splitting, and global tests such as the design-by-treatment interaction model. These tests have low power and a non-significant result is not proof that the assumption holds. Transitivity should therefore be examined first, by comparing the distribution of potential effect modifiers across the comparisons, and inconsistency testing treated as a check and not as a guarantee. See the guide on transitivity and inconsistency.

Statistical models

Network meta-analysis can be carried out in a frequentist or a Bayesian framework, and they usually give similar answers when the same assumptions are made. Frequentist approaches, often based on a graph-theoretical or multivariate meta-analysis formulation, are quick and report confidence intervals. Bayesian approaches, usually fitted by Markov chain Monte Carlo, are flexible with complex networks and sparse data, give direct probabilities for rankings and allow prior information, at the cost of needing convergence checks and a considered choice of priors. See Bayesian meta-analysis.

Most analyses use a random-effects model, because the trials differ. A common assumption is that the between-trial variance is the same for all comparisons, which makes the estimate from sparse comparisons more stable but should itself be questioned. The effect measure is chosen as for any meta-analysis, with risk ratios, odds ratios, hazard ratios or standardized mean differences according to the outcome. Where an effect modifier is thought to differ across comparisons, network meta-regression can adjust for it, although it needs enough trials to be informative.

Treatment rankings and how to read them

Network meta-analysis outputs often include a ranking of the treatments. The usual summaries are the surface under the cumulative ranking curve (SUCRA) in Bayesian analyses and the P-score, its frequentist analogue. Each summarizes, on a scale from zero to one, the extent to which a treatment is certain to be better than the others.

Rankings are attractive and easy to misuse. A ranking is not an effect size: a treatment can be ranked first while being only trivially better than the second, and with wide intervals the ranking can be very uncertain. Rankings also ignore the certainty of the evidence, so a treatment ranked first on the strength of a few small, biased trials is not necessarily the best. For these reasons the ranking should be shown together with the relative effects and their intervals, and the interpretation should rest on the effects and on the certainty of the evidence and not on the ranking alone. See the guide on SUCRA and rankings.

Heterogeneity, small-study effects and sensitivity analyses

The usual concerns of a pairwise meta-analysis apply to every comparison in the network, and some are harder to assess. Heterogeneity is estimated for the network as a whole and for individual comparisons, and prediction intervals describe the uncertainty in a future trial. Small-study effects, including publication bias, are assessed using comparison-adjusted funnel plots, which are interpreted with care because the number of trials per comparison is often small.

Sensitivity analyses test whether the conclusions depend on analytic choices: excluding trials at high risk of bias, using a different model or heterogeneity assumption, changing how treatments are grouped, or removing a node or a comparison that drives the results. The aim is to show how robust the findings are, not to find the most favorable analysis.

Certainty of the evidence

The certainty of the evidence for each comparison is assessed separately, because it differs: one comparison may rest mostly on direct evidence from large, well-conducted trials and another entirely on indirect evidence from small ones. Approaches designed for networks extend the GRADE framework. The CINeMA approach, for example, evaluates within-study bias, reporting bias, indirectness, imprecision, heterogeneity and incoherence for each estimate and combines them into a rating. The results are presented with the relative effects so that readers see both the size of an effect and the confidence that can be placed in it.

Reporting

A network meta-analysis is reported against PRISMA 2020 together with its extension for network meta-analyses, which adds items about the geometry of the network, the assessment of inconsistency, the effect measures and the presentation of results. The report includes the network diagram, a description of how treatments were grouped, the model and its assumptions, the software, and the results for all comparisons, usually as a league table with the certainty of each estimate. The analysis code and the data are provided so that others can reproduce the results. See PRISMA extensions.

Deliverables

Optional extension: full manuscript and submission

Limitations

A network meta-analysis can answer questions that no single trial addresses, but it relies on assumptions that cannot be verified from the data. Indirect evidence is observational in nature, because patients were not randomized between the treatments being compared indirectly, so it is vulnerable to confounding by differences between trials. Sparse networks give imprecise estimates, and tests for inconsistency have limited power. Rankings can mislead when they are read without the effect sizes and the certainty. Differences between trials in doses, background therapy and outcome definitions are common, and decisions about grouping treatments are partly subjective.

For these reasons results should be interpreted with the certainty of the evidence in mind and treated as support for decisions, not as a replacement for judgment. The results of a network meta-analysis are not advice about the care of an individual patient.

Frequently asked questions