Multivariate meta-analysis

What a multivariate meta-analysis is

A conventional meta-analysis deals with one outcome at a time. If a review concerns five outcomes, it runs five separate analyses, each ignoring the others. But outcomes measured on the same people in the same study are usually related: patients who improve on one scale tend to improve on another, and sensitivity and specificity of a test are linked through the threshold. A multivariate meta-analysis fits one model to the vector of effect estimates from each study, and estimates the pooled effects for all outcomes at once together with the covariance between them.

The model has two levels. Within each study, the estimates have a covariance matrix, with the variances of each outcome and the correlation between them induced by measuring the same participants. Between studies, the true effects vary, with a between-study covariance matrix that describes how the true effects on different outcomes move together. The pooled effects are weighted averages in which weights come from both layers.

When it is useful

The method suits several situations.

• Multiple outcomes in the same population. For example, benefit and harm, or several domains of a symptom scale, where some studies report only a subset.
• Diagnostic accuracy. The bivariate model of sensitivity and specificity is a special case, and is standard for diagnostic test reviews.
• Multiple time points. Effects at several follow-up times from the same participants are correlated.
• Surrogate endpoints. Joint modelling of effects on a surrogate and a final outcome across trials, to assess how well the surrogate predicts the final effect.
• Multiple treatment comparisons. Network meta-analysis can be written as a multivariate model, with the relative effects in a multi-arm trial correlated through the shared arm.
• Multiple effect measures from one study, such as several correlated comparisons against a shared control group.

It is not needed when a single primary outcome answers the question, when outcomes are reported by all studies, when there are few studies, or when the correlations cannot be obtained or reasonably assumed.

The two kinds of correlation

Within-study correlation arises because the same participants contribute to both outcomes. It is rarely reported in published papers, which is the main practical obstacle. It can be calculated from individual participant data, taken from an external source, such as a similar study, or estimated from the correlation in the participant-level outcomes if available. Where it is unknown, analysts may assume a value, such as 0.5, and run a sensitivity analysis over a range. Some methods avoid the need for it, for example a model with a common correlation parameter between the outcomes, or robust variance estimation.

Between-study correlation reflects whether studies with a higher true effect on one outcome also have a higher true effect on another. It must be estimated from the data, which requires enough studies. With only a handful, it is often estimated at the boundary (plus or minus one) or with great uncertainty, and the model may fail to converge. Alternatives are to simplify the model, for example by assuming a shared correlation, or to adopt a Bayesian approach with an informative prior.

A worked example of borrowing strength

Four simulated studies report an efficacy outcome, and three of them also report a second outcome. Study 3 reports only the first. The within-study correlation is taken as 0.5 where both are reported.

The univariate fixed-effect analysis gives a pooled estimate for outcome 1 of 0.427 (standard error 0.095), and for outcome 2 of 0.194 (standard error 0.113). The multivariate fixed-effect analysis, with the same within-study covariances, gives 0.427 (standard error 0.095) and 0.184 (standard error 0.106). The estimate for outcome 1 is unchanged, since all four studies already report it. For outcome 2, the standard error falls from 0.113 to 0.106, because information on outcome 1 in study 3 helps through the correlation, and the point estimate moves from 0.194 to 0.184. The data are simulated.

This is a small gain, which is typical. The benefit of borrowing strength depends on the strength of the correlation and the proportion of studies with missing outcomes. It is largest when the correlation is high and many studies lack the outcome of interest. It also carries a risk. The multivariate estimate relies on the model and on the assumed correlation, and if the correlation is wrong, the borrowed information can mislead. For this reason, results from the multivariate model are usually presented beside the univariate ones.

Fitting the model

Multivariate random-effects models can be fitted by restricted maximum likelihood, by method of moments, or by Bayesian methods. Software includes the metafor package in R (the rma.mv function), the mvmeta package in R and the mvmeta command in Stata, as well as Bayesian tools such as Stan and JAGS. Multilevel models, which treat several effect sizes from the same study as nested, are an alternative way to deal with dependence when the correlation is not known, and robust variance estimation gives valid standard errors without requiring the correlations to be correct.

Convergence can be a problem, particularly with many outcomes, since the number of between-study parameters grows rapidly: two outcomes need three, three need six, and four need ten. With few studies, the data cannot support them, and simplified structures are used. Check the estimates for implausible values, such as correlations at the boundary or tiny variances, and examine the profile likelihood where software allows.

Interpreting and presenting results

Present the pooled effects with confidence intervals for every outcome, the between-study variances and correlations, and the within-study correlation assumptions. Compare the multivariate results with univariate ones, and say where the gain was material. For decision making, joint prediction regions and the probability that a new study shows benefit on one outcome and harm on another can be informative, and a multivariate analysis lets such questions be asked, which separate analyses cannot. When the aim is to compare outcomes on a common scale, standardization requires care.

Multivariate meta-analysis does not solve the problem of selective outcome reporting. If outcomes are missing because they were unfavorable, the model assumption that data are missing at random fails, and borrowing strength can then increase bias. Compare the studies that report the outcome with those that do not, and consider sensitivity analyses.

A closer look at surrogate endpoints

One important use is the evaluation of surrogate endpoints. Trials often report an effect on a quickly measured marker, such as progression-free survival or a laboratory value, long before the effect on a clinically final outcome such as overall survival or disability is known. If the treatment effects on the surrogate predict the effects on the final outcome across many trials, the surrogate may be used in new trials, shortening them. A bivariate meta-analysis of trial-level effects on both endpoints estimates the correlation between the effects, and a regression of the final effect on the surrogate effect gives a prediction with its uncertainty. A strong trial-level association supports using the surrogate, and a weak one does not.

Published evaluations have found that associations vary considerably, and strong correlations in one setting do not necessarily carry over to another drug class or population. The analysis should also allow for the uncertainty in each trial's estimates, and for the possibility that a new treatment works through a different mechanism than the ones studied. Reporting should include the number of trials, the correlation with an interval, the predicted effect and its interval for a new trial, and a clear statement of the setting in which the surrogate was assessed.

Choosing between the options

A practical decision can follow three steps. If one outcome is primary and the others are secondary, analyse the primary one on its own and report the others separately, perhaps with a multivariate sensitivity analysis. If several outcomes are of equal interest and many studies report only some of them, a multivariate model can add information, provided there are enough studies, the correlation can be justified and sensitivity analyses show the result is not an artifact of it. If the aim is simply to handle dependence between several effect sizes from the same study, a multilevel model or robust variance estimation is often simpler and nearly as good. Writing the choice in the protocol protects against the temptation to try several and report the one that gives the preferred answer.

Limitations

The method is more complex than univariate analysis, needs information that is usually not published, and can be hard to fit and to explain. The gains are often modest, and sometimes the univariate results are nearly identical. With few studies, estimates of between-study correlation are unreliable. Reviewers and readers may find it less transparent, so the report must describe the model, the assumptions and the software. The results are for evidence synthesis and policy questions, and do not give advice for individual patients.

How we can help

Frequently asked questions