Choosing priors in Bayesian meta-analysis

What the prior does

In Bayesian analysis, the prior distribution expresses what is known or assumed about a parameter before seeing the data. The data enter through the likelihood, and the two combine to give the posterior distribution, which is the basis for all conclusions. With a lot of information in the data, the choice of prior matters little. With little information, which is the usual situation in a meta-analysis of a handful of studies, the prior can matter a great deal.

Bayesian meta-analysis is attractive for several reasons. It handles few studies without the large-sample approximations of conventional random-effects methods, it carries the uncertainty about heterogeneity through to the pooled effect, it gives direct probability statements, and it can incorporate external information. The price is that every choice of prior is a modeling choice that has to be justified, and an unexamined default can mislead.

Priors for the average effect

The usual choice for the average effect, often denoted mu, is a normal distribution centered on no effect, with a large variance, which makes it vague relative to the data. On the log scale for a ratio measure, a standard deviation of 1 is already weakly informative, because it covers risk ratios from about 0.14 to 7 within two standard deviations, and values larger than that are implausible for most interventions. A standard deviation of 10 or more is nearly flat and is used when the analyst wants the data to dominate.

Where there is good reason, a more informative prior can be justified, for instance a skeptical prior centered at zero with a narrow spread, representing doubt about large effects, or an enthusiastic prior. The example shows the effect on a simulated pooled estimate of 0.40 with standard error 0.20.

With the vague and weakly informative priors the answer is essentially the data. The skeptical prior pulls the estimate down to 0.24. Neither answer is wrong. They answer different questions, and the skeptical analysis is useful when the question is how strong the evidence must be to overcome prior doubt. The point is to choose the prior for a reason and to say what it was.

Priors for heterogeneity

The between-study standard deviation, tau, is the parameter that drives the width of the random-effects interval, and with few studies the data say little about it. A prior that is nominally vague can place a lot of weight on very large values, producing a wide posterior for tau and an implausibly wide interval for the effect. A prior that is too tight can understate heterogeneity.

The table shows a small simulation: four studies with effect estimates [0.1, 0.45, 0.3, 0.7] and standard errors [0.15, 0.2, 0.18, 0.25]. The posterior median and 95th percentile for tau were computed on a fine grid under five priors.

The median varies from 0.15 to 0.25, and the 95th percentile from 0.39 to 1.08. That is a large difference for a single choice, and the upper tail is where it shows most. With four studies the data cannot pin down tau, so the prior fills the gap. This is why it is essential to examine more than one prior and to report the range.

Commonly used priors for tau include the half-normal and half-Cauchy distributions, suggested by Gelman for hierarchical models, and uniform priors on tau, which can be informative in disguise when the upper bound is chosen carelessly. Friede and colleagues examined half-normal priors for meta-analyses of few studies and recommended weakly informative choices, with scales selected according to the effect measure. Inverse-gamma priors on the variance, once popular, can behave poorly when heterogeneity is small, and are no longer generally advised.

Informative and empirical priors

Instead of a generic weakly informative prior, one can use empirical evidence about typical heterogeneity. Turner and colleagues, and Rhodes and colleagues, derived distributions for between-study variance from large collections of meta-analyses in the Cochrane Database, classified by outcome type and by the kind of comparison. Their log-normal distributions can be used as informative priors, which is especially valuable when the review has only a few studies. Using them requires a match between the review and the data from which they were derived, in outcome type, intervention type and effect measure, and a statement of the source.

Informative priors for the effect itself can come from earlier trials or reviews. This is an appropriate use, provided that the earlier evidence is not also included in the data, which would count it twice, and provided that its relevance is argued. Expert elicitation, with a documented process, is another source. For regulatory and decision settings, there are structured methods for eliciting and checking priors.

Sensitivity analysis for priors

A prior sensitivity analysis fits the model with several reasonable priors and compares the posterior of the key quantities: the pooled effect, tau and the prediction interval. If conclusions agree, the prior does not drive the result. If they differ, the report must say which conclusions depend on the prior and why the primary choice was made. Some practical guidance follows.

• Specify the primary prior in the protocol, before seeing results, with the reasons.
• Choose two or three alternatives that span reasonable beliefs, including one more informative and one less.
• Compare the prior and posterior of tau on one plot. If they overlap heavily, the data contribute little.
• Check the scale. A half-normal prior with scale 1 means something very different on a log odds ratio scale from a standardized mean difference.
• Do not tune the prior to get a desired answer. That makes the analysis non-credible.

Practical points

Software defaults deserve attention. Packages differ in their default priors, and some defaults were designed for other settings. Always state the prior explicitly in the code and report it, along with the parameterization used, since the same name can mean a standard deviation in one package and a precision in another. Check convergence with several chains, using R-hat and effective sample size, and look for divergent transitions in Hamiltonian Monte Carlo software, since a hierarchical model with a heterogeneity parameter near zero can sample badly with a naive parameterization. A non-centered parameterization often helps. Posterior predictive checks and a posterior predictive distribution for a new study give a useful view of what to expect in practice, and the Bayesian prediction interval is the natural analogue of the frequentist one.

Finally, interpret probabilities carefully. A statement such as "there is a 95 percent probability that the effect is beneficial" is conditional on the model and the prior. It is not a statement about the world independent of those choices.

Reporting

Report the model, the priors for every parameter with their parameters and the reasoning, the software and version, the number of chains, iterations and burn-in, convergence diagnostics, the posterior summaries with credible intervals and the prediction interval, and the prior sensitivity analyses. Reporting guidance for Bayesian analyses, such as the Bayesian analysis reporting guidelines proposed by Kruschke, asks for these elements, and journals increasingly expect them. Share the code so that others can reproduce the analysis.

What to expect with very few studies

With two or three studies, no prior can make the data informative about heterogeneity. A Bayesian analysis will not invent information. It will transparently combine whatever the prior implies with the little that the data contain, and the posterior for tau will resemble the prior. That is a feature, since it shows honestly how little is known, but it makes the choice of prior part of the result. In such a case the report should present the effect estimate under the primary prior together with at least one alternative, state the posterior for tau and the prediction interval, and avoid statements of certainty. Conventional random-effects methods in this situation tend to estimate tau as zero or as very large, with a narrow interval that conveys false confidence, and this is a good reason for the interest in Bayesian methods. It is not a reason to treat the Bayesian interval as certain. A helpful device is to plot the prior and the posterior for tau on the same axes: where the two curves almost coincide, the data contributed nothing, and readers can see this at a glance without reading any numbers. Another is to report how the prediction interval, which depends on tau, changes across the alternative priors, since decisions about applying the result to a new setting depend on that interval more than on the average effect.

Common mistakes

• Accepting software default priors without looking at what they imply.
• Using an inverse-gamma prior with very small parameters as if it were non-informative.
• Applying a prior designed for one effect scale to another, for example a log odds ratio prior to a standardized mean difference.
• Choosing the prior after seeing which gives the preferred result.
• Using earlier evidence as a prior and also including it in the data.
• Reporting posterior probabilities without saying they depend on the model and prior.
• Skipping convergence checks and posterior predictive checks.

How we can help

We can specify and justify priors, fit Bayesian meta-analysis and network models in Stan, JAGS or R, run convergence checks and prior sensitivity analyses, and report results transparently with shared code. [OWNER VERIFICATION REQUIRED] The relevant services are statistical analysis and meta-analysis.

Frequently asked questions