Living systematic review

What a living systematic review is

A conventional systematic review is a snapshot. It is searched at one time, published months or years later, and begins to age at once. Some reviews are updated every few years, if resources allow. A living systematic review changes the model: it is searched and updated continually, and new studies are added soon after they appear, so that the review at any date reflects the evidence at close to that date.

The idea was proposed by Elliott and colleagues in 2014 and tested widely during the COVID-19 pandemic, when evidence on treatments and prevention changed within weeks. The Living Systematic Review Network and Cochrane have published guidance. A living review remains a systematic review in every other respect: it has a protocol, a documented search, explicit criteria, risk of bias assessment and a transparent report. What is different is the workflow after the first publication, and the commitment to maintain it.

When a living review is justified

Maintaining a living review takes sustained work, so it should be used selectively. Elliott and colleagues proposed three conditions, which many groups have adopted.

• The question is a priority for decision making. A guideline, a policy or a research agenda depends on the answer.
• The certainty of the existing evidence is low or very low. If evidence is already conclusive, more studies are unlikely to change it. Where certainty is low, new evidence could shift conclusions.
• New evidence is emerging quickly. A pipeline of trials, as with a new class of drug or a new public health threat, makes updating worthwhile.

If these conditions are not met, a conventional review with a plan for periodic updating is usually better. A review that is announced as living but cannot be sustained gives readers a false impression of currency, and is worse than an honest dated review.

The workflow

The main elements of a living review differ from a one-off review in the following ways.

A living review needs a standing team with defined roles, a screening and extraction process that can handle small batches repeatedly, and software that keeps decisions and data in order. Many groups use automation to reduce effort, such as machine-learning classifiers to prioritize or filter records, and tools that maintain the study database. These tools need validation, and the use of automation should be reported.

The frequency of searching and of analysis can differ. A team may search weekly or monthly but update the analysis only when a new eligible study has been found, or at fixed intervals. The protocol should state both, and say what will count as an update to the published version and what will be a minor revision.

Publication models

Traditional journals publish a fixed article, which does not fit a review that changes. Several models have been used.

• A living online version with archived snapshots. The current version is on a website or platform, with each update dated, and numbered versions are kept. This is the model of the Cochrane Library for living Cochrane reviews.
• A series of journal updates. The first version is published as a paper and each major update is published as a new article or as an addendum, sometimes with a link to the living platform.
• A preprint with versions. Updates are posted as new versions of a preprint, with peer review at selected points.
• An evidence platform or dashboard. An interactive resource updated in real time, which is used by guideline groups and health agencies.

Whatever the model, the update history must be transparent, with dates, changes to the studies included, changes to the results and any change in conclusions. Citations should identify the version used.

Statistical issues

Updating a meta-analysis repeatedly, and examining the result each time, is repeated significance testing on accumulating data. Left unaddressed it raises the chance of a false-positive conclusion and of overestimating effects early in the history of a question. The problem is the same as that in sequential clinical trials, and similar solutions have been proposed.

• Frequentist sequential methods. Trial sequential analysis and related approaches compute a required information size and monitoring boundaries that are stricter at early updates.
• Bayesian updating. Posterior probabilities are updated naturally as data accrue, and are not subject to the same type I error issue, though the choice of prior and the decision rules still need to be explicit.
• Prespecified decision rules. The protocol can say in advance what size and certainty of effect will count as a change of conclusion, and the conclusion is not altered by a result that crosses a threshold once.
• Attention to heterogeneity. Early updates have few studies and unstable variance estimates, so random-effects intervals can be misleading, and the Hartung-Knapp adjustment or Bayesian methods with careful priors may be needed.

The statistical plan should be written before the first search, since adopting a method after seeing the updates opens the door to selective choices.

Certainty of evidence and conclusions

Each update requires the certainty of evidence to be reviewed, since new studies can change risk of bias, inconsistency, imprecision and the assessment of publication bias. If recommendations or guidelines depend on the review, a clear process is needed for deciding when a change in the evidence requires a change in the recommendation. Living guidelines, which are paired with living reviews, have developed such processes, in which an expert panel reviews each substantial update. A living review should state what it will do when the conclusions change, and who will be informed.

Reporting and retirement

PRISMA 2020 includes guidance for updated reviews, including a flow diagram that distinguishes studies identified in the previous version from those identified in the update. Each version should report the dates of the searches, the number of new records and studies, the changes to results and conclusions and the reasons for any change in methods. The protocol and registration, for example in PROSPERO, should be updated and the status noted as living.

A living review also needs an exit. Criteria for retiring it may include evidence becoming conclusive and unlikely to change, the question losing priority, a loss of new studies, or a loss of resources. When it is retired, the final version should say so, give the date and state how the evidence stood at that time, so that readers do not mistake a stale review for a current one.

Resources and sustainability

Living reviews are costly in staff time, and many have ended for lack of funds. Before starting, estimate the volume of new records, the time for screening and extraction, the cost of statistical updating and the cost of publication. Decide who will do the work if a key person leaves. Funders and journals vary in how they support continuing work, and some require plans for sustainability. A review that begins as living and moves to a regular update schedule is a reasonable compromise, provided the change is reported.

Lessons from the pandemic

The COVID-19 period was the largest test of the model, and the experience is instructive. Many living reviews were started, some very quickly, and a number of them were abandoned as the flow of studies slowed or funding ended. Others produced updates that fed directly into guidelines. Common lessons were that data quality suffered when volume exceeded capacity, that the inclusion of preprints raised questions about changes between versions and retractions, and that duplication was widespread, with several groups reviewing the same question in parallel. Collaboration between groups, shared data on included studies and common platforms reduced duplication. For anyone planning a living review, the lesson is to check what already exists, to share the work where possible, to set up automation with validation and to write down the retirement criteria at the start, when it is easy to be realistic.

What an update looks like

Each update follows a fixed routine. The team runs the search and exports new records, removes duplicates against the master list, screens titles and abstracts, retrieves full texts and applies the same criteria as before. Included studies are extracted and assessed for risk of bias, entered into the analysis dataset and the meta-analysis is rerun with the same code. The results and the certainty ratings are compared with the previous version. If nothing has changed, the update is recorded as a search with no new eligible studies, with the date. If a new study has been added, the report is revised, the changes are described in a short summary at the top and the version number is incremented. A decision is taken, according to the rules in the protocol, on whether the change is large enough to be flagged to the users of the review, such as a guideline panel, who may then need to reconsider a recommendation.

Limitations

Living reviews can create false confidence if the evidence is changing but the review is not kept current in practice. They cost more than conventional reviews, they raise statistical issues that are not always handled, and the changing text can confuse readers and citation. Rapid updating can compromise quality when screening and extraction are rushed, and automation may introduce errors. The question needs to be narrow enough for sustained updating. Finally, updated results do not provide clinical advice for individual patients.

How we can help

Frequently asked questions