Realist review

What a realist review is

Many programs are complex. A community health program, a school reform or a new payment model is not a single pill given in a fixed dose. It is a set of activities, delivered by people in organizations, in particular social settings. The same program may succeed in one place and fail in another. A standard review of such programs can report an average effect and a measure of heterogeneity, but it does not say why the results differ.

Realist review addresses this. Pawson and colleagues proposed it in 2005 as a method for complex policy interventions, building on Pawson and Tilley's realist evaluation. Its guiding idea is that programs do not work by themselves. They offer resources or opportunities, and people respond to them in ways that depend on their circumstances. The response, the mechanism, is triggered in a particular context and produces an outcome. A realist review therefore aims to explain, in terms of context, mechanism and outcome, what works for whom, in what circumstances, in what respects and why.

It is a form of theory-driven synthesis. The product is not a pooled effect but a refined theory about how a class of programs operates, expressed so that decision makers can judge whether it applies to their situation.

Context, mechanism and outcome

The central device is the context-mechanism-outcome configuration (CMOC). A context is a feature of the setting that affects how people respond: the culture of an organization, the resources available, the characteristics of participants. A mechanism is not the activity of the program but the underlying response to it: trust, fear of sanction, a sense of fairness, a change in beliefs. An outcome is what follows, intended or not.

A simple illustration: a program gives schools feedback on pupil attendance. In schools with a culture of shared responsibility (context), the feedback prompts staff to discuss and act together (mechanism), and attendance rises (outcome). In schools with a punitive culture (context), the same feedback leads staff to hide data (mechanism), and attendance records worsen (outcome). The activity is the same in both. The explanation lies in the configuration. This example is hypothetical and is given only to show the structure.

Distinguishing context from mechanism takes care, and reviewers often find that an element can be one or the other depending on the question. The team should discuss and record how they used the terms, since the usefulness of the result depends on clear categories.

The steps of a realist review

The process is iterative and does not follow a fixed linear protocol, but a sequence of tasks is commonly described. The table sets out steps drawn from the original guidance and later training materials.

The first step is to clarify the scope and to build an initial program theory, often as a diagram or a set of if-then statements. The theory is drawn from policy documents, earlier evaluations and discussion with stakeholders. It gives the review its direction. Searching and appraisal are then tied to the theory, and the review's task is to test and refine it.

Searching and selecting evidence

A realist review does not try to find every study in a fixed list of databases. It uses iterative, purposive searching. An initial search finds sources to build and test the theory, and later searches follow leads: a program mentioned in one paper, a mechanism proposed in another, a context that needs more evidence. Sources can include trials, qualitative studies, process evaluations, case studies, policy documents and grey literature. The test is whether a source contains information about the context, mechanism or outcome in the theory.

The search must still be transparent. The reviewers should record the sources searched, the terms, the stopping point and the reason for stopping, which is usually theoretical saturation, the point at which new sources no longer change the theory. They should explain how the searches changed during the review. Realist reviews have been criticized when they offer no account of how sources were chosen, so a clear audit trail is essential.

Appraisal: relevance and rigor

Appraisal is done differently from conventional reviews. Because a realist review draws on many types of evidence, a single hierarchy of study designs is not used. Each piece of evidence is judged for relevance, meaning whether it contributes to the theory being tested, and rigor, meaning whether the method used to produce it is credible enough for the weight placed on it. A single interview study with rich information on mechanisms may be used for a mechanism claim but not for an outcome claim. A trial with a robust effect estimate may support an outcome but say little about mechanism.

Judgment is made at the level of the proposition: the same document can be strong evidence for one claim and weak for another. The team should record these judgments, and the report should be clear about which claims rest on strong evidence and which on limited evidence. This approach asks for more reasoning than a checklist does, and it makes the reviewers' judgments visible.

Analysis and synthesis

Reviewers extract fragments of evidence that bear on contexts, mechanisms and outcomes. They then ask of each piece what it shows about the theory: does it support, challenge or refine a proposed configuration? Reasoning forms include juxtaposition (setting different sources side by side), reconciliation (explaining apparent conflicts by differences in context), adjudication (judging between rival explanations) and consolidation (combining sources into a fuller account). The result is a set of refined CMO configurations, often with a diagram of the revised program theory.

Analysis is typically done by a team, with regular meetings to test interpretations. Stakeholder involvement, where the review has a panel of practitioners or policy makers, can help to check whether the emerging theory is plausible and useful. This involvement has to be reported, because it can influence the findings.

The conclusions should be written in terms of what can be said about contexts and mechanisms, with the strength of evidence for each. Realist reviews do not claim to generalize by statistics. They claim transferable theory: if a similar mechanism is triggered in a similar context, a similar outcome is plausible. Decision makers need to judge how well their setting matches.

Reporting and quality standards

Wong and colleagues developed the RAMESES publication standards for realist syntheses in 2013, with a checklist of items for the title, abstract, rationale, objectives, methods, results and discussion. Items include the program theory, the rationale for using realist methods, the search process and its changes, the selection and appraisal methods, the data extraction and the analysis and synthesis processes. Following RAMESES helps to demonstrate that the review is rigorous despite its flexible process.

Protocol registration is possible in some places, and some teams publish a protocol in a journal. Because the process is iterative, the protocol states the starting theory and the intended approach, and changes are reported in the final review. PRISMA does not fit well, although a flow diagram of the search and selection can still be useful.

Indicators of quality include a clear initial theory, a justified search, appraisal of relevance and rigor, an audit trail, explicit CMO configurations, a clear statement of the strength of evidence behind each and conclusions that follow from the synthesis. A realist review should also say how it relates to other reviews of the same program and what a conventional review could add.

When to choose a realist review

A realist review is a good choice when the intervention is complex, when previous results vary and the reason is unknown, when decision makers want to know how to implement a program locally, or when the aim is to refine a theory for later evaluation. It is a poor choice when the question is how large an effect is, when the intervention is simple and the setting matters little, or when the stakeholders need a result that others can replicate exactly. In many projects a realist review is combined with a conventional review of effects, with each answering its part of the question.

Limitations

Realist reviews rely on interpretation and on the judgment of the team, so the result depends on the reviewers' skill and cannot be fully reproduced. The distinction between context and mechanism is hard to apply consistently. Purposive searching can miss evidence, and the stopping point is a judgment. The method gives no pooled estimate of effect size. Findings are theories and are best seen as plausible explanations to be tested. The method has been criticized for variable quality in published reports, and RAMESES standards help but do not guarantee rigor. The review informs program design and policy; it is not clinical advice.

How we can help

Frequently asked questions