Evidence synthesis in the social sciences

Evidence synthesis in the social sciences

Policy makers and practitioners in justice, welfare, migration, housing and governance make decisions that affect many people, and they want to know what the evidence says. The social sciences have built their own review institutions. The Campbell Collaboration, founded in 2000, produces systematic reviews of the effects of social interventions, and the EPPI-Centre in London has developed methods for reviews that combine different kinds of evidence. Petticrew and Roberts wrote a widely used guide to systematic reviews for the field, and Gough, Oliver and Thomas edited a standard textbook.

The social sciences differ from clinical research in four ways that affect reviews. The interventions are complex and context dependent, often delivered to groups or communities, so effects depend on how and where a program is run. Randomization is less common, so reviews often rely on quasi-experimental designs. Outcomes are measured in many ways, from administrative records to attitude scales. And a large part of the knowledge consists of qualitative studies and theory that explain mechanisms, which a review of effects alone would miss.

The choice of review type therefore depends on the question. The pages on systematic review, qualitative evidence synthesis, mixed-methods review and realist review explain the options.

Choosing the review type

An evidence and gap map classifies studies in a matrix of interventions by outcomes, shows where evidence is concentrated and where it is missing, and often links to the underlying reviews. It is a good first step for a broad policy field, and it helps funders decide what to commission. It does not estimate effects.

Designs without randomization

In many social science questions randomization is not possible. Evaluations of laws, large programs or events rely on difference-in-differences, regression discontinuity, interrupted time series, instrumental variables and matching. Each has assumptions that can be examined: parallel trends for difference-in-differences, continuity at the cutoff for regression discontinuity, and a valid instrument. A review should record the design and the assumptions checked in each study, assess the risk of bias with a tool such as ROBINS-I or the design-specific criteria used by Campbell reviews, and not pool designs with different assumptions without comment.

Observational estimates can be combined with trial evidence, but they should be shown separately and compared. In some topics, such as parenting programs or offender treatment, the two sources agree. In others they do not, and the difference is informative. Waddington and colleagues give detailed guidance for reviews of effects in international development, where many programs are evaluated with quasi-experimental methods.

Equity, context and complex interventions

Social interventions can help one group and not another. A review that reports only an average may hide differences by gender, income, ethnicity or place. PRISMA-Equity, an extension of PRISMA, asks reviewers to say whether and how they considered equity: in the question, in the selection of subgroups, in the analysis and in the reporting of applicability. Subgroup analyses should be planned in advance, based on theory, and tested for interaction; unplanned subgroup findings are hypotheses.

Complex interventions have several interacting components, and a pooled effect for the whole package may not show which part matters. Component analysis, meta-regression on features of the intervention, and qualitative comparative analysis are approaches to this. They are best used with caution, as the number of studies is usually small compared with the number of components.

Context is both a source of variation and the subject of study. A program that works in a high-trust community may fail elsewhere, and reviews should record context information in a systematic way: setting, population, scale, delivery agent, policy environment. This information supports judgments about applicability.

Qualitative and theory-driven synthesis

Interviews, ethnographies and case studies explain the experiences of people who take part in programs, and why programs succeed or fail. Qualitative evidence synthesis integrates such findings with methods such as thematic synthesis and meta-ethnography, appraisal tools such as CASP, and the GRADE-CERQual approach to judge confidence in each finding. ENTREQ and eMERGe guide reporting. When the aim is to build or test a theory about how a program works in context, a realist review is the appropriate design, and RAMESES standards apply.

These approaches complement meta-analysis. In a mixed-methods review, qualitative findings about barriers can help explain why trials show a small average effect, and trials can show whether the themes identified in interviews are linked to outcomes. Planning this integration before the work starts is essential.

Search, grey literature and reporting

A large part of social science evidence is outside journals: government reports, evaluations by non-governmental organizations, working papers and theses. Searches should cover Scopus, Web of Science, ERIC, PsycINFO, Sociological Abstracts, Social Policy and Practice, EconLit and relevant repositories, plus websites of agencies and research institutes. Searching grey literature takes effort but reduces the risk of publication bias, and the search should be documented in a way that others can repeat. Reporting follows PRISMA 2020, with ENTREQ, RAMESES or other guidelines for qualitative and realist strands.

Protocols can be registered with the Campbell Collaboration, on the Open Science Framework or, for health-related questions, in PROSPERO. Registering helps to prevent changes of question after results are known and to reduce duplicate reviews.

Specialties and sub-fields

Sub-fields differ in designs, outcomes and data. Pages for sub-fields are added as they are completed.

A worked difference-in-differences reading

A city introduces a youth mentoring program. Before the program, the treated neighborhoods recorded 42 incidents per 1,000 young people and the comparison neighborhoods 40. After the program the treated neighborhoods recorded 36 and the comparison neighborhoods 38. The numbers are invented to show the arithmetic.

The treated areas fell by 6 and the comparison areas by 2. The difference-in-differences estimate is (36 - 42) - (38 - 40) = -4 incidents per 1,000. The comparison areas show that part of the fall would have occurred without the program, so a before-and-after comparison in the treated areas alone (a fall of 6) would overstate the effect by 2. The estimate is credible only if the two groups would have followed parallel trends without the program. A review that includes such studies records whether the authors showed several pre-program periods with parallel trends, whether other policies changed at the same time and whether standard errors were clustered. These details determine the risk of bias.

Administrative data, outcomes and measurement

Many outcomes in the social sciences come from administrative records: arrests, school attendance, benefit claims, hospital use or earnings. Such data cover large populations and avoid recall bias, but definitions change over time and across places, and records reflect the decisions of institutions as well as the behavior of individuals. An increase in recorded crime, for example, can follow more policing as well as more crime. Reviews should note how each study defined and obtained the outcome, and consider whether measurement could differ between treated and comparison groups.

Attitudes and beliefs are measured with scales whose meaning depends on language and culture. Combining them needs standardization and a check that the measures capture the same concept. The effect size is often a standardized mean difference or a correlation, and the choice and its interpretation are explained in the protocol. Where outcomes cannot be compared, the review reports them narratively with a table of results.

From findings to policy

Policy makers need findings that they can act on. A good review states how large the effect is in practical terms, how certain the evidence is, what is known about cost and what is known about implementation. It reports where the evidence comes from, since a result drawn from studies in one country may not apply in another. It says what is not known. Reviews commissioned by governments may follow a fixed timeline, which can be met with rapid review methods, with limits on search and screening that are described openly.

The role of the review is to describe the evidence. Weighing the evidence against values, budgets and politics is a task for decision makers, who may also consider fairness and feasibility. Review teams that engage stakeholders from the start, and report how they did so, produce findings that are more likely to be used. Plain-language summaries, short policy briefs and open data help findings travel beyond the academic audience, and a stated date of the last search tells readers how current the evidence is. When a topic moves quickly, such as a new law or a new technology, the review can be planned as a living review with scheduled updates.

How we support research projects in this area

Boundaries of this service

A review summarizes what studies found. It does not replace a local evaluation, and results from one jurisdiction may not hold in another. Social interventions are complex, so effects depend on context and implementation. We do not provide legal, policy or casework advice, and we do not make decisions about individual cases.

We recommend the review type that fits the question, which may be qualitative, realist or a map rather than a meta-analysis, and say so at the start.

Frequently asked questions