Evidence synthesis in environmental science and sustainability

Evidence synthesis for the environment

Environmental management often relies on expert opinion and on single studies from one site. Pullin and Stewart argued in 2006 that conservation should adopt the systematic review methods developed in medicine, and the Collaboration for Environmental Evidence (CEE) now publishes guidelines and a journal, Environmental Evidence, for reviews in this field. Gurevitch and colleagues, writing in Nature in 2018, described how meta-analysis changed ecology and called for greater rigor in its use.

Environmental evidence has special difficulties. Study sites differ in climate, soil, species and history, so effects are variable. Experiments are often small, short and hard to replicate, while observational monitoring covers large areas without a control. Interventions such as protected areas, restoration or policy changes are rarely assigned at random. Data are often spatially and temporally correlated. And many questions concern long-term effects that few studies can observe.

Our work follows the approach in systematic review and meta-analysis, adapted to these features.

Systematic review, systematic map or meta-analysis

A systematic map, as James, Randall and Haddaway described in 2016, uses the same transparent search and screening as a systematic review but does not synthesize results. It classifies studies by features such as habitat, intervention and outcome, so that users can see what exists. Maps are useful for broad questions and for planning later reviews. They are not a substitute for a review of effects.

Effect sizes and a worked calculation

Ratio-based effect sizes are common in environmental work because outcomes such as biomass, abundance or concentration are measured on ratio scales and effects are often proportional. The log response ratio is the natural logarithm of the ratio of the treatment mean to the control mean, and its sampling variance is approximated from the standard deviations, sample sizes and means.

As an illustration, suppose a restoration treatment gives a mean plant cover of 12 percent (standard deviation 3.0, n = 10) and the control gives 8 percent (standard deviation 2.5, n = 10). The log response ratio is ln(12/8) = 0.405. Its variance is 9.0/(10 x 144) + 6.25/(10 x 64) = 0.0160, so the standard error is 0.127 and the 95 percent confidence interval is 0.16 to 0.65. Back-transformed, the ratio is 1.50, or 50 percent more cover, with an interval of 1.17 to 1.92. The data are invented, and the calculation is shown only to explain the measure.

The log response ratio assumes means are positive and not close to zero, and its variance approximation can be poor when sample sizes are small. These checks belong in the protocol. When effects are expressed as proportional change, the review should present them in this form and not as absolute change, since the baseline differs between sites.

Spatial, temporal and phylogenetic dependence

Environmental effect sizes are dependent in several ways. Several effects come from one experiment, several experiments from one research program, several sites from one region, and several species from one genus. As in other biological work, multilevel models, robust variance estimation and sensitivity analyses address the dependence. Spatial dependence, where nearby sites are more alike, can be examined with a spatial random effect or by testing for geographic clustering of residuals.

Time matters, too. Responses to disturbance or restoration change as years pass, and a meta-analysis that pools short and long studies will blur them. Time since the intervention is a natural moderator, and the review should show how many studies contribute at each stage. Long-term data sets are valuable, but they are few, so conclusions about long-term effects often rest on little evidence.

Study design, bias and critical appraisal

Controlled experiments give the strongest causal evidence but often lack realism. Before-after-control-impact (BACI) designs compare changes at treated and control sites and are common in environmental assessment. Simple before-after and space-for-time comparisons are weaker, since other changes may occur at the same time or the sites may differ. The CEE critical appraisal tool and similar instruments assess the risk of bias from confounding, selection of sites, blinding (rare), data handling and selective reporting, with the findings used in sensitivity analysis.

Publication bias is a concern, as is the tendency to report positive restoration outcomes. The same tools for small-study effects apply, with the cautions given for dependent data. Searching should include grey literature, such as reports by agencies and non-governmental organizations, in languages relevant to the region, since much environmental evidence is published locally and not in English. Language restrictions are a source of bias, and a review should state them.

Reporting: ROSES and PRISMA

Haddaway and colleagues developed ROSES (RepOrting standards for Systematic Evidence Syntheses) in 2018 for systematic reviews and systematic maps in environmental sciences. It has separate forms for reviews and maps and asks for the details that matter in this field, such as how the search was supplemented with grey literature and how evidence was classified. PRISMA 2020 can be used alongside it for meta-analyses. CEE also provides a protocol and review submission process through its journal, which includes peer review of the protocol.

Specialties and sub-fields

Sub-fields differ in data and designs. Pages for sub-fields are added as they are completed.

Moderators and context dependence

Environmental effects vary with context, and the main aim of many reviews is to explain the variation. Typical moderators include climate zone, latitude, soil or water type, habitat, species group, the size of the area treated, the duration and the intensity of the intervention. A meta-regression can test these, and it should include a measure of the extent of each moderator in the data, such as the number of studies in each climate zone. Geographic coverage is often uneven: studies cluster in North America, Europe and a few other regions, and tropical and low-income regions are under-represented. A review should map where its studies come from and say how far the conclusions extend.

Moderators are correlated, so interpretation needs care. Studies in warmer places may also be shorter, and studies of larger plots may also use different methods. Reporting the full model and simpler models, and the correlation between moderators, helps readers judge. A moderator that explains differences between studies does not prove that it causes differences in the effect, and the review should say so.

Modeling studies, scenarios and projections

Climate, hydrology and energy research rely on models. Model outputs are not data from independent observations: they depend on assumptions, inputs and structure, and several outputs from one model family are related. A review that includes modeling studies should keep them separate from empirical studies, record the model, the scenario and the assumptions, and avoid treating the number of models that agree as a measure of the strength of evidence. Intercomparison projects, in which many groups run the same scenarios, have their own methods, and a review should use those where possible. Model evaluation against observations is a central part of judging credibility.

Life-cycle assessments and techno-economic analyses are similar. Their results depend on system boundaries, data sources and allocation rules. A review of such studies compares those choices first and numbers second, and a table of harmonized assumptions lets readers see why results differ. Harmonization, in which results are adjusted to a common basis, can reduce spread but adds assumptions that must be reported.

Using evidence in management and policy

Reviews of environmental evidence serve land managers, regulators and funders who need to know whether an action is likely to work in a particular place. Decision makers often need results within months, so the scope should be set with them: the habitat, the action, the outcome and the region. A report should state the conditions in which studies were done, the confidence in the findings, what is unknown and what monitoring would reduce the uncertainty. For adaptive management, where decisions are revised as data accumulate, a living review or a regularly updated map can be useful, and the plan for updates should be set at the start. A short summary for non-specialists, with the main figures and the confidence in them, helps the findings to be used. Where results will inform regulation or funding, an independent peer review of the protocol and the final report adds credibility, and conflicts of interest of the review team and funders should be declared in the report.

Conservation and sustainability decisions also involve costs, social acceptance and values, which a review of effects does not address. Where cost data are reported, they can be extracted and summarized, with the caution that costs vary greatly between places. Traditional and local knowledge is a source of evidence in many contexts, and synthesis methods that include it, such as qualitative and mixed-methods reviews, may be suitable when the question concerns community experience or practice.

How we support research projects in this area

Boundaries of this service

A synthesis of environmental studies describes what the evidence shows. It does not replace site-specific assessment, and effects measured in one ecosystem may differ in another. Modeling studies depend on their assumptions and are handled differently from empirical studies. We do not provide environmental consulting, impact assessment sign-off or legal opinions.

When there are too few comparable studies to pool, we recommend a systematic map or narrative synthesis and say so at the start.

Frequently asked questions