Data extraction and coding

Why extraction matters

Extraction is the step in which the numbers in published papers become the data of a review. It looks routine and is anything but. Papers report results in different ways, with different units, time points and effect measures, and often with errors or omissions. The extractor must decide which of several reported results to use, convert it to a common form, and enter it correctly. Each of these decisions can change the pooled estimate.

The error rate in extraction is not small. Studies of published meta-analyses have found extraction errors in a substantial share of them, including cases in which correcting the errors changed the conclusion, and meta-research has shown that independent double extraction reduces errors compared with a single extractor. For this reason extraction is treated as a measurement process that needs a protocol, a form, a check and a record. It is part of the systematic review process and is the input for the meta-analysis.

What the service includes

Designing the extraction form

A good form is specific to the review, as short as it can be and unambiguous. Every item has a definition, and the allowed values are listed, so that two extractors would enter the same thing. The form usually captures the identity of the study and its reports, the design, setting and dates, the characteristics of the participants, the details of the intervention or exposure and the comparator, the definitions and timing of the outcomes, the results needed for the analysis, the funding and declared conflicts, and the risk-of-bias information if the appraisal is done on the same form. Items that will not be used in the analysis or the report are left out, because each one adds time and error.

The form is piloted on a few studies of different kinds by the people who will use it. Problems that appear, such as unclear definitions or results that fit nowhere, are fixed before the main extraction. The final version is fixed and any later change is recorded, with the studies already extracted updated to match.

Extraction in practice

Two people extract independently from the full texts and the results are compared, with discrepancies resolved by discussion or a third person. Where resources are limited, a documented alternative, such as one extractor with a second person checking all the data used in the analysis, can be used, and its risk is acknowledged. The data should be taken from the most complete source: the main paper first, then supplements, registry entries and any additional reports, with a rule for deciding which to use when they disagree.

The unit of extraction is the study, not the paper. One trial may be reported in several papers, and one paper may report several studies. Linking reports to the parent study avoids counting the same participants twice, which would inflate the weight of that study. Where it is unclear whether two reports share participants, the authors are asked, and the uncertainty is handled in a sensitivity analysis.

Computing effect sizes

The analysis needs, for each study and outcome, an effect estimate and a measure of its precision. Studies report a bewildering variety of formats, and the conversion between them is part of the work. For binary outcomes the counts of events and participants in each group give a risk ratio, odds ratio or risk difference. For continuous outcomes the means, standard deviations and sample sizes give a mean difference or a standardized mean difference, with a correction for small samples for Hedges' g. Correlations are transformed to Fisher's z for pooling. Hazard ratios and their standard errors are taken from the paper or derived from confidence intervals. See the guide on effect sizes.

Formulas exist for many conversions: a standard error from a confidence interval, a standard deviation from a standard error, and effect sizes from t statistics or p-values when nothing else is reported. They are applied with care, because each depends on assumptions, and they are recorded so that the reader can see which values were reported and which were derived. A sensitivity analysis that excludes studies that required substantial conversion shows how much they matter.

Awkward cases

Medians and ranges

Studies often report a median with a range or quartiles instead of a mean and standard deviation. Methods exist for estimating the mean and standard deviation from these summaries, and they work better for some distributions and sample sizes than for others, so they are used with caution and flagged.

Multi-arm trials

A trial with several intervention arms and one control cannot enter an analysis as several independent comparisons without double-counting the control group. The arms are combined or the control group is split, according to the question.

Change from baseline and final values

Studies report one or the other, and a pooled analysis must not mix them without care, because their variances differ. The choice and the handling of the correlation between baseline and final values are documented.

Cluster and crossover designs

Analyses that ignore clustering or pairing give standard errors that are wrong, and adjustments, such as the design effect for clusters, need information the paper may not give.

Survival data

When a hazard ratio is not reported, it may be estimated from reported statistics or reconstructed from published curves. These routes are less reliable and should be flagged.

Data only in figures

Values can be read from graphs with digitizing software, which adds measurement error that must be recorded.

Coding study characteristics

Beyond the outcome data, a review often codes characteristics of the studies for use in subgroup analysis, meta-regression and description: the population, the setting, the dose or duration of the intervention, the year, the country, the design features and the risk of bias. Coding turns descriptions into variables, and the way it is done affects what can be analyzed. Categories are defined in advance with explicit rules and examples, continuous characteristics are recorded as numbers where the paper allows and not forced into bands, and the codes for rare categories are decided with the analysis in mind, since a category with one or two studies cannot be analyzed. Where coding involves judgment, two coders work independently on a sample, and their agreement is measured, typically with a chance-corrected statistic, and disagreements are resolved and the rules clarified. The coded variables feed the analyses described under meta-regression.

Data management and version control

The dataset is a research record, and it is managed as one. Raw extraction entries are kept separate from the cleaned analysis dataset, so that every change can be traced. Each version of the dataset is labeled with a date and a description, and the analysis code is tied to a specific version. Study and report identifiers are consistent across files, and the codebook is kept with the data so that the meaning of every variable is recorded. The data are stored securely, and where they include any information that could identify people, for example in individual participant data projects, they are handled according to the permissions and the privacy rules that apply. These habits are unglamorous, and they are what makes a result reproducible months later, when a reviewer asks a question about a single study.

Checking the dataset

Deliverables

Optional extension: from dataset to published paper

Limitations

Extraction cannot improve the reporting of the studies. When a paper does not give the data, they cannot be invented, and where an estimate has to be derived, its uncertainty is passed to the analysis. Different reasonable extractors can make different choices when a paper reports several results for an outcome, so rules should be set in advance and the choices recorded. Contact with authors helps, but replies are incomplete and slow. Because extracted data are the foundation of the analysis, errors cannot be eliminated, only reduced and documented, which is why the audit and the sensitivity analyses matter. The service does not select results to favor a conclusion, and every conversion is recorded.

How long does it take?

The time depends on the number of studies and outcomes, how clearly the papers report their results, how many conversions are needed, whether double extraction is used, and how many author queries are sent. Studies that report results in full are quick to extract, and those that report little take far longer. After a look at a sample of the studies, a schedule and its dependencies are agreed.

Frequently asked questions