Grey literature in systematic reviews

What counts as grey literature

The term covers material produced outside the channels of commercial publishing and distribution, where the primary activity of the producer is not publishing. In evidence synthesis it usually means material that cannot be found easily through the main bibliographic databases. The category is broad and the boundaries are not sharp: a conference abstract in a journal supplement is partly in the formal literature, and a preprint is on its way to it.

It is useful to separate three situations. Some grey sources are complete studies that were never published in a journal, such as theses and reports. Some are incomplete accounts of studies that were published later, such as conference abstracts and preprints. And some are records about studies rather than their results, such as registry entries. Each plays a different role in a review.

Why it matters

The main reason to search grey sources is to reduce the risk of publication bias. Studies with statistically significant or favorable results are more likely to be published, to be published sooner and in higher-profile journals, and to be indexed in major databases. A review that draws only on published, indexed articles may therefore overestimate effects. Empirical studies comparing meta-analyses with and without grey literature have found that excluding it tends to give larger effect estimates on average, although the difference varies between reviews and is not always large.

Grey literature also matters for the completeness of the evidence in fields where much of the work never appears in journals, such as program evaluations, education policy, health technology assessment, management and development economics. In these areas, reports from agencies and working papers are the primary form of publication, and omitting them leaves out much of the evidence.

There are costs. Grey sources are scattered, poorly indexed, difficult to search systematically, and uneven in quality and detail. The work needed to find them is much greater per eligible study than for bibliographic databases. For this reason the decision about how far to go should be made in the protocol, with reasons.

Where to look

Sources depend on the topic. The table lists common categories and examples, which should be checked for current availability and coverage.

Other approaches complement these. Searching the websites of relevant organizations, professional societies and funders can identify reports. Contacting experts, authors of included studies and manufacturers can reveal unpublished work. Checking the reference lists of included studies and relevant reviews finds items that mention grey sources. Google and Google Scholar are used to find reports and theses, with a stated number of screened results, because their rankings are not reproducible and the search interface is limited.

Registries and regulatory documents

These two sources deserve special mention for reviews of interventions. Trial registries show which trials were started, which have completed and what outcomes were planned. Comparing registry entries with published reports helps to detect selective outcome reporting, and an unpublished completed trial is a signal of possible publication bias. Some registries also hold results.

Regulatory submissions, such as the reviews and clinical study reports provided to drug and device regulators, can contain more complete data than journal articles, including outcomes that were not published and detailed adverse events. Studies that compared regulatory documents with journal publications have found differences in the outcomes reported and in the conclusions. Using them is demanding, because the documents are long and not designed for meta-analysis, but for drug reviews with safety questions, they may be the best source.

Appraising and handling grey sources

Grey literature is not necessarily poor in quality, and peer-reviewed literature is not necessarily good. The principle is to apply the same eligibility criteria and the same risk-of-bias assessment to all sources. In practice, grey sources often have less reporting detail, which makes risk-of-bias judgments harder and often leads to more items rated as unclear. That information gap is a finding, not a reason to exclude.

• Apply the same criteria. Eligibility depends on the study, not on where it was published.
• Link reports. A conference abstract may be an early report of a trial later published in full. Treat them as one study and use the most complete data.
• Check for later versions. Preprints may be revised or published, and results may change. Record the version and date accessed, and recheck before submission.
• Consider sensitivity analysis. Run the meta-analysis with and without grey sources, and report the difference.
• Note limited data. Abstracts rarely contain enough for a full risk-of-bias assessment or for extraction of adjusted results. Decide in advance how to treat studies that cannot be assessed.
• Look at retractions and corrections for items that later appeared in journals.

Preprints

Preprints have become a major route for new evidence, as the COVID-19 period showed, and many reviews now search servers such as medRxiv and bioRxiv. They give early access and reduce publication delay, and they include studies that may never be accepted by a journal. They have not been through peer review, a fact that should be recorded, and their results sometimes change before publication. A review that includes preprints should say so, say how they were identified, and plan to check for published versions. Journal policies differ, and some reviewers question the inclusion of preprints, so the choice and its reasons belong in the protocol.

Deciding how much to search

The amount of grey literature searching is a trade-off between completeness and resources, and the right answer depends on the review. A review of drug effects that informs a guideline should search registries and regulatory documents. A review of an education program should search education research repositories and agency reports. A rapid review may justify fewer sources, with the limitation stated. A useful rule is to consider where eligible but unpublished studies are likely to be held for this topic and to search there. Record how many included studies came from grey sources, since that gives evidence about whether the effort was worthwhile for this question and informs future reviews.

Documenting the search

PRISMA 2020 asks reviewers to list all sources searched, with dates, and PRISMA-S asks for details of registries, web searching, citation searching and contact with experts. Record each website, the search terms or browsing method, the date, the number of records screened and the number retained. Web searches change over time and cannot be rerun exactly, so saving the results and screening decisions matters. The flow diagram should show records from databases separately from records from other methods.

Searching websites and the open web

Web searching is a different activity from database searching, and it is easy to do badly. Search engines personalize and reorder results, apply their own ranking and truncate lists, so a web search cannot be repeated exactly. Plan it so that it is as transparent as it can be. List the organizations and sites to be searched, based on who produces research on the topic. Use the site's own search box where there is one, and record the terms. For general engines, use a small set of defined phrases, set a stopping rule such as the first 100 results for each, use a private browsing window to limit personalization, and record the date and the number screened.

Screening web results is slow because titles and snippets are often uninformative. A short form that asks whether the page reports a study with an eligible population, intervention and outcome, and where the full report can be found, speeds the work. Many results lead to documents that describe, but do not report, a study, so tracing the original source takes time.

Limits of the approach

Even a thorough grey search will not find everything, and it can introduce its own biases. Items that are easy to find online are not a random sample of what exists, and organizations that publish reports may be those with an interest in a particular result. Quality varies, and reporting is often brief. Duplicates are frequent, because the same report appears on several sites and in several versions. A review that includes grey material should say so, and its limitations section should recognize that completeness cannot be shown, only approached. The best protection against publication bias is the combination of registry searching, regulatory data, contact with investigators and clear reporting of what was and was not searched, so that readers can judge the risk for themselves. If time is short, prioritize: registries for trials, regulators for drugs and devices, theses for fields where graduate work is common, and the sites of the two or three organizations most likely to publish on the topic.

How we can help

We can plan the grey literature strategy for your topic, search registries, regulatory sources, theses and preprint servers, contact experts and manufacturers where appropriate, assess and link the records found and document the search to PRISMA-S. [OWNER VERIFICATION REQUIRED] The relevant services are literature search strategy and systematic review.

Frequently asked questions