What the service is for
Original research is where most scientific claims begin, and it is also where many go wrong. A study may ask a question that its design cannot answer, enroll too few participants, measure the wrong outcome, analyze the data in a way that does not match the design, or report so incompletely that readers cannot judge it. Such problems are often discovered only at peer review, when they are costly to fix, or after publication, when they are not fixable at all.
This service helps researchers avoid these outcomes. It supports the design of a study before data collection, the analysis of data after it, and the preparation of the manuscript, with attention to the reporting guideline that applies. It does not replace the researchers. They conceive the study, collect the data, interpret the findings and are accountable for the paper, as the research integrity and authorship statement explains. Our role is to support, to advise on method and to improve the quality of the report.
What the service includes
- Design review. An assessment of the question, the design, the population, the outcomes and the threats to validity, with suggestions for improvement.
- Sample size and power, where data are not yet collected.
- Statistical analysis plan, written before the analysis.
- Statistical analysis of the data, as described under statistical analysis.
- Reporting-guideline check for the study design, with a completed checklist.
- Manuscript support, from structure and drafting of methods and results to editing, as agreed.
- Journal selection and submission materials, described under publication support.
- Revision support in response to reviewers.
Reviewing the design
The design is the strongest determinant of what a study can show, and it is far easier to improve before the data are collected. A design review begins with the question: what is being asked, of whom, about which exposure or intervention, compared with what, and measured how. It then asks whether the design can answer it. A comparison without randomization is vulnerable to confounding, and the review considers which confounders are measured and how they will be handled. A single-arm study cannot establish that an effect is due to the intervention. A cross-sectional study cannot establish the order of events. An outcome measured with an unvalidated instrument may not mean what the authors say it does.
Specific threats are checked. Selection bias arises when the people studied differ systematically from those the question concerns. Measurement bias arises when exposure or outcome is recorded differently in different groups. Attrition and missing data can distort results when they are related to the outcome. Multiplicity, the testing of many outcomes, inflates false positives. The review does not aim to discourage research but to see that the limits of the design are understood and reported, and, where possible, that they are reduced.
Sample size and the plan
Studies with too few participants are common and costly. They are unlikely to detect real effects, and when they do find an effect they tend to overestimate it. A sample-size calculation needs a clear primary outcome, an assumed effect size that is justified from earlier research or from the smallest effect that would matter in practice, an estimate of variability, the acceptable error rates and an allowance for loss to follow-up. If the calculation shows that the necessary sample is out of reach, the honest options are to change the design, to collaborate to increase the sample, or to describe the study as exploratory or a pilot.
The statistical analysis plan fixes the primary outcome and analysis, the handling of missing data, the covariates, the secondary analyses and the planned subgroups before the data are analyzed. Registering the study and the plan, in a trial registry or an open-science registry, adds transparency. Studies that are analyzed first and planned afterwards are common, and the report should say honestly which analyses were planned and which came from exploring the data.
Reporting guidelines by design
Most designs have a reporting guideline that lists what a paper should contain. Following the right one makes the paper easier to assess and is increasingly required by journals. The EQUATOR Network catalogs them. The most commonly needed ones are in the table.
| Design | Guideline |
|---|---|
| Randomized trials | CONSORT; protocols with SPIRIT |
| Observational studies (cohort, case-control, cross-sectional) | STROBE |
| Diagnostic accuracy studies | STARD |
| Prediction models | TRIPOD |
| Qualitative research | COREQ for interviews and focus groups; SRQR for qualitative studies in general |
| Animal studies | ARRIVE |
| Case reports | CARE; see case report support |
See reporting guidelines. The checklist should be completed by finding each item in the manuscript, and the manuscript changed where an item is missing.
Analytic flexibility and false positives
There are many defensible ways to analyze almost any dataset: which participants to include, how to define the outcome, which covariates to adjust for, how to handle outliers, when to stop collecting data. Each choice is made after seeing something about the data, and together they let a researcher find a significant result that would not survive a single pre-planned analysis. Research on this flexibility has shown that it can raise the false-positive rate far above its nominal level without any intent to mislead. The defenses are the ones this service uses: a prespecified analysis plan, a single primary outcome, honest labeling of exploratory analyses, adjustment for multiple comparisons where it applies, and a full report of what was done. They protect the researchers as much as the readers, since a result that was planned, and that holds up, is much harder to criticize.
Pilot and feasibility studies
Many studies are really pilots: they test whether a design can work, whether participants can be recruited and retained, whether a measure behaves well, or what variability to expect. They are valuable, and they should be reported as pilots, with feasibility outcomes such as recruitment and retention rates, and not as small, underpowered tests of effectiveness. The effect estimate from a small pilot is imprecise and tends to be inflated, so it is a poor basis for the sample size of a larger trial, and guidance exists on how to plan one. Calling a pilot a definitive study invites criticism, and calling it a pilot, when that is what it is, is honest and protects the credibility of the findings.
Analysis
The analysis follows the plan and the design. It is carried out in scripts, with checks of assumptions, a considered handling of missing data, and a distinction between planned and exploratory analyses. Results are reported with effect estimates and confidence intervals and not only with p-values, and the clinical or practical meaning of the effects is discussed. The work is described in more detail under statistical analysis, and the code and the logs are delivered so that the results can be reproduced.
The manuscript
The manuscript tells the story of the study in the order readers expect: the question and why it matters, the methods in enough detail to be repeated, the results, and a discussion that interprets them and acknowledges limitations. The reporting guideline defines what each section should contain. The authors write or approve the interpretation, and the support concentrates on structure, accuracy and clarity: that the methods are complete, that the results are reported fully and consistently with the tables, that limitations are stated plainly, and that the conclusions follow from the results. The paper is then edited and formatted for the target journal, as described under manuscript editing.
Ethics, registration and data
Studies involving people or animals need ethics approval, and journals require a statement of it, and of consent where applicable. Clinical trials are registered, and many journals will not consider a trial that was not registered before enrollment. Data availability statements are increasingly expected, and the data should be managed according to the permissions and the privacy rules that apply. The researchers are responsible for confirming that the study was approved and conducted ethically, and we ask to see the approval documents. We cannot support a study that appears to lack the approvals it needs. Concerns about integrity are handled as described in the research integrity statement.
Authorship and acknowledgment
Authorship of an original study belongs to the people who meet the criteria set by the journal, which in most cases follow the ICMJE recommendations: a substantial contribution to the work, drafting or critical revision, approval of the final version and accountability for it. The researchers who conceive the study, collect the data and interpret the findings are in that position. Support with design, statistical analysis and writing is acknowledged as the journal requires, and where a person on our side contributes at a level that meets the authorship criteria, the matter is settled openly with the research team at the start and in line with the journal's policy. We do not take part in arrangements that hide a contribution which must be declared.
Deliverables
- Design review with recommendations.
- Sample-size calculation and analysis plan, where in scope.
- Results, tables and figures, and analysis code.
- Completed reporting checklist for the study design.
- Manuscript support as agreed, from structure to a formatted submission.
- Support with reviewer comments.
Get a quoteDescribe your study, stage and target journal.
Optional extension: full manuscript and submission
Full manuscript and submission package
When the scope includes manuscript preparation and submission, the package also contains the following.
Full manuscript draft
Written to the reporting guideline for your design, with the analysis, tables and figures integrated.
Reporting checklist
CONSORT, STROBE, STARD, TRIPOD or the guideline for your design, completed against the final text.
Submission package
A journal recommendation, the manuscript formatted to that journal, a cover letter and supplementary files.
Revision round
A point-by-point response to editor and reviewer comments, with re-analysis where requested.
Manuscript preparation follows the research integrity and authorship statement. The researchers who conceived the study and interpret its findings remain responsible for the content and its conclusions, and contributions that do not meet authorship criteria are acknowledged.
Limitations and boundaries
Support cannot rescue a study that was designed in a way that cannot answer its question, and it cannot compensate for data that are missing or poorly measured. We do not collect, generate or alter data, and we will not fit analyses to a desired conclusion. We do not guarantee acceptance by any journal. If the data do not support the planned conclusion, we say so and discuss what the data can support, which may be a more modest paper. The results of a study are not advice about the care of an individual patient. This service provides research support and does not provide clinical advice.
The researchers lead the study, own its conclusions and are accountable for the paper.
Frequently asked questions
Can you help before I collect data?
Yes. Design review and sample-size planning are most valuable before data collection, when problems are easiest to fix.
Do you collect data or run experiments?
No. We support design, analysis and reporting of studies that the researchers conduct. We do not collect, generate or alter data.
Which reporting guideline applies to my study?
It depends on the design. CONSORT applies to randomized trials, STROBE to observational studies, STARD to diagnostic accuracy and TRIPOD to prediction models, among others catalogued by the EQUATOR Network.
Who will be listed as authors?
The people who meet the journal's authorship criteria, usually the researchers who lead and interpret the study. Support is acknowledged as the journal requires, and any larger contribution is discussed openly at the start.
What if my results are not what I expected?
They are reported as they are. An honest analysis plan and a clear separation of planned and exploratory analyses protect the paper, and unexpected results are often informative.
Is this service clinical advice?
No. It is research support, and the results of a study are not advice about the care of an individual patient.
References
- EQUATOR Network. Enhancing the QUAlity and Transparency Of health Research. Available at equator-network.org.
- Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.
- Chan AW, Tetzlaff JM, Altman DG, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158(3):200-207.
- von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335:806-808.
- Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.
- Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594.
- Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349-357.
- O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245-1251.
- Percie du Sert N, Hurst V, Ahluwalia A, et al. The ARRIVE guidelines 2.0: updated guidelines for reporting animal research. PLoS Biol. 2020;18(7):e3000410.
- Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011;22(11):1359-1366.
- Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2(8):e124.
- Button KS, Ioannidis JPA, Mokrysz C, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14(5):365-376.
- Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci U S A. 2018;115(11):2600-2606.
- International Committee of Medical Journal Editors. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Available at icmje.org (check the current version).