Paper Claims Reviewer

Papers earn their conclusions or they borrow them. The abstract says one thing; somewhere in the middle, that claim rests on a specific design, a specific sample, and specific numbers — or it does not. This skill reads a paper the way a careful referee does: extract every substantive claim, check whether the design can support that type of claim, read the actual effect sizes, and issue a verdict per claim rather than a vibe per paper.

When to use this skill

Deciding whether a paper's findings should influence a product, investment, or policy decision
Reviewing a manuscript, preprint, or whitepaper someone is about to circulate as evidence
A press release or viral thread makes a paper sound revolutionary and someone should read the methods section
Comparing two papers that appear to disagree, before citing either one

Workflow

Read the methods before the abstract's spell sets in. Note the design — randomized experiment, natural experiment, observational with controls, cross-sectional survey, simulation, case study — the sample (size, recruitment, population), and what was actually measured, including how far the measured proxy sits from the concept in the title.
Extract the claims verbatim. From the abstract, results, and conclusion, list each substantive claim as stated. Classify each one: causal ("X improves Y"), associational ("X is linked to Y"), descriptive ("Y occurs at rate Z"), or extrapolative ("this suggests X would work in setting W").
Match claim type to design ceiling. An observational design caps out at association. A lab experiment on volunteers caps its generalization. A simulation supports "under these assumptions" and nothing louder. Flag every claim written above its design's ceiling — the single most common failure, and it usually lives in the abstract.
Read the numbers, not the stars. For each key result: the effect size and its practical meaning (a reduction of what, for whom), the uncertainty interval, and the raw group sizes. Ask two separate questions — is the effect large enough to matter if real, and is the sample large enough to trust the estimate? Statistically detectable and practically important are different properties.
Probe the flexibility. How many outcomes were measured versus reported? Were subgroups pre-specified or discovered after the fact? Do the tables, figures, and text tell the same story? Selective reporting rarely announces itself; count what should be present and is not.
Compare the limitations section against the conclusions. Authors confess constraints on one page and forget them two pages later. The gap between those two sections is the paper's honesty gradient — measure it.
Issue per-claim verdicts: supported (design and data carry it), overstated (real finding, inflated language or scope), unsupported (the data does not carry it), or cannot-tell (missing information — name exactly what is missing).

Output format

A claims table — claim verbatim, type, where stated, evidence offered, verdict, one-line reasoning — followed by a short overall read: what the paper genuinely establishes, what it merely suggests, the strongest single reason to doubt it, and what follow-up evidence would upgrade the verdicts.

Guardrails

Review the evidence, never the authors; motive speculation is out of scope
Do not demand perfection — every study has limits; the question is whether the limits are fatal to the specific claims made
Small samples justify wide uncertainty, not automatic dismissal; large samples do not launder a biased design
Say plainly when the field is outside your depth and mark the affected verdicts as provisional
A paper that fails this review is unproven here, not disproven — keep that distinction in every summary sentence

Worked example: one verdict row

Claim (conclusion section): "daily use of the tool improves team productivity." Type: causal. Evidence offered: a cross-sectional survey where heavy users self-report higher output. Verdict: overstated. Reasoning: the design supports association only — productive teams may simply adopt tools faster — and the outcome is self-reported by the treated group. Upgrade path: any design with assignment, or at minimum a lagged panel with objective output measures.