A/B Test Designer

An experiment is a bet with a referee, and the referee only works if appointed before the coin flip. This skill designs A/B tests that can actually settle their question: a falsifiable hypothesis with a mechanism, one pre-committed decision metric, samples sized against the minimum effect worth acting on, and an analysis plan written — and frozen — before the first user is assigned.

When to use this skill

A product, pricing, growth, or lifecycle change needs causal evidence before rollout
Someone proposes "let's just test it" without a hypothesis, a metric, or a size
Reviewing an experiment design, or a suspicious result someone wants to ship on
Deciding whether a question is testable at all, or better served by a holdout or a pre/post read

Workflow

Write the hypothesis as a falsifiable sentence with a mechanism: "Showing delivery dates on the product page will increase checkout completion by reducing uncertainty at the decision moment." A bare "B beats A" teaches nothing transferable whether it wins or loses.
Choose one primary decision metric — the single number the ship-or-kill call reads — plus two to four guardrail metrics that must not degrade: latency, unsubscribes, refunds, downstream conversion. Secondary metrics may be watched; they may not promote themselves to primary after the fact.
Fix the unit of randomization — user, account, session, geography — and keep it consistent with the unit of analysis. Randomize at the level where interference is tolerable: users who share a team, a household, or a marketplace bleed treatments into each other.
Size against the minimum effect worth acting on, not the effect you hope for. Ask "what lift would justify shipping and maintaining this?", then power the test to detect it — conventionally 80% or better at your chosen significance level. If the required sample takes longer than the decision can wait, redesign: a more frequent metric event, a coarser question, or a sequential design — never a knowingly underpowered run.
Set duration by calendar logic as well as sample size: whole weeks to absorb weekday-weekend cycles, at least one full business cycle for business-to-business products, and no distorting season unless the launch will live in one. Put the stop date in writing.
Pre-commit the analysis plan: exact metric definitions, the test statistic, one-sided or two-sided, the outlier and bot rules, the few pre-named segments that will be examined, and the multiple-comparison treatment if there is more than one look. Write the three verdict branches now: ship if X, kill if Y, iterate only under pre-named condition Z.
Instrument a health check, not just an outcome check: a sample-ratio-mismatch alarm, assignment logging, and a first-day sanity read of exposure counts. A 52/48 split that should be 50/50 invalidates the test no matter what p-value it goes on to produce.
Run the pre-mortem: "the test ended and the result convinced nobody — why?" The usual answers — the metric got argued about, someone peeked, it was underpowered, novelty wore off, one whale segment ruled the mean — are all fixable at design time and none of them afterwards.

Output format

## Experiment brief: <name>

Hypothesis: <change> will <effect> because <mechanism>
Primary metric: <exact definition, window>     Guardrails: <list, with thresholds>
Unit: <randomization / analysis>   Split: <A/B %>   Eligibility: <who enters>
MDE: <x>%   Power: <p>% at alpha <a>   Required n: <per arm>   Window: <start-stop>
Analysis: <test, sidedness, outlier rule, named segments, correction>
Decision rules: ship if <X> / kill if <Y> / iterate if <Z>
Health checks: <SRM alert, exposure sanity, assignment log>
Owner: <role>   Reviewed by: <role>   Frozen on: <date>

Guardrails

No peeking-and-stopping on raw p-values; interim looks require a sequential method chosen up front
The primary metric cannot change after launch — a better metric discovered mid-test is input to the next test's design
Findings from unplanned slices are hypotheses, not results; label them that way
A neutral result on a well-powered test is an answer; do not rerun until randomness cooperates
Never ship on a win that violates a guardrail — the guardrail was the point

Design review checklist

Mechanism stated, so someone learns something from a loss
One primary metric, defined down to the filter level
Sample size computed from an MDE someone defended in writing
Stop date on the calendar; analysis plan frozen and shared before launch
SRM and exposure monitoring live from day one