Automation Pipeline Designer

The difference between automation and a mess that runs unattended is where you put the checkpoints. This skill designs LLM-in-the-loop pipelines — triage, enrichment, drafting, routing, reporting — by mapping the manual process first, then deciding which steps become plain code, which get a model's judgment, and which keep a human hand on the lever, with failure handling designed before the first happy path ever runs.

When to use this skill

• Turning a recurring manual workflow into a scheduled or event-triggered pipeline
• An existing automation misbehaves and nobody can say at which step or why
• Deciding which parts of a process are safe to hand to a model and which are not
• Designing the human-approval surface for actions that cannot be taken back

Workflow

1. Map the manual process as performed, not as documented. List each step with its inputs, outputs, decisions, and the quiet exception handling humans do without noticing — "if the attachment is missing, I ask for it." The exceptions are the real specification.
2. Classify every step into one of three kinds:
   • Deterministic: rules fully cover it — plain code, no model involved
   • Judgment: pattern recognition or language work — a model step with an explicit "unsure" outcome
   • Consequential: it sends, deletes, pays, or publishes — it gets a gate, either human approval or a hard allowlist, regardless of how confident anything upstream is
3. Design each model step as a contract: input shape, output schema validated by code (not by hope), the unsure route, and two or three anchoring examples. A model step without an "unsure" outcome will guess, and a pipeline amplifies guesses.
4. Place gates directly before irreversibility. Batch approvals into a digest where volume allows. The approval view shows exactly what will happen and the exact content; a one-click rejection captures a reason, and rejection reasons are logged as future training material.
5. Make every step idempotent and the whole run resumable: a dedupe key on the trigger, steps safe to re-run, and state persisted between steps so a crash resumes instead of restarting — or worse, double-sending.
6. Set the failure policy per step: retry (how many times, what backoff), skip-and-flag, or halt-the-line. Consequential steps fail closed: when in doubt, do nothing and notify.
7. Instrument from day one: a per-run log with input reference, each step's outcome and latency, model confidence where applicable, and cost. Add a weekly human review of a random sample of auto-handled items — drift arrives quietly, and sampling is how you hear it.
8. Pilot on a shadow lane. Run read-only against live inputs, compare the pipeline's decisions with what humans actually did, and graduate to write-mode step by step — consequential steps last, each on the strength of its own shadow record.

Output format

Pipeline: <name>
Trigger: <event or schedule>     Dedupe key: <field>

| # | Step | Kind (code/model/gate) | Contract | On failure |
|---|------|------------------------|----------|------------|

Gates: <what humans approve, where it surfaces, expected turnaround>
Kill switch: <the one action that pauses everything>
Drift check: <sample size, cadence, who reviews>
Graduation plan: <shadow → assisted → autonomous, per step, with thresholds>

Guardrails

• No unattended consequential actions in version one; autonomy is earned per step with shadow-mode evidence, never granted by optimism
• Model outputs are validated by code before anything downstream consumes them
• A pipeline without a kill switch is a liability with a schedule
• Cap the blast radius structurally: rate-limit outbound actions so a bad loop is bounded by design
• Unsure items exit to a human queue; they never loop back in for a second guess

Worked example: support-inbox triage

Trigger: new inbound message, deduped on message ID. Code step extracts sender, product area, and attachments. Model step classifies intent and drafts a reply, with "unsure" routing straight to a human. Gate: humans approve all drafts in refund-class intents; password-reset replies go out automatically only after four weeks of shadow mode agreeing with humans above an agreed threshold. Failures: classification errors route to the human queue; the send step fails closed. Every message leaves an audit row — who or what decided, and on what evidence.