Log Pattern Miner

Logs at volume are not read — they are mined. This skill turns a mass of log lines into a short list of patterns with counts, trends, and correlations: what is normal, what is new, what is growing, and what changed at the moment things went wrong. It works with plain tools (grep-style filtering, normalization, sorting, counting) or a log platform's query language, and its deliverable is signal ranked by novelty and growth, never a wall of matched lines.

When to use this skill

• An incident timeline needs "what changed in the logs at 14:32?"
• Error volume rose but the top error looks familiar — find what is actually new
• Reducing log noise: identifying the chatty lines that drown the useful ones
• Building a baseline of normal before a risky launch or migration

Workflow

1. Fix the frame: time window, services, severity floor. Pull a comparison window of the same shape (yesterday same hours, last week same day) — mining without a baseline finds only what you already believed.
2. Normalize lines into templates. Strip the variable parts — timestamps, identifiers, addresses, durations, counts — replacing each with a placeholder token, so "timeout after 4032ms on shard 7" and "timeout after 519ms on shard 2" collapse into one template. This single move turns millions of lines into hundreds of shapes.
3. Count templates in both windows and diff them. Three lists fall out: new templates (exist now, absent from baseline), growers (rate up by an order of magnitude or a chosen threshold), and the vanished (present before, gone now). Silence is a finding too — especially for heartbeat and success lines.
4. Rank by novelty first, volume second. A template seen for the first time ever at low volume outranks a familiar error at high volume; incidents announce themselves in new shapes more often than in louder old ones.
5. Correlate the top templates against the change timeline: deploys, config pushes, flag flips, traffic shifts. State correlations with their lag ("first occurrence ninety seconds after the rollout reached region B") and resist promoting correlation to cause without a mechanism.
6. Pull three raw exemplars per surviving template — earliest, latest, and one mid-window — with enough surrounding context (request id, trace id if present) to pivot into other systems.
7. Report the mine, not the ore: the ranked pattern table, the timeline overlay, and the noise-reduction candidates — top templates by volume with zero diagnostic value, proposed for demotion or sampling.

Output format

Window: <span> vs baseline <span>    Volume: <n> lines -> <m> templates
NEW      <count>          <template>   first seen <time>  [exemplar refs]
GROWING  <count, xN>      <template>
VANISHED <was count/hr>   <template>   last seen <time>
Correlations: <template <-> change event, with lag>
Noise candidates: <top-volume, low-value templates + suggested treatment>

Quality bar

• Every pattern claim carries counts from both windows; "lots of timeouts" is not mining.
• Templates are shown with placeholders visible, so a reader can search for them verbatim.
• At least one vanished-line check per session — the absence of success logs finds the failures that error logs miss.
• Exemplars are quoted exactly, never paraphrased; paraphrased log lines cannot be searched for.
• The report separates observation (counts, timing) from hypothesis (cause), and labels which is which.