What is actually worth waking someone for?

tocsin groups a log stream into patterns, asks TypeSafe's Jev about each pattern once, and routes every line to page, ticket or log. This is what it cost and how well it worked across 27.6 million lines of real supercomputer logs, in two datasets.

Thunderbird sample, jev-1.13.0, priced at TypeSafe's published $0.042 per million input tokens.

How it works

Numbers, IPs, UUIDs, hex values and emails get masked. HTTP status codes keep their class, so GET /checkout 500 becomes GET /checkout <5xx> and never merges with a 200. Lines are then grouped with Drain, using the streaming engine from codag-drain. A pattern that repeats a million times is judged once.

Each new pattern goes to Jev in a single request carrying the paging policy and four typed questions.

Code owns the decision, not the model: attention = pageable × (1 − detail). Above 0.75 it pages, above 0.5 it opens a ticket, everything else is kept as a log line. Pages reach a webhook once per pattern per cooldown, carrying the count since the last alert.

Cost and speed

Two separate things make it cheap, and they are worth keeping apart. Grouping decides how many questions get asked, and that is Drain's doing, not the model's. Choosing Jev to answer them decides what each question costs.

Grouping runs at roughly 390k lines per second on one core of an M-series laptop. Jev answered in about 370 ms at the median and 450 ms at p95, called from India, with no failed calls in either run. Two independent cold runs routed 99.87% of BGL lines and 99.29% of Thunderbird lines the same way.

Accuracy, including the part that went badly

Both datasets ship with line level alert labels written by the administrators who ran those machines. The baseline is the rule most teams actually have: page on lines matching error and fatal keywords.

What happened on Thunderbird

99.7% of its alert lines come from one pattern:

kernel: <*> <*> failed, return code = -<NUM> (Fatal error (Local Catastrophic Error))

It scored 0.65, which lands in ticket, one notch below a page. The wildcards hide what actually failed, and the default policy had nothing to match on. Meanwhile the patterns it did page were these:

What was tuned where

The split matters more than the numbers, so here it is in full.

• BGL was the development set. The Drain similarity of 0.7 was chosen on its first million lines. The detail question, the default policy and the 0.75 threshold all came out of error analysis on it.
• An earlier version, before that analysis, scored F1 0.296 and PR-AUC 0.420 on BGL's held-out later 3.75M lines.
• Thunderbird stayed untouched until one final run, over a sample fixed in advance: every tenth of its 74 shards, 22.8M lines, 1.8% labeled alerts.
• Nothing was changed after that run. The numbers above come from the shipped code, re-run cold.

If you want to know whether this works for your logs, the answer is to label a sample of your own incidents and run tocsin eval against them. That path is the product.

Data and reproduction

• LogHub, the log collection both datasets come from (Zhu et al., ISSRE 2023).
• BGL on Zenodo: Blue Gene/L, 4,747,963 lines, 348,460 labeled alerts.
• Thunderbird on Hugging Face: 211M lines total, sampled to 22,833,750 for this run.
• Both logs were collected and labeled by Sandia and LLNL administrators, described in Oliner and Stearley, What Supercomputers Say (DSN 2007).

scripts/fetch-loghub.sh && scripts/fetch-thunderbird-sample.sh
tocsin eval data/bgl.tsv --out results/bgl.json
tocsin eval data/tbird-sample.tsv --out results/thunderbird.json

Limits

• Lines longer than 16 KiB are truncated before grouping.
• One process per cache file. Two writers on the same file overwrite each other's new entries.
• jev-latest is an alias. Pin a version if you need verdicts to stay fixed.
• Webhook alerts retry four times with backoff, then get logged and dropped.
• Calibration was not validated on these labels, and this report makes no claim about it.