Loading…
Loading…
Postmortems / PM-2026-10-10-05
…_oldest_request_age_seconds{state="inprogress"} misreadBlameless postmortem. Incident record: incident-log.md INC-2026-10-10-05. Times UTC.
During INC-01 the prover dashboard showed execution,inprogress = 1492.8 s, which read as a proof stuck for 25 min. The proof had actually been running for 107 s. The metric measured the age of the request file (mtime, set when the coordinator wrote it), so for an in-progress request it was queue wait plus proving time, not time since pickup.
| what | how much |
|---|---|
| system | none |
| operator | a false "stuck proof" reading during an incident; could have caused a needless prover restart |
The metric's name and HELP text said "age" without saying from which event. The prover renames the request to .inprogress on pickup, which updates the file's ctime, not its mtime; the exporter used mtime.
The operator cross-checked against the prover log (pickup 107 s earlier) before acting.
lineth_prover_inprogress_seconds{proof} = now − ctime of the .inprogress file (commit 6264bf2); old metric's HELP now says "queue wait + proving". First reading for the same request: old 16825 s, new 35 sthe reading was checked against a second source before acting.
a metric whose meaning depends on file-system details was shipped without stating the start event. - A duration metric must say which two events it spans (same rule later used for the W4 trace stages).
—
Use it in the prover dashboard and a "proof running too long" alert
open — checked 10-13: no rule or dashboard uses lineth_prover_inprogress_seconds yet
New metric (time since pickup)
done (6264bf2)
Old vs new metric for the same request: 16825 s vs 35 s (10-11 01:30).