Loading…
Loading…
Postmortems / PM-2026-10-11-08
.largeBlameless postmortem. Incident record: incident-log.md INC-2026-10-11-08. Times UTC. Block numbers refer to the chain before the 2026-10-12 reset (archived).
At 10:15 the kernel's global OOM killer killed the prover (RSS 10.2 GiB, below its 11 Gi limit) while it proved batch 13589-13701: the kind node had run out of memory, not the container. The prover's recovery renamed the dangling job to .large for a human and moved on to later batches, so finalization stayed at 13588. A retry by hand died the same way. Freeing ~2.4 GiB (block explorer off) let the batch prove at 11:31, and finalization caught up by 11:37. The automatic one-time restore of .large jobs did not run because it lived in an initContainer, which runs on pod start, not on a container restart.
| what | how much |
|---|---|
| finalization | stalled at 13588 for ~1 h 22 min; head − finalized 837 blocks at 11:11 |
| W1 rollup, W3 withdrawal | delayed by the same time |
| SLOs | settlement-freshness budget spent (14.4x/1 h page fired) |
| not affected | block production, user txs, deposits; no data lost |
blockhash module counts 6 × (n + 257) rows, so any batch over 84 blocks overflows.dmesg had the truth.restore-large-once was an initContainer; a container restart inside the same pod skips it.Alerts fired in order of cause to effect: LinethL2ProverLargeRetry 10:21:29, LinethL2FinalizationStalled 10:27:59, LinethL2SLOBurnRatePage 10:47:58, then BlobSubmissionStalled, MessagesStuck, WorkflowStuck. They only reach the alert-sink log; a person saw the stall at 11:11 during alert practice (C), ~56 min after it began.
.large, continues with 13702+.large → .json by hand (runbook step 2); normal mode exit 77 → large modeflux suspend kustomization lineth-l2-explorer, blockscout + frontend → 0; batch renamed backthe alerts told the right story in the right order; freeing memory was a fast, reversible lever.
nobody receives alerts; the retry procedure did not consider that the same peak would recur; the memory request hid the real peak from Kubernetes.
it happened during a practice session, so a person was at the keyboard within the hour. The next occurrence (INC-11) had no such luck and lasted 27 h.
Restore .large on container restarts too (in the prover container command)
done 10-11
Prover memory request near its real peak (was 6 Gi vs ~10 GiB)
10-13 1fb7983: request 10 Gi (normal-mode peak 9.44 GiB working set); node requests 24 of 29.3 GiB
Stop batches overflowing normal mode (smaller batches or large mode by default)
done 10-13 in INC-12 (BLOCK_HASH = 1440, 80-block batches) — not done before INC-11, which repeated this incident
Route alerts to a person
10-13 11:00 934a2bb: Telegram receiver on lab alerts (critical + warning, repeat 1 h); see INC-15
dmesg on the kind node: global_oom … Killed process (prover) anon-rss:10213936kB (10:15), anon-rss:10693008kB (11:19)..large-restored had no entry for 13589 (initContainer did not run).