Loading…
Loading…
Postmortems / PM-2026-10-13-12
Blameless postmortem. Incident record: incident-log.md INC-2026-10-13-12. Times UTC.
After the security programme was rolled back (02:01–02:08, every pod restarted), an end-to-end run showed the W3 withdrawal waiting 36 min for finalization, and the gap between L2 head and finalized block kept growing (486 at 02:53, 1,087 at 04:13). Two independent causes were found.
BLOCK_HASH module 18 rows per block, so 113 blocks = 2034 ≤ 2048. The prover counts the same module as 6 × (n + 257) rows (a fixed 256-block window), so 113 blocks = 2220 > 2048: every batch failed normal mode after ~21 s and was re-proven in large mode (~9 min, ~10 GiB). Loki shows 7–8 such overflows per hour since at least 10-12 18:00 — this was the hidden root cause of INC-08 and INC-11, not new.BLOCK_HASH was lowered to 1440 (80-block batches, prover 2022 < 2048: normal mode) and the VM to 30 GiB. Proofs dropped to ~3 min 20 s per batch, finalization ran at ~1,440 blocks/h against ~860–960 produced, and the lag was back to its structural minimum (~50–160 blocks) by ~05:23.
| what | how much |
|---|---|
| settlement lag (head − L1 finalized) | ~100–190 blocks on 10-12 evening → 1,087 at 04:13 → 51–109 from ~05:23 |
| W3 withdrawals | waited longer in proportion (the E2E withdrawal: 36 min) |
| W1 rollup | slower; never stopped |
| SLOs | settlement-freshness burn-rate ticket fired |
| not affected | L2 block production, deposits, user txs (until INC-13, a separate incident) |
Trigger: the rollback restarted the prover; afterwards large proofs ran slower (5 min 44 s on 10-12 22:09 vs 9 min 6 s on 10-13 02:22), so the lag that large mode had always cost now grew.
Cause 1 — trace-limit mismatch (pre-existing).
blockhash; its normal limit is 2048.traces-limits-v5.toml BLOCK_HASH = 2048) is checked against the besu line counters, 18 per block: 2034 for 113 blocks.BLOCKHASH.Cause 2 — host memory overcommit.
kernel_task ~198 % CPU, ~178k decompressions/s.Found by hand during the post-rollback E2E run (W3 slow), then by the prover log ([LIMIT OVERFLOW] … module 'blockhash' … requested=2220 limit=2048). LinethL2ProverBacklog and LinethL2SLOBurnRateTicket were firing into the alert-sink log. The host memory cause was found only by measuring on the Mac (vm_stat, top, sysctl vm.swapusage).
requested=2220)6e6b138 BLOCK_HASH 1800 (100-block batches); re-conflation from 6112; 113-block orphans → orphans-20261013-bh1800/the investigation separated "always true" from "new" (Loki history showed the overflow on every batch since 10-12); the first numeric fix was tested on one batch before relying on it, and the wrong model was corrected from two measurements; orphaned requests were moved, not deleted; in-flight batches were allowed to finish before each restart.
the overflow had been logged 7–8 times an hour for days and was treated as normal ("every 113-block batch goes large", INC-08 Next 3); the 10-12 VM memory increase was sized for the VM, not for the host.
the l2-node-besu history limit had been raised to 20,000 in INC-11, so re-conflation far behind head was possible.
Host memory pressure check (Mac vm_stat decompressions / swap) with a threshold — nothing inside the VM sees it
Alert on prover normal-mode overflows (LIMIT OVERFLOW rate > 0): a regression of A1 must be visible
Review blocks-limit (no longer binding) and conflation-deadline for 80-block batches; update INC-11 throughput notes
Prover scale-out only with host headroom: 2 replicas, WORKER_ID from the pod name, lower GOMAXPROCS and limits each
|= "LIMIT OVERFLOW"): 7–8 every hour from 10-12 18:00 to 10-13 03:00.linea_getBlockTracesCountersV2 BLOCK_HASH = 18 for every block in 5434–5772.sysctl vm.swapusage 9.3/10 GB; decompressions 178k/s before, 41–65k/s after; swap-ins 60–101/s → 7/s.3572a7e BLOCK_HASH 1440 (80 blocks); re-conflation from 6212; 100-block orphans → orphans-20261013-bh1440/Watch the gap back to 1–2 batches
done (~05:23)
Prover memory request to the normal-mode peak (80-block batches: working set 9.44 GiB; the 7.3 GiB of INC-11 was a 60-block batch)
10-13 1fb7983 (10 Gi)
BLOCK_HASH = 1440 with the measured model in the config comment
done (6e6b138, 3572a7e)
Docker VM 30 GiB
done (settings backup in the session scratchpad)
Route alerts to a person
10-13 11:00 934a2bb: Telegram receiver on lab alerts (critical + warning, repeat 1 h); see INC-15