L2 finalization halted for 27 hours

Trigger: batch 14832–14944, the first batch after the last finalization, needed ~10 GiB of prover memory while the node had a few GiB free (MemAvailable fell to 0.3 GiB at 17:10 and ~0.06 GiB at 17:50).

Root causes (5 whys):

1. *Why did finalization stop?* The batch after the last finalized block was never proven; finalization needs consecutive proofs.
2. *Why was it never proven?* The prover was OOM-killed on every attempt (48 restarts; kernel global_oom … Killed process (prover) anon-rss:9599592kB).
3. *Why OOM?* The node was overcommitted. 113-block batches overflow normal-mode trace limits (exit 77) and are re-proven in large mode, peaking at ~9.5–10.7 GiB. The prover requests only 6 Gi, so the scheduler and kubelet never accounted for its real peak, and after INC-08 the block explorer (~2.3 GiB) was running again alongside sequencer 2.6, l2-node-besu 1.9, L1 besu 1.7, teku 1.1 GiB … A global OOM kills the largest process: the prover.
4. *Why did it not recover on its own?* By design: the restore-once automation retries a crashed batch once in normal mode and then parks it as .large, because blind retries under memory pressure destabilise the node. The prover controller does not process .large, and the coordinator stops conflating when the proven gap exceeds 4,000 blocks (BlockCreationMonitor … gapLimit=4000). Together they formed a deadlock that only a human can break.
5. *Why did no human act for 27 hours?* Alerts (FinalizationStalled, ProverLargeRetry, settlement SLO pages, WorkflowStuck) all fired, but Alertmanager routes everything to an alert-sink that only writes a log. No person receives alerts.

Contributing factors

• INC-08 (10-11 10:15–11:37) had the same cause. Its fix covered only one gap (restore on in-container restarts). Its Next items (2) prover memory request ~10 Gi and (3) smaller batches / large mode by default were not done, and the explorer was turned back on after the recovery.
• 30 s memory scrapes miss the peak (Prometheus showed ≤ 9.5 GiB for the prover; the kernel saw it die at 9.2 GiB with the node out of memory), so dashboards understated the risk.
• ProverLargeRetry is a warning (ticket), although it always means "a human must act now".
• Recovery had a hidden time limit. The coordinator's conflated traces are generated by l2-node-besu, which can only rebuild state for the last bonsai-historical-block-limit = 1024 blocks (~1–2 h of traffic). Blocks older than that could not be traced, so a prover backlog beyond ~1024 blocks could not be worked off at all until the limit was raised during the recovery (INC-08's 1 h 22 min stall fitted inside it by luck).
• Single-prover capacity vs a block-production floor. Even with all synthetic traffic stopped, the sequencer's liveness txs produce ~460 blocks/h; one prover does ~1,130 blocks/h with 113-block batches (and ~700 with 60-block batches, its fixed cost dominates). Catching up 16,700 blocks therefore takes ~25 h, not ~4 h.

The monitoring itself worked: the SLO, the pipeline alerts and the workflow traces all pointed at the problem. The failure was routing: time to detect ≈ 26 h 50 min, of which ~22 h with critical alerts firing.

What went well

• The parking design held: no endless crash loop after the one retry per batch; nothing was lost, and the queue restarted in order from a single rename.
• The INC-08 runbook ("free memory first, then rename") was correct and fast once someone read it (~20 min).
• Observability was sufficient to diagnose without guessing: span-metric SLOs, exporter queue metrics, workflow traces (stuck=true spans), kernel logs, and component logs each confirmed one link of the chain.
• Recovery did not need a restart of any component; the coordinator resumed conflation automatically, keeping a 4,000-block window ahead of the proofs.

What went wrong

• The same incident recurred within hours because the previous postmortem's preventive items were not done.
• 22 h of critical alerts reached nobody.
• Restoring the explorer after INC-08 put the node back into the overcommitted state without anyone checking the memory headroom against the prover's peak.
• The prover's resource request (6 Gi) does not reflect its real peak (~10 Gi), so Kubernetes could not protect it.

Where we got lucky

• The halt was found during an unrelated learning session; nothing else would have surfaced it.
• The first retry succeeded with only 1.06 GiB of node memory to spare.
• The kernel always picked the prover as OOM victim. A different victim (sequencer, L1 besu, Postgres) would have turned a settlement delay into a chain or data outage.

• Prover restarts (10-11): first at 12:05, 20 by 14:10, 48 by 18:02 (kube_pod_container_status_restarts_total).
• Node MemAvailable min per 30 min (10-11): 11:40 4.9 · 12:10 2.4 · 13:40 1.9 · 16:40 4.8 · 17:10 0.3 · 17:40 1.7 GiB.
• Kernel: [10-11 17:50:15] oom-kill:constraint=CONSTRAINT_NONE … global_oom … Killed process 3777670 (prover) anon-rss:9599592kB.
• Coordinator: BlockCreationMonitor | Gap between highest consecutive proven block and L2 block is too big: lastProvenBlock=14831 nextBlockToFetch=18832 gapOverflow=4001 gapLimit=4000 (every second; nextBlockToFetch frozen at 18832 until the first proof on 10-12, then sliding: 15:17 lastProvenBlock=15509 nextBlockToFetch=19510).
• Prover: restore-once log keep 14832-14944-getZkProof.json.large: already restored once; found no jobs in the queue from 10-11 18:10:27.
• L1: last DataFinalizedV3 before the halt at L1 block 0x1665d (10-11 11:52:36), start 14719, end 14831 (645th finalization).
• Postman DB: L2_TO_L1 SENT 376, min(sent_block_number) 14848.
• How it was found: W3 withdrawal trace walk-through — { span.workflow = "withdrawal" && span.stuck = "true" }.