Prover killed by memory pressure, batches parked as .large
criticalreviewed 2026-10-13What it meansmoreThe prover was killed while proving a batch (almost always the node's global OOM killer: a 113-block batch peaks at ~10 GiB in large mode, above the prover's 6 Gi request). The restore-once automation retries a crashed batch once, then parks it as <range>-getZkProof.json.large. The prover controller does not pick up .large files, so the batch waits for a human. Finalization needs consecutive proofs, so it stops at the batch before the parked one.
ImpactmoreL2 → L1 finalization stops; withdrawals (L2 → L1 claims) wait. Block production, deposits and user txs are not affected. SEV2 once LinethL2FinalizationStalled fires (settlement-freshness pages). The longer it waits, the harder the catch-up: one prover does ~1,130 blocks/h against ~460–575 produced, and blocks older than bonsai-historical-block-limit cannot be traced at all (INC-11). Every .large file needs a human now, even though LinethL2ProverLargeRetry is a warning.
Quick mitigation0/6
Free memory first, then requeue. Renaming without freeing memory gets the batch killed again (INC-08, 11:19).
flux suspend kustomization lineth-l2-explorer and scale blockscout, blockscout-frontend to 0 (~2.4 GiB)<range>-getZkProof.json.large → .json in the prover requests volumeflux suspend kustomization lineth-l2-traffic) to shorten the catch-upDiagnosis6 queries
Parked batches and the oldest pending request:
sum by (state) (lineth_prover_requests)Prover restarts and OOM kills in the last 2 hours:
max(max_over_time(kube_pod_container_status_last_terminated_reason{namespace="lineth-l2", pod=~"prover-.*", container="prover", reason="OOMKilled"}[2h])) or vector(0)Node memory headroom (the prover needs ~10 GiB free while proving a large batch):
min(node_memory_MemAvailable_bytes) / 1024^3How far finalization is behind the head:
max(lineth_l2_head_block) - max(lineth_l2_rollup_finalized_block{l1_block_tag="latest"})Prover and coordinator logs (the coordinator stops conflating when the proven gap exceeds 4,000 blocks):
{namespace="lineth-l2", pod=~"prover-.*"} |~ "(?i)oom|killed|large|no jobs"{namespace="lineth-l2", pod=~"coordinator-.*"} |= "gapOverflow"The kernel log on the kind node confirms a global OOM (docker exec lineth-local-control-plane dmesg | grep -i oom).
Fix0/3
blocks-limit or run large mode by default (PM-11 A3)Escalation
Solo operator: page yourself via the SEV2 flow. If the parked batch dies again after memory was freed, or the proven gap approaches bonsai-historical-block-limit, treat it as SEV1 risk to settlement and stop all synthetic traffic before anything else.
Seen in INC-2026-10-11-08, INC-2026-10-11-11 · SLOs settlement-freshness, withdrawal-1h · owner s1ns3nz0