Prover requests failing

Probable cause prover · affects coordinator

Detected
Declared
Mitigated
Resolved
Closed

…

open for

alerts, all cleared

7/20

checklist

…

next update

acked 2m

paging

Batch 18412-18525 proved in 11 min on the old heap; 9 parked batches requeued
10-13 18:17 UTC · bob
AI briefing v1: Prover was OOM-killed on every large batch for ~2 h after the heap change (CHG-02); rolled back, catch-up running. (2 next steps, see Response)
10-13 17:27 UTC · AI
Issue #1002 created (mock) for: Load-test heap changes against .large batches before rollout
10-13 17:19 UTC · alice
Follow-up added (P1 detect): Page when the prover restarts more than twice in 30 min
10-13 17:19 UTC · alice
Follow-up added (P1 prevent): Load-test heap changes against .large batches before rollout
10-13 17:19 UTC · alice
Ops added to checklist: Resume blockscout after catch-up
10-13 17:19 UTC · alice
Ops ✓ Record each mitigation as a decision, with when to revert it
How: rollback, then suspend blockscout
10-13 17:19 UTC · alice
Comms ✓ Post the first status update
How: status page: withdrawals delayed
10-13 17:19 UTC · alice
Ops ✓ Apply the runbook's quick mitigation and post the result as an update
How: rolled back the heap, requeued 9 batches
10-13 17:19 UTC · alice
Ops ✓ Open the runbook for each firing alert
How: prover-oom-loop
10-13 17:19 UTC · alice
Ops ✓ Check recent changes (open CHGs, Flux reconciles) on the affected components
How: CHG-02 heap tuning 45 min before the first OOM: the suspect
10-13 17:19 UTC · alice
IC ✓ Assign the Ops and Comms leads, or note that you hold them
How: bob ops, carol comms
10-13 17:19 UTC · alice
IC ✓ Confirm the severity and write the impact in Current status
How: FinalizationStalled + ProverFailures, finalized head stopped at 18,410
10-13 17:19 UTC · alice
summary: Catch-up running: finalized head 840 blocks behind, ETA 2 h
10-13 17:19 UTC · alice
Checkpoint: Catch-up running: finalized head 840 blocks behind, closing ~6 blocks/min
Wake when max(changes(lineth_l2_rollup_finalized_block{l1_block_tag="latest"}[30m])) == 0
Handed to dave
10-13 17:19 UTC · alice
Suspend blockscout to give the prover memory during catch-up
Why
frees 3.1 GiB; catch-up ~2.5 h
Instead of
catch up with blockscout running (~6 h)
Revert when
finalized head within 100 blocks of the L2 head
10-13 16:15 UTC · alice
Deployed Kustomization lineth-l2-proving main@b7d03f1
Deployment/lineth-l2/prover configured
10-13 15:46 UTC · flux
Flux failed Kustomization lineth-l2-proving main@b7d03f1
Deployment/lineth-l2/prover dry-run failed: spec.template.spec.containers[0].env[2].value: Invalid value: "-Xmx20g "
10-13 15:39 UTC · flux
state declared → mitigated
10-13 15:31 UTC · alice
✓ Free memory: `flux suspend kustomization lineth-l2-explorer` and scale `blockscout`, `blockscout-frontend` to 0 (~2.4 GiB) (prover-oom-loop)
10-13 14:10 UTC · bob
✓ Check node memory headroom against the prover's ~10 GiB peak (query in Diagnosis) (prover-oom-loop)
10-13 14:10 UTC · bob
Checkpoint: Rollback PR open; prover still OOM on large batches
Wake when increase(kube_pod_container_status_restarts_total{namespace="lineth-l2",container="prover"}[15m]) > 1
10-13 14:10 UTC · alice
Roll back the prover heap tuning (CHG-02)
Why
OOMs started 45 min after -Xmx went to 23g; the 24 GiB limit leaves no off-heap room
Instead of
raise the container limit to 28 GiB (node has no headroom)
Revert when
large batches prove under the old heap
10-13 14:01 UTC · alice
prover restartCount 6, lastState OOMKilled at 21.9 of 24 GiB; 9 batches parked as .large
10-13 13:55 UTC · bob
summary: Prover OOM-killed on every large batch since the heap change
10-13 13:43 UTC · alice
comms: carol — withdrawal delay notice every hour
10-13 13:43 UTC · alice
ops: bob — prover memory and batch requeue
10-13 13:43 UTC · alice
Acknowledged the page, 2m after it was sent
10-13 13:38 UTC · alice
Paged via ntfy (step 1, mock)
10-13 13:36 UTC · paging
state candidate → declared
10-13 13:36 UTC · alice
SEV — → 2
10-13 13:36 UTC · alice
Deployed Kustomization lineth-l2-proving main@4e1c9a2
Deployment/lineth-l2/prover configured
10-13 12:32 UTC · flux

Loading…

Prover requests failing

Mark resolved?

Prover requests failing

Mark resolved?

Record a decision

Post an update