Demo data Sample data for a portfolio demo; not real incidents.

3 alerts firingLinethL2MessagesStuck (warning)Open incident #3

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

Postmortems / PM-2026-10-10-01

SEV3INC-2026-10-10-01 · 2026-10-10

prover backlog never drained after the T3 prover outage

Draft
Reviewed
Closed

Impact

backlog from the T3 drill (prover back 13:02) → drained 16:08 (~3 h)

Severity

settlement latency grew without bound; no workflow stopped, budgets still 100 %

Components

provercoordinator

Authors

s1ns3nz0, Claude

Blameless postmortem. Incident record: incident-log.md INC-2026-10-10-01. Times UTC. Block numbers refer to the chain before the 2026-10-12 reset (archived).

1Summary

The T3 drill stopped the prover on purpose. When it came back at 13:02 it had 4–5 execution batches waiting, and the queue never shrank: every batch waited longer than the one before (queue + prove 1351 s → 1637 s, +~40 s per batch). The coordinator cut a 60-block batch every ~5 min while one execution proof took 5–7 min, so the prover was at ≥ 100 % utilisation and had no spare capacity to catch up. The P1 change (14:32: blocks-limit 120, conflation-deadline PT12M) made batches of 113 blocks every ~9.4 min; the proof time per block fell from 6.0 s to 3.7–3.8 s because a large part of a proof is fixed cost. Utilisation dropped to ~77 % and the backlog drained by 16:08.

2Impact

what	how much
settlement latency (head − finalized)	570 blocks at 14:58 and growing; structural minimum ~170–230
W1 rollup, W3 withdrawal	slower in proportion; nothing stuck
SLOs	`withdrawal-1h` and `settlement-freshness` at risk, budgets still 100 %
not affected	L2 block production, user txs, deposits

3Root causes and trigger

Trigger: the planned T3 prover outage left a backlog.
Root cause: capacity. Batch interval (~5 min) ≤ proof time (5–7 min) means utilisation ≥ 1; a queue with utilisation ≥ 1 never drains, it only grows. The proof cost is dominated by a fixed part, so small batches were the most expensive way to prove a block.
Nothing in the configuration related batch size to prover throughput; the 60-block default came from upstream settings sized for production hardware.

4Detection

LinethL2ProverBacklog, LinethL2BlobSubmissionStalled, LinethL2WorkflowStuck and lineth_prover_oldest_request_age_seconds showed it while the drill results were being watched, so detection was immediate (a human was looking). The alerts themselves only reach the alert-sink log.

5Response and recovery

13:02
prover back after T3; backlog 4–5 batches
14:29
queue + prove per batch 1351 s → 1637 s: the backlog grows
14:32
P1: coordinator blocks-limit 120, conflation-deadline PT12M → 113-block batches (cut by the trace limit) every ~9.4 min
14:38
orphaned 60-block requests found and moved (INC-02)
14:58
execution backlog 3, finalization lag 570 blocks
15:58, 16:08
execution pending 0, head − finalized 156–165 blocks two checks in a row; queue wait per batch 1637 s → 66 s; LinethL2ProverBacklog cleared

Measured after the change: 9 batches of 113 blocks took 352–441 s (mean ~410 s, 3.7 s/block vs 6.0 s/block before).

6Lessons learned

Went well

the fix came from measurement (per-block proof cost), not guessing; the backlog drained as predicted.

Went badly

batch size was a throughput setting nobody had sized; a single drill was enough to tip it over.

Lucky

—

7Action items

P2
Second prover (or a pool per proof type) for catch-up capacity
open (INC-04, INC-11 A11)
Prevents1ns3nz0Open
P3
Recompute settlement baselines after 24 h
open — redo after the 10-12 reset and the 10-13 batch change
Processs1ns3nz0Open
P1
Keep blocks-limit 120 / deadline 12 m as the lab default
done — superseded 10-13 by BLOCK_HASH = 1440 (80-block batches, INC-12)
Prevents1ns3nz0✓ Done
P2
Size batch changes against memory and trace limits, not only proof time
done in INC-12 (model in traces-limits-v5.toml)
Processs1ns3nz0✓ Done

8Supporting data

Proof times before/after P1 and queue waits: incident record.
Utilisation: 410 s per 113-block batch every ~9.4 min (564 s) ≈ 0.73–0.77.

Loading…

1Summary

what

how much

settlement latency (head − finalized)

570 blocks at 14:58 and growing; structural minimum ~170–230

W1 rollup, W3 withdrawal

slower in proportion; nothing stuck

SLOs

withdrawal-1h and settlement-freshness at risk, budgets still 100 %

not affected

L2 block production, user txs, deposits