Loading…
Loading…
Postmortems / PM-2026-10-10-04
Blameless postmortem. Incident record: incident-log.md INC-2026-10-10-04. Times UTC. Block numbers refer to the chain before the 2026-10-12 reset (archived).
At 14:46:54 the coordinator created the aggregation request for 3961-4073, three seconds after the single prover had started execution proof 4074-4186. The prover is not pre-emptive: the aggregation wins the next pick but waits for the running proof, so finalization stayed at 3960 until ~14:54. This happens to every aggregation whose request arrives while an execution proof runs.
| what | how much |
|---|---|
| finalization latency | + one execution proof per aggregation: ~7 min with 113-block batches then; ~3 min 20 s with 80-block batches since 10-13 (INC-12) |
| W1 rollup / W3 withdrawal | slower by the same amount |
| not affected | correctness, block production, deposits |
One prover process takes one job at a time. Job order is Score = 100 * endBlock + priority, so the aggregation is picked next, but a running execution proof is never interrupted. The production pattern is separate prover pools per proof type (execution vs. compression + aggregation), which the lab does not run.
lineth_l2_rollup_finalized_block flat for 25 min and LinethL2WorkflowStuck (rollup) while watching INC-01.
No action: accepted as a design limit. 10-13: smaller batches (INC-12) roughly halved the wait.
the scheduling rule was read from the prover, not guessed. - Latency SLOs for settlement must include one execution proof of queueing as their floor.
—
—
Second, small prover for compression + aggregation (or pools per proof type)
open — needs host memory headroom (INC-12 A3)
Settlement SLO thresholds: include one execution proof of queueing
Coordinator request time 14:46:54 vs prover pickup of 4074-4186 three seconds earlier; finalization flat until ~14:54.