Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
After the security programme was rolled back (02:01–02:08, every pod restarted), an end-to-end run showed the W3 withdrawal waiting 36 min for finalization, and the gap between L2 head and finalized block kept growing (486 at 02:53, 1,087 at 04:13). Two independent causes were found.
The T3 drill stopped the prover on purpose. When it came back at 13:02 it had 4–5 execution batches waiting, and the queue never shrank: every batch waited longer than the one before (queue + prove 1351 s → 1637 s, +~40 s per batch). The coordinator cut a 60-block batch every ~5 min while one execution proof took 5–7 min, so the prover was at ≥ 100 % utilisation and had no spare capacity to catch up. The P1 change (14:32: blocks-limit 120, conflation-deadline PT12M) made batches of 113 blocks every ~9.4 min; the proof time per block fell from 6.0 s to 3.7–3.8 s because a large part of a proof is fixed cost. Utilisation dropped to ~77 % and the backlog drained by 16:08.