Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
After the security programme was rolled back (02:01–02:08, every pod restarted), an end-to-end run showed the W3 withdrawal waiting 36 min for finalization, and the gap between L2 head and finalized block kept growing (486 at 02:53, 1,087 at 04:13). Two independent causes were found.
On 10-11 the prover began running out of memory on every attempt to prove the L2 block range right after the last finalized block (14832–14944). Each 113-block batch needs ~10 GiB in the prover; the kind node (27.4 GiB) had less than that free, so the kernel's global OOM killer killed the prover 48 times between 12:05 and 18:02. The prover's automatic recovery retries a crashed batch once and then parks it as .large for a human; all 22 queued batches ended up parked. With no .json request left, the prover went idle at 18:10. The coordinator, seeing that the highest consecutively proven block (14831) was more than 4,000 blocks behind the next block to conflate, stopped creating requests. The two components waited on each other and finalization stayed at L2 block 14831.