Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
After the security programme was rolled back (02:01–02:08, every pod restarted), an end-to-end run showed the W3 withdrawal waiting 36 min for finalization, and the gap between L2 head and finalized block kept growing (486 at 02:53, 1,087 at 04:13). Two independent causes were found.
Two prover large-mode peaks at 12:00 and 12:05 brought the kind node close to zero free memory and stalled it (kubelet lease timeouts, containerd stream errors). The launchd-managed kubectl port-forward for Grafana lost its stream at 12:06:05 and logged the same error on every request for 12 hours. kubectl port-forward does not exit on these errors, and launchd's KeepAlive only restarts a process that exits — so the process stayed alive and the forward stayed dead.