Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
Two prover large-mode peaks at 12:00 and 12:05 brought the kind node close to zero free memory and stalled it (kubelet lease timeouts, containerd stream errors). The launchd-managed kubectl port-forward for Grafana lost its stream at 12:06:05 and logged the same error on every request for 12 hours. kubectl port-forward does not exit on these errors, and launchd's KeepAlive only restarts a process that exits — so the process stayed alive and the forward stayed dead.
The T3 drill stopped the prover on purpose. When it came back at 13:02 it had 4–5 execution batches waiting, and the queue never shrank: every batch waited longer than the one before (queue + prove 1351 s → 1637 s, +~40 s per batch). The coordinator cut a 60-block batch every ~5 min while one execution proof took 5–7 min, so the prover was at ≥ 100 % utilisation and had no spare capacity to catch up. The P1 change (14:32: blocks-limit 120, ) made batches of 113 blocks every ~9.4 min; the proof time per block fell from 6.0 s to 3.7–3.8 s because a large part of a proof is fixed cost. Utilisation dropped to ~77 % and the backlog drained by 16:08.
conflation-deadline PT12M