Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
Two prover large-mode peaks at 12:00 and 12:05 brought the kind node close to zero free memory and stalled it (kubelet lease timeouts, containerd stream errors). The launchd-managed kubectl port-forward for Grafana lost its stream at 12:06:05 and logged the same error on every request for 12 hours. kubectl port-forward does not exit on these errors, and launchd's KeepAlive only restarts a process that exits — so the process stayed alive and the forward stayed dead.
.largeAt 10:15 the kernel's global OOM killer killed the prover (RSS 10.2 GiB, below its 11 Gi limit) while it proved batch 13589-13701: the kind node had run out of memory, not the container. The prover's recovery renamed the dangling job to .large for a human and moved on to later batches, so finalization stayed at 13588. A retry by hand died the same way. Freeing ~2.4 GiB (block explorer off) let the batch prove at 11:31, and finalization caught up by 11:37. The automatic one-time restore of jobs did not run because it lived in an initContainer, which runs on pod start, not on a container restart.
.large