Loading…
Loading…
Postmortems / PM-2026-10-11-09
Blameless postmortem. Incident record: incident-log.md INC-2026-10-11-09. Times UTC.
Two prover large-mode peaks at 12:00 and 12:05 brought the kind node close to zero free memory and stalled it (kubelet lease timeouts, containerd stream errors). The launchd-managed kubectl port-forward for Grafana lost its stream at 12:06:05 and logged the same error on every request for 12 hours. kubectl port-forward does not exit on these errors, and launchd's KeepAlive only restarts a process that exits — so the process stayed alive and the forward stayed dead.
| what | how much |
|---|---|
| lineth Grafana on 127.0.0.1:23000 | unreachable ~12 h (1921 failed requests in the forward log) |
| not affected | the Grafana pod (0 restarts, Ready), Prometheus, Loki, Tempo, alert evaluation |
kubectl port-forward survives a broken API connection in a wedged state.None automatic: nothing watched the local port. Found while writing scripts/labq.sh (23000 → HTTP 000) and root-caused at 23:59.
error creating error stream for port 23000 -> 3000: Timeout occurredkill -STOP on the kubectl childFix: launchd runs scripts/grafana-forward.sh, which starts the forward, checks /api/health through it every 30 s and exits after 3 failures in a row. Old plist backed up at ~/.local/state/grafana-forward/io.lineth.grafana-forward.plist.bak-20261012.
the fix was tested by injecting the failure (SIGSTOP), not only by restarting.
a supervisor that checks liveness instead of function gives false confidence — the same lesson as INC-13 (Besu up with 0 peers) and INC-10 (maru up with a full heap).
—
Ad-hoc 2xxxx forwards from labq.sh can wedge the same way: pkill -f "port-forward.*<port>" before re-creating
open (documented)
Remove the trigger: node memory headroom for the prover
done 10-13 (INC-12: normal-mode 80-block batches, peak ~9.4 GiB, no large-mode retry; VM 30 GiB with host headroom)
Health-checked forward wrapper under launchd
Forward log: 1921 × error creating error stream for port 23000 -> 3000 from 10-11 12:06:05 to 10-12 00:00. Also seen then: tempo-0 OOMKilled 23:49:37 (its own ~1 GiB limit, not node memory).