Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
The Docker VM was restarted at 04:17 to reduce its memory (INC-12). On a full node restart all pods start at once and cluster DNS answers only after some seconds. l2-node-besu — the public RPC node — started at 04:17:48 and tried its only static peer, enode://…@sequencer.lineth-l2.svc.cluster.local:30303. The name did not resolve yet; Besu resolves static-peer names once at start, drops a failed one without a log line, and never retries. The RPC node ran for 49 minutes with zero peers: transactions sent to it stayed in its own pool and never reached the sequencer, so blocks carried only the sequencer's liveness transactions. Restarting the pod fixed it; a startup probe now restarts the node if it has no peer within 2 minutes.
Maru (the L2 consensus client) leaks heap: its old generation grew ~8 MiB/h from 64 MiB on 10-10 to the 256 MiB cap on 10-11 ~12:00. At 19:39 it threw and at 19:54 it stopped driving the sequencer through the engine API, so no more L2 blocks were produced. The JVM stayed alive, its readiness check was a TCP port, and there was no liveness probe — Kubernetes saw with 0 restarts. Alerts fired within minutes but only into a log. A person noticed at 00:26 while reading l2-traffic logs; restarting maru brought blocks back in ~5 s.
OutOfMemoryErrorRunning