Loading…
Loading…
Postmortems / PM-2026-10-13-13
Blameless postmortem. Incident record: incident-log.md INC-2026-10-13-13. Times UTC.
The Docker VM was restarted at 04:17 to reduce its memory (INC-12). On a full node restart all pods start at once and cluster DNS answers only after some seconds. l2-node-besu — the public RPC node — started at 04:17:48 and tried its only static peer, enode://…@sequencer.lineth-l2.svc.cluster.local:30303. The name did not resolve yet; Besu resolves static-peer names once at start, drops a failed one without a log line, and never retries. The RPC node ran for 49 minutes with zero peers: transactions sent to it stayed in its own pool and never reached the sequencer, so blocks carried only the sequencer's liveness transactions. Restarting the pod fixed it; a startup probe now restarts the node if it has no peer within 2 minutes.
| what | how much |
|---|---|
| W4 user-tx, W1 transfers, ERC-20 transfers, withdrawals sent via the RPC node | not included for ~49 min; 67 txs in the RPC pool; generators gave up after 3 attempts (not mined, Replacement transaction underpriced) |
| W2 deposits | claim txs from the postman (sent to the RPC node) were lost → INC-14 |
| L2 head | +205 blocks in 30 min (liveness blocks only) instead of ~430 |
| not affected | sequencer, block production, finalization (W1 rollup) |
Temporary failure in name resolution at 04:17:52–57).LinethL2NoPeers (critical, l2-node-besu and l2-sequencer) fired at 04:20:38 and LinethL2RpcTxStuck at 04:23:53 — into the alert-sink log. Found by hand at 04:55 while measuring INC-12: L2 head growth was half the usual and the traffic generators logged failures.
Connecting to 1 static nodes — never peersLinethL2NoPeers fires (unseen)net_peerCount 0 on both nodessequencer.lineth-l2.svc.cluster.local → 10.96.18.95); l2-node-besu-0 deletedok7bf79fc: startupProbe GET /readiness?minPeers=1 (Besu built-in; 200 with ≥ 1 peer, 503 otherwise), 24 × 5 s; new pod Ready with a peer in 6 sThe check confirming recovery was itself delayed ~10 min by a broken shell wait loop in the operator's tooling.
the other components were checked for the same "resolve once" pattern from their logs before choosing a fix; the fix uses a built-in endpoint and covers any cause of "no peer", not only DNS.
a planned restart (fix for INC-12) had no post-restart check of user-facing workflows; the second incident in two days caused by "started before DNS" (web3signer in INC-11, this one); the alert was correct and nobody saw it.
—
Check other components for once-at-start name resolution
done (only l2-node-besu affected)
After any full node restart: run the W1–W5 checks before calling it done (runbook)
— docs/runbooks/node-restart.md + scripts/check-workflows.sh (52b00b5)
Fix the static-peer comment
done (7bf79fc)
l2-node-besu startupProbe /readiness?minPeers=1 within 2 min
done (7bf79fc)
Route alerts to a person
10-13 11:00 934a2bb: Telegram receiver on lab alerts (critical + warning, repeat 1 h); see INC-15
l2-node-besu log 04:17:48: Connecting to 1 static nodes; no later peer lines until the restart./readiness?minPeers=1 → 200, /readiness?minPeers=2 → 503, /liveness → 200.