Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
The postman claims deposits on L2 by sending a claim transaction per message from one signer. During INC-13 its claim transactions went to the peer-less RPC node and never reached the sequencer; two restarts of that node dropped them. 41 messages were left PENDING with nonces 197–237 while the chain's nonce for the signer stayed at 197. The postman chooses the next nonce as max(on-chain, highest PENDING nonce + 1), so every new claim also landed above the gap and could not be mined. The postman has a repair path for exactly this — cancel the stuck nonce with a self-transfer and reset the message — but it crashed on every attempt with TypeError: Cannot mix BigInt and other types. The 41 rows were reset by hand the way the repair path would have, the postman was restarted, and it re-claimed from nonce 197.
l2-node-besu — the public RPC node — started at 04:17:48 and tried its only static peer, enode://…@sequencer.lineth-l2.svc.cluster.local:30303. The name did not resolve yet; Besu resolves static-peer names once at start, drops a failed one without a log line, and never retries. The RPC node ran for 49 minutes with zero peers: transactions sent to it stayed in its own pool and never reached the sequencer, so blocks carried only the sequencer's liveness transactions. Restarting the pod fixed it; a startup probe now restarts the node if it has no peer within 2 minutes.