Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
The postman claims deposits on L2 by sending a claim transaction per message from one signer. During INC-13 its claim transactions went to the peer-less RPC node and never reached the sequencer; two restarts of that node dropped them. 41 messages were left PENDING with nonces 197–237 while the chain's nonce for the signer stayed at 197. The postman chooses the next nonce as max(on-chain, highest PENDING nonce + 1), so every new claim also landed above the gap and could not be mined. The postman has a repair path for exactly this — cancel the stuck nonce with a self-transfer and reset the message — but it crashed on every attempt with TypeError: Cannot mix BigInt and other types. The 41 rows were reset by hand the way the repair path would have, the postman was restarted, and it re-claimed from nonce 197.
OutOfMemoryError and at 19:54 it stopped driving the sequencer through the engine API, so no more L2 blocks were produced. The JVM stayed alive, its readiness check was a TCP port, and there was no liveness probe — Kubernetes saw Running with 0 restarts. Alerts fired within minutes but only into a log. A person noticed at 00:26 while reading l2-traffic logs; restarting maru brought blocks back in ~5 s.