Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
The postman claims deposits on L2 by sending a claim transaction per message from one signer. During INC-13 its claim transactions went to the peer-less RPC node and never reached the sequencer; two restarts of that node dropped them. 41 messages were left PENDING with nonces 197–237 while the chain's nonce for the signer stayed at 197. The postman chooses the next nonce as max(on-chain, highest PENDING nonce + 1), so every new claim also landed above the gap and could not be mined. The postman has a repair path for exactly this — cancel the stuck nonce with a self-transfer and reset the message — but it crashed on every attempt with TypeError: Cannot mix BigInt and other types. The 41 rows were reset by hand the way the repair path would have, the postman was restarted, and it re-claimed from nonce 197.
.large for a human; all 22 queued batches ended up parked. With no .json request left, the prover went idle at 18:10. The coordinator, seeing that the highest consecutively proven block (14831) was more than 4,000 blocks behind the next block to conflate, stopped creating requests. The two components waited on each other and finalization stayed at L2 block 14831.