Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
The local L1 runs one beacon node and one validator with 1-second slots. At 07:30:58, with slots bunching up under slow block production, teku re-orged its head back by one slot (to EL block 42490) while besu had already imported the block of the orphaned slot (42491). From then on teku asked besu, every slot, to build on 42490. The Engine API says an EL must not start a payload build when the requested head is an ancestor of its canonical head, so besu answered VALID without a payload id; teku logged ExecutionPayloadContext is not provided and produced no block. Without a new block nothing could change: a permanent deadlock. On L2 the coordinator stopped creating batches and its /health and /metrics stopped answering; it stayed wedged after L1 came back. The alerts fired within a minute and went to the alert-sink log and the console only. A person ran the workflow check at 10:42; debug_setHead on besu (back to 42490) restarted L1 at 10:45 and a coordinator restart resumed settlement at 10:52.
The postman claims deposits on L2 by sending a claim transaction per message from one signer. During INC-13 its claim transactions went to the peer-less RPC node and never reached the sequencer; two restarts of that node dropped them. 41 messages were left PENDING with nonces 197–237 while the chain's nonce for the signer stayed at 197. The postman chooses the next nonce as max(on-chain, highest PENDING nonce + 1), so every new claim also landed above the gap and could not be mined. The postman has a repair path for exactly this — cancel the stuck nonce with a self-transfer and reset the message — but it crashed on every attempt with TypeError: Cannot mix BigInt and other types. The 41 rows were reset by hand the way the repair path would have, the postman was restarted, and it re-claimed from nonce 197.