Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
A planned rotation of the L2 web3signer keys (CHG-01) mounted the new key files under the old Secret name; web3signer restarted with 2 of its 4 keys and blob submissions paused 26 min.
The local L1 runs one beacon node and one validator with 1-second slots. At 07:30:58, with slots bunching up under slow block production, teku re-orged its head back by one slot (to EL block 42490) while besu had already imported the block of the orphaned slot (42491). From then on teku asked besu, every slot, to build on 42490. The Engine API says an EL must not start a payload build when the requested head is an ancestor of its canonical head, so besu answered VALID without a payload id; teku logged ExecutionPayloadContext is not provided and produced no block. Without a new block nothing could change: a permanent deadlock. On L2 the coordinator stopped creating batches and its /health and /metrics stopped answering; it stayed wedged after L1 came back. The alerts fired within a minute and went to the alert-sink log and the console only. A person ran the workflow check at 10:42; debug_setHead on besu (back to 42490) restarted L1 at 10:45 and a coordinator restart resumed settlement at 10:52.