Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
The sequencer logs ERROR LineaLivenessService Error building bundle for sequencer uptime transactions java.io.IOException: Web3Signer request was interrupted about 190 times an hour. The first guess was that ~24 % of liveness attempts failed. Measuring the outcome showed none failed: Maru re-sends forkchoiceUpdated with new payload attributes, Besu cancels the superseded block build while it is waiting for Web3Signer, and logs the interruption at ERROR. The new build signs again and succeeds; every block that needs liveness txs has them.
On 10-10 at 14:34:18 the coordinator logged eth_sendRawTransaction for aggregation finalization failed: aggregation=[3901..3960], right after it had been restarted for the P1 change. The next attempt landed and finalization reached 3960. The exact JSON-RPC error was not captured, so the cause stayed open with the action "capture the full error next time". On 10-13 the full errors were pulled from Loki for the whole period since the 10-12 chain reset: every such line is one of two benign duplicates.