Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Contributing factors across 16 postmortems. Select one to filter.
What the postmortems asked for, and how much is done.
56%done
Postmortems per day, by severity.
The sequencer logs ERROR LineaLivenessService Error building bundle for sequencer uptime transactions java.io.IOException: Web3Signer request was interrupted about 190 times an hour. The first guess was that ~24 % of liveness attempts failed. Measuring the outcome showed none failed: Maru re-sends forkchoiceUpdated with new payload attributes, Besu cancels the superseded block build while it is waiting for Web3Signer, and logs the interruption at ERROR. The new build signs again and succeeds; every block that needs liveness txs has them.
…_oldest_request_age_seconds{state="inprogress"} misreadDuring INC-01 the prover dashboard showed execution,inprogress = 1492.8 s, which read as a proof stuck for 25 min. The proof had actually been running for 107 s. The metric measured the age of the request file (mtime, set when the coordinator wrote it), so for an in-progress request it was queue wait plus proving time, not time since pickup.