Loading…
Loading…
Postmortems / PM-2026-10-10-07
Blameless postmortem. Incident record: incident-log.md INC-2026-10-10-07. Times UTC. Block numbers refer to the chain before the 2026-10-12 reset (archived).
The sequencer logs ERROR LineaLivenessService Error building bundle for sequencer uptime transactions java.io.IOException: Web3Signer request was interrupted about 190 times an hour. The first guess was that ~24 % of liveness attempts failed. Measuring the outcome showed none failed: Maru re-sends forkchoiceUpdated with new payload attributes, Besu cancels the superseded block build while it is waiting for Web3Signer, and logs the interruption at ERROR. The new build signs again and succeeds; every block that needs liveness txs has them.
| what | how much |
|---|---|
| liveness txs | none missing: of 2000 blocks, 580 were > 8 s after their parent and 580/580 carry the liveness txs |
| operator | noisy ERRORs; a log-based alert on them would have fired forever |
Besu interrupts the superseded build (New payloadId received so cancelling block creation tasks); 33 of 43 errors fell within 50 ms of such a cancel. The plugin logs the cancellation at ERROR instead of checking the interrupt cause. Web3Signer itself signs in ~1 ms (p99 6 ms) — it was never the bottleneck.
Read in the besu logs at 10-11 08:30. kubectl logs only reached back to 08:11 (rotation); Loki had the history.
Fix: l2-exporter metrics lineth_l2_liveness_needed_blocks_total / lineth_l2_liveness_included_blocks_total (on-chain scan of every sequencer block), alert LinethL2LivenessTxMissing (< 99 % over 1 h, for 15 m), L2 Chain panels "Liveness Tx Inclusion · 1h" and "Liveness Blocks · per 15m".
the first hypothesis was tested against chain data before anything was changed.
the first guess sounded plausible and was nearly acted on. - Measure the outcome, not the ERROR count; keep logs centrally (rotation hid the start).
—
Upstream note: log a cancelled (superseded) build at DEBUG
Outcome metrics + LinethL2LivenessTxMissing alert + panels
done (rule present in the cluster 10-13)
Blocks 12270-12300 each contain 2 txs to the liveness contract 0x7917…60C1.