Loading…
Loading…
Postmortems / PM-2026-10-10-03
Blameless postmortem. Incident record: incident-log.md INC-2026-10-10-03. Times UTC.
On 10-10 at 14:34:18 the coordinator logged eth_sendRawTransaction for aggregation finalization failed: aggregation=[3901..3960], right after it had been restarted for the P1 change. The next attempt landed and finalization reached 3960. The exact JSON-RPC error was not captured, so the cause stayed open with the action "capture the full error next time". On 10-13 the full errors were pulled from Loki for the whole period since the 10-12 chain reset: every such line is one of two benign duplicates.
| what | how much |
|---|---|
| finalization | none: every range in the error lines was finalized on L1 |
| operator | noise: ERROR lines that look like failed finalizations |
Loki, coordinator, 2026-10-12 19:30 → 2026-10-13 06:22: 42 lines … aggregation finalization failed:
| count | call | error | meaning |
|---|---|---|---|
| 37 | eth_sendRawTransaction | code=-32000 message=Known transaction | the coordinator re-sends a finalization tx it already sent; the L1 node already has it in its pool. The original tx is mined |
| 5 | eth_call | revertReason=StartingRootHashDoesNotMatch (0xead4c30e) | the coordinator simulates a finalization that has just been finalized, so the parent state root moved on |
Both are logged at ERROR by the coordinator although the outcome is success. They are not tied to restarts: they appear on roughly one in three aggregations. The 10-10 line was most likely the first of them; its text was not kept, so that is a strong guess, not a fact.
Read in the coordinator log. Correctly not alerted: the outcome metric (lineth_l2_rollup_finalized_block) kept moving.
the outcome (finalized block) was checked before acting; nothing was changed for a log line.
"capture the error next time" only worked once there was central logging with history (Loki). - Same lesson as INC-07: judge components by outcome metrics, not by ERROR counts.
—
Upstream note: log Known transaction and a post-finalization StartingRootHashDoesNotMatch at INFO/WARN
Capture the full JSON-RPC error
done 10-13 (Loki)
Do not alert on these log lines; alert on finalization progress only (exists: LinethL2FinalizationStalled)
LogQL: {namespace="lineth-l2", container="coordinator"} |= "aggregation finalization failed", first line of each entry; classification by errorMessage=.