Loading…
Loading…
Postmortems / PM-2026-10-13-15
Blameless postmortem. Incident record: incident-log.md INC-2026-10-13-15. Times UTC.
The local L1 runs one beacon node and one validator with 1-second slots. At 07:30:58, with slots bunching up under slow block production, teku re-orged its head back by one slot (to EL block 42490) while besu had already imported the block of the orphaned slot (42491). From then on teku asked besu, every slot, to build on 42490. The Engine API says an EL must not start a payload build when the requested head is an ancestor of its canonical head, so besu answered VALID without a payload id; teku logged ExecutionPayloadContext is not provided and produced no block. Without a new block nothing could change: a permanent deadlock. On L2 the coordinator stopped creating batches and its /health and /metrics stopped answering; it stayed wedged after L1 came back. The alerts fired within a minute and went to the alert-sink log and the console only. A person ran the workflow check at 10:42; debug_setHead on besu (back to 42490) restarted L1 at 10:45 and a coordinator restart resumed settlement at 10:52.
| what | how much |
|---|---|
| L1 | no blocks 07:30:57 → 10:45; L1 finality stuck at epoch 10770, beacon chain in inactivity leak |
| W1 rollup / finalization | head − finalized grew to 3,184 blocks; catching up at ~3.5 min per 80-block batch |
| W3 withdrawals (L2 → L1) | 57 unclaimed > 30 min at 10:42 (49 at 11:05); none lost |
| W2 deposits (L1 → L2) | none possible (no L1 blocks) |
| not affected | L2 block production, L2 user txs (W4/W5), postman L2 nonce |
Slow Block Production / Slow Block Publishing warnings and several slots processed in the same second before it; teku Reorg Event *** New Head … (43093), Previous Head … (43095) at 07:30:58.431.0x9ff3f… (42491), teku forkchoice head 0x3bcbb… (42490, its parent). besu logged only FCU(VALID) every slot (no Start building proposals); teku logged ExecutionPayloadContext is not provided for production of post-merge block every slot (120 per 2 min from 07:32). With one validator nothing else could produce the block that breaks the tie./health and /metrics timed out from ~07:38 (LinethL2ScrapeFailed), conflation stopped at block 9912 (block time 07:36:23), and between 07:56 and 09:36 batches already conflated were proven one per ~20 min (normal ~3.5 min), then nothing. It still timed out at 10:50 with L1 healthy since 10:45; a restart fixed it at once. The pod had readiness and startup probes only, so Kubernetes marked it unready 1,150 times and never restarted it. Why it wedged is unknown (no thread dump was taken before the restart); log lines around it: block=LATEST not found (09:05), BlockCreationMonitor … block=10464 / 12314 not found (08:17, 10:32).LinethL1ExecutionHeadStalled (critical) 07:31:27, LinethL2FinalizationStalled 07:35, LinethL2ScrapeFailed 07:38, SLO burn-rate page 07:59 — all into alert-sink and the console, as in INC-08 … 14.Alerts were correct and fast (L1 stall alert 30 s after the last block). A person found it at 10:42 by running scripts/check-workflows.sh (4 FAIL: coordinator not Ready, L1 head stuck at 42491, finalization gap 3184, 57 withdrawals unclaimed). Since 934a2bb (11:00) lab alerts also go to Telegram; tested with a synthetic alert.
LinethL1ExecutionHeadStalled fires (unseen)/metrics stops answering (LinethL2ScrapeFailed)latest = 42491, FCU head = 42490)debug_setHead("0xa5fa") → Success; 20 s later head 42509, no more ExecutionPayloadContext errorsResuming conflation from block=9913; prover gets 9913-9992934a2bb)the cause was found from logs and two RPC calls in ~3 min; the fix was the smallest possible (move one head pointer back by one empty, already orphaned block) and checked before running (head still 42491).
3 h 11 min between a correct critical alert and a human — the eighth incident with the same gap; "running but not working" again (INC-09, 10, 13) — a process with only a readiness probe can be broken forever; the coordinator was restarted before a thread dump was taken, so its root cause is lost.
the orphaned block 42491 was empty, so moving the head back dropped nothing.
Coordinator liveness probe on /health (5 min of no answer → restart)
open (prepared, see CHG)
Longer L1 slots (2–4 s) to remove the trigger; needs a new L1 genesis = chain reset
open — do with the next chain reset
Runbook for an L1 stall incl. the deadlock check and debug_setHead
Before restarting a wedged JVM, save a thread dump (kill -3 1, in the runbook)
done (runbook); find the coordinator wedge cause next time
Automatic deadlock repair (watchdog calling debug_setHead)
not planned: rewinding a chain head automatically is riskier than alert + runbook; revisit if it recurs
Route lab alerts to a person (Telegram)
934a2bb (also closes INC-08 … 14 "route alerts")
eth_blockNumber = 0xa5fb; block 0xa5fa hash 0x3bcbb58d…, block 0xa5fb hash 0x9ff3fa8a…; besu log FCU(VALID) | head: 3bcbb.....29727 every second.Failed to produce block per 2 min (Loki): 1–4 from 06:58 to 07:26, 12 at 07:30, 70 at 07:32, ~120 (every slot) from 07:34.