L1 head stalled or not finalizing
criticalreviewed 2026-10-13What it meansmoreThe local L1 is one besu (execution), one teku beacon node and one teku validator with 1-second slots. Execution head stalled means no new L1 block for 30 s; finality delayed means the beacon chain stopped finalizing; optimistic or execution offline means teku cannot validate payloads with besu. In INC-15 a one-slot re-org left besu one block ahead of teku (EL/CL deadlock): teku kept asking besu to build on an ancestor, besu refused, nothing moved by itself.
ImpactmoreL2 keeps producing blocks, but nothing settles: no finalization on L1, no withdrawals, no message anchoring. The coordinator can wedge and stay wedged after the L1 recovers (INC-15). SEV2 if it lasts over 15 minutes (settlement-freshness burns); SEV1 if finalization on L1 is also stalled for an hour.
Quick mitigation0/5
Tell which stall it is:
kubectl -n local-l1 get pods and kubectl -n local-l1 logs teku-0 --since=2m | grep -c 'ExecutionPayloadContext is not provided'Crashed client (pod not Ready, CrashLoop): read its logs, then restart that pod only
EL/CL deadlock (teku logs the message every slot, besu logs only
FCU(VALID)): set besu's head back to teku's head with debug_setHead (lineth-l2-gitops docs/runbooks/l1-stall.md, step 2)If the head does not move within a minute:
kubectl -n local-l1 delete pod besu-0, then teku-0After the L1 moves: check the coordinator is Ready and producing proofs; restart it if
/health times out (after a thread dump)Diagnosis4 queries
Seconds since the last L1 execution block:
time() - lab_execution_head_timestamp_seconds{network="lineth-local-l1"}Epochs between the head and the finalized checkpoint:
floor(lab_beacon_head_slot{network="lineth-local-l1"} / 4) - lab_beacon_finalized_epoch{network="lineth-local-l1"}Teku execution client state (1 = optimistic or offline):
max by (__name__) ({__name__=~"lab_beacon_is_optimistic|lab_beacon_el_offline", network="lineth-local-l1"})Teku errors about payloads:
{namespace="local-l1", pod="teku-0"} |~ "ExecutionPayloadContext|optimistic|execution client"Fix0/3
Record whether it was a deadlock, a crash or resource pressure, with the besu and teku head hashes
If it was a deadlock, check that the coordinator liveness probe restarted it; tune it in lineth-l2-gitops if not
If it recurs, plan longer slots or a second beacon node for the lab (one node with 1-second slots re-orgs easily)
Escalation
If the L1 does not move after debug_setHead and a restart of both clients, page the IC: settlement is stopped. Do not delete PVCs: the L1 data cannot be rebuilt without resetting the L2 bridge state.
Seen in INC-2026-10-13-15 · SLOs settlement-freshness, withdrawal-1h, deposit-10m · owner s1ns3nz0