L2 sequencer stopped producing blocks
criticalreviewed 2026-10-13What it meansmoreThe sequencer's head has not moved for 3 minutes. Blocks are driven by maru over the engine API; when maru stops (heap full, crash, signer missing), the sequencer idles and logs Execution engine not called in 120 seconds.
ImpactmoreNo L2 blocks: every user tx waits in the RPC pool; block-production and tx-inclusion SLOs burn. SEV1 (block production stopped).
Quick mitigation0/4
Check maru: heap (see
maru-heap-full), pod Ready, recent restartsCheck web3signer L2: signing keys loaded (a restart with 0 keys halted the chain once)
If maru is the cause:
kubectl -n lineth-l2 delete pod maru-0Confirm blocks resume and pending txs are included
Diagnosis4 queries
Head movement in the last 3 minutes (0 = stalled):
increase(ethereum_blockchain_height{job="l2-sequencer"}[3m])maru old-gen heap:
max(jvm_memory_used_bytes{job="l2-maru", id=~"Tenured Gen|G1 Old Gen|PS Old Gen"}) / max(jvm_memory_max_bytes{job="l2-maru", id=~"Tenured Gen|G1 Old Gen|PS Old Gen"}){namespace="lineth-l2", pod=~"sequencer-.*"} |= "Execution engine not called"{namespace="lineth-l2", pod=~"maru-.*"} |~ "(?i)outofmemory|engine"Fix0/1
Liveness probe on maru that fails when it stops driving the engine API (INC-10: the pod stayed Running for 4.5 h)
Escalation
If neither maru nor the signer explains it within 10 minutes, page for help and capture sequencer and maru logs before restarting anything else.
Seen in INC-2026-10-11-10 · SLOs block-production, tx-inclusion-5s, tx-inclusion-60s · owner s1ns3nz0