L2 web3signer has fewer than 4 keys loaded
criticalreviewed 2026-10-13What it meansmoreThe L2 web3signer signs for the sequencer (liveness txs), the coordinator (blob and finalization submissions) and the postman. It loads its 4 keys from Vault at start. If Vault was not reachable when it started (after a node restart Vault starts later and needs unsealing), it runs with fewer keys and never reloads them (INC-11 A12).
ImpactmoreMissing the sequencer key stops L2 block production within minutes (SEV1). Missing the coordinator key stops blob submission and finalization (SEV2). The pod stays Running and Ready, so only this alert shows it.
Quick mitigation0/3
Check Vault is unsealed:
kubectl -n vault exec vault-0 -- vault statusRestart web3signer so it loads the keys again:
kubectl -n lineth-l2 rollout restart deployment web3signerConfirm 4 keys (query in Diagnosis) and that L2 blocks and blob submissions resume
Diagnosis3 queries
Keys loaded by the L2 web3signer:
max(signing_signers_loaded_count{job="l2-web3signer"})Is the L2 still producing blocks:
increase(ethereum_blockchain_height{job="l2-sequencer"}[3m])Web3signer key loading:
{namespace="lineth-l2", pod=~"web3signer-.*"} |~ "(?i)vault|key|loaded|error"Fix0/2
Make web3signer wait for Vault before it starts (init container or a readiness check on the key count)
Validate the key count in CI before a web3signer Secret or key rotation merges
Escalation
If Vault is sealed and cannot be unsealed, page the IC: signing for the whole L2 depends on it. The unseal keys are in the break-glass procedure, not in this runbook.
Seen in INC-2026-10-11-11 · SLOs block-production, settlement-freshness · owner s1ns3nz0