L1 validators not signing: keys, external signer or slashing protection
criticalreviewed 2026-10-13What it meansmoreThe four L1 validators run in teku-validator and sign through the L1 web3signer (keys from Vault, slashing protection in web3signer-postgres). Fewer than 4 active validators, a key count other than 4, failed or timed out signing requests, signing prevented by slashing protection, or a slashed validator all mean blocks or attestations are being missed. With one node and four validators, missing signatures stop the L1 quickly.
ImpactmoreMissed proposals slow the L1; missed attestations stop finality. Settlement on L1 follows. Slashed or slashing-protection blocks: SEV1. Never "fix" these by deleting the slashing database. Key count or signing failures: SEV2 while blocks are missed; SEV3 if the chain still moves.
Quick mitigation0/4
Check the key count and signing errors (queries in Diagnosis)
Web3signer with fewer than 4 keys after a restart: Vault was not reachable at start; once Vault is unsealed,
kubectl -n local-l1 rollout restart deployment web3signerSigning timeouts: check
web3signer-postgres is Ready and web3signer logs; restart web3signer onceSlashing protection prevented signing or a validator is slashed: stop here, do not restart with a fresh database; page the IC
Diagnosis4 queries
Validators by status:
sum by (status) (validator_local_validator_counts{network="lineth-local-l1", job="teku-validator"})Keys loaded in the L1 web3signer:
signing_signers_loaded_count{network="lineth-local-l1", job="web3signer"}Failed or timed-out signing requests in the last 15 minutes:
sum by (result) (increase(validator_external_signer_requests_total{network="lineth-local-l1", job="teku-validator"}[15m]))Web3signer errors:
{namespace="local-l1", pod=~"web3signer-[a-z0-9]+-[a-z0-9]+"} |~ "(?i)error|vault|slashing"Fix0/3
Make web3signer wait for Vault at start instead of starting with fewer keys (same class as INC-11 A12)
If keys were rotated, validate the key count before the Secret change merges
For a slashing event, write down both conflicting messages and how the double signature happened before any restart
Escalation
Any slashing alert: page the IC immediately and stop all validator restarts until the cause is understood.
Seen in INC-2026-10-11-11 · SLOs settlement-freshness · owner s1ns3nz0