L2 web3signer started with 2 of 4 keys after a key rotation; blob submission paused 26 min

Draft
Reviewed
Closed

Impact

11:19 → 11:45 (26 min)

Severity

SEV-2

Components

web3signer-l2coordinator

Authors

bob

Blameless postmortem (sample data for the portfolio demo). Times UTC.

1Summary

A planned rotation of the L2 web3signer keys (CHG-01) mounted the new key files under the old Secret name. web3signer restarted with 2 of its 4 keys, so the coordinator could not sign blob submissions. The keys alert paged within a minute; restoring the previous Secret and restarting web3signer brought all four keys back.

2Impact

what	how much
blob submissions to L1	paused 26 min, caught up within 10 min after
users	none visible: L2 blocks kept coming; settlement was 26 min late

3Root causes and trigger

Trigger: the rotation changed the Secret name in one manifest and not in the web3signer volume.
Root cause: nothing checks the number of keys web3signer will load before the change merges.

4Detection

LinethL2SignerKeysMissing (critical) fired 1 min after the restart and paged the on-call.

5Response and recovery

11:19
web3signer restarts with 2 keys; the alert fires
11:23
declared SEV2 (bob)
11:36
decision: restore the previous Secret and restart web3signer
11:45
4 keys loaded; blob submission resumes

6Lessons learned

Went well

the alert paged within a minute and the rollback was one command.

Went badly

the rotation had no rollback step written down.

Lucky

the sequencer key was not among the missing ones, so L2 blocks kept coming.

7Action items

P1
Validate the key count in CI before a web3signer Secret change merges
console follow-up
PreventbobOpen
P2
Key rotations go through a CHG with a rollback step
ProcessbobOpen
P2
Page on LinethL2SignerKeysMissing (critical) instead of a ticket
Detectalice✓ Done

Loading…

1Summary

what

how much

blob submissions to L1

paused 26 min, caught up within 10 min after

users

none visible: L2 blocks kept coming; settlement was 26 min late