Loading…

AI SRE OpsUTC …

Overview Infrastructure Detection Incidents Handoff Reliability Postmortems AI

Demo data Sample data for a portfolio demo; not real incidents.

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

Incident queue

Loading…

3 alerts firingLinethL2MessagesStuck (warning)Open incident #3

Prover requests failing

cause: prover · 2 open follow-ups

SEV2Mitigated⚠ change…

Bridge messages in flight but nothing claimed for 45 minutes

SEV3Declared2 firing…

Sequencer liveness txs missing

SEV3suggested⚠ change1 firing…

#1SEV2Postmortem⚠ Suspected change

L2 web3signer has fewer than 4 keys loaded

Probable cause web3signer-l2

Detected
Declared
Mitigated
Resolved
Closed

Close blocked: 1 follow-up without an issue

33m

to resolve

1

alerts, all cleared

0/19

checklist

—

next update

acked 60s

paging

Summary Response Updates11 Alerts1 Roles0/19 Runbook2

PostmortemPM-2026-10-13-16

Status

All 4 keys loaded since 01:28; blob submission caught up

ImpactBlob submissions to L1 paused 26 min

Checkpoint

No checkpoint was written.

Follow-ups

1

0/1 have issues

Add a follow-up

P1preventValidate the key count in CI before a web3signer Secret change merges

Paging

mock delivery

Page 1 · ntfy high
10-13 11:23 UTC
✓
Acknowledged by bob
60s after the first page

Changes

2

Before detection

Kustomization lineth-l2-signing main@9f8e7d6
CHG-2026-10-13-01 Rotate the L2 web3signer keys

First signs

map at detection

web3signer-l2Breachat detection
LinethL2SignerKeysMissing fired · 10-13 11:19 UTC

Action

TypePriorityOwner