Loading…

AI SRE OpsUTC …

Overview Infrastructure Detection Incidents Handoff Reliability Postmortems AI

Demo data Sample data for a portfolio demo; not real incidents.

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

Incident queue

Loading…

3 alerts firingLinethL2MessagesStuck (warning)Open incident #3

Prover requests failing

cause: prover · 2 open follow-ups

SEV2Mitigated⚠ change…

Bridge messages in flight but nothing claimed for 45 minutes

SEV3Declared2 firing…

Sequencer liveness txs missing

SEV3suggested⚠ change1 firing…

#1SEV2Postmortem⚠ Suspected change

L2 web3signer has fewer than 4 keys loaded

Probable cause web3signer-l2

Detected
Declared
Mitigated
Resolved
Closed

Close blocked: 1 follow-up without an issue

33m

to resolve

1

alerts, all cleared

0/19

checklist

—

next update

acked 60s

paging

Summary Response Updates11 Alerts1 Roles0/19 Runbook2

Incident commander

0/6

Owns the incident and every role nobody else holds.

bob, incident command

▶Right after declaring0/3

Confirm the severity and write the impact in Current status
Assign the Ops and Comms leads, or note that you hold them
Set the next status update time

▶Before mitigation0/1

Decide rollback or fix forward with the Ops lead and record the decision

▶Before resolving0/1

Confirm the affected SLIs have been normal for 30 minutes

▶Before closing0/1

Hand postmortem ownership to the Planning lead

Ops lead

0/6

Works the system: diagnosis and mitigation.

Held by the commander

▶Right after declaring0/2

Check recent changes (open CHGs, Flux reconciles) on the affected components
Open the runbook for each firing alert

▶Before mitigation0/2

Record each mitigation as a decision, with when to revert it
Apply the runbook's quick mitigation and post the result as an update

▶Before resolving0/2

Verify the fix on the dashboard and that the alerts have cleared
Revert temporary mitigations, or schedule the revert

Comms lead

0/3

Keeps people outside the response informed.

Held by the commander

▶Right after declaring0/1

Post the first status update

▶Before mitigation0/1

Keep updates on cadence (SEV1 every 30 min, SEV2 every hour)

▶Before resolving0/1

Post the all-clear

Planning lead

0/4

Tracks follow-ups, handoffs and the longer view.

Held by the commander

▶Right after declaring0/1

Plan a handoff if the incident will outlast this shift

▶Before mitigation0/1

Note follow-up work as it comes up

▶Before closing0/2

Draft the postmortem with its action items
Create an issue for every action item