Loading…

AI SRE OpsUTC …

Overview Infrastructure Detection Incidents Handoff Reliability Postmortems AI

Demo data Sample data for a portfolio demo; not real incidents.

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

Incident queue

Loading…

3 alerts firingLinethL2MessagesStuck (warning)Open incident #3

Prover requests failing

cause: prover · 2 open follow-ups

SEV2Mitigated⚠ change…

Bridge messages in flight but nothing claimed for 45 minutes

SEV3Declared2 firing…

Sequencer liveness txs missing

SEV3suggested⚠ change1 firing…

#2SEV2Mitigated⚠ Suspected change

Prover requests failing

Probable cause prover · affects coordinator

Detected
Declared
Mitigated
Resolved
Closed

Mark resolved?

10 checklist items are still open, by role:

IC3 open

Set the next status update time
Decide rollback or fix forward with the Ops lead and record the decision
Confirm the affected SLIs have been normal for 30 minutes

Planning2 open

Plan a handoff if the incident will outlast this shift
Note follow-up work as it comes up

Comms2 open

Keep updates on cadence (SEV1 every 30 min, SEV2 every hour)
Post the all-clear

Ops3 open

Verify the fix on the dashboard and that the alerts have cleared
Revert temporary mitigations, or schedule the revert
Resume blockscout after catch-up

…

open for

4

alerts, all cleared

7/20

checklist

…

next update

acked 2m

paging

Summary Response Updates29 Alerts4 Roles7/20 Runbook5

Assign roles

Leave the name empty to hand the role back to the commander.

Incident commander

2/6

Owns the incident and every role nobody else holds.

alice, incident command

▶Right after declaring2/3

Confirm the severity and write the impact in Current status
FinalizationStalled + ProverFailures, finalized head stopped at 18,410
alice · 10-13 17:19 UTC
How was it handled?
Confirm the severity and write the impact in Current status
FinalizationStalled + ProverFailures, finalized head stopped at 18,410
Assign the Ops and Comms leads, or note that you hold them
bob ops, carol comms
alice · 10-13 17:19 UTC
How was it handled?
Mark done

▶Before mitigation0/1

Mark done

▶Before resolvingNow0/1

Mark done

▶Before closing0/1

Mark done

Add to the incident commander checklist

Ops lead

4/7

Works the system: diagnosis and mitigation.

bob, prover memory and batch requeue

▶Right after declaring✓ 2/2

Check recent changes (open CHGs, Flux reconciles) on the affected components
CHG-02 heap tuning 45 min before the first OOM: the suspect
alice · 10-13 17:19 UTC
How was it handled?
Open the runbook for each firing alert
prover-oom-loop
alice · 10-13 17:19 UTC

Comms lead

1/3

Keeps people outside the response informed.

carol, withdrawal delay notice every hour

▶Right after declaring✓ 1/1

Post the first status update
status page: withdrawals delayed
alice · 10-13 17:19 UTC
How was it handled?

▶Before mitigation0/1

Mark done

Planning lead

0/4

Tracks follow-ups, handoffs and the longer view.

Held by the commander

▶Right after declaring0/1

Mark done

▶Before mitigation0/1

Mark done

Incident commander

Ops lead

Comms lead

Planning lead

How was it handled?

Open the runbook for each firing alert

▶Before mitigation✓ 2/2

Apply the runbook's quick mitigation and post the result as an update
rolled back the heap, requeued 9 batches
alice · 10-13 17:19 UTC
How was it handled?
Apply the runbook's quick mitigation and post the result as an update
rolled back the heap, requeued 9 batches
Record each mitigation as a decision, with when to revert it
rollback, then suspend blockscout
alice · 10-13 17:19 UTC
How was it handled?
Record each mitigation as a decision, with when to revert it
rollback, then suspend blockscout

▶Before resolvingNow0/3

Mark done
Verify the fix on the dashboard and that the alerts have cleared
How was it handled?
Mark done
Revert temporary mitigations, or schedule the revert
How was it handled?
Mark done

Add to the ops lead checklist

ItemWhen

Keep updates on cadence (SEV1 every 30 min, SEV2 every hour)

Keep updates on cadence (SEV1 every 30 min, SEV2 every hour)

How was it handled?

▶Before resolvingNow0/1

Mark done
Post the all-clear
How was it handled?

Add to the comms lead checklist

ItemWhen

▶Before closing0/2

Mark done
Draft the postmortem with its action items
How was it handled?
Mark done
Create an issue for every action item
How was it handled?

Add to the planning lead checklist

ItemWhen

Assign the Ops and Comms leads, or note that you hold them

Set the next status update time

How was it handled?

Decide rollback or fix forward with the Ops lead and record the decision

How was it handled?

Confirm the affected SLIs have been normal for 30 minutes

How was it handled?

Hand postmortem ownership to the Planning lead

How was it handled?

ItemWhen

Check recent changes (open CHGs, Flux reconciles) on the affected components

Post the first status update

Plan a handoff if the incident will outlast this shift

How was it handled?

Note follow-up work as it comes up

How was it handled?

Resume blockscout after catch-up

How was it handled?