Loading…

AI SRE OpsUTC …

Overview Infrastructure Detection Incidents Handoff Reliability Postmortems AI

Demo data Sample data for a portfolio demo; not real incidents.

Incident queue

Loading…

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

3 alerts firingLinethL2MessagesStuck (warning)Open incident #3

Prover requests failing

cause: prover · 2 open follow-ups

SEV2Mitigated⚠ change…

Bridge messages in flight but nothing claimed for 45 minutes

SEV3Declared2 firing…

Sequencer liveness txs missing

SEV3suggested⚠ change1 firing…

#2SEV2Mitigated⚠ Suspected change

Prover requests failing

Probable cause prover · affects coordinator

Detected
Declared
Mitigated
Resolved
Closed

Mark resolved?

10 checklist items are still open, by role:

IC3 open

Set the next status update time
Decide rollback or fix forward with the Ops lead and record the decision
Confirm the affected SLIs have been normal for 30 minutes

Planning2 open

Plan a handoff if the incident will outlast this shift
Note follow-up work as it comes up

Comms2 open

Keep updates on cadence (SEV1 every 30 min, SEV2 every hour)
Post the all-clear

Ops3 open

Verify the fix on the dashboard and that the alerts have cleared
Revert temporary mitigations, or schedule the revert
Resume blockscout after catch-up

…

open for

4

alerts, all cleared

7/20

checklist

…

next update

acked 2m

paging

Summary Response Updates29 Alerts4 Roles7/20 Runbook5

Wrongly grouped? Tick alerts and move them to a new candidate.

LinethL2ProverFailuresprovercriticalCleared
Prover requests failing
10-13 13:15 UTC → 10-13 15:29 UTC, 2h 14m
LinethL2ProverBacklogproverwarningCleared
Prover requests waiting > 5 minutes
10-13 13:19 UTC → 10-13 15:57 UTC, 2h 38m
LinethL2FinalizationStalledcoordinatorcriticalCleared
No L2 finalization reached L1 for 20 minutes
10-13 13:29 UTC → 10-13 15:47 UTC, 2h 18m
LinethL2WorkflowStuckcoordinatorwarningCleared
A finalization workflow is stuck in stage wait
10-13 13:41 UTC → 10-13 17:13 UTC, 3h 32m