Loading…

AI SRE OpsUTC …

Overview Infrastructure Detection Incidents Handoff Reliability Postmortems AI

Demo data Sample data for a portfolio demo; not real incidents.

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

Incident queue

Loading…

3 alerts firingLinethL2MessagesStuck (warning)Open incident #3

Prover requests failing

cause: prover · 2 open follow-ups

SEV2Mitigated⚠ change…

Bridge messages in flight but nothing claimed for 45 minutes

SEV3Declared2 firing…

Sequencer liveness txs missing

SEV3suggested⚠ change1 firing…

#3SEV3Declared

Bridge messages in flight but nothing claimed for 45 minutes

Probable cause postman

Detected
Declared
Mitigated
Resolved
Closed

Mark mitigated?

7 checklist items are still open, by role:

IC2 open

Confirm the severity and write the impact in Current status
Set the next status update time

Ops4 open

Check recent changes (open CHGs, Flux reconciles) on the affected components
Open the runbook for each firing alert
Apply the runbook's quick mitigation and post the result as an update
Record each mitigation as a decision, with when to revert it

Planning1 open

Note follow-up work as it comes up

…

open for

2

alerts firing

0/12

checklist

…

next update

Slack

paging

Summary Response Updates5 Alerts2 Roles0/12 Runbook3

Status

Change severity

Edit status

Claims reverting with nonce too low; postman retrying

ImpactSome deposits delayed; no funds at risk

Checkpoint

IC checkpoint

None yet. Write one before you hand over or step away; a wake condition pages you when it matters.

Priority

21queue score

Queue score 21

SLO1
deposit-10m ×1, 98% budget left; deposit-2m ×1, 99% budget left
Impact0
nothing depends on it
Age20
1 per 10 min open, max 20
Trend0
no new alert in 15 min

Paging

mock delivery

Slack notice · Slack default
10-13 12:52 UTC

Changes

3

During the incident

CHG-2026-10-13-05 Raise the kind node memory to 64 GiB
Kustomization lineth-l2-explorer main@c1d2e3f
CHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up

First signs

map at detection

postmanAt riskat detection
LinethL2MessagesStuck fired · 10-13 12:40 UTC

StatusImpactNext update (UTC)

Status, in one lineCurrent hypothesis

Action in progressExpected to finish (UTC)

Wake me when (PromQL, pages once when it returns any series)Next stepHanding over to (optional)