Loading…

AI SRE OpsUTC …

Overview Infrastructure Detection Incidents Handoff Reliability Postmortems AI

Demo data Sample data for a portfolio demo; not real incidents.

Incident queue

Loading…

3 alerts firingLinethL2MessagesStuck (warning)Open incident #3

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

Prover requests failing

cause: prover · 2 open follow-ups

SEV2Mitigated⚠ change…

Bridge messages in flight but nothing claimed for 45 minutes

SEV3Declared2 firing…

Sequencer liveness txs missing

SEV3suggested⚠ change1 firing…

#2SEV2Mitigated⚠ Suspected change

Prover requests failing

Probable cause prover · affects coordinator

Detected
Declared
Mitigated
Resolved
Closed

Mark resolved?

10 checklist items are still open, by role:

IC3 open

Set the next status update time
Decide rollback or fix forward with the Ops lead and record the decision
Confirm the affected SLIs have been normal for 30 minutes

Planning2 open

Plan a handoff if the incident will outlast this shift
Note follow-up work as it comes up

Comms2 open

Keep updates on cadence (SEV1 every 30 min, SEV2 every hour)
Post the all-clear

Ops3 open

Verify the fix on the dashboard and that the alerts have cleared
Revert temporary mitigations, or schedule the revert
Resume blockscout after catch-up

…

open for

4

alerts, all cleared

7/20

checklist

…

next update

acked 2m

paging

Summary Response Updates29 Alerts4 Roles7/20 Runbook5

AIProver was OOM-killed on every large batch for ~2 h after the heap change (CHG-02); rolled back, catch-up running.2 next steps →

Status

Change severity

Edit status

Catch-up running: finalized head 840 blocks behind, ETA 2 h

ImpactWithdrawals delayed ~4 h; no funds at risk

Checkpoint

alice · 10-13 13:26 UTC · 1 earlier

IC checkpoint

Catch-up running: finalized head 840 blocks behind, closing ~6 blocks/min

Hypothesis

cause confirmed: heap change; rolled back

In progress

watch the catch-up; blockscout stays suspendeddone by 15:44 UTC · …

Handed to

dave

Next step

Resume blockscout when the finalized head is within 100 blocks

Wake conditionWatchingnot checked yetshow query

max(changes(lineth_l2_rollup_finalized_block{l1_block_tag="latest"}[30m])) == 0

Fires it now instead of waiting for the query. Mock delivery.

Follow-ups

2

1/2 have issues

Add a follow-up

Action

TypePriorityOwner

P1preventLoad-test heap changes against .large batches before rollout#1002
P1detectPage when the prover restarts more than twice in 30 min

Priority

50queue score

Queue score 50

SLO30
settlement-freshness ×2, 39% budget left; withdrawal-1h ×1, 71% budget left
Impact0
nothing depends on it
Age20
1 per 10 min open, max 20
Trend0
no new alert in 15 min

Paging

mock delivery

Page 1 · ntfy high
10-13 09:43 UTC
✓
Acknowledged by alice
2m after the first page

Changes

8

Before detection

Kustomization lineth-l2-proving main@4e1c9a2
CHG-2026-10-13-02 Tune the prover JVM heap for large batches (-Xmx 20g to 23g)
CHG-2026-10-13-01 Rotate the L2 web3signer keys

During the incident

CHG-2026-10-13-05 Raise the kind node memory to 64 GiB
Kustomization lineth-l2-explorer main@c1d2e3f
CHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up
Kustomization lineth-l2-proving main@b7d03f1applied after a failure

1 more

First signs

map at detection

proverBreachat detection
LinethL2ProverFailures fired · 10-13 09:22 UTC

StatusImpactNext update (UTC)

Status, in one lineCurrent hypothesis

Action in progressExpected to finish (UTC)

Wake me when (PromQL, pages once when it returns any series)

Next stepHanding over to (optional)

CHG-2026-10-13-03 Roll back the prover heap tuning