Postmortems
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.
Loading…
| Priority | Action | Type | Owner | Status | From |
|---|---|---|---|---|---|
| P1 | LinethL2ProverLargeRetry → critical: a parked batch always needs a human | detect | s1ns3nz0 | open | PM-2026-10-11-11 |
| P1 | Alert when node MemAvailable < prover peak + margin while prover requests are pending (capacity alert before the OOM) |
| detect |
| s1ns3nz0 |
| open |
| PM-2026-10-11-11 |
| P1 | Host memory pressure check (Mac vm_stat decompressions / swap) with a threshold — nothing inside the VM sees it | detect | s1ns3nz0 | open | PM-2026-10-13-12 |
| P1 | Alert on prover normal-mode overflows (LIMIT OVERFLOW rate > 0): a regression of A1 must be visible | detect | s1ns3nz0 | open | PM-2026-10-13-12 |
| P2 | Alert on prover restarts (increase(kube_pod_container_status_restarts_total{container="prover"}[30m]) > 0) | detect | s1ns3nz0 | open | PM-2026-10-11-11 |
| P2 | Alert when a signer's pending-in-DB nonce is ahead of its on-chain nonce for > 10 min (nonce gap) | detect | s1ns3nz0 | open | PM-2026-10-13-14 |
| P3 | Capture the full stack trace and the block number for one failure | detect | s1ns3nz0 | open | PM-2026-10-10-06 |
| P3 | Check whether the same block succeeds on retry and on the sequencer (its own counters) | detect | s1ns3nz0 | open | PM-2026-10-10-06 |
| P3 | Use it in the prover dashboard and a "proof running too long" alert | detect | s1ns3nz0 | open | PM-2026-10-10-05 |
| P4 | Check the import-time ZkBlockImportTracerProvider WARNs together (same plugin) | detect | s1ns3nz0 | open | PM-2026-10-10-06 |