Demo data Sample data for a portfolio demo; not real incidents.

3 alerts firingLinethL2MessagesStuck (warning)Open incident #3

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

Postmortems

Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.

Open AllEvery type prevent detect mitigate process

Priority	Action	Type	Owner	Status	From
P1	`LinethL2ProverLargeRetry` → critical: a parked batch always needs a human	detect	s1ns3nz0	open	PM-2026-10-11-11
P1	Alert when node MemAvailable < prover peak + margin while prover requests are pending (capacity alert before the OOM)	detect	s1ns3nz0	open	PM-2026-10-11-11
P1	Postmortem review rule: a P0/P1 action from a postmortem is done or explicitly re-scheduled before the incident is closed	process	s1ns3nz0	open	PM-2026-10-11-11
P1	Capacity plan: prover throughput vs block production incl. liveness blocks; a second prover or larger node so catch-up after an outage is hours, not a day	prevent	s1ns3nz0	open	PM-2026-10-11-11
P1	Find the leak: run without the OTel agent for a day and compare the old-gen slope, or heap dump on OOM	prevent	s1ns3nz0	open	PM-2026-10-11-10
P1	Validate the key count in CI before a web3signer Secret change merges	prevent	bob	open	PM-2026-10-13-16
P1	Coordinator liveness probe on `/health` (5 min of no answer → restart)	mitigate	s1ns3nz0	open	PM-2026-10-13-15
P1	Host memory pressure check (Mac `vm_stat` decompressions / swap) with a threshold — nothing inside the VM sees it	detect	s1ns3nz0	open	PM-2026-10-13-12
P1	Alert on prover normal-mode overflows (`LIMIT OVERFLOW` rate > 0): a regression of A1 must be visible	detect	s1ns3nz0	open	PM-2026-10-13-12
P2	Second prover (or a pool per proof type) for catch-up capacity	prevent	s1ns3nz0	open	PM-2026-10-10-01
P2	Alert on prover restarts (`increase(kube_pod_container_status_restarts_total{container="prover"}[30m]) > 0`)	detect	s1ns3nz0	open	PM-2026-10-11-11
P2	Runbook: before re-enabling the explorer (or any big workload), compare free memory with the prover peak	process	s1ns3nz0	open	PM-2026-10-11-11
P2	Key rotations go through a CHG with a rollback step	process	bob	open	PM-2026-10-13-16
P2	Find why two claims were sent for the same message during the re-claim	prevent	s1ns3nz0	open	PM-2026-10-13-14
P2	Alert when a signer's pending-in-DB nonce is ahead of its on-chain nonce for > 10 min (nonce gap)	detect	s1ns3nz0	open	PM-2026-10-13-14
P2	Review `blocks-limit` (no longer binding) and `conflation-deadline` for 80-block batches; update INC-11 throughput notes	process	s1ns3nz0	open	PM-2026-10-13-12
P3	Capture the full stack trace and the block number for one failure	detect	s1ns3nz0	open	PM-2026-10-10-06
P3	Check whether the same block succeeds on retry and on the sequencer (its own counters)	detect	s1ns3nz0	open	PM-2026-10-10-06
P3	Use it in the prover dashboard and a "proof running too long" alert	detect	s1ns3nz0	open	PM-2026-10-10-05
P3	Second, small prover for compression + aggregation (or pools per proof type)	prevent	s1ns3nz0	open	PM-2026-10-10-04
P3	Settlement SLO thresholds: include one execution proof of queueing	process	s1ns3nz0	open	PM-2026-10-10-04
P3	Script it: list requests whose ranges do not start at the coordinator's "Resuming conflation from block=…"	prevent	s1ns3nz0	open	PM-2026-10-10-02
P3	Recompute settlement baselines after 24 h	process	s1ns3nz0	open	PM-2026-10-10-01
P3	Longer L1 slots (2–4 s) to remove the trigger; needs a new L1 genesis = chain reset	prevent	s1ns3nz0	open	PM-2026-10-13-15
P3	Prover scale-out only with host headroom: 2 replicas, `WORKER_ID` from the pod name, lower `GOMAXPROCS` and limits each	prevent	s1ns3nz0	open	PM-2026-10-13-12
P4	Upstream note: log a cancelled (superseded) build at DEBUG	process	s1ns3nz0	open	PM-2026-10-10-07
P4	Check the import-time `ZkBlockImportTracerProvider` WARNs together (same plugin)	detect	s1ns3nz0	open	PM-2026-10-10-06
P4	Upstream note: log `Known transaction` and a post-finalization `StartingRootHashDoesNotMatch` at INFO/WARN	process	s1ns3nz0	open	PM-2026-10-10-03
P4	Delete the 10-10 quarantine dir once P1 settled	process	s1ns3nz0	open	PM-2026-10-10-02
P4	Ad-hoc 2xxxx forwards from `labq.sh` can wedge the same way: `pkill -f "port-forward.*<port>"` before re-creating	process	s1ns3nz0	open	PM-2026-10-11-09
P0	Raise `bonsai-historical-block-limit` permanently (done at runtime via env, `fcd76bc`; size it to the longest prover outage we must survive) and alert when the proven gap approaches it	prevent	s1ns3nz0	open	PM-2026-10-11-11

Loading…