Demo data Sample data for a portfolio demo; not real incidents.

3 alerts firingLinethL2MessagesStuck (warning)Open incident #3

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

Postmortems

Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.

Open AllEvery type prevent detect mitigate process

Priority	Action	Type	Owner	Status	From
P1	`LinethL2ProverLargeRetry` → critical: a parked batch always needs a human	detect	s1ns3nz0	open	PM-2026-10-11-11
P1	Alert when node MemAvailable < prover peak + margin while prover requests are pending (capacity alert before the OOM)	detect	s1ns3nz0	open	PM-2026-10-11-11
P1	Postmortem review rule: a P0/P1 action from a postmortem is done or explicitly re-scheduled before the incident is closed	process	s1ns3nz0	open	PM-2026-10-11-11
P1	Capacity plan: prover throughput vs block production incl. liveness blocks; a second prover or larger node so catch-up after an outage is hours, not a day	prevent	s1ns3nz0	open	PM-2026-10-11-11
P1	Find the leak: run without the OTel agent for a day and compare the old-gen slope, or heap dump on OOM	prevent	s1ns3nz0	open	PM-2026-10-11-10
P1	Validate the key count in CI before a web3signer Secret change merges	prevent	bob	open	PM-2026-10-13-16
P1	Coordinator liveness probe on `/health` (5 min of no answer → restart)	mitigate	s1ns3nz0	open	PM-2026-10-13-15
P1	Host memory pressure check (Mac `vm_stat` decompressions / swap) with a threshold — nothing inside the VM sees it	detect	s1ns3nz0	open	PM-2026-10-13-12
P1	Alert on prover normal-mode overflows (`LIMIT OVERFLOW` rate > 0): a regression of A1 must be visible	detect	s1ns3nz0	open	PM-2026-10-13-12
P2	Second prover (or a pool per proof type) for catch-up capacity	prevent	s1ns3nz0	open	PM-2026-10-10-01
P2	Alert on prover restarts (`increase(kube_pod_container_status_restarts_total{container="prover"}[30m]) > 0`)	detect	s1ns3nz0	open	PM-2026-10-11-11
P2	Runbook: before re-enabling the explorer (or any big workload), compare free memory with the prover peak	process	s1ns3nz0	open	PM-2026-10-11-11
P2	Key rotations go through a CHG with a rollback step	process	bob	open	PM-2026-10-13-16
P2	Find why two claims were sent for the same message during the re-claim	prevent	s1ns3nz0	open	PM-2026-10-13-14
P2	Alert when a signer's pending-in-DB nonce is ahead of its on-chain nonce for > 10 min (nonce gap)	detect	s1ns3nz0	open	PM-2026-10-13-14
P2	Review `blocks-limit` (no longer binding) and `conflation-deadline` for 80-block batches; update INC-11 throughput notes	process	s1ns3nz0	open	PM-2026-10-13-12
P3	Capture the full stack trace and the block number for one failure	detect	s1ns3nz0	open	PM-2026-10-10-06
P3	Check whether the same block succeeds on retry and on the sequencer (its own counters)	detect	s1ns3nz0	open	PM-2026-10-10-06
P3	Use it in the prover dashboard and a "proof running too long" alert	detect	s1ns3nz0	open	PM-2026-10-10-05
P3	Second, small prover for compression + aggregation (or pools per proof type)	prevent	s1ns3nz0	open	PM-2026-10-10-04
P3	Settlement SLO thresholds: include one execution proof of queueing	process	s1ns3nz0	open	PM-2026-10-10-04
P3	Script it: list requests whose ranges do not start at the coordinator's "Resuming conflation from block=…"	prevent	s1ns3nz0	open	PM-2026-10-10-02
P3	Recompute settlement baselines after 24 h	process	s1ns3nz0	open	PM-2026-10-10-01
P3	Longer L1 slots (2–4 s) to remove the trigger; needs a new L1 genesis = chain reset	prevent	s1ns3nz0	open	PM-2026-10-13-15
P3	Prover scale-out only with host headroom: 2 replicas, `WORKER_ID` from the pod name, lower `GOMAXPROCS` and limits each	prevent	s1ns3nz0	open	PM-2026-10-13-12
P4	Upstream note: log a cancelled (superseded) build at DEBUG	process	s1ns3nz0	open	PM-2026-10-10-07
P4	Check the import-time `ZkBlockImportTracerProvider` WARNs together (same plugin)	detect	s1ns3nz0	open	PM-2026-10-10-06
P4	Upstream note: log `Known transaction` and a post-finalization `StartingRootHashDoesNotMatch` at INFO/WARN	process	s1ns3nz0	open	PM-2026-10-10-03
P4	Delete the 10-10 quarantine dir once P1 settled	process	s1ns3nz0	open	PM-2026-10-10-02
P4	Ad-hoc 2xxxx forwards from `labq.sh` can wedge the same way: `pkill -f "port-forward.*<port>"` before re-creating	process	s1ns3nz0	open	PM-2026-10-11-09
P0	Raise `bonsai-historical-block-limit` permanently (done at runtime via env, `fcd76bc`; size it to the longest prover outage we must survive) and alert when the proven gap approaches it	prevent	s1ns3nz0	open	PM-2026-10-11-11
P1	Keep `blocks-limit 120` / deadline 12 m as the lab default	prevent	s1ns3nz0	done	PM-2026-10-10-01
P1	Stop batches overflowing normal mode: lower conflation `blocks-limit` / trace limits, or run large mode by default	prevent	s1ns3nz0	done	PM-2026-10-11-11
P1	web3signer must fail (not run with 0 keys) when key files cannot be loaded, or retry: readiness probe on loaded key count / startup dependency on DNS+Vault; found when a node restart halted the chain 8 min	prevent	s1ns3nz0	done	PM-2026-10-11-11
P1	Larger heap (`-Xmx768m`, limit 1536Mi) — buys time	mitigate	s1ns3nz0	done	PM-2026-10-11-10
P1	Alert on old gen > 90 % before the halt	detect	s1ns3nz0	done	PM-2026-10-11-10
P1	Cap unreachable burn-rate thresholds	detect	s1ns3nz0	done	PM-2026-10-11-10
P1	Remove the trigger: node memory headroom for the prover	prevent	s1ns3nz0	done	PM-2026-10-11-09
P1	Restore `.large` on container restarts too (in the prover container command)	mitigate	s1ns3nz0	done	PM-2026-10-11-08
P1	Prover memory request near its real peak (was 6 Gi vs ~10 GiB)	prevent	s1ns3nz0	done	PM-2026-10-11-08
P1	Stop batches overflowing normal mode (smaller batches or large mode by default)	prevent	s1ns3nz0	done	PM-2026-10-11-08
P1	Runbook for an L1 stall incl. the deadlock check and `debug_setHead`	process	s1ns3nz0	done	PM-2026-10-13-15
P1	Patch the postman: convert the two fee columns with `BigInt(...)` in `messageMappers.ts` (or a TypeORM transformer); build a lab image; test by dropping a pending claim	prevent	s1ns3nz0	done	PM-2026-10-13-14
P1	After restarting a node that holds others' pending txs (RPC node), check every signer's on-chain vs DB nonce (postman L1/L2, traffic) — runbook step	process	s1ns3nz0	done	PM-2026-10-13-14
P1	Check other components for once-at-start name resolution	prevent	s1ns3nz0	done	PM-2026-10-13-13
P1	After any full node restart: run the W1–W5 checks before calling it done (runbook)	process	s1ns3nz0	done	PM-2026-10-13-13
P1	Watch the gap back to 1–2 batches	process	s1ns3nz0	done	PM-2026-10-13-12
P2	Make orphan cleanup part of every coordinator config change (checklist)	process	s1ns3nz0	done	PM-2026-10-10-02
P2	Size batch changes against memory and trace limits, not only proof time	process	s1ns3nz0	done	PM-2026-10-10-01
P2	Health-checked forward wrapper under launchd	mitigate	s1ns3nz0	done	PM-2026-10-11-09
P2	Page on LinethL2SignerKeysMissing (critical) instead of a ticket	detect	alice	done	PM-2026-10-13-16
P2	Before restarting a wedged JVM, save a thread dump (`kill -3 1`, in the runbook)	process	s1ns3nz0	done	PM-2026-10-13-15
P2	Prover memory request to the normal-mode peak (80-block batches: working set 9.44 GiB; the 7.3 GiB of INC-11 was a 60-block batch)	prevent	s1ns3nz0	done	PM-2026-10-13-12
P3	Outcome metrics + `LinethL2LivenessTxMissing` alert + panels	detect	s1ns3nz0	done	PM-2026-10-10-07
P3	New metric (time since pickup)	detect	s1ns3nz0	done	PM-2026-10-10-05
P3	Capture the full JSON-RPC error	detect	s1ns3nz0	done	PM-2026-10-10-03
P3	Fix the static-peer comment	process	s1ns3nz0	done	PM-2026-10-13-13
P4	Do not alert on these log lines; alert on finalization progress only (exists: `LinethL2FinalizationStalled`)	detect	s1ns3nz0	done	PM-2026-10-10-03
P4	Automatic deadlock repair (watchdog calling `debug_setHead`)	mitigate	s1ns3nz0	dropped	PM-2026-10-13-15
P0	Route alerts to a person (Telegram receiver; `severity=critical` pages)	detect	s1ns3nz0	done	PM-2026-10-11-11
P0	Prover memory request ~10 Gi (= measured peak) so the scheduler sees the overcommit; check the node fits it	prevent	s1ns3nz0	done	PM-2026-10-11-11
P0	After catch-up: resume explorer + traffic, confirm 376 withdrawals claimed, close INC-11, restart CHG-2026-10-12-02 D3-b	mitigate	s1ns3nz0	done	PM-2026-10-11-11
P0	`-XX:+ExitOnOutOfMemoryError` so OOM kills the JVM and Kubernetes restarts it	mitigate	s1ns3nz0	done	PM-2026-10-11-10
P0	Route alerts to a person	detect	s1ns3nz0	done	PM-2026-10-11-10
P0	Route alerts to a person	detect	s1ns3nz0	done	PM-2026-10-11-08
P0	Route lab alerts to a person (Telegram)	detect	s1ns3nz0	done	PM-2026-10-13-15
P0	Route alerts to a person	detect	s1ns3nz0	done	PM-2026-10-13-14
P0	`l2-node-besu` startupProbe `/readiness?minPeers=1` within 2 min	mitigate	s1ns3nz0	done	PM-2026-10-13-13
P0	Route alerts to a person	detect	s1ns3nz0	done	PM-2026-10-13-13
P0	`BLOCK_HASH = 1440` with the measured model in the config comment	prevent	s1ns3nz0	done	PM-2026-10-13-12
P0	Docker VM 30 GiB	prevent	s1ns3nz0	done	PM-2026-10-13-12
P0	Route alerts to a person	detect	s1ns3nz0	done	PM-2026-10-13-12

Loading…