Demo data Sample data for a portfolio demo; not real incidents.

Change in progressCHG-2026-10-13-04 Pause synthetic traffic during finalization catch-up (low risk, 1 component)Details

Postmortems

Blameless write-ups of what the system and the process allowed, with action items that prevent, detect or mitigate a recurrence. The GitHub repository is the single source of truth; drafts open as pull requests.

Open AllEvery type prevent detect mitigate process

Priority	Action	Type	Owner	Status	From
P1	Capacity plan: prover throughput vs block production incl. liveness blocks; a second prover or larger node so catch-up after an outage is hours, not a day	prevent	s1ns3nz0	open	PM-2026-10-11-11
P1	Find the leak: run without the OTel agent for a day and compare the old-gen slope, or heap dump on OOM	prevent	s1ns3nz0	open	PM-2026-10-11-10
P1	Validate the key count in CI before a web3signer Secret change merges	prevent	bob	open	PM-2026-10-13-16
P2	Second prover (or a pool per proof type) for catch-up capacity	prevent	s1ns3nz0	open	PM-2026-10-10-01
P2	Find why two claims were sent for the same message during the re-claim	prevent	s1ns3nz0	open	PM-2026-10-13-14
P3	Second, small prover for compression + aggregation (or pools per proof type)	prevent	s1ns3nz0	open	PM-2026-10-10-04
P3	Script it: list requests whose ranges do not start at the coordinator's "Resuming conflation from block=…"	prevent	s1ns3nz0	open	PM-2026-10-10-02
P3	Longer L1 slots (2–4 s) to remove the trigger; needs a new L1 genesis = chain reset	prevent	s1ns3nz0	open	PM-2026-10-13-15
P3	Prover scale-out only with host headroom: 2 replicas, `WORKER_ID` from the pod name, lower `GOMAXPROCS` and limits each	prevent	s1ns3nz0	open	PM-2026-10-13-12
P0	Raise `bonsai-historical-block-limit` permanently (done at runtime via env, `fcd76bc`; size it to the longest prover outage we must survive) and alert when the proven gap approaches it	prevent	s1ns3nz0	open	PM-2026-10-11-11

Loading…