Reliability

How fast incidents are seen, answered, contained and fixed; where they come from; what is still owed. Declared incidents only.

12m

MTTD median

first alert → declared · p90 21m · n=3

MTTA median

first page → acknowledged · p90 2m · n=2

16m

MTTM median

declared → mitigated · p90 1h 55m · n=2

33m

MTTR median

first alert → resolved · p90 33m · n=1

Incidents by probable cause

No repeated cause.

P1Validate the key count in CI before a web3signer Secret change mergesno issue#1
P1Page when the prover restarts more than twice in 30 minno issue#2
P1Load-test heap changes against .large batches before rollout#1002 mock#2

Incidents

Incident	Cause	MTTD	MTTA	MTTM	MTTR	Detected
#3 SEV3 Bridge messages in flight but nothing claimed for 45 minutes	postman	12m	—	—	—	10-13 12:40 UTC
#2 SEV2 Prover requests failing	prover	21m	2m	1h 55m	—	10-13 09:22 UTC
#1 SEV2 L2 web3signer has fewer than 4 keys loaded	web3signer-l2	4m	1m	16m	33m	10-13 07:26 UTC

Loading…