EU → US · 08:00–16:00 UTC
- 1Writtens1ns3nz0
- 2Handed overby the outgoing shift
- 3Acknowledgedpaged at 16:15 UTC
3
open incidents
3
alerts firing
2
got worse
-12.0%
worst budget move
1
CHGs in progress
1
failed deploys
The shift at a glance
08:0010:0012:0014:0016:00
- candidatebefore
- declared09:43
- mitigated11:38
- resolved
- closed
Decisions 2
Roll back the prover heap tuning (CHG-02)
10:08Why: OOMs started 45 min after -Xmx went to 23g; the 24 GiB limit leaves no off-heap room
Suspend blockscout to give the prover memory during catch-up
12:22Why: frees 3.1 GiB; catch-up ~2.5 h
Latest checkpoint 13:26
Catch-up running: finalized head 840 blocks behind, closing ~6 blocks/min
Next: Resume blockscout when the finalized head is within 100 blocks
Also: Severity changed: set to SEV2 (09:43) · Page answered: after 120s (09:45)
Needs you
3- SEV2Prover requests failing
#2 mitigated · cause prover
Next updateoverdue 2h 32mOpen for8h 19mICaliceChecklist5 openWhere it standsCatch-up running: finalized head 840 blocks behind, closing ~6 blocks/min. watch the catch-up; blockscout stays suspended, done by 15:44 UTC.
Open incident →Your next stepfrom the last checkpointResume blockscout when the finalized head is within 100 blocks
SLO error budget
7| SLO | Budget left | This shift | |
|---|---|---|---|
| settlement-freshness | 39% left | -12.0% | |
| withdrawal-1h | 71% left | -6.5% | |
| tx-inclusion-5s | 92% left | -2.4% | |
| block-production | 86% left | -1.6% | |
| tx-inclusion-60s | 95% left | -1.6% | |
| rpc-availability | 95% left | -0.8% |