maru Java heap exhausted, L2 block production halted 4 h 36 min

Draft
Reviewed
Closed

Impact

10-11 19:54:19 → 10-12 00:29:56 (4 h 36 min)

Severity

no L2 blocks for 4 h 36 min — users could not transact

Components

marusequencer

Authors

s1ns3nz0, Claude

Blameless postmortem. Incident record: incident-log.md INC-2026-10-11-10. Times UTC. Block numbers refer to the chain before the 2026-10-12 reset (archived).

1Summary

Maru (the L2 consensus client) leaks heap: its old generation grew ~8 MiB/h from 64 MiB on 10-10 to the 256 MiB cap on 10-11 ~12:00. At 19:39 it threw OutOfMemoryError and at 19:54 it stopped driving the sequencer through the engine API, so no more L2 blocks were produced. The JVM stayed alive, its readiness check was a TCP port, and there was no liveness probe — Kubernetes saw Running with 0 restarts. Alerts fired within minutes but only into a log. A person noticed at 00:26 while reading l2-traffic logs; restarting maru brought blocks back in ~5 s.

2Impact

what	how much
L2 block production	none 19:54:19 → 00:29:56 (last block 20102)
W4 user-tx, W1 transfers	every user tx stuck in the RPC pool for up to 4.5 h; included after the restart
SLOs	tx-inclusion and block-production budgets burned; `block-production` / `settlement-freshness` paged
not affected	no data lost

3Root causes and trigger

Trigger: the heap reached its cap (old gen = 2/3 of -Xmx384m with SerialGC = 256 MiB).
Root cause 1 — leak: Tenured Gen grows steadily (~8 MiB/h then; ~11–14 MiB/h after the fix). Source unknown; the OTel Java agent is a suspect. No jcmd/jmap in the image.
Root cause 2 — a dead process looked healthy: no ExitOnOutOfMemoryError, TCP readiness only, no liveness probe; a JVM after OOM is alive but useless.
Root cause 3 — detection: alerts reach no human. The event-based tx-inclusion-5s SLO never alerted: its 14.4x page threshold was 144 % (unreachable), and a halt produces few events (l2-send waits up to 180 s per tx), so an event-based SLI goes quiet exactly when nothing works.

4Detection

LinethL2ScrapeFailed 19:40:14, LinethL2SequencerStalled 19:59:44, LinethL2RpcTxStuck 20:01:59 and burn-rate pages — all into the alert-sink log. A human noticed at 00:26 (l2-traffic FAILED … Replacement transaction underpriced). Earlier the same evening a single cast block-number reading (20101) looked normal: one sample cannot show a halt.

5Response and recovery

10-11 19:39:05
first java.lang.OutOfMemoryError: Java heap space
19:54:19
last block 20102
10-12 00:26
noticed during practice 2-3
00:29:02
kubectl delete pod maru-0
00:29:56
Ready; 00:30:01 head 20109; 00:30:17 l2-traffic ok
05:41
2d2bf4a: -Xmx768m (old gen 512 MiB), -XX:+ExitOnOutOfMemoryError, limit 1536Mi, alert LinethL2MaruHeapNearlyFull (old gen > 90 % for 10 m); SLO generator caps unreachable burn thresholds at 0.9 error ratio (backtest: the tx-inclusion page would have fired 52 min into the halt)

6Lessons learned

Went well

the evidence (6 h of maru log) was saved before the restart; the fix turned a multi-hour halt into a ~1 min restart.

Went badly

health checks tested "port open", not "doing its job"; an SLO whose page threshold cannot be reached gives false comfort; one reading of a counter is not a rate.

Lucky

the halt was found the same night; the leak would have refilled the heap every ~1.5 days.

7Action items

P1
Find the leak: run without the OTel agent for a day and compare the old-gen slope, or heap dump on OOM
open — 10-13: old gen 58 → 126 MiB in 6 h (~11 MiB/h), max 512 MiB
Prevents1ns3nz0Open
P1
Larger heap (-Xmx768m, limit 1536Mi) — buys time
done (2d2bf4a)
Mitigates1ns3nz0✓ Done
P1
Alert on old gen > 90 % before the halt
done (LinethL2MaruHeapNearlyFull)
Detects1ns3nz0✓ Done
P1
Cap unreachable burn-rate thresholds
Detects1ns3nz0✓ Done
P0
-XX:+ExitOnOutOfMemoryError so OOM kills the JVM and Kubernetes restarts it
done (2d2bf4a)
Mitigates1ns3nz0✓ Done
P0
Route alerts to a person
10-13 11:00 934a2bb: Telegram receiver on lab alerts (critical + warning, repeat 1 h); see INC-15
Detects1ns3nz0✓ Done

8Supporting data

Maru log 6 h before the restart: ~/.local/state/lineth-incidents/inc10/maru-pre-restart.log (16330 lines).
Old gen growth: 121 MiB (10-10 16:31) → 151 (10-11 00:31) → 194 (04:31) → 250 (08:31) → 256 (~12:00).

Loading…