maru old-gen heap nearly full
warningreviewed 2026-10-13What it meansmoremaru leaks old-gen heap (~8–14 MiB/h). When it fills, maru stops calling the sequencer's engine API, so no blocks are built, while the process stays Running. Since 2026-10-12 -XX:+ExitOnOutOfMemoryError makes it exit and restart (~1 min halt), but a planned restart is still better than an unplanned one.
ImpactmoreBefore the OOM: none yet. At the OOM: L2 block production stops (~1 min with the exit flag; 4h36m in INC-10 without it). Warning; SEV3 while only the heap is high. LinethL2SequencerStalled turns it into SEV1 (see sequencer-stalled).
Quick mitigation0/3
Confirm old gen above 90% and still growing (query in Diagnosis)
Restart maru at a quiet moment:
kubectl -n lineth-l2 delete pod maru-0Confirm blocks resume within a minute (head moves) and the heap dropped
Diagnosis3 queries
Old-gen use as a share of its max:
max(jvm_memory_used_bytes{job="l2-maru", id=~"Tenured Gen|G1 Old Gen|PS Old Gen"}) / max(jvm_memory_max_bytes{job="l2-maru", id=~"Tenured Gen|G1 Old Gen|PS Old Gen"})Is the sequencer still building blocks?
increase(ethereum_blockchain_height{job="l2-sequencer"}[3m]){namespace="lineth-l2", pod=~"maru-.*"} |~ "(?i)outofmemory|heap"Fix0/1
Find the leak: run maru for a day without the OTel java agent and compare the old-gen slope, or take a heap dump (INC-10 Next 3)
Escalation
If the heap refills within hours of a restart, the leak got worse: open an incident and plan restarts on a schedule until it is found.
Seen in INC-2026-10-11-10 · SLOs block-production, tx-inclusion-5s · owner s1ns3nz0