L1 under resource pressure: OOM kill, JVM heap, container memory or disk
warningreviewed 2026-10-13What it meansmoreAn L1 client was OOM-killed recently, its JVM heap stays high, a container is close to its memory limit, or the node filesystem is running out of space. The lab runs everything on one kind node inside the Docker VM, so memory pressure in one namespace hurts all of them (INC-08, INC-12: the prover slowed or died when the host was short).
ImpactmoreNothing yet while it is pressure; an OOM kill restarts the client and can stall the L1 for a minute or more. Disk full stops besu and teku writing: the L1 halts. Treat low disk as SEV2 when under 10 %. Warning otherwise; open an incident if an OOM kill repeats within an hour.
Quick mitigation0/4
Find which container and how close it is (queries in Diagnosis)
OOM-killed once: check it came back Ready and the head moves; note it on the incident
Disk low: find the largest PVC on the node and prune what is safe (old logs, Loki retention), not chain data
Memory near limit on many pods at once: free node memory (suspend the L2 explorer:
flux suspend kustomization lineth-l2-explorer)Diagnosis4 queries
Containers by memory use against their limit (percent):
100 * max by (pod, container) (container_memory_working_set_bytes{namespace="local-l1", container!=""}) / max by (pod, container) (kube_pod_container_resource_limits{namespace="local-l1", resource="memory"})Recent OOM kills in local-l1:
max by (pod, container) (kube_pod_container_status_last_terminated_reason{namespace="local-l1", reason="OOMKilled"})Node memory available (GiB) and disk free (percent) on the node (/var) and the volumes (/srv/lineth, hostPath PVs):
min(node_memory_MemAvailable_bytes) / 1024^3100 * node_filesystem_avail_bytes{job="node-exporter", mountpoint=~"/var|/srv/lineth"} / node_filesystem_size_bytes{job="node-exporter", mountpoint=~"/var|/srv/lineth"}Fix0/3
Raise the container limit or the JVM
-Xmx in lineth-l2-gitops only with the node headroom checked firstFor disk, set retention (Loki, Prometheus) so the lab cannot fill the node again
If the Docker VM is too small for the lab, resize it in a change window (CHG), following the node restart runbook
Escalation
If the node itself is short of memory (MemAvailable under 2 GiB) or disk is under 5 %, page the IC: every component on the node is at risk, not only the L1.
Seen in INC-2026-10-11-08, INC-2026-10-13-12 · owner s1ns3nz0