L1 telemetry missing: a scrape target, a chain-exporter collection or a JVM metric is gone
criticalreviewed 2026-10-13What it meansmorePrometheus can no longer see part of the local L1. Either a scrape target is down or gone (up == 0 or absent), the chain-exporter in local-l1 cannot reach a client API (execution, beacon or finality collection failed or stale), or a JVM heap series disappeared. The L1 itself may be fine: these alerts say we are blind, not that the chain stopped. Every other L1 alert depends on these series, so while this fires the L1 has no working detection.
ImpactmoreNo direct user impact. Indirect: an L1 stall or a validator problem would go unseen while this fires. Critical (target or collection missing): treat as SEV3 until you know whether the chain is moving; if the L1 head is also stalled, follow l1-chain-stalled instead. JVM metric missing alone is a warning: the client runs, only its heap panel and LinethL1JVMHeapPressure are blind.
Quick mitigation0/4
Check whether the L1 still moves (query in Diagnosis); if not, switch to
l1-chain-stalledFind which target is down:
kubectl -n local-l1 get pods and the up query belowIf the pod is down or not Ready, follow its logs before restarting:
kubectl -n local-l1 logs <pod> --tail=200If only
chain-exporter collections fail while the clients are Ready, restart it: kubectl -n local-l1 rollout restart deployment chain-exporterDiagnosis5 queries
Which L1 targets are up:
up{network="lineth-local-l1"}Chain-exporter collection status per API (1 = ok):
{__name__=~"lab_(execution|beacon|beacon_finality)_collection_success", network="lineth-local-l1"}Seconds since the chain-exporter last collected:
time() - lab_sync_exporter_collection_timestamp_seconds{network="lineth-local-l1"}Is the L1 head still moving (seconds since the last execution block):
time() - lab_execution_head_timestamp_seconds{network="lineth-local-l1"}Chain-exporter errors:
{namespace="local-l1", pod=~"chain-exporter-.*"} |~ "(?i)error|timeout|refused"Fix0/3
If a ServiceMonitor or Service changed, fix it in lineth-l2-gitops (
infrastructure/ or local-l1/) rather than by handIf a client's metrics port moved or its metrics flag was dropped, restore the flag in its StatefulSet
Add the cause to the incident and check the other L1 alerts were not silently broken by the same change
Escalation
If the clients are Ready but the APIs do not answer the chain-exporter for more than 15 minutes, treat the L1 as stalled: page the IC and follow l1-chain-stalled. Settlement on L1 (finalization, withdrawals) depends on it.
Seen in INC-2026-10-13-15 · owner s1ns3nz0