L2 telemetry missing: a scrape target is down or the l2-exporter stopped collecting
criticalreviewed 2026-10-13What it meansmorePrometheus cannot scrape an L2 target (up == 0), or the l2-exporter in lineth-l2 stopped collecting (its last collection is older than 60 s). The l2-exporter turns chain state into the metrics most L2 SLOs and alerts use (finalized block, liveness, workflows), so when it stops, the L2 SLOs freeze and real problems go unseen.
ImpactmoreNo direct user impact; detection for finalization, liveness and the SLO burn alerts is blind while it lasts. Critical because it hides other incidents: SEV3, SEV2 if it lasts over 30 minutes during another open incident.
Quick mitigation0/4
See which target is down (query in Diagnosis) and whether the pod is Ready:
kubectl -n lineth-l2 get podsl2-exporter stale but Ready: check its logs for RPC errors, then
kubectl -n lineth-l2 rollout restart deployment l2-exporterA component's target down: the component itself is probably down; follow that component's runbook
Note on the open incidents that L2 detection was blind from when to when
Diagnosis3 queries
L2 targets that are down:
up{network="lineth-local-l2"} == 0Seconds since the l2-exporter last collected:
time() - max(lineth_l2_exporter_last_collection_timestamp_seconds)l2-exporter errors:
{namespace="lineth-l2", pod=~"l2-exporter-.*"} |~ "(?i)error|timeout|refused"Fix0/2
If the exporter failed on an RPC change (method removed, node not synced), fix the exporter or its target in lineth-l2-gitops
Make sure a stale exporter fails its readiness probe so Kubernetes shows it
Escalation
If the exporter cannot be brought back within 30 minutes, page the IC and watch the L2 by hand (block height, finalized block, prover queue) until it is back.
Seen in INC-2026-10-11-09 · owner s1ns3nz0