`linea_getBlockTracesCountersV2` returns INTERNAL_ERROR on the public RPC

Draft
Reviewed
Closed

Impact

first seen 2026-10-10 12:40 (after the T1 RPC restart)

Severity

the coordinator retries; no batch lost; spends the rpc-server-errors budget

Components

coordinatorl2-node-besu

Authors

s1ns3nz0, Claude

Blameless postmortem (interim: cause not found). Incident record: incident-log.md INC-2026-10-10-06. Times UTC.

1Summary

Since the T1 drill restarted the public RPC node (l2-node-besu), some calls to linea_getBlockTracesCountersV2 fail with INTERNAL_ERROR. The coordinator uses this call to count trace lines per block when it cuts batches; it retries the block after 1 s and gets an answer, so no batch has been lost. The stack trace points into the line counter's fork lookup (Fork.getForkFromBesuBlockchainService in LineCounterGenerator.createLineCountingTracer). The error survived the 10-12 chain reset, so it is not tied to old chain data.

2Impact

what	how much
batches	none lost; per-block retries with 1 s backoff
`rpc-server-errors` SLO	budget spend (99.39 % left on 10-10)
error rate	10-10: burst after the restart, then ~8 / 30 min; 10-13: 5–8 / 30 min, 102 in 6 h on `l2-node-besu` (`besu_rpc_errors_count_total{errorType="INTERNAL_ERROR"}`)

3Root causes and trigger

Trigger: the T1 restart of l2-node-besu (burst right after it).
Root cause: unknown. Hypotheses, none tested yet: (a) a race between block import and the line-counting tracer creation (fork lookup on a block the blockchain service does not expose yet); (b) a plugin version mismatch between the RPC node's tracer and the fork schedule.

4Detection

The rpc-server-errors SLI and coordinator counters WARN logs.

5Response and recovery

None yet beyond observation; the coordinator's retry contains it.

6Lessons learned

Went well

an SLI on server errors made a silent retry loop visible.

Went badly

the investigation steps were written down on 10-10 and not done for three days.

Lucky

—

7Action items

P3
Capture the full stack trace and the block number for one failure
Detects1ns3nz0Open
P3
Check whether the same block succeeds on retry and on the sequencer (its own counters)
Detects1ns3nz0Open
P4
Check the import-time ZkBlockImportTracerProvider WARNs together (same plugin)
Detects1ns3nz0Open

8Supporting data

PromQL (10-13): sum by (job, errorType) (increase(besu_rpc_errors_count_total[6h])) > 0 → l2-node-besu INTERNAL_ERROR 102.

Loading…

1Summary

what

how much

batches

none lost; per-block retries with 1 s backoff

rpc-server-errors SLO

budget spend (99.39 % left on 10-10)

error rate

10-10: burst after the restart, then ~8 / 30 min; 10-13: 5–8 / 30 min, 102 in 6 h on l2-node-besu (besu_rpc_errors_count_total{errorType="INTERNAL_ERROR"})

3Root causes and trigger

Trigger: the T1 restart of l2-node-besu (burst right after it).
Root cause: unknown. Hypotheses, none tested yet: (a) a race between block import and the line-counting tracer creation (fork lookup on a block the blockchain service does not expose yet); (b) a plugin version mismatch between the RPC node's tracer and the fork schedule.