L2 RPC node behind the sequencer, without peers, or holding user transactions
criticalreviewed 2026-10-13What it meansmoreUsers send transactions to the public RPC node l2-node-besu, which gossips them to the sequencer over P2P and imports its blocks. With no peer, txs stay in the RPC node's pool and its head falls behind; shomei (the state manager) follows the RPC node, so it lags too. In INC-13 the RPC node started before cluster DNS answered, never resolved its one static peer and never retried: no user tx was included for 49 minutes while pods looked healthy.
ImpactmoreUsers' transactions are not included (tx-inclusion SLOs burn); reads from the RPC are stale. Deposits claimed by the postman through the RPC node wait too (INC-14 followed INC-13). LinethL2NoPeers is SEV1 while user txs are not included; lagging without stuck txs is SEV3.
Quick mitigation0/4
Check peers and the head gap (queries in Diagnosis)
No peer: restart the RPC node so it re-resolves its static peer:
kubectl -n lineth-l2 delete pod l2-node-besu-0Peer back but still behind: wait a few minutes and watch the gap close; restart only if it does not move
After recovery, check the postman did not lose pending claims (INC-14): see
l2-bridge-messagesDiagnosis4 queries
Peers per L2 node (0 = isolated):
max by (job) (ethereum_peer_count{job=~"l2-sequencer|l2-node-besu"})Blocks the RPC node and shomei are behind the sequencer:
max(ethereum_blockchain_height{job="l2-sequencer"}) - max(ethereum_blockchain_height{job="l2-node-besu"})max(ethereum_blockchain_height{job="l2-node-besu"}) - max(shomei_blockchain_head)Age of the oldest tx in the RPC node's pool (seconds):
max(lineth_l2_rpc_txpool_oldest_age_seconds)Fix0/2
Keep the startupProbe on
/readiness?minPeers=1 so a node without a peer restarts on its own (INC-13 A1)If the static peer is a DNS name, make sure every client that resolves names once at start has the same guard
Escalation
If the RPC node does not peer after two restarts, page the IC: check the sequencer's P2P port and Service, and consider pointing users at the sequencer RPC until it is fixed.
Seen in INC-2026-10-13-13 · SLOs tx-inclusion-5s, tx-inclusion-60s, rpc-availability · owner s1ns3nz0