deposits not claimed for ~1 h 50 min — postman nonce gap and a broken self-repair

Draft
Reviewed
Closed

Impact

04:20 (first unclaimed deposit) → 06:09 (all claimed) ≈ 1 h 50 min

Severity

the deposit bridge (L1 → L2) stopped delivering for ~1 h 50 min

Components

postmanl2-node-besu

Authors

s1ns3nz0, Claude

Blameless postmortem. Incident record: incident-log.md INC-2026-10-13-14. Times UTC.

1Summary

The postman claims deposits on L2 by sending a claim transaction per message from one signer. During INC-13 its claim transactions went to the peer-less RPC node and never reached the sequencer; two restarts of that node dropped them. 41 messages were left PENDING with nonces 197–237 while the chain's nonce for the signer stayed at 197. The postman chooses the next nonce as max(on-chain, highest PENDING nonce + 1), so every new claim also landed above the gap and could not be mined. The postman has a repair path for exactly this — cancel the stuck nonce with a self-transfer and reset the message — but it crashed on every attempt with TypeError: Cannot mix BigInt and other types. The 41 rows were reset by hand the way the repair path would have, the postman was restarted, and it re-claimed from nonce 197.

2Impact

what	how much
W2 deposits (ETH and TERC20)	41 not claimed on L2, up to ~1 h 50 min late (04:20 → 06:09); every message claimed exactly once in the end
duplicates	2 messages got a second claim tx that reverted (`MessageDoesNotExistOrHasAlreadyBeenClaimed`): gas only
not affected	W1, W3 (L1 claims use another signer), W4, W5

3Root causes and trigger

Trigger: INC-13 — 41 signed claim txs lost from the RPC node's pool.
Root cause 1 — no re-broadcast: nothing re-sends a pending tx whose hash vanished from every pool; the DB still counted it as pending, which pinned the nonce floor above the gap.
Root cause 2 — postman bug (consensys/linea-postman:1.0.1-20260629-7358b2f): the entity declares claim_tx_max_fee_per_gas / claim_tx_max_priority_fee_per_gas as bigint, but the pg driver returns int8 as a string and messageMappers.ts passes it through; ViemTransactionRetrier.cancelTransaction multiplies it by 110n → TypeError. The cancel-and-reset path can never succeed in this deployment.
Open question: why the two token-bridge messages got two claim txs during the re-claim.

4Detection

LinethL2MessagesStuck fired at 05:07:38 into the alert-sink log. Found by hand at 05:56 while checking all six workflows after INC-13: deposit spans had been 0 since ~04:56, 41 messages PENDING in the postman DB.

5Response and recovery

04:17:08
last deposit claim before the VM restart
04:24–05:46
41 claim txs (nonces 197–237) sent to the peer-less RPC node; lost at its restarts (04:56, 05:26)
05:56
found; on-chain nonce 197 vs postman nonces 232+; no tx of the signer in any pool
06:02
postman's own repair starts, fails every 4 s (TypeError, source read to confirm)
06:06
41 rows backed up to CSV (session scratchpad)
06:06:43
postman scaled to 0; rows reset (status='SENT', claim tx fields cleared, claim_cycle_count+1) in one transaction
06:06:44
postman scaled to 1: NonceManager initialized startNonce=197 onChainNonce=197 dbMaxNonce=null
06:07–06:09
nonce 197 → 248: 39 CLAIMED_SUCCESS, 2 CLAIMED_REVERTED (already claimed on-chain: L2 inbox status 2)
06:12–06:14
new deposits claimed (4), deposit spans back (~46 / 15 min)

6Lessons learned

Went well

the repair copied the code's own intended reset instead of inventing one; the rows were backed up first; the source was read to explain the crash before touching the DB.

Went badly

one incident's recovery (INC-13: restart the RPC node) silently caused this one; the six-workflow check that found it was done ~50 min after INC-13 was "resolved".

Lucky

the token-bridge duplicates reverted harmlessly because the contract rejects a second claim.

7Action items

P2
Find why two claims were sent for the same message during the re-claim
open (after A1)
Prevents1ns3nz0Open
P2
Alert when a signer's pending-in-DB nonce is ahead of its on-chain nonce for > 10 min (nonce gap)
Detects1ns3nz0Open
P1
Patch the postman: convert the two fee columns with BigInt(...) in messageMappers.ts (or a TypeORM transformer); build a lab image; test by dropping a pending claim
10-13 06:45 1fb7983: lineth-postman:1.0.1-7358b2f-lab1 (images/lineth-postman/; 439/439 postman tests, 2 new mapper tests fail without the fix). Upstream checked 10-13: no issue/PR, bug still on LFDT-Lineth/lineth-monorepo main 5c0c4caa6 (introduced with the cancel path in #2617); filing upstream not done. Live drill of the cancel path not done
Prevents1ns3nz0✓ Done
P1
After restarting a node that holds others' pending txs (RPC node), check every signer's on-chain vs DB nonce (postman L1/L2, traffic) — runbook step
— scripts/check-workflows.sh (nonce check) + docs/runbooks/node-restart.md (52b00b5)
Processs1ns3nz0✓ Done
P0
Route alerts to a person
10-13 11:00 934a2bb: Telegram receiver on lab alerts (critical + warning, repeat 1 h); see INC-15
Detects1ns3nz0✓ Done

8Supporting data

Signer 0xc8c9…7a62: eth_getTransactionCount latest = pending = 197 on sequencer and RPC node at 05:5x.
Postman DB before the reset: 41 PENDING, claim_tx_nonce 197–237, distinct, created 04:20:38–05:46:48.
Crash: TypeError: Cannot mix BigInt and other types at ViemTransactionRetrier.js:56 (source: postman/src/infrastructure/blockchain/viem/ViemTransactionRetrier.ts cancelTransaction).
Reverted duplicate: tx 0x80eec5…6994 for message 0xf83eb1…c918; the successful claim was nonce 225, L2 block 8630.

Loading…