orphan prover requests after the P1 coordinator config change

Draft
Reviewed
Closed

Impact

14:32:27 (coordinator restart) → 14:38 (mitigated)

Severity

~6 min of wasted proving, no user-visible effect

Components

provercoordinator

Authors

s1ns3nz0, Claude

Blameless postmortem. Incident record: incident-log.md INC-2026-10-10-02. Times UTC. Block numbers refer to the chain before the 2026-10-12 reset (archived).

1Summary

The P1 change restarted the coordinator with new conflation settings. On restart the coordinator re-conflates from the last proven block, so request files written before the restart (60-block boundaries) no longer matched any batch it would aggregate. The prover kept working on them: at 14:38 it was proving 4021-4080, which nothing would use. The runbook says to remove request files older than the coordinator pod after such a change; that step was skipped.

2Impact

what	how much
prover time wasted	~6 min on an already saturated prover (INC-01)
orphan files	10 (execution + compression for 4021-4080, 4081-4140, 4141-4200, 4201-4260, 4261-4320)
not affected	correctness: orphans are never aggregated

3Root causes and trigger

Trigger: coordinator restart with a different batch size.
Root cause: the coordinator does not clean up requests it no longer needs, and the prover does not know which requests are current; the cleanup is a manual runbook step that was not part of the change.
Tooling gap: the prover image's BusyBox find has no -newermt, so the runbook command needs a marker file.

4Detection

Seen at 14:38 while watching the backlog of INC-01 (the prover was on an unexpected range). No alert covers "prover works on a request the coordinator will not use".

5Response and recovery

14:32:27
coordinator restarted with blocks-limit 120, deadline PT12M; re-conflates from 3961 in 113-block batches
14:38
prover found on 4021-4080 (old boundaries); 10 files moved to /data/prover/v3/orphans-20261010-p1/ (kept); kubectl -n lineth-l2 rollout restart deploy/prover; prover goes to 3961-4073

6Lessons learned

Went well

files were quarantined, not deleted, so nothing was lost if the analysis was wrong.

Went badly

a known runbook step depended on memory. The same step was needed again in INC-11 (twice) and INC-12 (twice) — there it was done on purpose, but always by hand.

Lucky

—

7Action items

P3
Script it: list requests whose ranges do not start at the coordinator's "Resuming conflation from block=…"
Prevents1ns3nz0Open
P4
Delete the 10-10 quarantine dir once P1 settled
open (the chain was reset 10-12; the dir is on the archived data)
Processs1ns3nz0Open
P2
Make orphan cleanup part of every coordinator config change (checklist)
done in practice (INC-11, INC-12 moved orphans); not scripted
Processs1ns3nz0✓ Done

8Supporting data

Coordinator log line to anchor the cleanup: Resuming conflation from block=<n> inclusive (seen in INC-12).

Loading…

1Summary

what

how much

prover time wasted

~6 min on an already saturated prover (INC-01)

orphan files

10 (execution + compression for 4021-4080, 4081-4140, 4141-4200, 4201-4260, 4261-4320)

not affected

correctness: orphans are never aggregated

3Root causes and trigger

Trigger: coordinator restart with a different batch size.
Root cause: the coordinator does not clean up requests it no longer needs, and the prover does not know which requests are current; the cleanup is a manual runbook step that was not part of the change.
Tooling gap: the prover image's BusyBox find has no -newermt, so the runbook command needs a marker file.

5Response and recovery

14:32:27
coordinator restarted with blocks-limit 120, deadline PT12M; re-conflates from 3961 in 113-block batches
14:38
prover found on 4021-4080 (old boundaries); 10 files moved to /data/prover/v3/orphans-20261010-p1/ (kept); kubectl -n lineth-l2 rollout restart deploy/prover; prover goes to 3961-4073

7Action items

P3
Script it: list requests whose ranges do not start at the coordinator's "Resuming conflation from block=…"
Prevents1ns3nz0Open
P4
Delete the 10-10 quarantine dir once P1 settled
open (the chain was reset 10-12; the dir is on the archived data)
Processs1ns3nz0Open
P2
Make orphan cleanup part of every coordinator config change (checklist)
done in practice (INC-11, INC-12 moved orphans); not scripted
Processs1ns3nz0✓ Done