L2 database backup stale
warningreviewed 2026-10-13What it meansmoreThe db-backup CronJob in lineth-l2 dumps the L2 Postgres databases (coordinator, postman, transaction exclusion, blockscout) once a day. No successful run in 26 hours means the last good backup is older than a day.
ImpactmoreNo impact now. If a database is lost before the next good backup, the coordinator or postman state since the last backup is lost too (batches, proofs and claims would have to be rebuilt from chain data). Warning; SEV3 if it stays stale over 48 hours.
Quick mitigation0/3
Check the last jobs:
kubectl -n lineth-l2 get jobs --sort-by=.metadata.creationTimestamp | tail -5Read the failed job's logs:
kubectl -n lineth-l2 logs job/<name>Run one now once the cause is fixed:
kubectl -n lineth-l2 create job --from=cronjob/db-backup db-backup-manual-$(date +%s)Diagnosis2 queries
Hours since the last successful backup:
(time() - max(kube_cronjob_status_last_successful_time{namespace="lineth-l2", cronjob="db-backup"})) / 3600Failed backup jobs:
sum(kube_job_status_failed{namespace="lineth-l2", job_name=~"db-backup-.*"})Fix0/2
Fix the cause (storage full, a database renamed, credentials rotated) in lineth-l2-gitops
Restore one backup into a scratch database to prove it works; an untested backup is not a backup
Escalation
If backups cannot run for 48 hours, open an incident and take a manual dump of the coordinator and postman databases before any risky change.