[Nic-dorman]

Closes V2-1773

Problem

The stuck-upload sweep can put a live upload back in the queue, and with more than one worker slot it then gets paid for twice:

1. DequeueNext sets processing_at once and never moves it.
2. Every minute the reconcile loop ran RequeueStuck(60), resetting any upload "processing" for over 60 minutes back to "queued". Nothing told it which uploads this process was still working on.
3. With the seeded max_concurrent_uploads of 4, the next 2-second poll dequeued that row into a second goroutine, which prepared and paid again while the first was still paying or storing.

Any upload that takes over an hour hits this. Multi-batch merkle (#162) makes it likely: each batch can wait up to 5 minutes for confirmation, and finalizing ~10 GiB stores ~2,500 chunks.

Startup has a related gap: RequeueStuck(0) requeues everything left "processing", and the next attempt paid again for chunks a crashed attempt had already paid for.

Fix

• In-flight guard.
  • The worker keeps the set of uploads it is processing, from dequeue until processOne returns.
  • The sweep (RequeueStuckExcept) never requeues those uploads.
  • An in-flight upload with no progress past the window is logged once as a warning instead.
  • Indelible runs as a single active writer, so a per-process guard is enough.
• Phase heartbeats. processing_at moves forward at each phase boundary (TouchProcessing): after prepare, after each payment (every merkle batch, and the wave-batch payment), and before finalize. The window now measures time without progress, not time since dequeue.
• No second automatic payment.
  • Before any payment (merkle approve and batches, wave-batch payForQuotes), refuseRepayment checks for a payment recorded since the upload was created or last retried by an admin: a transaction for the upload with transactions.id > uploads.payment_floor_txn_id (migration 015).
  • The floor is 0 at creation, and an admin Retry raises it to the upload's latest transaction id. Sweep and startup requeues, in-attempt retries and gas backoff leave it alone. Transaction ids only grow, so the check doesn't depend on second-precision timestamps.
  • So an automatic re-run of an upload that already paid fails as paid_unfinalized, with its source kept and nothing paid.
  • An admin Retry authorizes one new payment. That's the existing recovery path for paid_unfinalized.
  • If the ledger write for a confirmed payment fails, recordPayment returns the error and the rest of that attempt is paid/no-retry (paidNoRetryIfUnrecorded). Any later failure ends as paid_unfinalized, so no retry runs without the row the check needs. A merkle plan still pays its remaining batches, because finalize needs them.
• Operator control for the window.
  • GET/PUT/DELETE /api/v2/admin/upload-requeue-window and an "Upload queue" card on the admin System page.
  • The default is 60 minutes; the accepted range is 1 minute to 7 days.
  • The override lives in process memory only, so a restart restores the default. That's by design: a window raised to get one long upload through shouldn't outlive the reason it was raised.
  • Every change is audited (upload_requeue_window_changed, warn).
  • Reader instances (workers_enabled=false) report the window but refuse changes with 409.

Behaviour change

A wave-batch finalize failure used to retry by preparing again, and that paid again for any chunks still unstored. That second payment is now refused, and the upload goes to paid_unfinalized. A re-prepare that finds nothing left to pay completes as before.

V2-1774 (stacked PR, next) restores automatic recovery without paying again: it retries the finalize against the payment antd keeps (antd ≥ 0.14).

Known residual

A crash after a payment confirms but before it is recorded (recordPayment) leaves no row for the check to see. That gap is unchanged from today. A failed ledger write without a crash is now handled (above).

Files

• internal/services: RequeueStuckExcept, TouchProcessing, HasPaymentSinceQueued; Retry raises the payment floor. RequeueStuck keeps its signature.
• internal/database/migrations/{sqlite,postgres}/015_upload_payment_floor.sql: uploads.payment_floor_txn_id, backfilled so every existing upload keeps the answer the timestamp comparison gave: just below the lowest payment at or after queued_at if there is one (paid), otherwise the highest payment id (not paid), whatever the order of ids and timestamps.
• internal/worker: requeue_window.go (the override) and upload_inflight.go (the in-flight set, heartbeat, sweep and check), wired into upload.go.
• internal/handlers: admin_upload_requeue.go and the routes. Swagger is regenerated with swag v1.16.6 (additions only).
• web: the System page card.
• FEATURES.md §5.3. The CHANGELOG entry comes with the release.

Verified

On dev1 (Linux, Go 1.26.10), at 3524b6e:

• gofmt and go vet ./...;
• the swagger-up-to-date check;
• go test ./... on sqlite and on postgres 16;
• go test -race on worker, services and handlers;
• npm ci, the web build (vue-tsc + vite) and web unit tests;
• golangci-lint v1.64.8.

New tests:

• services: the exclusion keeps in-flight rows; a heartbeat un-sticks a row and never revives a finished one; paid-since-queued is true after a payment and after a sweep requeue, false after an admin Retry even in the same second as the payment (fails on the old timestamp check), and true again after the payment the Retry authorized; migration 015 on a pre-upgrade database gives each upload the old check's answer: retried after one or two payments (not paid), never retried, paid in the queued second, and ids and timestamps out of order (paid), and no payments (fails without the backfill, and the out-of-order case fails on the round-2 backfill).
• worker: the window override and its bounds; the sweep skips in-flight uploads; the override decides what counts as stuck; heartbeats; the check refuses a second payment and classifies the refusal as preserve-paid; a failed ledger write is reported; after an unrecorded payment a finalize failure classifies as preserve-paid, not retry (a confirmation timeout keeps preserve-unconfirmed).
• handlers: get, set, bad values, reset, audit count, non-admin 403, reader 409.

Devnet (2026-10-29, ant-node v0.23.1, antd 0.17.0, at ee96c9a + #171): the guard fired in every run with a 1-minute window, and no live upload was requeued or paid twice. A SIGKILL after payment followed by a restart refused the second payment and ended paid_unfinalized with one transaction. An admin Retry paid only for the unstored remainder. (Report on V2-1775 / V2-1778.) The review fixes in 3d8c10e are covered by the tests above, not by a devnet run.

Review round 1 (Hermes, ee96c9a): both findings are fixed in 3d8c10e (review range ee96c9a..3d8c10e).

Review round 2 (Hermes, 3d8c10e): the upgrade-path finding (no backfill in 015) is fixed in 38f9380 (review range 3d8c10e..38f9380).

Review round 3 (Hermes, 38f9380): the id/timestamp-order finding is fixed in 3524b6e (review range 38f9380..3524b6e). The backfill now reproduces the old predicate exactly.

🤖 Generated with Claude Code

[MaidSafe-Hermes]

REQUEST_CHANGES

Findings:

1. Retry can be blocked immediately after a paid attempt. internal/services/transaction.go:70-75 treats created_at == queued_at as already paid. SQLite stores both timestamps at one-second precision (internal/database/migrations/sqlite/001_initial.sql:245; uploads.Retry writes CURRENT_TIMESTAMP at internal/services/upload.go:565). If an admin retries a paid_unfinalized upload in the same second as its previous payment record, the old transaction still compares equal to the new queued_at; refuseRepayment then refuses the authorized retry. The test backdates the transaction (internal/services/upload_requeue_guard_test.go:138) and misses this boundary. Use a distinct retry/payment-attempt marker or sufficient timestamp precision, and add a same-second retry test.

2. The no-repayment guard can miss a successful payment when ledger recording fails. recordPayment ignores errors from txnSvc.Record (internal/worker/upload.go:727-734), while refuseRepayment relies only on that row (internal/worker/upload_inflight.go:93-104). If payment succeeds, the insert fails, and a later finalize/retry path runs after DB recovery, the guard sees no payment and may pay again. Return and handle the record error as a paid/no-retry outcome before another payment can be attempted; cover the failure path. The disclosed crash-before-record gap remains a separate residual unless external idempotency or durable intent is added.

Coverage: 2/2 seats reviewed (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3); both seats agree (two model families) on CHANGES_REQUIRED. The reported every-minute discrepancy was checked against reconcileLoop and is not a defect (ticker is one minute). CI checks are green; local Go tests could not run here because Go is not installed.

[MaidSafe-Hermes]

REQUEST_CHANGES

The two prior findings are fixed in this revision: retries now use a monotonic transaction-ID floor, and a failed ledger write prevents a later failure from triggering another payment attempt.

Finding — migration blocks uploads whose retry was already authorized before upgrade. Both internal/database/migrations/{sqlite,postgres}/015_upload_payment_floor.sql add payment_floor_txn_id with default 0, without backfilling existing uploads. If an upload was paid, then an admin retried it before this upgrade, Retry had advanced queued_at but the payment transaction ID remains above the new floor 0. Before this change, created_at < queued_at made that queued retry eligible; after migration, t.id > payment_floor_txn_id reports a payment and refuseRepayment rejects it. I reproduced the old predicate returning false and the new one true for that state using SQLite. Backfill the floor to the latest transaction ID whose created_at < queued_at (preserving the previous authorization semantics), and add an upgrade test with a paid, pre-upgrade retried upload.

The restart window between on-chain confirmation and recording remains the disclosed residual; I did not treat it as a new finding in this re-review. CI checks are green. Local Go tests remain unavailable here because Go is not installed.

Coverage: 2/2 seats reviewed at 3d8c10e (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3). The seats disagreed on the migration behavior; I verified the regression against the old/new predicates and the stated pre-upgrade Retry semantics. The GLM seat approved the incremental change; sol-1 flagged the migration and restart residual. This request is based on the confirmed migration regression, not the disclosed restart residual.

[MaidSafe-Hermes]

REQUEST_CHANGES

The earlier findings are addressed: the transaction-ID floor fixes same-second retries, the worker now handles ledger-write errors, and migration 015 backfills pre-upgrade retries.

Finding — migration can turn a previously blocked payment into an allowed re-payment when transaction IDs and timestamps are not in the same order. In both internal/database/migrations/{sqlite,postgres}/015_upload_payment_floor.sql, the backfill sets the floor to MAX(t.id) among transactions with created_at < queued_at. The replacement guard blocks payment only when some transaction ID is above that floor. If an upload has a row at or after queued_at with a lower ID than another row before queued_at, the backfill puts the floor above the qualifying row and the new guard reports unpaid. The old timestamp guard reported paid. I reproduced this with IDs 100 (at queued_at) and 101 (one second before): old guard true, new guard false. This can authorize a repeat payment during upgrade. Preserve the old blocked/allowed result for all existing rows (for example, backfill a state that blocks whenever the old EXISTS(created_at >= queued_at) was true, and otherwise sets the floor above all existing transaction IDs), or prove and enforce a per-upload ID/timestamp ordering invariant; add a migration test with mixed ID/timestamp order.

CI checks are green. Local Go tests could not run here because Go is not installed.

Coverage: 2/2 seats reviewed at 38f9380 (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3). The seats disagreed: sol-1 flagged this migration edge; glm-1 approved. I verified the failing migration state with SQLite. This request is based on the potential re-payment path, not the remaining disclosed crash-before-ledger-record residual.

[MaidSafe-Hermes]

APPROVE

The migration now preserves the old paid/unpaid result for existing uploads even when transaction IDs and timestamps are out of order. The upgrade test covers that regression, prior retries, same-second payments, multiple pre-retry payments, and uploads with no payments. The earlier review findings are resolved.

Coverage: 2/2 seats reviewed at 3524b6e (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3); both seats approve. CI checks are green, including SQLite and PostgreSQL tests. I could not run local Go tests because Go is not installed here. The documented crash-between-payment-and-record residual remains acknowledged.