Retry temporary collector rejections per run instead of making the run local-only #1237

Problem

Since #1147 (0e80b03), any 4xx from the telemetry collector other than 408 and 429 makes a run local-only. Local-only is permanent: EventSpool.make_local_only excludes the run from upload, and nothing resets it, including re-registration and restarts.

That rule was added because the old behaviour retried a rejected batch forever and wedged delivery. The fix trades that for silent data loss in cases that would otherwise recover:

• A 422 from an older collector is temporary. The main case is a blocked event (staging contract v3) sent to a collector without calibration-diagnostics#206. Once the collector is deployed, the same batch is accepted. Today the run never reaches the hosted view.
• The whole run is dropped, not only the rejected event. Every later event stays local, including the run's final completed, failed or blocked event.
• 404 counts as permanent. A collector that is mid-deploy or mis-routed costs each affected run its hosted telemetry.

Root cause of the original wedge

The wedge came from how retries are scheduled, not from retrying itself:

• CollectorDelivery._next_attempt_at and _retry_seconds are one timer for the whole service. A run that keeps failing delays the delivery pass for every other run, and the backoff only resets when another run succeeds.
• spool.batch() always returns the same first 100 unacknowledged events, so a refused batch is head-of-line for that run. That part is correct, because events must arrive in order.

Proposal

1. Back off per run. Store next_attempt_at and retry_seconds on TelemetryRunRecord (an Alembic migration) instead of on CollectorDelivery. A failing run waits on its own schedule, and other runs keep delivering at full speed.
2. Keep temporary rejections pending. Retry 404, 409 on events and 422 with per-run backoff and a long ceiling (for example, one hour). The run stays pending until delivery succeeds or the 7-day retention removes its events.
3. Keep the permanent list narrow. Make a run local-only only on:
   • authorization refusals, as before: no Hugging Face token, a rejected credential, a 403, or a 409 registration conflict;
   • requests that can never succeed: 400 and 413.
4. Warn once while retrying. On the first temporary rejection for a run, print one warning naming the status and saying delivery will be retried, so an operator can see a version mismatch.

Acceptance

• A collector that returns 422 and then 202 delivers every event of the run, in order, with no local-only state.
• While one run is getting 422s, another run's events are delivered on the next pass without waiting for the failing run's backoff.
• 400 and 413 still make the run local-only, with one warning.
• 401 still refreshes the token. 408 and 429 still retry.
• Existing spools migrate with every run's per-run backoff at its starting value.

Context

Review round 1, finding 2 on #1147 introduced the current rule to soften the deployment-order constraint with calibration-diagnostics#206. That constraint still holds for releases. This issue is about not losing runs whenever a collector is behind.