Fix job resubmission loop: retry cap, backoff, atomic claim

[shakee93]

Problem

Production evidence: 81% of critical-CSS jobs received by the API were repeats of an earlier job for the same URL; one site submitted 1,475 jobs in 24h for 7 URLs. A failing URL was resubmitted on every queue tick (60s) with no cap and no backoff. (The API is also getting a server-side dedupe in a separate PR; this PR fixes the plugin side.)

Root causes (file:line on master)

• A. RapidLoad_CriticalCSS_Store::update_css (~236-246) and RapidLoad_UnusedCSS_Store::update_css (~250-260) call requeue() unconditionally on Job processing failed in queue / Error.
• B. RapidLoad_CriticalCSS_Store::purge_css (~121-150) and the UnusedCSS equivalent: a response with neither id nor completed (and no errors) is ignored, so the row stays queued and is resent every tick.
• C. No atomic claim: the row leaves queued only after the POST returns (timeout 360s, RapidLoad_Api::post); WP-Cron and rapidload_cron.php can overlap.
• D. RapidLoad_Job_Data::mark_as_successful_hit reset attempts on every page view (defeats any cap); RapidLoad_CriticalCSS::cleanCacheFiles used get_success_data() with default LIMIT 1, so only one URL's files were kept and the rest deleted, forcing requeues. RapidLoad_UnusedCSS::cleanCacheFiles is an empty stub, so UUCSS is not affected.
• E. gmdate("Y-m-d H:m:s") (month in the minutes slot) in RapidLoad_Job_Data (x2), RapidLoad_Job, RapidLoad_Job_Optimization.

Changes (one commit each)

1. Date format H:m:s -> H:i:s (needed for correct backoff timestamps).
2. Attempts are never reset by hits or by success (resetting on success re-opened the loop for success-with-warnings / missing-file pages, because the page-view guards attempts <= 2 / failed && attempts >= 2 rely on attempts growing). Attempts are reset only by explicit actions: requeue(0) for immediate/requeue args and clear_cache (save_post, manual purge), and the bulk admin SQL (requeue_where_status, clear_data(soft), requeueJob, update*JobDataStatusWhere now set attempts = 0, created_at = now). Page-view/automatic requeues (both Enqueue classes, cache_cpcss/cache_uucss without immediate/requeue) use new requeue_delayed(): attempts + 1 plus the same exponential backoff as retry_or_fail.
3. cleanCacheFiles passes limit 0 to RapidLoad_CriticalCSS_DB::get_success_data; 0 means all rows, default stays 1 for other callers.
4. Retry cap + backoff: new RapidLoad_Job_Data::retry_or_fail(). Max attempts default 3 (RAPIDLOAD_JOB_MAX_ATTEMPTS or filter rapidload/job/max_attempts); backoff 120s, 240s, 480s... capped 1h (filter rapidload/job/retry_base_delay). Backoff is stored as a future created_at (no schema change). The queued-job SELECTs (CPCSS and UUCSS) add AND (attempts IS NULL OR attempts < 1 OR created_at <= <now>). After the cap: mark_as_failed + uucss/cache_cleared.
5. Unexpected purge response (no id, no completed, e.g. {"message":"rate limited"}) counts an attempt via retry_or_fail, failed after the cap, for CPCSS and UUCSS. Also guards non-object results.
6. Atomic claim + lock (hardened in follow-up commits: unique lock token <token>|<time>, release/refresh only by the holder, touch_lock() before every job, release also from a shutdown function so PHP Errors/fatals/exit are covered without Throwable; cache_cpcss/cache_uucss and the immediate/titan Store guards never requeue or re-submit a row that is submitting; transient submit errors (null/timeout, cURL string, 5xx, 429) go through retry_or_fail, 4xx fail at once; the admin UI sees submitting as waiting via display_status()): claim() does UPDATE ... SET status='submitting' WHERE id=%d AND status='queued' before the POST and skips if 0 rows. release_stale_claims() runs at the start of fetch_job_id: a row in submitting for > 600s (longer than the 360s HTTP timeout) is treated as a failed attempt (back to queued with backoff, or failed at the cap), so a crashed run can neither strand a row nor loop forever. RapidLoad_Queue::run_task() takes an INSERT IGNORE options-row lock (expires after 600s) and is used by both WP-Cron and rapidload_cron.php.

Worst-case submissions per URL per day (max attempts 3, base delay 120s)

Path	Bound
API error (poll returns Error / Job processing failed, or unexpected/5xx/timeout submit response)	3 submissions total (attempt 1 -> wait 2 min, attempt 2 -> wait 4 min, attempt 3 -> failed). Then terminal: failed and attempts >= 2 blocks automatic requeues; only an explicit action restarts it.
Success but not injected (warnings)	Page-view requeue allowed while attempts <= 2: at most 3 resubmissions (waits 2, 4, 8 min between), then at most 1 per 24h (created_at age guard, refreshed at each requeue). So <= 3 on day one, <= 1/day afterwards.
CPCSS success but file missing (cache dir not writable)	Same guard and same bound as above (<= 3, then <= 1/day).
save_post / importer storm	Throttled: the content-change path (clear_on_actions -> clear_cache(throttle)) skips the requeue if the row was (re)queued or claimed < 10 min ago (rapidload/job/save_post_min_interval, minutes), keeping files. At most ~6 submissions/hour/URL (144/day). Manual purge and immediate/requeue args are not throttled.

Late review fixes

• clear_cache(job) skips (and logs) when the row is submitting, so a save/purge mid-POST cannot flip it back to queued and cause a duplicate claim. Chosen over a dirty marker: no existing field fits without a schema change, and the next change/purge requeues normally.
• update{Url,Rule}JobDataStatusWhere bound a comma-joined id list to one %d (only the first id was updated, including "requeue all"); now builds IN (%d,%d,...) from intval'd ids via prepare, returns early on an empty list.

Backward compatibility

• No DB schema change, no new options (lock is a transient-style options row, removed after each run), no API contract change.
• PHP 5.4/5.6 compatible syntax (no ??, no finally, no scalar types); php -l clean on all changed files.
• Existing rows: attempts 0/NULL are unaffected; user/page-view requeues set created_at = now, so they stay immediately eligible. Legacy H:m:s timestamps can be at most ~12 min in the future (only matters for rows with attempts >= 1).
• New transient status submitting (varchar(15) fits) exists only during the POST.
• Out of scope, noticed: RapidLoad_UnusedCSS_DB::get_current_processing_tasks_job_ids has an unparenthesised OR in its WHERE.

Test plan / scenarios

Verified with a stubbed harness (no WordPress available) for the Job_Data logic, plus php -l and phpcs (no new findings vs baseline):

• (a) poll returns Error repeatedly: attempt 1 -> requeued, +120s; attempt 2 -> requeued, +240s; attempt 3 -> failed (3 submissions total).
• (b) POST returns {"message":"rate limited"}: same path via retry_or_fail; stored error is the message.
• (c) two claims on the same row: first true, second false (second run does no POST).
• (d) mark_as_successful_hit leaves attempts unchanged (2 -> 2); success does not reset attempts either; page-view requeues stop after 3 within the first window.
• Lock: second runner is refused; a stale lock is taken over; the old holder's release does not delete the new holder's lock (stub harness).
• (e) cleanCacheFiles now builds the in-use list from all success rows (code review; not run against a live site).

Manual to do on a staging site: point at a failing URL and watch rapidload_job_data (attempts/created_at/status) and the API request log across several 60s ticks; run php rapidload_cron.php concurrently with a WP-Cron tick and confirm one POST per row.

Review follow-up (551e1d7)

Supersedes the parts above that disagree with it:

• Retry cap is per failure episode. retry_or_fail now caps consecutive failures (transient keyed by job-data id + the attempts value, no schema change). Before, attempts grew with page-view requeues and never reset on success, so a long-lived URL at attempts >= 2 failed permanently on its first 5xx/timeout. Success, explicit resets and page-view requeues end the episode. Backoff follows the retry step (120s, 240s), not lifetime attempts.
• 429 + Retry-After (rapidload-api#4): the row is deferred until the API window resets (clamped 60s..1 day) and the retry cap is not used. RapidLoad_Api records Retry-After. A 429 without the header is still a counted retry.
• save_post throttle: 60 min by default (was 10), so content changes produce at most 24 submits per URL per day, matching the API per-URL limit. Throttled changes, and changes made while a submit is in flight, are no longer dropped: one deferred requeue (a single WP-Cron event rapidload/job/deferred_requeue per job) replays clear_cache after the interval, so the latest content is always regenerated.
• Titan second CPCSS strategy claims the row from any status except submitting (it was dropped whenever the row wasn't queued, e.g. after the immediate strategy succeeded or failed).
• release_stale_claims deletes orphaned rows, orders by age and fires uucss/cache_cleared when it gives up.
• Bulk requeue SQL skips submitting rows.
• Fixed the unparenthesised OR in RapidLoad_UnusedCSS_DB waiting/processing queries. It counted and polled cpcss rows as uucss.

Updated worst case for the save_post row in the table: at most 1 requeue per 60 min per URL from content changes (24/day), plus at most 2 backoff retries per episode on transient errors.

Verified: php -l clean. phpcs adds one finding vs this branch's previous head: the existing uucss/cache_cleared hook name, which is already flagged throughout the codebase. A stub harness for RapidLoad_Job_Data passed 20/20 checks: per-episode cap, success/explicit/page-view resets, 429 deferral and clamps, 4xx/timeouts, deferred-requeue scheduling and dedupe. The failing deploy check is the preview environment's "Configure SSH" step (infra/secrets), unrelated to this code.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
autoptimize-unusedcss	Ignored		Oct 12, 2026 5:55am UTC