Cancel source extraction deadlines and isolate scheduler failures

docs/implementation-checkpoint-20260912.md

@@ -0,0 +1,113 @@
+# Implementation checkpoint — 12 September 2026
+
+## Recovered baseline
+
+The September 6 local branch is superseded. Continue from backend
+`e9eb7fa8b2d10146480377e8c51854991c728194` and app/schema
+`19a402ca28c61b53d17a6b12f14ca047ed5917e9`. Reapplying that old branch would
+regress the manual deployment controls and duplicate already merged work.
+
+The persistent staging rehearsal and production schema cutover are complete
+according to the September 9 rollout attestation in
+[`reliability-repair-progress.md`](reliability-repair-progress.md). The guarded
+[deployment run](https://github.com/OneClickPostFactory/social-agents/actions/runs/34354662402)
+confirms Worker version `773bb17d-59ef-484d-ae74-2f7c004e3447`, from
+`3a27a993742151a6558089c9ab0ade6b1a762ba4`, with generation and provider dispatch
+disabled. This is an inert canary, not end-to-end product acceptance. No new
+production deployment, schema change, paid API call or social post was made
+during this continuation.
+
+## Repairs in this continuation
+
+The worker cancels angle extraction through the HTTP transport, response read
+and retry wait. It awaits settlement before releasing the source claim. This
+prevents abandoned local work; cancellation cannot guarantee that a provider
+has not already incurred a charge. Durable generation accounting remains a
+separate release requirement.
+
+Each scheduling stage, tenant and due queue row has an error boundary. A failed
+fetch, tenant lookup, recovery query or log write cannot suppress independent
+publish work. `SchedulerStats.errors` reports failures by stage. Existing
+entitlement, canary, immutable hold and publication ledger gates still apply.
+Scheduling remains serial; this change does not provide durable fairness,
+enqueue uniqueness, or an overall tick deadline.
+
+The build uses Node's `--import tsx` entrypoint, avoiding the tsx CLI's unnecessary
+IPC listener. Both new regression files are included in the required test command.
+Two existing static wiring assertions now accept the scheduler's stage wrapper;
+their prohibition on legacy publication reconciliation remains intact.
+
+The companion app repair checks billing database results, requires immutable
+customer ownership, makes X OAuth state single use, and permits authenticated
+credential clearing without paid access. Webhook acknowledgement and delivery
+order are separate concerns; the former is repaired and the latter still needs
+the durable inbox/projection release described below.
+
+## Reviewed completion plan
+
+Every phase must leave its current consumer, schema contract, negative tests,
+rollout controls and recovery procedure executable before moving to the next.
+Merge and deployment are distinct checkpoints.
+
+| Phase | Current state | Remaining implementation and acceptance gate |
+| --- | --- | --- |
+| Release identity and CI | Merged; inert deployment verified | Deploy only an approved exact SHA through the existing guarded workflow. |
+| Tenant runtime and database claims | Scope isolation, source/angle fencing and publication ledger merged | Add durable job-enqueue uniqueness and terminal job ownership; demonstrate competing Workers cannot create duplicate logical jobs or finish a replacement owner's job. |
+| Worker cancellation and isolation | Implemented in this repair | Full CI and exact-commit PR checks; retain all publication ambiguity tests. |
+| Billing | Error acknowledgement, ownership and period compatibility repaired in companion app branch | Environment-owned configuration and environment/customer mapping; durable event inbox; serial canonical Stripe reconciliation; atomic projection/audit commit; checkout intent and lifetime trial eligibility. Test duplicate, reversed, concurrent and crash/replay deliveries without changing another tenant or environment. |
+| Connection lifecycle | Single-use X callback and unpaid credential clearing implemented | Credential versions and compare-and-swap persistence for callback, refresh, reconnect and disconnect across app and Worker. A delayed refresh/callback must never restore a disconnected credential. |
+| Generation | Claims and cancellation exist | Reserve a durable generation operation and budget before any paid request; classify ambiguous attempts; preserve outputs; enforce one spend for a logical operation across restarts. Never infer a durable budget from best-effort logs. |
+| Scheduling | Tenant/stage failures isolated | Durable cursor, bounded tenant batches, per-tenant fairness, exact enqueue identity and queue-age monitoring. Test a healthy tenant behind more than one page of blocked rows. |
+| Providers and ingestion | Existing retirement/quarantine boundaries enforced | Validate the installed Reddit connector end to end. Establish the hosted tenant contract for Threads/Instagram without reviving retired adapters. Verify current LinkedIn compatibility and tenant X identity. Facebook needs an explicit supported target; legacy Groups and frontend Page credentials are not interchangeable. |
+| Product status and release acceptance | Health and ledger provide partial truth | UI consumes connection, generation and per-platform publication states, including unknown outcomes. Align marketing with enabled capabilities. Complete one authorised scheduled canary per supported provider, then a monitored soak and recovery drill. |
+
+## Original defect register reconciliation
+
+“Repaired here” means code and regression coverage on the repair branch. It does
+not mean deployed. “Merged” refers to the recovered baseline; it does not prove
+live account compatibility or every failure mode.
+
+| Finding | Current disposition |
+| --- | --- |
+| D01 release/deploy verification | Merged: manual exact-SHA gate and full CI. |
+| D02 hosted Meta publisher gap | Open; retired adapters remain closed. |
+| D03 shared tenant runtime | Runtime scope isolation merged; connection versioning remains. |
+| D04 angle/source races | Atomic database claims and fenced finalisation merged. |
+| D05 job creation races | Open for worker-owned enqueue paths. |
+| D06 ambiguous database mutation retries | Merged: ordinary mutations single attempt; explicit idempotent RPC retry. |
+| D07 provider success followed by bookkeeping failure | Publication ledger/executor merged; ambiguity cannot authorise resend. |
+| D08 source URL used as publication identity | Removed from production reconciliation; exact queue/attempt identity merged. |
+| D09 stale recovery ownership | Publication/source/angle protections merged; terminal agent-job fencing remains. |
+| D10 uncancelled extraction deadline | Repaired here, including transport/body/backoff cancellation tests. |
+| D11 generation spending from best-effort logs | Open: durable budget and generation operation ledger required. |
+| D12 scheduler failure propagation | Repaired here at stage, tenant and row boundaries. |
+| D13 first-page starvation | Open: durable fair scheduling and bounded cursors required. |
+| D14 incomplete regression command | Full existing suite merged; new tests added to the same required command. |
+| D15 readiness/typed UI truth | Health repaired; complete application state projection remains. |
+| D16 ignored billing writes | Companion repair returns failures and requires affected profile rows; durable inbox remains. |
+| D17 Stripe arrival-order projection and late deduplication | Open; trial reminders are now audit-only, but general ordering still needs transactional reconciliation. |
+| D18 Stripe environment selection/partition | Incoming signed mode checked in companion repair; deployment selection and database partition remain open. |
+| D19 first email customer fallback | Removed in companion repair; metadata conflicts fail closed for support review. |
+| D20 duplicate checkout/trials | Open; customer-create idempotency alone is not checkout/trial protection. |
+| D21 paid access required to clear credentials | App/API restriction removed in companion repair; disconnect/refresh fencing remains. |
+| D22 implicit platform enablement | Explicit stored-true policy merged in app and Worker. |
+| D23 concurrent X callback reuse | Atomic unexpired DELETE RETURNING implemented in companion repair. |
+| D24 LinkedIn compatibility | Open; verify current Posts API contract before claiming availability. |
+| D25 unsupported learning/marketing claims | Open: reconcile product promises with implemented evidence. |
+| Additional: Stripe period fields | Companion repair reads current subscription item periods, with legacy-event compatibility. |
+
+## Evidence and limitations
+
+Local worker validation: `npm run ci` passes, including the new transport and
+real scheduler failure tests. App validation and the exact review links are
+recorded in the companion repository's September 12 checkpoint.
+
+Existing production data was not read again during the source repairs. The
+148-row/11-hold inventory is the September 9 attestation, not a new snapshot.
+The 16 real-Postgres scenarios are prior rehearsal evidence, not new tests run
+in this continuation. The new tests intercept external transport and make no
+paid or public calls.
+
+There is no defensible unconditional guarantee that the entire service will
+work after these patches. Completion means measured success for the declared
+platforms and failure/recovery contracts above, followed by monitored operation.

docs/implementation-checkpoint-20260913.md

@@ -0,0 +1,49 @@
+# Publication concurrency repair checkpoint — 13 September 2026
+
+The full engineering blueprint remains the completion scope. This change closes
+a newly reproduced database concurrency defect; it does not complete the
+remaining billing, job ownership, connection lifecycle, generation, fairness,
+provider or account-acceptance work in [the full register](implementation-checkpoint-20260912.md).
+
+## Reproduction and repair
+
+App PR #9 merged as c46dff045e79b537661c48b92b5f0b32c7be6b4b. Its post-merge
+integration failed when eight concurrent publishers produced no provider write.
+That run did not retain SQLSTATE diagnostics, so its exact historical cause
+cannot be recovered from the assertion alone.
+
+A controlled test against the unchanged schema reproduced a begin/claim
+deadlock with PostgreSQL SQLSTATE 40P01:
+https://github.com/AyobamiH/oneclickpostfactory/actions/runs/34741743158
+
+Claims lock queue then intent. Other publication mutations locked intent or
+attempt first, then needed the queue row, including the attempt foreign-key
+check. The companion forward migration takes the tenant-owned queue lock first
+in every publication mutation. This serialises one publication without blocking
+unrelated queue identities.
+
+The worker requires publication-queue-lock-order-v1 before dispatch. The v1
+contract migration identity remains compatible with the previous worker, while
+the database advertises lock_order_migration=20260913061000 separately.
+
+## Paired review and verification
+
+- Canonical worker PR: https://github.com/OneClickPostFactory/social-agents/pull/11
+- Schema/app follow-up: https://github.com/AyobamiH/oneclickpostfactory/pull/10
+- The app workflow pins this worker candidate and checks out the canonical
+  OneClickPostFactory/social-agents repository.
+- Controlled begin/claim and release/claim contention, twenty rounds of eight
+  Worker claimers, and the existing publication failure scenarios run against
+  real isolated Supabase/Postgres with provider transport intercepted.
+- A missing lock-order capability must block the worker before provider dispatch.
+- Passing receipts belong to the exact commits recorded in the paired PR checks.
+  Do not infer a pass from the existence of this checkpoint.
+
+Promote schema first, then the paired worker, through the existing guarded
+release process. The old deployed worker remains compatible with the additive
+schema. Do not re-enable generation or provider dispatch as part of source
+integration. No live-provider acceptance or production migration is claimed.
+
+Next implementation slice: durable logical agent-job identity and fenced
+terminal ownership (D05 and the remaining D09), with real database competing
+claim/recovery tests before consumer promotion.

docs/reliability-repair-progress.md

@@ -1,5 +1,9 @@
 # Reliability repair progress
 
+Current continuation: [12 September implementation checkpoint](implementation-checkpoint-20260912.md).
+It reconciles the superseded September 6 branch, current repairs and remaining
+release gates. The September 9 deployment below remains an inert canary.
+
 This file records implementation and rollout status. Exact-SHA CI receipts remain on the relevant pull requests and deployment runs. A passing component test is not live account evidence.
 
 ## Sequence 1: release identity and complete CI gate

package.json

@@ -4,9 +4,9 @@
   "description": "Tenant-scoped source records -> OpenAI -> LinkedIn + Threads + X + Instagram autoposter with durable Cloudinary image persistence.",
   "main": "dist/src/agent.js",
   "scripts": {
-    "build": "tsx scripts/build.ts",
+    "build": "node --import tsx scripts/build.ts",
     "typecheck": "tsc --noEmit --project tsconfig.json",
-    "test": "npm run build && node dist/test/security-hardening.test.js && node dist/test/source-ssrf.test.js && node dist/test/browser-collector-ingest.test.js && node dist/test/slot-scheduler.test.js && node dist/test/daily-inventory-planner.test.js && node dist/test/refresh-queue-finalization.test.js && node dist/test/publish-summary.test.js && node dist/test/threads-refresh.test.js && node dist/test/linkedin-refresh.test.js && node dist/test/instagram-image-timeout.test.js && node dist/test/cost-control.test.js && node dist/test/recovery-scheduler.test.js && node dist/test/social-connector.test.js && node dist/test/meta-publication-boundary.test.js && node dist/test/cloudflare-health.test.js && node dist/test/canary-policy.test.js && node dist/test/exclusive-run-gate.test.js && node dist/test/runtime-scope.test.js && node dist/test/tenant-platform-policy.test.js && node dist/test/supabase-client-retry.test.js && node dist/test/worker-claims.test.js && node dist/test/publication-ledger.test.js && node dist/test/publication-outcome.test.js && node dist/test/publication-executor.test.js && node dist/test/provider-single-dispatch.test.js && node dist/test/publication-receipts.test.js && node dist/test/legacy-revision-hold.test.js",
+    "test": "npm run build && node dist/test/security-hardening.test.js && node dist/test/source-ssrf.test.js && node dist/test/browser-collector-ingest.test.js && node dist/test/slot-scheduler.test.js && node dist/test/daily-inventory-planner.test.js && node dist/test/refresh-queue-finalization.test.js && node dist/test/publish-summary.test.js && node dist/test/threads-refresh.test.js && node dist/test/linkedin-refresh.test.js && node dist/test/instagram-image-timeout.test.js && node dist/test/cost-control.test.js && node dist/test/recovery-scheduler.test.js && node dist/test/social-connector.test.js && node dist/test/meta-publication-boundary.test.js && node dist/test/cloudflare-health.test.js && node dist/test/canary-policy.test.js && node dist/test/exclusive-run-gate.test.js && node dist/test/runtime-scope.test.js && node dist/test/tenant-platform-policy.test.js && node dist/test/supabase-client-retry.test.js && node dist/test/worker-claims.test.js && node dist/test/publication-ledger.test.js && node dist/test/publication-outcome.test.js && node dist/test/publication-executor.test.js && node dist/test/provider-single-dispatch.test.js && node dist/test/publication-receipts.test.js && node dist/test/legacy-revision-hold.test.js && node dist/test/http-cancellation.test.js && node dist/test/scheduler-isolation.test.js",
     "smoke:dist": "node dist/src/cli.js status",
     "ci": "npm run typecheck && npm test && npm run smoke:dist",
     "dev": "tsx src/agent.ts",

src/ai.ts

@@ -543,8 +543,10 @@ async function chatComplete(
   userPrompt: string,
   maxTokens = 500,
   temperature = 0.8,
-  usageContext?: OpenAIUsageContext
+  usageContext?: OpenAIUsageContext,
+  signal?: AbortSignal
 ): Promise<string> {
+  signal?.throwIfAborted();
   const model = config.OPENAI_MODEL || 'gpt-4o';
   const startedAt = Date.now();
   const inputSizeEstimate = systemPrompt.length + userPrompt.length;
@@ -566,6 +568,7 @@ async function chatComplete(
   });
 
   try {
+    signal?.throwIfAborted();
     const { data } = await requestJson<ChatCompletionResponse>('https://api.openai.com/v1/chat/completions', {
       method: 'POST',
       headers: {
@@ -574,6 +577,7 @@ async function chatComplete(
       },
       body,
       timeoutMs: config.HTTP_TIMEOUT_MS,
+      signal,
     });
     if (data.error) {
       throw new Error('OpenAI: ' + (data.error.message || 'Unknown error'));
@@ -763,9 +767,10 @@ function chatCompleteJson<T>(
   userPrompt: string,
   maxTokens = 500,
   temperature = 0.6,
-  usageContext?: OpenAIUsageContext
+  usageContext?: OpenAIUsageContext,
+  signal?: AbortSignal
 ): Promise<T> {
-  return chatComplete(systemPrompt, userPrompt, maxTokens, temperature, usageContext)
+  return chatComplete(systemPrompt, userPrompt, maxTokens, temperature, usageContext, signal)
     .then(raw => extractJson<T>(raw));
 }
 
@@ -1672,6 +1677,7 @@ export async function extractSourceBank(
     contentStrategyProfile?: unknown;
     contentStrategyProfileVersion?: string | null;
     usageContext?: OpenAIUsageContext;
+    signal?: AbortSignal;
   } = {}
 ): Promise<SourceExtraction> {
   const source = [post.title, post.selftext].filter(Boolean).join('\n\n').substring(0, 2400);
@@ -1728,7 +1734,8 @@ ${source}
       ...options.usageContext,
       promptVersion: options.usageContext?.promptVersion || TEXT_PROMPT_VERSION,
       stage: OPENAI_TEXT_ANGLE_EXTRACTION_STAGE,
-    }
+    },
+    options.signal
   );
 
   return normalizeSourceExtraction(parsed, post);