diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md index d28ee9a4a..e7e48f2ad 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md @@ -23,8 +23,8 @@ they cross multiple L1 boundaries. | Chunk | Goal | Risk | Entry gate/status | |---|---|---:|---| | `WS-ART-001-PLAN3` | Reconcile the complete remaining v0.1 custody chain and AUTH/REV/CON handoffs. | L1 | Planning only; proposed | -| `WS-ART-001-03C` | Clean-cut legacy guide identity/excerpts and make the verified same-generation pipeline live. | L1 | Proposed; AUTH-04B entry gate satisfied by PR #245 | -| `WS-ART-001-04A1` | Remove legacy multi-step contributor intake reachability and schema without adding the replacement route. | L1 | Proposed after 03C | +| `WS-ART-001-03C` | Clean-cut legacy guide identity/excerpts and make the verified same-generation pipeline live. | L1 | Merged PR #249 | +| `WS-ART-001-04A1` | Remove legacy multi-step contributor intake reachability and schema without adding the replacement route. | L1 | Implemented; internal review passed; external PR gates pending | | `WS-ART-001-04A2` | Add bounded one-outer-ZIP intake and archive-safety inspection in private scratch. | L1 | Proposed after 04A1 | | `WS-ART-001-04A3` | Add canonical semantic manifest, executable normalization, and unchanged-work gate. | L1 | Proposed after 04A2 | | `WS-ART-001-04B` | Run non-bypassable platform and locked-guide prechecks against that exact scratch tree and persist bounded evidence. | L1 | Proposed after 04A3 | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md index 82ea3fef9..97382d3e7 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md @@ -37,8 +37,8 @@ capability, browser-to-provider path, or client provider credential. ## D6 - Verification Before Binding -Provider acknowledgement sets the upload item to -`stored_pending_verification` and creates a pending replica, never a binding. +Provider acknowledgement sets the durable put attempt to `object_confirmed` +and creates a pending replica, never a binding. Celery independently reads and hashes the complete object. Only a matching object becomes bindable. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md index f081d32f6..5a5481c50 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md @@ -184,3 +184,62 @@ binding, checker materialization/output custody, reviewer packet byte access, and accepted-contribution identity projection. REV owns review lifecycle and notes/findings; CON owns ContributionRecord; client delivery remains a future owner and is not silently implemented by ART v0.1. + +## 2026-08-04 ART-04A1 Legacy Contributor Intake Discovery + +Observations on merged `main` at `2feaf47d`: + +- No HTTP route currently exposes upload-session or upload-item creation, but + `ArtifactUploadSession` and `ArtifactUploadItem` remain active SQLAlchemy + metadata in `app/modules/artifacts/models.py` and `app/db/models.py`. +- `ContributorArtifactAdmissionRequest` in + `app/modules/artifacts/schemas.py` remains an internal command that accepts + caller authorization plus an upload-item id. `ArtifactAdmissionService` + still dispatches that type through `_contributor_facts`, so the obsolete + contributor intake remains reachable to internal callers even without an + HTTP route. +- `ArtifactRepository` still exposes upload-item/session locks, contributor + relationship lookup, and upload-item receipt lookup. These methods are used + only by the retired contributor path and compatibility state projection. +- Shared put recovery and verification still conditionally mutate an upload + item when `ArtifactPutAttempt.upload_item_id` is present. Removing the two + ledgers therefore also requires removing those compatibility mutations while + preserving guide and checker-output recovery. +- `ArtifactPutAttempt.upload_item_id` and + `ArtifactOperationReceipt.upload_item_id` still foreign-key the legacy item + table. Receipt contract version 1 identifies historical acknowledgements only + through `upload_item_id`; version 2 may also carry it for contributor puts. + Historical audit values cannot remain readable if the column is dropped. +- Existing migration tests already seed populated legacy sessions/items, + attempts, and receipts and exercise database invariants. ART-04A1 needs a new + head migration that refuses unsafe populated cutover instead of fabricating a + replacement identity or silently deleting evidence. +- AUTH has already removed the obsolete action identifiers from the active + catalogue and service matrix; `tests/test_authorization.py` retains the + deterministic historical-only proof. ART must not edit AUTH availability or + create aliases. + +Implementation constraints derived from the current code: + +- Remove the contributor admission request variant, service dispatch, repository + relationship methods, upload-ledger models, and conditional upload-item state + projection from shared recovery/verification. +- Do not detach or preserve nullable `upload_item_id` compatibility columns. + Refuse any populated historical reference before dropping the columns and + ledgers, leaving that deployment unchanged for a separate maintenance decision. +- The migration must fail closed when rows exist whose deletion would discard + non-represented contributor state. Upgrade/downgrade behavior and the exact + safe-empty condition require plan-review approval before implementation. +- No replacement route, ZIP parser, scratch orchestration, provider write, AUTH + action activation, Submission, checker, or review behavior belongs to 04A1. + +Plan-review resolution: + +- 04A1 is a complete safe-empty clean cut. It takes exclusive locks and refuses + atomically when any session/item row, contributor put attempt, contract-v1 + receipt, or non-null upload-item reference exists. Refusal preserves the old + schema and all historical identifiers for a separately approved maintenance + decision. A successful upgrade therefore removes the contributor columns and + ledger tables completely; it does not retain detached compatibility fields. +- Downgrade recreates only the exact empty legacy schema proven by the upgrade + precondition. It never fabricates a session/item lineage from newer facts. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index b272c88b4..2934fb81c 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -204,8 +204,7 @@ durable acknowledgement-unknown attempt rather than committing a terminal fact. 7. Transaction B records provider acknowledgement, completes the provisional admission charges, sets the `ArtifactPutAttempt` to `object_confirmed`, and creates the replica with pending verification and unknown - availability/integrity; while a legacy contributor upload item exists, it - alone moves to `stored_pending_verification`. No binding exists. + availability/integrity. No binding exists. 8. A durable verification job is committed in PostgreSQL and published to Celery after commit. A periodic scanner republishes pending work within the configured SLA. @@ -218,8 +217,7 @@ charges provisional. A PostgreSQL scanner publishes ambiguous and expired in-flight attempts; a fixed service principal runs read-only `observe_put_result` plus a complete hash. Matching bytes complete Transaction B once, authoritative absence releases charges and moves the put attempt to -`absent_replay_required`; while a legacy contributor upload item exists, it -alone moves to `replay_required`. Mismatched bytes quarantine the key. No background +`absent_replay_required`. Mismatched bytes quarantine the key. No background resolver repeats a provider write. Exact replay after absence must atomically reacquire capacity before another provider call. Workstream never stores upload bytes in Postgres, Redis, or Celery payloads. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A1-legacy-contributor-intake-removal.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A1-legacy-contributor-intake-removal.md index e5cb3dca6..2866c24da 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A1-legacy-contributor-intake-removal.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A1-legacy-contributor-intake-removal.md @@ -1,6 +1,6 @@ # Chunk Contract: WS-ART-001-04A1 — Legacy Contributor Intake Removal -Parent initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03C +Parent initiative: `WS-ART-001` | Risk: L1 | Status: Implemented; external PR gates pending ## Goal @@ -9,24 +9,107 @@ unused schema before building the one-ZIP replacement. ## Allowed Files -ART upload models/migration/repository/routes/schemas, static architecture and -migration tests, stale-contract/docs, and scoped CI evidence. +- `backend/app/modules/artifacts/models.py` +- `backend/app/db/models.py` +- `backend/app/modules/artifacts/schemas.py` +- `backend/app/modules/artifacts/service.py` +- `backend/app/modules/artifacts/repository.py` +- one new linear `0051` Alembic revision after `0050_guide_source_v2` +- `backend/tests/test_alembic.py` +- `backend/tests/test_artifact_architecture.py` +- contributor-only removals plus guide/checker regression coverage in + `backend/tests/test_artifact_admission.py`, + `backend/tests/test_artifact_recovery.py`, and + `backend/tests/test_artifact_verification.py` +- `backend/tests/conftest.py` +- contributor-only cleanup in `backend/tests/test_artifact_authorization.py` +- stale-contract/spec/glossary text and this chunk's loop evidence + +No runtime route file is allowed because no upload-session/item HTTP route is +currently composed and 04A1 adds no replacement. ## Not Allowed Changes Replacement upload route, ZIP parsing, provider I/O, Submission/checker/review, AUTH catalogue/availability, compatibility aliases, or fabricated backfill. +Old Alembic revisions are immutable. No detached historical columns, legacy +tables, compatibility models, or write aliases remain after the safe-empty +cutover. + +## Locked Migration Policy + +The new migration obtains exclusive locks on the legacy ledgers, put attempts, +and operation receipts before checking or changing schema. Upgrade refuses in +the same transaction if any of these facts exist: + +- any `artifact_upload_sessions` row; +- any `artifact_upload_items` row; +- any `artifact_put_attempts.producer_request_type = 'contributor'` row; +- any non-null `artifact_put_attempts.upload_item_id`; +- any `artifact_operation_receipts.contract_version = 1` row; or +- any non-null `artifact_operation_receipts.upload_item_id`. + +Refusal preserves the prior revision, schema, rows, foreign keys, and readable +historical identifiers without deletion or fabricated translation. Such a +deployment requires a separately approved maintenance/audit migration; 04A1 +does not migrate populated legacy intake. + +On a safe-empty deployment, upgrade removes contributor branches from put and +receipt constraints, removes their upload-item foreign keys/indexes/columns, +then drops `artifact_upload_items` and `artifact_upload_sessions`. Runtime code +simultaneously loses every contributor request, dispatch, lookup, and state +projection path. + +Downgrade recreates the exact empty legacy columns, constraints, indexes, and +tables because a successful upgrade proved that no legacy facts were deleted. +It refuses atomically if the post-cutover database contains a contributor +producer fact that cannot be represented truthfully. No downgrade invents +sessions, items, actors, roles, or state. ## Acceptance Criteria -No route, command, service matrix lookup, model, or schema can create/use the -old intake; historical audit values remain readable where required; populated -unsafe downgrade/upgrade refuses honestly; no new intake becomes reachable. +- No route, request command, admission union member, service dispatch, + repository lookup, ORM model/import, SQL constraint, or recovery/verification + mutation can create or use the old intake. +- Architecture/OpenAPI proof shows 04A1 exposes neither the retired surface nor + the later submission-bundle replacement. +- Safe-empty upgrade removes both ledgers and every writable contributor + reference; safe-empty downgrade recreates the exact empty prior schema. +- Every populated legacy condition above refuses atomically and leaves revision, + schema, data, and historical identifier readability unchanged. +- Direct SQL cannot create contributor put attempts or upload-item-backed + receipts after cutover. +- Guide and checker-output admission, put confirmation, missing-object recovery, + integrity mismatch, and verification terminalization continue without any + upload-item mutation branch. +- AUTH's historical-only deletion proof continues to pass without catalogue, + matrix, availability, grant, or alias edits. ## Verification Commands -Focused Alembic/architecture/route tests, Ruff, stale scans, hosted Backend and -Agent Gates, repository 78% and changed subsystem 90% coverage. +Exact minimum: + +```bash +(cd backend && .venv/bin/python -m pytest -q \ + tests/test_artifact_architecture.py tests/test_alembic.py) +(cd backend && .venv/bin/python -m pytest -q \ + tests/test_artifact_admission.py tests/test_artifact_recovery.py \ + tests/test_artifact_verification.py) +(cd backend && .venv/bin/python -m ruff check app tests scripts) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_stale_authorization_docs.py +python3 scripts/check_stale_workstream_wording.py +python3 scripts/check_markdown_links.py +git diff --check +``` + +Alembic tests must prove empty upgrade/downgrade, every populated-upgrade +refusal condition, atomic unchanged state after refusal, exact schema removal +and recreation, and post-cutover direct-SQL denial. Architecture tests must +prove runtime/import/metadata and route/OpenAPI absence. Focused guide/checker +tests must cover the shared recovery/verification outcomes named above. Hosted +Backend and Agent Gates remain authoritative for repository 78% and ART 90% +coverage; no threshold or gate may be weakened. ## Required Reviewers @@ -35,3 +118,7 @@ Architecture, security/auth, QA, product/ops, senior, CI, docs, reuse, test delt ## Human Review Focus And Stop Conditions Prove deletion without opening a replacement or losing historical evidence. +Stop if populated legacy rows require migration, an AUTH/catalogue change seems +necessary, a replacement route/ZIP/scratch/provider/Submission/checker/review +behavior is needed, an old Alembic revision would need editing, or any required +test/coverage gate would need weakening. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-04A1-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-04A1-external-review-response.md new file mode 100644 index 000000000..44f820719 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-04A1-external-review-response.md @@ -0,0 +1,91 @@ +# WS-ART-001-04A1 External Review Response + +## CodeRabbit + +CodeRabbit completed its review on PR #264 without actionable comments. + +CodeRabbit's earlier detailed review still contained one major finding and two +nitpicks, so the absence of a new summary comment was not treated as closure: + +- **Major — resolved:** recovery tests no longer attach or read the unmapped + `recovery_submission_id` attribute on `ArtifactVerificationJob`. The fixture + now derives the real immutable `Submission.id` from the persisted + `CheckerRun` lineage and passes it explicitly in every task-scoped recovery + request, including retries. +- **Receipt nullability — resolved:** migration 0052 now makes + `artifact_operation_receipts.put_attempt_id` non-null after its locked + populated-legacy preflight. The ORM matches that v2-only invariant, the + downgrade restores legacy nullability before recreating the v1 shape, and a + model contract assertion protects the mapping. +- **Test-helper import — documented, no code move:** moving the shared + checker-output helper would also move its large project/guide/task/submission + relationship fixture across domain test modules. That broad fixture + refactor is outside this removal chunk and would increase this PR's coupling + and review surface. The import remains test-only and has no runtime effect. +- **Docstring heuristic — no change:** the repository's hosted docstring gate + passed. No repository standard was weakened and no unrelated docstrings were + added solely for a standalone reviewer heuristic. + +Local correction evidence: + +- Ruff on all changed Python files: passed; +- `git diff --check`: passed; +- no `recovery_submission_id` references remain under `backend/tests`; +- focused PostgreSQL rerun was interrupted by the known local Python exit 139 + before pytest produced a result; hosted sharded Backend and Agent Gates are + the authoritative execution evidence for this correction. + +The first hosted correction run then exposed the expected schema-custody delta: +making `put_attempt_id` non-null changed the canonical public-schema +fingerprint. The database migrations completed successfully, but fixture reset +failed closed because `EXPECTED_PUBLIC_SCHEMA_SHA256` still named the prior +nullable schema. The constant now records the hosted schema digest +`8acef1c1d96ced0a4d4723ce71aa2e675ab841ec4305d9421ed0584313b98b55`; +no reset guard was removed or weakened. After REV-03A1 merged its own migration +as revision 0051, ART was rebased as revision 0052 with +`0051_review_queue_foundation` as its parent; REV's schema and tests were +preserved. A fresh merged-head migration produced the combined canonical schema +digest `f30127e0acd66d6e2e0cc4d56741ddc8b7cd09175d8c3251749624a0fc353aee`, +which replaces both branch-local fingerprints. + +The first reconciled hosted run passed project lifecycle, shared foundations, +and schema contracts B, then found two REV downgrade assertions that still +named REV's former branch-local head. Both downgrade refusals behaved correctly +and atomically retained the merged current head. Their expected revision is now +`0052_legacy_intake_removal`; no REV downgrade guard, model, or product behavior +changed. + +## Hosted CI correction + +The first Backend sharded run failed one `shared_foundations` test. Replacing +the deleted contributor fixture with a current checker-output fixture made the +recovery resource submission-scoped, but the operator HTTP test requests still +omitted the canonical `submission_id`. Production correctly failed closed with +`409 artifact recovery resource facts changed`. + +The test now carries the exact submission lineage for denied, stale, successful, +replayed, altered, ineligible, and cross-project recovery requests. No production +authorization or recovery guard was weakened. + +Focused correction evidence: + +- the formerly failing operator HTTP test: `1 passed`; +- complete operator API and recovery files: `15 passed`; +- Ruff on the corrected test file: passed. + +The correction is pushed for a fresh hosted Backend and Agent Gates run. + +The fresh run proved `shared_foundations`, `project_lifecycle`, +`task_lifecycle`, and `schema_contracts_a`, then exposed one stale assertion in +`schema_contracts_b`: the broad current-schema contract still required the two +tables this chunk intentionally removes. The assertion now classifies +`artifact_upload_sessions.id` and `artifact_upload_items.id` as discarded +columns. The exact formerly failing schema test passes locally. + +All five shards then passed, while the aggregate ART subsystem coverage gate +reported `89.52%`. The deleted contributor tests had also carried ambiguous-put +terminal coverage. That proof is now restored on the current task-scoped +checker-output producer for mismatch, provider conflict, and collision with an +already verified replica. All three cases pass. Combining those exact tests +with the authenticated hosted shard coverage reaches `90.01%`; the hosted gate +must confirm the final commit. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-04A1-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-04A1-internal-review-evidence.md new file mode 100644 index 000000000..ff51a82fd --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-04A1-internal-review-evidence.md @@ -0,0 +1,65 @@ +# WS-ART-001-04A1 Internal Review Evidence + +## Evidence gate + +Result: PASS. + +- Scope stayed within the removal-only chunk. No route, ZIP parser, submission, + checker-policy, review, provider, or AUTH activation was added. +- Ruff passed across `backend/app`, `backend/tests`, and `backend/scripts`. +- PostgreSQL proof passed for the exact empty upgrade/downgrade and all six + populated refusal predicates. The final enriched refusal snapshot also + preserves physical schema, foreign keys, indexes, row identifiers, and row + values. +- The focused surviving ART paths passed: 59 admission/recovery/verification + tests before review corrections, followed by 64 migration, architecture, + authorization, checker lifecycle, and task-scoped recovery tests after the + corrections. +- After hosted CI exposed missing submission lineage in the replacement + checker-output operator fixture, the corrected operator/recovery pair passed + all 15 tests. +- After the schema-contract shard exposed a stale broad-schema expectation, the + exact current-schema test passed with the retired intake tables asserted + absent instead of present. +- Current checker-output mismatch, provider-conflict, and verified-replica + collision proofs pass. Against the authenticated hosted coverage baseline, + they restore the ART subsystem projection from 89.52% to 90.01% without + changing the required threshold. +- Stale artifact/auth/wording scans, Markdown links, and `git diff --check` + passed. No dependency, workflow, coverage-threshold, or package-script change + was made. + +## Reviewer results + +- Architecture: PASS after exact downgrade-schema correction. +- Security/auth: PASS. +- QA: PASS after checker-output, OpenAPI, SQL, and refusal-state proofs. +- Product/operations: PASS after operator migration guidance. +- Senior engineering: PASS after stale fixture and downgrade correction. +- CI integrity: PASS with no workflow, threshold, script, skip, or dependency + weakening. +- Documentation: PASS. +- Reuse/dedup: PASS with one non-blocking test-helper locality risk. +- Test delta: PASS after restoring task-scoped checker recovery and after-I/O + stale-race coverage. + +## Findings resolved + +- Recreated legacy timestamps are non-null with their predecessor defaults. +- Downgrade comparison now covers the relevant physical schema, not only table + names. +- The stale `_AdmissionFacts.upload_item_id` fixture was removed. +- Checker-output put, verified, missing, integrity-mismatch, and recovery paths + replace generic coverage formerly carried by contributor fixtures. +- Direct SQL cannot relabel a put attempt as contributor or downgrade a receipt + to contract version 1. +- Canonical decisions and operator guidance no longer describe upload items as + current runtime state. +- Every operator recovery request backed by the replacement checker-output + fixture now carries its canonical submission identity. + +## Residual risk + +The recovery test imports one private checker-admission helper from another ART +test module. Review found no existing shared helper and treated this as low, +non-blocking locality debt. Move it only if another consumer appears. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-04A1-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-04A1-pr-trust-bundle.md new file mode 100644 index 000000000..73d10c359 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-04A1-pr-trust-bundle.md @@ -0,0 +1,84 @@ +# WS-ART-001-04A1 PR Trust Bundle + +## Chunk + +`WS-ART-001-04A1 — Legacy Contributor Intake Removal` (L1) + +## Goal and approved intent + +Remove the inactive multi-step contributor upload-session/item path before the +one-outer-ZIP submission pipeline is built. Preserve historical evidence by +refusing populated deployments; do not translate, detach, delete, or fabricate +legacy lineage. + +## What changed and why + +- Removed legacy upload session/item ORM records, contributor admission request + and dispatch, repository lookups, and shared mutation branches. +- Narrowed put-attempt and receipt constraints to current guide/checker + producers only. +- Added migration `0052_legacy_intake_removal`: exclusive-lock preflight, + atomic populated refusal, safe-empty removal, and exact empty downgrade. +- Replaced useful generic contributor-fixture coverage with guide and + task-scoped checker-output proofs. +- Updated current specification, glossary, decisions, and operator deployment + guidance. + +## Design and rejected alternatives + +Chosen: a complete safe-empty clean cut. Rejected: detached compatibility +columns, inferred backfill, automatic deletion, dual runtime paths, and adding +the replacement submission route in this chunk. + +## Scope and product behavior + +No public contributor intake is added. Existing guide and checker storage, +verification, and recovery behavior remains available. A deployment with any +legacy intake evidence remains on `0050_guide_source_v2` and requires separately +approved maintenance work. + +## Acceptance proof + +- Empty `0050 -> 0051 -> 0050 -> 0051` round trip passed with exact physical + schema comparison. +- All six populated preflight predicates refused atomically. +- Direct contributor producer and v1 receipt mutations were rejected. +- OpenAPI contains neither retired routes nor the future submission-bundle + surface. +- Guide/checker put, missing, mismatch, verification, stale-race, retry, + concurrency, and recovery lineage tests passed. + +## CI and test integrity + +No CI, threshold, dependency, or package-script changes. Ruff, stale scans, +Markdown links, and diff checks pass. Contributor-specific tests were removed; +their surviving generic guarantees were re-established on current producers. + +## Reviewer results and external review + +Architecture, security, QA, product/operations, senior engineering, +documentation, reuse, and test-delta reviews pass. CodeRabbit completed without +actionable comments. The first hosted Backend run exposed one stale test request +that omitted the replacement checker's canonical submission lineage; the test +was corrected without changing production guards, and the complete operator and +recovery pair passes 15 tests. A subsequent schema-contract shard exposed one +remaining broad assertion that required the intentionally retired tables; it now +asserts those table columns are discarded, and the exact schema test passes. +All shards subsequently passed, but aggregate ART coverage was 89.52%. Current +checker-output observation tests now replace the removed contributor coverage +for mismatch, provider conflict, and verified-replica collision; all pass and +project 90.01% against the authenticated hosted coverage baseline. Fresh hosted +Backend/Agent Gates are required on the final correction commit. + +## Remaining risks and follow-up + +- Populated deployments intentionally cannot apply 0051 without separately + approved maintenance/audit work. +- The replacement one-ZIP preparation begins only in the next approved ART + chunk; 04A1 creates no submission behavior. + +## Human review focus and merge ownership + +Confirm the preflight predicates, exact downgrade shape, absence of a +replacement route, and the replacement checker/guide coverage. Human approval +is required for merge; the agent will not merge this PR. diff --git a/backend/alembic/versions/0052_legacy_intake_removal.py b/backend/alembic/versions/0052_legacy_intake_removal.py new file mode 100644 index 000000000..afee9b6b5 --- /dev/null +++ b/backend/alembic/versions/0052_legacy_intake_removal.py @@ -0,0 +1,343 @@ +"""remove the inactive multi-step contributor artifact intake + +Revision ID: 0052_legacy_intake_removal +Revises: 0051_review_queue_foundation +Create Date: 2026-08-04 +""" + +from __future__ import annotations + +from alembic import op +import sqlalchemy as sa + + +revision = "0052_legacy_intake_removal" +down_revision = "0051_review_queue_foundation" +branch_labels = depends_on = None + + +_PUT_PRODUCER_REFERENCE = ( + "(producer_request_type = 'guide' and guide_source_item_id is not null " + "and checker_run_id is null and task_id is null and logical_role is null) or " + "(producer_request_type = 'checker_output' and guide_source_item_id is null " + "and checker_run_id is not null and task_id is not null " + "and octet_length(logical_role) between 1 and 100)" +) +_PUT_PRODUCER_IDENTITY = ( + "((producer_request_type = 'guide' and producer_type = 'actor_profile' and " + "producer_ref ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-" + "[89ab][0-9a-f]{3}-[0-9a-f]{12}$') or " + "(producer_request_type = 'checker_output' and producer_type = 'service_identity' " + "and producer_ref = 'workstream.artifact.checker_output'))" +) +_LEGACY_PUT_PRODUCER_IDENTITY = ( + "((producer_request_type in ('guide', 'contributor') and " + "producer_type = 'actor_profile' and producer_ref ~ " + "'^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-" + "[89ab][0-9a-f]{3}-[0-9a-f]{12}$') or " + "(producer_request_type = 'checker_output' and producer_type = 'service_identity' " + "and producer_ref = 'workstream.artifact.checker_output'))" +) +_LEGACY_PUT_PRODUCER_REFERENCE = ( + "(producer_request_type = 'guide' and guide_source_item_id is not null " + "and upload_item_id is null and checker_run_id is null and task_id is null " + "and logical_role is null) or " + "(producer_request_type = 'contributor' and guide_source_item_id is null " + "and upload_item_id is not null and checker_run_id is null and task_id is not null " + "and logical_role is null) or " + "(producer_request_type = 'checker_output' and guide_source_item_id is null " + "and upload_item_id is null and checker_run_id is not null and task_id is not null " + "and octet_length(logical_role) between 1 and 100)" +) +_RECEIPT_PRODUCER_REFERENCE = ( + "contract_version = 2 and put_attempt_id is not null and " + "((guide_source_item_id is not null)::int + " + "(checker_run_id is not null)::int) = 1" +) +_LEGACY_RECEIPT_PRODUCER_REFERENCE = ( + "(contract_version = 1 and put_attempt_id is null and upload_item_id is not null " + "and guide_source_item_id is null and checker_run_id is null) or " + "(contract_version = 2 and put_attempt_id is not null and " + "((upload_item_id is not null)::int + (guide_source_item_id is not null)::int + " + "(checker_run_id is not null)::int) = 1)" +) + + +def _lock_and_refuse_populated_legacy_intake() -> None: + bind = op.get_bind() + bind.execute( + sa.text( + "lock table artifact_upload_sessions, artifact_upload_items, " + "artifact_put_attempts, artifact_operation_receipts in access exclusive mode" + ) + ) + populated = bind.execute( + sa.text( + "select exists(select 1 from artifact_upload_sessions) " + "or exists(select 1 from artifact_upload_items) " + "or exists(select 1 from artifact_put_attempts " + "where producer_request_type = 'contributor' or upload_item_id is not null) " + "or exists(select 1 from artifact_operation_receipts " + "where contract_version = 1 or upload_item_id is not null)" + ) + ).scalar_one() + if populated: + raise RuntimeError( + "legacy contributor artifact intake is populated; preserve the existing " + "schema and use a separately approved maintenance migration" + ) + + +def upgrade() -> None: + """Remove only a proven-empty legacy contributor intake.""" + _lock_and_refuse_populated_legacy_intake() + + op.drop_constraint("producer_reference", "artifact_put_attempts", type_="check") + op.drop_constraint("producer_identity", "artifact_put_attempts", type_="check") + op.drop_constraint("producer_request_type", "artifact_put_attempts", type_="check") + op.drop_constraint("contract_producer_reference", "artifact_operation_receipts", type_="check") + op.drop_constraint( + "fk_artifact_put_attempts_upload_item_id_artifact_upload_items", + "artifact_put_attempts", + type_="foreignkey", + ) + op.drop_constraint( + "fk_artifact_operation_receipts_upload_item_id_artifact__cc40", + "artifact_operation_receipts", + type_="foreignkey", + ) + op.drop_index("ix_artifact_put_attempts_upload_item_id", table_name="artifact_put_attempts") + op.drop_index( + "ix_artifact_operation_receipts_upload_item_id", + table_name="artifact_operation_receipts", + ) + op.drop_column("artifact_put_attempts", "upload_item_id") + op.drop_column("artifact_operation_receipts", "upload_item_id") + op.alter_column( + "artifact_operation_receipts", + "put_attempt_id", + existing_type=sa.String(length=36), + nullable=False, + ) + op.create_check_constraint( + "producer_request_type", + "artifact_put_attempts", + "producer_request_type in ('guide', 'checker_output')", + ) + op.create_check_constraint("producer_identity", "artifact_put_attempts", _PUT_PRODUCER_IDENTITY) + op.create_check_constraint( + "producer_reference", "artifact_put_attempts", _PUT_PRODUCER_REFERENCE + ) + op.create_check_constraint( + "contract_producer_reference", + "artifact_operation_receipts", + _RECEIPT_PRODUCER_REFERENCE, + ) + op.drop_table("artifact_upload_items") + op.drop_table("artifact_upload_sessions") + + +def downgrade() -> None: + """Recreate the exact empty legacy schema without fabricating lineage.""" + bind = op.get_bind() + bind.execute( + sa.text( + "lock table artifact_put_attempts, artifact_operation_receipts in access exclusive mode" + ) + ) + if bind.execute( + sa.text( + "select exists(select 1 from artifact_put_attempts " + "where producer_request_type = 'contributor') " + "or exists(select 1 from artifact_operation_receipts where contract_version = 1)" + ) + ).scalar_one(): + raise RuntimeError("cannot truthfully recreate legacy contributor intake") + + op.create_table( + "artifact_upload_sessions", + sa.Column("id", sa.String(36), primary_key=True), + sa.Column("actor_id", sa.String(100), nullable=False), + sa.Column("project_id", sa.String(36), nullable=False), + sa.Column("task_id", sa.String(36), nullable=True), + sa.Column("guide_id", sa.String(36), nullable=True), + sa.Column("permitted_roles", sa.JSON(), nullable=False), + sa.Column("state", sa.String(30), nullable=False), + sa.Column("maximum_bytes", sa.Integer(), nullable=False), + sa.Column("current_bytes", sa.Integer(), nullable=False), + sa.Column("reserved_bytes", sa.Integer(), nullable=False), + sa.Column("maximum_items", sa.Integer(), nullable=False), + sa.Column("current_items", sa.Integer(), nullable=False), + sa.Column("reserved_items", sa.Integer(), nullable=False), + sa.Column("artifact_set_hash", sa.String(71), nullable=True), + sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("consumed_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("cas_version", sa.Integer(), nullable=False), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.Column( + "updated_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.ForeignKeyConstraint(["project_id"], ["projects.id"], ondelete="RESTRICT"), + sa.CheckConstraint( + "state in ('open', 'sealed', 'consumed', 'expired', 'cancelled')", name="state" + ), + sa.CheckConstraint( + "maximum_bytes >= 0 and current_bytes >= 0 and reserved_bytes >= 0", + name="byte_counts_nonnegative", + ), + sa.CheckConstraint( + "maximum_items >= 0 and current_items >= 0 and reserved_items >= 0", + name="item_counts_nonnegative", + ), + sa.CheckConstraint("current_bytes + reserved_bytes <= maximum_bytes", name="byte_limit"), + sa.CheckConstraint("current_items + reserved_items <= maximum_items", name="item_limit"), + sa.CheckConstraint("cas_version >= 0", name="cas_nonnegative"), + sa.CheckConstraint( + "artifact_set_hash is null or artifact_set_hash ~ '^sha256:[0-9a-f]{64}$'", + name="artifact_set_hash_shape", + ), + sa.CheckConstraint( + "(state = 'consumed') = (consumed_at is not null)", name="consumed_timestamp" + ), + sa.CheckConstraint( + "state not in ('sealed', 'consumed') or artifact_set_hash is not null", + name="sealed_hash_required", + ), + ) + for column in ("actor_id", "project_id", "task_id", "guide_id", "state"): + op.create_index( + f"ix_artifact_upload_sessions_{column}", + "artifact_upload_sessions", + [column], + ) + + op.create_table( + "artifact_upload_items", + sa.Column("id", sa.String(36), primary_key=True), + sa.Column("session_id", sa.String(36), nullable=False), + sa.Column("logical_role", sa.String(100), nullable=False), + sa.Column("display_name", sa.String(500), nullable=False), + sa.Column("media_type", sa.String(200), nullable=True), + sa.Column("reserved_bytes", sa.Integer(), nullable=False), + sa.Column("expected_sha256", sa.String(71), nullable=True), + sa.Column("expected_size", sa.Integer(), nullable=True), + sa.Column("idempotency_key", sa.String(200), nullable=False), + sa.Column("request_digest", sa.String(71), nullable=False), + sa.Column("state", sa.String(30), nullable=False), + sa.Column("cas_version", sa.Integer(), nullable=False), + sa.Column("provider_object_ref", sa.String(1024), nullable=True), + sa.Column("content_id", sa.String(36), nullable=True), + sa.Column("error_code", sa.String(100), nullable=True), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.Column( + "updated_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.ForeignKeyConstraint( + ["session_id"], ["artifact_upload_sessions.id"], ondelete="CASCADE" + ), + sa.ForeignKeyConstraint(["content_id"], ["artifact_contents.id"], ondelete="RESTRICT"), + sa.UniqueConstraint("session_id", "idempotency_key", name="uq_artifact_item_operation"), + sa.CheckConstraint( + "state in ('reserved', 'uploading', 'replay_required', " + "'stored_pending_verification', 'ready', 'failed', 'cancelled')", + name="state", + ), + sa.CheckConstraint( + "reserved_bytes >= 0 and cas_version >= 0 and " + "(expected_size is null or expected_size >= 0)", + name="counts_nonnegative", + ), + sa.CheckConstraint("request_digest ~ '^sha256:[0-9a-f]{64}$'", name="request_digest_shape"), + sa.CheckConstraint( + "expected_sha256 is null or expected_sha256 ~ '^sha256:[0-9a-f]{64}$'", + name="expected_sha256_shape", + ), + sa.CheckConstraint( + "((state in ('stored_pending_verification', 'ready')) and content_id is not null " + "and provider_object_ref is not null) or " + "((state not in ('stored_pending_verification', 'ready')) and content_id is null " + "and provider_object_ref is null)", + name="stored_result_required", + ), + sa.CheckConstraint( + "state != 'failed' or error_code is not null", name="failed_error_required" + ), + ) + for column in ("session_id", "content_id", "state"): + op.create_index(f"ix_artifact_upload_items_{column}", "artifact_upload_items", [column]) + + op.drop_constraint("producer_reference", "artifact_put_attempts", type_="check") + op.drop_constraint("producer_identity", "artifact_put_attempts", type_="check") + op.drop_constraint("producer_request_type", "artifact_put_attempts", type_="check") + op.drop_constraint("contract_producer_reference", "artifact_operation_receipts", type_="check") + op.alter_column( + "artifact_operation_receipts", + "put_attempt_id", + existing_type=sa.String(length=36), + nullable=True, + ) + op.add_column( + "artifact_put_attempts", sa.Column("upload_item_id", sa.String(36), nullable=True) + ) + op.add_column( + "artifact_operation_receipts", + sa.Column("upload_item_id", sa.String(36), nullable=True), + ) + op.create_foreign_key( + "fk_artifact_put_attempts_upload_item_id_artifact_upload_items", + "artifact_put_attempts", + "artifact_upload_items", + ["upload_item_id"], + ["id"], + ondelete="RESTRICT", + ) + op.create_foreign_key( + "fk_artifact_operation_receipts_upload_item_id_artifact__cc40", + "artifact_operation_receipts", + "artifact_upload_items", + ["upload_item_id"], + ["id"], + ondelete="RESTRICT", + ) + op.create_index( + "ix_artifact_put_attempts_upload_item_id", + "artifact_put_attempts", + ["upload_item_id"], + ) + op.create_index( + "ix_artifact_operation_receipts_upload_item_id", + "artifact_operation_receipts", + ["upload_item_id"], + ) + op.create_check_constraint( + "producer_request_type", + "artifact_put_attempts", + "producer_request_type in ('guide', 'contributor', 'checker_output')", + ) + op.create_check_constraint( + "producer_identity", "artifact_put_attempts", _LEGACY_PUT_PRODUCER_IDENTITY + ) + op.create_check_constraint( + "producer_reference", "artifact_put_attempts", _LEGACY_PUT_PRODUCER_REFERENCE + ) + op.create_check_constraint( + "contract_producer_reference", + "artifact_operation_receipts", + _LEGACY_RECEIPT_PRODUCER_REFERENCE, + ) diff --git a/backend/app/db/models.py b/backend/app/db/models.py index a81928f4b..5631cb51d 100644 --- a/backend/app/db/models.py +++ b/backend/app/db/models.py @@ -20,8 +20,6 @@ ArtifactOperationReceipt, ArtifactReplica, ArtifactRecoveryAttempt, - ArtifactUploadItem, - ArtifactUploadSession, ) from app.modules.authorization.models import ( # noqa: F401 AdminRoleGrant, diff --git a/backend/app/modules/artifacts/models.py b/backend/app/modules/artifacts/models.py index b02c19f31..010a0512f 100644 --- a/backend/app/modules/artifacts/models.py +++ b/backend/app/modules/artifacts/models.py @@ -30,130 +30,6 @@ ) -class ArtifactUploadSession(Base): - """Mutable staging authority for one bounded artifact set.""" - - __tablename__ = "artifact_upload_sessions" - __table_args__ = ( - CheckConstraint( - "state in ('open', 'sealed', 'consumed', 'expired', 'cancelled')", - name="state", - ), - CheckConstraint( - "maximum_bytes >= 0 and current_bytes >= 0 and reserved_bytes >= 0", - name="byte_counts_nonnegative", - ), - CheckConstraint( - "maximum_items >= 0 and current_items >= 0 and reserved_items >= 0", - name="item_counts_nonnegative", - ), - CheckConstraint( - "current_bytes + reserved_bytes <= maximum_bytes", - name="byte_limit", - ), - CheckConstraint( - "current_items + reserved_items <= maximum_items", - name="item_limit", - ), - CheckConstraint("cas_version >= 0", name="cas_nonnegative"), - CheckConstraint( - "artifact_set_hash is null or " + SHA256_CHECK.format(column="artifact_set_hash"), - name="artifact_set_hash_shape", - ), - CheckConstraint( - "(state = 'consumed') = (consumed_at is not null)", - name="consumed_timestamp", - ), - CheckConstraint( - "state not in ('sealed', 'consumed') or artifact_set_hash is not null", - name="sealed_hash_required", - ), - ) - - id: Mapped[str] = mapped_column(String(36), primary_key=True) - actor_id: Mapped[str] = mapped_column(String(100), nullable=False, index=True) - project_id: Mapped[str] = mapped_column( - ForeignKey("projects.id", ondelete="RESTRICT"), nullable=False, index=True - ) - task_id: Mapped[str | None] = mapped_column(String(36), index=True) - guide_id: Mapped[str | None] = mapped_column(String(36), index=True) - permitted_roles: Mapped[list[str]] = mapped_column(JSON, nullable=False, default=list) - state: Mapped[str] = mapped_column(String(30), nullable=False, default="open", index=True) - maximum_bytes: Mapped[int] = mapped_column(Integer, nullable=False) - current_bytes: Mapped[int] = mapped_column(Integer, nullable=False, default=0) - reserved_bytes: Mapped[int] = mapped_column(Integer, nullable=False, default=0) - maximum_items: Mapped[int] = mapped_column(Integer, nullable=False) - current_items: Mapped[int] = mapped_column(Integer, nullable=False, default=0) - reserved_items: Mapped[int] = mapped_column(Integer, nullable=False, default=0) - artifact_set_hash: Mapped[str | None] = mapped_column(String(71)) - expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) - consumed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) - cas_version: Mapped[int] = mapped_column(Integer, nullable=False, default=0) - created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) - updated_at: Mapped[datetime] = mapped_column( - DateTime(timezone=True), server_default=func.now(), onupdate=func.now() - ) - - -class ArtifactUploadItem(Base): - """Per-item staging and recovery ledger under an upload session.""" - - __tablename__ = "artifact_upload_items" - __table_args__ = ( - UniqueConstraint("session_id", "idempotency_key", name="uq_artifact_item_operation"), - CheckConstraint( - "state in ('reserved', 'uploading', 'replay_required', " - "'stored_pending_verification', 'ready', 'failed', 'cancelled')", - name="state", - ), - CheckConstraint( - "reserved_bytes >= 0 and cas_version >= 0 and " - "(expected_size is null or expected_size >= 0)", - name="counts_nonnegative", - ), - CheckConstraint(SHA256_CHECK.format(column="request_digest"), name="request_digest_shape"), - CheckConstraint( - "expected_sha256 is null or " + SHA256_CHECK.format(column="expected_sha256"), - name="expected_sha256_shape", - ), - CheckConstraint( - "((state in ('stored_pending_verification', 'ready')) and " - "content_id is not null and provider_object_ref is not null) or " - "((state not in ('stored_pending_verification', 'ready')) and " - "content_id is null and provider_object_ref is null)", - name="stored_result_required", - ), - CheckConstraint( - "state != 'failed' or error_code is not null", - name="failed_error_required", - ), - ) - - id: Mapped[str] = mapped_column(String(36), primary_key=True) - session_id: Mapped[str] = mapped_column( - ForeignKey("artifact_upload_sessions.id", ondelete="CASCADE"), nullable=False, index=True - ) - logical_role: Mapped[str] = mapped_column(String(100), nullable=False) - display_name: Mapped[str] = mapped_column(String(500), nullable=False) - media_type: Mapped[str | None] = mapped_column(String(200)) - reserved_bytes: Mapped[int] = mapped_column(Integer, nullable=False) - expected_sha256: Mapped[str | None] = mapped_column(String(71)) - expected_size: Mapped[int | None] = mapped_column(Integer) - idempotency_key: Mapped[str] = mapped_column(String(200), nullable=False) - request_digest: Mapped[str] = mapped_column(String(71), nullable=False) - state: Mapped[str] = mapped_column(String(30), nullable=False, default="reserved", index=True) - cas_version: Mapped[int] = mapped_column(Integer, nullable=False, default=0) - provider_object_ref: Mapped[str | None] = mapped_column(String(1024)) - content_id: Mapped[str | None] = mapped_column( - ForeignKey("artifact_contents.id", ondelete="RESTRICT"), index=True - ) - error_code: Mapped[str | None] = mapped_column(String(100)) - created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) - updated_at: Mapped[datetime] = mapped_column( - DateTime(timezone=True), server_default=func.now(), onupdate=func.now() - ) - - class ArtifactContent(Base): """Immutable provider-neutral identity for exact stored bytes.""" @@ -759,7 +635,7 @@ class ArtifactPutAttempt(Base): ), UniqueConstraint("operation_identity", name="uq_artifact_put_attempt_operation"), CheckConstraint( - "producer_request_type in ('guide', 'contributor', 'checker_output')", + "producer_request_type in ('guide', 'checker_output')", name="producer_request_type", ), CheckConstraint( @@ -767,7 +643,7 @@ class ArtifactPutAttempt(Base): name="producer_type", ), CheckConstraint( - "((producer_request_type in ('guide', 'contributor') " + "((producer_request_type = 'guide' " "and producer_type = 'actor_profile' and " + UUID_CHECK.format(column="producer_ref") + ") or (producer_request_type = 'checker_output' " @@ -830,13 +706,10 @@ class ArtifactPutAttempt(Base): ), CheckConstraint( "(producer_request_type = 'guide' and guide_source_item_id is not null " - "and upload_item_id is null and checker_run_id is null and task_id is null " - "and logical_role is null) or " - "(producer_request_type = 'contributor' and guide_source_item_id is null " - "and upload_item_id is not null and checker_run_id is null and task_id is not null " + "and checker_run_id is null and task_id is null " "and logical_role is null) or " "(producer_request_type = 'checker_output' and guide_source_item_id is null " - "and upload_item_id is null and checker_run_id is not null and task_id is not null " + "and checker_run_id is not null and task_id is not null " "and octet_length(logical_role) between 1 and 100)", name="producer_reference", ), @@ -855,9 +728,6 @@ class ArtifactPutAttempt(Base): guide_source_item_id: Mapped[str | None] = mapped_column( ForeignKey("guide_source_snapshot_items.id", ondelete="RESTRICT"), index=True ) - upload_item_id: Mapped[str | None] = mapped_column( - ForeignKey("artifact_upload_items.id", ondelete="RESTRICT"), index=True - ) checker_run_id: Mapped[str | None] = mapped_column( ForeignKey("checker_runs.id", ondelete="RESTRICT"), index=True ) @@ -972,24 +842,17 @@ class ArtifactOperationReceipt(Base): CheckConstraint("outcome = 'stored_pending_verification'", name="outcome"), CheckConstraint("attempt_number > 0", name="attempt_positive"), CheckConstraint( - "(contract_version = 1 and put_attempt_id is null and upload_item_id is not null " - "and guide_source_item_id is null and checker_run_id is null) or " - "(contract_version = 2 and put_attempt_id is not null and " - "((upload_item_id is not null)::int + " - "(guide_source_item_id is not null)::int + " - "(checker_run_id is not null)::int) = 1)", + "contract_version = 2 and put_attempt_id is not null and " + "((guide_source_item_id is not null)::int + " + "(checker_run_id is not null)::int) = 1", name="contract_producer_reference", ), ) id: Mapped[str] = mapped_column(String(36), primary_key=True) contract_version: Mapped[int] = mapped_column(Integer, nullable=False, default=2) - put_attempt_id: Mapped[str | None] = mapped_column( - ForeignKey("artifact_put_attempts.id", ondelete="RESTRICT"), index=True - ) - upload_item_id: Mapped[str | None] = mapped_column( - ForeignKey("artifact_upload_items.id", ondelete="RESTRICT"), - index=True, + put_attempt_id: Mapped[str] = mapped_column( + ForeignKey("artifact_put_attempts.id", ondelete="RESTRICT"), nullable=False, index=True ) guide_source_item_id: Mapped[str | None] = mapped_column( ForeignKey("guide_source_snapshot_items.id", ondelete="RESTRICT"), index=True diff --git a/backend/app/modules/artifacts/repository.py b/backend/app/modules/artifacts/repository.py index 368422899..95bdbeb66 100644 --- a/backend/app/modules/artifacts/repository.py +++ b/backend/app/modules/artifacts/repository.py @@ -25,8 +25,6 @@ ArtifactReplica, ArtifactRecoveryAttempt, ArtifactStorageNamespace, - ArtifactUploadItem, - ArtifactUploadSession, ) from app.modules.checkers.models import CheckerRun from app.modules.projects.models import ( @@ -73,21 +71,6 @@ class VerifiedGuideContentCandidate: byte_count: int -@dataclass(frozen=True, slots=True) -class ContributorAdmissionFacts: - """Authoritative upload-item ownership and state.""" - - upload_item_id: str - project_id: str - task_id: str | None - actor_profile_id: str - session_state: str - item_state: str - expected_sha256: str | None - expected_size: int | None - media_type: str | None - - @dataclass(frozen=True, slots=True) class CheckerOutputAdmissionFacts: """Authoritative project/task ownership for one checker run.""" @@ -164,28 +147,12 @@ async def database_now(self) -> datetime: raise RuntimeError("PostgreSQL clock did not return a timestamp") return value - async def lock_upload_item(self, item_id: str) -> ArtifactUploadItem | None: - """Load one upload item with a row lock.""" - result = await self._session.execute( - select(ArtifactUploadItem).where(ArtifactUploadItem.id == item_id).with_for_update() - ) - return result.scalar_one_or_none() - async def lock_checker_run(self, checker_run_id: str) -> CheckerRun | None: """Lock one checker run for canonical recovery resource derivation.""" return await self._session.scalar( select(CheckerRun).where(CheckerRun.id == checker_run_id).with_for_update() ) - async def lock_upload_session(self, session_id: str) -> ArtifactUploadSession | None: - """Load one upload session with a row lock.""" - result = await self._session.execute( - select(ArtifactUploadSession) - .where(ArtifactUploadSession.id == session_id) - .with_for_update() - ) - return result.scalar_one_or_none() - async def get_guide_admission_facts( self, guide_source_item_id: str ) -> GuideAdmissionFacts | None: @@ -316,50 +283,6 @@ async def get_guide_lineage(self, guide_source_item_id: str) -> GuideLineageFact project_id=lineage.project_id, ) - async def get_contributor_admission_facts( - self, upload_item_id: str - ) -> ContributorAdmissionFacts | None: - """Load canonical contributor upload ownership and state.""" - row = ( - await self._session.execute( - select( - ArtifactUploadItem.id, - WorkstreamTask.project_id, - WorkstreamTask.id.label("task_id"), - ArtifactUploadSession.actor_id, - ArtifactUploadSession.state.label("session_state"), - ArtifactUploadItem.state.label("item_state"), - ArtifactUploadItem.expected_sha256, - ArtifactUploadItem.expected_size, - ArtifactUploadItem.media_type, - ) - .join( - ArtifactUploadSession, - ArtifactUploadSession.id == ArtifactUploadItem.session_id, - ) - .join( - WorkstreamTask, - (WorkstreamTask.id == ArtifactUploadSession.task_id) - & (WorkstreamTask.project_id == ArtifactUploadSession.project_id), - ) - .where(ArtifactUploadItem.id == upload_item_id) - .with_for_update(of=(ArtifactUploadSession, ArtifactUploadItem, WorkstreamTask)) - ) - ).one_or_none() - if row is None: - return None - return ContributorAdmissionFacts( - upload_item_id=row.id, - project_id=row.project_id, - task_id=row.task_id, - actor_profile_id=row.actor_id, - session_state=row.session_state, - item_state=row.item_state, - expected_sha256=row.expected_sha256, - expected_size=row.expected_size, - media_type=row.media_type, - ) - async def get_checker_output_admission_facts( self, checker_run_id: str ) -> CheckerOutputAdmissionFacts | None: @@ -811,19 +734,6 @@ async def add_receipt(self, receipt: ArtifactOperationReceipt) -> ArtifactOperat await self._session.flush() return receipt - async def get_receipt_for_item(self, upload_item_id: str) -> ArtifactOperationReceipt | None: - """Load the Workstream put receipt for one upload item.""" - result = await self._session.execute( - select(ArtifactOperationReceipt) - .where(ArtifactOperationReceipt.upload_item_id == upload_item_id) - .order_by( - ArtifactOperationReceipt.created_at.desc(), - ArtifactOperationReceipt.id, - ) - .limit(1) - ) - return result.scalar_one_or_none() - async def claim_storage_namespace( self, namespace: ArtifactStorageNamespace ) -> ArtifactStorageNamespace: diff --git a/backend/app/modules/artifacts/schemas.py b/backend/app/modules/artifacts/schemas.py index aece39ac4..a6cba3172 100644 --- a/backend/app/modules/artifacts/schemas.py +++ b/backend/app/modules/artifacts/schemas.py @@ -27,16 +27,6 @@ class GuideArtifactAdmissionRequest: guide_source_snapshot_id: UUID | None = None -@final -@dataclass(frozen=True, slots=True) -class ContributorArtifactAdmissionRequest: - """One prepared contributor item admitted under its upload session.""" - - authorization_context: AuthorizationContext - upload_item_id: UUID - source: CommittedArtifactSource - - @final @dataclass(frozen=True, slots=True) class CheckerOutputArtifactAdmissionRequest: @@ -49,9 +39,7 @@ class CheckerOutputArtifactAdmissionRequest: ArtifactAdmissionRequest: TypeAlias = ( - GuideArtifactAdmissionRequest - | ContributorArtifactAdmissionRequest - | CheckerOutputArtifactAdmissionRequest + GuideArtifactAdmissionRequest | CheckerOutputArtifactAdmissionRequest ) diff --git a/backend/app/modules/artifacts/service.py b/backend/app/modules/artifacts/service.py index 53c45171c..a916cb5b8 100644 --- a/backend/app/modules/artifacts/service.py +++ b/backend/app/modules/artifacts/service.py @@ -78,7 +78,6 @@ ArtifactRecoveryNotFoundError, DenyArtifactInternalAuthority, CheckerOutputArtifactAdmissionRequest, - ContributorArtifactAdmissionRequest, GuideArtifactAdmissionRequest, GuideArtifactIngestAuthorityFacts, ) @@ -160,7 +159,6 @@ class _AdmissionFacts: task_id: str | None guide_source_item_id: str | None guide_source_snapshot_id: str | None - upload_item_id: str | None checker_run_id: str | None logical_role: str | None operation_identity: str @@ -849,7 +847,6 @@ async def _complete_transaction_b( ArtifactOperationReceipt( id=str(uuid4()), put_attempt_id=attempt.id, - upload_item_id=attempt.upload_item_id, guide_source_item_id=attempt.guide_source_item_id, checker_run_id=attempt.checker_run_id, logical_role=attempt.logical_role, @@ -875,13 +872,6 @@ async def _complete_transaction_b( maximum_attempts=self._settings.artifact_provider_observation_maximum_attempts, ) ) - if attempt.upload_item_id is not None: - item = await self._repo.lock_upload_item(attempt.upload_item_id) - if item is not None: - item.state = "stored_pending_verification" - item.content_id = content.id - item.provider_object_ref = provider_object_ref - item.cas_version += 1 attempt.status = "object_confirmed" attempt.replica_id = replica.id # observation receipts are not acknowledgement receipts; retain the @@ -992,13 +982,6 @@ async def _record_put_absence( expected_byte_count=attempt.byte_count, ) ) - if attempt.upload_item_id is not None: - item = await self._repo.lock_upload_item(attempt.upload_item_id) - if item is not None: - item.state = "replay_required" - item.content_id = None - item.provider_object_ref = None - item.cas_version += 1 attempt.status = "absent_replay_required" attempt.terminal_result_code = "missing" attempt.terminal_at = now @@ -1128,17 +1111,6 @@ async def _record_put_terminal_observation( replica.availability_state = "available" replica.integrity_state = "invalid" attempt.replica_id = replica.id - if outcome == "observed_integrity_mismatch" and attempt.upload_item_id is not None: - item = await self._repo.lock_upload_item(attempt.upload_item_id) - binding = await self._repo.lock_binding_for_content(content.id) - if ( - item is not None - and binding is None - and item.state in {"reserved", "replay_required"} - ): - item.state = "failed" - item.error_code = "artifact_integrity_failure" - item.cas_version += 1 await self._repo.add_put_observation_receipt( ArtifactPutObservationReceipt( id=str(uuid4()), @@ -1296,33 +1268,6 @@ async def _complete_verification( observed_byte_count=observed_size, ) ) - if attempt.upload_item_id is not None: - item = await self._repo.lock_upload_item(attempt.upload_item_id) - if item is not None: - item_changed = False - if outcome == "verified": - item.state = "ready" - item_changed = True - elif outcome == "missing": - content = await self._repo.lock_content(replica.content_id) - if content is None: - raise ArtifactIngestStateError( - "artifact replica content is unavailable" - ) - binding = await self._repo.lock_binding_for_content(replica.content_id) - if binding is None: - item.state = "replay_required" - item.content_id = None - item.provider_object_ref = None - item_changed = True - elif outcome == "integrity_mismatch": - item.state = "failed" - item.content_id = None - item.provider_object_ref = None - item.error_code = "artifact_integrity_failure" - item_changed = True - if item_changed: - item.cas_version += 1 job.status = outcome job.next_run_at = None job.terminal_result_code = outcome @@ -1925,7 +1870,6 @@ async def admit( "project_id": facts.project_id, "task_id": facts.task_id, "guide_source_item_id": facts.guide_source_item_id, - "upload_item_id": facts.upload_item_id, "checker_run_id": facts.checker_run_id, "logical_role": facts.logical_role, "sha256": commitment.sha256, @@ -1980,7 +1924,6 @@ async def admit( project_id=facts.project_id, task_id=facts.task_id, guide_source_item_id=facts.guide_source_item_id, - upload_item_id=facts.upload_item_id, checker_run_id=facts.checker_run_id, logical_role=facts.logical_role, sha256=commitment.sha256, @@ -2014,7 +1957,6 @@ def _validate_request_boundary(request: ArtifactAdmissionRequest) -> None: """Reject open-ended or forged internal request shapes.""" if type(request) not in { GuideArtifactAdmissionRequest, - ContributorArtifactAdmissionRequest, CheckerOutputArtifactAdmissionRequest, }: raise TypeError("invalid artifact admission request") @@ -2046,8 +1988,6 @@ async def _derive_admission_facts(self, request: ArtifactAdmissionRequest) -> _A """Load every product and producer relationship from authoritative rows.""" if type(request) is GuideArtifactAdmissionRequest: return await self._guide_facts(request) - if type(request) is ContributorArtifactAdmissionRequest: - return await self._contributor_facts(request) if type(request) is CheckerOutputArtifactAdmissionRequest: return await self._checker_output_facts(request) raise TypeError("invalid artifact admission request") @@ -2078,51 +2018,6 @@ async def _guide_facts(self, request: GuideArtifactAdmissionRequest) -> _Admissi task_id=None, guide_source_item_id=item_id, guide_source_snapshot_id=row.guide_source_snapshot_id, - upload_item_id=None, - checker_run_id=None, - logical_role=None, - operation_identity=operation_identity, - ) - - async def _contributor_facts( - self, request: ContributorArtifactAdmissionRequest - ) -> _AdmissionFacts: - """Bind committed bytes to one contributor-owned upload item.""" - context = request.authorization_context - if context.actor_kind is not ActorKind.HUMAN: - raise ArtifactAdmissionRelationshipError( - "contributor artifact producer must be a human actor" - ) - await self._require_active_human_actor(context) - item_id = str(request.upload_item_id) - row = await self._repo.get_contributor_admission_facts(item_id) - commitment = request.source.commitment - if ( - row is None - or row.actor_profile_id != str(context.actor_profile_id) - or row.task_id is None - or row.session_state != "open" - or row.item_state not in {"reserved", "replay_required"} - or row.expected_sha256 != commitment.sha256 - or row.expected_size != commitment.byte_count - or row.media_type != commitment.media_type - ): - raise ArtifactAdmissionRelationshipError( - "contributor upload item relationship is unavailable" - ) - operation_identity = canonical_json_hash( - {"request_type": "contributor", "upload_item_id": item_id} - ) - return _AdmissionFacts( - request_type="contributor", - producer_type="actor_profile", - producer_ref=str(context.actor_profile_id), - project_id=row.project_id, - guide_id=None, - task_id=row.task_id, - guide_source_item_id=None, - guide_source_snapshot_id=None, - upload_item_id=item_id, checker_run_id=None, logical_role=None, operation_identity=operation_identity, @@ -2174,7 +2069,6 @@ async def _checker_output_facts( task_id=row.task_id, guide_source_item_id=None, guide_source_snapshot_id=None, - upload_item_id=None, checker_run_id=checker_run_id, logical_role=logical_role, operation_identity=operation_identity, diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py index 10eec036c..fa64254aa 100644 --- a/backend/tests/conftest.py +++ b/backend/tests/conftest.py @@ -21,7 +21,7 @@ from scripts.run_isolated_tests import LOOPBACK, NAME_RE, ROLE_RE DDL_LOCK_DIRECTORY = Path("/tmp") -EXPECTED_PUBLIC_SCHEMA_SHA256 = "80a21fac3c65f252f4b0aaa9ec037a2327fb20ff941e6d4c6a840db80e66ff1c" +EXPECTED_PUBLIC_SCHEMA_SHA256 = "f30127e0acd66d6e2e0cc4d56741ddc8b7cd09175d8c3251749624a0fc353aee" PROTECTED_TEST_TABLES = ( "actor_profile_migration_state", "alembic_version", @@ -42,8 +42,6 @@ "artifact_recovery_attempts", "artifact_replicas", "artifact_storage_namespaces", - "artifact_upload_items", - "artifact_upload_sessions", "artifact_verification_jobs", "artifact_verification_receipts", "audit_events", diff --git a/backend/tests/test_alembic.py b/backend/tests/test_alembic.py index 7fc62be2a..73c9a1f42 100644 --- a/backend/tests/test_alembic.py +++ b/backend/tests/test_alembic.py @@ -73,7 +73,7 @@ snapshot_existing_service_rows, ) -HEAD_REVISION = "0051_review_queue_foundation" +HEAD_REVISION = "0052_legacy_intake_removal" pytestmark = pytest.mark.postgres_schema_contract @@ -230,6 +230,261 @@ def test_alembic_upgrade_and_downgrade(isolated_database_env: str, migration_loc command.downgrade(config, "base") +def test_0051_legacy_intake_safe_empty_round_trip( + isolated_database_env: str, migration_lock +) -> None: + """The clean cut removes the namespace and recreates only an empty legacy shape.""" + config = _alembic_config() + with migration_lock(): + try: + command.downgrade(config, "base") + command.upgrade(config, "0050_guide_source_v2") + legacy = asyncio.run(_legacy_intake_shape(isolated_database_env)) + assert legacy["revision"] == "0050_guide_source_v2" + assert legacy["tables"] == (True, True) + assert legacy["upload_columns"] == (True, True) + + command.upgrade(config, HEAD_REVISION) + removed = asyncio.run(_legacy_intake_shape(isolated_database_env)) + assert removed["revision"] == HEAD_REVISION + assert removed["tables"] == (False, False) + assert removed["upload_columns"] == (False, False) + assert removed["contributor_constraints"] == () + + command.downgrade(config, "0050_guide_source_v2") + restored = asyncio.run(_legacy_intake_shape(isolated_database_env)) + assert restored == legacy + command.upgrade(config, HEAD_REVISION) + finally: + command.downgrade(config, "base") + + +@pytest.mark.parametrize( + "blocker", + ( + "upload_session", + "upload_item", + "contributor_attempt", + "attempt_upload_item", + "v1_receipt", + "receipt_upload_item", + ), +) +def test_0051_legacy_intake_refuses_each_populated_condition_atomically( + isolated_database_env: str, migration_lock, blocker: str +) -> None: + """Every historical row class preserves the entire predecessor schema on refusal.""" + config = _alembic_config() + with migration_lock(): + try: + command.downgrade(config, "base") + command.upgrade(config, "0050_guide_source_v2") + asyncio.run(_seed_0051_legacy_blocker(isolated_database_env, blocker)) + before = asyncio.run(_legacy_intake_shape(isolated_database_env)) + with pytest.raises( + RuntimeError, match="legacy contributor artifact intake is populated" + ): + command.upgrade(config, HEAD_REVISION) + assert asyncio.run(_legacy_intake_shape(isolated_database_env)) == before + finally: + asyncio.run(_reset_0051_test_schema(isolated_database_env)) + + +async def _reset_0051_test_schema(database_url: str) -> None: + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + await connection.execute(text("drop schema public cascade")) + await connection.execute(text("create schema public")) + finally: + await engine.dispose() + + +async def _legacy_intake_shape(database_url: str) -> dict[str, object]: + engine = create_async_engine(database_url) + try: + async with engine.connect() as connection: + has_legacy_tables = bool( + await connection.scalar( + text("select to_regclass('artifact_upload_sessions') is not null") + ) + ) + physical_schema = tuple( + tuple(row) + for row in ( + await connection.execute( + text( + "select 'column',table_name,column_name,data_type,udt_name," + "is_nullable,coalesce(column_default,'')," + "coalesce(character_maximum_length::text,'') " + "from information_schema.columns where table_schema='public' and " + "table_name in ('artifact_upload_sessions','artifact_upload_items'," + "'artifact_put_attempts','artifact_operation_receipts') union all " + "select 'constraint',c.relname,q.conname,q.contype::text," + "pg_get_constraintdef(q.oid,true),'','','' from pg_constraint q " + "join pg_class c on c.oid=q.conrelid join pg_namespace n " + "on n.oid=c.relnamespace where n.nspname='public' and c.relname in " + "('artifact_upload_sessions','artifact_upload_items'," + "'artifact_put_attempts','artifact_operation_receipts') union all " + "select 'index',tablename,indexname,indexdef,'','','','' " + "from pg_indexes where schemaname='public' and tablename in " + "('artifact_upload_sessions','artifact_upload_items'," + "'artifact_put_attempts','artifact_operation_receipts') " + "order by 1,2,3,4" + ) + ) + ).all() + ) + constraints = tuple( + await connection.scalars( + text( + "select pg_get_constraintdef(oid) from pg_constraint " + "where conrelid='artifact_put_attempts'::regclass and contype='c' " + "and pg_get_constraintdef(oid) ilike '%contributor%' order by conname" + ) + ) + ) + return { + "revision": str( + await connection.scalar(text("select version_num from alembic_version")) + ), + "tables": ( + bool( + await connection.scalar( + text("select to_regclass('artifact_upload_sessions') is not null") + ) + ), + bool( + await connection.scalar( + text("select to_regclass('artifact_upload_items') is not null") + ) + ), + ), + "upload_columns": ( + bool( + await connection.scalar( + text( + "select exists(select 1 from information_schema.columns where table_name='artifact_put_attempts' and column_name='upload_item_id')" + ) + ) + ), + bool( + await connection.scalar( + text( + "select exists(select 1 from information_schema.columns where table_name='artifact_operation_receipts' and column_name='upload_item_id')" + ) + ) + ), + ), + "contributor_constraints": constraints, + "physical_schema": physical_schema, + "legacy_rows": tuple( + tuple(row) + for row in ( + await connection.execute( + text( + "select 'session',id,state from artifact_upload_sessions " + "union all select 'item',id,state from artifact_upload_items " + "union all select 'attempt',id,producer_request_type " + "from artifact_put_attempts union all " + "select 'receipt',id,contract_version::text " + "from artifact_operation_receipts order by 1,2" + ) + ) + ).all() + ) + if has_legacy_tables + else (), + "row_counts": tuple( + ( + await connection.execute( + text( + "select (select count(*) from artifact_put_attempts)," + "(select count(*) from artifact_operation_receipts)," + "(select count(*) from artifact_upload_sessions)," + "(select count(*) from artifact_upload_items)" + ) + ) + ).one() + ) + if has_legacy_tables + else (), + } + finally: + await engine.dispose() + + +async def _seed_0051_legacy_blocker(database_url: str, blocker: str) -> None: + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + table = { + "upload_session": "artifact_upload_sessions", + "upload_item": "artifact_upload_items", + "contributor_attempt": "artifact_put_attempts", + "attempt_upload_item": "artifact_put_attempts", + "v1_receipt": "artifact_operation_receipts", + "receipt_upload_item": "artifact_operation_receipts", + }[blocker] + constraint_names = tuple( + await connection.scalars( + text( + "select conname from pg_constraint where conrelid=cast(:table as regclass) " + "and contype in ('c','f') order by conname" + ), + {"table": table}, + ) + ) + for name in constraint_names: + await connection.execute(text(f'alter table {table} drop constraint "{name}"')) + identifier = str(uuid4()) + if blocker == "upload_session": + await connection.execute( + text( + "insert into artifact_upload_sessions (id,actor_id,project_id,permitted_roles,state,maximum_bytes,current_bytes,reserved_bytes,maximum_items,current_items,reserved_items,expires_at,cas_version) values (:id,'actor','project','[]'::json,'open',1,0,0,1,0,0,now(),0)" + ), + {"id": identifier}, + ) + elif blocker == "upload_item": + await connection.execute( + text( + "insert into artifact_upload_items (id,session_id,logical_role,display_name,reserved_bytes,idempotency_key,request_digest,state,cas_version) values (:id,'session','result','result.zip',1,'key',:digest,'reserved',0)" + ), + {"id": identifier, "digest": "sha256:" + "1" * 64}, + ) + elif blocker in {"contributor_attempt", "attempt_upload_item"}: + await connection.execute( + text( + "insert into artifact_put_attempts (id,producer_request_type,producer_type,producer_ref,project_id,task_id,upload_item_id,sha256,byte_count,media_type,storage_namespace_id,namespace_fingerprint,canonical_target,operation_identity,request_digest,status,execution_generation,observation_count,maximum_observations,cas_version) values (:id,:request_type,'actor_profile',:actor,'project',:task,:item,:digest,1,'application/zip','primary',:digest,'sha256/11/' || repeat('1',62),:digest,:digest,'prepared',0,0,5,0)" + ), + { + "id": identifier, + "request_type": "contributor" + if blocker == "contributor_attempt" + else "guide", + "actor": str(uuid4()), + "task": "task" if blocker == "contributor_attempt" else None, + "item": None if blocker == "contributor_attempt" else "item", + "digest": "sha256:" + "1" * 64, + }, + ) + else: + await connection.execute( + text( + "insert into artifact_operation_receipts (id,contract_version,put_attempt_id,upload_item_id,replica_id,operation,idempotency_key,request_digest,provider_object_ref,replayed,outcome,attempt_number,correlation_id,details) values (:id,:version,:attempt,:item,'replica','put','key',:digest,'object',false,'stored_pending_verification',1,'correlation','[]'::json)" + ), + { + "id": identifier, + "version": 1 if blocker == "v1_receipt" else 2, + "attempt": None if blocker == "v1_receipt" else "attempt", + "item": "item", + "digest": "sha256:" + "1" * 64, + }, + ) + finally: + await engine.dispose() + + async def _project_setup_run_check_constraint_names(database_url: str) -> set[str]: """Return physical check-constraint names for the setup-run table.""" engine = create_async_engine(database_url) @@ -3672,8 +3927,6 @@ def test_current_schema_uses_project_policy_contract( "checker_policies.pre_submit_checker_bundle_hash", "payment_policies.base_amount", "payment_policies.currency", - "artifact_upload_sessions.id", - "artifact_upload_items.id", "artifact_contents.sha256", "artifact_bindings.scope_version", "artifact_storage_namespaces.namespace_fingerprint", @@ -3706,6 +3959,8 @@ def test_current_schema_uses_project_policy_contract( "workstream_tasks.locked_checker_policy_version", "submissions.locked_checker_policy_version", "checker_runs.locked_checker_policy_version", + "artifact_upload_sessions.id", + "artifact_upload_items.id", "artifact_replicas.provider_artifact_id", "artifact_replicas.provider_manifest_id", "artifact_replicas.retention_state", diff --git a/backend/tests/test_artifact_admission.py b/backend/tests/test_artifact_admission.py index e01c94bbe..e8e69f493 100644 --- a/backend/tests/test_artifact_admission.py +++ b/backend/tests/test_artifact_admission.py @@ -6,7 +6,7 @@ import asyncio from collections.abc import Iterator from dataclasses import replace -from datetime import UTC, datetime, timedelta +from datetime import UTC, datetime from pathlib import Path from types import SimpleNamespace from unittest.mock import AsyncMock @@ -32,7 +32,6 @@ from app.modules.artifacts.models import ( ArtifactAdmissionCharge, ArtifactAdmissionScope, - ArtifactBinding, ArtifactContent, ArtifactOperationReceipt, ArtifactPutObservationReceipt, @@ -40,8 +39,6 @@ ArtifactPutAttemptCharge, ArtifactReplica, ArtifactStorageNamespace, - ArtifactUploadItem, - ArtifactUploadSession, ArtifactVerificationJob, ArtifactVerificationReceipt, ) @@ -62,12 +59,9 @@ ArtifactAuthorityDeniedError, ArtifactInternalResourceType, CheckerOutputArtifactAdmissionRequest, - ContributorArtifactAdmissionRequest, GuideArtifactAdmissionRequest, ) from app.modules.artifacts.service import ( - ArtifactAdmissionCapacityError, - ArtifactAdmissionConflictError, ArtifactAdmissionRelationshipError, ArtifactAdmissionService, ArtifactStorageNamespaceSpec, @@ -343,85 +337,6 @@ async def _seed_guide( return project_id, item_id -async def _seed_contributor_items( - session, - *, - context: AuthorizationContext, - commitments: tuple[tuple[str, int, str], ...], -) -> tuple[str, str, tuple[str, ...]]: - await _seed_human_actor(session, context) - actor_profile_id = str(context.actor_profile_id) - project_id = str(uuid4()) - task_id = str(uuid4()) - upload_session_id = str(uuid4()) - await seed_historical_project( - session, - project_id=project_id, - name="Contributor project", - slug=f"contributor-{project_id}", - ) - await session.flush() - session.add( - WorkstreamTask( - id=task_id, - project_id=project_id, - title="Admission task", - description="Prove artifact admission.", - status="draft", - created_by="test", - ) - ) - await session.flush() - total_bytes = sum(byte_count for _, byte_count, _ in commitments) - session.add( - ArtifactUploadSession( - id=upload_session_id, - actor_id=actor_profile_id, - project_id=project_id, - task_id=task_id, - permitted_roles=["submission"], - state="open", - maximum_bytes=max(total_bytes, 1), - current_bytes=0, - reserved_bytes=total_bytes, - maximum_items=len(commitments), - current_items=0, - reserved_items=len(commitments), - expires_at=datetime.now(UTC) + timedelta(minutes=10), - cas_version=0, - ) - ) - await session.flush() - item_ids = [] - for index, (sha256, byte_count, media_type) in enumerate(commitments): - item_id = str(uuid4()) - item_ids.append(item_id) - session.add( - ArtifactUploadItem( - id=item_id, - session_id=upload_session_id, - logical_role=f"submission-{index}", - display_name=f"result-{index}.bin", - media_type=media_type, - reserved_bytes=byte_count, - expected_sha256=sha256, - expected_size=byte_count, - idempotency_key=f"put-{item_id}", - request_digest=canonical_json_hash( - { - "sha256": sha256, - "byte_count": byte_count, - "media_type": media_type, - } - ), - state="reserved", - cas_version=0, - ) - ) - await session.commit() - return project_id, task_id, tuple(item_ids) - - async def _seed_checker_output_relationships(session) -> tuple[str, str, str]: """Persist one complete checker-run ownership chain for admission proof.""" project_id = str(uuid4()) @@ -827,6 +742,48 @@ async def _admit_guide_source(session, settings, namespace, context, source): ) +async def _admit_checker_output(session, settings, namespace, source): + """Create one exact task-scoped checker-output attempt for shared-path tests.""" + actor_id, link_id = uuid4(), uuid4() + context = _context( + actor_profile_id=actor_id, + identity_link_id=link_id, + actor_kind=ActorKind.SERVICE, + ) + project_id, task_id, checker_run_id = await _seed_checker_output_relationships(session) + session.add( + ActorProfile( + id=str(actor_id), + actor_kind="service", + status="active", + provisioning_method="manual_service_provisioning", + service_identity=ServiceIdentity.ARTIFACT_CHECKER_OUTPUT.value, + created_by="test", + ) + ) + session.add( + ActorIdentityLink( + id=str(link_id), + actor_profile_id=str(actor_id), + issuer="https://issuer.example.test", + subject=f"checker-output-{actor_id}", + subject_kind="service", + status="active", + linked_by="test", + ) + ) + await session.commit() + result = await ArtifactAdmissionService(session, settings, namespace).admit( + CheckerOutputArtifactAdmissionRequest( + authorization_context=context, + checker_run_id=UUID(checker_run_id), + logical_role="platform-review", + source=source, + ) + ) + return project_id, task_id, checker_run_id, result + + async def test_committed_put_and_independent_verification_are_fenced( admission_database_env: str, tmp_path: Path, @@ -1188,203 +1145,6 @@ async def observe_put_result(_commitment: object) -> ArtifactPutObservation: await engine.dispose() -async def test_preacknowledgement_mismatch_fails_contributor_item_and_keeps_charge( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - assert settings.artifact_local_root is not None - bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) - store = bootstrap.initialize_after_namespace_claim( - ArtifactStoreNamespaceClaim( - adapter_identity=bootstrap.identity, - namespace_identity=bootstrap.namespace_identity, - namespace_fingerprint=namespace.namespace_fingerprint, - ) - ) - authority = _AllowArtifactAuthority() - try: - async with factory() as session: - async with minted_source( - tmp_path / "mismatch-contributor-source", - b"expected immutable bytes", - media_type="text/plain", - ) as source: - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - admission = await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) - ) - attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) - assert attempt is not None - prefix, filename = attempt.canonical_target.removeprefix("sha256/").split("/") - provider_path = settings.artifact_local_root / "objects" / "sha256" / prefix - provider_path.mkdir(mode=0o700) - corrupt_object = provider_path / filename - corrupt_object.write_bytes(b"different provider bytes") - corrupt_object.chmod(0o400) - await session.rollback() - orchestrator = ArtifactStorageOrchestrator( - session, store, namespace, settings, authority - ) - assert ( - await orchestrator.resolve_put_attempt(admission.attempt_id) - == "integrity_mismatch" - ) - attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) - assert attempt is not None - assert attempt.status == "integrity_mismatch" - assert attempt.replica_id is not None - replica = await session.get(ArtifactReplica, attempt.replica_id) - assert replica is not None - assert [ - replica.verification_state, - replica.availability_state, - replica.integrity_state, - ] == ["integrity_mismatch", "available", "invalid"] - charges = (await session.execute(select(ArtifactAdmissionCharge))).scalars().all() - assert charges and {charge.state for charge in charges} == {"completed"} - item = await session.get(ArtifactUploadItem, item_ids[0]) - assert item is not None - assert item.state == "failed" - assert item.error_code == "artifact_integrity_failure" - assert item.cas_version == 1 - assert await _count(session, ArtifactPutObservationReceipt) == 1 - assert await _count(session, ArtifactOperationReceipt) == 0 - finally: - bootstrap.close() - await engine.dispose() - - -async def test_mismatch_transition_serializes_concurrent_binding_insert( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - assert settings.artifact_local_root is not None - bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) - store = bootstrap.initialize_after_namespace_claim( - ArtifactStoreNamespaceClaim( - adapter_identity=bootstrap.identity, - namespace_identity=bootstrap.namespace_identity, - namespace_fingerprint=namespace.namespace_fingerprint, - ) - ) - content_locked = asyncio.Event() - allow_mismatch_completion = asyncio.Event() - binding_flush_started = asyncio.Event() - - class PausingContentLockRepository(ArtifactRepository): - async def lock_content(self, content_id: str): - content = await super().lock_content(content_id) - content_locked.set() - await allow_mismatch_completion.wait() - return content - - try: - async with factory() as session: - async with minted_source(tmp_path / "mismatch-binding-race", b"expected") as source: - project_id, task_id, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - admission = await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) - ) - attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) - assert attempt is not None - prefix, filename = attempt.canonical_target.removeprefix("sha256/").split("/") - provider_path = settings.artifact_local_root / "objects" / "sha256" / prefix - provider_path.mkdir(mode=0o700) - corrupt_object = provider_path / filename - corrupt_object.write_bytes(b"different") - corrupt_object.chmod(0o400) - content = ArtifactContent( - id=str(uuid4()), - sha256=source.commitment.sha256, - byte_count=source.commitment.byte_count, - media_type=source.commitment.media_type, - normalized_display_name=None, - ) - session.add(content) - await session.commit() - content_id = content.id - orchestrator = ArtifactStorageOrchestrator( - session, store, namespace, settings, _AllowArtifactAuthority() - ) - orchestrator._repo = PausingContentLockRepository(session) - resolution = asyncio.create_task( - orchestrator.resolve_put_attempt(admission.attempt_id) - ) - await asyncio.wait_for(content_locked.wait(), timeout=2) - - async def insert_binding() -> None: - async with factory() as binding_session, binding_session.begin(): - binding_session.add( - ArtifactBinding( - id=str(uuid4()), - content_id=content_id, - project_id=project_id, - resource_type="task", - resource_id=task_id, - logical_role="submission", - scope_version=1, - actor_id=str(context.actor_profile_id), - attribution_type="human", - supersedes_binding_id=None, - ) - ) - binding_flush_started.set() - await binding_session.flush() - - binding_insert = asyncio.create_task(insert_binding()) - await asyncio.wait_for(binding_flush_started.wait(), timeout=2) - with pytest.raises(TimeoutError): - await asyncio.wait_for(asyncio.shield(binding_insert), timeout=0.2) - allow_mismatch_completion.set() - assert await resolution == "integrity_mismatch" - await asyncio.wait_for(binding_insert, timeout=2) - item = await session.get(ArtifactUploadItem, item_ids[0]) - assert item is not None and item.state == "failed" - assert await _count(session, ArtifactBinding) == 1 - finally: - allow_mismatch_completion.set() - bootstrap.close() - await engine.dispose() - - async def test_simultaneous_put_claims_have_one_generation_winner( admission_database_env: str, tmp_path: Path, @@ -2023,15 +1783,12 @@ async def test_existing_replica_immutable_fact_conflict_is_fenced( await engine.dispose() -async def test_verification_resource_drift_returns_stale_without_mutation( +async def test_verification_resource_drift_after_read_is_stale_without_terminal_facts( admission_database_env: str, tmp_path: Path, ) -> None: settings = _settings(tmp_path) namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) assert settings.artifact_local_root is not None bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) store = bootstrap.initialize_after_namespace_claim( @@ -2041,26 +1798,13 @@ async def test_verification_resource_drift_returns_stale_without_mutation( namespace_fingerprint=namespace.namespace_fingerprint, ) ) + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) try: async with factory() as session: async with minted_source(tmp_path / "verification-drift", b"expected") as source: - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - admission = await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) + admission = await _admit_guide_source( + session, settings, namespace, _context(), source ) orchestrator = ArtifactStorageOrchestrator( session, store, namespace, settings, _AllowArtifactAuthority() @@ -2070,8 +1814,7 @@ async def test_verification_resource_drift_returns_stale_without_mutation( ) attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) job = await session.scalar(select(ArtifactVerificationJob)) - assert attempt is not None and attempt.replica_id is not None - assert job is not None + assert attempt is not None and attempt.replica_id is not None and job is not None original_replica_id = attempt.replica_id unrelated_content = ArtifactContent( id=str(uuid4()), @@ -2113,15 +1856,11 @@ async def drift_job_after_claim(_provider_object_ref: str): await session.refresh(job) await session.refresh(unrelated_replica) original_replica = await session.get(ArtifactReplica, original_replica_id) - item = await session.get(ArtifactUploadItem, item_ids[0]) - assert original_replica is not None and item is not None + assert original_replica is not None assert job.status == "running" assert unrelated_replica.verification_state == "pending" assert original_replica.verification_state == "pending" - assert item.state == "stored_pending_verification" - assert item.content_id == original_replica.content_id - receipt = await session.scalar(select(ArtifactVerificationReceipt)) - assert receipt is None + assert await _count(session, ArtifactVerificationReceipt) == 0 finally: bootstrap.close() await engine.dispose() @@ -2362,9 +2101,6 @@ async def test_verification_rechecks_authorized_object_ref_after_io( ) -> None: settings = _settings(tmp_path) namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) assert settings.artifact_local_root is not None bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) store = bootstrap.initialize_after_namespace_claim( @@ -2374,26 +2110,13 @@ async def test_verification_rechecks_authorized_object_ref_after_io( namespace_fingerprint=namespace.namespace_fingerprint, ) ) + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) try: async with factory() as session: async with minted_source(tmp_path / "postread-object-ref-drift", b"expected") as source: - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - admission = await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) + admission = await _admit_guide_source( + session, settings, namespace, _context(), source ) orchestrator = ArtifactStorageOrchestrator( session, store, namespace, settings, _AllowArtifactAuthority() @@ -2404,8 +2127,7 @@ async def test_verification_rechecks_authorized_object_ref_after_io( job = await session.scalar(select(ArtifactVerificationJob)) attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) assert job is not None and attempt is not None and attempt.replica_id is not None - job_id = UUID(job.id) - replica_id = attempt.replica_id + job_id, replica_id = UUID(job.id), attempt.replica_id await session.commit() async def drift_object_ref_during_read(_provider_object_ref: str): @@ -2421,26 +2143,30 @@ async def drift_object_ref_during_read(_provider_object_ref: str): assert await orchestrator.verify_object(job_id) == "stale" await session.refresh(job) replica = await session.get(ArtifactReplica, replica_id) - item = await session.get(ArtifactUploadItem, item_ids[0]) - assert replica is not None and item is not None + assert replica is not None assert job.status == "running" assert replica.verification_state == "pending" - assert item.state == "stored_pending_verification" - assert await session.scalar(select(func.count(ArtifactVerificationReceipt.id))) == 0 + assert await _count(session, ArtifactVerificationReceipt) == 0 finally: bootstrap.close() await engine.dispose() -@pytest.mark.parametrize("bound", [False, True]) -async def test_post_ack_missing_replays_only_unbound_contributor_item( +@pytest.mark.parametrize( + ("provider_result", "expected"), + [ + (("sha256:" + "e" * 64, 7), "integrity_mismatch"), + (ArtifactStoreError("conflict"), "conflict"), + ], +) +async def test_verification_terminal_result_matrix( admission_database_env: str, tmp_path: Path, - bound: bool, + provider_result: tuple[str, int] | Exception, + expected: str, ) -> None: settings = _settings(tmp_path) namespace = _namespace(settings) - context = _context() engine = create_async_engine(admission_database_env) factory = async_sessionmaker(engine, expire_on_commit=False) assert settings.artifact_local_root is not None @@ -2454,227 +2180,17 @@ async def test_post_ack_missing_replays_only_unbound_contributor_item( ) try: async with factory() as session: - async with minted_source(tmp_path / f"missing-{bound}", b"contributor") as source: - project_id, task_id, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - admission = await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) + async with minted_source(tmp_path / expected, b"verification matrix") as source: + admission = await _admit_guide_source( + session, settings, namespace, _context(), source ) orchestrator = ArtifactStorageOrchestrator( session, store, namespace, settings, _AllowArtifactAuthority() ) - assert ( - await orchestrator.execute_committed_put( - attempt_id=admission.attempt_id, - source=source, - ) - == "stored_pending_verification" - ) - attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) - assert attempt is not None and attempt.replica_id is not None - replica = await session.get(ArtifactReplica, attempt.replica_id) - assert replica is not None - if bound: - session.add( - ArtifactBinding( - id=str(uuid4()), - content_id=replica.content_id, - project_id=project_id, - resource_type="task", - resource_id=task_id, - logical_role="submission", - scope_version=1, - actor_id=str(context.actor_profile_id), - attribution_type="human", - supersedes_binding_id=None, - ) - ) - await session.commit() - job = await session.scalar( - select(ArtifactVerificationJob).where( - ArtifactVerificationJob.originating_put_attempt_id == attempt.id - ) + await orchestrator.execute_committed_put( + attempt_id=admission.attempt_id, source=source ) - assert job is not None - job_id = UUID(job.id) - await session.rollback() - orchestrator._read_complete = AsyncMock( - side_effect=ArtifactObjectMissingError("missing") - ) - assert await orchestrator.verify_object(job_id) == "missing" - item = await session.get(ArtifactUploadItem, item_ids[0]) - assert item is not None - assert item.state == ("stored_pending_verification" if bound else "replay_required") - assert (item.content_id is not None) is bound - assert (item.provider_object_ref is not None) is bound - finally: - bootstrap.close() - await engine.dispose() - - -async def test_missing_transition_serializes_concurrent_binding_insert( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - assert settings.artifact_local_root is not None - bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) - store = bootstrap.initialize_after_namespace_claim( - ArtifactStoreNamespaceClaim( - adapter_identity=bootstrap.identity, - namespace_identity=bootstrap.namespace_identity, - namespace_fingerprint=namespace.namespace_fingerprint, - ) - ) - content_locked = asyncio.Event() - allow_missing_completion = asyncio.Event() - binding_flush_started = asyncio.Event() - content_lock_count = 0 - - class PausingContentLockRepository(ArtifactRepository): - async def lock_content(self, content_id: str): - nonlocal content_lock_count - content = await super().lock_content(content_id) - content_lock_count += 1 - if content_lock_count == 2: - content_locked.set() - await allow_missing_completion.wait() - return content - - try: - async with factory() as session: - async with minted_source(tmp_path / "binding-race", b"serialize") as source: - project_id, task_id, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - admission = await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) - ) - orchestrator = ArtifactStorageOrchestrator( - session, store, namespace, settings, _AllowArtifactAuthority() - ) - await orchestrator.execute_committed_put( - attempt_id=admission.attempt_id, source=source - ) - attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) - assert attempt is not None and attempt.replica_id is not None - replica = await session.get(ArtifactReplica, attempt.replica_id) - assert replica is not None - content_id = replica.content_id - job = await session.scalar(select(ArtifactVerificationJob)) - assert job is not None - job_id = UUID(job.id) - await session.rollback() - orchestrator._repo = PausingContentLockRepository(session) - orchestrator._read_complete = AsyncMock( - side_effect=ArtifactObjectMissingError("missing") - ) - verification = asyncio.create_task(orchestrator.verify_object(job_id)) - await asyncio.wait_for(content_locked.wait(), timeout=2) - - async def insert_binding() -> None: - async with factory() as binding_session, binding_session.begin(): - binding_session.add( - ArtifactBinding( - id=str(uuid4()), - content_id=content_id, - project_id=project_id, - resource_type="task", - resource_id=task_id, - logical_role="submission", - scope_version=1, - actor_id=str(context.actor_profile_id), - attribution_type="human", - supersedes_binding_id=None, - ) - ) - binding_flush_started.set() - await binding_session.flush() - - binding_insert = asyncio.create_task(insert_binding()) - await asyncio.wait_for(binding_flush_started.wait(), timeout=2) - with pytest.raises(TimeoutError): - await asyncio.wait_for(asyncio.shield(binding_insert), timeout=0.2) - allow_missing_completion.set() - assert await verification == "missing" - await asyncio.wait_for(binding_insert, timeout=2) - item = await session.get(ArtifactUploadItem, item_ids[0]) - assert item is not None and item.state == "replay_required" - assert await _count(session, ArtifactBinding) == 1 - finally: - allow_missing_completion.set() - bootstrap.close() - await engine.dispose() - - -@pytest.mark.parametrize( - ("provider_result", "expected"), - [ - (("sha256:" + "e" * 64, 7), "integrity_mismatch"), - (ArtifactStoreError("conflict"), "conflict"), - ], -) -async def test_verification_terminal_result_matrix( - admission_database_env: str, - tmp_path: Path, - provider_result: tuple[str, int] | Exception, - expected: str, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - assert settings.artifact_local_root is not None - bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) - store = bootstrap.initialize_after_namespace_claim( - ArtifactStoreNamespaceClaim( - adapter_identity=bootstrap.identity, - namespace_identity=bootstrap.namespace_identity, - namespace_fingerprint=namespace.namespace_fingerprint, - ) - ) - try: - async with factory() as session: - async with minted_source(tmp_path / expected, b"verification matrix") as source: - admission = await _admit_guide_source( - session, settings, namespace, _context(), source - ) - orchestrator = ArtifactStorageOrchestrator( - session, store, namespace, settings, _AllowArtifactAuthority() - ) - await orchestrator.execute_committed_put( - attempt_id=admission.attempt_id, source=source - ) - job = await session.scalar(select(ArtifactVerificationJob)) + job = await session.scalar(select(ArtifactVerificationJob)) assert job is not None job_id = UUID(job.id) await session.rollback() @@ -2823,139 +2339,6 @@ async def test_verification_unavailable_retries_then_exhausts( await engine.dispose() -@pytest.mark.postgres_schema_contract -async def test_0030_populated_contributor_receipt_upgrade_and_guarded_downgrade( - admission_database_env: str, - tmp_path: Path, -) -> None: - """Prove populated v1 receipt compatibility and refusal to erase v2 evidence.""" - settings = _settings(tmp_path) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - receipt_id = str(uuid4()) - try: - async with factory() as session: - async with minted_source(tmp_path / "migration-0029", b"legacy receipt") as source: - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - admission = await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) - ) - attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) - assert attempt is not None - content = ArtifactContent( - id=str(uuid4()), - sha256=attempt.sha256, - byte_count=attempt.byte_count, - media_type=attempt.media_type, - normalized_display_name=None, - ) - session.add(content) - await session.flush() - replica = ArtifactReplica( - id=str(uuid4()), - content_id=content.id, - storage_namespace_id=attempt.storage_namespace_id, - namespace_fingerprint=attempt.namespace_fingerprint, - adapter="local", - provider_profile=namespace.provider_profile, - provider_object_ref=attempt.canonical_target, - verification_state="pending", - availability_state="unknown", - integrity_state="unknown", - ) - session.add(replica) - await session.flush() - session.add( - ArtifactOperationReceipt( - id=receipt_id, - contract_version=1, - put_attempt_id=None, - upload_item_id=item_ids[0], - guide_source_item_id=None, - checker_run_id=None, - logical_role=None, - replica_id=replica.id, - operation="put", - idempotency_key="legacy-put", - request_digest=attempt.request_digest, - provider_object_ref=attempt.canonical_target, - replayed=False, - outcome="stored_pending_verification", - attempt_number=1, - correlation_id="legacy-correlation", - details=[], - ) - ) - await session.flush() - attempt.status = "object_confirmed" - attempt.replica_id = replica.id - attempt.receipt_id = receipt_id - attempt.terminal_result_code = "acknowledged" - attempt.terminal_at = datetime.now(UTC) - await session.commit() - finally: - await engine.dispose() - - config = _alembic_config() - await asyncio.to_thread(command.downgrade, config, "0028_artifact_admission") - await asyncio.to_thread(command.upgrade, config, "0030_artifact_verification") - engine = create_async_engine(admission_database_env) - try: - async with engine.begin() as session: - migrated = ( - ( - await session.execute( - text( - "select contract_version, put_attempt_id, upload_item_id " - "from artifact_operation_receipts where id = :id" - ), - {"id": receipt_id}, - ) - ) - .mappings() - .one() - ) - assert dict(migrated) == { - "contract_version": 2, - "put_attempt_id": str(admission.attempt_id), - "upload_item_id": item_ids[0], - } - await session.execute( - text( - "insert into artifact_put_observation_receipts " - "(id, put_attempt_id, execution_generation, outcome, expected_sha256, " - "expected_byte_count) values (:id, :attempt, 1, 'observed_missing', " - ":sha256, :byte_count)" - ), - { - "id": str(uuid4()), - "attempt": str(admission.attempt_id), - "sha256": attempt.sha256, - "byte_count": attempt.byte_count, - }, - ) - finally: - await engine.dispose() - with pytest.raises(RuntimeError, match="cannot downgrade populated artifact verification"): - await asyncio.to_thread(command.downgrade, config, "0028_artifact_admission") - - async def test_guide_admission_derives_three_scopes_without_provider_evidence( admission_database_env: str, tmp_path: Path, @@ -3112,6 +2495,15 @@ async def test_guide_admission_derives_three_scopes_without_provider_evidence( assert await _count(session, ArtifactContent) == 0 assert await _count(session, ArtifactReplica) == 0 assert await _count(session, ArtifactOperationReceipt) == 0 + with pytest.raises(DBAPIError): + await session.execute( + text( + "update artifact_put_attempts " + "set producer_request_type='contributor' where id=:attempt_id" + ), + {"attempt_id": str(result.attempt_id)}, + ) + await session.rollback() finally: await engine.dispose() @@ -3317,116 +2709,6 @@ async def test_guide_admission_consumes_real_project_manager_prep_atomically( await engine.dispose() -async def test_human_admission_revalidates_exact_active_profile_and_link( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - try: - async with factory() as session: - async with minted_source( - tmp_path / "guide-source", - b"guide", - media_type="text/markdown", - ) as guide_source: - _, _guide_item_id = await _seed_guide( - session, - context=context, - content_hash=guide_source.commitment.sha256, - media_type=guide_source.commitment.media_type, - ) - async with minted_source( - tmp_path / "contributor-source", - b"work", - ) as contributor_source: - _, _, upload_item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - contributor_source.commitment.sha256, - contributor_source.commitment.byte_count, - contributor_source.commitment.media_type, - ), - ), - ) - requests = ( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(upload_item_ids[0]), - source=contributor_source, - ), - ) - forged_context = context.model_copy(update={"identity_link_id": uuid4()}) - for request in requests: - with pytest.raises( - ArtifactAdmissionRelationshipError, - match="artifact admission human identity is unavailable", - ): - await ArtifactAdmissionService(session, settings, namespace).admit( - replace( - request, - authorization_context=forged_context, - ) - ) - - link = await session.get( - ActorIdentityLink, - str(context.identity_link_id), - ) - assert link is not None - link.status = "revoked" - link.revoked_by = "test" - link.revoked_at = datetime.now(UTC) - link.revoked_reason = "test revocation" - await session.commit() - for request in requests: - with pytest.raises( - ArtifactAdmissionRelationshipError, - match="artifact admission human identity is unavailable", - ): - await ArtifactAdmissionService(session, settings, namespace).admit( - request - ) - - link.status = "active" - link.revoked_by = None - link.revoked_at = None - link.revoked_reason = None - link.reactivated_by = "test" - link.reactivated_at = datetime.now(UTC) - link.reactivation_reason = "test reactivation" - profile = await session.get( - ActorProfile, - str(context.actor_profile_id), - ) - assert profile is not None - profile.status = "suspended" - profile.suspended_by = "test" - profile.suspended_at = datetime.now(UTC) - profile.suspension_reason = "test suspension" - await session.commit() - for request in requests: - with pytest.raises( - ArtifactAdmissionRelationshipError, - match="artifact admission human identity is unavailable", - ): - await ArtifactAdmissionService(session, settings, namespace).admit( - request - ) - - assert await _count(session, ArtifactStorageNamespace) == 0 - assert await _count(session, ArtifactAdmissionScope) == 0 - assert await _count(session, ArtifactAdmissionCharge) == 0 - assert await _count(session, ArtifactPutAttempt) == 0 - finally: - await engine.dispose() - - async def test_guide_admission_facts_lock_snapshot_and_item( admission_database_env: str, tmp_path: Path, @@ -3558,640 +2840,43 @@ async def test_actor_admission_proof_locks_exact_profile_then_link( await engine.dispose() -async def test_exact_replay_returns_one_attempt_and_one_charge_set( +async def test_checker_output_requires_exact_active_fixed_service_identity( admission_database_env: str, tmp_path: Path, ) -> None: settings = _settings(tmp_path) namespace = _namespace(settings) - context = _context() + actor_id = uuid4() + link_id = uuid4() + context = _context( + actor_profile_id=actor_id, + identity_link_id=link_id, + actor_kind=ActorKind.SERVICE, + ) engine = create_async_engine(admission_database_env) factory = async_sessionmaker(engine, expire_on_commit=False) try: async with factory() as session: - async with minted_source(tmp_path / "scratch-source", b"same") as source: - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), + project_id, task_id, checker_run_id = await _seed_checker_output_relationships(session) + session.add( + ActorProfile( + id=str(actor_id), + actor_kind="service", + status="active", + provisioning_method="manual_service_provisioning", + service_identity=ServiceIdentity.ARTIFACT_CHECKER_OUTPUT.value, + created_by="test", ) - request = ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) - first = await ArtifactAdmissionService( - session, - settings, - namespace, - ).admit(request) - replay = await ArtifactAdmissionService( - session, - settings, - namespace, - ).admit(request) - - assert replay.replayed is True - assert replay.attempt_id == first.attempt_id - assert replay.charge_ids == first.charge_ids - assert await _count(session, ArtifactPutAttempt) == 1 - assert await _count(session, ArtifactAdmissionCharge) == 4 - assert await _count(session, ArtifactPutAttemptCharge) == 4 - finally: - await engine.dispose() - - -async def test_exact_replay_reacquires_released_charges_under_capacity( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path, maximum_bytes=4) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - try: - async with factory() as session: - async with minted_source(tmp_path / "first-source", b"aaaa") as first_source: - async with minted_source(tmp_path / "second-source", b"bbbb") as second_source: - first_sha256 = first_source.commitment.sha256 - second_sha256 = second_source.commitment.sha256 - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - first_source.commitment.sha256, - first_source.commitment.byte_count, - first_source.commitment.media_type, - ), - ( - second_source.commitment.sha256, - second_source.commitment.byte_count, - second_source.commitment.media_type, - ), - ), - ) - first_request = ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=first_source, - ) - second_request = ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[1]), - source=second_source, - ) - first = await ArtifactAdmissionService(session, settings, namespace).admit( - first_request - ) - first_attempt = await session.get( - ArtifactPutAttempt, - str(first.attempt_id), - ) - assert first_attempt is not None - first_attempt.status = "absent_replay_required" - counters = ( - (await session.execute(select(ArtifactAdmissionScope))).scalars().all() - ) - first_charges = ( - ( - await session.execute( - select(ArtifactAdmissionCharge).where( - ArtifactAdmissionCharge.sha256 == first_sha256 - ) - ) - ) - .scalars() - .all() - ) - released_at = datetime.now(UTC) - for charge in first_charges: - charge.state = "released" - charge.released_at = released_at - charge.cas_version += 1 - for counter in counters: - counter.counted_bytes = 0 - counter.cas_version += 1 - await session.commit() - - await ArtifactAdmissionService(session, settings, namespace).admit( - second_request - ) - with pytest.raises(ArtifactAdmissionCapacityError): - await ArtifactAdmissionService(session, settings, namespace).admit( - first_request - ) - - second_charges = ( - ( - await session.execute( - select(ArtifactAdmissionCharge).where( - ArtifactAdmissionCharge.sha256 == second_sha256 - ) - ) - ) - .scalars() - .all() - ) - counters = ( - (await session.execute(select(ArtifactAdmissionScope))).scalars().all() - ) - for charge in second_charges: - charge.state = "released" - charge.released_at = datetime.now(UTC) - charge.cas_version += 1 - for counter in counters: - counter.counted_bytes = 0 - counter.cas_version += 1 - await session.commit() - - replay = await ArtifactAdmissionService(session, settings, namespace).admit( - first_request - ) - - assert replay.replayed is True - assert replay.attempt_id == first.attempt_id - refreshed_first_charges = ( - ( - await session.execute( - select(ArtifactAdmissionCharge).where( - ArtifactAdmissionCharge.sha256 == first_sha256 - ) - ) - ) - .scalars() - .all() - ) - refreshed_counters = ( - (await session.execute(select(ArtifactAdmissionScope))).scalars().all() - ) - assert {charge.state for charge in refreshed_first_charges} == {"provisional"} - assert {charge.released_at for charge in refreshed_first_charges} == {None} - assert {counter.counted_bytes for counter in refreshed_counters} == {4} - assert await _count(session, ArtifactPutAttempt) == 2 - finally: - await engine.dispose() - - -async def test_contributor_admission_rejects_cross_project_task_relationship( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - try: - async with factory() as session: - async with minted_source(tmp_path / "scratch-source", b"contributor") as source: - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - item = await session.get(ArtifactUploadItem, item_ids[0]) - assert item is not None - upload_session = await session.get(ArtifactUploadSession, item.session_id) - assert upload_session is not None - unrelated_project_id = str(uuid4()) - await seed_historical_project( - session, - project_id=unrelated_project_id, - name="Unrelated admission project", - slug=f"unrelated-{unrelated_project_id}", - ) - await session.flush() - upload_session.project_id = unrelated_project_id - authority_audit_count = await _count(session, AuditEvent) - await session.commit() - - with pytest.raises( - ArtifactAdmissionRelationshipError, - match="contributor upload item relationship is unavailable", - ): - await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) - ) - - assert await _count(session, ArtifactStorageNamespace) == 0 - assert await _count(session, ArtifactAdmissionScope) == 0 - assert await _count(session, ArtifactAdmissionCharge) == 0 - assert await _count(session, ArtifactPutAttempt) == 0 - assert await _count(session, AuditEvent) == authority_audit_count - finally: - await engine.dispose() - - -async def test_same_content_distinct_operations_deduplicate_scope_bytes( - admission_database_env: str, - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - contexts = (_context(), _context()) - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - original_reserve = ArtifactRepository.ensure_and_lock_admission_scopes - ready_count = 0 - ready_lock = asyncio.Lock() - start = asyncio.Event() - - async def synchronized_reserve(repository, scopes): - nonlocal ready_count - async with ready_lock: - ready_count += 1 - if ready_count == 2: - start.set() - await start.wait() - return await original_reserve(repository, scopes) - - monkeypatch.setattr( - ArtifactRepository, - "ensure_and_lock_admission_scopes", - synchronized_reserve, - ) - try: - async with minted_source(tmp_path / "scratch-source", b"same") as source: - async with factory() as seed_session: - commitment = ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ) - item_ids = [] - for context in contexts: - _, _, context_item_ids = await _seed_contributor_items( - seed_session, - context=context, - commitments=(commitment,), - ) - item_ids.append(context_item_ids[0]) - seed_session.add( - ArtifactStorageNamespace( - id="primary", - backend=namespace.backend, - adapter=namespace.adapter, - provider_profile=namespace.provider_profile, - namespace_descriptor=namespace.namespace_descriptor, - namespace_fingerprint=namespace.namespace_fingerprint, - ) - ) - await seed_session.commit() - - async def admit(item_id: str, context: AuthorizationContext): - async with factory() as session: - return await ArtifactAdmissionService( - session, - settings, - namespace, - ).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_id), - source=source, - ) - ) - - results = await asyncio.gather( - *(admit(item_id, context) for item_id, context in zip(item_ids, contexts)) - ) - - async with factory() as session: - assert all(result.replayed is False for result in results) - counters = (await session.execute(select(ArtifactAdmissionScope))).scalars().all() - assert {counter.counted_bytes for counter in counters} == {4} - assert await _count(session, ArtifactAdmissionCharge) == 7 - assert await _count(session, ArtifactPutAttempt) == 2 - assert await _count(session, ArtifactPutAttemptCharge) == 8 - finally: - await engine.dispose() - - -async def test_completed_charge_deduplicates_and_released_charge_is_reacquired( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - try: - async with minted_source(tmp_path / "scratch-source", b"same") as source: - async with factory() as session: - commitment = ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ) - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=(commitment, commitment, commitment), - ) - service = ArtifactAdmissionService(session, settings, namespace) - await service.admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) - ) - - charges = (await session.execute(select(ArtifactAdmissionCharge))).scalars().all() - completed_at = datetime.now(UTC) - for charge in charges: - charge.state = "completed" - charge.completed_at = completed_at - await session.commit() - - await service.admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[1]), - source=source, - ) - ) - counters = (await session.execute(select(ArtifactAdmissionScope))).scalars().all() - assert {counter.counted_bytes for counter in counters} == {4} - - released_at = datetime.now(UTC) - for charge in charges: - charge.state = "released" - charge.completed_at = None - charge.released_at = released_at - for counter in counters: - counter.counted_bytes = 0 - counter.cas_version += 1 - await session.commit() - - await service.admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[2]), - source=source, - ) - ) - - refreshed_charges = ( - (await session.execute(select(ArtifactAdmissionCharge))).scalars().all() - ) - refreshed_counters = ( - (await session.execute(select(ArtifactAdmissionScope))).scalars().all() - ) - assert {charge.state for charge in refreshed_charges} == {"provisional"} - assert {charge.released_at for charge in refreshed_charges} == {None} - assert {charge.cas_version for charge in refreshed_charges} == {1} - assert {counter.counted_bytes for counter in refreshed_counters} == {4} - assert await _count(session, ArtifactAdmissionCharge) == 4 - assert await _count(session, ArtifactPutAttempt) == 3 - assert await _count(session, ArtifactPutAttemptCharge) == 12 - finally: - await engine.dispose() - - -async def test_concurrent_distinct_content_cannot_oversubscribe_any_scope( - admission_database_env: str, - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - settings = _settings(tmp_path, maximum_bytes=6) - namespace = _namespace(settings) - contexts = (_context(), _context()) - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - original_reserve = ArtifactRepository.ensure_and_lock_admission_scopes - ready_count = 0 - ready_lock = asyncio.Lock() - start = asyncio.Event() - - async def synchronized_reserve(repository, scopes): - nonlocal ready_count - async with ready_lock: - ready_count += 1 - if ready_count == 2: - start.set() - await start.wait() - return await original_reserve(repository, scopes) - - monkeypatch.setattr( - ArtifactRepository, - "ensure_and_lock_admission_scopes", - synchronized_reserve, - ) - try: - async with minted_source(tmp_path / "scratch-a", b"aaaa") as first_source: - async with minted_source(tmp_path / "scratch-b", b"bbbb") as second_source: - async with factory() as seed_session: - item_ids = [] - for context, source in zip( - contexts, - (first_source, second_source), - ): - _, _, context_item_ids = await _seed_contributor_items( - seed_session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - item_ids.append(context_item_ids[0]) - seed_session.add( - ArtifactStorageNamespace( - id="primary", - backend=namespace.backend, - adapter=namespace.adapter, - provider_profile=namespace.provider_profile, - namespace_descriptor=namespace.namespace_descriptor, - namespace_fingerprint=namespace.namespace_fingerprint, - ) - ) - await seed_session.commit() - - async def admit(item_id: str, source, context: AuthorizationContext): - async with factory() as session: - return await ArtifactAdmissionService( - session, - settings, - namespace, - ).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_id), - source=source, - ) - ) - - outcomes = await asyncio.gather( - admit(item_ids[0], first_source, contexts[0]), - admit(item_ids[1], second_source, contexts[1]), - return_exceptions=True, - ) - - assert sum(not isinstance(value, BaseException) for value in outcomes) == 1 - assert sum(isinstance(value, ArtifactAdmissionCapacityError) for value in outcomes) == 1 - async with factory() as session: - counters = (await session.execute(select(ArtifactAdmissionScope))).scalars().all() - assert {counter.counted_bytes for counter in counters} == {4} - assert await _count(session, ArtifactPutAttempt) == 1 - assert await _count(session, ArtifactAdmissionCharge) == 4 - finally: - await engine.dispose() - - -async def test_capacity_failure_rolls_back_namespace_scopes_charges_and_attempt( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path, maximum_bytes=3) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - try: - async with factory() as session: - async with minted_source(tmp_path / "scratch-source", b"four") as source: - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - source.commitment.sha256, - source.commitment.byte_count, - source.commitment.media_type, - ), - ), - ) - with pytest.raises(ArtifactAdmissionCapacityError): - await ArtifactAdmissionService( - session, - settings, - namespace, - ).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=source, - ) - ) - - assert await _count(session, ArtifactStorageNamespace) == 0 - assert await _count(session, ArtifactAdmissionScope) == 0 - assert await _count(session, ArtifactAdmissionCharge) == 0 - assert await _count(session, ArtifactPutAttempt) == 0 - finally: - await engine.dispose() - - -async def test_changed_input_for_existing_operation_fails_closed( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - context = _context() - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - try: - async with factory() as session: - async with minted_source(tmp_path / "scratch-a", b"aaaa") as first_source: - _, _, item_ids = await _seed_contributor_items( - session, - context=context, - commitments=( - ( - first_source.commitment.sha256, - first_source.commitment.byte_count, - first_source.commitment.media_type, - ), - ), - ) - await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=first_source, - ) - ) - - async with minted_source(tmp_path / "scratch-b", b"bbbb") as changed_source: - item = await session.get(ArtifactUploadItem, item_ids[0]) - assert item is not None - item.expected_sha256 = changed_source.commitment.sha256 - item.expected_size = changed_source.commitment.byte_count - await session.commit() - with pytest.raises(ArtifactAdmissionConflictError): - await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_ids[0]), - source=changed_source, - ) - ) - finally: - await engine.dispose() - - -async def test_checker_output_requires_exact_active_fixed_service_identity( - admission_database_env: str, - tmp_path: Path, -) -> None: - settings = _settings(tmp_path) - namespace = _namespace(settings) - actor_id = uuid4() - link_id = uuid4() - context = _context( - actor_profile_id=actor_id, - identity_link_id=link_id, - actor_kind=ActorKind.SERVICE, - ) - engine = create_async_engine(admission_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - try: - async with factory() as session: - project_id, task_id, checker_run_id = await _seed_checker_output_relationships(session) - session.add( - ActorProfile( - id=str(actor_id), - actor_kind="service", - status="active", - provisioning_method="manual_service_provisioning", - service_identity=ServiceIdentity.ARTIFACT_CHECKER_OUTPUT.value, - created_by="test", - ) - ) - session.add( - ActorIdentityLink( - id=str(link_id), - actor_profile_id=str(actor_id), - issuer="https://issuer.example.test", - subject="checker-output-service", - subject_kind="service", - status="active", - linked_by="test", + ) + session.add( + ActorIdentityLink( + id=str(link_id), + actor_profile_id=str(actor_id), + issuer="https://issuer.example.test", + subject="checker-output-service", + subject_kind="service", + status="active", + linked_by="test", ) ) await session.commit() @@ -4344,6 +3029,229 @@ async def test_checker_output_requires_exact_active_fixed_service_identity( await engine.dispose() +@pytest.mark.parametrize( + ("read_result", "expected_outcome"), + ( + (None, "verified"), + (ArtifactObjectMissingError("missing"), "missing"), + (("sha256:" + "0" * 64, 1), "integrity_mismatch"), + ), +) +async def test_checker_output_shared_put_and_verification_lifecycle( + admission_database_env: str, + tmp_path: Path, + read_result: object, + expected_outcome: str, +) -> None: + """Task-scoped checker output survives every shared terminal byte outcome.""" + settings = _settings(tmp_path) + namespace = _namespace(settings) + assert settings.artifact_local_root is not None + bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) + store = bootstrap.initialize_after_namespace_claim( + ArtifactStoreNamespaceClaim( + adapter_identity=bootstrap.identity, + namespace_identity=bootstrap.namespace_identity, + namespace_fingerprint=namespace.namespace_fingerprint, + ) + ) + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + async with minted_source( + tmp_path / f"checker-{expected_outcome}", b"checker lifecycle" + ) as source: + project_id, task_id, checker_run_id, admission = await _admit_checker_output( + session, settings, namespace, source + ) + orchestrator = ArtifactStorageOrchestrator( + session, store, namespace, settings, _AllowArtifactAuthority() + ) + assert ( + await orchestrator.execute_committed_put( + attempt_id=admission.attempt_id, source=source + ) + == "stored_pending_verification" + ) + attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) + receipt = await session.scalar( + select(ArtifactOperationReceipt).where( + ArtifactOperationReceipt.put_attempt_id == str(admission.attempt_id) + ) + ) + job = await session.scalar( + select(ArtifactVerificationJob).where( + ArtifactVerificationJob.originating_put_attempt_id + == str(admission.attempt_id) + ) + ) + assert attempt is not None and receipt is not None and job is not None + job_id = UUID(job.id) + assert attempt.producer_request_type == "checker_output" + assert (attempt.project_id, attempt.task_id, attempt.checker_run_id) == ( + project_id, + task_id, + checker_run_id, + ) + assert receipt.checker_run_id == checker_run_id + with pytest.raises(DBAPIError): + await session.execute( + text( + "update artifact_operation_receipts set contract_version=1 " + "where id=:receipt_id" + ), + {"receipt_id": receipt.id}, + ) + await session.rollback() + if isinstance(read_result, Exception): + orchestrator._read_complete = AsyncMock(side_effect=read_result) + elif read_result is not None: + orchestrator._read_complete = AsyncMock(return_value=read_result) + assert await orchestrator.verify_object(job_id) == expected_outcome + job = await session.get(ArtifactVerificationJob, str(job_id)) + assert job is not None + assert job.status == expected_outcome + verification = await session.scalar( + select(ArtifactVerificationReceipt).where( + ArtifactVerificationReceipt.verification_job_id == job.id + ) + ) + assert verification is not None + assert verification.outcome == expected_outcome + finally: + bootstrap.close() + await engine.dispose() + + +@pytest.mark.parametrize( + "expected_outcome", ["integrity_mismatch", "conflict", "existing_replica_conflict"] +) +async def test_checker_output_put_observation_terminal_outcomes( + admission_database_env: str, + tmp_path: Path, + expected_outcome: str, +) -> None: + """Current checker output preserves ambiguous-put mismatch and conflict evidence.""" + settings = _settings(tmp_path) + namespace = _namespace(settings) + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + async with minted_source( + tmp_path / f"checker-observation-{expected_outcome}", b"checker observation" + ) as source: + _project_id, _task_id, checker_run_id, admission = await _admit_checker_output( + session, settings, namespace, source + ) + attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) + assert attempt is not None + provider_object_ref = attempt.canonical_target + if expected_outcome == "existing_replica_conflict": + content_id = str(uuid4()) + session.add_all( + [ + ArtifactContent( + id=content_id, + sha256=attempt.sha256, + byte_count=attempt.byte_count, + media_type=attempt.media_type, + normalized_display_name=None, + ), + ArtifactReplica( + id=str(uuid4()), + content_id=content_id, + storage_namespace_id=attempt.storage_namespace_id, + namespace_fingerprint=attempt.namespace_fingerprint, + adapter=namespace.adapter, + provider_profile=namespace.provider_profile, + provider_object_ref=provider_object_ref, + verification_state="verified", + availability_state="available", + integrity_state="valid", + ), + ] + ) + await session.commit() + else: + await session.rollback() + observe = ( + AsyncMock( + return_value=ArtifactPutObservation( + provider_object_ref, committed=True + ) + ) + if expected_outcome != "conflict" + else AsyncMock(side_effect=ArtifactStoreError("provider conflict")) + ) + provider = SimpleNamespace( + identity=SimpleNamespace(provider_key=namespace.adapter), + observe_put_result=observe, + ) + orchestrator = ArtifactStorageOrchestrator( + session, provider, namespace, settings, _AllowArtifactAuthority() + ) + orchestrator._read_complete = AsyncMock( + return_value=("sha256:" + "0" * 64, 1) + ) + + assert ( + await orchestrator.resolve_put_attempt(admission.attempt_id) + == ( + "conflict" + if expected_outcome == "existing_replica_conflict" + else expected_outcome + ) + ) + attempt = await session.get(ArtifactPutAttempt, str(admission.attempt_id)) + assert attempt is not None + receipt = await session.scalar( + select(ArtifactPutObservationReceipt).where( + ArtifactPutObservationReceipt.put_attempt_id == attempt.id + ) + ) + charges = ( + await session.execute( + select(ArtifactAdmissionCharge) + .join( + ArtifactPutAttemptCharge, + ArtifactPutAttemptCharge.charge_id == ArtifactAdmissionCharge.id, + ) + .where(ArtifactPutAttemptCharge.attempt_id == attempt.id) + ) + ).scalars().all() + assert receipt is not None + assert receipt.outcome == ( + "observed_integrity_mismatch" + if expected_outcome == "integrity_mismatch" + else "conflict" + ) + terminal_outcome = ( + "conflict" + if expected_outcome == "existing_replica_conflict" + else expected_outcome + ) + assert attempt.status == attempt.terminal_result_code == terminal_outcome + assert {charge.state for charge in charges} == {"completed"} + if expected_outcome == "integrity_mismatch": + assert attempt.replica_id is not None + assert receipt.observed_sha256 == "sha256:" + "0" * 64 + assert receipt.observed_byte_count == 1 + elif expected_outcome == "existing_replica_conflict": + assert attempt.replica_id is not None + assert receipt.observed_sha256 is None + assert receipt.observed_byte_count is None + else: + assert attempt.replica_id is None + assert receipt.observed_sha256 is None + assert receipt.observed_byte_count is None + assert receipt.execution_generation == attempt.execution_generation + assert checker_run_id == attempt.checker_run_id + finally: + await engine.dispose() + + async def test_invalid_checker_role_precedes_namespace_drift( admission_database_env: str, tmp_path: Path, diff --git a/backend/tests/test_artifact_architecture.py b/backend/tests/test_artifact_architecture.py index b107a9a58..320ba8975 100644 --- a/backend/tests/test_artifact_architecture.py +++ b/backend/tests/test_artifact_architecture.py @@ -6,6 +6,8 @@ from pathlib import Path from app.interfaces import artifact_operations +from app.db.base import Base +from app.main import create_app BACKEND_ROOT = Path(__file__).parents[1] @@ -69,9 +71,20 @@ "ArtifactAdmissionService", "ArtifactAdmissionResult", "CheckerOutputArtifactAdmissionRequest", - "ContributorArtifactAdmissionRequest", "GuideArtifactAdmissionRequest", } + +RETIRED_CONTRIBUTOR_INTAKE_NAMES = { + "ArtifactUploadItem", + "ArtifactUploadSession", + "ContributorArtifactAdmissionRequest", + "ContributorAdmissionFacts", + "get_contributor_admission_facts", + "get_receipt_for_item", + "lock_upload_item", + "lock_upload_session", + "_contributor_facts", +} PROVIDER_METHODS = {"put", "observe_put_result", "open", "head"} CONCRETE_ADAPTER_MODULES = { "app.adapters.artifacts.local", @@ -116,6 +129,37 @@ def _declared_annotation_names(tree: ast.AST) -> set[str]: return names +def test_retired_contributor_intake_has_no_runtime_declaration_or_reference() -> None: + runtime_files = _python_files(APP_ROOT) + for path in runtime_files: + tree = _tree(path) + names = {node.id for node in ast.walk(tree) if isinstance(node, ast.Name)} | { + node.attr for node in ast.walk(tree) if isinstance(node, ast.Attribute) + } + declared = { + node.name + for node in ast.walk(tree) + if isinstance(node, (ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef)) + } + assert not ((names | declared) & RETIRED_CONTRIBUTOR_INTAKE_NAMES), path + + assert "artifact_upload_sessions" not in Base.metadata.tables + assert "artifact_upload_items" not in Base.metadata.tables + + +def test_no_retired_or_replacement_contributor_intake_route_is_composed() -> None: + paths = set(create_app().openapi()["paths"]) + forbidden_fragments = ( + "artifact-upload", + "artifact_upload", + "upload-session", + "upload_session", + "submission-bundle", + "submission_bundle", + ) + assert not {path for path in paths if any(fragment in path for fragment in forbidden_fragments)} + + def test_product_api_and_workers_cannot_import_or_inject_raw_artifact_types() -> None: product_modules = [ path diff --git a/backend/tests/test_artifact_authorization.py b/backend/tests/test_artifact_authorization.py index 4750e00d1..291ce106f 100644 --- a/backend/tests/test_artifact_authorization.py +++ b/backend/tests/test_artifact_authorization.py @@ -189,7 +189,6 @@ async def test_quota_reconciliation_is_configuration_driven_and_rollback_safe() task_id=None, guide_source_item_id=str(uuid4()), guide_source_snapshot_id=None, - upload_item_id=None, checker_run_id=None, logical_role=None, operation_identity="sha256:" + "a" * 64, diff --git a/backend/tests/test_artifact_operator_api.py b/backend/tests/test_artifact_operator_api.py index c33d0fcc8..22d6cba68 100644 --- a/backend/tests/test_artifact_operator_api.py +++ b/backend/tests/test_artifact_operator_api.py @@ -131,9 +131,14 @@ async def test_real_http_operator_path_returns_redacted_lineage_and_recovery( async with factory() as session: context = _context() settings = _settings(tmp_path) - project_id, task_id, source_job, _orchestrator, bootstrap = await _exhausted_job( - session, settings, tmp_path, context - ) + ( + project_id, + task_id, + submission_id, + source_job, + _orchestrator, + bootstrap, + ) = await _exhausted_job(session, settings, tmp_path, context) assert {scope_type for scope_type, _band in artifact_admission_metrics.snapshot()} == { "deployment", "project", @@ -312,6 +317,7 @@ async def context_override(): json={ "project_id": project_id, "task_id": task_id, + "submission_id": submission_id, "reason": "authority race", "client_idempotency_key": "authority-race", "expected_source_job_cas_version": source_job_cas_version, @@ -331,6 +337,7 @@ async def context_override(): json={ "project_id": project_id, "task_id": task_id, + "submission_id": submission_id, "reason": "identity race", "client_idempotency_key": str(unavailable_context.actor_status), "expected_source_job_cas_version": source_job_cas_version, @@ -343,6 +350,7 @@ async def context_override(): json={ "project_id": project_id, "task_id": task_id, + "submission_id": submission_id, "reason": "stale source fence", "client_idempotency_key": "stale-source", "expected_source_job_cas_version": source_job_cas_version + 1, @@ -355,12 +363,13 @@ async def context_override(): json={ "project_id": project_id, "task_id": task_id, + "submission_id": submission_id, "reason": "provider remained unavailable", "client_idempotency_key": "operator-http-retry", "expected_source_job_cas_version": source_job_cas_version, }, ) - assert retry.status_code == 202 + assert retry.status_code == 202, retry.text recovery_id = retry.json()["recovery_attempt_id"] retry_job_id = retry.json()["retry_verification_job_id"] replay = await client.post( @@ -368,6 +377,7 @@ async def context_override(): json={ "project_id": project_id, "task_id": task_id, + "submission_id": submission_id, "reason": "provider remained unavailable", "client_idempotency_key": "operator-http-retry", "expected_source_job_cas_version": source_job_cas_version, @@ -381,6 +391,7 @@ async def context_override(): json={ "project_id": project_id, "task_id": task_id, + "submission_id": submission_id, "reason": "altered replay", "client_idempotency_key": "operator-http-retry", "expected_source_job_cas_version": source_job_cas_version, @@ -392,6 +403,7 @@ async def context_override(): json={ "project_id": project_id, "task_id": task_id, + "submission_id": submission_id, "reason": "pending retry is ineligible", "client_idempotency_key": "ineligible-retry", "expected_source_job_cas_version": 0, @@ -495,6 +507,7 @@ async def context_override(): json={ "project_id": project_id, "task_id": task_id, + "submission_id": submission_id, "reason": "cross-project probe", "client_idempotency_key": "cross-project-probe", "expected_source_job_cas_version": source_job_cas_version, diff --git a/backend/tests/test_artifact_recovery.py b/backend/tests/test_artifact_recovery.py index 2162e5476..ab79cbe08 100644 --- a/backend/tests/test_artifact_recovery.py +++ b/backend/tests/test_artifact_recovery.py @@ -3,7 +3,7 @@ from __future__ import annotations import asyncio -from datetime import UTC, datetime, timedelta +from datetime import UTC, datetime from pathlib import Path from unittest.mock import AsyncMock from uuid import UUID, uuid4 @@ -25,12 +25,9 @@ ArtifactStoreNamespaceClaim, ArtifactStoreUnavailableError, ) -from app.modules.actors.models import ActorIdentityLink, ActorProfile from app.modules.artifacts.models import ( ArtifactPutAttempt, ArtifactRecoveryAttempt, - ArtifactUploadItem, - ArtifactUploadSession, ArtifactVerificationJob, ) from app.modules.artifacts.schemas import ( @@ -39,7 +36,6 @@ ArtifactRecoveryIneligibleError, ArtifactAuthorityDeniedError, DenyArtifactRecoveryAuthority, - ContributorArtifactAdmissionRequest, GuideArtifactAdmissionRequest, ) from app.modules.artifacts.service import ( @@ -48,6 +44,8 @@ ArtifactStorageOrchestrator, artifact_storage_namespace_spec, ) +from app.modules.checkers.models import CheckerRun +from app.modules.actors.models import ActorIdentityLink, ActorProfile from app.modules.authorization.runtime import ( ActorKind, ActorStatus, @@ -59,12 +57,12 @@ from app.modules.projects.models import ( GuideSourceSnapshot, GuideSourceSnapshotItem, - Project, ProjectGuide, ) from project_create_fixtures import seed_historical_project, suspend_historical_product_custody -from app.modules.tasks.models import AuditEvent, WorkstreamTask +from app.modules.tasks.models import AuditEvent from tests.artifact_store_helpers import artifact_admission_limit_settings, minted_source +from tests.test_artifact_admission import _admit_checker_output class _AllowArtifactAuthority: @@ -152,9 +150,8 @@ def _context() -> HumanAuthorizationContext: ) -async def _seed_contributor(session, context, sha256: str, size: int) -> tuple[str, str, str]: +async def _seed_recovery_actor(session, context: HumanAuthorizationContext) -> None: actor_id = str(context.actor_profile_id) - project_id, task_id, upload_id, item_id = (str(uuid4()) for _ in range(4)) session.add( ActorProfile( id=actor_id, @@ -177,107 +174,19 @@ async def _seed_contributor(session, context, sha256: str, size: int) -> tuple[s last_verified_at=datetime.now(UTC), ) ) + + +async def _seed_guide_owner(session, context: HumanAuthorizationContext) -> str: + await _seed_recovery_actor(session, context) + project_id = str(uuid4()) await seed_historical_project( session, project_id=project_id, - name="Recovery project", - slug=f"recovery-{project_id}", - ) - await session.flush() - session.add( - WorkstreamTask( - id=task_id, - project_id=project_id, - title="Recovery task", - description="Prove recovery.", - status="draft", - created_by="test", - ) + name="Guide recovery project", + slug=f"guide-recovery-{project_id}", ) await session.flush() - session.add( - ArtifactUploadSession( - id=upload_id, - actor_id=actor_id, - project_id=project_id, - task_id=task_id, - permitted_roles=["submission"], - state="open", - maximum_bytes=size, - current_bytes=0, - reserved_bytes=size, - maximum_items=1, - current_items=0, - reserved_items=1, - expires_at=datetime.now(UTC) + timedelta(minutes=10), - cas_version=0, - ) - ) - await session.flush() - session.add( - ArtifactUploadItem( - id=item_id, - session_id=upload_id, - logical_role="submission", - display_name="result.bin", - media_type="application/octet-stream", - reserved_bytes=size, - expected_sha256=sha256, - expected_size=size, - idempotency_key=f"put-{item_id}", - request_digest=canonical_json_hash({"sha256": sha256, "size": size}), - state="reserved", - cas_version=0, - ) - ) - await session.commit() - return project_id, task_id, item_id - - -async def _exhausted_job(session, settings, tmp_path, context): - namespace = artifact_storage_namespace_spec( - settings, - LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)), - ) - bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) - store = bootstrap.initialize_after_namespace_claim( - ArtifactStoreNamespaceClaim( - adapter_identity=bootstrap.identity, - namespace_identity=bootstrap.namespace_identity, - namespace_fingerprint=namespace.namespace_fingerprint, - ) - ) - source_cm = minted_source(tmp_path / "source", b"recover me") - source = await source_cm.__aenter__() - project_id, task_id, item_id = await _seed_contributor( - session, context, source.commitment.sha256, source.commitment.byte_count - ) - admission = await ArtifactAdmissionService(session, settings, namespace).admit( - ContributorArtifactAdmissionRequest( - authorization_context=context, - upload_item_id=UUID(item_id), - source=source, - ) - ) - orchestrator = ArtifactStorageOrchestrator( - session, store, namespace, settings, _AllowArtifactAuthority() - ) - await orchestrator.execute_committed_put(attempt_id=admission.attempt_id, source=source) - job = await session.scalar(select(ArtifactVerificationJob)) - assert job is not None - job_id = UUID(job.id) - await session.rollback() - orchestrator._read_complete = AsyncMock( - side_effect=ArtifactStoreUnavailableError("unavailable") - ) - await orchestrator.verify_object(job_id) - job = await session.get(ArtifactVerificationJob, str(job_id)) - assert job is not None - await session.refresh(job) - await session.commit() - await source_cm.__aexit__(None, None, None) - assert job.terminal_result_code == "provider_unavailable" - return project_id, task_id, job, orchestrator, bootstrap + return project_id async def _exhausted_guide_job(session, settings, tmp_path, context): @@ -295,12 +204,7 @@ async def _exhausted_guide_job(session, settings, tmp_path, context): ) source_cm = minted_source(tmp_path / "guide-source", b"recover guide") source = await source_cm.__aenter__() - await _seed_contributor( - session, context, source.commitment.sha256, source.commitment.byte_count - ) - project = await session.scalar(select(Project)) - assert project is not None - project_id = project.id + project_id = await _seed_guide_owner(session, context) guide_id, snapshot_id, item_id = (str(uuid4()) for _ in range(3)) async with suspend_historical_product_custody( session, @@ -397,7 +301,52 @@ async def _exhausted_guide_job(session, settings, tmp_path, context): await session.refresh(job) await session.commit() await source_cm.__aexit__(None, None, None) - return project_id, job, bootstrap + return project_id, job, orchestrator, bootstrap + + +async def _exhausted_job(session, settings, tmp_path, context): + await _seed_recovery_actor(session, context) + namespace = artifact_storage_namespace_spec( + settings, + LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)), + ) + bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) + store = bootstrap.initialize_after_namespace_claim( + ArtifactStoreNamespaceClaim( + adapter_identity=bootstrap.identity, + namespace_identity=bootstrap.namespace_identity, + namespace_fingerprint=namespace.namespace_fingerprint, + ) + ) + async with minted_source(tmp_path / "checker-output", b"recover checker output") as source: + project_id, task_id, checker_run_id, admission = await _admit_checker_output( + session, settings, namespace, source + ) + orchestrator = ArtifactStorageOrchestrator( + session, store, namespace, settings, _AllowArtifactAuthority() + ) + put_outcome = await orchestrator.execute_committed_put( + attempt_id=admission.attempt_id, source=source + ) + assert put_outcome == "stored_pending_verification" + job = await session.scalar( + select(ArtifactVerificationJob).where( + ArtifactVerificationJob.originating_put_attempt_id == str(admission.attempt_id) + ) + ) + assert job is not None + job_id = job.id + await session.rollback() + orchestrator._read_complete = AsyncMock(side_effect=ArtifactStoreUnavailableError("down")) + await orchestrator.verify_object(UUID(job_id)) + job = await session.get(ArtifactVerificationJob, job_id) + assert job is not None + checker_run = await session.get(CheckerRun, checker_run_id) + assert checker_run is not None + submission_id = checker_run.submission_id + await session.refresh(job) + await session.commit() + return project_id, task_id, submission_id, job, orchestrator, bootstrap def _request( @@ -406,6 +355,7 @@ def _request( task_id: str | None, job: ArtifactVerificationJob, *, + submission_id: str | None = None, reason: str = "provider remained unavailable", client_idempotency_key: str = "recovery-1", ) -> ArtifactRecoveryRequest: @@ -413,7 +363,7 @@ def _request( authorization_context=context, project_id=UUID(project_id), task_id=UUID(task_id) if task_id is not None else None, - submission_id=None, + submission_id=UUID(submission_id) if submission_id is not None else None, source_verification_job_id=UUID(job.id), reason=reason, client_idempotency_key=client_idempotency_key, @@ -431,11 +381,16 @@ async def test_exact_replay_creates_one_recovery_job_and_audit( async with factory() as session: context = _context() settings = _settings(tmp_path) - project_id, task_id, source, _orchestrator, bootstrap = await _exhausted_job( - session, settings, tmp_path, context - ) + ( + project_id, + task_id, + submission_id, + source, + _orchestrator, + bootstrap, + ) = await _exhausted_job(session, settings, tmp_path, context) service = ArtifactRecoveryService(session, settings, _AllowRecoveryAuthority()) - request = _request(context, project_id, task_id, source) + request = _request(context, project_id, task_id, source, submission_id=submission_id) first = await service.create(request) replay = await service.create(request) assert first.retry_verification_job_id == replay.retry_verification_job_id @@ -465,7 +420,7 @@ async def test_taskless_recovery_and_deny_only_authority_boundary( async with factory() as session: context = _context() settings = _settings(tmp_path) - project_id, source, bootstrap = await _exhausted_guide_job( + project_id, source, _orchestrator, bootstrap = await _exhausted_guide_job( session, settings, tmp_path, context ) request = _request(context, project_id, None, source) @@ -514,11 +469,18 @@ async def test_changed_or_ineligible_recovery_has_no_side_effects( async with factory() as session: context = _context() settings = _settings(tmp_path) - project_id, task_id, source, _orchestrator, bootstrap = await _exhausted_job( - session, settings, tmp_path, context - ) + ( + project_id, + task_id, + submission_id, + source, + _orchestrator, + bootstrap, + ) = await _exhausted_job(session, settings, tmp_path, context) service = ArtifactRecoveryService(session, settings, _AllowRecoveryAuthority()) - created = await service.create(_request(context, project_id, task_id, source)) + created = await service.create( + _request(context, project_id, task_id, source, submission_id=submission_id) + ) with pytest.raises(ArtifactRecoveryConflictError): await service.create( _request( @@ -526,6 +488,7 @@ async def test_changed_or_ineligible_recovery_has_no_side_effects( project_id, task_id, source, + submission_id=submission_id, reason="changed", ) ) @@ -539,6 +502,7 @@ async def test_changed_or_ineligible_recovery_has_no_side_effects( project_id, task_id, source, + submission_id=submission_id, client_idempotency_key="different", ) ) @@ -553,6 +517,7 @@ async def test_changed_or_ineligible_recovery_has_no_side_effects( project_id, task_id, retry, + submission_id=submission_id, client_idempotency_key="retry-pending", ) await session.rollback() @@ -573,13 +538,24 @@ async def test_terminal_recovery_authority_change_rolls_back_all_facts( async with factory() as session: context = _context() settings = _settings(tmp_path) - project_id, task_id, source, _orchestrator, bootstrap = await _exhausted_job( - session, settings, tmp_path, context - ) + ( + project_id, + task_id, + submission_id, + source, + _orchestrator, + bootstrap, + ) = await _exhausted_job(session, settings, tmp_path, context) authority = _AllowThenDenyRecoveryAuthority() with pytest.raises(ArtifactAuthorityDeniedError): await ArtifactRecoveryService(session, settings, authority).create( - _request(context, project_id, task_id, source) + _request( + context, + project_id, + task_id, + source, + submission_id=submission_id, + ) ) assert authority.calls == 2 assert await session.scalar(select(func.count(ArtifactRecoveryAttempt.id))) == 0 @@ -607,12 +583,17 @@ async def test_retry_terminalizes_recovery_under_verification_fence( async with factory() as session: context = _context() settings = _settings(tmp_path) - project_id, task_id, source, orchestrator, bootstrap = await _exhausted_job( - session, settings, tmp_path, context - ) + ( + project_id, + task_id, + submission_id, + source, + orchestrator, + bootstrap, + ) = await _exhausted_job(session, settings, tmp_path, context) created = await ArtifactRecoveryService( session, settings, _AllowRecoveryAuthority() - ).create(_request(context, project_id, task_id, source)) + ).create(_request(context, project_id, task_id, source, submission_id=submission_id)) orchestrator._read_complete = ArtifactStorageOrchestrator._read_complete.__get__( orchestrator ) @@ -645,12 +626,17 @@ async def test_terminal_authority_drift_writes_no_recovery_terminal_facts( async with factory() as session: context = _context() settings = _settings(tmp_path) - project_id, task_id, source, orchestrator, bootstrap = await _exhausted_job( - session, settings, tmp_path, context - ) + ( + project_id, + task_id, + submission_id, + source, + orchestrator, + bootstrap, + ) = await _exhausted_job(session, settings, tmp_path, context) created = await ArtifactRecoveryService( session, settings, _AllowRecoveryAuthority() - ).create(_request(context, project_id, task_id, source)) + ).create(_request(context, project_id, task_id, source, submission_id=submission_id)) orchestrator._authority = _DenyTerminalArtifactAuthority() orchestrator._read_complete = ArtifactStorageOrchestrator._read_complete.__get__( orchestrator @@ -700,12 +686,17 @@ async def test_every_failed_retry_outcome_terminalizes_recovery_once( async with factory() as session: context = _context() settings = _settings(tmp_path) - project_id, task_id, source, orchestrator, bootstrap = await _exhausted_job( - session, settings, tmp_path, context - ) + ( + project_id, + task_id, + submission_id, + source, + orchestrator, + bootstrap, + ) = await _exhausted_job(session, settings, tmp_path, context) created = await ArtifactRecoveryService( session, settings, _AllowRecoveryAuthority() - ).create(_request(context, project_id, task_id, source)) + ).create(_request(context, project_id, task_id, source, submission_id=submission_id)) if isinstance(provider_result, Exception): orchestrator._read_complete = AsyncMock(side_effect=provider_result) else: @@ -744,10 +735,15 @@ async def test_concurrent_exact_replay_has_one_envelope_and_retry_job( async with factory() as setup: context = _context() settings = _settings(tmp_path) - project_id, task_id, source, _orchestrator, bootstrap = await _exhausted_job( - setup, settings, tmp_path, context - ) - request = _request(context, project_id, task_id, source) + ( + project_id, + task_id, + submission_id, + source, + _orchestrator, + bootstrap, + ) = await _exhausted_job(setup, settings, tmp_path, context) + request = _request(context, project_id, task_id, source, submission_id=submission_id) async with factory() as first_session, factory() as second_session: first, second = await asyncio.gather( ArtifactRecoveryService(first_session, settings, _AllowRecoveryAuthority()).create( @@ -778,11 +774,18 @@ async def test_exhausted_retry_can_form_only_the_next_linear_chain_link( async with factory() as session: context = _context() settings = _settings(tmp_path) - project_id, task_id, source, orchestrator, bootstrap = await _exhausted_job( - session, settings, tmp_path, context - ) + ( + project_id, + task_id, + submission_id, + source, + orchestrator, + bootstrap, + ) = await _exhausted_job(session, settings, tmp_path, context) service = ArtifactRecoveryService(session, settings, _AllowRecoveryAuthority()) - first = await service.create(_request(context, project_id, task_id, source)) + first = await service.create( + _request(context, project_id, task_id, source, submission_id=submission_id) + ) orchestrator._read_complete = AsyncMock( side_effect=ArtifactStoreUnavailableError("still unavailable") ) @@ -802,6 +805,7 @@ async def test_exhausted_retry_can_form_only_the_next_linear_chain_link( project_id, task_id, retry, + submission_id=submission_id, client_idempotency_key="recovery-2", ) first_attempt_id = first_attempt.id diff --git a/backend/tests/test_artifacts.py b/backend/tests/test_artifacts.py index 67ccfcaee..4aa330bc0 100644 --- a/backend/tests/test_artifacts.py +++ b/backend/tests/test_artifacts.py @@ -105,4 +105,5 @@ def test_models_retain_only_v2_provider_evidence_fields() -> None: assert "retention_state" not in ArtifactReplica.__table__.columns assert "provider_receipt_id" not in ArtifactOperationReceipt.__table__.columns assert "retention_reference" not in ArtifactOperationReceipt.__table__.columns + assert ArtifactOperationReceipt.__table__.c.put_attempt_id.nullable is False assert "namespace_fingerprint" in ArtifactStorageNamespace.__table__.columns diff --git a/backend/tests/test_review_queue_persistence.py b/backend/tests/test_review_queue_persistence.py index a52a3d8c2..8fa789480 100644 --- a/backend/tests/test_review_queue_persistence.py +++ b/backend/tests/test_review_queue_persistence.py @@ -587,7 +587,7 @@ def downgrade() -> None: async with db_session.get_session_factory()() as session: assert await session.scalar(text("select version_num from alembic_version")) == ( - "0051_review_queue_foundation" + "0052_legacy_intake_removal" ) assert await session.scalar( select(ReviewAdmissionIdempotencyRecord.id).where( @@ -623,6 +623,6 @@ def downgrade() -> None: async with db_session.get_session_factory()() as session: assert await session.scalar(text("select version_num from alembic_version")) == ( - "0051_review_queue_foundation" + "0052_legacy_intake_removal" ) assert await session.get(ReviewQueueEntry, queue_value.id) is not None diff --git a/docs/decision_0013_immutable_artifact_storage_boundary.md b/docs/decision_0013_immutable_artifact_storage_boundary.md index f62dd068c..a93f00747 100644 --- a/docs/decision_0013_immutable_artifact_storage_boundary.md +++ b/docs/decision_0013_immutable_artifact_storage_boundary.md @@ -134,9 +134,9 @@ uploads. ## Independent Verification -A provider acknowledgement sets the upload item to -`stored_pending_verification` and creates a pending replica, not a bindable -artifact. A durable Celery job performs a fresh complete-object read, +A provider acknowledgement sets the durable put attempt to +`object_confirmed` and creates a pending replica, not a bindable artifact. A +durable Celery job performs a fresh complete-object read, computes SHA-256 and byte count, and records an immutable verification receipt. Only a matching replica becomes bindable. @@ -200,9 +200,10 @@ or human ownership. ## Failure Meaning Storage conditions do not introduce task or review decisions. Stable failures -include `artifact_storage_unavailable`, `artifact_input_mismatch`, -`artifact_upload_expired`, `artifact_upload_consumed`, and -`artifact_integrity_failure`. +include `artifact_storage_unavailable`, `artifact_input_mismatch`, and +`artifact_integrity_failure`. The retired upload-session failures +`artifact_upload_expired` and `artifact_upload_consumed` are historical only and +are not current runtime outcomes. Transient post-submit retrieval failure keeps the task in `evaluation_pending`. Integrity mismatch quarantines the replica and blocks diff --git a/docs/glossary.md b/docs/glossary.md index 42611acdb..1bc0141dd 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -102,12 +102,6 @@ immutable Submission and binding, while proven task/predecessor/locked-context drift may make it stale. No state expires, releases storage capacity, or authorizes deletion in v0.1. -## ArtifactUploadItem - -Legacy unavailable multi-item staging metadata. ART-04A must remove it or make -it statically unreachable before submission-bundle preparation activates. It is -not the v0.1 contributor intake, `ArtifactContent`, or an `ArtifactBinding`. - ## ArtifactBinding The immutable logical association between `ArtifactContent` and one exact diff --git a/docs/operations_artifact_storage.md b/docs/operations_artifact_storage.md index 401f14c98..7c8c4b668 100644 --- a/docs/operations_artifact_storage.md +++ b/docs/operations_artifact_storage.md @@ -84,3 +84,16 @@ charges or directly editing database state. These routes do not delete, retain, release, or mutate provider objects; change admission configuration; activate AWS; or cut over any guide, task, submission, checker, review, contribution, payment, or reputation lifecycle. + +## Legacy contributor-intake migration + +Migration `0051_legacy_intake_removal` is a safe-empty cut. Before deploying, +confirm that the legacy upload-session/item tables contain no rows and that no +put attempt or operation receipt carries contributor/upload-item lineage or a +version-1 receipt contract. The migration takes exclusive locks and refuses +before changing schema when any such evidence exists. + +Do not delete, detach, or rewrite those rows to force deployment. Preserve the +database at revision `0050_guide_source_v2` and escalate for a separately +approved maintenance and audit migration. A refusal is an expected evidence- +preservation outcome, not permission to bypass the preflight. diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 469ef434a..cf5e0ac80 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -63,11 +63,10 @@ mandatory platform/locked-guide checks. It is not candidate storage, carries no provider reference, cannot cross a process boundary, and is discarded on failure or process loss. -Only a passing preparation enters generic durable admission. Existing -`ArtifactUploadSession`/`ArtifactUploadItem` rows and planned actions are legacy -unavailable staging input and must be removed or made statically unreachable by -ART-04A plus the separately reviewed AUTH registration handoff before the new -surface activates. +Only a passing preparation enters generic durable admission. The retired +`ArtifactUploadSession`/`ArtifactUploadItem` staging path is absent from the +current runtime and schema. Submission-bundle preparation enters generic +durable admission directly after its bounded checks pass. ### ArtifactStorageAdmissionLedger @@ -140,8 +139,7 @@ A bounded PostgreSQL scanner publishes `prepared`, uses read-only `observe_put_result`, then opens and hashes any observed object. Matching bytes complete the original charges and Transaction B facts exactly once. Authoritative absence releases charges and moves the producer attempt to -`absent_replay_required`; a legacy contributor upload item, while it exists, -moves to `replay_required`. Mismatched bytes remain charged, quarantine the key, and +`absent_replay_required`. Mismatched bytes remain charged, quarantine the key, and create an incident. Terminal writes require matching executor and generation. No background resolver performs another provider write. @@ -172,9 +170,8 @@ availability_state = unknown | available | unavailable integrity_state = unknown | valid | invalid ``` -Transaction B sets the `ArtifactPutAttempt` to `object_confirmed`, sets a legacy -contributor upload item, while it exists, to `stored_pending_verification`, and -creates the replica as `pending/unknown/unknown`. A matching complete read makes the +Transaction B sets the `ArtifactPutAttempt` to `object_confirmed` and creates +the replica as `pending/unknown/unknown`. A matching complete read makes the admission bindable and replica `verified/available/valid`. A provider-unavailable or conflict job result does not fabricate a replica observation. @@ -182,8 +179,7 @@ A confirmed absent object sets the replica to `missing/unavailable/unknown`. Before any binding exists, and only while the original producer attempt remains eligible, an exact replay by the original authorized producer may move that same replica to `pending/unknown/unknown`, -set the attempt back to `object_confirmed`, set a legacy contributor upload item, -while it exists, to `stored_pending_verification`, append a new operation receipt, +set the attempt back to `object_confirmed`, append a new operation receipt, and create a new verification job. After a binding exists, the missing replica is terminal and cannot return to pending in v0.1. A digest/size mismatch sets the replica to `integrity_mismatch/available/invalid`, fails the pre-binding @@ -762,8 +758,7 @@ in the v2 clean cut. No compatibility adapter or dual format remains. stores or resolves an exact replay candidate. 6. Transaction B validates the reservation/CAS, completes every provisional charge, records content, replica, and operation receipt, sets the - `ArtifactPutAttempt` to `object_confirmed`, sets a legacy contributor upload item, while it exists, - to `stored_pending_verification`, and sets the replica to + `ArtifactPutAttempt` to `object_confirmed`, and sets the replica to `pending/unknown/unknown`. 7. The transaction creates an outbox/publication obligation for one verification job. @@ -786,8 +781,7 @@ The attempt scanner publishes resolution after an ambiguous outcome or expired execution lease. Resolution performs a fresh read-only observation and full hash. A confirmed object completes the same charges and Transaction B facts. Fresh authoritative absence releases them and moves the `ArtifactPutAttempt` to -`absent_replay_required`; while a legacy contributor upload item exists, it -alone moves to `replay_required`. Replay must atomically reacquire every +`absent_replay_required`. Replay must atomically reacquire every applicable charge before another provider call. Integrity-mismatched or quarantined existing bytes remain completed and charged.