Skip to content

feat: implement object storage upload foundation#77

Merged
kiaquila merged 10 commits into
mainfrom
codex/object-storage-foundation
Jul 12, 2026
Merged

feat: implement object storage upload foundation#77
kiaquila merged 10 commits into
mainfrom
codex/object-storage-foundation

Conversation

@kiaquila

@kiaquila kiaquila commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Summary

  • add strict OBJECT_STORAGE_* configuration and an AWS SDK v2 S3-compatible private-storage adapter with bounded retries/timeouts, readiness, signed PUT/GET, metadata, and delete primitives
  • add owner-bound photo upload init/complete routes, durable upload_jobs / unattached original item_assets, strict validation, and concurrency-safe idempotent completion
  • add Object Storage to health while caching only the serialized external probe; keep Postgres/Kratos fresh and prevent canceled public requests from poisoning the shared result
  • add OpenAPI/generated client, Compose/env wiring, canonical Hetzner policy/CORS artifacts, and synchronized spec/runtime documentation
  • provision asset buckets in project 15203114 / HEL and the Object-Locked backup bucket in project 15296835 / FSN; move runtime and backup credentials to bucketless key-only projects 15302873 and 15302925
  • apply and live-test cross-project runtime/public/backup policies, rotate the protected server env, revoke the old same-project keys and both temporary policy operators, and pass a post-revocation signed 10 MiB PUT/HEAD/GET/checksum/delete smoke
  • keep production upload routes fail-closed with OBJECT_STORAGE_UPLOADS_ENABLED=false
  • address both Codex P2 findings test-first: make storage-smoke independent of DATABASE_URL, then isolate container liveness on dependency-free /livez while /api/health remains fail-closed release readiness

Scope Context

  • Scope id or feature name: 040-object-storage-upload-foundation
  • Durable docs updated: yes

Reuse check: extended config.Load, auth.RequireSession/auth.UserID, httpx, the health handler, pgx migration/repository patterns, and existing OpenAPI upload routes; new storage/uploads packages were required because no Go storage/upload bounded context existed.

Module-size note: the composition root and upload handlers exceed the ~60-line soft signal only as linear wiring/request pipelines; provider and persistence logic remain split into focused packages and every new Go file remains below the ~500-line review signal.

SENAR Done Gate

  • Feature memory names the goal and scope (spec.md Goal + Scope sections).
  • Every acceptance criterion has final-head evidence. Rows 1–13 are complete; head 789405ba96 has green baseline-checks / guard / test / AI Review / osv-scan and an exact-head Codex no-major-issues summary. Row 14 waits only for human-authorized resolution of the two addressed review threads and the residual-risk gate below.
  • At least one negative scenario is covered.
  • tasks.md records dead ends, decisions, and known issues under ## Process Memory.
  • TDD evidence (specs ≥ 025): failing-test commits precede implementation: 7f25927, 85b7eaa, and f7577c2; implementation begins in a4dd84c. Review fixes also remain test-first: 8933ce4a1a895b, and 47f4fb5789405b.
  • Remaining provider residual accepted by the human merge owner: Hetzner/RGW accepts Object Lock mode/retain-until/legal-hold headers carried by an allowed backup PutObject. The writer still cannot read/delete/list existing data or directly mutate ACL/Object Lock state, but a compromised writer could create retained/held versions and amplify storage cost. Backup automation remains disabled and gated on header sanitization plus explicit acceptance or a provider fix.

Validation

  • exact head 789405ba96: green baseline-checks, guard, test, AI Review, and osv-scan; Codex reported no major issues
  • container liveness regression: red 47f4fb5 → green 789405b; /livez is dependency-free and release readiness stays on /api/health
  • cd api && go vet ./... && go test ./...
  • cd api && go test -race ./internal/storage ./internal/uploads ./internal/httpx ./cmd/api
  • fresh postgres:16: go test -tags=integration -race -count=1 -run TestRepoCompleteIsConcurrentAndIdempotent ./internal/uploads
  • npm run check:repo && npm run check:feature-memory && npm run check:api-contract
  • npm --prefix app run typecheck
  • production and dev docker compose ... config --quiet
  • frontend lint/stylelint: 0 errors; pre-existing soft-gate warnings only
  • Next.js production build
  • redacted live policy audit: runtime private data plane, public s3:* deny, backup normal Put plus read/delete/list/control/ACL denies and dangerous ACL header guards
  • redacted live CORS audit: exact production origin allowed for intended asset flows; attacker origins denied; backup CORS absent
  • redacted post-revocation smoke: readiness, signed 10 MiB PUT, HEAD metadata, signed GET checksum match, delete and cleanup

Security Notes

  • Runtime access is limited to private-bucket readiness and item-originals/* / smoke/spec-040/* object actions; the public bucket explicitly denies that principal s3:*.
  • Backup access is a hybrid write boundary: normal postgres/* puts work; object/version reads, ACL/direct retention/legal-hold controls, deletes, governance bypass, listing, and bucket control reads were denied in live probes. Dangerous canned ACL and AllUsers grant-read puts were also denied.
  • The Object Lock Put-header exception above is documented without overstating strict write-only isolation. It does not change upload activation: uploads still wait for owner quota, orphan cleanup, and wardrobe attachment.
  • No credential or presigned URL is stored in the repository, PR text, or captured evidence.

Follow-up Work

  • Upload activation: owner quota, orphan cleanup, wardrobe attachment, then frontend/mobile wiring.
  • Backup activation: age-encrypted pg_dump, scheduling, retention, restore drill, Object Lock header rejection, and explicit residual acceptance/provider fix.
  • Later slices: public catalog application behavior, image processing, and imgproxy.

Co-authored-by: Codex codex@openai.com

kiaquila and others added 5 commits July 10, 2026 19:09
Add the feature memory and compile-red tests for strict storage config, presigning, readiness, upload validation, fail-closed completion, and completion idempotency.

Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Add the strict Hetzner S3 adapter, storage readiness, owner-bound signed photo upload lifecycle, durable migrations, production topology artifacts, and default-off activation gates.

Co-authored-by: OpenAI Codex <codex@openai.com>
Record dedicated cross-project credentials, live policy/CORS evidence, and the backup Object Lock residual.

Co-authored-by: OpenAI Codex <codex@openai.com>
@kiaquila kiaquila marked this pull request as ready for review July 12, 2026 00:33

Copy link
Copy Markdown
Owner Author

@codex review

Please review the current PR head 0d32510e14fe79a6804d1874329414a4437ad3d6, including the cross-project Object Storage policy hardening and documented Hetzner Object Lock Put-header residual.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0d32510e14

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread api/cmd/storage-smoke/main.go Outdated
kiaquila and others added 3 commits July 11, 2026 21:57
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>

Copy link
Copy Markdown
Owner Author

@codex review

Please review current head 16c5065e9b78878ac4be12369fc8debc1f8eac6c. The previous P2 is addressed test-first: cmd/storage-smoke now uses the reusable Object-Storage-only loader, and the regression test proves it does not require DATABASE_URL. This head also merges current main (#78). Upload routes remain fail-closed and production-disabled. The documented Hetzner backup PutObject-header residual remains an explicit future backup-activation gate; backup automation is not enabled by this PR.

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 16c5065e9b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread api/cmd/api/main.go
kiaquila and others added 2 commits July 11, 2026 22:20
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 789405ba96

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@kiaquila kiaquila merged commit 903fbad into main Jul 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant