diff --git a/CLAUDE.md b/CLAUDE.md index 6b00f2908..3cbfc6d96 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,7 +5,7 @@ - **extralit-frontend/**: Vue 3 / Nuxt 4 (Vite); Pinia state management - **extralit/**: Python SDK client - **extralit-hf-space/**: Self-contained HF Spaces deployment bundle (Docker; bundles Elasticsearch + Redis + OCR) — git submodule -- **Vector DB**: Elasticsearch/OpenSearch (separate service) +- **Vector DB**: Elasticsearch/OpenSearch (migrating to Lancedb) ### Key Patterns - Backend: SQLAlchemy ORM, Alembic migrations, async pytest @@ -41,44 +41,7 @@ cd extralit-server && uv run alembic -c src/extralit_server/alembic.ini upgrade docker-compose up -d ``` -## Development Workflow - -# Python Package Management with uv -Use uv exclusively for Python package management in this project. - -## Package Management Commands -- All Python dependencies **must be installed, synchronized, and locked** using uv -- Never use pip, pip-tools, poetry, or conda directly for dependency management - -Use these commands: -- Install dependencies: `uv add `, avoid adding to pyproject.toml directly -- Remove dependencies: `uv remove ` -- Sync environment: `uv sync` -- Lock dependencies: `uv lock` - -## Running Python Code -- Run a Python script with `uv run .py` -- Run Python tools with `uv run ` (e.g. `uv run pytest`, `uv run ruff check`, `uv run ruff format`, `uv run ty check`, `uv run pre-commit`, `npm run lint`, `npm run format`) -- Launch a Python REPL with `uv run python` - -### Running Services -```bash -cd extralit-server && uv run python -m extralit_server server-dev # Server + worker -cd extralit-frontend && npm run dev # Frontend -``` - -### Testing -```bash -cd extralit-server && uv run pytest tests # Server tests -cd extralit && uv run pytest tests # SDK tests -cd extralit-frontend && npm run test # Frontend tests -``` - -## Branching - -Trunk-based. **`main` is the trunk and the default branch** — every change, code or docs, -branches from `main` and squash-merges back via PR. There is no `develop` branch and no -`releases/**` branches. +## Deployment and Branching | Ref | Role | Deploys to | |---|---|---| @@ -87,11 +50,6 @@ branches from `main` and squash-merges back via PR. There is no `develop` branch | `vX.Y.Z` tag | the release itself | PyPI, versioned docs, GitHub Release | | PR (non-fork) | preview | ephemeral `extralit-dev/pr-N` | -Branch names: `feat/*`, `fix/*`, `docs/*`, short-lived. PR titles use the matching -`feat:` / `fix:` / `docs:` / `chore:` prefix. Note the generated release notes group PRs by -**label**, not by title prefix (see `.github/release.yml`) — label a PR to place it under a -heading; the title convention is for humans. - **Never push to `release` or create tags by hand.** Releases are one dispatch: ```bash @@ -103,10 +61,4 @@ That stamps the version via `scripts/bump_version.py` and pushes `main`, `releas tag atomically. The version lives in three files — always change it with `python scripts/bump_version.py set --version X.Y.Z`, never by hand. -See `docs/architecture/deployment.md` for the full pipeline and -`extralit/docs/community/release_guide.md` for the release runbook. - -## Git Workgrees -When creating a git workgree, place it at `.worktree/` relative to the repo root, normalizing `/` to `-` in the branch-name. - -Example: branch `feature/add-new-feature` should be placed at `.worktree/feature-add-new-feature`. +See `docs/architecture/deployment.md` for the full pipeline diff --git a/docs/architecture/deployment.md b/docs/architecture/deployment.md index a01e23656..2cb346cf5 100644 --- a/docs/architecture/deployment.md +++ b/docs/architecture/deployment.md @@ -81,8 +81,8 @@ flowchart TD srvbuild -->|build & push| srvimg srvbuild -->|repository_dispatch: build-hf-space| bhs bhs -->|FROM server image + ES/Redis/OCR| spaceimg - bhs -->|"restart (is_release=true)"| demo - bhs -->|"restart (branch=main)"| dev + bhs -->|"commit digest pin (is_release=true)"| demo + bhs -->|"commit digest pin (branch=main)"| dev bhs -->|duplicate + retarget| prspace ``` @@ -112,7 +112,7 @@ on **`pull_request`**. | `extralit/docs/` | `extralit.docs.yml` | `extralit/docs/**`, `mkdocs.yml` | Versioned docs via `mike` → `gh-pages` | No | | *(whole repo)* | `release.yml` | manual dispatch only | version stamp on `main` + `release` + `vX.Y.Z` tag | **Yes** (indirectly) | | *(none)* | `github-release.yml` | `vX.Y.Z` tag | GitHub Release, once PyPI serves both packages | No³ | -| `extralit-hf-space/`| `build-hf-space.yml` *(other repo)* | repository_dispatch / manual | `extralit-hf-space` Docker image + Space restart/deploy | **Yes** (terminal) | +| `extralit-hf-space/`| `build-hf-space.yml` *(other repo)* | repository_dispatch / manual | `extralit-hf-space` Docker image + Space deploy | **Yes** (terminal) | ¹ The frontend's own workflow tests, lints, and uploads a prerendered SPA artifact; the **live** UI is deployed separately by Vercel's native Git @@ -289,8 +289,13 @@ Remove it once the trunk flip has been verified. `env_name` selects the GitHub **Environment** (`production` vs `staging`), which is how per-environment secrets/vars are scoped: `DOCKER_REPO`, -`EXTRALIT_SERVER_IMAGE`, `HF_SPACE_ID`, `HF_TOKEN`, -`DOCKER_USERNAME`/`DOCKER_PASSWORD`. +`EXTRALIT_SERVER_IMAGE`, `HF_SPACE_ID`, +`DOCKER_USERNAME`/`DOCKER_PASSWORD` (plus `HF_TOKEN` on `staging`, which only +`deploy-pr-space` still uses — see §4). + +Note the environments carry **no protection rules and no branch policy**, so +`environment:` here is a scoping mechanism, not an approval gate. What actually +confines production access is the per-job `permissions:` block (§3). ### Job `build` Builds the self-contained Space image **on top of the server image**: @@ -309,13 +314,56 @@ multi-process runtime (elastic + redis + RQ workers + FastAPI). Pushed to (staging), tagging `:latest` when `tag_latest=true`. ### Job `deploy-space` — *non-PR builds only* (`pr_space_slug == ''`) -Restarts the live Space so it pulls the freshly pushed image: +Deploys by **committing to the Space repo**, not by restarting it. The Space is a +thin `FROM ` Dockerfile; the job rewrites that `FROM` line to the +**digest** `build` just pushed and commits it, and HF rebuilds on the new commit. -```bash -curl -X POST "https://huggingface.co/api/spaces/${HF_SPACE_ID}/restart" \ - -H "Authorization: Bearer $HF_TOKEN" +```yaml +permissions: + contents: read + id-token: write # mint the OIDC token; see below +env: + HF_OIDC_RESOURCE: spaces/${{ vars.HF_SPACE_ID }} + IMAGE_DIGEST: ${{ needs.build.outputs.image_digest }} +``` +```python +after = re.sub(r"^FROM\s+\S+", f"FROM {repo}@{digest}", before, count=1, flags=re.M) +api.upload_file(path_or_fileobj=after.encode(), path_in_repo="Dockerfile", + repo_id=space, repo_type="space", commit_message=f"Deploy {pin}") ``` +**Why a commit and not `restart_space()`.** The restart API answers **401** to an +OIDC token: a repo publisher grants *write access to that repo*, and restarting is +a runtime operation rather than a repo write. Committing is what the credential is +for, and is HF's own +[documented GitHub Actions pattern](https://huggingface.co/docs/hub/en/spaces-github-actions). +A 401 here is specifically the restart endpoint — a misconfigured publisher fails +earlier and differently, as `OIDCError`/`invalid_grant` from the exchange. + +**Why a digest and not a tag.** When the `FROM` line was `:latest`, HF reused the +base image it had already built and never re-pulled — the job went green while the +Space served the old build. That is the v0.7.0 failure: it cycled +`RUNNING_APP_STARTING → RUNNING` and still reported 0.6.1. A digest cannot resolve +to a previously-built image, which retires the `factory_reboot=True` workaround +that used to paper over this. + +Re-deploying an unchanged digest is a no-op commit and so triggers no rebuild; that +path skips the wait and asserts the Space's *current* stage is `RUNNING`, so an +earlier failed build is never reported as a green redeploy. + +**This job holds no HF credential.** It authenticates with +[Trusted Publishers](https://huggingface.co/docs/hub/en/trusted-publishers): +GitHub Actions mints a short-lived OIDC id token, and HF exchanges it (RFC 8693) +for a token scoped to that one Space for ~1h. `huggingface_hub` does the whole +dance inside `get_token()` when `HF_OIDC_RESOURCE` is set, and raises `OIDCError` +rather than falling back to an ambient credential. + +Each Space registers a publisher pinned to repo `Extralit/extralit-hf-space`, +branch `main`, workflow `build-hf-space.yml`. Those claims are satisfied by +*every* job in this file, so the `id-token: write` grant is deliberately scoped to +this job alone — `build` and `deploy-pr-space` inherit only the workflow-level +`contents: read` and therefore cannot mint a token to exchange at all. + `HF_SPACE_ID` is the environment-scoped Space (see §5): the **`production`** environment points at `extralit/public-demo` — the live public demo served at **** — while the **`staging`** @@ -382,10 +430,26 @@ resolve **per environment**. | Secret | Repo-level | `production` env | `staging` env | | ----------------- | :--------: | :--------------: | :-----------: | -| `HF_TOKEN` | ✅ (default) | ✅ (override) | ✅ (override) | +| `HF_TOKEN` | — | — | ✅ | | `DOCKER_USERNAME` | — | ✅ | ✅ | | `DOCKER_PASSWORD` | — | ✅ | ✅ | +`HF_TOKEN` survives **only** on `staging`, and only for `deploy-pr-space`. +Trusted Publishers scope a token to an *existing* repo, so they cannot cover +`duplicate_space()`, which creates `extralit-dev/pr-N` on demand. That leaves the +remaining token's write access confined to the `extralit-dev` org — nothing can +reach `extralit/public-demo` with a stored credential. + +> Deleting the repo-level `HF_TOKEN` is part of this, not an afterthought: +> `secrets.HF_TOKEN` silently falls back to it, so leaving it in place would make +> removing the `production` override purely cosmetic. +> +> **Order matters.** This table is the state *after* a production release has +> deployed keyless end-to-end. Until then both overrides stay, so reverting +> `deploy-space` to a stored token remains a one-commit rollback — deleting them +> early costs nothing on the deploy path (nothing reads `secrets.HF_TOKEN` there +> anymore) but throws away that escape hatch. + ### `extralit/extralit` (monorepo) - **Repo variables:** none. @@ -442,7 +506,7 @@ The cross-repo `client-payload` carries the handoff state: | `image_tag` *(output)* | payload tag, or `latest` (manual) | Docker tag built & deployed | | `tag_latest` *(output)* | `true` on release/trunk | also tag/push `:latest` | | `platforms` *(output)* | `amd64` (staging) / `amd64,arm64` (release) | buildx target platforms | -| `pr_space_slug` *(output)* | `pr-N` / slug for preview refs | empty → restart live Space; set → PR preview | +| `pr_space_slug` *(output)* | `pr-N` / slug for preview refs | empty → deploy live Space; set → PR preview | | `DOCKER_TAGS` | `${DOCKER_REPO}:${IMAGE_TAG}[,:latest]` | tags pushed by build job | | `EXTRALIT_SERVER_IMAGE` *(build-arg)* | `vars.EXTRALIT_SERVER_IMAGE` | base image the Space is built `FROM` | | `EXTRALIT_VERSION` *(build-arg)* | `image_tag` | base image tag (→ Dockerfile `ARG`) | @@ -508,7 +572,7 @@ The cross-repo `client-payload` carries the handoff state: 4. `repository_dispatch(build-hf-space, {tag: main, is_release: false, branch: main})`. 5. `build-hf-space.yml` → `resolve-env` (env=`staging`) → builds `extralitdev/extralit-hf-space:main` `FROM` the server image. -6. `deploy-space` → `POST /spaces/extralit-dev/develop/restart`. +6. `deploy-space` → commits the digest pin to `extralit-dev/develop`, which rebuilds. 7. Live at ****. **Production untouched.** ### Release → public demo (`release` + tag) @@ -521,8 +585,8 @@ The cross-repo `client-payload` carries the handoff state: (`is_release=true`) → `extralit/extralit-server:vX.Y.Z` (+`:latest`), amd64+arm64. 4. `repository_dispatch(build-hf-space, {tag: vX.Y.Z, is_release: true})`. 5. `build-hf-space.yml` → `resolve-env` (env=`production`) → builds - `extralit/extralit-hf-space:vX.Y.Z` → `deploy-space` restarts - `extralit/public-demo`. + `extralit/extralit-hf-space:vX.Y.Z` → `deploy-space` commits that image's + digest to `extralit/public-demo`, which rebuilds. 6. In parallel, the **tag** push drives PyPI (`extralit`, `extralit-server`), versioned docs (`mike deploy X.Y` + `stable`), and the GitHub Release. 7. Public demo live at ****.