Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 3 additions & 51 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
- **extralit-frontend/**: Vue 3 / Nuxt 4 (Vite); Pinia state management
- **extralit/**: Python SDK client
- **extralit-hf-space/**: Self-contained HF Spaces deployment bundle (Docker; bundles Elasticsearch + Redis + OCR) — git submodule
- **Vector DB**: Elasticsearch/OpenSearch (separate service)
- **Vector DB**: Elasticsearch/OpenSearch (migrating to Lancedb)

### Key Patterns
- Backend: SQLAlchemy ORM, Alembic migrations, async pytest
Expand Down Expand Up @@ -41,44 +41,7 @@ cd extralit-server && uv run alembic -c src/extralit_server/alembic.ini upgrade
docker-compose up -d
```

## Development Workflow

# Python Package Management with uv
Use uv exclusively for Python package management in this project.

## Package Management Commands
- All Python dependencies **must be installed, synchronized, and locked** using uv
- Never use pip, pip-tools, poetry, or conda directly for dependency management

Use these commands:
- Install dependencies: `uv add <package>`, avoid adding to pyproject.toml directly
- Remove dependencies: `uv remove <package>`
- Sync environment: `uv sync`
- Lock dependencies: `uv lock`

## Running Python Code
- Run a Python script with `uv run <script-name>.py`
- Run Python tools with `uv run <tool>` (e.g. `uv run pytest`, `uv run ruff check`, `uv run ruff format`, `uv run ty check`, `uv run pre-commit`, `npm run lint`, `npm run format`)
- Launch a Python REPL with `uv run python`

### Running Services
```bash
cd extralit-server && uv run python -m extralit_server server-dev # Server + worker
cd extralit-frontend && npm run dev # Frontend
```

### Testing
```bash
cd extralit-server && uv run pytest tests # Server tests
cd extralit && uv run pytest tests # SDK tests
cd extralit-frontend && npm run test # Frontend tests
```

## Branching

Trunk-based. **`main` is the trunk and the default branch** — every change, code or docs,
branches from `main` and squash-merges back via PR. There is no `develop` branch and no
`releases/**` branches.
## Deployment and Branching

| Ref | Role | Deploys to |
|---|---|---|
Expand All @@ -87,11 +50,6 @@ branches from `main` and squash-merges back via PR. There is no `develop` branch
| `vX.Y.Z` tag | the release itself | PyPI, versioned docs, GitHub Release |
| PR (non-fork) | preview | ephemeral `extralit-dev/pr-N` |

Branch names: `feat/*`, `fix/*`, `docs/*`, short-lived. PR titles use the matching
`feat:` / `fix:` / `docs:` / `chore:` prefix. Note the generated release notes group PRs by
**label**, not by title prefix (see `.github/release.yml`) — label a PR to place it under a
heading; the title convention is for humans.

**Never push to `release` or create tags by hand.** Releases are one dispatch:

```bash
Expand All @@ -103,10 +61,4 @@ That stamps the version via `scripts/bump_version.py` and pushes `main`, `releas
tag atomically. The version lives in three files — always change it with
`python scripts/bump_version.py set --version X.Y.Z`, never by hand.

See `docs/architecture/deployment.md` for the full pipeline and
`extralit/docs/community/release_guide.md` for the release runbook.

## Git Workgrees
When creating a git workgree, place it at `.worktree/<branch-name>` relative to the repo root, normalizing `/` to `-` in the branch-name.

Example: branch `feature/add-new-feature` should be placed at `.worktree/feature-add-new-feature`.
See `docs/architecture/deployment.md` for the full pipeline
92 changes: 78 additions & 14 deletions docs/architecture/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,8 @@ flowchart TD
srvbuild -->|build & push| srvimg
srvbuild -->|repository_dispatch: build-hf-space| bhs
bhs -->|FROM server image + ES/Redis/OCR| spaceimg
bhs -->|"restart (is_release=true)"| demo
bhs -->|"restart (branch=main)"| dev
bhs -->|"commit digest pin (is_release=true)"| demo
bhs -->|"commit digest pin (branch=main)"| dev
bhs -->|duplicate + retarget| prspace
```

Expand Down Expand Up @@ -112,7 +112,7 @@ on **`pull_request`**.
| `extralit/docs/` | `extralit.docs.yml` | `extralit/docs/**`, `mkdocs.yml` | Versioned docs via `mike` → `gh-pages` | No |
| *(whole repo)* | `release.yml` | manual dispatch only | version stamp on `main` + `release` + `vX.Y.Z` tag | **Yes** (indirectly) |
| *(none)* | `github-release.yml` | `vX.Y.Z` tag | GitHub Release, once PyPI serves both packages | No³ |
| `extralit-hf-space/`| `build-hf-space.yml` *(other repo)* | repository_dispatch / manual | `extralit-hf-space` Docker image + Space restart/deploy | **Yes** (terminal) |
| `extralit-hf-space/`| `build-hf-space.yml` *(other repo)* | repository_dispatch / manual | `extralit-hf-space` Docker image + Space deploy | **Yes** (terminal) |

¹ The frontend's own workflow tests, lints, and uploads a prerendered SPA
artifact; the **live** UI is deployed separately by Vercel's native Git
Expand Down Expand Up @@ -289,8 +289,13 @@ Remove it once the trunk flip has been verified.

`env_name` selects the GitHub **Environment** (`production` vs `staging`), which
is how per-environment secrets/vars are scoped: `DOCKER_REPO`,
`EXTRALIT_SERVER_IMAGE`, `HF_SPACE_ID`, `HF_TOKEN`,
`DOCKER_USERNAME`/`DOCKER_PASSWORD`.
`EXTRALIT_SERVER_IMAGE`, `HF_SPACE_ID`,
`DOCKER_USERNAME`/`DOCKER_PASSWORD` (plus `HF_TOKEN` on `staging`, which only
`deploy-pr-space` still uses — see §4).

Note the environments carry **no protection rules and no branch policy**, so
`environment:` here is a scoping mechanism, not an approval gate. What actually
confines production access is the per-job `permissions:` block (§3).
Comment on lines +296 to +298

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

# Inspect the documented authorization controls and the workflow paths that implement them.
set -eu
printf '%s\n' '--- deployment documentation ---'
sed -n '280,355p' docs/architecture/deployment.md
printf '%s\n' '--- workflow files and relevant controls ---'
git ls-files '.github/workflows' | sed -n '1,120p'
rg -n -C 4 'permissions:|id-token|HF_OIDC_RESOURCE|Trusted Publisher|huggingface|environment:' .github/workflows docs/architecture/deployment.md

Repository: Extralit/extralit

Length of output: 25782


🏁 Script executed:

set -eu
printf '%s\n' '--- deployment workflow ---'
sed -n '1,260p' .github/workflows/extralit-server.build-docker-images.yml
printf '%s\n' '--- remaining deployment documentation ---'
sed -n '355,430p' docs/architecture/deployment.md
printf '%s\n' '--- relevant implementation references ---'
rg -n -C 5 'get_token|HF_OIDC_RESOURCE|OIDC|restart_space|huggingface_hub|HF_TOKEN' .github/workflows docs pyproject.toml setup.cfg setup.py requirements*.txt 2>/dev/null || true

Repository: Extralit/extralit

Length of output: 22451


🏁 Script executed:

python3 - <<'PY'
from pathlib import Path
import re

doc = Path("docs/architecture/deployment.md").read_text()
workflow = Path(".github/workflows/extralit-server.build-docker-images.yml").read_text()
tracked = set(__import__("subprocess").check_output(["git", "ls-files"], text=True).splitlines())

deploy_section = doc[doc.index("### Job `deploy-space`"):doc.index("### Job `deploy-pr-space`")]
print("doc_permission_mints_oidc=", "id-token: write" in deploy_section)
print("doc_resource_scopes_space=", bool(re.search(r"HF_OIDC_RESOURCE: spaces/\\$\\{\\{ vars\\.HF_SPACE_ID \\}\\}", deploy_section)))
print("doc_trusted_publisher_exchange=", "Trusted Publishers" in deploy_section and "scoped to that one Space" in deploy_section)
print("doc_boundary_assigns_only_permissions=", "What actually\nconfines production access is the per-job `permissions:` block" in doc)
print("build_workflow_has_id_token_write=", bool(re.search(r"(?m)^\\s+id-token:\\s+write\\b", workflow)))
print("build_workflow_dispatches_companion=", "repository: extralit/extralit-hf-space" in workflow)
print("companion_workflow_tracked_here=", any(p.endswith("build-hf-space.yml") for p in tracked))
PY

Repository: Extralit/extralit

Length of output: 423


🏁 Script executed:

rg -n '^## |^### ' docs/architecture/deployment.md | sed -n '1,100p'

Repository: Extralit/extralit

Length of output: 1226


Security Misconfiguration (CWE-16)

Exploitability: Theoretical

State the production authorization boundary correctly.

id-token: write only allows deploy-space to mint a GitHub OIDC token. HF_OIDC_RESOURCE and the Trusted Publisher claims scope the exchanged Hugging Face token to the target Space.

Proposed wording
-What actually confines production access is the per-job `permissions:` block (§3).
+The per-job `permissions:` block (§3) allows only `deploy-space` to mint an OIDC token; `HF_OIDC_RESOURCE` and the Trusted Publisher claims (§4) scope the exchanged Hugging Face token.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Note the environments carry **no protection rules and no branch policy**, so
`environment:` here is a scoping mechanism, not an approval gate. What actually
confines production access is the per-job `permissions:` block (§3).
Note the environments carry **no protection rules and no branch policy**, so
`environment:` here is a scoping mechanism, not an approval gate. The per-job
`permissions:` block (§3) allows only `deploy-space` to mint an OIDC token;
`HF_OIDC_RESOURCE` and the Trusted Publisher claims (§4) scope the exchanged
Hugging Face token.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/architecture/deployment.md` around lines 296 - 298, Update the
deployment authorization explanation near the environment scoping note to state
that id-token: write only lets deploy-space mint a GitHub OIDC token, while
HF_OIDC_RESOURCE and Trusted Publisher claims scope the exchanged Hugging Face
token to the target Space; do not describe these permissions as the production
access boundary.


### Job `build`
Builds the self-contained Space image **on top of the server image**:
Expand All @@ -309,13 +314,56 @@ multi-process runtime (elastic + redis + RQ workers + FastAPI). Pushed to
(staging), tagging `:latest` when `tag_latest=true`.

### Job `deploy-space` — *non-PR builds only* (`pr_space_slug == ''`)
Restarts the live Space so it pulls the freshly pushed image:
Deploys by **committing to the Space repo**, not by restarting it. The Space is a
thin `FROM <pushed image>` Dockerfile; the job rewrites that `FROM` line to the
**digest** `build` just pushed and commits it, and HF rebuilds on the new commit.

```bash
curl -X POST "https://huggingface.co/api/spaces/${HF_SPACE_ID}/restart" \
-H "Authorization: Bearer $HF_TOKEN"
```yaml
permissions:
contents: read
id-token: write # mint the OIDC token; see below
env:
HF_OIDC_RESOURCE: spaces/${{ vars.HF_SPACE_ID }}
IMAGE_DIGEST: ${{ needs.build.outputs.image_digest }}
```
```python
after = re.sub(r"^FROM\s+\S+", f"FROM {repo}@{digest}", before, count=1, flags=re.M)
api.upload_file(path_or_fileobj=after.encode(), path_in_repo="Dockerfile",
repo_id=space, repo_type="space", commit_message=f"Deploy {pin}")
```

**Why a commit and not `restart_space()`.** The restart API answers **401** to an
OIDC token: a repo publisher grants *write access to that repo*, and restarting is
a runtime operation rather than a repo write. Committing is what the credential is
for, and is HF's own
[documented GitHub Actions pattern](https://huggingface.co/docs/hub/en/spaces-github-actions).
A 401 here is specifically the restart endpoint — a misconfigured publisher fails
earlier and differently, as `OIDCError`/`invalid_grant` from the exchange.

**Why a digest and not a tag.** When the `FROM` line was `:latest`, HF reused the
base image it had already built and never re-pulled — the job went green while the
Space served the old build. That is the v0.7.0 failure: it cycled
`RUNNING_APP_STARTING → RUNNING` and still reported 0.6.1. A digest cannot resolve
to a previously-built image, which retires the `factory_reboot=True` workaround
that used to paper over this.

Re-deploying an unchanged digest is a no-op commit and so triggers no rebuild; that
path skips the wait and asserts the Space's *current* stage is `RUNNING`, so an
earlier failed build is never reported as a green redeploy.

**This job holds no HF credential.** It authenticates with
[Trusted Publishers](https://huggingface.co/docs/hub/en/trusted-publishers):
GitHub Actions mints a short-lived OIDC id token, and HF exchanges it (RFC 8693)
for a token scoped to that one Space for ~1h. `huggingface_hub` does the whole
dance inside `get_token()` when `HF_OIDC_RESOURCE` is set, and raises `OIDCError`
rather than falling back to an ambient credential.

Each Space registers a publisher pinned to repo `Extralit/extralit-hf-space`,
branch `main`, workflow `build-hf-space.yml`. Those claims are satisfied by
*every* job in this file, so the `id-token: write` grant is deliberately scoped to
this job alone — `build` and `deploy-pr-space` inherit only the workflow-level
`contents: read` and therefore cannot mint a token to exchange at all.

`HF_SPACE_ID` is the environment-scoped Space (see §5): the **`production`**
environment points at `extralit/public-demo` — the live public demo served at
**<https://extralit-public-demo.hf.space>** — while the **`staging`**
Expand Down Expand Up @@ -382,10 +430,26 @@ resolve **per environment**.

| Secret | Repo-level | `production` env | `staging` env |
| ----------------- | :--------: | :--------------: | :-----------: |
| `HF_TOKEN` | ✅ (default) | ✅ (override) | ✅ (override) |
| `HF_TOKEN` | — | — | ✅ |
| `DOCKER_USERNAME` | — | ✅ | ✅ |
| `DOCKER_PASSWORD` | — | ✅ | ✅ |

`HF_TOKEN` survives **only** on `staging`, and only for `deploy-pr-space`.
Trusted Publishers scope a token to an *existing* repo, so they cannot cover
`duplicate_space()`, which creates `extralit-dev/pr-N` on demand. That leaves the
remaining token's write access confined to the `extralit-dev` org — nothing can
reach `extralit/public-demo` with a stored credential.
Comment on lines +437 to +441

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- workflow files ---'
git ls-files .github/workflows
printf '%s\n' '--- HF_TOKEN and staging references ---'
rg -n -C 4 'HF_TOKEN|deploy-pr-space|deploy-space|environment:\s*staging|duplicate_space|extralit/public-demo|extralit-dev/pr-' .github/workflows docs/architecture/deployment.md

Repository: Extralit/extralit

Length of output: 24628


🏁 Script executed:

#!/bin/bash
set -e
repo='extralit/extralit-hf-space'
printf '%s\n' '--- workflow listing ---'
curl -fsSL "https://api.github.com/repos/$repo/contents/.github/workflows" |
  jq -r '.[] | .path' |
  sort
printf '%s\n' '--- HF_TOKEN references in workflow files ---'
for path in $(curl -fsSL "https://api.github.com/repos/$repo/contents/.github/workflows" |
  jq -r '.[].path'); do
  body=$(curl -fsSL "https://raw.githubusercontent.com/$repo/main/$path")
  if printf '%s' "$body" | rg -n -q 'HF_TOKEN|deploy-pr-space|deploy-space|environment:|repository_dispatch|workflow_dispatch'; then
    printf '\n### %s\n' "$path"
    printf '%s\n' "$body" | rg -n -C 5 'HF_TOKEN|deploy-pr-space|deploy-space|environment:|repository_dispatch|workflow_dispatch'
  fi
done

Repository: Extralit/extralit

Length of output: 324


🏁 Script executed:

#!/bin/bash
set -e
url='https://raw.githubusercontent.com/extralit/extralit-hf-space/main/.github/workflows/build-hf-space.yml'
printf '%s\n' '--- build-hf-space.yml ---'
curl -fsSL "$url" |
  rg -n -C 6 'HF_TOKEN|deploy-pr-space|deploy-space|environment:|repository_dispatch|workflow_dispatch'

Repository: Extralit/extralit

Length of output: 6239


🏁 Script executed:

#!/bin/bash
set -e
url='https://raw.githubusercontent.com/extralit/extralit-hf-space/main/.github/workflows/build-hf-space.yml'
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
curl -fsSL "$url" > "$tmp"
python3 - "$tmp" <<'PY'
import re, sys
text = open(sys.argv[1]).read()
checks = {
    "build_has_HF_TOKEN": bool(re.search(r"(?ms)^  build:.*?(?=^  [A-Za-z0-9_-]+:|\Z).*?HF_TOKEN", text)),
    "deploy_space_environment": re.search(r"(?ms)^  deploy-space:.*?^    environment:\s*(.+)$", text),
    "deploy_space_condition": re.search(r"(?ms)^  deploy-space:.*?^    if:\s*(.+)$", text),
    "deploy_space_HF_TOKEN": bool(re.search(r"(?ms)^  deploy-space:.*?HF_TOKEN:\s*\$\{\{\s*secrets\.HF_TOKEN\s*\}\}", text)),
    "deploy_pr_environment": re.search(r"(?ms)^  deploy-pr-space:.*?^    environment:\s*(.+)$", text),
    "deploy_pr_condition": re.search(r"(?ms)^  deploy-pr-space:.*?^    if:\s*(.+)$", text),
    "deploy_pr_HF_TOKEN": bool(re.search(r"(?ms)^  deploy-pr-space:.*?HF_TOKEN:\s*\$\{\{\s*secrets\.HF_TOKEN\s*\}\}", text)),
}
for key, value in checks.items():
    if hasattr(value, "group"):
        value = value.group(1).strip()
    print(f"{key}={value!r}")
print("\n--- resolve-env and job headers ---")
for i, line in enumerate(text.splitlines(), 1):
    if i <= 95 or 140 <= i <= 195:
        print(f"{i}:{line}")
PY

Repository: Extralit/extralit

Length of output: 21558


Sensitive Data Exposure (CWE-732): Incorrect Permission Assignment for Critical Resource

Reachability: Internal · Exploitability: Theoretical

Align the documentation with the workflow’s token boundary.

build-hf-space.yml passes secrets.HF_TOKEN to both deploy-space and deploy-pr-space. On staging main dispatches, deploy-space restarts extralit-dev/develop; build does not receive the token. Update lines 419–423 to describe both consumers and document the token’s Hugging Face scope.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/architecture/deployment.md` around lines 419 - 423, The deployment
documentation incorrectly limits HF_TOKEN to deploy-pr-space and omits
deploy-space. Update the section around deploy-space and deploy-pr-space to
document that build-hf-space.yml passes the token to both consumers, that
staging main dispatches use deploy-space to restart extralit-dev/develop, and
that build does not receive it; accurately state the token’s Hugging Face scope.


> Deleting the repo-level `HF_TOKEN` is part of this, not an afterthought:
> `secrets.HF_TOKEN` silently falls back to it, so leaving it in place would make
> removing the `production` override purely cosmetic.
>
> **Order matters.** This table is the state *after* a production release has
> deployed keyless end-to-end. Until then both overrides stay, so reverting
> `deploy-space` to a stored token remains a one-commit rollback — deleting them
> early costs nothing on the deploy path (nothing reads `secrets.HF_TOKEN` there
> anymore) but throws away that escape hatch.

### `extralit/extralit` (monorepo)

- **Repo variables:** none.
Expand Down Expand Up @@ -442,7 +506,7 @@ The cross-repo `client-payload` carries the handoff state:
| `image_tag` *(output)* | payload tag, or `latest` (manual) | Docker tag built & deployed |
| `tag_latest` *(output)* | `true` on release/trunk | also tag/push `:latest` |
| `platforms` *(output)* | `amd64` (staging) / `amd64,arm64` (release) | buildx target platforms |
| `pr_space_slug` *(output)* | `pr-N` / slug for preview refs | empty → restart live Space; set → PR preview |
| `pr_space_slug` *(output)* | `pr-N` / slug for preview refs | empty → deploy live Space; set → PR preview |
| `DOCKER_TAGS` | `${DOCKER_REPO}:${IMAGE_TAG}[,:latest]` | tags pushed by build job |
| `EXTRALIT_SERVER_IMAGE` *(build-arg)* | `vars.EXTRALIT_SERVER_IMAGE` | base image the Space is built `FROM` |
| `EXTRALIT_VERSION` *(build-arg)* | `image_tag` | base image tag (→ Dockerfile `ARG`) |
Expand Down Expand Up @@ -508,7 +572,7 @@ The cross-repo `client-payload` carries the handoff state:
4. `repository_dispatch(build-hf-space, {tag: main, is_release: false, branch: main})`.
5. `build-hf-space.yml` → `resolve-env` (env=`staging`) → builds
`extralitdev/extralit-hf-space:main` `FROM` the server image.
6. `deploy-space` → `POST /spaces/extralit-dev/develop/restart`.
6. `deploy-space` → commits the digest pin to `extralit-dev/develop`, which rebuilds.
7. Live at **<https://extralit-dev-develop.hf.space>**. **Production untouched.**

### Release → public demo (`release` + tag)
Expand All @@ -521,8 +585,8 @@ The cross-repo `client-payload` carries the handoff state:
(`is_release=true`) → `extralit/extralit-server:vX.Y.Z` (+`:latest`), amd64+arm64.
4. `repository_dispatch(build-hf-space, {tag: vX.Y.Z, is_release: true})`.
5. `build-hf-space.yml` → `resolve-env` (env=`production`) → builds
`extralit/extralit-hf-space:vX.Y.Z` → `deploy-space` restarts
`extralit/public-demo`.
`extralit/extralit-hf-space:vX.Y.Z` → `deploy-space` commits that image's
digest to `extralit/public-demo`, which rebuilds.
6. In parallel, the **tag** push drives PyPI (`extralit`, `extralit-server`),
versioned docs (`mike deploy X.Y` + `stable`), and the GitHub Release.
7. Public demo live at **<https://extralit-public-demo.hf.space>**.
Expand Down