Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 31 additions & 24 deletions docs/architecture/llm-architecture.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
# LLM and embedding architecture

Loom has two pluggable providers: an embedder (`server/embedding/`) that turns text into
vectors, and an LLM client (`server/llm/`) that completes prompts. Each is chosen once
per process from settings. Each falls back to a deterministic fake when its key is
missing, so a keyless install boots and answers. Operator setup is in
vectors, and an LLM client (`server/llm/`) that completes prompts. Their active
configuration comes from server settings. API-key providers can fall back to a
deterministic fake when their key is missing; native Bedrock uses deployment AWS
credentials and propagates credential failures. Operator setup is in
[deployment](../operations/deployment.md); every setting is in the
[configuration reference](../reference/configuration.md).

## Goals and non-goals

| Goals | Non-goals |
| --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Every correctness gate on the write path works without an LLM. | A general LLM gateway. Loom calls models for its own features only. |
| A keyless install boots and says clearly that it runs on fakes. | Provider-agnostic prompts. Prompts are tuned to a model and re-benched on a change. |
| Swapping the embedder does not silently miscalibrate recall. | Mixing vector spaces. Vectors from different models are never compared. |
| Every provider call is attributed to the op or worker pass that made it, with its cost. | Runtime prompt management. Prompts are code. |
| Goals | Non-goals |
| ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Every correctness gate on the write path works without an LLM. | A general LLM gateway. Loom calls models for its own features only. |
| A keyless install boots and says clearly that it runs on fakes. | Provider-agnostic prompts. Prompts are tuned to a model and re-benched on a change. |
| Swapping the embedder does not silently miscalibrate recall. | Mixing vector spaces. Vectors from different models are never compared. |
| Every provider call is attributed to its op or worker pass, with usage and cost when known. | Runtime prompt management. Prompts are code. |

## Embedders

Expand All @@ -25,21 +26,25 @@ once per process:

1. A label resolves first. `prefix:<dim>:<model>` and `ctx[<variant>]:<model>` are views
over the served embedder and cannot be the served model.
2. `LOOM_EMBEDDING_PROVIDER` (`fake`, `local`, `openai`, `gemini`) when set; any other
value raises.
2. `LOOM_EMBEDDING_PROVIDER` (`fake`, `local`, `openai`, `gemini`, `compatible`,
`bedrock`) when set; any other value raises.
3. Otherwise inferred from `LOOM_EMBEDDING_MODEL`: `fake-ngram*` is fake, `local` or
`local:*` is local, `text-embedding-*` is OpenAI, `gemini*` is Gemini. Any other name
raises `unknown embedding model`.

| Provider | Module | Model names | Needs | Notes |
| -------- | ---------------------------- | -------------------------------------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------- |
| `fake` | `server/embedding/fake.py` | `fake-ngram-<dim>` | nothing | Hashed character trigrams, L2-normalised. Deterministic; the dev default. |
| `local` | `server/embedding/local.py` | `local` (`bge-small`), `local:bge-small`, `local:bge-base`, `local:minilm` | the `local-embedder` extra (`fastembed`) | CPU ONNX; 384, 768 and 384 dimensions. No network. |
| `openai` | `server/embedding/openai.py` | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` | Requests `dimensions=LOOM_EMBEDDING_DIM`. |
| `gemini` | `server/embedding/gemini.py` | `gemini-embedding-001` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | Requests `outputDimensionality=LOOM_EMBEDDING_DIM` (up to 3072). |

**Width.** The served width is `LOOM_EMBEDDING_DIM` (default 3072). OpenAI and Gemini are
asked for that width; choose a model that supports it. Local models emit their native width. Boot probes the real
`local:*` is local, `text-embedding-*` is OpenAI, `gemini*` is Gemini and `bedrock:*`
is Bedrock. Any other name raises `unknown embedding model`; a compatible endpoint
needs an explicit provider.

| Provider | Module | Model names | Needs | Notes |
| ------------ | -------------------------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `fake` | `server/embedding/fake.py` | `fake-ngram-<dim>` | nothing | Hashed character trigrams, L2-normalised. Deterministic; the dev default. |
| `local` | `server/embedding/local.py` | `local` (`bge-small`), `local:bge-small`, `local:bge-base`, `local:minilm` | the `local-embedder` extra (`fastembed`) | CPU ONNX; 384, 768 and 384 dimensions. No network. |
| `openai` | `server/embedding/openai.py` | `text-embedding-3-small`, `text-embedding-3-large` | `OPENAI_API_KEY` | Requests `dimensions=LOOM_EMBEDDING_DIM`. |
| `gemini` | `server/embedding/gemini.py` | `gemini-embedding-001` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` | Requests `outputDimensionality=LOOM_EMBEDDING_DIM` (up to 3072). |
| `compatible` | `server/embedding/compatible.py` | the endpoint's embedding model ID | `LOOM_EMBEDDING_BASE_URL`, `LOOM_EMBEDDING_API_KEY` or configured workload-token authentication | [OpenAI-compatible embeddings](../getting-started/models-and-keys.md#claude-for-the-llm); optional dimensions parameter. |
| `bedrock` | `server/embedding/bedrock.py` | Cohere Embed v4 model or inference-profile ID | deployment AWS credentials | [Native Bedrock](#native-bedrock); no API-key fallback. |

**Width.** The served width is `LOOM_EMBEDDING_DIM` (default 3072, or 1536 for native
Bedrock when unset). OpenAI and Gemini are asked for that width; choose a model that
supports it. Local models emit their native width. Boot probes the real
width (`probe_embedding_dim`); a mismatch shows as `embedding_dim_mismatch` on
`/v1/health`.

Expand Down Expand Up @@ -324,7 +329,9 @@ them. `LOOM_LLM_ANTHROPIC_EFFORT` sends a thinking effort to Claude models.
(transport error, retryable status, timeout) or a rejected request (400, 413, 422);
401, 403 and 404 propagate. Each hand-over is logged and counted (`loom.llm.fallback`),
and the result names the model that answered. Fallbacks do not chain.
- **Missing key.** The client is the fake (`_build_llm_client`), logged at WARNING.
- **Missing API key.** An OpenAI or Anthropic client without its key uses the fake
(`_build_llm_client`), logged at WARNING, unless configured workload-token
authentication supplies credentials. Native Bedrock credential failures propagate instead.
- **One identity refused.** With identity headers set, an answer from a gateway host that
`LOOM_INFERENCE_IDENTITY_REFUSALS_JSON` lists is the gateway refusing the call's identity or
asking it to wait ([identity headers](#identity-headers)): never retried, never handed to the
Expand Down
61 changes: 60 additions & 1 deletion docs/features/code-index/repositories.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,65 @@ that:
A `code.repo.set_access` row wins over the configured classification and join policy
and survives restarts. `code.refresh` (writer) and the admin ops always need a grant.

### Separate repositories for two teams

Keep repositories with different readers in different code namespaces. For example,
index a billing repository as `code-billing` and an operations repository as
`code-operations`, both `classification: "restricted"` with `join_policy: "approval"`.
Use the [operator-configured entry format](../../operations/code-index-operations.md)
or the [self-service access controls](#self-service-repositories) for your deployment.
Bootstrap maintainer grants before restricting an existing namespace, following the
[restricted-namespace procedure](../../guides/tokens-and-grants.md#restricted-namespaces-classificationrestricted).

Give a billing service a live viewer grant and a token explicitly naming only
`code-billing`. Its expected read behavior is:

| Request | Expected scope outcome |
| ---------------------------------- | ------------------------------------------------------------------------------ |
| `code.search` in `code-billing` | Authorized; useful hits still depend on the index and query |
| `code.search` in `code-operations` | Denied; the billing grant and token do not cover it |
| `code.repos` | Can include request-only metadata for the other repository; inspect `readable` |

Check the allowed request with a symbol you know is indexed, then repeat an equivalent
request against the other namespace. Over HTTP, a denied scoped code read returns
`AUTH_INSUFFICIENT`; a known object ID can instead read as missing to avoid revealing its
existence ([authorization diagnostics](../../operations/troubleshooting.md#an-id-you-know-exists-reads-as-missing)).
Test revocation separately from an empty search result. Repeat for each indexed branch:
namespace naming and maintainer inheritance do not give every reader access to every
branch.

A same-organization repository with approval-based joining can appear in `code.repos`
as `readable: false`. That row exposes metadata for requesting access, with empty index
counts and watermarks. It does not authorize reading code. Do not treat a listing entry
as a successful access check.

For example, with the billing service's token, replace the query with a known billing
symbol. Repeat with `namespace` set to `code-operations` and expect an authorization
failure, not an empty successful result:

```sh
curl --fail-with-body -X POST "$LOOM_BASE_URL/v1/code.search" \
-H "Authorization: Bearer $LOOM_API_KEY" \
-H 'Content-Type: application/json' \
--data '{"org":"example","namespace":"code-billing","query":"invoice authorization","top_k":3}'
```

The Git token or SSH identity that fetches source is separate from the Loom token and
grants used by readers. A chat or document connector login does not grant a native Loom
service repository access. Agent names and repository tags record attribution or
association; they are not access boundaries.

### Check indexing isolation separately from read access

Repository read grants do not sandbox indexing jobs. Self-service indexing requires
the sandbox unless an operator explicitly enables its restricted unsandboxed mode;
operator-configured repositories can run without it. To require isolation for every
repository job, set `LOOM_CODE_SANDBOX_REQUIRED=true` and satisfy the host requirements.
When the sandbox is unavailable, jobs are refused or skipped rather than silently run
outside it. The [sandbox runbook](../../operations/code-index-operations.md#repository-job-sandbox)
describes namespaces, mounts, egress, process limits, supported stacks and remaining
container/host responsibilities; Loom's per-process limits do not impose a job cgroup.

## Failure modes

| Symptom | Cause and fix |
Expand All @@ -226,7 +285,7 @@ and survives restarts. `code.refresh` (writer) and the admin ops always need a g
| Adding a repository answers `AUTH_INSUFFICIENT` | the message names `LOOM_CODE_SELF_SERVICE_PLATFORM_ADMIN_ONLY`: only platform admins add repositories and branches on this deployment, with a token of their own that is not scoped to namespaces |
| A self-service card stays at `error` | read `stage_detail` in `code.repos`; a failed clone retries next pass (timeout `LOOM_CODE_BOOTSTRAP_CLONE_TIMEOUT_S`) |
| A private self-service repository stops fetching | `LOOM_JWT_SECRET` was rotated; re-enter the token |
| An entry is missing from `code.repos` | check `loom.code.repos_config_errors`; `code.repos` also lists only what the caller may read |
| An entry is missing from `code.repos` | check `loom.code.repos_config_errors` and the caller's listing eligibility; request-only entries have `readable: false` and do not grant code access |
| Revoking or lowering an admin answers busy | a revoke on a code namespace first re-reads `loom.code_repos`, so it sees the branches added here from another process, and answers busy while that read fails; retry once the read succeeds |
| A restricted branch added here has no admin | its admins were all copies from branches added here, which the sync revokes; grant admin on one of the repository's operator entries (with none, the operator lists a branch of it in `LOOM_CODE_REPOS_JSON` first) |
| A new maintainer lacks the other branches | the admin grant was made before this process read `loom.code_repos` (that process logs `not synced` the first time); the next admin grant or approval on a branch of the repository, or a branch added, copies it; or grant it again |
Expand Down
28 changes: 28 additions & 0 deletions docs/getting-started/evaluate.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,34 @@ agent tasks with ordinary repository tools only. Report task success first, then
tokens, cost and wall time. Tool-schema overhead depends on the client, so record whether
it defers MCP tool definitions.

#### Label your own repository before querying

The built-in acceptance report is not a repository-neutral answer-quality test. Make a
task sheet for your own repository with these columns before collecting results:

| Column | What to record |
| ---------------------- | ---------------------------------------------------------------------------------------------- |
| Task and frozen source | Question, repository commit, branch and task category |
| Expected evidence | Definition path and qualified symbol, accepted source ranges, or a reviewed caller/test set |
| Expected answer | Claims the answer must support and claims it must not make |
| Expected refusal | Unanswerable, wrong-repository and denied-access cases |
| Retrieval outcome | Returned identifiers, ranks, citations, freshness and completeness limits |
| Agent outcome | Actual calls and evidence used, answer or patch, objective test result, latency and total cost |

This is a proposed evaluation record, not an input schema accepted by the supplied
harness. Call `code.symbol`, `code.callers`, `code.impact`, `code.search` or `code.ask`
for the corresponding labelled cases using the
[operation reference](../reference/operations/code.md). Read each result's uncertainty:
an incomplete graph should not be graded as an exhaustive caller set, and changed or
missing watermarks leave a freshness comparison unverified. Keep stale, failed and
permission-denied requests visible alongside successful retrieval.

For task-level comparisons and a CTO decision scorecard, use the
[paired agent-platform evaluation](../guides/agent-platforms.md#give-a-cto-an-evaluation-they-can-check).
It includes application, hook and model-selected calls, zero-use tasks, citation checks,
objective tests, construction cost and uncertainty. That procedure supplies no measured
result until you run and report it on the chosen workload.

### Keep the comparison fair

- Use the same model, prompt, data snapshot and question order in paired arms.
Expand Down
Loading