Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
3284a6c
Bound delegated Workspace grants and Directory reads
zlareb1 Nov 18, 2026
d37d3fb
Require a subject for delegated token grants
zlareb1 Nov 18, 2026
67b5c35
Preserve delegated Directory quota failures
zlareb1 Nov 18, 2026
a7069c1
Prepare bounded crawl source observations
zlareb1 Nov 18, 2026
21cfada
Keep the crawl row fixture in canonical source form
zlareb1 Nov 18, 2026
a7fa6d5
Fence durable crawl exclusions through cleanup and repair
zlareb1 Nov 18, 2026
9f02606
Retain source approval as checked authorization metadata
zlareb1 Nov 18, 2026
3f1d580
Bind delegated crawl tokens to retained source authority
zlareb1 Nov 18, 2026
b9bc571
Require explicit denial of every crawl write scope
zlareb1 Nov 18, 2026
7daf0f8
Observe complete bounded crawl unions before publishing
zlareb1 Nov 18, 2026
aae036d
Reseal crawl refusal keys without changing their identity
zlareb1 Nov 18, 2026
ca5fcd3
Generate crawl control settings references
zlareb1 Nov 18, 2026
b9aa2c5
Pin complete crawl control and membership row builders
zlareb1 Nov 18, 2026
4b1aa7f
Prepare verified person capture and historical crawl ACL redaction
zlareb1 Nov 18, 2026
88c0c8c
Redact exact person references from crawl controls and membership his…
zlareb1 Nov 18, 2026
3eead92
Prepare stable Workspace identity before canonical person deletion
zlareb1 Nov 18, 2026
497922b
Fence canonical person cleanup through raw capsule erasure
zlareb1 Nov 18, 2026
916437c
Generate bounded crawl person cleanup configuration
zlareb1 Nov 18, 2026
9d86c73
Erase verified person aliases from historical crawl approvals
zlareb1 Nov 18, 2026
5c9b445
Refuse unknown restrictions in retained crawl approval metadata
zlareb1 Nov 18, 2026
6b5da35
Bind delegated crawl registration and indexed reads to completed sour…
zlareb1 Nov 18, 2026
484eb1f
Preserve crawl person refusals across readmission rotation and restore
zlareb1 Nov 18, 2026
e316c02
Cover crawl registry loss and worker retry boundaries
zlareb1 Nov 18, 2026
54760e5
Preserve crawl refusal and sync retry proofs
zlareb1 Nov 18, 2026
01e6f2c
Pin delegated crawl control defaults
zlareb1 Nov 18, 2026
4151bef
Format delegated crawl storage rows
zlareb1 Nov 18, 2026
f095513
Generate crawl source companions on an isolated hosted runner
zlareb1 Nov 18, 2026
3b6e739
Adopt canonical crawl companions and classify capsule resealing
zlareb1 Nov 18, 2026
500f8b4
Align crawl fixtures with live authorization and schema contracts
zlareb1 Nov 18, 2026
9dee829
fix: preserve crawl deadlines and lifecycle fixture contracts
zlareb1 Nov 18, 2026
8c4a61b
test: model canonical person cleanup and verified token metadata
zlareb1 Nov 18, 2026
192f3a8
test: cover crawl restore and historical approval resealing
zlareb1 Nov 18, 2026
04a2d3f
Prevent source resealing from reviving erased metadata
zlareb1 Nov 18, 2026
353be99
Keep uncertain metadata writes fenced until claim expiry
zlareb1 Nov 18, 2026
6a0fed9
Crawl test claims retain settlement and distinguish completed cleanup
zlareb1 Nov 18, 2026
4296e41
Test native crawl publication through authenticated HTTP
zlareb1 Nov 18, 2026
8e7e7a5
Use account sessions in the native crawl witness
zlareb1 Nov 18, 2026
b7d9a85
Use valid collections in the native crawl witness
zlareb1 Nov 19, 2026
e5ca7a8
Use the next migration number for workspace crawls
zlareb1 Nov 20, 2026
55a5fa9
Match the crawl migration journal filename
zlareb1 Nov 20, 2026
a234cf8
Compose workspace crawls with current main
zlareb1 Nov 20, 2026
ff0645b
Compose workspace crawls with the latest main fixes
zlareb1 Nov 20, 2026
4e35400
Require the crawl migration across the serving fleet
zlareb1 Nov 20, 2026
113d808
Crawl fixtures advertise the required schema migration
zlareb1 Nov 20, 2026
c748697
Classify the crawl erasure column update in the SQL census
zlareb1 Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions .github/workflows/crawl-native-privacy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
name: Crawl native privacy

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

jobs:
crawl-native-privacy:
runs-on: ubuntu-latest
timeout-minutes: 20
env:
LOOM_ENV_FILE: ""
LOOM_CH_CONTAINER: docsrc-crawl-${{ github.run_id }}-${{ github.run_attempt }}
LOOM_CH_HOST: 127.0.0.1
LOOM_CH_PORT: "18123"
LOOM_CH_HOST_PORT: "18123"
LOOM_CH_NATIVE_HOST_PORT: "19000"
LOOM_CH_USER: loom
LOOM_CH_PASSWORD: loomdev
LOOM_CH_DATABASE: loom
LOOM_DEPLOYMENT: "1"
LOOM_INSTANCE_LABEL: ci-crawl-${{ github.run_id }}-${{ github.run_attempt }}
LOOM_BUILD_COMMIT: ${{ github.sha }}
LOOM_AUTH_REQUIRED: "true"
LOOM_STORAGE_AUTHORITY: clickhouse
LOOM_WORKERS_PROCESS: inline
LOOM_BACKGROUND_WORKERS: auto
LOOM_EMBEDDING_MODEL: fake-ngram-1024
LOOM_LLM_PROVIDER: fake
LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED: "true"
LOOM_KNOWLEDGE_SOURCE_SYNC_ENABLED: "true"
LOOM_KNOWLEDGE_SOURCE_SYNC_INTERVAL_SECONDS: "60"
LOOM_KNOWLEDGE_SOURCE_SYNC_IDLE_INTERVAL_SECONDS: "60"
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
persist-credentials: false
- uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: "22"
- name: Install locked dependencies and build dashboard assets
run: |
uv sync --locked --python 3.12 --extra dev --extra code
npm --prefix frontend ci
npm --prefix frontend run build
- name: Configure the test signer
run: |
crawl_signer_material="$(openssl rand -hex 32)"
printf '::add-mask::%s\n' "$crawl_signer_material"
printf 'LOOM_JWT_SECRET=%s\n' "$crawl_signer_material" >> "$GITHUB_ENV"
- name: Start the owned backend
id: clickhouse
run: |
if docker container inspect "$LOOM_CH_CONTAINER" >/dev/null 2>&1; then
echo "refusing an existing backend container" >&2
exit 1
fi
LOOM_CLICKHOUSE_IMAGE="$(bash dev/clickhouse_release.sh)"
export LOOM_CLICKHOUSE_IMAGE
echo "admitted=true" >> "$GITHUB_OUTPUT"
bash dev/ci_start_clickhouse.sh
- name: Run the native privacy witness
run: >-
.venv/bin/python -m pytest tests/integration/crawl_native_privacy.py
-q --tb=short --junitxml=crawl-native-privacy.xml
- name: Preserve backend diagnostics
if: always() && steps.clickhouse.outputs.admitted == 'true'
run: docker logs "$LOOM_CH_CONTAINER" > crawl-native-clickhouse.log 2>&1
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
if: always()
with:
name: crawl-native-privacy
path: |
crawl-native-privacy.xml
crawl-native-clickhouse.log
if-no-files-found: ignore
- name: Remove the owned backend
if: always() && steps.clickhouse.outputs.admitted == 'true'
run: docker rm -f "$LOOM_CH_CONTAINER"
2 changes: 2 additions & 0 deletions dev/restore_connected_accounts.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
from server.cli._apply import add_apply_flags
from server.cli._ch import reads, run_connected, writes
from server.knowledge.app.connected_restore import restore_all
from server.knowledge.app.crawl_restore import restore_all as restore_crawls


def main(argv: list[str] | None = None) -> int:
Expand All @@ -26,6 +27,7 @@ def main(argv: list[str] | None = None) -> int:

async def run(ch: Any) -> int:
await restore_all(apply=not args.dry_run, ch=ch)
await restore_crawls(apply=not args.dry_run, ch=ch)
print(
"connected-account restore cleanup completed; connections must be established again"
if not args.dry_run
Expand Down
19 changes: 19 additions & 0 deletions docs/architecture/clickhouse.md
Original file line number Diff line number Diff line change
Expand Up @@ -694,6 +694,25 @@ with `purge=true`, runner state, and the supersession rows `knowledge.restore` r
refused because an erasure removed the document it was built from deletes the chunk
rows it inserted (lightweight `DELETE FROM loom.chunks` by source and version,
`chunks.delete_versions`), so its text and author do not outlive the erasure.
- **Source credential resealing** conditionally updates only the matching old
ciphertext for an exact org, namespace, name and source id, under the source
lease. It rewrites raw versions and checks for old-ciphertext residue before
the previous root is retired. A full-row replacement could revive selectors
or identity metadata erased during a delayed write; the mutation preserves
every other column, including the version and unrelated sources.
- **Canonical-person metadata publishers** bound every source and SCIM INSERT
under their claim's receiver deadline. An unanswered send retains its Keeper
key until expiry; the receiver's latest possible completion precedes that
expiry. Nested claims retain the earliest deadline and every unanswered send.
Cleanup cannot acquire the same claim and report raw history erased while an
older publisher may still append it. Scoped statement bounds also limit the
guarded receiver; an insufficient claim window refuses before publishing.
- **SCIM cleanup capsule resealing** updates matching historical `scim_users` rows
in place under the identity lease, by exact SCIM row id and old ciphertext. The
replacement retains the captured canonical/provider identity and cleanup phase.
It waits for mutation completion and checks raw-history residue before the old
sealing root can be retired. Appending a version would leave older capsules
encrypted by the retired root (`server/auth/person_erasure.py`).
- **Remaining `ALTER … UPDATE/DELETE` call sites** are listed in
`tests/unit/test_clickhouse_mutation_inventory.py`, which fails on a new one until it is
classified there.
Expand Down
43 changes: 38 additions & 5 deletions docs/architecture/document-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -373,7 +373,7 @@ directory, so it expands no group ([below](#who-may-read)).

A `crawl` source's credential is a service-account key whose client id the org's
workspace administrator granted domain-wide delegation for `drive.readonly`,
`admin.directory.group.member.readonly` and `admin.directory.user.readonly`. Its config
`admin.directory.group.member.readonly` and `admin.directory.user.readonly`. Its `crawl_options` field
holds `directory_subject` (a workspace administrator the directory reads impersonate)
and `subjects`: people (emails) and groups (group emails), whose members the directory
read lists at each turn, at most 2000 people (not a setting). For each person the sync
Expand All @@ -385,8 +385,9 @@ included, leaves out the people an erasure excluded ([erasure](#erasure-and-audi

The probe checks that the delegation grants no write on what the crawl reads: it asks
for each of the three scopes, which must be granted, and for
`https://www.googleapis.com/auth/drive`, `admin.directory.user` and
`admin.directory.group`, which the store must refuse; a credential the store grants any
`https://www.googleapis.com/auth/drive`, `admin.directory.user`,
`admin.directory.group` and the distinct `admin.directory.group.member` write scope,
which the store must refuse; a credential the store grants any
of them is refused (`CREDENTIAL_REFUSED`). It cannot prove the delegation grants nothing
else (a mail scope, say), and the page says so to the admin who registers it. The audit
rows of `add`, `update` and `rotate` record the configured subjects as keyed hashes (an
Expand All @@ -395,6 +396,24 @@ addresses does not reverse them). Audit rows also record the scopes granted and
number of members resolved: the audit trail outlives erasure and keeps no crawled
person's address.

Registration, update and rotation require a current org administrator who also
administers the source namespace. Each approval binds the signing-key lifecycle,
canonical registrant, exact selectors, items, credential hash, verified Workspace
customer and source generation. Workers recheck that authority after provider and
model waits and under the publication leases. An indexed read uses the same local
approval and exclusion fence without opening a credential or contacting Google.
Only a complete union listing stamped with both the source generation and current
exclusion epoch admits copies; updating a registration cannot reuse its old listing.
A check reports delegation standing without document titles or counts.

`knowledge.source.update.crawl_readmit` explicitly readmits named, Directory-verified
people after SCIM reactivation. It requires the same current org and namespace
administration, refuses a still-deactivated contact and advances the durable refusal
epoch. It never republishes old copies: a complete subsequent crawl must establish
their permissions and freshness. Ordinary selector edits do not readmit anyone.
A dry run writes no registry or approval; an org whose refusal key has not been
initialized must complete its first real registration before a dry-run probe.

## Access follows the source

A source's `access`:
Expand Down Expand Up @@ -1242,8 +1261,8 @@ statements and census keys against these, and its tests pin each one:
- A person's connected accounts, the sources they own with every mirrored copy those
wrote, and their entries in every crawl are erased when an org admin asks; afterwards
no source row, list or members row names them, and a crawl reads them again only once
an org admin readmits them. The pull request that builds connected accounts and crawls
designs that erasure.
an org admin readmits them. Canonical-person deletion runs from the required SCIM
lifecycle; it never infers that a memory `user_id` names a sign-in subject.
- Private canonical-owner cleanup uses the exact `(org, subject, kind)` key independently
of namespace and end-user erasure. Disconnect and deactivation fence the generation,
clear the current credential, queue every owned source for retirement, then physically
Expand All @@ -1254,6 +1273,20 @@ statements and census keys against these, and its tests pin each one:
mean an end-user principal and never guesses a sign-in subject from an id or email.
Source retirement remains queued until the existing mirror retirement worker finishes;
account cleanup alone does not certify that every source copy or crawl entry is erased.
- Canonical-person deletion captures verified sign-in subject aliases and contact
ties before removing them. A bounded Directory lookup separately proves Workspace
customer and provider user ids; neither an email nor an end-user `user_id` is a
canonical subject. A required cleanup capsule fences crawls until raw document ACL,
group membership, source selector, approval and capsule histories have been scrubbed
and their censuses pass. Other writers' content and the registering administrator's
service-account key remain intact. The capsule is physically erased; only keyed
refusal intent survives until explicit readmission. Reactivation cannot clear it.
- The canonical-person control registry is cross-namespace and uses its real identity
axes rather than inventing a namespace or memory `user_id`. SCIM capsules have an
identity-lease-protected history reseal; exclusion keys and pending manifests reseal
without changing the keyed identities, cleanup phase or Keeper fence. An unknown
historical deletion remains a refusal pending verified identity repair, rather than
guessing whom an old email denoted.
- `erasure.request` refuses an `acl-` scope for every caller but a platform admin with
auth on, and `erasure.export` refuses one for every caller (an export would read the
mirror's chunks with no permission check); with auth off both refuse it for everyone.
Expand Down
Loading
Loading