Skip to content

Google Drive sources crawl selected Workspace people and groups - #920

Draft
zlareb1 wants to merge 45 commits into
mainfrom
feat/workspace-crawl-publication
Draft

zlareb1 wants to merge 45 commits into
mainfrom
feat/workspace-crawl-publication

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Related: #883

Changelog category

  • New Feature

Changelog entry

Add delegated Workspace crawling for explicitly selected people and groups, with mirrored document permissions and durable person exclusions. Source creation remains disabled pending qualification.

Why

An organization-wide source needs the complete union of selected people's accessible files while preserving each document's permissions. A service-account key alone does not establish the registering administrator's continuing authority, a person's current identity or permission to ingest someone excluded by erasure.

Scope and risk

Delegated crawling extends the existing gdrive source kind. Registration binds its canonical registrant, signing-key lifecycle, verified Workspace customer, credential, selectors, items and source generation. Registration and mutations require current org and namespace administration. Publication rechecks authority after external work; indexed reads require a completed listing for the current source generation and exclusion epoch without opening provider credentials.

Directory expansion independently verifies provider identities, bounds nested groups, pages, response bytes and the full file union, and caps people at 2000. Delegation must grant the three fixed read-only Drive and Directory scopes and refuse the distinct Drive, user, group and group-member write scopes. These checks do not establish that every unrelated scope is absent.

A turn completes every eligible person's listing before publishing content, ACLs, membership or a completed-listing marker. Selected folders must stay visible. Retirement requires every eligible presenter to confirm the file is no longer reached, and consumes the shared document budget. Retry state belongs to the mirror and source identifier; whole enumeration does not grant a periodic retry every turn.

Canonical-person SCIM deletion captures verified identity before removing contact ties, then fences historical ACL, membership, selector, approval and capsule cleanup with residue checks. It preserves other writers' content and shared service-account credentials. Only keyed refusal intent survives; ordinary reactivation or selector changes cannot readmit a person. Missing historical identity remains deleted and pending verified repair rather than being reconstructed from a reused email.

The two exclusion tables are org-wide authorization registries keyed by org and (org, digest). Their sealed keys and refusal state therefore have no fabricated namespace or memory identity tuple. Canonical-person cleanup has a separate census classification, and encrypted personal capsules are physically erased. Root-secret resealing preserves refusal identity and pending cleanup. Restore reconciliation invalidates crawl authority before publication and refuses missing refusal proof.

Claims retain their existing Keeper key when a sent write lacks confirmed completion. Guarded sends must end before that key expires; nested claims retain the earliest deadline and every unanswered send. Acknowledged writes release immediately. Positive scoped statement timeouts also bound the receiver. At the defaults, the 24-second work window plus 29-second send bound fits inside the 60-second claim; insufficient windows refuse before work. No timeout or TTL default increases.

The main risks are identity confusion, stale authority after an await, incomplete expansion, lost refusal state after restore, and concurrent cleanup or publication. Fake-I/O regressions cover these paths; runtime qualification remains incomplete.

Compatibility, rollout and rollback

knowledge.source.add adds credential_mode="crawl" and crawl_options with directory_subject and person/group subjects. Update accepts replacement options and explicit crawl_readmit addresses. Add, update and rotate may return crawl_proof with subject hashes, read scopes, write denials and members. List and inspect disclose crawl options only to current org and namespace administrators. Check proves delegation standing without document titles or counts. The default credential mode stays shared; no operation or existing field is removed.

Additive migration 114 creates the two exclusion registries and adds approval/selection fields to sources, cleanup capsules to SCIM users, and source/exclusion generation markers to listings. Existing migrations 112 and 113 remain unchanged. Erasure, placement, row builders, resealing and restore companions are included. There is no new public erasure scope.

New settings and defaults:

Setting Default
knowledge_crawl_directory_max_response_bytes 1048576
knowledge_crawl_directory_max_pages 256
knowledge_crawl_max_groups 2000
knowledge_crawl_max_files 10000
knowledge_crawl_registry_scan_limit 4096
knowledge_crawl_registry_page_size 256
knowledge_crawl_lease_ttl_s 60
knowledge_crawl_lease_wait_s 5
knowledge_crawl_identity_capture_timeout_s 30
knowledge_crawl_person_cleanup_timeout_s 60

Apply the additive migration and deploy the validating build everywhere before qualification and source enablement. Crawl admission requires fleet support for schema 114. Configure the fixed read-only scope ceiling, Directory reader and explicitly selected people/groups. Provider registration and live acceptance are separate requirements. Unknown historical deletions cannot initialize an empty refusal registry. Restored data requires offline reconciliation before serving.

Older builds continue serving existing modes on the additive schema. Crawl rows leave legacy kind and credential columns empty so older builds cannot serve them. Rollback retains the new rows and durable exclusions. It must not discard refusal state or serve restored crawl copies with an unverifiable epoch. Fresh approval and a complete listing are required before those copies become readable again.

Tests and remaining verification

Fleet admission and the composed positive fixture require migration 114. Four focused cases passed at 113d808ad07357a55953f19926502474b093e494: 112 and 113 are rejected, 114 is admitted, and the composed fixture satisfies the real fleet gate. The fixture regression first failed with its stale version 112.

The SQL inventory classifies the crawl-erasure column-form UPDATE fragment by its exact file, function, verb, target expression and count. The fragment selects one of two source metadata columns and mutates the fixed, scoped source table; it does not name a cell control table. The scanner and complete inventory comparison remain unchanged, so unclassified sites and changed counts still fail. This metadata-only follow-up passes the whitespace check; final-head hosted validation is pending. No broad local source-scanning test was run.

Canonical migration replay, dry run, schema dump and source generation previously completed in the hosted generation run, and the verified generated outputs were adopted exactly. Replay, schema and mixed-version checks are still required for the newly numbered migration on the final composed head. No hand-edited schema dump replaces that validation.

Runtime qualification remains incomplete. Four earlier indexed-read positive assertions returned no admission; their mechanism remains unknown and their assertions remain required. The latest completed native privacy witness passed setup, build, signer and backend startup, then its canary summary reported SESSION_PROTECTION_REQUIRED. That permitted summary contains no HTTP status. An earlier witness reported 409 SESSION_PROTECTION_REQUIRED with retryable: false. Its raw HTTP client does not implement capture suppression or durable client privacy state. A genuine protecting client must be qualified before a capability declaration can be used. No raw-client declaration, guard change or replay substitutes for that behavior.

The witness retains real signed-user verification, canonical contacts, grants, fleet leadership, publication and HTTP reads, with controlled provider inputs. It requires an authorized positive before same-namespace and cross-scope privacy negatives, submits registration and sync once, and refuses unknown provider requests. Trusted contact fixtures do not cover upstream enrollment. Source creation stays disabled.

Full required hosted CI, a fresh independent whole permitted-source review and the unchanged positive assertions must pass on the final head before merge. Restricted serving dependencies, genuine native-client behavior and live provider qualification remain unresolved. Broad local suites are not run.

AI assistance in the code

  • AI-generated: An agent wrote the change.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • Required format, lint, typecheck, unit and integration gates pass at the final head.
  • Final migration replay, schema and mixed-version checks pass.
  • Native privacy and indexed-read positives pass.
  • Fresh independent review has no unresolved findings or nits.
  • Canonical generated companions are preserved.
  • Migration is additive; no merged migration is edited.
  • Operator steps, wire changes and validation limits are documented.
  • No plugin or frontend files change.
  • Commits are signed off.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 7af4238f1ae28db853b2cdaa61a0dc82f9bed366)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 56f93f0a66c1a64524176f8146c2818724feac0c)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit db318a243ada359c31a7f3b346b0b85292898c7b)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 841200529b6ae66d56d73f0dc741b3c37ebf7c8d)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 66fb97246035aa2b582983ef8fd0db5b9d894430)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 396467268abb9ccb56bf4ca77de4900790805cef)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit b0b6acc3f253b43a7319c10766bf63e9b1bebfde)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 6ae8f81fd1924e6277cc0bc853ea69943b38ad26)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 504c995f1200b20e223c3c4e545faea4b5d36cfb)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 9346322f70d4cf3f361064dce87973691b47b4eb)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 81ec69be978eb590a063a190f9e4ed932c402164)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 088102f08d6b097cfe1b9a9ab64a50e07dee9ffe)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 1694ff9d052c23cb4a75f4c902d35191c75b464e)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit d1286a815d5f2230cf1d5e9afbe78faf636380a8)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…tory

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 0ccdf915e29e115fc2410e8507b3283242d5c935)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 318e60a504c0d3a009df32d9c22e86cf6979fc4e)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit fb4b8544a88eb33d03198a77de020f7086d69f22)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit e7b5bde6eb91a460572e7cc9a78460a9e09cbb45)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 09aa2a9c891acae0f2fa3cf5ff960ab600b2b592)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 2d7355935fd13ae18b468512a8a03e34fe853104)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…ce generations

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit b76db25f1e003345395b2c69b314b775a0e9f06a)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 2a67107712a8eafde524e96e68d88db467c1b8b0)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 8c7fd3b899cf2231913a538518270e0c786cf7a1)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit b258dd6020d519174124fdc8b041c24fc351a02b)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Run the full app and inline workers on an owned backend, with controlled provider I/O. Require an authorized publication witness before ACL and tenant denials.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep token subjects aligned with verified login handles and the actual grant resolver. Use the established in-process origin and pin the workflow actions to verified commits.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Preserve the merged migrations and update the journal filename to the available slot.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the journal expectation aligned with the renumbered migration.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Preserve provisioning checks, canonical-person erasure and both migration histories. Keep the enclosing deadline rule and analyzer coverage from each branch.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 21, 2026 12:15am UTC

Request Review

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant