Make queries more efficient

dev/TODO.md

@@ -1,21 +1,5 @@
 # TODO
 
-## High priority: Sync modes
-
-### Fast
-
-- Additive modes, to add new users’ perms quickly,
-  without the extraneous load on the database of a full sync
-- Take a list of usernames and/or email addresses as input,
-  query users on the instance for these,
-  then trigger a perms sync for found users
-- Query the instance for all new users, which do not yet have explicit perms
-- Query the instance for all new repos, which do not yet have explicit perms
-
-### Full: Overwrite all perms
-
-- Separate full sync mode with an arg
-
 ## High priority: Remote trigger on demand
 
 - Sourcegraph webhook for new user coming in v7.4.0

dev/memory-efficiency-generate.py

@@ -551,7 +551,7 @@ def list_external_services(client: src.SourcegraphClient) -> list[ExternalServic
             )
         )
     if not services:
-        raise SystemExit("No external services found on the Sourcegraph instance")
+        raise SystemExit("No code hosts found on the Sourcegraph instance")
     return services
 
 

dev/memory-efficiency.md

@@ -298,6 +298,39 @@ Important requirements:
 Expected benefit: replace hundreds or thousands of per-repo resolver SQL spans
 per request with one indexed `user_repo_permissions` join per user batch.
 
+The `get --users-without-explicit-perms` path also needs a cheaper presence
+check. Today it has to ask
+`User.permissionsInfo.repositories(source: API, first: 1)` for every candidate
+user, in aliased batches. Recent test runs show the client can parallelize
+those batches, but the Sourcegraph frontend / load balancer can still return
+502/503s under that resolver load. Add one or both direct APIs:
+
+```graphql
+type ExplicitRepositoryPermissionPresence {
+  userID: ID!
+  hasExplicitRepositoryPermissions: Boolean!
+}
+
+extend type Query {
+  explicitRepositoryPermissionPresenceForUsers(
+    userIDs: [ID!]!
+    source: PermissionSource = API
+  ): [ExplicitRepositoryPermissionPresence!]!
+
+  usersWithoutExplicitRepositoryPermissions(
+    createdAt: DateTimeFilter
+    source: PermissionSource = API
+    first: Int
+    after: String
+  ): UserConnection!
+}
+```
+
+Expected benefit: `src-auth-perms-sync get --users-without-explicit-perms`
+can either check explicit-permission presence for candidate users in one indexed
+batch query, or ask Sourcegraph for the filtered user set directly instead of
+probing every user through the expensive permissions connection resolver.
+
 The stress profile also needs attention on the write path. A purpose-built
 bulk overwrite API that accepts many repo/user edges at once, streams or stages
 the input server-side, and avoids repeated per-repo permission reconciliation
@@ -324,7 +357,10 @@ Request: add a bulk explicit-permissions read API that accepts many user IDs and
 returns compact permission edges (`userID`, `repositoryID`, `repositoryName`,
 `updatedAt`) for `source: API`, without resolving full `Repository` GraphQL
 objects. A single indexed query over `user_repo_permissions` joined to `repo`
-should be enough for each user batch.
+should be enough for each user batch. Also add a cheaper presence/filter path
+for `get --users-without-explicit-perms`: either `userID -> has explicit API
+repo permissions` for many users, or a direct query for users without explicit
+API repo permissions, optionally filtered by `createdAt`.
 
 Acceptance criteria:
 
@@ -336,6 +372,9 @@ Acceptance criteria:
   latency visible.
 - `src-auth-perms-sync` can replace its aliased
   `User.permissionsInfo.repositories(source: API)` calls with this API.
+- `src-auth-perms-sync get --users-without-explicit-perms` can stop probing
+  every candidate user through `User.permissionsInfo.repositories(source: API,
+  first: 1)`.
 - Follow-up: evaluate a bulk overwrite API for large full-set applies. The
   stress run planned roughly 10 million grants and observed
   `permsStore.upsertUserRepoPermissions-range1` averaging about 2.5s per call.

dev/test-end-to-end.py

@@ -1575,6 +1575,12 @@ def invalid_configuration_cases(config: EndToEndConfig) -> list[CommandCase]:
                 expected_exit_code=2,
                 must_contain=("choose at most one",),
             ),
+            CommandCase(
+                name="invalid-set-full-and-created-after",
+                arguments=("set", "--full", "--created-after", config.future_date),
+                expected_exit_code=2,
+                must_contain=("--full cannot be combined with --created-after",),
+            ),
             CommandCase(
                 name="invalid-user-filter-conflict",
                 arguments=("get", "--users", config.user, "--users-without-explicit-perms"),
@@ -1682,19 +1688,18 @@ def read_only_cases(config: EndToEndConfig) -> list[CommandCase]:
 def run_safe_set_cases(config: EndToEndConfig, runner: CommandPermutationRunner) -> None:
     runner.run(
         CommandCase(
-            name="set-explicit-full-no-op-apply",
+            name="set-created-after-no-op-apply",
             arguments=(
                 "set",
-                "--full",
                 "--created-after",
                 config.future_date,
                 "--apply",
                 "--no-backup",
                 "--parallelism",
                 str(config.parallelism),
             ),
-            expected_log_command="set_full",
-            must_contain=("No repos resolved across any mapping",),
+            expected_log_command="set_created_after",
+            must_contain=("No users selected",),
         )
     )