fix(security): deduplicate imports using authorized tracker body index

plugins/magpie-security/skills/issue-import/existing-tracker-dedup.md

@@ -5,35 +5,57 @@
 
 ## Step 2 — Deduplicate against existing <tracker> issues
 
-For each candidate `threadId`, check whether that ID already appears in an `<tracker>` issue body
-(the *"Security mailing list thread"* field holds it, as a `<mail-archive-url>/thread/<id>` URL or a textual note containing the Gmail `threadId`).
-Check every candidate in one pass: OR the `threadId`s into one `gh search issues` query,
-at most 6 IDs per query (GitHub search allows at most five `OR` operators).
-Write the ids as a JSON list to `<scratch>/thread-ids.json` and let the tool batch them
-(`<sm>` is `uv run --project <framework>/tools/security-management security-management issue-import`):
+For each candidate `threadId`, check the authoritative *Security mailing list
+thread* field of existing tracker issues. Do **not** use GitHub issue-body
+search as proof that a thread is untracked: search indexing can miss opaque
+identifiers enclosed in Markdown backticks.
+
+Write all candidate IDs to `<scratch>/thread-ids.json`. Use the configured
+vetted-ops policy and its `security-issue-import` caller to read the complete
+bounded GitHub tracker index:
 
 ```bash
-<sm> dedup-queries --thread-ids <scratch>/thread-ids.json --tracker <tracker>
+vetted-op-read --caller security-issue-import \
+  --save security-import-index.json issue-list-with-bodies
 ```
 
-It accepts the identifiers the mail-source backends emit — a Gmail or PonyMail id, a maildir file name, a root `Message-ID` such as `<report@example.org>` —
-and quotes each as a search phrase.
-**Exit `2` lists `rejected_ids`** it cannot search safely: stop and show them to the user; never search without them, or a tracked report re-imports.
-Run each batch's printed `command` (`gh search issues '<query>' --repo <tracker> --match body --limit 100 --json number,title,state,url,body`) as a plain command,
-saving its output to `<scratch>/dedup-hits-<n>.json`.
-
-Then attribute the hits with `<sm> dedup-attribute --thread-ids <scratch>/batch-<n>.json --hits <scratch>/dedup-hits-<n>.json`
-(the batch's `ids`; a hit can match more than one candidate; the same exit `2` applies).
-A batch that returns exactly 100 hits may be truncated: the tool reports `truncated`, no `untracked` ids, and `rerun` batches —
-run those rather than treating the unmatched `threadId`s as untracked.
+The operation reads up to 1000 issues, including open and closed issues.
+It must be explicitly authorized in the adopter's vetted-ops `[callers]`
+policy. Never bypass a missing grant with direct `gh` calls; ask the user to
+approve the minimal caller-policy change instead. The `--save` target must
+be a pre-existing, marked, owner-controlled directory, as documented in
+`tools/vetted-ops/README.md`. Do not print saved issue bodies into model
+context. Reuse the saved index for this import run, including Step 2b.
 
-If the search returns any hit for a candidate, the thread is already imported — skip it; do **not** propose re-importing.
-If the user explicitly passed `import thread:<id>` and the thread is already imported, tell the user and link the existing issue.
+Run the deterministic matcher against the saved JSON file:
 
-The remaining candidates proceed to the on-thread-handling check below;
-only threads that survive both filters reach the user in Step 5.
+```bash
+<sm> dedup-index \
+  --thread-ids <scratch>/thread-ids.json \
+  --issues <saved-index-path>
+```
 
-**Budget guardrail**: if the de-dup step knocks the candidate set down to zero, say so and stop, without reading any email bodies.
+If the adopting project customizes tracker body headings, also pass
+`--body-fields <scratch>/body-fields.json`, using the configured role-to-heading
+mapping. The matcher extracts the authoritative `mailing_thread` field and
+compares exact normalized thread identifiers. It returns only matching issue
+summaries, never full issue bodies.
+
+**Exit 2** for unsupported identifiers or invalid inputs means stop and
+surface the refusal. If `complete` is false, or any field is `undecided`,
+do not classify unmatched threads as new; ask for a manual resolution.
+A full 1000-issue index is potentially truncated and cannot support
+negative deduplication claims. On Jira or local-files trackers, the
+body-index operation is currently unsupported: stop and request a
+backend-supported check rather than assuming no matches.
+
+Any ID in `tracked` has already been imported — skip it and link the existing
+issue. If an explicitly requested thread is already tracked, tell the user.
+Only IDs in `untracked` may proceed to the on-thread check below.
+
+**Budget guardrail:** fetch one bounded tracker index per import run; never
+re-fetch the index for every candidate. If no candidates remain, stop without
+reading email bodies.
 
 ### 2-bis. Drop threads already answered on-thread without a tracker
 

plugins/magpie-security/skills/issue-import/screening-and-proposal.md

@@ -98,11 +98,16 @@ The per-candidate budget is ≤ 2 archive searches (whichever backend) for the p
    -from:notifications@github.com -from:noreply@github.com
    ```
 
-   Cross-reference the hits' `threadId`s against existing trackers in one batched query, as in Step 2 —
-   the `dedup-queries` batches (≤ 6 IDs per query) run as
-   `gh search issues "<query>" --repo <tracker> --match body --limit 100 --json number,body`,
-   with `dedup-attribute` attributing each hit to the `threadId` its body contains
-   (the *Security mailing list thread* field or the rollup's threadId backfill note).
+   Cross-reference those hits' `threadId`s against the **same saved
+   tracker body index** fetched in Step 2. Write their IDs as JSON and
+   run `<sm> dedup-index --thread-ids <scratch>/precedent-ids.json
+   --issues <saved-index-path>` (also supplying the configured
+   `--body-fields` mapping when applicable).
+
+   Do not repeat the `gh search issues` queries. An empty GitHub search
+   result does not prove that a tracker is absent. If the issue index
+   is incomplete or its authoritative mailing-thread fields are
+   ambiguous, do not classify the thread as tracker-free.
    The hits with **no** corresponding tracker are the "rejected without tracker" precedents.
 
 **Surfacing in Step 5.** For each precedent found, attach to the

plugins/magpie-vetted-ops/tools/vetted-ops/README.md

@@ -216,7 +216,7 @@ board_status_field_id = "PVTSSF_…"    # its Status field id
                            "body-field-get", "body-field-set"]
 "security-issue-fix"    = ["rollup-append", "rollup-fold",
                            "body-field-get", "body-field-set"]
-"security-issue-import" = ["rollup-append"]
+"security-issue-import" = ["issue-list-with-bodies", "rollup-append"]
 "security-issue-import-from-md" = ["rollup-append"]
 "security-issue-import-from-pr" = ["pr-view-with-body", "rollup-append"]
 "security-issue-invalidate" = ["rollup-append", "rollup-amend-latest"]
@@ -554,6 +554,7 @@ Nine operations exist for it:
 | `issue-create` | write | `title body` | GitHub: `gh api …/issues -f title=… -F body=@-`. |
 | `issue-reopen` | write | `number` | |
 | `issue-list-label` | read | `state label` | `issue-list` filtered by one configured label. |
+| `issue-list-with-bodies` | read | none | GitHub tracker only: up to 1000 open and closed issues with bodies; requires an explicit caller grant. Use `--save` to keep confidential bodies out of model context. Unsupported on Jira/local-files. |
 | `issue-search` | read | `query` | Plain words; a `:` is refused, so a `repo:` qualifier cannot re-target the search. |
 | `issue-body` | read | `number` | Non-GitHub only: the body as markdown sections, for `generate-cve-json --stdin`. |
 | `issue-attach` | write | `number file attachment` | Non-GitHub only: attach a workspace file under a plain name (no path, no `..`). |

plugins/magpie-vetted-ops/tools/vetted-ops/src/vetted_ops/ops.py

@@ -603,6 +603,27 @@ def _register(op: Op) -> None:
     )
 )
 
+_register(
+    Op(
+        name="issue-list-with-bodies",
+        params=(),
+        summary="Read up to 1000 tracker issues with bodies for authorized import deduplication.",
+        build=lambda cfg: [
+            "gh",
+            "issue",
+            "list",
+            "--repo",
+            _tracker(cfg),
+            "--state",
+            "all",
+            "--limit",
+            "1000",
+            "--json",
+            "number,title,state,url,body",
+        ],
+    )
+)
+
 _register(
     Op(
         name="issue-list-label",

plugins/magpie-vetted-ops/tools/vetted-ops/src/vetted_ops/tracker_backends.py

@@ -186,6 +186,7 @@ def _close_reason(reason: str) -> str:
 
 #: Tracker operations with no counterpart on a non-GitHub backend, and why.
 NOT_ON_BACKEND: dict[str, str] = {
+    "issue-list-with-bodies": "the non-GitHub list contract does not return issue bodies; refuse instead of declaring unmatched reports untracked",
     "milestone-list": "milestones are free values here; the policy's `milestones` list is the set",
     "milestone-create": "milestones are free values here and need no creating",
     "milestone-close": "milestones are free values here and are never closed",

tools/security-management/README.md

@@ -111,6 +111,7 @@ Every command prints one JSON object; rendered text is in its `text` field.
 | `prefilter --threads F` | Per thread: `decision` (`keep` / `drop`), `rule` (1 GHSA relay, 2 CVE-tool bookkeeping, 3 anything else), `class`, `pretag`; `dropped` and `counts`. Every message must match rule 2 to drop; a missing subject or sender keeps; `named_thread_id` is never dropped. |
 | `dedup-queries --thread-ids F [--tracker T]` | `batches` of at most six ids, each a quoted phrase OR'd into `query`, and the shell-quoted `command`; ids are Gmail / PonyMail ids, maildir names or root `Message-ID`s, and any other id exits `2` with `rejected_ids`. |
 | `dedup-attribute --thread-ids F --hits F` | `tracked` (id → hits), `untracked`; at 100 hits `truncated`, no `untracked`, and `rerun` batches; exit `2` with `rejected_ids` as above. |
+| `dedup-index --thread-ids F --issues F [--body-fields F] [--limit N]` | Match identifiers against a bounded, saved tracker issue-body index using the configured `mailing_thread` field. Returns matching issue summaries, `untracked` only when the index is complete, and `undecided`/`truncated` when a negative result is unsafe. Never prints private issue bodies. |
 | `relay-prescreen --message F --forwarders F` | `invoke_subskill`: false only when no enabled adapter's `sender_pattern` / `preamble_match` (first 400 characters) matches; an unresolved or invalid pattern invokes. |
 | `keyword --file F [--kind keywords\|email-local]` | `cleaned`, `empty`. |
 | `title --input F` | `title`: the subject without its mail prefixes. |

tools/security-management/src/security_management/issue_import/cli.py

@@ -68,6 +68,22 @@ def command(name: str, help_text: str, *flags: tuple[str, str]) -> None:
         ("--thread-ids", "JSON list of the batch's threadIds"),
         ("--hits", "the batch's `gh search issues --json number,title,state,url,body` output"),
     )
+    command(
+        "dedup-index",
+        "Step 2: exact-match threadIds against a saved, bounded tracker issue-body index",
+        ("--thread-ids", "JSON list of candidate threadIds"),
+        ("--issues", "JSON array from the authorized saved tracker read"),
+    )
+    commands.choices["dedup-index"].add_argument(
+        "--limit",
+        type=int,
+        default=1000,
+        help="maximum number of tracker issues fetched (default: 1000)",
+    )
+    commands.choices["dedup-index"].add_argument(
+        "--body-fields",
+        help="optional JSON object mapping canonical roles to configured body headings",
+    )
     command(
         "relay-prescreen",
         "Step 3: whether the via-forwarder sub-skill can match this root message",
@@ -146,6 +162,28 @@ def dispatch(args: argparse.Namespace) -> int:
             return refuse(result["refused"], rejected_ids=result["rejected_ids"])
         emit(result)
         return 0
+    if command == "dedup-index":
+        issues = read_json(args.issues)
+        body_fields = read_json(args.body_fields) if args.body_fields else None
+        if not isinstance(issues, list) or any(not isinstance(item, dict) for item in issues):
+            return refuse("issue index must be a JSON array of issue objects")
+        if body_fields is not None and (
+            not isinstance(body_fields, dict)
+            or any(not isinstance(k, str) or not isinstance(v, str) for k, v in body_fields.items())
+        ):
+            return refuse("body-fields must be a JSON object of string headings")
+        if args.limit < 1 or args.limit > 1000:
+            return refuse("issue index limit must be between 1 and 1000")
+        result = dedup.attribute_index(
+            read_json(args.thread_ids),
+            issues,
+            body_fields=body_fields,
+            limit=args.limit,
+        )
+        if not result["ok"]:
+            return refuse(result["refused"], rejected_ids=result["rejected_ids"])
+        emit(result)
+        return 0
     if command == "relay-prescreen":
         emit(relay.prescreen(read_object(args.message), read_object(args.forwarders)))
         return 0

tools/security-management/src/security_management/issue_import/dedup.py

@@ -35,6 +35,7 @@
 from collections.abc import Iterable, Mapping, Sequence
 from typing import Any
 
+from security_management.shared import fields, threads
 from security_management.shell import w
 
 BATCH = 6
@@ -106,3 +107,109 @@ def attribute(thread_ids: Iterable[object], hits: Sequence[Mapping[str, Any]]) -
         "rerun": batches(unmatched, half) if truncated and unmatched else [],
         "rejected_ids": bad,
     }
+
+
+def attribute_index(
+    thread_ids: Iterable[object],
+    issues: Sequence[Mapping[str, Any]],
+    *,
+    body_fields: Mapping[str, str] | None = None,
+    limit: int = 1000,
+) -> dict[str, Any]:
+    """Match IDs against a bounded, complete issue-body index.
+
+    Never expose body contents in the result. Missing or ambiguous
+    authoritative fields prevent a definitive negative result when
+    they cannot be interpreted safely.
+    """
+    good, bad = valid_ids(thread_ids)
+    if bad:
+        return {
+            "ok": False,
+            "refused": "thread ids no mail-source backend emits",
+            "rejected_ids": bad,
+        }
+
+    tracked: dict[str, list[dict[str, Any]]] = {}
+    undecided: list[dict[str, Any]] = []
+
+    for issue in issues:
+        summary = {key: issue.get(key) for key in ("number", "title", "state", "url")}
+        if not isinstance(issue.get("body"), str):
+            undecided.append(
+                {
+                    "number": issue.get("number"),
+                    "reason": "missing or invalid issue body",
+                }
+            )
+            continue
+
+        parsed = fields.read(
+            issue["body"],
+            ["mailing_thread"],
+            body_fields,
+        )
+
+        if parsed["undecided"]:
+            undecided.append(
+                {
+                    "number": issue.get("number"),
+                    "reason": "ambiguous mailing_thread field",
+                }
+            )
+            continue
+
+        field_value = parsed["values"].get("mailing_thread", "")
+        if not field_value:
+            continue
+
+        identity = threads.classify(field_value)
+        if identity["needs_decision"]:
+            undecided.append(
+                {
+                    "number": issue.get("number"),
+                    "reason": "mailing_thread needs a decision",
+                }
+            )
+            continue
+
+        known = {
+            value
+            for value in (
+                identity["thread_id"],
+                identity["root_message_id"],
+            )
+            if value
+        }
+
+        # Archive-only entries carry the PonyMail identifier in the
+        # /thread/<id> path, rather than in a Gmail thread note.
+        archive_urls = [identity["archive_url"]]
+        for secondary in identity["secondary_threads"]:
+            if secondary.get("thread_id"):
+                known.add(secondary["thread_id"])
+            archive_urls.append(secondary.get("url"))
+
+        for url in archive_urls:
+            if not isinstance(url, str):
+                continue
+            match = re.search(r"/thread/([A-Za-z0-9_.,:=+-]{6,128})(?:[/?#]|$)", url)
+            if match:
+                known.add(match.group(1))
+
+        for candidate in good:
+            if candidate in known:
+                tracked.setdefault(candidate, []).append(summary)
+
+    truncated = len(issues) >= limit
+    complete = not truncated and not undecided
+
+    return {
+        "ok": True,
+        "tracked": tracked,
+        "untracked": [candidate for candidate in good if candidate not in tracked] if complete else [],
+        "complete": complete,
+        "truncated": truncated,
+        "undecided": undecided,
+        "rejected_ids": [],
+    }