diff --git a/docs/designs/2026-10-15-security-tracker-backends-plan.md b/docs/designs/2026-10-15-security-tracker-backends-plan.md index 8bbea9cc7..86d2acbdb 100644 --- a/docs/designs/2026-10-15-security-tracker-backends-plan.md +++ b/docs/designs/2026-10-15-security-tracker-backends-plan.md @@ -31,7 +31,7 @@ Per the [designs convention](README.md), this plan is folded into the design and | 10 | Setup: Step 3a — choose backend, detect format, confirm access, write descriptor | `plugins/magpie-setup/skills/setup/config.md` | Done | | 11 | Taxonomy: `contract:security-tracker` in the validator, capability docs, labeler | `tools/skill-and-tool-validator`, `docs/labels-and-capabilities.md`, `.github/labeler.yml` | Done | | 12 | Token restamp for the edited skills; `prek run --all-files` | `measured_tokens:` frontmatter | Done | -| 12a | vetted-ops: `[tracker]` policy, translation of every tracker operation, six new operations, refusal of the rest, tests incl. an end-to-end run against the local tracker | `tools/vetted-ops/` | Done | +| 12a | vetted-ops: `[tracker]` policy, translation of every tracker operation, nine new operations, refusal of the rest, tests incl. an end-to-end run against the local tracker | `tools/vetted-ops/` | Done | | 12b | Contract: skill-command → vetted-operation → backend table; shared block points skills at vetted operations | `tools/security-tracker/contract.md`, `tools/dev/blocks/tracker-backend.md` | Done | | 13 | Security-skill eval suites (self-eval) — 308/313; the 5 failures come from prompt files this change does not touch | `tools/skill-evals/evals/security-*` | Done (cross-model pass to do) | | 14 | Create the `contract:security-tracker` label on `apache/magpie` | GitHub labels | To do (maintainer) | diff --git a/docs/designs/2026-10-15-security-tracker-backends.md b/docs/designs/2026-10-15-security-tracker-backends.md index 2a7b5d9e7..0d595d4f1 100644 --- a/docs/designs/2026-10-15-security-tracker-backends.md +++ b/docs/designs/2026-10-15-security-tracker-backends.md @@ -56,12 +56,13 @@ named body fields, the single status rollup, attachments for the CVE record, and Skills never call a backend CLI. Every tracker `gh` command in a skill maps to a [vetted operation](../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations), and vetted-ops translates that operation for the backend its policy names (`[tracker] platform`). - The translation is reviewed code: the policy supplies the backend's tool directory and pinned target, never a command, + The translation is reviewed code: the policy supplies only the pinned target, and the backend's code runs from the tool installed beside vetted-ops, never from a path a policy names, so the closed-shape guarantees — fixed program, unaddressable target, parameters as argv elements, bodies on stdin, reads unreachable from the write entry point — hold on all three backends. - Six operations were added so every tracker shape the skills use has a vetted form: + Nine operations were added so every tracker shape the skills use has a vetted form: `issue-create`, `issue-reopen`, `issue-list-label`, `issue-search` (plain words, no `:` qualifiers), - `tracker-pending` and `tracker-publish`. + `issue-body` and `issue-attach` (so CVE JSON generation works without GitHub), + `tracker-doctor` (the pre-flight on another backend), `tracker-pending` and `tracker-publish`. Tracker operations with no counterpart (boards, milestone administration, the roster, `rollup-fold`) refuse by name, and a test derives the tracker operations from the catalogue so a new one cannot go unmapped. 5. **Skills switch through one shared block**, `tools/dev/blocks/tracker-backend.md`, @@ -131,7 +132,7 @@ the Groovy bridge stays the general `contract:tracker` write path for the `issue one question in the existing batch picks the backend. For local files it finds the folder, runs `detect`, shows the decisions and notes, asks who can read the repository, and writes the descriptor only after that answer. The descriptor is the one file this step writes outside the personal layer. It is left uncommitted, and the first publish proposal includes it. -For both non-GitHub backends the step also shows the vetted-ops `[tracker]` table and the six new caller grants, and writes them only on confirmation. +For both non-GitHub backends the step also shows the vetted-ops `[tracker]` table and the nine new caller grants, and writes them only on confirmation. ## Alternatives considered diff --git a/plugins/magpie-security/skills/cve-allocate/SKILL.md b/plugins/magpie-security/skills/cve-allocate/SKILL.md index 02cc2acf9..75f13022d 100644 --- a/plugins/magpie-security/skills/cve-allocate/SKILL.md +++ b/plugins/magpie-security/skills/cve-allocate/SKILL.md @@ -21,7 +21,7 @@ argument-hint: "[issue-number] [CVE-YYYY-NNNNN]" capability: capability:resolve surface_hash: sha256:25c2286c4e159c66 license: Apache-2.0 -measured_tokens: 7366 +measured_tokens: 7638 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-deduplicate/SKILL.md b/plugins/magpie-security/skills/issue-deduplicate/SKILL.md index 723084571..691610717 100644 --- a/plugins/magpie-security/skills/issue-deduplicate/SKILL.md +++ b/plugins/magpie-security/skills/issue-deduplicate/SKILL.md @@ -25,7 +25,7 @@ argument-hint: "[kept-issue] [duplicate-issue]" capability: capability:resolve surface_hash: sha256:3c9f502b802d2037 license: Apache-2.0 -measured_tokens: 5989 +measured_tokens: 6261 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-fix/SKILL.md b/plugins/magpie-security/skills/issue-fix/SKILL.md index ab1b82263..fedf56bbd 100644 --- a/plugins/magpie-security/skills/issue-fix/SKILL.md +++ b/plugins/magpie-security/skills/issue-fix/SKILL.md @@ -23,7 +23,7 @@ capability: - capability:resolve surface_hash: sha256:550eabe3b9bb3689 license: Apache-2.0 -measured_tokens: 7163 +measured_tokens: 7435 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-import-from-md/SKILL.md b/plugins/magpie-security/skills/issue-import-from-md/SKILL.md index b8a5476a4..8129c9fea 100644 --- a/plugins/magpie-security/skills/issue-import-from-md/SKILL.md +++ b/plugins/magpie-security/skills/issue-import-from-md/SKILL.md @@ -27,7 +27,7 @@ argument-hint: "[path-to-markdown-file]" capability: capability:intake surface_hash: sha256:e2437e84cdf98aa5 license: Apache-2.0 -measured_tokens: 7095 +measured_tokens: 7367 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-import-from-pr/SKILL.md b/plugins/magpie-security/skills/issue-import-from-pr/SKILL.md index 579a0450b..150d5dceb 100644 --- a/plugins/magpie-security/skills/issue-import-from-pr/SKILL.md +++ b/plugins/magpie-security/skills/issue-import-from-pr/SKILL.md @@ -20,7 +20,7 @@ argument-hint: "[pr-number] [repo:owner/name]" capability: capability:intake surface_hash: sha256:c746e4aecc3cf315 license: Apache-2.0 -measured_tokens: 8447 +measured_tokens: 8719 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-import-from-scan/SKILL.md b/plugins/magpie-security/skills/issue-import-from-scan/SKILL.md index 337a6942e..9ea423307 100644 --- a/plugins/magpie-security/skills/issue-import-from-scan/SKILL.md +++ b/plugins/magpie-security/skills/issue-import-from-scan/SKILL.md @@ -19,7 +19,7 @@ argument-hint: "[scan-source ...] (one or more GitHub issues and/or report fold capability: capability:intake surface_hash: sha256:7adfbe73f79606b1 license: Apache-2.0 -measured_tokens: 5964 +measured_tokens: 6236 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-import-via-forwarder/SKILL.md b/plugins/magpie-security/skills/issue-import-via-forwarder/SKILL.md index cd152b64b..895bf6a0b 100644 --- a/plugins/magpie-security/skills/issue-import-via-forwarder/SKILL.md +++ b/plugins/magpie-security/skills/issue-import-via-forwarder/SKILL.md @@ -20,7 +20,7 @@ when_to_use: | capability: capability:intake surface_hash: sha256:43894008501eb56d license: Apache-2.0 -measured_tokens: 6213 +measured_tokens: 6485 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-import/SKILL.md b/plugins/magpie-security/skills/issue-import/SKILL.md index 4fb44b2d8..24407d825 100644 --- a/plugins/magpie-security/skills/issue-import/SKILL.md +++ b/plugins/magpie-security/skills/issue-import/SKILL.md @@ -20,7 +20,7 @@ argument-hint: "[import] [last Nd|all] [skip threadId]" capability: capability:intake surface_hash: sha256:e9c563d494821baf license: Apache-2.0 -measured_tokens: 9864 +measured_tokens: 10136 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-invalidate/SKILL.md b/plugins/magpie-security/skills/issue-invalidate/SKILL.md index 0d6b8e2f5..ec7f88243 100644 --- a/plugins/magpie-security/skills/issue-invalidate/SKILL.md +++ b/plugins/magpie-security/skills/issue-invalidate/SKILL.md @@ -20,7 +20,7 @@ argument-hint: "[issue-number]" capability: capability:resolve surface_hash: sha256:43a163bcbdb06f11 license: Apache-2.0 -measured_tokens: 7582 +measured_tokens: 7854 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-sync/SKILL.md b/plugins/magpie-security/skills/issue-sync/SKILL.md index 52f07ebff..77190a99d 100644 --- a/plugins/magpie-security/skills/issue-sync/SKILL.md +++ b/plugins/magpie-security/skills/issue-sync/SKILL.md @@ -25,7 +25,7 @@ argument-hint: "[issue-number]" capability: capability:intake surface_hash: sha256:5c4aad61d458d98f license: Apache-2.0 -measured_tokens: 6734 +measured_tokens: 7006 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/issue-triage/SKILL.md b/plugins/magpie-security/skills/issue-triage/SKILL.md index cc5499958..2a1d038af 100644 --- a/plugins/magpie-security/skills/issue-triage/SKILL.md +++ b/plugins/magpie-security/skills/issue-triage/SKILL.md @@ -23,7 +23,7 @@ when_to_use: | capability: capability:triage surface_hash: sha256:4fb1fcfc79d7f275 license: Apache-2.0 -measured_tokens: 7130 +measured_tokens: 7402 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). diff --git a/plugins/magpie-security/skills/tracker-stats-dashboard/SKILL.md b/plugins/magpie-security/skills/tracker-stats-dashboard/SKILL.md index 6a11079c7..1fef821d2 100644 --- a/plugins/magpie-security/skills/tracker-stats-dashboard/SKILL.md +++ b/plugins/magpie-security/skills/tracker-stats-dashboard/SKILL.md @@ -18,7 +18,7 @@ when_to_use: | capability: capability:stats surface_hash: sha256:779340f8eb985eeb license: Apache-2.0 -measured_tokens: 4232 +measured_tokens: 4591 --- -Read `tracker.platform` from `/project.md` (default `github`) before the first tracker operation. +Read `tracker.platform` from `/project.md` before the first tracker operation; unset or `TODO` means `github`. With `github`, every recipe below runs as written. With `local-files` or `jira`, never run a `gh` command against ``: look the command up in the [skill-command → vetted-operation table](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations) and run that vetted operation instead — `vetted-op-read` for reads, `vetted-op-tracker` for the rollup, body-field and publish operations, `vetted-op` for the other writes. -Vetted-ops translates it for the configured backend; `` issue numbers become the backend's IDs, and the returned `url` is the link. +Vetted-ops translates it for the configured backend; issue numbers become the backend's IDs (`SEC-0012`, `FOOSEC-12`), and the returned `url` is the link. Operations on `` (PRs, code search) do not change. A refusal — the backend's confidentiality guard, or an operation the backend does not have — stops the step: report it, never route around it. +**Pre-flight on another backend.** The GitHub checks in Step 0 — `gh api repos/` access, the collaborator roster, project-board ids — do not apply. +Run the `tracker-doctor` vetted operation instead and stop on any failing check; take the security-team roster from `/release-trains.md`, and treat the project board as disabled. + +**CVE JSON on another backend.** `generate-cve-json --attach` reads and attaches through GitHub, so do not run it. +Instead: save the body with the `issue-body` vetted operation (`vetted-op-read … --save cve-body issue-body `), +run `generate-cve-json --stdin --title "" --output /.json` on that saved file — +adding `--review` when the issue's labels (from `issue-view`) include the release-vote label or a forward-state label the generator's config names, since `--stdin` cannot see labels and would otherwise produce DRAFT — +and attach the result with the `issue-attach` vetted operation (` /.json .json`). + **Publish local tracker changes before ending.** With `local-files`, nothing reaches the team until it is committed. -If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files, +If this run wrote to the tracker, run the `tracker-pending` vetted operation as the last step, show the changed files and any unpushed commits, draft a commit message naming the trackers and the action, ending with the project's commit-attribution trailer, and ask. Only on the user's confirmation run `tracker-publish` with that message file (Git: commit the tracker folder and push; SVN: commit, which publishes). +**This skill is GitHub-only today.** Its fetch pipeline (`run.sh` and the scripts it runs) reads `` with `gh` directly, so the vetted-operation translation above does not reach it. +With `tracker.platform` set to `local-files` or `jira`, stop before fetching anything and tell the user the dashboard does not support that backend yet; never point the pipeline at a GitHub repository instead. + *Renderer* configuration (bucket granularity, milestones, categories, scope labels, triage keywords, …) lives in a separate YAML file at `.apache-magpie-overrides/security-tracker-stats.yaml` (path set by `tracker_stats_config:` in [`/security-tracker-stats.md`](../../../magpie-setup/templates/security-tracker-stats.md)). diff --git a/plugins/magpie-setup/skills/setup/config.md b/plugins/magpie-setup/skills/setup/config.md index 8c89e982e..c8b3a0cb0 100644 --- a/plugins/magpie-setup/skills/setup/config.md +++ b/plugins/magpie-setup/skills/setup/config.md @@ -215,12 +215,27 @@ Step 3c's harness command files are the two named exceptions). Never stage anyth ## Step 3a — Security tracker backend Run this when a `security-*` skill is in scope and `tracker.platform` in `project.md` is still unset or `TODO`. +A freshly scaffolded `project.md` carries `platform: TODO`, so the question is part of every first setup; +a `TODO` the user skips reads as `github`. Put one question in Step 3's batch, offering the three backends of the [security-tracker contract](../../../../tools/security-tracker/contract.md): -- **`github`** (default) — a private GitHub repository, `tracker_repo`. +- **`github`** (default) — a private GitHub repository, `tracker_repo`; write `tracker.platform: github`. - **`local-files`** — Markdown files in a folder of this repository (Git or SVN), committed and pushed at the end of each session. -- **`jira`** — a Jira project whose issues carry an issue-security level; ask for `url`, `project` and `security_level`, and run the backend's `doctor`. +- **`jira`** — a Jira project whose issues carry an issue-security level; follow the Jira steps below. + +For **`jira`**: + +1. **Ask for the target**: `tracker.jira.url`, `tracker.jira.project`, `tracker.jira.security_level` and `tracker.jira.deployment` (`server` or `cloud`). + Write them, with `tracker.platform: jira`, to the personal layer's `project.md`. +2. **Make every label Jira-safe.** Jira labels cannot contain whitespace and the backend refuses any label that does, so the template's names (`needs triage`, `cve allocated`, …) fail on the first write. + Map each configured label to a whitespace-free name (`needs triage` → `needs-triage`) everywhere it is configured: + the `tracker.labels.*` values and the `scope_detection.labels` keys in `project.md`, + the `any_label` lists in `security-tracker-stats.md`, + and the vetted-ops policy's `[values] labels` list. + Show the mapping and apply it on the user's confirmation; the policy change goes in the same diff as the `[tracker]` table below. +3. **Run `doctor`**: `uv run --project /tools/jira magpie-jira-tracker --url --project --security-level --deployment doctor`, and show its checks. + Do not declare the backend ready until every label is mapped and `doctor` passes. For **`local-files`**: @@ -246,10 +261,10 @@ For **`local-files`**: For both `local-files` and `jira`, the skills reach the tracker only through vetted operations ([mapping](../../../../tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations)). When a vetted-ops policy exists, show the `[tracker]` table it needs — -`platform`, the backend's absolute `project` directory, and `root` or the `jira_*` target +`platform`, and `root` or the `jira_*` target (the backend's code comes from the tool installed beside vetted-ops, never from the policy) ([vetted-ops → Tracker backends](../../../../tools/vetted-ops/README.md#tracker-backends)) — and write it only on the user's confirmation; the policy is a security control, not configuration this step may change unasked. -Grant the security callers `issue-create`, `issue-search`, `issue-list-label`, `issue-reopen`, `tracker-pending` and `tracker-publish` in the same diff. +Grant the security callers `issue-create`, `issue-search`, `issue-list-label`, `issue-reopen`, `issue-body`, `issue-attach`, `tracker-doctor`, `tracker-pending` and `tracker-publish` in the same diff. ## Step 3b — Record what this run reconciled diff --git a/plugins/magpie-setup/templates/project.md b/plugins/magpie-setup/templates/project.md index c123d9f62..0bdba0408 100644 --- a/plugins/magpie-setup/templates/project.md +++ b/plugins/magpie-setup/templates/project.md @@ -457,8 +457,9 @@ tracker: # Git or SVN working copy (tools/local-tracker/). `jira`: a Jira project # whose issues carry an issue-security level (tools/jira/). # `/magpie-setup config` asks and, for local-files, learns the format of - # the reports already in the folder. - platform: github + # the reports already in the folder. `TODO` (not yet asked) reads as + # `github`; the question writes the confirmed choice here. + platform: TODO # local-files only. local: diff --git a/plugins/magpie-vetted-ops/tools/jira/README.md b/plugins/magpie-vetted-ops/tools/jira/README.md new file mode 100644 index 000000000..0388b76d0 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/jira/README.md @@ -0,0 +1,405 @@ + + + + +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [JIRA bridge](#jira-bridge) + - [Prerequisites](#prerequisites) + - [Layout](#layout) + - [Invocation](#invocation) + - [Read subcommands](#read-subcommands) + - [`search `](#search-jql) + - [`issue `](#issue-key) + - [`projects`](#projects) + - [Write subcommands](#write-subcommands) + - [`comment --body-file `](#comment-key---body-file-path) + - [`transition `](#transition-key-transition-name) + - [`label --add --remove `](#label-key---add-name---remove-name) + - [`assign `](#assign-key-username) + - [`field --value ` / `--value-json `](#field-key-field-name---value-value----value-json-json) + - [`attach `](#attach-key-file) + - [People and contributor-activity reads](#people-and-contributor-activity-reads) + - [Security tracker (`magpie-jira-tracker`)](#security-tracker-magpie-jira-tracker) + - [Configuration](#configuration) + - [Output contract](#output-contract) + - [Write-path discipline](#write-path-discipline) + - [Testing](#testing) + - [Cross-references](#cross-references) + + + + + +# JIRA bridge + +**Capability:** contract:tracker + contract:people + contract:security-tracker + +**Kind:** implementation + +**Vendor:** Atlassian + +JIRA REST helpers for the `issue-*` skill family. +Adopters with JIRA-based issue trackers wire this in as their +tracker bridge; adopters using GitHub Issues or other trackers +contribute a parallel `tools//` directory. + +The bridge provides both **read** and **write** subcommands. +The read-only contributor-activity queries of `contract:tracker` run in Python, in the `jira` backend of [`tools/contributor-metrics`](https://github.com/apache/magpie/blob/main/tools/contributor-metrics/README.md#backends), through this package's REST client, `jira_bridge.rest`; the profile lookup of `contract:people` is a single REST read. See [People and contributor-activity reads](#people-and-contributor-activity-reads). +The same package ships `magpie-jira-tracker`, the private-Jira backend of the [security-tracker contract](https://github.com/apache/magpie/blob/main/tools/security-tracker/contract.md), whose reads and writes go through that client too — see [Security tracker](#security-tracker-magpie-jira-tracker). +Write operations require `JIRA_API_TOKEN` and follow the same +write-path discipline as the GitHub bridge: every mutation is +gated on explicit user confirmation in the calling skill — the +bridge only executes confirmed actions. + +## Prerequisites + +- **Runtime:** Groovy 4.x+ on `PATH` (`groovy tools/jira/bridge.groovy …`); `@Grab` pulls the HTTP-client dependencies on first run, no separate install step. Python 3.11+ via `uv` (standard library only) for `jira_bridge.rest`, `magpie-jira-tracker` and the pytest harness. +- **CLIs:** `groovy` (4.x — the `@Grab` coordinate uses the `org.apache.groovy` group ID); `uv` only to run the tests. +- **Credentials / auth:** `ISSUE_TRACKER_URL` (required) and `ISSUE_TRACKER_PROJECT` exported by the caller; write subcommands require `JIRA_API_TOKEN` (`JIRA_AUTH_SCHEME` = `Basic` default, or `Bearer` for ASF PATs). Anonymous-read trackers need no auth for read subcommands. +- **Network:** the configured `` JIRA host (e.g. `issues.apache.org/jira`); `@Grab` reaches Maven Central on first run to resolve dependencies. +- **Optional:** `groovy` on `PATH` for the pytest suite — tests auto-skip when Groovy is absent. + +## Layout + +```text +tools/jira/ +├── README.md (this file) +├── bridge.groovy (Groovy reference implementation) +├── pyproject.toml (Python test harness config) +├── src/jira_bridge/ (Python REST client, rest.py; the security tracker, +│ security_tracker.py + tracker_client.py + markup.py) +└── tests/ (pytest test suite) +``` + +Other languages (Python, Bash + curl) are welcome via PR. + +## Invocation + +```bash +groovy tools/jira/bridge.groovy [args] +``` + +The Groovy implementation uses `@Grab` for HTTP client dependencies +— no separate install step. Requires Groovy 4.x or newer on +`PATH` (the `@Grab` coordinate uses `org.apache.groovy`, which +is the Groovy 4 group ID). + +## Read subcommands + +### `search ` + +Run a JQL query against `` and emit matching issues +as JSON to stdout: + +```bash +groovy tools/jira/bridge.groovy search \ + 'project = AND status = Open AND resolution = Unresolved' +``` + +Output (truncated): + +```json +{ + "total": 42, + "issues": [ + {"key": "-9999", "title": "...", "status": "Open", "components": [...], "fixVersion": "..."}, + ... + ] +} +``` + +The `--limit ` flag caps the result count (default: 50). + +### `issue ` + +Fetch a single issue's full state (body, comments, attachments +list, labels, fixVersion, etc.) as JSON: + +```bash +groovy tools/jira/bridge.groovy issue -9999 +``` + +Output is the JIRA REST `/rest/api/2/issue/` response, +shaped for skill consumption. + +### `projects` + +List the JIRA projects available at the configured +`` URL. Useful during initial adoption to confirm +the project key is correct. + +```bash +groovy tools/jira/bridge.groovy projects +``` + +## Write subcommands + +All write subcommands require `JIRA_API_TOKEN` to be set and +follow the write-path discipline described below. + +### `comment --body-file ` + +Post a comment on an issue. The comment body is read from a file +to avoid shell-quoting issues: + +```bash +groovy tools/jira/bridge.groovy comment FOO-9999 --body-file /tmp/comment.txt +``` + +Output: + +```json +{"ok": true, "key": "FOO-9999", "commentId": "12345"} +``` + +### `transition ` + +Move an issue to a new workflow state. The transition name is +resolved case-insensitively against the issue's available +transitions: + +```bash +groovy tools/jira/bridge.groovy transition FOO-9999 "Resolve Issue" +``` + +Output: + +```json +{"ok": true, "key": "FOO-9999", "transition": "Resolve Issue", "transitionId": "21"} +``` + +If the transition name does not match any available transition, +the command exits with an error listing the valid names. + +### `label --add --remove ` + +Toggle labels on an issue. Both `--add` and `--remove` can be +specified multiple times in a single call: + +```bash +groovy tools/jira/bridge.groovy label FOO-9999 --add security --remove needs-triage +``` + +Output: + +```json +{"ok": true, "key": "FOO-9999", "added": ["security"], "removed": ["needs-triage"]} +``` + +Uses JIRA's atomic `update` API — no read-modify-write race. + +### `assign ` + +Set the assignee on an issue. Data Center only — Cloud uses +`accountId`, which is not currently supported: + +```bash +groovy tools/jira/bridge.groovy assign FOO-9999 jdoe +``` + +Output: + +```json +{"ok": true, "key": "FOO-9999", "assignee": "jdoe"} +``` + +### `field --value ` / `--value-json ` + +Edit a single field (including custom fields) on an issue. +Use `--value` for plain string/number values. Use `--value-json` +for structured values (priority, version, single-select, user +picker, etc.): + +```bash +# String value +groovy tools/jira/bridge.groovy field FOO-9999 customfield_10100 --value "high" + +# Structured value (e.g. priority) +groovy tools/jira/bridge.groovy field FOO-9999 priority --value-json '{"name":"High"}' + +# Array value (e.g. fixVersions) +groovy tools/jira/bridge.groovy field FOO-9999 fixVersions --value-json '[{"name":"1.2.3"}]' +``` + +Output: + +```json +{"ok": true, "key": "FOO-9999", "field": "priority", "value": {"name": "High"}} +``` + +### `attach ` + +Attach a file to an issue: + +```bash +groovy tools/jira/bridge.groovy attach FOO-9999 /tmp/report.txt +``` + +Output: + +```json +{"ok": true, "key": "FOO-9999", "attachments": [{"id": "99", "filename": "report.txt"}]} +``` + +## People and contributor-activity reads + +The read-only activity queries of [`contract:tracker`](https://github.com/apache/magpie/blob/main/tools/tracker/README.md) and the profile lookup of [`contract:people`](https://github.com/apache/magpie/blob/main/tools/people/README.md), against the same Jira Data Center REST API v2 the bridge uses and with the same configuration (`ISSUE_TRACKER_URL`, `ISSUE_TRACKER_PROJECT`, `JIRA_API_TOKEN`, `JIRA_AUTH_SCHEME`). +The per-person streams are implemented by the `jira` backend of [`tools/contributor-metrics`](https://github.com/apache/magpie/blob/main/tools/contributor-metrics/README.md#backends) over `jira_bridge.rest`; the others are plain REST reads a skill makes. + +`jira_bridge.rest` treats the token as the user's personal credential and the URL as project configuration, which anyone with write access can commit: + +- the URL must be `https://` (`http://` only for `localhost`, `127.0.0.1` or `::1`); +- the token is sent only to the host the user confirmed for it — `JIRA_API_HOST`, or a `host=` line beside a `token=` line in `~/.config/apache-magpie/jira-token` — and only on requests to that host; for any other host it is withheld, reads go out anonymously, and the caller is told why; +- redirects are refused, so the token never follows one; +- the token never appears in an error message. +`` is the person's Jira username, validated against `[A-Za-z0-9._@+-]{1,255}` before it reaches a query. + +| Verb | Jira resolution | +|---|---| +| tracker `list_filed(, since, end)` | `GET /rest/api/2/search` with JQL `project = "" AND reporter = "" AND created >= "" AND created <= " 23:59"` | +| tracker `list_triaged(, since, end)` | `GET /rest/api/2/search?expand=changelog&fields=comment,…` over `project = "" AND updated >= "" AND created <= " 23:59"`; an issue the user did not report counts when its changelog shows the user changing `status`, `labels`, `component`, `priority`, `assignee`, `resolution` or `Fix Version`, or the user commented, inside the window — dated by the first such action | +| tracker `list_commented(, since, end)` | the same search; an issue counts when the user commented inside the window, dated by the first such comment | +| tracker `list_created(since, until)` | `project = "" AND created >= "" AND created <= " 23:59"`; `author_first_time` from one `reporter = "" AND created < ""` count per author | +| tracker `first_reply()` | `GET /rest/api/2/issue//comment`, the first comment by a rostered maintainer (Jira has no author-association signal) | +| people `get_profile()` | `GET /rest/api/2/user?username=` → `displayName`; a 404 means the account does not exist. Organisation, website and linked accounts are not exposed: `null`. | +| people `list_collaborators`, `add_team_member` | not provided — Jira has no repository collaborators or forge teams (`NotApplicable`) | + +Jira has no pull requests: authored changes and reviews come from the project's code host through `contract:change-request`. +Comment bodies are read inside the backend to find pushback candidates and never leave it. + +## Security tracker (`magpie-jira-tracker`) + +The `jira` backend of the [security-tracker contract](https://github.com/apache/magpie/blob/main/tools/security-tracker/contract.md): a project's security trackers in a **private Jira project** whose issues carry an issue-security level. +It exposes the same subcommands and JSON as the [`local-files` backend](../local-tracker/README.md), and skills reach it through vetted operations, never directly +([mapping](https://github.com/apache/magpie/blob/main/tools/security-tracker/contract.md#from-skill-commands-to-vetted-operations)). + +```bash +uv run --project /tools/jira magpie-jira-tracker --url --project --security-level [args…] +``` + +**Why the level is mandatory.** Jira projects are often browsable by any logged-in user; the issue-security level is what makes a tracker private. +`create` always sets the configured level and refuses to run without one. +Before every write on an existing issue (`edit`, `field-set`, `comment`, `close`, `reopen`, `rollup-*`, `attach`) the backend reads the issue's `security` field and refuses with exit `4` unless it is exactly the configured level, before sending any content: a scheme can define levels open to every logged-in user. +It never sets a level on an existing issue itself: an issue without one may already have been exposed, and a human decides what to do. + +**Credentials are `jira_bridge.rest`'s.** Every request goes through `JiraRest.send`, so the token comes from `JIRA_API_TOKEN` or the token file +`~/.config/apache-magpie/jira-token`, is bound to a host the user confirmed (`JIRA_API_HOST` or a `host=` line), is sent only to that host over https, +follows no redirect, and is never printed. +When the token is withheld — no confirmed host, or a different one — every write is refused (exit `4`) rather than sent anonymously; reads go out anonymously. +`JIRA_AUTH_SCHEME` (`Basic` default, `Bearer` for Data Center PATs) and `JIRA_USER` (pairs a raw token for `Basic`) work as for the bridge. + +| Flag | Environment | `project.md` key | Meaning | +|---|---|---|---| +| `--url` | `ISSUE_TRACKER_URL` | `tracker.jira.url` | Jira base URL. | +| `--project` | `SECURITY_TRACKER_PROJECT` | `tracker.jira.project` | Key of the private project. Deliberately separate from `ISSUE_TRACKER_PROJECT`, which names the public tracker; an ID from another project is a usage error. | +| `--security-level` | `SECURITY_TRACKER_SECURITY_LEVEL` | `tracker.jira.security_level` | The level `create` sets. Required for `create`. | +| `--issue-type` | `SECURITY_TRACKER_ISSUE_TYPE` | `tracker.jira.issue_type` | Default `Bug`. | +| `--deployment {server,cloud}` | `JIRA_DEPLOYMENT` | `tracker.jira.deployment` | Cloud uses the search endpoint `/search/jql` and account-ID assignees. When given, it wins over `JIRA_DEPLOYMENT` and `--jira-cloud`; vetted-ops always passes it. | +| `--jira-cloud` | `JIRA_DEPLOYMENT=cloud` | — | Compatibility alias for `--deployment cloud`. | + +Under vetted-ops the URL, project and level are pinned by the policy's `[tracker]` table, not by the caller. + +| Operation | Jira mapping | +|---|---| +| IDs | `FOOSEC-12`, or a bare `12` expanded with the project key. | +| `view` | `closed` when resolved or the status category is *Done*; `milestone` is the first fix version; adds `security_level` and `warnings` (one when the issue has no level). | +| `list` | JQL `project = KEY`, the same open / closed rule as `view` (resolution or *Done* category), one `labels = "L"` per label, `text ~ "…"` for `--search`, newest first. | +| `labels` | Distinct labels across the project (at most 5000 issues scanned). | +| `create` / `edit` | `summary`, `description`, `labels`, `issuetype`, `security`; labels via `update.labels`, milestone via `fixVersions`, one assignee. | +| `comment` | `--author` is echoed but Jira records the authenticated user. | +| `close` / `reopen` | A transition into (or out of) the *Done* category; `close` sets `Fixed` or `Won't Fix` and retries without a resolution when the screen rejects one. | +| `attach` | Multipart upload; `--name` reduced to a base name. | +| `body` | The description with `h1.`–`h6.` headings as markdown `#`–`######`, raw (not JSON), for `generate-cve-json --stdin`. | +| `pending` / `publish` | No-ops: Jira writes are immediate. | +| `doctor` | URL, project, confirmed token, configured level, `/myself`, project reachability, and whether the level exists (403/404 on the listing reports *unverified*). | + +**Storage.** Body fields are `h3. ` wiki-markup sections of the description (markdown `###` sections are read too); headings inside `{code}`, `{noformat}` and fences are ignored. +The status rollup is one comment opening with the invisible anchor `{anchor:magpie-status-rollup-v1}`, entries `*YYYY-MM-DD · @user · Action*` separated by `----`, oldest first; +`rollup-amend-latest` exits `4` when the latest action differs. Neither rollup command prints the rollup text. + +## Configuration + +The bridge reads its configuration from the environment: + +| Variable | Notes | +|---|---| +| `ISSUE_TRACKER_URL` | required; e.g. `https://issues.apache.org/jira` | +| `ISSUE_TRACKER_PROJECT` | project key (e.g. `FOO`) | +| `JIRA_API_TOKEN` | required for write subcommands — see auth notes below | +| `JIRA_AUTH_SCHEME` | `Basic` (default) or `Bearer` — see auth notes below | + +The caller is responsible for exporting these (a skill resolves them +from [`/issue-tracker-config.md`](https://github.com/apache/magpie/blob/main/plugins/magpie-setup/templates/issue-tracker-config.md) +and passes them in the environment). Direct file-fallback inside the +bridge is a possible future enhancement — it is **not** implemented +today; the bridge exits if `ISSUE_TRACKER_URL` is unset. + +For anonymous-read trackers, no auth is required for read +subcommands. Write subcommands always require `JIRA_API_TOKEN` and +exit with an error if it is unset. + +**Authentication:** This bridge targets JIRA Data Center (DC), +specifically ASF JIRA at `issues.apache.org/jira`. Cloud is not +currently supported (`assign` uses DC `name`, not Cloud +`accountId`). + +- **Basic auth (default):** set `JIRA_API_TOKEN` to the + base64-encoded `username:password` or `username:pat` string. +- **Bearer auth (ASF PATs):** set `JIRA_AUTH_SCHEME=Bearer` and + `JIRA_API_TOKEN` to the raw PAT string. ASF JIRA DC PATs use + `Authorization: Bearer `. + +## Output contract + +Every subcommand emits JSON to stdout on success, or a non-zero +exit code with a human-readable error to stderr on failure. + +Write subcommands return `{"ok": true, "key": "", ...}` with +operation-specific fields as documented per subcommand above. + +The output schema is documented per subcommand above. Skills +parse the JSON via standard JSON tooling — no special envelope, +no wrapper. + +## Write-path discipline + +The bridge executes mutations but does **not** decide whether to +mutate. Every write operation is gated on **explicit user +confirmation** in the calling skill — the bridge only executes +confirmed actions. + +This mirrors the GitHub bridge's write-path discipline (see +[`tools/github/operations.md`](https://github.com/apache/magpie/blob/main/tools/github/operations.md)): skills +surface the proposed action to the maintainer, wait for +confirmation, then call the bridge to execute. + +## Testing + +The Groovy bridge tests use a mock HTTP server and require `groovy` on +`PATH`; they are skipped automatically when Groovy is not +available. The Python tests (`test_rest.py`, `test_security_tracker.py`, +`test_markup.py`) need no network: `JiraRest` takes a fake opener, and the +security tracker a fake transport. + +```bash +cd tools/jira +uv run pytest +``` + +## Cross-references + +- [`issue-triage`](https://github.com/apache/magpie/blob/main/plugins/magpie-issue/skills/triage/SKILL.md) — + primary consumer (selector resolution + per-issue fetch). +- [`issue-reassess`](https://github.com/apache/magpie/blob/main/plugins/magpie-issue/skills/reassess/SKILL.md) — + campaign-level consumer (pool fetch). +- [`security-issue-sync`](https://github.com/apache/magpie/blob/main/plugins/magpie-security/skills/issue-sync/SKILL.md) — + write-path consumer (label, transition, comment, field updates). +- [`security-issue-invalidate`](https://github.com/apache/magpie/blob/main/plugins/magpie-security/skills/issue-invalidate/SKILL.md) — + write-path consumer (close with label + comment). +- [`tools/github/operations.md`](https://github.com/apache/magpie/blob/main/tools/github/operations.md) — + write-path discipline reference. +- [`/issue-tracker-config.md`](https://github.com/apache/magpie/blob/main/plugins/magpie-setup/templates/issue-tracker-config.md) — + the adopter's tracker URL + project key. diff --git a/plugins/magpie-vetted-ops/tools/jira/pyproject.toml b/plugins/magpie-vetted-ops/tools/jira/pyproject.toml new file mode 100644 index 000000000..9460598f6 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/jira/pyproject.toml @@ -0,0 +1,63 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "jira-bridge" +version = "0.1.0" +description = "JIRA REST bridge: test harness for bridge.groovy, a stdlib Python REST client, and the private-Jira security tracker." +readme = "README.md" +requires-python = ">=3.11" +license = { text = "Apache-2.0" } +# General-tracker writes live in bridge.groovy. The package ships a stdlib-only REST +# client (jira_bridge.rest) and, on top of it, the security-tracker backend +# (jira_bridge.security_tracker), whose writes go through the same client. +dependencies = [] + +[project.scripts] +magpie-jira-tracker = "jira_bridge.security_tracker:main" + +[tool.hatch.build.targets.wheel] +packages = ["src/jira_bridge"] + +[tool.mypy] +python_version = "3.11" +files = ["src", "tests"] +warn_unused_ignores = true +warn_redundant_casts = true +warn_unreachable = true + +[[tool.mypy.overrides]] +module = "tests.*" +disallow_untyped_defs = false +disallow_incomplete_defs = false + +[tool.pytest.ini_options] +minversion = "8.0" +addopts = "-ra -q" +testpaths = ["tests"] + +# No `[dependency-groups] dev` here: this project also ships inside the +# `magpie-vetted-ops` plugin (vetted-ops runs it as a security-tracker backend +# with `uv run --project /tools/`), and `magpie-dev` only resolves +# through the workspace root's `[tool.uv.sources]`, which the plugin does not +# ship — a `dev` group would make every plugin invocation fail. The shared +# toolchain still reaches the tests: CI and the workspace checks sync the root +# `dev` group. See tools/vetted-ops/pyproject.toml. diff --git a/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/__init__.py b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/__init__.py new file mode 100644 index 000000000..13a83393a --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/__init__.py @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. diff --git a/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/markup.py b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/markup.py new file mode 100644 index 000000000..cc810b619 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/markup.py @@ -0,0 +1,221 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 + +"""Body-field sections and the status rollup, in Jira wiki markup. + +A body field is a level-3 section: ``h3. `` (wiki markup), or +``### `` (Markdown, read compatibility), running until the next +level 1-3 heading of either syntax. Headings inside ``{code}`` / +``{noformat}`` macros and Markdown code fences are ignored. + +The rollup is one comment whose first line is the invisible anchor macro +:data:`ROLLUP_MARKER`; each entry is a bold summary line +``*YYYY-MM-DD · @user · *``, a blank line, the entry body, and +entries are separated by a ``----`` line. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass + +ROLLUP_MARKER = "{anchor:magpie-status-rollup-v1}" + +_MARKER_RE = re.compile(r"^\{anchor:magpie-status-rollup-v\d+\}\s*$") +_SUMMARY_RE = re.compile(r"^\*(\d{4}-\d{2}-\d{2}) · @(.+?) · (.+)\*\s*$") +_WIKI_HEADING_RE = re.compile(r"^\s*h([1-6])\.\s+(.*?)\s*$") +_MD_HEADING_RE = re.compile(r"^\s{0,3}(#{1,6})\s+(.*?)(?:\s+#+)?\s*$") +_WIKI_BLOCK_RE = re.compile(r"\{(code|noformat)(?::[^}]*)?\}") +_FENCE_RE = re.compile(r"^\s{0,3}(`{3,}|~{3,})") +_FENCE_CLOSE_RE = re.compile(r"^\s{0,3}(`{3,}|~{3,})\s*$") + + +@dataclass(frozen=True) +class Heading: + """A heading line found in a body.""" + + index: int + level: int + text: str + wiki: bool + + +def _normalise(name: str) -> str: + return " ".join(name.split()).casefold() + + +def find_headings(lines: list[str]) -> list[Heading]: + """Return every heading outside code blocks, in document order.""" + headings: list[Heading] = [] + in_wiki_block: str | None = None + # Open Markdown fence as (character, length); per CommonMark only a + # fence of the same character, at least as long, with nothing but + # whitespace after it, closes it. + fence: tuple[str, int] | None = None + for index, line in enumerate(lines): + if fence is not None: + close = _FENCE_CLOSE_RE.match(line) + if close: + run = close.group(1) + if run[0] == fence[0] and len(run) >= fence[1]: + fence = None + continue + if in_wiki_block is not None: + # A {code} / {noformat} block closes on the bare macro name. + if "{" + in_wiki_block + "}" in line: + in_wiki_block = None + continue + opener = _FENCE_RE.match(line) + if opener: + fence = (opener.group(1)[0], len(opener.group(1))) + continue + macros = _WIKI_BLOCK_RE.findall(line) + if len(macros) % 2 == 1: + in_wiki_block = macros[-1] + continue + wiki = _WIKI_HEADING_RE.match(line) + if wiki: + headings.append(Heading(index, int(wiki.group(1)), wiki.group(2), True)) + continue + md = _MD_HEADING_RE.match(line) + if md: + headings.append(Heading(index, len(md.group(1)), md.group(2), False)) + return headings + + +def _locate(lines: list[str], heading: str) -> tuple[int, int] | None: + """Return ``(heading_index, end_index)`` of the field section, or None.""" + wanted = _normalise(heading) + headings = find_headings(lines) + for pos, found in enumerate(headings): + if found.level == 3 and _normalise(found.text) == wanted: + end = len(lines) + for later in headings[pos + 1 :]: + if later.level <= 3: + end = later.index + break + return found.index, end + return None + + +def get_field(body: str, heading: str) -> str | None: + """Return the stripped value of the ``heading`` section, or None when missing.""" + lines = body.splitlines() + span = _locate(lines, heading) + if span is None: + return None + start, end = span + return "\n".join(lines[start + 1 : end]).strip() + + +def set_field(body: str, heading: str, value: str) -> tuple[str, bool]: + """Replace (or append) the ``heading`` section; return ``(new_body, created)``.""" + lines = body.splitlines() + value_lines = value.strip("\n").splitlines() + span = _locate(lines, heading) + if span is not None: + start, end = span + tail = lines[end:] + new_lines = [*lines[: start + 1], *value_lines] + if tail: + new_lines.append("") + new_lines.extend(tail) + return "\n".join(new_lines) + "\n", False + headings = find_headings(lines) + markdown_only = bool(headings) and not any(h.wiki for h in headings) + heading_line = f"### {heading}" if markdown_only else f"h3. {heading}" + prefix = body.rstrip() + section = "\n".join([heading_line, *value_lines]) + "\n" + return (f"{prefix}\n\n{section}" if prefix else section), True + + +@dataclass +class RollupEntry: + """One rollup entry.""" + + date: str + user: str + action: str + body: str + + +_ANCHOR_RE = re.compile(r"\{(?=anchor:magpie-status-rollup)", re.IGNORECASE) + + +def neutralize(text: str) -> str: + """Make untrusted comment or rollup text unable to forge rollup structure. + + The text usually comes from outside the team. Written verbatim, a body that + opens with the rollup anchor would be taken for the rollup comment, and a + line shaped like ``*YYYY-MM-DD · @user · Action*`` would read back as a + rollup entry by someone else — misleading `rollup-amend-latest` too. Both + are escaped with a wiki-markup backslash, which renders the same visible + text and is never parsed as structure. + """ + text = _ANCHOR_RE.sub("\\{", text) + # Split exactly as parse_rollup does (str.splitlines: \r, \u2028, \x85 …), + # so no line boundary the parser honours can hide a forged summary line. + out = [] + for part in text.splitlines(keepends=True): + line = part.splitlines()[0] if part.splitlines() else "" + out.append("\\" + part if _SUMMARY_RE.match(line) else part) + return "".join(out) + + +def to_markdown_headings(text: str) -> str: + """Wiki-markup headings (`h3. Field`) as markdown (`### Field`); nothing else changes. + + Only the headings `find_headings` reports are converted — the same + code-block-aware scan the field readers use — so heading-like text inside a + `{code}` / `{noformat}` PoC never becomes a field generate-cve-json could + read a CVE ID from. + """ + lines = text.splitlines() + for heading in find_headings(lines): + if heading.wiki: + lines[heading.index] = "#" * heading.level + " " + heading.text + return "\n".join(lines) + + +def is_rollup(text: str) -> bool: + """True when ``text`` is a rollup comment (first non-empty line is the marker).""" + for line in text.splitlines(): + if line.strip(): + return bool(_MARKER_RE.match(line.strip())) + return False + + +def parse_rollup(text: str) -> list[RollupEntry]: + """Parse a rollup comment into its entries, oldest first.""" + entries: list[RollupEntry] = [] + body_lines: list[list[str]] = [] + seen_marker = False + for line in text.splitlines(): + if not seen_marker: + if _MARKER_RE.match(line.strip()): + seen_marker = True + continue + match = _SUMMARY_RE.match(line) + if match: + entries.append( + RollupEntry(match.group(1), match.group(2), match.group(3), "") + ) + body_lines.append([]) + elif entries: + body_lines[-1].append(line) + for entry, lines in zip(entries, body_lines, strict=True): + while lines and not lines[-1].strip(): + lines.pop() + if lines and lines[-1].strip() == "----": + lines.pop() + entry.body = "\n".join(lines).strip("\n") + return entries + + +def render_rollup(entries: list[RollupEntry]) -> str: + """Render rollup entries into the comment body.""" + blocks = [ + f"*{e.date} · @{e.user} · {e.action}*\n\n{e.body.strip(chr(10))}" + for e in entries + ] + return ROLLUP_MARKER + "\n" + "\n\n----\n\n".join(blocks) + "\n" diff --git a/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/py.typed b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/py.typed new file mode 100644 index 000000000..e69de29bb diff --git a/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/rest.py b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/rest.py new file mode 100644 index 000000000..5d088ecc6 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/rest.py @@ -0,0 +1,299 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 +"""Jira REST v2 client, stdlib only — the Python counterpart of bridge.groovy. + +It keeps the bridge's conventions: Jira Data Center REST API v2, a token in +`JIRA_API_TOKEN` sent as `Authorization: ` (`Basic` by default, +`Bearer` for ASF personal access tokens), and anonymous reads when there is no token. +With `JIRA_USER` set and a raw (not base64 `user:secret`) token, `Basic` encodes +`user:token` itself. + +`get()` / `search()` only ever send GET requests. `send()` is the one write path, +used by the security tracker (`jira_bridge.security_tracker`): it applies every rule +below, and refuses any non-GET request outright when the token was withheld, so a +write can never go out anonymously or to an unconfirmed host. + +**The token is the user's personal credential; the URL is not theirs to vouch for.** +The Jira URL comes from project configuration, which can be committed by anyone with +write access, so a changed URL must never silently receive the token. Four rules: + +1. The URL must be `https://`; `http://` is accepted only for loopback hosts + (`localhost`, `127.0.0.1`, `::1`), for tests and local instances. +2. The token is bound to a host the user confirmed: `JIRA_API_HOST`, or a `host=` line in + the token file `~/.config/apache-magpie/jira-token` (which then holds `token=` and + `host=` lines; a file with only the token is read as an unconfirmed token). When the + configured URL's host is not the confirmed one, or none is confirmed, the token is + withheld, reads go out anonymously, and `auth_note` says why. +3. `Authorization` is attached only to a request whose host equals the configured host. +4. Redirects are refused, so the token never follows one; the error names the target. + +The token never appears in an error message or an exception. + +Consumers: the `jira` backend of tools/contributor-metrics (contributor-activity reads of +`contract:tracker`), and the private-Jira security tracker (`contract:security-tracker`). +""" + +from __future__ import annotations + +import base64 +import binascii +import json +import os +import re +import time +import urllib.error +import urllib.parse +import urllib.request +from collections.abc import Callable +from pathlib import Path +from typing import Any + +URL_RE = re.compile(r"^https?://[^\s\"'<>]+$") +LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"}) +PAGE_SIZE = 100 +MAX_TRIES = 6 +RETRYABLE_STATUS = {429, 502, 503, 504} +TOKEN_FILE = Path.home() / ".config" / "apache-magpie" / "jira-token" + +Opener = Callable[[urllib.request.Request], Any] + + +class JiraRestError(RuntimeError): + """A request failed: an HTTP error, or an unreachable host after retries.""" + + +class JiraRestConfigError(ValueError): + """The base URL or the auth scheme is not usable.""" + + +def _read_token_file() -> tuple[str, str]: + """(token, host) from the token file: `token=` / `host=` lines, or a bare token (no host).""" + if not TOKEN_FILE.is_file(): + return "", "" + lines = [ln.strip() for ln in TOKEN_FILE.read_text().splitlines() if ln.strip()] + fields = dict( + ln.split("=", 1) for ln in lines if ln.startswith(("token=", "host=")) + ) + if fields: + return fields.get("token", "").strip(), fields.get("host", "").strip().lower() + return (lines[0] if lines else ""), "" + + +def default_credentials() -> tuple[str, str]: + """(token, confirmed host): the environment first, then the token file; "" when unset.""" + file_token, file_host = _read_token_file() + token = os.environ.get("JIRA_API_TOKEN", "").strip() or file_token + host = os.environ.get("JIRA_API_HOST", "").strip().lower() or file_host + return token, host + + +def _is_basic_encoded(token: str) -> bool: + try: + decoded = base64.b64decode(token, validate=True).decode("utf-8") + except (binascii.Error, UnicodeDecodeError, ValueError): + return False + return ":" in decoded + + +def authorization_header(token: str, scheme: str, user: str = "") -> str: + """The `Authorization` value for a token, the way bridge.groovy builds it. + + `Bearer` sends the token as-is. `Basic` sends it as-is when it already is a + base64 `user:secret` string; when `user` is set and the token is a raw secret, + it encodes `user:token`. + """ + if scheme == "Basic" and user and not _is_basic_encoded(token): + return "Basic " + base64.b64encode(f"{user}:{token}".encode()).decode("ascii") + return f"{scheme} {token}" + + +class _RefuseRedirects(urllib.request.HTTPRedirectHandler): + def redirect_request( + self, + req: urllib.request.Request, + fp: Any, + code: int, + msg: str, + headers: Any, + newurl: str, + ) -> urllib.request.Request | None: + raise JiraRestError( + f"refusing HTTP {code} redirect to {newurl}; check the configured Jira URL" + ) + + +_OPENER = urllib.request.build_opener(_RefuseRedirects()) + + +def _urlopen(req: urllib.request.Request) -> Any: + return _OPENER.open(req, timeout=60) + + +class JiraRest: + """GET-only access to one Jira instance.""" + + def __init__( + self, + url: str, + *, + token: str | None = None, + token_host: str | None = None, + auth_scheme: str | None = None, + user: str | None = None, + opener: Opener | None = None, + sleep: Callable[[float], None] | None = None, + ) -> None: + self.url = url.rstrip("/") + if not URL_RE.match(self.url): + raise JiraRestConfigError(f"Jira URL must be https://…, got {self.url!r}") + parts = urllib.parse.urlsplit(self.url) + self.host = (parts.hostname or "").lower() + self.netloc = parts.netloc.lower() + if parts.scheme != "https" and self.host not in LOOPBACK_HOSTS: + raise JiraRestConfigError( + f"Jira URL must be https:// (http:// only for localhost), got {self.url!r}" + ) + env_token, env_host = default_credentials() + token = env_token if token is None else token + confirmed = (env_host if token_host is None else token_host).strip().lower() + self.auth_note = "" + if token and confirmed != self.host: + self.auth_note = ( + f"the Jira token is confirmed for {confirmed!r}, not {self.host!r}; reading anonymously" + if confirmed + else f"the Jira token has no confirmed host (set JIRA_API_HOST or a host= line in " + f"{TOKEN_FILE}); not sending it to {self.host!r}, reading anonymously" + ) + token = "" + self._token = token + self.auth_scheme = auth_scheme or os.environ.get("JIRA_AUTH_SCHEME", "Basic") + if self.auth_scheme not in ("Basic", "Bearer"): + raise JiraRestConfigError( + f"JIRA_AUTH_SCHEME must be Basic or Bearer, got {self.auth_scheme!r}" + ) + self.user = (os.environ.get("JIRA_USER", "") if user is None else user).strip() + self.opener: Opener = opener or _urlopen + self.sleep: Callable[[float], None] = sleep or time.sleep + + def __repr__(self) -> str: + return f"JiraRest(url={self.url!r}, authenticated={bool(self._token)})" + + @property + def authenticated(self) -> bool: + return bool(self._token) + + def get(self, path: str, params: dict[str, str | int]) -> dict[str, Any]: + """One GET, retrying rate limits and transient failures with exponential backoff.""" + req = urllib.request.Request( + f"{self.url}{path}?{urllib.parse.urlencode(params)}", + headers={"Accept": "application/json"}, + ) + if ( + self._token + and urllib.parse.urlsplit(req.full_url).netloc.lower() == self.netloc + ): + req.add_header( + "Authorization", + authorization_header(self._token, self.auth_scheme, self.user), + ) + delay = 2.0 + for attempt in range(MAX_TRIES): + last = attempt == MAX_TRIES - 1 + try: + with self.opener(req) as resp: + data: dict[str, Any] = json.loads(resp.read().decode("utf-8")) + return data + except urllib.error.HTTPError as exc: + if exc.code not in RETRYABLE_STATUS or last: + raise JiraRestError( + f"HTTP {exc.code} from {path}: {exc.reason}" + ) from exc + except urllib.error.URLError as exc: + if last: + raise JiraRestError( + f"cannot reach {self.url}: {exc.reason}" + ) from exc + self.sleep(delay) + delay = min(delay * 2, 120.0) + raise JiraRestError("Jira failed after retries") # pragma: no cover + + def search( + self, jql: str, *, fields: str, pages: int, expand: str = "" + ) -> tuple[list[dict[str, Any]], int]: + """Up to `pages` pages of 100 issues for `jql`, and the total Jira reports.""" + issues: list[dict[str, Any]] = [] + total = 0 + for page in range(pages): + params: dict[str, str | int] = { + "jql": jql, + "startAt": page * PAGE_SIZE, + "maxResults": PAGE_SIZE, + "fields": fields, + } + if expand: + params["expand"] = expand + data = self.get("/rest/api/2/search", params) + total = int(data.get("total", 0)) + batch = data.get("issues") or [] + issues += batch + if not batch or len(issues) >= total: + break + return issues, total + + def send( + self, + method: str, + path: str, + *, + params: dict[str, str | int] | None = None, + body: bytes | None = None, + headers: dict[str, str] | None = None, + ) -> tuple[int, bytes]: + """One request of any method; returns (status, body) and never raises for an HTTP status. + + A non-GET request needs the token: when it was withheld (no host confirmed, or a + different one), the write is refused here rather than sent anonymously. GET + retries transient failures like `get()`; a write is sent once. + """ + if method != "GET" and not self._token: + raise JiraRestError( + f"refusing {method} {path}: writes need a Jira token confirmed for {self.host!r}" + + ( + f" ({self.auth_note})" + if self.auth_note + else " (no JIRA_API_TOKEN or token file)" + ) + ) + query = f"?{urllib.parse.urlencode(params)}" if params else "" + req = urllib.request.Request( + f"{self.url}{path}{query}", + data=body, + method=method, + headers={"Accept": "application/json", **(headers or {})}, + ) + if ( + self._token + and urllib.parse.urlsplit(req.full_url).netloc.lower() == self.netloc + ): + req.add_header( + "Authorization", + authorization_header(self._token, self.auth_scheme, self.user), + ) + tries = MAX_TRIES if method == "GET" else 1 + delay = 2.0 + for attempt in range(tries): + last = attempt == tries - 1 + try: + with self.opener(req) as resp: + return int(resp.status), resp.read() + except urllib.error.HTTPError as exc: + if exc.code not in RETRYABLE_STATUS or last: + return int(exc.code), exc.read() or b"" + except urllib.error.URLError as exc: + if last: + raise JiraRestError( + f"cannot reach {self.url}: {exc.reason}" + ) from exc + self.sleep(delay) + delay = min(delay * 2, 120.0) + raise JiraRestError("Jira failed after retries") # pragma: no cover diff --git a/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/security_tracker.py b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/security_tracker.py new file mode 100644 index 000000000..0f4ababd1 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/security_tracker.py @@ -0,0 +1,862 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 + +"""Command line interface: the security-tracker contract over a private Jira project. + +See ``tools/security-tracker/contract.md`` for the operations, their JSON +output and the exit codes. Every write on an existing issue first checks that +the issue carries an issue-security level and refuses (exit 4) otherwise. + +Every request goes through `jira_bridge.rest.JiraRest`, so credentials follow its +rules: `JIRA_API_TOKEN` or the token file, bound to a host the user confirmed +(`JIRA_API_HOST` or a `host=` line), never sent elsewhere, and no write without it. +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import os +import re +import sys +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from jira_bridge import markup +from jira_bridge.rest import JiraRest, JiraRestConfigError +from jira_bridge.tracker_client import ( + BackendError, + JiraClient, + NotFoundError, + RefusedError, + TrackerError, + Transport, + UsageError, +) + +_KEY_RE = re.compile(r"^[A-Z][A-Z0-9_]*-\d+$") +_RAW = object() # an operation that already wrote its own (non-JSON) output +_VIEW_FIELDS = "summary,status,resolution,labels,assignee,fixVersions,reporter,created,updated,description,security" +_LIST_FIELDS = "summary,status,resolution,labels,updated" +_LABEL_SCAN_CAP = 5000 +_RESOLUTIONS = {"completed": "Fixed", "not-planned": "Won't Fix"} + + +@dataclass +class Config: + url: str + project: str + security_level: str + issue_type: str + cloud: bool + + +# -- helpers ----------------------------------------------------------------- + + +def _emit(data: Any) -> None: + print(json.dumps(data, indent=2, ensure_ascii=False)) + + +def _read_text(path: str, what: str) -> str: + if path == "-": + return sys.stdin.read() + try: + return Path(path).read_text(encoding="utf-8") + except OSError as exc: + raise UsageError(f"cannot read {what} file {path}: {exc.strerror}") from exc + + +def jql_quote(value: str) -> str: + """Quote ``value`` as a JQL string literal.""" + return '"' + value.replace("\\", "\\\\").replace('"', '\\"') + '"' + + +def expand_id(raw: str, project: str) -> str: + """Expand a bare number with the project key and validate the key form.""" + value = raw.strip().upper() + if value.isdigit(): + if not project: + raise UsageError( + f"bare id {raw!r} needs a project key (--project / SECURITY_TRACKER_PROJECT)" + ) + value = f"{project}-{int(value)}" + if not _KEY_RE.match(value): + raise UsageError(f"invalid issue id {raw!r}: expected KEY-123 or a bare number") + if project and value.rsplit("-", 1)[0] != project: + raise UsageError( + f"issue {value} is not in the configured security project {project}" + ) + return value + + +def _person(person: Any) -> str | None: + if not isinstance(person, dict): + return None + for key in ("name", "accountId", "displayName"): + if person.get(key): + return str(person[key]) + return None + + +def _state(fields: Mapping[str, Any]) -> str: + if fields.get("resolution"): + return "closed" + status = fields.get("status") or {} + category = (status.get("statusCategory") or {}).get("key") + return "closed" if category == "done" else "open" + + +def _security_name(fields: Mapping[str, Any]) -> str | None: + security = fields.get("security") + if not isinstance(security, dict): + return None + return str(security.get("name") or security.get("id") or "") or None + + +def _date(now: str | None) -> str: + if not now: + return dt.datetime.now(dt.UTC).date().isoformat() + try: + return dt.datetime.fromisoformat(now.replace("Z", "+00:00")).date().isoformat() + except ValueError as exc: + raise UsageError(f"--now is not an ISO 8601 date/time: {now!r}") from exc + + +def _single_line(value: str, what: str) -> str: + if "\n" in value or "\r" in value: + raise UsageError(f"{what} must be a single line") + return value.strip() + + +# -- the guard --------------------------------------------------------------- + + +def guarded_issue( + client: JiraClient, key: str, extra_fields: str = "", *, level: str +) -> dict[str, Any]: + """Fetch ``key`` and refuse unless it carries exactly the configured + issue-security level. + + Any level is not enough: a scheme can define levels open to every logged-in + user, and an issue created under one of those must not receive embargoed + content. + """ + if not level: + raise RefusedError( + f"refusing to write to {key}: no issue-security level is configured " + "(--security-level / SECURITY_TRACKER_SECURITY_LEVEL / tracker.jira.security_level)" + ) + fields = "security" + ("," + extra_fields if extra_fields else "") + issue = client.issue(key, fields) + actual = _security_name(issue.get("fields") or {}) + if actual is None: + raise RefusedError( + f"refusing to write to {key}: the issue has no issue-security level, so it is not " + "access-restricted, and this backend will not write embargoed content to it. " + "Ask a Jira project admin to set the security level; the backend never sets it on an existing issue." + ) + if actual.strip().casefold() != level.strip().casefold(): + raise RefusedError( + f"refusing to write to {key}: its issue-security level is {actual!r}, not the configured " + f"{level!r}; a different level may be visible to people outside the security team" + ) + return issue + + +# -- operations -------------------------------------------------------------- + + +def op_view(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + key = expand_id(args.id, cfg.project) + issue = client.issue(key, _VIEW_FIELDS + (",comment" if args.comments else "")) + fields = issue.get("fields") or {} + level = _security_name(fields) + warnings = [] + if level is None: + warnings.append( + f"{key} has no issue-security level: it is not access-restricted; writes are refused" + ) + assignee = _person(fields.get("assignee")) + versions = fields.get("fixVersions") or [] + result: dict[str, Any] = { + "id": issue.get("key", key), + "title": fields.get("summary") or "", + "state": _state(fields), + "labels": list(fields.get("labels") or []), + "assignees": [assignee] if assignee else [], + "milestone": versions[0].get("name") if versions else None, + "author": _person(fields.get("reporter")), + "created": fields.get("created"), + "updated": fields.get("updated"), + "body": fields.get("description") or "", + "url": client.browse_url(issue.get("key", key)), + "security_level": level, + "warnings": warnings, + } + if args.comments: + result["comments"] = [ + { + "id": str(c.get("id")), + "author": _person(c.get("author")), + "created": c.get("created"), + "body": c.get("body") or "", + } + for c in client.comments(key) + ] + return result + + +def op_list(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + clauses = [f"project = {jql_quote(cfg.project)}"] + # The same definition as _state(): closed is resolved *or* in the Done category. + if args.state == "open": + clauses.append("resolution is EMPTY AND statusCategory != Done") + elif args.state == "closed": + clauses.append("(resolution is not EMPTY OR statusCategory = Done)") + clauses.extend(f"labels = {jql_quote(label)}" for label in args.label or []) + if args.search: + clauses.append(f"text ~ {jql_quote(args.search)}") + jql = " AND ".join(clauses) + " ORDER BY updated DESC" + if args.limit < 1: + raise UsageError("--limit must be at least 1") + return [ + { + "id": issue.get("key"), + "title": (issue.get("fields") or {}).get("summary") or "", + "state": _state(issue.get("fields") or {}), + "labels": list((issue.get("fields") or {}).get("labels") or []), + "updated": (issue.get("fields") or {}).get("updated"), + "url": client.browse_url(str(issue.get("key"))), + } + for issue in client.search(jql, _LIST_FIELDS, args.limit) + ] + + +def op_labels(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + jql = f"project = {jql_quote(cfg.project)} AND labels is not EMPTY ORDER BY updated DESC" + found: set[str] = set() + for issue in client.search(jql, "labels", _LABEL_SCAN_CAP): + found.update((issue.get("fields") or {}).get("labels") or []) + return sorted(found) + + +def op_field_get(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + key = expand_id(args.id, cfg.project) + fields = client.issue(key, "description").get("fields") or {} + value = markup.get_field(fields.get("description") or "", args.heading) + return {"id": key, "heading": args.heading, "value": value} + + +def op_field_set(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + key = expand_id(args.id, cfg.project) + heading = _single_line(args.heading, "heading") + value = _read_text(args.value_file, "value") + issue = guarded_issue(client, key, "description", level=cfg.security_level) + body = (issue.get("fields") or {}).get("description") or "" + new_body, created = markup.set_field(body, heading, value) + client.call( + "PUT", f"/rest/api/2/issue/{key}", payload={"fields": {"description": new_body}} + ) + return {"id": key, "heading": heading, "created": created} + + +def _check_labels(labels: Sequence[str]) -> list[str]: + for label in labels: + if not label or any(ch.isspace() for ch in label): + raise UsageError( + f"invalid Jira label {label!r}: labels cannot be empty or contain whitespace" + ) + return list(labels) + + +def _title_arg(args: argparse.Namespace) -> str | None: + """The title from ``--title`` (argv, never a shell) or ``--title-file``.""" + if getattr(args, "title", None) is not None and args.title_file: + raise UsageError("--title and --title-file are mutually exclusive") + if getattr(args, "title", None) is not None: + raw = str(args.title) + elif args.title_file: + raw = _read_text(args.title_file, "title") + else: + return None + title = " ".join(raw.split()) + if not title: + raise UsageError("title is empty") + return title + + +def op_create(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + if not cfg.security_level: + raise RefusedError( + "refusing to create: no issue-security level configured " + "(--security-level / SECURITY_TRACKER_SECURITY_LEVEL / tracker.jira.security_level); " + "an issue without one is not access-restricted" + ) + title = _title_arg(args) + if title is None: + raise UsageError("create needs --title or --title-file") + body = _read_text(args.body_file, "body") + fields: dict[str, Any] = { + "project": {"key": cfg.project}, + "summary": title, + "description": body, + "issuetype": {"name": cfg.issue_type}, + "security": {"name": cfg.security_level}, + "labels": _check_labels(args.label or []), + } + data = client.call("POST", "/rest/api/2/issue", payload={"fields": fields}) + key = str((data or {}).get("key") or "") + if not key: + raise BackendError("Jira did not return the created issue key") + return {"id": key, "url": client.browse_url(key)} + + +def _assignee_ref(cfg: Config, user: str) -> dict[str, str]: + return {"accountId": user} if cfg.cloud else {"name": user} + + +def op_edit(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + key = expand_id(args.id, cfg.project) + if args.milestone is not None and args.clear_milestone: + raise UsageError("--milestone and --clear-milestone are mutually exclusive") + adds = args.add_assignee or [] + if len(adds) > 1: + raise UsageError( + "a Jira issue has a single assignee: pass --add-assignee at most once" + ) + fields: dict[str, Any] = {} + update: dict[str, Any] = {} + label_ops = [{"add": lb} for lb in _check_labels(args.add_label or [])] + label_ops += [{"remove": lb} for lb in _check_labels(args.remove_label or [])] + if label_ops: + update["labels"] = label_ops + if args.milestone is not None: + fields["fixVersions"] = [{"name": args.milestone}] + elif args.clear_milestone: + fields["fixVersions"] = [] + title = _title_arg(args) + if title is not None: + fields["summary"] = title + if args.body_file: + fields["description"] = _read_text(args.body_file, "body") + removes = args.remove_assignee or [] + if not (fields or update or adds or removes): + raise UsageError("edit: nothing to change") + issue = guarded_issue(client, key, "assignee", level=cfg.security_level) + current = _person((issue.get("fields") or {}).get("assignee")) + if adds: + fields["assignee"] = _assignee_ref(cfg, adds[0]) + elif removes and current is not None and current in removes: + fields["assignee"] = None + payload: dict[str, Any] = {} + if fields: + payload["fields"] = fields + if update: + payload["update"] = update + if payload: + client.call("PUT", f"/rest/api/2/issue/{key}", payload=payload) + return {"id": key, "url": client.browse_url(key)} + + +def op_comment(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + key = expand_id(args.id, cfg.project) + body = markup.neutralize(_read_text(args.body_file, "body")) + guarded_issue(client, key, level=cfg.security_level) + data = client.call( + "POST", f"/rest/api/2/issue/{key}/comment", payload={"body": body} + ) + comment_id = str((data or {}).get("id") or "") + result: dict[str, Any] = { + "id": key, + "comment_id": comment_id, + "url": client.comment_url(key, comment_id), + } + if args.author: + # Jira records the authenticated user as the author; --author is informational only. + result["author"] = args.author + result["author_ignored"] = True + return result + + +def _transitions(client: JiraClient, key: str) -> list[dict[str, Any]]: + data = client.call( + "GET", + f"/rest/api/2/issue/{key}/transitions", + params={"expand": "transitions.fields"}, + ) + return list((data or {}).get("transitions") or []) + + +def _target_category(transition: Mapping[str, Any]) -> str | None: + return ((transition.get("to") or {}).get("statusCategory") or {}).get("key") + + +def op_close(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + key = expand_id(args.id, cfg.project) + issue = guarded_issue(client, key, "status,resolution", level=cfg.security_level) + if _state(issue.get("fields") or {}) == "closed": + return {"id": key, "state": "closed"} + done = [t for t in _transitions(client, key) if _target_category(t) == "done"] + if not done: + raise BackendError( + f"no transition to a done status is available on {key} for this user" + ) + transition = done[0] + path = f"/rest/api/2/issue/{key}/transitions" + screen = transition.get("fields") + with_resolution = not isinstance(screen, dict) or "resolution" in screen + base: dict[str, Any] = {"transition": {"id": str(transition.get("id"))}} + if with_resolution: + payload = { + **base, + "fields": {"resolution": {"name": _RESOLUTIONS[args.reason]}}, + } + try: + client.call("POST", path, payload=payload) + return {"id": key, "state": "closed"} + except BackendError as exc: + if exc.status != 400: + raise + client.call("POST", path, payload=base) + return {"id": key, "state": "closed"} + + +def op_reopen(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + key = expand_id(args.id, cfg.project) + issue = guarded_issue(client, key, "status,resolution", level=cfg.security_level) + if _state(issue.get("fields") or {}) == "open": + return {"id": key, "state": "open"} + candidates = [ + t + for t in _transitions(client, key) + if _target_category(t) not in (None, "done") + ] + if not candidates: + raise BackendError( + f"no transition to an open status is available on {key} for this user" + ) + named = [t for t in candidates if "reopen" in str(t.get("name", "")).lower()] + transition = (named or candidates)[0] + client.call( + "POST", + f"/rest/api/2/issue/{key}/transitions", + payload={"transition": {"id": str(transition.get("id"))}}, + ) + return {"id": key, "state": "open"} + + +def _find_rollup(client: JiraClient, key: str) -> dict[str, Any] | None: + for comment in client.comments(key): + if markup.is_rollup(str(comment.get("body") or "")): + return comment + return None + + +def _rollup_user(client: JiraClient, user: str | None) -> str: + if user: + return _single_line(user, "--user").lstrip("@") + me = client.myself() + name = me.get("name") or me.get("displayName") + if not name: + raise BackendError("cannot determine the authenticated Jira user; pass --user") + return str(name) + + +def op_rollup_append(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + key = expand_id(args.id, cfg.project) + action = _single_line(args.action, "--action") + entry_body = markup.neutralize(_read_text(args.entry_body_file, "entry body")) + date = _date(args.now) + guarded_issue(client, key, level=cfg.security_level) + user = _rollup_user(client, args.user) + existing = _find_rollup(client, key) + entry = markup.RollupEntry(date, user, action, entry_body) + if existing is None: + data = client.call( + "POST", + f"/rest/api/2/issue/{key}/comment", + payload={"body": markup.render_rollup([entry])}, + ) + comment_id = str((data or {}).get("id") or "") + else: + comment_id = str(existing.get("id")) + entries = [*markup.parse_rollup(str(existing.get("body") or "")), entry] + client.call( + "PUT", + f"/rest/api/2/issue/{key}/comment/{comment_id}", + payload={"body": markup.render_rollup(entries)}, + ) + return {"id": key, "url": client.comment_url(key, comment_id)} + + +def op_rollup_amend_latest( + client: JiraClient, cfg: Config, args: argparse.Namespace +) -> Any: + key = expand_id(args.id, cfg.project) + action = _single_line(args.action, "--action") + entry_body = markup.neutralize(_read_text(args.entry_body_file, "entry body")) + guarded_issue(client, key, level=cfg.security_level) + existing = _find_rollup(client, key) + entries = markup.parse_rollup(str(existing.get("body") or "")) if existing else [] + if existing is None or not entries: + raise NotFoundError(f"{key} has no status rollup to amend") + latest = entries[-1] + if latest.action.strip() != action: + raise RefusedError( + f"latest rollup entry on {key} is {latest.action.strip()!r}, not {action!r}; not amending" + ) + latest.body = entry_body + comment_id = str(existing.get("id")) + client.call( + "PUT", + f"/rest/api/2/issue/{key}/comment/{comment_id}", + payload={"body": markup.render_rollup(entries)}, + ) + return {"id": key, "url": client.comment_url(key, comment_id)} + + +def op_body(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + """The description as markdown `### Field` sections, for generate-cve-json --stdin.""" + key = expand_id(args.id, cfg.project) + data = client.issue(key, "description") + description = str((data.get("fields") or {}).get("description") or "") + sys.stdout.write(markup.to_markdown_headings(description).strip("\n") + "\n") + return _RAW + + +def op_attach(client: JiraClient, cfg: Config, args: argparse.Namespace) -> Any: + key = expand_id(args.id, cfg.project) + if args.file == "-": + if not args.name: + raise UsageError("attaching from stdin needs --name") + content, name = sys.stdin.buffer.read(), Path(args.name).name + else: + path = Path(args.file) + if not path.is_file(): + raise UsageError(f"attachment {args.file} is not a readable file") + name = Path(args.name).name if args.name else path.name + content = path.read_bytes() + if not name: + raise UsageError("attachment name is empty") + guarded_issue(client, key, level=cfg.security_level) + data = client.attach(key, name, content) + first = data[0] if isinstance(data, list) and data else {} + url = ( + first.get("content") if isinstance(first, dict) else None + ) or client.browse_url(key) + return { + "id": key, + "name": (first.get("filename") if isinstance(first, dict) else None) or name, + "url": url, + } + + +def op_pending(client: JiraClient | None, cfg: Config, args: argparse.Namespace) -> Any: + return {"backend": "jira", "changes": [], "needs_publish": False} + + +def op_publish(client: JiraClient | None, cfg: Config, args: argparse.Namespace) -> Any: + return { + "committed": False, + "pushed": False, + "revision": None, + "note": "jira writes are immediate", + } + + +def op_doctor(client: JiraClient | None, cfg: Config, args: argparse.Namespace) -> Any: + checks: list[dict[str, Any]] = [] + + def check(name: str, ok: bool, detail: str) -> None: + checks.append({"name": name, "ok": ok, "detail": detail}) + + check("url", bool(cfg.url), cfg.url or "ISSUE_TRACKER_URL / --url not set") + check( + "project", + bool(cfg.project), + cfg.project or "SECURITY_TRACKER_PROJECT / --project not set", + ) + rest = client.rest if client is not None else None + if rest is not None: + check( + "token", + rest.authenticated, + f"token confirmed for {rest.host} ({rest.auth_scheme})" + if rest.authenticated + else rest.auth_note + or "no JIRA_API_TOKEN or token file; writes are refused", + ) + check( + "security_level", + bool(cfg.security_level), + cfg.security_level + or "SECURITY_TRACKER_SECURITY_LEVEL / --security-level not set; create is refused", + ) + if client is not None and cfg.url: + try: + me = client.myself() + check( + "auth", + True, + f"authenticated as {me.get('name') or me.get('displayName') or 'unknown'}", + ) + except TrackerError as exc: + check("auth", False, str(exc)) + if cfg.project: + try: + project = client.call("GET", f"/rest/api/2/project/{cfg.project}") + check( + "project_reachable", + True, + str((project or {}).get("name") or cfg.project), + ) + except TrackerError as exc: + check("project_reachable", False, str(exc)) + if cfg.security_level: + check(*_security_level_check(client, cfg)) + ok = all(c["ok"] for c in checks) + return {"backend": "jira", "ok": ok, "checks": checks} + + +def _security_level_check(client: JiraClient, cfg: Config) -> tuple[str, bool, str]: + name = "security_level_exists" + try: + data = client.call("GET", f"/rest/api/2/project/{cfg.project}/securitylevel") + except NotFoundError: + return name, True, "unverified: the security-level endpoint is not available" + except BackendError as exc: + if exc.status in (401, 403): + return name, True, "unverified: not permitted to list security levels" + return name, False, str(exc) + levels = [str(level.get("name")) for level in (data or {}).get("levels") or []] + if cfg.security_level in levels: + return ( + name, + True, + f"{cfg.security_level!r} is a security level of {cfg.project}", + ) + return ( + name, + False, + f"{cfg.security_level!r} not among the levels visible to this user: {levels}", + ) + + +# -- parser ------------------------------------------------------------------ + +_Op = Callable[[Any, Config, argparse.Namespace], Any] +_OFFLINE = {"pending", "publish", "doctor"} + + +def _common(parser: argparse.ArgumentParser, default: Any) -> None: + parser.add_argument( + "--url", default=default, help="Jira base URL (ISSUE_TRACKER_URL)." + ) + parser.add_argument( + "--project", default=default, help="Project key (SECURITY_TRACKER_PROJECT)." + ) + parser.add_argument( + "--security-level", + default=default, + help="Issue-security level name set on create (SECURITY_TRACKER_SECURITY_LEVEL).", + ) + parser.add_argument( + "--issue-type", + default=default, + help="Issue type for create (default Bug; SECURITY_TRACKER_ISSUE_TYPE).", + ) + parser.add_argument( + "--jira-cloud", + action="store_true", + default=default, + help="Jira Cloud deployment: assignees are accountIds (JIRA_DEPLOYMENT=cloud).", + ) + parser.add_argument( + "--deployment", + choices=["server", "cloud"], + default=default, + help="Jira deployment; when given, wins over JIRA_DEPLOYMENT and --jira-cloud.", + ) + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="magpie-jira-tracker", + description="Security-tracker contract over a private Jira project (issue-security level required).", + ) + _common(parser, None) + common = argparse.ArgumentParser(add_help=False) + _common(common, argparse.SUPPRESS) + sub = parser.add_subparsers(dest="op", required=True) + + def add(name: str, func: _Op, help_text: str) -> argparse.ArgumentParser: + p = sub.add_parser(name, parents=[common], help=help_text) + p.set_defaults(func=func) + return p + + p = add("view", op_view, "Show one tracker.") + p.add_argument("id") + p.add_argument("--comments", action="store_true") + + p = add("list", op_list, "List trackers.") + p.add_argument("--state", choices=["open", "closed", "all"], default="open") + p.add_argument("--label", action="append") + p.add_argument("--search") + p.add_argument("--limit", type=int, default=30) + + add("labels", op_labels, "Labels in use in the project.") + + p = add("field-get", op_field_get, "Read a body-field section.") + p.add_argument("id") + p.add_argument("heading") + + p = add("field-set", op_field_set, "Replace or append a body-field section.") + p.add_argument("id") + p.add_argument("heading") + p.add_argument("--value-file", required=True) + + p = add("create", op_create, "Create a tracker with the configured security level.") + p.add_argument("--title-file") + p.add_argument( + "--title", help="title as one argv value (callers that never use a shell)" + ) + p.add_argument("--body-file", required=True) + p.add_argument("--label", action="append") + + p = add("edit", op_edit, "Edit labels, assignee, milestone, title or body.") + p.add_argument("id") + p.add_argument("--add-label", action="append") + p.add_argument("--remove-label", action="append") + p.add_argument("--add-assignee", action="append") + p.add_argument("--remove-assignee", action="append") + p.add_argument("--milestone") + p.add_argument("--clear-milestone", action="store_true") + p.add_argument("--title-file") + p.add_argument( + "--title", help="title as one argv value (callers that never use a shell)" + ) + p.add_argument("--body-file") + + p = add("comment", op_comment, "Add a comment.") + p.add_argument("id") + p.add_argument("--body-file", required=True) + p.add_argument("--author") + + p = add("close", op_close, "Close via a transition to a done status.") + p.add_argument("id") + p.add_argument("--reason", choices=sorted(_RESOLUTIONS), default="completed") + + p = add("reopen", op_reopen, "Reopen via a transition to a not-done status.") + p.add_argument("id") + + p = add("rollup-append", op_rollup_append, "Append a status-rollup entry.") + p.add_argument("id") + p.add_argument("--action", required=True) + p.add_argument("--entry-body-file", required=True) + p.add_argument("--user") + p.add_argument("--now") + + p = add( + "rollup-amend-latest", + op_rollup_amend_latest, + "Replace the latest rollup entry's body.", + ) + p.add_argument("id") + p.add_argument("--action", required=True) + p.add_argument("--entry-body-file", required=True) + + p = add( + "body", + op_body, + "Print the description as markdown sections (for generate-cve-json --stdin).", + ) + p.add_argument("id") + + p = add("attach", op_attach, "Attach a file.") + p.add_argument("id") + p.add_argument("file") + p.add_argument("--name") + + add("pending", op_pending, "Unpublished changes (always none for Jira).") + + p = add("publish", op_publish, "No-op: Jira writes are immediate.") + p.add_argument("--message-file") + p.add_argument("--trailer", action="append") + p.add_argument("--push", action="store_true") + + add("doctor", op_doctor, "Check configuration, credentials and the security level.") + return parser + + +def resolve_config(args: argparse.Namespace, env: Mapping[str, str]) -> Config: + explicit = getattr(args, "deployment", None) + if explicit: + cloud = explicit == "cloud" + else: + cloud = ( + bool(args.jira_cloud) + or env.get("JIRA_DEPLOYMENT", "").strip().lower() == "cloud" + ) + return Config( + url=(args.url or env.get("ISSUE_TRACKER_URL", "")).strip().rstrip("/"), + project=(args.project or env.get("SECURITY_TRACKER_PROJECT", "")) + .strip() + .upper(), + security_level=( + args.security_level or env.get("SECURITY_TRACKER_SECURITY_LEVEL", "") + ).strip(), + issue_type=( + args.issue_type or env.get("SECURITY_TRACKER_ISSUE_TYPE", "") + ).strip() + or "Bug", + cloud=cloud, + ) + + +def main( + argv: Sequence[str] | None = None, + *, + transport: Transport | None = None, + env: Mapping[str, str] | None = None, +) -> int: + parser = build_parser() + try: + args = parser.parse_args(argv) + except SystemExit as exc: + return int(exc.code or 0) + cfg = resolve_config(args, os.environ if env is None else env) + try: + client: JiraClient | None = None + if cfg.url: + rest = None + if transport is None: + try: + rest = JiraRest(cfg.url) + except JiraRestConfigError as exc: + raise UsageError(str(exc)) from exc + client = JiraClient( + base_url=cfg.url, + project=cfg.project, + cloud=cfg.cloud, + rest=rest, + transport=transport, + ) + elif args.op not in _OFFLINE: + raise UsageError( + "Jira URL not configured: set ISSUE_TRACKER_URL or pass --url" + ) + if args.op not in _OFFLINE and not cfg.project: + raise UsageError( + "project key not configured: set SECURITY_TRACKER_PROJECT or pass --project" + ) + result = args.func(client, cfg, args) + if result is not _RAW: + _emit(result) + return 0 + except TrackerError as exc: + print(f"magpie-jira-tracker: error: {exc}", file=sys.stderr) + return exc.exit_code diff --git a/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/tracker_client.py b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/tracker_client.py new file mode 100644 index 000000000..e617f6fa9 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/jira/src/jira_bridge/tracker_client.py @@ -0,0 +1,273 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 + +"""Jira REST v2 operations for the security tracker, over the shared `JiraRest` client. + +Every request goes through `jira_bridge.rest.JiraRest.send`, so the security tracker +inherits its rules unchanged: https only, the token bound to a host the user confirmed +(`JIRA_API_HOST` or the token file), `Authorization` only to that host, redirects +refused, and writes refused outright when the token was withheld. Tests replace the +network with a `Transport`; production never takes that path. +""" + +from __future__ import annotations + +import json +import secrets +import urllib.parse +from collections.abc import Callable, Iterator, Mapping +from dataclasses import dataclass +from typing import Any, NamedTuple + +from jira_bridge.rest import JiraRest, JiraRestError + +EXIT_USAGE = 2 +EXIT_NOT_FOUND = 3 +EXIT_REFUSED = 4 +EXIT_BACKEND = 5 + + +class TrackerError(Exception): + """An error carrying the contract exit code.""" + + exit_code = EXIT_BACKEND + + +class UsageError(TrackerError): + exit_code = EXIT_USAGE + + +class NotFoundError(TrackerError): + exit_code = EXIT_NOT_FOUND + + +class RefusedError(TrackerError): + exit_code = EXIT_REFUSED + + +class BackendError(TrackerError): + """An HTTP or network failure; ``status`` is the HTTP status when there was one.""" + + exit_code = EXIT_BACKEND + + def __init__(self, message: str, status: int | None = None) -> None: + super().__init__(message) + self.status = status + + +class Response(NamedTuple): + status: int + body: bytes + + +# (method, url, headers, body) -> Response. Tests only; production uses JiraRest.send. +Transport = Callable[[str, str, Mapping[str, str], bytes | None], Response] + + +def _error_text(body: bytes) -> str: + try: + data = json.loads(body) + except (ValueError, UnicodeDecodeError): + return "" + if not isinstance(data, dict): + return "" + parts = [str(m) for m in data.get("errorMessages") or []] + errors = data.get("errors") or {} + if isinstance(errors, dict): + parts.extend(f"{k}: {v}" for k, v in errors.items()) + return "; ".join(parts) + + +@dataclass +class JiraClient: + """Jira REST v2 access for one project.""" + + base_url: str + project: str + cloud: bool = False + #: The shared client every production request goes through. + rest: JiraRest | None = None + #: Tests only: a fake network in place of `rest`. + transport: Transport | None = None + + def __post_init__(self) -> None: + self.base_url = self.base_url.rstrip("/") + if self.rest is None and self.transport is None: + raise ValueError("JiraClient needs a JiraRest (or, in tests, a transport)") + + # -- plumbing ------------------------------------------------------ + + def _send( + self, + method: str, + path: str, + params: Mapping[str, str | int] | None, + body: bytes | None, + headers: Mapping[str, str] | None, + ) -> Response: + if self.transport is not None: + url = self.base_url + path + if params: + url += "?" + urllib.parse.urlencode( + {k: str(v) for k, v in params.items()} + ) + return self.transport( + method, url, {"Accept": "application/json", **(headers or {})}, body + ) + assert self.rest is not None + try: + status, data = self.rest.send( + method, + path, + params={k: str(v) for k, v in params.items()} if params else None, + body=body, + headers=dict(headers or {}), + ) + except JiraRestError as exc: + message = str(exc) + if ( + message.startswith("refusing ") + and "writes need a Jira token" in message + ): + raise RefusedError(message) from exc + raise BackendError(message) from exc + return Response(status, data) + + def raw( + self, + method: str, + path: str, + *, + params: Mapping[str, str | int] | None = None, + body: bytes | None = None, + headers: Mapping[str, str] | None = None, + ) -> Any: + resp = self._send(method, path, params, body, headers) + if resp.status == 404: + raise NotFoundError(f"not found: {method} {path}") + if resp.status >= 400: + detail = _error_text(resp.body) + suffix = f": {detail}" if detail else "" + raise BackendError( + f"Jira returned HTTP {resp.status} for {method} {path}{suffix}", + resp.status, + ) + if not resp.body.strip(): + return None + try: + return json.loads(resp.body) + except ValueError as exc: + raise BackendError( + f"Jira returned a non-JSON response for {method} {path}" + ) from exc + + def call( + self, + method: str, + path: str, + *, + params: Mapping[str, str | int] | None = None, + payload: Any = None, + ) -> Any: + body = None if payload is None else json.dumps(payload).encode("utf-8") + headers = {"Content-Type": "application/json"} if body is not None else None + return self.raw(method, path, params=params, body=body, headers=headers) + + # -- urls ---------------------------------------------------------- + + def browse_url(self, key: str) -> str: + return f"{self.base_url}/browse/{key}" + + def comment_url(self, key: str, comment_id: str) -> str: + return ( + f"{self.browse_url(key)}?focusedCommentId={comment_id}#comment-{comment_id}" + ) + + # -- operations ---------------------------------------------------- + + def issue(self, key: str, fields: str) -> dict[str, Any]: + data = self.call("GET", f"/rest/api/2/issue/{key}", params={"fields": fields}) + if not isinstance(data, dict): + raise BackendError(f"unexpected response for issue {key}") + return data + + def search(self, jql: str, fields: str, limit: int) -> Iterator[dict[str, Any]]: + """Yield up to ``limit`` issues matching ``jql``, paginating. + + Server / Data Center paginate ``/search`` by ``startAt``; Cloud has + retired that endpoint in favour of ``/search/jql`` with a token. + """ + page = min(limit, 100) + returned = 0 + start_at = 0 + token: str | None = None + while returned < limit: + params: dict[str, str | int] = { + "jql": jql, + "fields": fields, + "maxResults": page, + } + if self.cloud: + if token: + params["nextPageToken"] = token + data = self.call("GET", "/rest/api/2/search/jql", params=params) + else: + params["startAt"] = start_at + data = self.call("GET", "/rest/api/2/search", params=params) + issues = (data or {}).get("issues") or [] + for issue in issues: + if returned >= limit: + return + returned += 1 + yield issue + if not issues: + return + if self.cloud: + token = data.get("nextPageToken") + if not token or data.get("isLast"): + return + else: + start_at += len(issues) + total = data.get("total") + if isinstance(total, int) and start_at >= total: + return + + def comments(self, key: str) -> list[dict[str, Any]]: + result: list[dict[str, Any]] = [] + start_at = 0 + while True: + data = self.call( + "GET", + f"/rest/api/2/issue/{key}/comment", + params={"startAt": start_at, "maxResults": 100}, + ) + batch = (data or {}).get("comments") or [] + result.extend(batch) + start_at += len(batch) + total = (data or {}).get("total") + if not batch or not isinstance(total, int) or start_at >= total: + return result + + def myself(self) -> dict[str, Any]: + data = self.call("GET", "/rest/api/2/myself") + return data if isinstance(data, dict) else {} + + def attach(self, key: str, name: str, content: bytes) -> Any: + boundary = "magpie-" + secrets.token_hex(16) + safe_name = name.replace('"', "_").replace("\r", "_").replace("\n", "_") + body = ( + ( + f"--{boundary}\r\n" + f'Content-Disposition: form-data; name="file"; filename="{safe_name}"\r\n' + "Content-Type: application/octet-stream\r\n\r\n" + ).encode() + + content + + f"\r\n--{boundary}--\r\n".encode() + ) + headers = { + "Content-Type": f"multipart/form-data; boundary={boundary}", + "X-Atlassian-Token": "no-check", + } + return self.raw( + "POST", f"/rest/api/2/issue/{key}/attachments", body=body, headers=headers + ) diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/README.md b/plugins/magpie-vetted-ops/tools/local-tracker/README.md new file mode 100644 index 000000000..06e9cda1a --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/README.md @@ -0,0 +1,178 @@ + + + + + +- [Local file tracker](#local-file-tracker) + - [Prerequisites](#prerequisites) + - [Invocation](#invocation) + - [The descriptor](#the-descriptor) + - [Default format](#default-format) + - [Confidentiality](#confidentiality) + - [Publishing](#publishing) + - [Testing](#testing) + + + + + +# Local file tracker + +**Capability:** contract:security-tracker + +**Kind:** implementation + +**Vendor:** agnostic + +A security tracker kept as Markdown files in a folder of a private Git or SVN working copy. +It implements the [security-tracker contract](https://github.com/apache/magpie/blob/main/tools/security-tracker/contract.md), +so the `security-*` skills run against it unchanged when a project sets `tracker.platform: local-files`. + +It is for teams that already keep their reports as files — often in a private SVN area or a private repository — +and for teams that want the tracker to be nothing more than text they can read, grep, diff and review. +When reports already exist, the tool learns their layout and writes new reports the same way; +when there are none, it uses the default format below. + +## Prerequisites + +- **Runtime:** Python 3.11+ run via `uv` (stdlib only, no third-party dependencies). +- **CLIs:** `uv`; `git` or `svn` for `pending` / `publish` and for history-preserving moves; `gh` is used by `doctor` only, to read a GitHub remote's visibility. +- **Credentials / auth:** None of its own. `publish --push` uses whatever credentials the working copy already pushes with. +- **Network:** Offline for every tracker operation. `publish --push` reaches the working copy's remote, and `doctor` may call the GitHub API through `gh`. + +## Invocation + +```bash +uv run --project /tools/local-tracker magpie-local-tracker --root [args] +``` + +`--root` is the tracker folder, resolved from `tracker.local.root` in `/project.md` +(or the `MAGPIE_LOCAL_TRACKER_ROOT` environment variable). +Every operation of the contract is a subcommand with the arguments, JSON output and exit codes the contract defines: +`view`, `list`, `labels`, `field-get`, `create`, `edit`, `field-set`, `comment`, `close`, `reopen`, +`rollup-append`, `rollup-amend-latest`, `attach`, `body`, `pending`, `publish`, `doctor`. + +Two subcommands are this backend's own, used by `/magpie-setup config`: + +| Subcommand | What it does | +|---|---| +| `detect [--write] [--force]` | Reads the existing reports, prints the inferred format with the evidence behind every decision, and with `--write` saves it as the descriptor. | +| `init [--force]` | Saves the default descriptor in an empty tracker folder. | + +Both take `--prefix` (ID prefix for new reports), `--url-template` (a browse URL with `{path}`), +and `--access-restricted` with an optional `--access-note` — see [Confidentiality](#confidentiality). + +## The descriptor + +The format lives in `/.magpie-tracker.json`, committed with the reports so everyone on the team writes the same shape. +It records: + +| Field | Meaning | +|---|---| +| `layout` | `folder-per-issue` (`//`) or `file-per-issue` (`/.md`). | +| `issue_file` | The report file inside a folder; `{id}` stands for the folder name. | +| `metadata` | `front-matter` (a `---` block) or `bold-fields` (`**Key:** value` lines). | +| `keys` | Contract role → the key the team uses: `title`, `state`, `labels`, `assignees`, `milestone`, `author`, `created`, `updated`. An empty key means the role is not stored. | +| `title_source` | `key`, or `h1` when the title is the report's first `# ` heading. | +| `id` | `prefix`, zero-pad `width`, and `slug` when names carry a title slug (`2024-007-xss-in-login`). | +| `state` | `mode: field` (a metadata value) or `mode: directory` (a sub-folder per state, such as `open/` and `closed/`), and the values or folder names that count as open and closed. | +| `field_heading_level` | The heading level of body fields (`###` by default). | +| `comments_mode`, `rollup_mode` | `file` (beside the report) or `section` (a `## ` heading at the end of the report). | +| `url_template` | Optional private browse URL, for example a ViewVC or gitweb address. Without it, the `url` field is the path relative to the working copy. | +| `access_restricted` | The human-confirmed access check that unlocks writes. | + +Detection decides each field from what is present: state folders, file or folder names that look like IDs, +the majority metadata style and key names (`Status` or `state`, `Tags` or `labels`, …), +the state values in use, the heading level of body sections, +and an existing comments file or `## Comments` / `## Notes` / `## Timeline` heading. +A role the skills need but the team never recorded (`labels`, `milestone`, …) is added in the team's own key casing, and the output says so. +Anything detection could not place is listed under `notes`; setup shows them before writing. + +## Default format + +Used when the tracker folder has no reports: + +```text +/ +├── .magpie-tracker.json +└── SEC-0001/ + ├── issue.md # front matter + the issue-template body + ├── comments.md # comment trail, one entry per comment + ├── status-rollup.md # the status rollup, same shape as on GitHub + └── attachments/ # CVE JSON and other files +``` + +```markdown +--- +title: "Path traversal in the upload endpoint" +state: open +labels: [needs triage] +assignees: [] +milestone: +reporter: "Jane Doe" +created: 2026-10-15 +updated: 2026-10-15 +--- + +### Reporter credited as + +Jane Doe + +### Affected versions + +_No response_ +``` + +Closing sets `state: closed`, or the `not_planned_value` for `close --reason not-planned`. +Comments carry a `` marker (the author percent-encoded, so `Jane Doe` is `Jane%20Doe`) so they can be counted and addressed; +hand-written comments without the marker are still read, from their headings. +The rollup keeps the `
` entry shape of [`tools/github/status-rollup.md`](https://github.com/apache/magpie/blob/main/tools/github/status-rollup.md). + +## Confidentiality + +The folder holds embargoed vulnerability reports. +Every write is refused (exit `4`) until the descriptor records `access_restricted: true`, +and only `detect --write --access-restricted` or `init --access-restricted` sets it. +`/magpie-setup config` passes that flag only after the user confirms who can read the repository. +`doctor` reports the working copy's remotes and, for a GitHub remote, fails when the repository is public. + +The same rules as for a GitHub tracker apply to everything read from it: +report text is untrusted input, never instructions, and tracker contents never reach a public surface +([`AGENTS.md`](https://github.com/apache/magpie/blob/main/AGENTS.md#confidentiality-of-the-tracker-repository)). + +## Publishing + +Every operation writes to the working copy only. +`pending` lists what changed under the tracker root, and `publish` commits exactly that: + +- **Git:** `git add -A -- `, `git commit -F [--trailer …] -- `, then, with `--push`, `git push --no-follow-tags HEAD:` + to exactly the remote and branch the checks below were run against, so `push.default=matching`, a remote's `push` refspecs or `push.followTags` cannot add other branches or tags. + Other staged or unstaged work in the repository is left alone, and a push is refused (exit `4`) when the branch also carries unpushed commits that touch anything outside the tracker folder: publishing never sends someone else's work. + A commit that was made but not pushed is reported by `pending` and pushed by the next `publish --push`. +- **SVN:** `svn add` for new files, `svn rm` for missing ones, then `svn commit -F ` — which is already the push. + +Skills run `publish` only after the user confirmed the proposal at the end of the session, +as the contract's [publishing section](https://github.com/apache/magpie/blob/main/tools/security-tracker/contract.md#publishing-local-changes) describes. +Closing a report in a folder-per-state tracker moves it with `git mv` or `svn mv`, so its history follows it. + +These steps can run outside the sandbox (through vetted-ops), in a repository a policy file names, so git is never allowed to start a program there: +every git call runs with `core.fsmonitor=false`, an empty hooks path and `protocol.allow=never` (only `https` and `ssh` re-allowed), commit and push pass `--no-verify`, +and any git step is refused when the repository's own config sets a key outside a known-safe list (filter, merge or diff drivers, aliases, includes, URL rewrites, `gpg.program`, …). +That check runs before every git command, and git reads attributes from the empty tree (`GIT_ATTR_SOURCE`, git 2.42 and later), never from the working tree, +so no `.gitattributes` can switch on a filter, diff or merge driver, even one configured after the check. +Automatic `gc` and maintenance are off, only the `https` and `ssh` transports are allowed, and `publish` pushes only to an `https://` / `ssh://` remote: a local-path remote would run that repository's receive hooks here. +The config check runs before every git command; a process that rewrites `.git/config` in the instant between the check and the command is the one window left, which only running the step sandboxed closes. + +Two people allocating an ID at the same time get the same number. +On Git the second push is rejected as non-fast-forward; on SVN the second commit fails as out of date. +Update, rename the later report with `git mv` / `svn mv`, and publish again. + +## Testing + +```bash +uv run --project tools/local-tracker --group dev pytest tools/local-tracker/tests +``` + +The Git publish test runs against a temporary bare repository; the SVN test runs when `svn` and `svnadmin` are installed. diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/pyproject.toml b/plugins/magpie-vetted-ops/tools/local-tracker/pyproject.toml new file mode 100644 index 000000000..b46f0434e --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/pyproject.toml @@ -0,0 +1,88 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "magpie-local-tracker" +version = "0.1.0" +description = "Local folder-and-markdown security tracker for Apache Magpie (security-tracker contract over a private working copy)." +readme = "README.md" +requires-python = ">=3.11" +license = { text = "Apache-2.0" } +dependencies = [] + +[project.scripts] +magpie-local-tracker = "magpie_local_tracker:main" + +[tool.hatch.build.targets.wheel] +packages = ["src/magpie_local_tracker"] + +[tool.ruff] +line-length = 110 +target-version = "py311" +src = ["src", "tests"] + +[tool.ruff.lint] +select = [ + "E", + "W", + "F", + "I", + "B", + "UP", + "SIM", + "C4", + "RUF", +] +ignore = [ + "E501", +] + +[tool.ruff.lint.per-file-ignores] +"tests/**" = ["B", "SIM"] + +[tool.mypy] +python_version = "3.11" +files = ["src", "tests"] +warn_unused_ignores = true +warn_redundant_casts = true +warn_unreachable = true +check_untyped_defs = true +no_implicit_optional = true +disallow_untyped_defs = true +disallow_incomplete_defs = true + +[[tool.mypy.overrides]] +module = "tests.*" +disallow_untyped_defs = false +disallow_incomplete_defs = false + +[tool.pytest.ini_options] +minversion = "8.0" +addopts = "-ra -q" +testpaths = ["tests"] + +# No `[dependency-groups] dev` here: this project also ships inside the +# `magpie-vetted-ops` plugin (vetted-ops runs it as a security-tracker backend +# with `uv run --project /tools/`), and `magpie-dev` only resolves +# through the workspace root's `[tool.uv.sources]`, which the plugin does not +# ship — a `dev` group would make every plugin invocation fail. The shared +# toolchain still reaches the tests: CI and the workspace checks sync the root +# `dev` group. See tools/vetted-ops/pyproject.toml. diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/__init__.py b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/__init__.py new file mode 100644 index 000000000..2d3876143 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/__init__.py @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 +"""Local, file-based security tracker backend for Apache Magpie.""" + +from .cli import main + +__all__ = ["main"] diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/cli.py b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/cli.py new file mode 100644 index 000000000..3d5118a86 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/cli.py @@ -0,0 +1,438 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 +"""``magpie-local-tracker`` — the security-tracker contract over local files. + +The subcommands, arguments, JSON outputs and exit codes are the ones in +``tools/security-tracker/contract.md``; ``detect`` and ``init`` are this +backend's own setup commands. +""" + +from __future__ import annotations + +import argparse +import datetime as _dt +import getpass +import json +import os +import subprocess +import sys +from pathlib import Path + +from . import detect as detect_mod +from .descriptor import DESCRIPTOR_NAME, Descriptor, load, save +from .store import Tracker, TrackerError +from .vcs import VcsError, VcsRefused +from .vcs import detect as detect_vcs + + +def _emit(payload: object) -> None: + json.dump(payload, sys.stdout, indent=2, ensure_ascii=False) + sys.stdout.write("\n") + + +def _read_file(path: str) -> str: + if path == "-": + return sys.stdin.read() + try: + return Path(path).read_text(encoding="utf-8") + except OSError as exc: + raise TrackerError(f"cannot read {path}: {exc}", 2) from exc + + +def _default_user() -> str: + for args in (["git", "config", "user.name"],): + try: + out = subprocess.run(args, capture_output=True, text=True, check=False).stdout.strip() + except FileNotFoundError: + out = "" + if out: + return out + return os.environ.get("USER") or getpass.getuser() + + +def _root(args: argparse.Namespace) -> Path: + raw = args.root or os.environ.get("MAGPIE_LOCAL_TRACKER_ROOT") + if not raw: + raise TrackerError("--root (or MAGPIE_LOCAL_TRACKER_ROOT) is required", 2) + return Path(raw) + + +def _tracker(args: argparse.Namespace) -> Tracker: + root = _root(args) + try: + desc = load(root) + except FileNotFoundError as exc: + raise TrackerError(str(exc), 3) from exc + except ValueError as exc: + raise TrackerError(f"invalid tracker descriptor: {exc}", 5) from exc + return Tracker(root, desc) + + +def _github_visibility(remote: str) -> str | None: + """Visibility of a GitHub remote via `gh`, or None when it cannot be told.""" + if "github.com" not in remote: + return None + slug = remote.rstrip("/").removesuffix(".git").split("github.com")[-1].lstrip(":/") + try: + proc = subprocess.run( + ["gh", "repo", "view", slug, "--json", "visibility", "--jq", ".visibility"], + capture_output=True, + text=True, + check=False, + ) + except FileNotFoundError: + return None + return proc.stdout.strip().lower() or None if proc.returncode == 0 else None + + +# ---------------------------------------------------------------- commands + + +def cmd_detect(args: argparse.Namespace) -> int: + root = _root(args) + desc, evidence = detect_mod.detect(root) + existing = root / DESCRIPTOR_NAME + if existing.is_file() and not args.force: + evidence["decisions"].insert(0, f"{existing} already exists; showing what detection would write") # type: ignore[attr-defined] + _apply_setup_flags(desc, args) + out: dict[str, object] = {"descriptor": json.loads(desc.to_json()), "summary": detect_mod.describe(desc)} + out.update(evidence) + if args.write: + if existing.is_file() and not args.force: + raise TrackerError(f"{existing} exists; pass --force to replace it", 4) + root.mkdir(parents=True, exist_ok=True) + out["written"] = str(save(root, desc)) + _emit(out) + return 0 + + +def _apply_setup_flags(desc: Descriptor, args: argparse.Namespace) -> None: + if getattr(args, "prefix", None): + desc.id.prefix = args.prefix + if getattr(args, "url_template", None): + desc.url_template = args.url_template + if getattr(args, "access_restricted", False): + desc.access_restricted = True + desc.access_note = args.access_note or ( + f"confirmed access-restricted by {_default_user()} on {_dt.date.today().isoformat()}" + ) + + +def cmd_init(args: argparse.Namespace) -> int: + root = _root(args) + existing = root / DESCRIPTOR_NAME + if existing.is_file() and not args.force: + raise TrackerError(f"{existing} exists; pass --force to replace it", 4) + desc = Descriptor() + _apply_setup_flags(desc, args) + root.mkdir(parents=True, exist_ok=True) + path = save(root, desc) + _emit({"written": str(path), "summary": detect_mod.describe(desc)}) + return 0 + + +def cmd_view(args: argparse.Namespace) -> int: + _emit(_tracker(args).view(args.id, with_comments=args.comments)) + return 0 + + +def cmd_list(args: argparse.Namespace) -> int: + _emit(_tracker(args).list_issues(args.state, args.label, args.search or "", args.limit)) + return 0 + + +def cmd_labels(args: argparse.Namespace) -> int: + _emit(_tracker(args).labels()) + return 0 + + +def cmd_field_get(args: argparse.Namespace) -> int: + _emit(_tracker(args).field_get(args.id, args.heading)) + return 0 + + +def _title_arg(args: argparse.Namespace) -> str | None: + """The title from ``--title`` (argv, never a shell) or ``--title-file``.""" + if args.title is not None and args.title_file: + raise TrackerError("--title and --title-file are mutually exclusive", 2) + if args.title is not None: + return str(args.title) + return _read_file(args.title_file) if args.title_file else None + + +def cmd_create(args: argparse.Namespace) -> int: + tracker = _tracker(args) + title = _title_arg(args) + if title is None: + raise TrackerError("create needs --title or --title-file", 2) + _emit(tracker.create(title, _read_file(args.body_file), args.label, args.author or "")) + return 0 + + +def cmd_edit(args: argparse.Namespace) -> int: + _emit( + _tracker(args).edit( + args.id, + args.add_label, + args.remove_label, + args.add_assignee, + args.remove_assignee, + args.milestone, + args.clear_milestone, + _title_arg(args), + _read_file(args.body_file) if args.body_file else None, + ) + ) + return 0 + + +def cmd_field_set(args: argparse.Namespace) -> int: + _emit(_tracker(args).field_set(args.id, args.heading, _read_file(args.value_file))) + return 0 + + +def cmd_comment(args: argparse.Namespace) -> int: + _emit(_tracker(args).comment(args.id, _read_file(args.body_file), args.author or _default_user())) + return 0 + + +def cmd_close(args: argparse.Namespace) -> int: + _emit(_tracker(args).set_state(args.id, True, args.reason)) + return 0 + + +def cmd_reopen(args: argparse.Namespace) -> int: + _emit(_tracker(args).set_state(args.id, False)) + return 0 + + +def _now(args: argparse.Namespace) -> str: + return args.now or _dt.datetime.now(_dt.UTC).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def _user(args: argparse.Namespace) -> str: + user = args.user or _default_user() + return user if user.startswith("@") else f"@{user}" + + +def cmd_rollup_append(args: argparse.Namespace) -> int: + body = _read_file(args.entry_body_file) + _emit(_tracker(args).rollup_append(args.id, args.action, body, _user(args), _now(args))) + return 0 + + +def cmd_rollup_amend_latest(args: argparse.Namespace) -> int: + _emit(_tracker(args).rollup_amend_latest(args.id, args.action, _read_file(args.entry_body_file))) + return 0 + + +def cmd_attach(args: argparse.Namespace) -> int: + if args.file == "-": + if not args.name: + raise TrackerError("attaching from stdin needs --name", 2) + content, name = sys.stdin.buffer.read(), args.name + else: + source = Path(args.file) + if not source.is_file(): + raise TrackerError(f"{source} is not a file", 3) + content, name = source.read_bytes(), args.name or source.name + _emit(_tracker(args).attach(args.id, content, name)) + return 0 + + +def cmd_body(args: argparse.Namespace) -> int: + # Raw markdown, not JSON: it is piped straight into generate-cve-json --stdin. + sys.stdout.write(_tracker(args).body(args.id)) + return 0 + + +def cmd_pending(args: argparse.Namespace) -> int: + root = _root(args) + wc = detect_vcs(root) + changes = wc.status(root) if root.exists() else [] + unpushed = wc.unpushed() + _emit( + { + "backend": "local-files", + "vcs": wc.kind, + "changes": changes, + "unpushed_commits": unpushed, + "push_destination": wc.push_destination(), + "needs_publish": bool(changes) or bool(unpushed), + } + ) + return 0 + + +def cmd_publish(args: argparse.Namespace) -> int: + tracker = _tracker(args) + tracker.require_writable() + message = _read_file(args.message_file) + if not message.strip(): + raise TrackerError("the commit message file is empty", 2) + result = tracker.wc.publish(tracker.root, message, args.trailer, args.push) + result["vcs"] = tracker.wc.kind + _emit(result) + return 0 + + +def cmd_doctor(args: argparse.Namespace) -> int: + root = _root(args) + checks: list[dict[str, object]] = [] + checks.append({"name": "root exists", "ok": root.is_dir(), "detail": str(root)}) + desc: Descriptor | None = None + try: + desc = load(root) + checks.append({"name": "descriptor", "ok": True, "detail": detect_mod.describe(desc)}) + except (FileNotFoundError, ValueError) as exc: + checks.append({"name": "descriptor", "ok": False, "detail": str(exc)}) + wc = detect_vcs(root) if root.exists() else None + checks.append( + { + "name": "working copy", + "ok": bool(wc and wc.kind != "none"), + "detail": f"{wc.kind} at {wc.top}" if wc else "root missing", + } + ) + remotes = wc.remotes() if wc else [] + checks.append({"name": "remotes", "ok": bool(remotes), "detail": remotes}) + for remote in remotes: + visibility = _github_visibility(remote) + if visibility: + checks.append( + {"name": f"visibility {remote}", "ok": visibility != "public", "detail": visibility} + ) + checks.append( + { + "name": "access restricted (human-confirmed)", + "ok": bool(desc and desc.access_restricted), + "detail": desc.access_note if desc else "", + } + ) + _emit({"backend": "local-files", "ok": all(bool(c["ok"]) for c in checks), "checks": checks}) + return 0 + + +# ---------------------------------------------------------------- parser + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(prog="magpie-local-tracker", description=__doc__) + parser.add_argument("--root", help="tracker folder (or MAGPIE_LOCAL_TRACKER_ROOT)") + sub = parser.add_subparsers(dest="command", required=True) + + def add(name: str, func: object, help_text: str) -> argparse.ArgumentParser: + p = sub.add_parser(name, help=help_text) + p.set_defaults(func=func) + return p + + for name, func, help_text in ( + ("detect", cmd_detect, "infer the format from existing reports"), + ("init", cmd_init, "write the default format descriptor"), + ): + p = add(name, func, help_text) + if name == "detect": + p.add_argument("--write", action="store_true", help="write the descriptor") + p.add_argument("--force", action="store_true", help="replace an existing descriptor") + p.add_argument("--prefix", help="ID prefix for new reports") + p.add_argument("--url-template", help="browse URL with {path}") + p.add_argument("--access-restricted", action="store_true", help="record the human access check") + p.add_argument("--access-note", help="who checked the access control, and how") + + p = add("view", cmd_view, "show one report") + p.add_argument("id") + p.add_argument("--comments", action="store_true") + + p = add("list", cmd_list, "list reports") + p.add_argument("--state", choices=["open", "closed", "all"], default="open") + p.add_argument("--label", action="append", default=[]) + p.add_argument("--search") + p.add_argument("--limit", type=int, default=100) + + add("labels", cmd_labels, "labels in use") + + p = add("field-get", cmd_field_get, "read one body field") + p.add_argument("id") + p.add_argument("heading") + + p = add("create", cmd_create, "create a report") + p.add_argument("--title-file") + p.add_argument("--title", help="title as one argv value (callers that never use a shell)") + p.add_argument("--body-file", required=True) + p.add_argument("--label", action="append", default=[]) + p.add_argument("--author") + + p = add("edit", cmd_edit, "edit labels, assignees, milestone, title or body") + p.add_argument("id") + for flag in ("--add-label", "--remove-label", "--add-assignee", "--remove-assignee"): + p.add_argument(flag, action="append", default=[]) + p.add_argument("--milestone") + p.add_argument("--clear-milestone", action="store_true") + p.add_argument("--title-file") + p.add_argument("--title", help="title as one argv value (callers that never use a shell)") + p.add_argument("--body-file") + + p = add("field-set", cmd_field_set, "replace one body field") + p.add_argument("id") + p.add_argument("heading") + p.add_argument("--value-file", required=True) + + p = add("comment", cmd_comment, "add a comment") + p.add_argument("id") + p.add_argument("--body-file", required=True) + p.add_argument("--author") + + p = add("close", cmd_close, "close a report") + p.add_argument("id") + p.add_argument("--reason", choices=["completed", "not-planned"], default="completed") + + p = add("reopen", cmd_reopen, "reopen a report") + p.add_argument("id") + + for name, func in ( + ("rollup-append", cmd_rollup_append), + ("rollup-amend-latest", cmd_rollup_amend_latest), + ): + p = add(name, func, "status rollup") + p.add_argument("id") + p.add_argument("--action", required=True) + p.add_argument("--entry-body-file", required=True) + p.add_argument("--user") + p.add_argument("--now") + + p = add("body", cmd_body, "print the report body as markdown (for generate-cve-json --stdin)") + p.add_argument("id") + + p = add("attach", cmd_attach, "attach a file") + p.add_argument("id") + p.add_argument("file") + p.add_argument("--name") + + add("pending", cmd_pending, "uncommitted tracker changes") + + p = add("publish", cmd_publish, "commit (and push) the tracker changes") + p.add_argument("--message-file", required=True) + p.add_argument("--trailer", action="append", default=[]) + p.add_argument("--push", action="store_true") + + add("doctor", cmd_doctor, "check the tracker setup") + return parser + + +def main(argv: list[str] | None = None) -> int: + parser = build_parser() + args = parser.parse_args(argv) + try: + return int(args.func(args)) + except TrackerError as exc: + print(f"error: {exc}", file=sys.stderr) + return exc.code + except VcsRefused as exc: + print(f"error: {exc}", file=sys.stderr) + return 4 + except VcsError as exc: + print(f"error: {exc}", file=sys.stderr) + return 5 + except OSError as exc: + print(f"error: {exc}", file=sys.stderr) + return 5 diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/descriptor.py b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/descriptor.py new file mode 100644 index 000000000..ed48070b6 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/descriptor.py @@ -0,0 +1,169 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 +"""The format descriptor: how one tracker folder lays out its reports. + +``detect`` writes it once, from the reports already in the folder (or the +default below when there are none); every other operation reads it, so the +tool writes new reports exactly the way the team already does. It lives at +``/.magpie-tracker.json`` and is committed with the reports, which keeps +everyone on the team writing the same shape. +""" + +from __future__ import annotations + +import json +import re +from dataclasses import asdict, dataclass, field +from pathlib import Path + +DESCRIPTOR_NAME = ".magpie-tracker.json" +SCHEMA_VERSION = 1 + + +@dataclass +class IdFormat: + # The prefix new reports get. + prefix: str = "SEC-" + width: int = 4 + # Existing names like ``2024-007-xss-in-login`` carry a slug after the + # number; new reports then get one too, derived from the title. + slug: bool = False + # Further prefixes existing reports use (``2025-`` next to ``2026-``). + # They are recognised, never allocated. + other_prefixes: list[str] = field(default_factory=list) + + def regex(self) -> re.Pattern[str]: + prefixes = sorted({self.prefix, *self.other_prefixes}, key=len, reverse=True) + alternatives = "|".join(re.escape(p) for p in prefixes) + return re.compile("^(?P" + alternatives + r")(?P\d+)(?:[-_](?P[A-Za-z0-9-]+))?$") + + def render(self, number: int, slug: str = "") -> str: + base = f"{self.prefix}{number:0{self.width}d}" + return f"{base}-{slug}" if self.slug and slug else base + + +@dataclass +class StateFormat: + # ``field``: a metadata key holds the state. ``directory``: the report sits + # in a sub-folder per state (``open/``, ``closed/``) and moving it is the + # state change. + mode: str = "field" + open_value: str = "open" + closed_value: str = "closed" + not_planned_value: str = "closed" + open_values: list[str] = field(default_factory=lambda: ["open"]) + closed_values: list[str] = field(default_factory=lambda: ["closed"]) + open_dir: str = "open" + closed_dir: str = "closed" + + +@dataclass +class Descriptor: + version: int = SCHEMA_VERSION + # ``folder-per-issue``: //. ``file-per-issue``: /.md. + layout: str = "folder-per-issue" + issue_file: str = "issue.md" + metadata: str = "front-matter" + list_style: str = "inline" + # role -> the concrete metadata key. A role whose key is ``""`` is not + # stored (e.g. no ``milestone`` in a team's reports). + keys: dict[str, str] = field( + default_factory=lambda: { + "title": "title", + "state": "state", + "labels": "labels", + "assignees": "assignees", + "milestone": "milestone", + "author": "reporter", + "created": "created", + "updated": "updated", + } + ) + # ``key`` (metadata) or ``h1`` (first ``# `` heading of the body). + title_source: str = "key" + field_heading_level: int = 3 + id: IdFormat = field(default_factory=IdFormat) + state: StateFormat = field(default_factory=StateFormat) + # ``file``: /comments.md. ``section``: a heading inside the issue file. + comments_mode: str = "file" + comments_file: str = "comments.md" + comments_heading: str = "Comments" + rollup_mode: str = "file" + rollup_file: str = "status-rollup.md" + rollup_heading: str = "Status rollup" + attachments_dir: str = "attachments" + # Optional URL to browse a report, ``{path}`` = path relative to the + # repository root (e.g. a private ViewVC / gitweb URL). + url_template: str = "" + # Set only by a human, through `init|detect --write --access-restricted`, + # after checking who can read the repository. Every write refuses while it + # is false: the folder holds embargoed vulnerability reports. + access_restricted: bool = False + access_note: str = "" + notes: list[str] = field(default_factory=list) + + def to_json(self) -> str: + return json.dumps(asdict(self), indent=2, sort_keys=False) + "\n" + + @classmethod + def from_dict(cls, data: dict[str, object]) -> Descriptor: + raw = dict(data) + id_raw = raw.pop("id", {}) or {} + state_raw = raw.pop("state", {}) or {} + known = set(cls.__dataclass_fields__) + desc = cls(**{k: v for k, v in raw.items() if k in known}) # type: ignore[arg-type] + if isinstance(id_raw, dict): + desc.id = IdFormat(**{k: v for k, v in id_raw.items() if k in IdFormat.__dataclass_fields__}) + if isinstance(state_raw, dict): + desc.state = StateFormat( + **{k: v for k, v in state_raw.items() if k in StateFormat.__dataclass_fields__} + ) + default_keys = Descriptor().keys + desc.keys = {**default_keys, **desc.keys} + return desc + + +# One path component: ordinary filename characters, spaces included (detection +# accepts a state folder named `in progress` or a report file with spaces), but +# no separator, NUL, control character or surrounding whitespace. +_NAME_RE = re.compile(r"^(?! )[^/\\\x00-\x1f\x7f]+(? None: + """Every file or folder name the descriptor supplies stays inside the tracker. + + The descriptor is committed with the reports, so anyone who can push to the + tracker can edit it; a name like ``../../.ssh/config`` must never become a + path the tool reads or writes. + """ + names = { + "issue_file": desc.issue_file, + "comments_file": desc.comments_file, + "rollup_file": desc.rollup_file, + "attachments_dir": desc.attachments_dir, + "state.open_dir": desc.state.open_dir, + "state.closed_dir": desc.state.closed_dir, + } + for key, value in names.items(): + if not _NAME_RE.match(value) or value in (".", "..") or ".." in value: + raise ValueError(f"descriptor {key} must be a plain file or folder name, got {value!r}") + + +def load(root: Path) -> Descriptor: + path = root / DESCRIPTOR_NAME + if not path.is_file(): + raise FileNotFoundError( + f"{path} not found — run `magpie-local-tracker detect --write` (or `init`) first" + ) + data = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(data, dict): + raise ValueError(f"{path}: expected a JSON object") + desc = Descriptor.from_dict(data) + _check_names(desc) + return desc + + +def save(root: Path, desc: Descriptor) -> Path: + path = root / DESCRIPTOR_NAME + path.write_text(desc.to_json(), encoding="utf-8") + return path diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/detect.py b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/detect.py new file mode 100644 index 000000000..c67dd437a --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/detect.py @@ -0,0 +1,285 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 +"""Infer a descriptor from the reports a team already keeps. + +Every choice is made from evidence — the names, folders, metadata keys and +headings actually present — and the evidence is returned with the result, so +the person running setup can see why the tool decided what it did and correct +it before anything is written. With no reports present the default +descriptor applies. +""" + +from __future__ import annotations + +import re +from collections import Counter +from pathlib import Path + +from . import metadata as md +from .descriptor import DESCRIPTOR_NAME, Descriptor, IdFormat, StateFormat + +OPEN_WORDS = [ + "open", "new", "triage", "needs-triage", "needs triage", "in-progress", "in progress", + "active", "accepted", "valid", "confirmed", "pending", "draft", "embargoed", +] # fmt: skip +CLOSED_WORDS = [ + "closed", "fixed", "resolved", "done", "published", "rejected", "invalid", "wontfix", + "won't fix", "duplicate", "archive", "archived", "disclosed", "not-planned", "not planned", +] # fmt: skip +NOT_PLANNED_WORDS = ["rejected", "invalid", "wontfix", "won't fix", "not-planned", "not planned"] + +ROLE_SYNONYMS: dict[str, list[str]] = { + "title": ["title", "summary", "subject"], + "state": ["state", "status"], + "labels": ["labels", "tags", "keywords"], + "assignees": ["assignees", "assignee", "owner", "owners", "handler", "assigned to"], + "milestone": ["milestone", "target", "fix version", "fix-version", "fix_version", "release"], + "author": ["reporter", "author", "reported by", "reported-by", "reported_by", "finder"], + "created": ["created", "date", "reported", "opened", "created_at", "reported on"], + "updated": ["updated", "last updated", "last-updated", "modified", "updated_at"], +} +# Roles the security skills need even when a team's reports never had them. +REQUIRED_ROLES = ("title", "state", "labels", "assignees", "milestone") + +COMMENT_HEADINGS = ["Comments", "Discussion", "Notes", "Timeline", "History", "Log"] +COMMENT_FILES = ["comments.md", "discussion.md", "notes.md", "timeline.md", "history.md"] + +_ID_RE = re.compile(r"^(?P.*?[-_])?(?P\d+)(?:[-_](?P[A-Za-z].*))?$") + + +def _candidates(base: Path) -> tuple[list[Path], list[Path]]: + files: list[Path] = [] + folders: list[Path] = [] + if not base.is_dir(): + return files, folders + for child in sorted(base.iterdir()): + if child.name.startswith(".") or child.name.lower() in {"readme.md", "index.md", "template.md"}: + continue + if child.is_file() and child.suffix == ".md" and _ID_RE.match(child.stem): + files.append(child) + elif child.is_dir() and _ID_RE.match(child.name) and any(child.glob("*.md")): + folders.append(child) + return files, folders + + +def _is_word(name: str, words: list[str]) -> bool: + return name.lower().replace("_", "-") in {w.replace(" ", "-") for w in words} or name.lower() in words + + +def detect(root: Path) -> tuple[Descriptor, dict[str, object]]: + desc = Descriptor() + evidence: dict[str, object] = {"root": str(root), "reports_found": 0, "decisions": []} + decisions: list[str] = evidence["decisions"] # type: ignore[assignment] + + if not root.is_dir() or not [p for p in root.iterdir() if p.name != DESCRIPTOR_NAME]: + decisions.append("no existing reports — using the default format (folder per report, front matter)") + evidence["confidence"] = "default" + return desc, evidence + + # 1. State folders. + state_dirs = [ + d + for d in sorted(root.iterdir()) + if d.is_dir() and (_is_word(d.name, OPEN_WORDS) or _is_word(d.name, CLOSED_WORDS)) + ] + bases: list[Path] = state_dirs or [root] + if state_dirs: + opens = [d.name for d in state_dirs if _is_word(d.name, OPEN_WORDS)] + closeds = [d.name for d in state_dirs if _is_word(d.name, CLOSED_WORDS)] + desc.state = StateFormat( + mode="directory", + open_dir=opens[0] if opens else "open", + closed_dir=closeds[0] if closeds else "closed", + open_values=opens or ["open"], + closed_values=closeds or ["closed"], + ) + decisions.append(f"state is the folder a report sits in: open={opens}, closed={closeds}") + + # 2. Layout. + files: list[Path] = [] + folders: list[Path] = [] + for base in bases: + f, d = _candidates(base) + files += f + folders += d + if not files and not folders: + decisions.append("no file or folder named like a report ID — using the default format") + evidence["confidence"] = "default" + evidence["unrecognized"] = sorted(p.name for p in root.iterdir())[:20] + return Descriptor(), evidence + + issue_paths: list[Path] + if len(folders) > len(files): + desc.layout = "folder-per-issue" + names: Counter[str] = Counter() + same_as_dir = 0 + for folder in folders: + mds = [p.name for p in folder.glob("*.md") if p.name.lower() not in COMMENT_FILES] + if f"{folder.name}.md" in mds: + same_as_dir += 1 + names.update(mds) + if same_as_dir > len(folders) / 2: + desc.issue_file = "{id}.md" + elif names: + desc.issue_file = names.most_common(1)[0][0] + issue_paths = [f / desc.issue_file.replace("{id}", f.name) for f in folders] + issue_paths = [p for p in issue_paths if p.is_file()] + decisions.append(f"one folder per report ({len(folders)} found), report file {desc.issue_file!r}") + stems = [f.name for f in folders] + else: + desc.layout = "file-per-issue" + issue_paths = files + decisions.append(f"one markdown file per report ({len(files)} found)") + stems = [f.stem for f in files] + desc.comments_mode = "section" + desc.rollup_mode = "section" + evidence["reports_found"] = len(stems) + + # 3. Identifier format. + matches = [m for m in (_ID_RE.match(s) for s in stems) if m] + prefixes = Counter(m.group("prefix") or "" for m in matches) + prefix = prefixes.most_common(1)[0][0] + nums = [m.group("num") for m in matches if (m.group("prefix") or "") == prefix] + padded = any(n.startswith("0") for n in nums) + desc.id = IdFormat( + prefix=prefix, + width=max(len(n) for n in nums) if padded else 1, + slug=any(m.group("slug") for m in matches), + ) + decisions.append( + f"IDs look like {desc.id.render(7, 'example')!r} (prefix {prefix!r}, width {desc.id.width}" + + (", with a title slug" if desc.id.slug else "") + + ")" + ) + if len(prefixes) > 1: + # Every prefix in use stays recognised; only the most common one is allocated. + desc.id.other_prefixes = sorted(p for p in prefixes if p != prefix) + desc.notes.append( + f"several ID prefixes in use {dict(prefixes)}; all are recognised, new reports use {prefix!r}" + ) + + # 4. Metadata, keys, state values, headings. + sample = issue_paths[:60] + styles: Counter[str] = Counter() + list_styles: Counter[str] = Counter() + keys_seen: Counter[str] = Counter() + state_values: Counter[str] = Counter() + heading_levels: Counter[int] = Counter() + h1_titles = 0 + comment_headings: Counter[str] = Counter() + for path in sample: + meta, body = md.split(path.read_text(encoding="utf-8", errors="replace")) + styles[meta.style] += 1 + if meta.style == "front-matter": + list_styles[meta.list_style] += 1 + keys_seen.update(meta.values.keys()) + skey = meta.find_key(ROLE_SYNONYMS["state"]) + sval = meta.values.get(skey) if skey else None + if isinstance(sval, str): + state_values[sval] += 1 + if re.search(r"^# .+$", body, re.MULTILINE): + h1_titles += 1 + for line in body.split("\n"): + m = re.match(r"^(#{2,3})\s+(.+?)\s*$", line) + if not m: + continue + text = m.group(2) + if text in COMMENT_HEADINGS and len(m.group(1)) == 2: + comment_headings[text] += 1 + else: + heading_levels[len(m.group(1))] += 1 + + style = styles.most_common(1)[0][0] + desc.metadata = style if style != "none" else "bold-fields" + if style == "none": + desc.notes.append("existing reports carry no metadata block; new reports get **Key:** value lines") + desc.list_style = list_styles.most_common(1)[0][0] if list_styles else "inline" + decisions.append(f"metadata style {style!r} ({dict(styles)})") + + capitalised = sum(1 for k in keys_seen if k[:1].isupper()) > len(keys_seen) / 2 + keys: dict[str, str] = {} + for role, synonyms in ROLE_SYNONYMS.items(): + lowered = {k.lower(): k for k in keys_seen} + found = next((lowered[s] for s in synonyms if s in lowered), "") + if not found and role in REQUIRED_ROLES: + found = role.capitalize() if capitalised else role + if not (role == "state" and desc.state.mode == "directory"): + desc.notes.append( + f"no {role!r} key in existing reports; new and edited reports add {found!r}" + ) + if role == "state" and desc.state.mode == "directory" and found not in keys_seen: + found = "" + keys[role] = found + if "title" not in {r for r, k in keys.items() if k in keys_seen} and h1_titles > len(sample) / 2: + desc.title_source = "h1" + keys["title"] = "" + decisions.append("title is the report's first '# ' heading") + desc.keys = keys + decisions.append(f"metadata keys: { {r: k for r, k in keys.items() if k} }") + + if state_values: + opens = [v for v in state_values if _is_word(v, OPEN_WORDS)] + closeds = [v for v in state_values if _is_word(v, CLOSED_WORDS)] + unknown = [v for v in state_values if v not in opens and v not in closeds] + top_open = max(opens, key=lambda v: state_values[v]) if opens else "open" + top_closed = max(closeds, key=lambda v: state_values[v]) if closeds else "closed" + not_planned = [v for v in closeds if _is_word(v, NOT_PLANNED_WORDS)] + st = desc.state + st.open_value = next((v for v in opens if _is_word(v, ["open", "new"])), top_open) + st.closed_value = next( + (v for v in closeds if _is_word(v, ["fixed", "closed", "resolved"])), top_closed + ) + st.not_planned_value = not_planned[0] if not_planned else st.closed_value + if st.mode == "directory": + # The folder decides; the values only keep the key in step with it. + st.open_values = [*st.open_values, *opens, *unknown] + st.closed_values = [*st.closed_values, *closeds] + else: + st.open_values = opens + unknown or ["open"] + st.closed_values = closeds or ["closed"] + decisions.append(f"state values: open={opens}, closed={closeds}") + if unknown and st.mode == "field": + desc.notes.append( + f"state values {unknown} were not recognised and are treated as open — check them" + ) + + if heading_levels: + desc.field_heading_level = heading_levels.most_common(1)[0][0] + decisions.append(f"report sections use level-{desc.field_heading_level} headings") + + # 5. Comments. + if desc.layout == "folder-per-issue": + files_seen: Counter[str] = Counter( + p.name for f in folders for p in f.iterdir() if p.name.lower() in COMMENT_FILES + ) + if files_seen: + desc.comments_mode = "file" + desc.comments_file = files_seen.most_common(1)[0][0] + decisions.append(f"comments live in {desc.comments_file!r} next to the report") + elif comment_headings: + desc.comments_mode = "section" + desc.comments_heading = comment_headings.most_common(1)[0][0] + decisions.append(f"comments live under '## {desc.comments_heading}' in the report") + elif comment_headings: + desc.comments_heading = comment_headings.most_common(1)[0][0] + decisions.append(f"comments live under '## {desc.comments_heading}' in the report") + + evidence["confidence"] = "high" if not desc.notes else "review" + evidence["sample_size"] = len(sample) + return desc, evidence + + +def describe(desc: Descriptor) -> str: + """A short human description of the format, for setup's confirmation.""" + where = ( + f"/{desc.id.render(1)}/{desc.issue_file.replace('{id}', desc.id.render(1))}" + if desc.layout == "folder-per-issue" + else f"/{desc.id.render(1)}.md" + ) + if desc.state.mode == "directory": + where = where.replace("/", f"/{{{desc.state.open_dir}|{desc.state.closed_dir}}}/") + level = "#" * desc.field_heading_level + return ( + f"{where}; metadata as {desc.metadata}; fields as '{level} ' sections; " + f"comments in {'section ' + desc.comments_heading if desc.comments_mode == 'section' else desc.comments_file}" + ) diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/metadata.py b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/metadata.py new file mode 100644 index 000000000..da7ab95c3 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/metadata.py @@ -0,0 +1,308 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 +"""Read and write the metadata block at the top of an issue file. + +Two styles are understood, because those are the two a hand-kept tracker +tends to grow: + +* ``front-matter`` — a ``---`` fenced block of ``key: value`` lines (the YAML + subset people actually write: scalars, ``[a, b]`` inline lists, ``- item`` + block lists). +* ``bold-fields`` — ``**Key:** value`` lines at the top of the file, before + the first heading. + +Key order and unknown keys survive a round trip, so rewriting one value never +reshuffles a file a human maintains. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field + +Value = str | list[str] | None + +_BOLD_RE = re.compile(r"^\*\*(?P[^*:]+?):?\*\*:?\s*(?P.*)$") +_KEY_RE = re.compile(r"^(?P[A-Za-z][\w .-]*?)\s*:\s*(?P.*)$") + + +@dataclass +class Metadata: + """Ordered metadata with the style it was read in.""" + + style: str = "front-matter" + values: dict[str, Value] = field(default_factory=dict) + list_style: str = "inline" + # Set when the front matter uses YAML this parser does not model (block + # scalars, nested mappings, comments, …); rewriting it would lose data. + unsupported: str = "" + + def find_key(self, candidates: list[str]) -> str | None: + """Return the stored key matching any candidate, case-insensitively.""" + lowered = {k.lower(): k for k in self.values} + for cand in candidates: + if cand.lower() in lowered: + return lowered[cand.lower()] + return None + + +def _unescape_double(inner: str) -> str: + """Undo exactly what `_quote` does: `\\\\` -> `\\`, `\\"` -> `"`; any other + backslash is literal.""" + out: list[str] = [] + i = 0 + while i < len(inner): + ch = inner[i] + if ch == "\\" and i + 1 < len(inner) and inner[i + 1] in '\\"': + out.append(inner[i + 1]) + i += 2 + continue + out.append(ch) + i += 1 + return "".join(out) + + +def _unquote(raw: str) -> str: + raw = raw.strip() + if len(raw) >= 2 and raw[0] == raw[-1] and raw[0] in "\"'": + inner = raw[1:-1] + return _unescape_double(inner) if raw[0] == '"' else inner.replace("''", "'") + return raw + + +def _has_inline_comment(raw: str) -> bool: + """A YAML ``# comment`` after a value. + + A quote only opens a quoted scalar when it *starts* the value (``Bob's + report`` is a plain scalar); inside double quotes a backslash escapes the + next character, inside single quotes ``''`` is a literal quote. After the + scalar, a ``#`` preceded by whitespace starts a comment. + """ + text = raw.strip() + rest = text + if text[:1] == "[": + # A flow sequence: quotes open quoted items at item boundaries only. + i, quote, at_item = 1, "", True + while i < len(text): + ch = text[i] + if quote == '"': + if ch == "\\": + i += 2 + continue + if ch == '"': + quote = "" + elif quote == "'": + if ch == "'": + if text[i + 1 : i + 2] == "'": + i += 2 + continue + quote = "" + elif at_item and ch in "\"'": + quote, at_item = ch, False + elif ch == ",": + at_item = True + elif ch == "]": + break + elif not ch.isspace(): + at_item = False + i += 1 + rest = text[i + 1 :] + return re.search(r"\s#", rest) is not None or rest.lstrip().startswith("#") + if text[:1] == '"': + i = 1 + while i < len(text): + if text[i] == "\\": + i += 2 + continue + if text[i] == '"': + break + i += 1 + rest = text[i + 1 :] + elif text[:1] == "'": + i = 1 + while i < len(text): + if text[i] == "'": + if text[i + 1 : i + 2] == "'": + i += 2 + continue + break + i += 1 + rest = text[i + 1 :] + else: + rest = " " + text + return re.search(r"\s#", rest) is not None or rest.lstrip().startswith("#") + + +def _split_inline_list(raw: str) -> list[str]: + """Items of a flow sequence, by the scalar rules `_has_inline_comment` uses: + a quote opens a quoted item only at an item boundary (``[Bob's report, + triaged]`` is two plain items), ``\\`` escapes inside double quotes, and + ``''`` is a literal quote inside single quotes.""" + inner = raw.strip()[1:-1] + items: list[str] = [] + buf = "" + quote = "" + at_item = True + i = 0 + while i < len(inner): + ch = inner[i] + if quote == '"': + buf += ch + if ch == "\\" and i + 1 < len(inner): + buf += inner[i + 1] + i += 2 + continue + if ch == '"': + quote = "" + elif quote == "'": + buf += ch + if ch == "'": + if inner[i + 1 : i + 2] == "'": + buf += "'" + i += 2 + continue + quote = "" + elif ch == ",": + items.append(buf) + buf, at_item = "", True + elif at_item and ch in "\"'": + quote, at_item = ch, False + buf += ch + else: + if not ch.isspace(): + at_item = False + buf += ch + i += 1 + if buf.strip(): + items.append(buf) + return [_unquote(i) for i in items if i.strip()] + + +def _parse_scalar_or_list(raw: str) -> Value: + raw = raw.strip() + if raw.startswith("[") and raw.endswith("]"): + return _split_inline_list(raw) + if raw in ("", "~", "null"): + return None + return _unquote(raw) + + +def split(text: str) -> tuple[Metadata, str]: + """Split ``text`` into its metadata and the remaining markdown body.""" + lines = text.splitlines(keepends=True) + if lines and lines[0].strip() == "---": + meta = Metadata(style="front-matter") + current_list: str | None = None + for idx in range(1, len(lines)): + line = lines[idx].rstrip("\n").rstrip("\r") + if line.strip() == "---": + body = "".join(lines[idx + 1 :]) + return meta, body.lstrip("\n") + if current_list and re.match(r"^\s*-\s+", line): + item = re.sub(r"^\s*-\s+", "", line) + if _has_inline_comment(item): + meta.unsupported = meta.unsupported or "comments" + existing = meta.values.get(current_list) + if not isinstance(existing, list): + existing = [] + meta.values[current_list] = existing + existing.append(_unquote(item)) + meta.list_style = "block" + continue + if line.strip() == "": + continue + if line.lstrip().startswith("#"): + meta.unsupported = meta.unsupported or "comments" + continue + m = _KEY_RE.match(line) + if not m or line[:1].isspace(): + meta.unsupported = meta.unsupported or "nested or multi-line values" + current_list = None + continue + key, value = m.group("key").strip(), m.group("value") + if _has_inline_comment(value): + meta.unsupported = meta.unsupported or "comments" + if value.strip()[:1] in ("|", ">", "{", "&", "*", "!"): + meta.unsupported = ( + meta.unsupported or f"the YAML construct {value.strip()[:1]!r} (key {key!r})" + ) + if value.strip() == "": + meta.values[key] = None + current_list = key + else: + meta.values[key] = _parse_scalar_or_list(value) + current_list = None + # Unterminated front matter: treat the whole file as body. + return Metadata(style="front-matter"), text + + meta = Metadata(style="bold-fields") + title_line = "" + consumed = 0 + for idx, raw_line in enumerate(lines): + line = raw_line.strip() + if line == "": + continue + if line.startswith("# ") and not title_line and not meta.values: + title_line = raw_line if raw_line.endswith("\n") else raw_line + "\n" + consumed = idx + 1 + continue + m = _BOLD_RE.match(line) + if not m: + break + meta.values[m.group("key").strip()] = _parse_bold_value( + m.group("key").strip(), m.group("value").strip() + ) + consumed = idx + 1 + if not meta.values: + return Metadata(style="none"), text + rest = "".join(lines[consumed:]).lstrip("\n") + return meta, (title_line + "\n" + rest) if title_line else rest + + +_LIST_KEYS = {"labels", "tags", "assignees", "assignee", "owners", "components"} + + +def _parse_bold_value(key: str, value: str) -> Value: + if value in ("", "-", "n/a", "N/A", "_No response_"): + return [] if key.lower() in _LIST_KEYS else None + if key.lower() in _LIST_KEYS: + return [v.strip().strip("`") for v in value.split(",") if v.strip()] + return value + + +def _quote(value: str) -> str: + if value == "" or re.search(r"[:#\[\]{},&*!|>'\"%@`]", value) or value != value.strip(): + return '"' + value.replace("\\", "\\\\").replace('"', '\\"') + '"' + return value + + +def render(meta: Metadata, body: str) -> str: + """Render metadata back above ``body`` in the style it was read in.""" + if meta.style == "none" or not meta.values: + return body + if meta.style == "bold-fields": + out = [] + for key, value in meta.values.items(): + text = ", ".join(value) if isinstance(value, list) else value or "" + out.append(f"**{key}:** {text}".rstrip()) + # Keep a leading H1 above the bold block. + if body.startswith("# "): + first, _, rest = body.partition("\n") + return first + "\n\n" + "\n".join(out) + "\n\n" + rest.lstrip("\n") + return "\n".join(out) + "\n\n" + body + out = ["---"] + for key, value in meta.values.items(): + if isinstance(value, list): + if meta.list_style == "block": + out.append(f"{key}:") + out.extend(f" - {_quote(v)}" for v in value) + if not value: + out[-1] = f"{key}: []" + else: + out.append(f"{key}: [" + ", ".join(_quote(v) for v in value) + "]") + elif value is None: + out.append(f"{key}:") + else: + out.append(f"{key}: {_quote(value)}") + out.append("---") + return "\n".join(out) + "\n\n" + body diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/sections.py b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/sections.py new file mode 100644 index 000000000..bd3b7fd41 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/sections.py @@ -0,0 +1,305 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 +"""Markdown section surgery, comment entries, and the status rollup.""" + +from __future__ import annotations + +import re +import urllib.parse +from collections.abc import Callable +from dataclasses import dataclass + +_HEADING_RE = re.compile(r"^(?P#{1,6})\s+(?P.+?)\s*#*\s*$") + + +_FENCE_RE = re.compile(r"^ {0,3}(?P`{3,}|~{3,})(?P.*)$") + + +def _fence_step(open_fence: str, line: str) -> tuple[str, bool]: + """Advance the fence state over one line, the CommonMark way. + + Returns (open fence after the line, whether the line is a fence line). A + fence opens with three or more backticks or tildes and closes only on a + line of the same character, at least as long, with nothing after it. + """ + m = _FENCE_RE.match(line) + if not m: + return open_fence, False + fence, rest = m.group("fence"), m.group("rest") + if open_fence: + if fence[0] == open_fence[0] and len(fence) >= len(open_fence) and not rest.strip(): + return "", True + return open_fence, False + if fence[0] == "`" and "`" in rest: + return "", False # an inline code span, not a fence + return fence, True + + +def _headings(lines: list[str]) -> list[tuple[int, int, str]]: + """(line index, level, text) for every ATX heading outside code fences.""" + out: list[tuple[int, int, str]] = [] + fence = "" + for idx, line in enumerate(lines): + fence, is_fence = _fence_step(fence, line) + if is_fence or fence: + continue + m = _HEADING_RE.match(line) + if m: + out.append((idx, len(m.group("hashes")), m.group("text").strip())) + return out + + +def open_fence(lines: list[str]) -> str: + """The fence marker still open after ``lines`` (```` ``` ```` / ``~~~``), or "". + + The same toggling `_headings` uses, so escaping and parsing agree on what is + code. + """ + fence = "" + for line in lines: + fence, _ = _fence_step(fence, line) + return fence + + +def close_fences(text: str) -> str: + """``text`` with a closing line added for a fence it leaves open. + + Every piece of untrusted text is balanced before it is written, and so is + the existing text it is appended to. Otherwise a body that *starts* with a + fence line could close a fence left open earlier in the file, and a heading + the body-local escaping skipped as "fenced" would parse as a real one. + """ + marker = open_fence(text.split("\n")) + return f"{text.rstrip(chr(10))}\n{marker}" if marker else text + + +def heading_lines(lines: list[str]) -> list[tuple[int, int, str]]: + """(line index, level, text) for every heading outside code fences.""" + return _headings(lines) + + +def find_section(body: str, heading: str, level: int | None = None) -> tuple[int, int, int] | None: + """Locate a section: (heading line, first content line, end line exclusive).""" + lines = body.split("\n") + heads = _headings(lines) + for pos, (idx, lvl, text) in enumerate(heads): + if text.lower() != heading.lower() or (level is not None and lvl != level): + continue + end = len(lines) + for nidx, nlvl, _ in heads[pos + 1 :]: + if nlvl <= lvl: + end = nidx + break + return idx, idx + 1, end + return None + + +# The markers this tool trusts when it reads a report back: comment markers, +# the rollup marker, and the rollup's
/ entry lines. +_FORGEABLE = re.compile( + r"<(?=/?details\b|summary\b|!--\s*comment\s+id=|!--[^\n]*?status rollup v\d)", + re.IGNORECASE, +) + + +def neutralize(text: str, managed_headings: tuple[str, ...] = ()) -> str: + """Make untrusted text unable to forge structure. + + Report, comment and rollup text usually comes from outside the team (a + reporter's mail, a scanner finding). Written verbatim, a line such as + ```` would read back as a comment by + someone else, a ``
…`` line as a rollup entry — + misleading `rollup-amend-latest` too — and a ``## Comments`` heading as the + start of the managed comment section. Each such marker is escaped + (``<`` becomes ``<``, a managed heading gets a leading backslash), which + renders as the same visible text and is never parsed as structure. + """ + text = close_fences(_FORGEABLE.sub("<", text)) + if not managed_headings: + return text + managed = {h.lower() for h in managed_headings} + return escape_headings(text, lambda level, title: level <= 2 and title.strip().lower() in managed) + + +def escape_headings(text: str, which: Callable[[int, str], bool] | None = None) -> str: + """Prefix heading lines with a backslash so they are never parsed as headings. + + Only real headings are touched — the same fence-aware scan the parsers use — + so a ``# comment`` inside a fenced PoC stays exactly as the reporter wrote it. + """ + lines = text.split("\n") + for idx, level, title in _headings(lines): + if which is None or which(level, title): + lines[idx] = "\\" + lines[idx] + return "\n".join(lines) + + +def get_field(body: str, heading: str) -> str | None: + found = find_section(body, heading) + if not found: + return None + _, start, end = found + return "\n".join(body.split("\n")[start:end]).strip("\n") + + +def _join(before: list[str], middle: list[str], after: list[str]) -> str: + """Join three line runs with one blank line at each seam. + + Only the blank lines *at the seams* are normalised. Everything inside + `before` and `after` — fenced PoCs, payloads whose blank lines matter — is + kept verbatim. + """ + while before and before[-1].strip() == "": + before = before[:-1] + marker = open_fence(before) + if marker: + before = [*before, marker] # never let inserted text continue an open fence + while after and after[0].strip() == "": + after = after[1:] + out = [*before, "", *middle] if before else list(middle) + if after: + out += ["", *after] + text = "\n".join(out) + return text.rstrip("\n") + "\n" + + +def set_field(body: str, heading: str, value: str, level: int) -> tuple[str, bool]: + """Replace a section's content, or append the section. Returns (body, created).""" + lines = body.split("\n") + found = find_section(body, heading) + value_lines = value.strip("\n").split("\n") + if found: + idx, _start, end = found + return _join(lines[: idx + 1], value_lines, lines[end:]), False + return _join(lines, ["#" * level + " " + heading, "", *value_lines], []), True + + +def append_to_section(body: str, heading: str, level: int, text: str) -> str: + """Append ``text`` at the end of a section, creating it at the end of the body.""" + lines = body.split("\n") + found = find_section(body, heading, level) + text_lines = text.strip("\n").split("\n") + if found: + _, _, end = found + return _join(lines[:end], text_lines, lines[end:]) + return _join(lines, ["#" * level + " " + heading, "", *text_lines], []) + + +# ---------------------------------------------------------------- comments + +_COMMENT_MARK = re.compile(r"^$") + + +@dataclass +class Comment: + id: int + author: str + created: str + body: str + + +def render_comment(comment_id: int, author: str, date: str, body: str, level: int) -> str: + # Hand-written comments are recognised by their headings, so a heading + # inside a comment body would read back as a comment by someone else: + # escape every heading line in it, as well as the markers. + safe = escape_headings(neutralize(body.strip())) + # The marker's author is percent-encoded: a git user.name such as + # `Jane Doe` (or one containing `-->`) stays a single token there. + shown = " ".join(author.split()) + return ( + f"\n" + f"{'#' * level} {date[:10]} — {shown}\n\n{safe}\n" + ) + + +def parse_comments(text: str) -> list[Comment]: + """Comments written by this tool carry a marker; hand-written ones are + recovered from their headings, so a mixed trail reads in order.""" + comments: list[Comment] = [] + lines = text.split("\n") + heading_at = {idx: title for idx, lvl, title in _headings(lines) if lvl >= 2} + current: Comment | None = None + body: list[str] = [] + skip_heading = False + + def flush() -> None: + if current is not None: + current.body = "\n".join(body).strip() + comments.append(current) + + for idx, line in enumerate(lines): + marker = _COMMENT_MARK.match(line.strip()) + if marker: + flush() + author = urllib.parse.unquote(marker.group("author")) + current = Comment(int(marker.group("id")), author, marker.group("date"), "") + body = [] + skip_heading = True + continue + if idx in heading_at: + if skip_heading: + skip_heading = False + continue + flush() + title = heading_at[idx] + parts = re.split(r"\s+[\u2014\u2013-]\s+", title, maxsplit=1) + date, author = (parts[0], parts[1]) if len(parts) == 2 else (title, "") + current = Comment(len(comments) + 1, author, date, "") + body = [] + continue + if line.strip(): + skip_heading = False + body.append(line) + flush() + return comments + + +def next_comment_id(text: str) -> int: + ids = [c.id for c in parse_comments(text)] + return max(ids, default=0) + 1 + + +# ---------------------------------------------------------------- rollup + +# Any name: a rollup written under a name with spaces must still be found, +# or the next append would start a new rollup and lose the history. +ROLLUP_MARKER_RE = re.compile(r"" + ) + + +def render_entry(date: str, user: str, action: str, body: str) -> str: + # Trim first, then escape: left-trimming must not expose a heading or a + # marker that the escaping did not see. Rollup entries, like comments, may + # sit inside the report (inline rollup section), so every heading outside + # fences is escaped too. + trimmed = "\n".join(line.lstrip() for line in body.strip("\n").split("\n")) + trimmed = escape_headings(neutralize(trimmed)) + return f"
{date} · {user} · {action}\n\n{trimmed}\n\n
" + + +def rollup_append(existing: str | None, name: str, entry: str) -> str: + if not existing or not ROLLUP_MARKER_RE.search(existing): + return rollup_marker(name) + "\n" + entry + "\n" + return close_fences(existing).rstrip("\n") + "\n\n---\n\n" + entry + "\n" + + +def rollup_amend_latest(existing: str, action: str, body: str) -> str | None: + """Replace the latest entry's body if its action is ``action``; else None.""" + start = existing.rfind("
") + if start < 0: + return None + line_end = existing.find("\n", start) + m = _SUMMARY_RE.match(existing[start:line_end]) + if not m: + return None + fields = [f.strip() for f in m.group("summary").split("·")] + if len(fields) < 3 or fields[2] != action: + return None + entry = render_entry(fields[0], fields[1], action, body) + return existing[:start] + entry + "\n" diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/store.py b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/store.py new file mode 100644 index 000000000..7739e4828 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/store.py @@ -0,0 +1,734 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 +"""Issues as files: the contract operations over a descriptor-shaped folder.""" + +from __future__ import annotations + +import contextlib +import datetime as _dt +import errno +import fcntl +import os +import re +from collections.abc import Iterator +from dataclasses import dataclass +from pathlib import Path + +from . import metadata as md +from . import sections +from .descriptor import DESCRIPTOR_NAME, Descriptor +from .vcs import WorkingCopy +from .vcs import detect as detect_vcs + + +class TrackerError(Exception): + """Carries the contract exit code.""" + + def __init__(self, message: str, code: int) -> None: + super().__init__(message) + self.code = code + + +def not_found(message: str) -> TrackerError: + return TrackerError(message, 3) + + +def refused(message: str) -> TrackerError: + return TrackerError(message, 4) + + +def _today() -> str: + return _dt.datetime.now(_dt.UTC).strftime("%Y-%m-%d") + + +def _now() -> str: + return _dt.datetime.now(_dt.UTC).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def slugify(title: str) -> str: + slug = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-") + return slug[:48].rstrip("-") or "report" + + +_SKIP_NAMES = {"readme.md", "index.md", "template.md", "_template.md", "license.md"} + + +@dataclass +class Entry: + """Where one issue lives.""" + + id: str + number: int + path: Path # the issue markdown file + container: Path # folder (folder layout) or the file itself + state_dir: str # "" unless state.mode == "directory" + prefix: str = "" # which ID prefix the name carries + + +class Tracker: + def __init__(self, root: Path, desc: Descriptor, wc: WorkingCopy | None = None) -> None: + self.root = root + self.desc = desc + self.wc = wc or detect_vcs(root) + # Resolved once, here: every later open walks to it component by component + # and refuses a symlink anywhere, so swapping the root (or any folder above + # it) for a symlink afterwards cannot redirect a read or a write. + self._real_root = Path(os.path.realpath(root)) + + # ------------------------------------------------------------ locating + + def _bases(self) -> list[tuple[str, Path]]: + if self.desc.state.mode != "directory": + return [("", self.root)] + out = [] + if self.root.is_dir(): + for child in sorted(self.root.iterdir()): + if child.is_dir() and not child.name.startswith(".") and self._is_state_dir(child.name): + out.append((child.name, child)) + names = {name for name, _ in out} + for name in (self.desc.state.open_dir, self.desc.state.closed_dir): + if name not in names: + out.append((name, self.root / name)) + return out + + def _is_state_dir(self, name: str) -> bool: + st = self.desc.state + known = {v.lower() for v in [*st.open_values, *st.closed_values, st.open_dir, st.closed_dir]} + return name.lower() in known + + def _issue_file(self, folder: Path, ident: str) -> Path: + return folder / self.desc.issue_file.replace("{id}", ident) + + def entries(self) -> list[Entry]: + rx = self.desc.id.regex() + out: list[Entry] = [] + for state_dir, base in self._bases(): + if not base.is_dir(): + continue + if base.is_symlink(): + continue + for child in sorted(base.iterdir()): + if child.name.startswith(".") or child.name.lower() in _SKIP_NAMES or child.is_symlink(): + continue + if self.desc.layout == "folder-per-issue": + if not child.is_dir(): + continue + m = rx.match(child.name) + issue = self._issue_file(child, child.name) + if m and issue.is_file() and not issue.is_symlink(): + out.append( + Entry(child.name, int(m.group("num")), issue, child, state_dir, m.group("prefix")) + ) + else: + if not (child.is_file() and child.suffix == ".md"): + continue + m = rx.match(child.stem) + if m: + out.append( + Entry(child.stem, int(m.group("num")), child, child, state_dir, m.group("prefix")) + ) + return out + + def resolve(self, ident: str) -> Entry: + ident = ident.strip() + entries = self.entries() + for entry in entries: + if entry.id == ident: + return entry + rx = self.desc.id.regex() + + def one(matches: list[Entry]) -> Entry | None: + if len(matches) > 1: + ids = ", ".join(sorted(e.id for e in matches)) + raise TrackerError(f"{ident!r} is ambiguous ({ids}); use the full ID", 2) + return matches[0] if matches else None + + # A bare number names every report with that number, whatever its + # prefix or slug: more than one is ambiguous, never a first match. + if ident.isdigit(): + found = one([e for e in entries if e.number == int(ident)]) + if found: + return found + # A slugged name (`SEC-001-xss`, `SEC-001_xss`) is also addressable by + # its ID without the slug, compared on the parsed prefix and number. + wanted = rx.match(ident) + if wanted and not wanted.group("slug"): + bare = [] + for entry in entries: + m = rx.match(entry.id) + if ( + m + and m.group("prefix") == wanted.group("prefix") + and int(m.group("num")) == int(wanted.group("num")) + ): + bare.append(entry) + found = one(bare) + if found: + return found + raise not_found(f"no tracker {ident!r} under {self.root}") + + # ------------------------------------------------------------ guards + + def require_writable(self) -> None: + if not self.desc.access_restricted: + raise refused( + f"{self.root / DESCRIPTOR_NAME} does not record access_restricted: true. " + "The local tracker holds embargoed reports; a human must confirm the repository " + "is access-restricted (`magpie-local-tracker init|detect --write --access-restricted`) " + "before anything is written to it." + ) + + # ------------------------------------------------------------ reading + + def _contained(self, path: Path) -> Path: + """Refuse a path that leaves the tracker root or passes through a symlink. + + The tracker is shared through version control, so anyone who can push + to it can plant a symlink; a write that followed one could overwrite a + file anywhere the tool can reach. + """ + root = self.root.resolve() + current = path + while True: + if current.is_symlink(): + raise refused(f"{path} goes through a symlink ({current}); refusing to follow it") + if current.resolve() == root or current.parent == current: + break + current = current.parent + try: + path.resolve().relative_to(root) + except ValueError: + raise refused(f"{path} is outside the tracker root {root}") from None + return path + + def _clean(self, text: str) -> str: + """Untrusted report text, with every marker the tracker parses neutralized.""" + managed = [] + if self.desc.comments_mode == "section": + managed.append(self.desc.comments_heading) + if self.desc.rollup_mode == "section": + managed.append(self.desc.rollup_heading) + return sections.neutralize(text, tuple(managed)) + + def _open(self, path: Path, flags: int, *, make_dirs: bool = False) -> int: + """Open a managed file by walking from the tracker root one directory + descriptor at a time, refusing a symlink at *every* component. + + A path-based check followed by `open()` leaves a window in which a + parent directory can be swapped for a symlink; opening each component + relative to the previous descriptor with `O_NOFOLLOW` closes it. + """ + rel = os.path.relpath(os.path.abspath(path), os.path.abspath(self.root)) + parts = Path(rel).parts + if not parts or parts[0] == ".." or os.path.isabs(rel): + raise refused(f"{path} is outside the tracker root {self.root}") + dir_flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW + dir_fd = self._root_fd() + try: + for name in parts[:-1]: + try: + next_fd = os.open(name, dir_flags, dir_fd=dir_fd) + except FileNotFoundError: + if not make_dirs: + raise + # Created concurrently is fine: open it below like any other. + with contextlib.suppress(FileExistsError): + os.mkdir(name, 0o755, dir_fd=dir_fd) + next_fd = os.open(name, dir_flags, dir_fd=dir_fd) + except OSError as exc: + if exc.errno in (errno.ELOOP, errno.ENOTDIR): + raise refused( + f"{path} goes through a symlink or non-folder ({name}); refusing" + ) from exc + raise + os.close(dir_fd) + dir_fd = next_fd + try: + return os.open(parts[-1], flags | os.O_NOFOLLOW, 0o644, dir_fd=dir_fd) + except OSError as exc: + if exc.errno == errno.ELOOP: + raise refused(f"{path} is a symlink; refusing to follow it") from exc + raise + finally: + os.close(dir_fd) + + def _root_fd(self) -> int: + """A descriptor on the tracker root, reached from `/` with `O_NOFOLLOW` + on every component of the path resolved when the tracker was opened.""" + flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW + fd = os.open("/", os.O_RDONLY | os.O_DIRECTORY) + for part in self._real_root.parts[1:]: + try: + next_fd = os.open(part, flags, dir_fd=fd) + except OSError as exc: + os.close(fd) + if exc.errno in (errno.ELOOP, errno.ENOTDIR): + raise refused( + f"the tracker root {self._real_root} changed into a symlink; refusing" + ) from exc + raise + os.close(fd) + fd = next_fd + return fd + + @contextlib.contextmanager + def _allocation_lock(self) -> Iterator[None]: + """Serialize ID allocation across processes: exclusive creation alone + reserves a file name, and with title slugs two writers could otherwise + both take number N under different names.""" + fd = self._root_fd() + try: + fcntl.flock(fd, fcntl.LOCK_EX) + yield + finally: + fcntl.flock(fd, fcntl.LOCK_UN) + os.close(fd) + + def _require_rewritable(self, entry: Entry, meta: md.Metadata) -> None: + if meta.unsupported: + raise refused( + f"{entry.path}: its front matter uses {meta.unsupported}, which this tool cannot " + "rewrite without losing it; edit that report by hand, or simplify its front matter" + ) + + def _read_file(self, path: Path) -> str: + with os.fdopen(self._open(path, os.O_RDONLY), encoding="utf-8") as handle: + return handle.read() + + def _write_file(self, path: Path, text: str) -> None: + fd = self._open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, make_dirs=True) + with os.fdopen(fd, "w", encoding="utf-8") as handle: + handle.write(text) + + def _read(self, entry: Entry) -> tuple[md.Metadata, str]: + meta, body = md.split(self._read_file(entry.path)) + if meta.style == "none" and self.desc.metadata in ("front-matter", "bold-fields"): + # A report written before the tracker had metadata: the first write + # that sets a label, state or milestone gives it the tracker's style + # instead of silently dropping the value. + meta.style = self.desc.metadata + meta.list_style = self.desc.list_style + return meta, body + + def _write(self, entry: Entry, meta: md.Metadata, body: str) -> None: + self._require_rewritable(entry, meta) + key = self.desc.keys.get("updated", "") + if key and (key in meta.values or meta.style != "none"): + meta.values[key] = _today() + self._write_file(entry.path, md.render(meta, body)) + + def _split_managed(self, body: str) -> tuple[str, str]: + """Split off the trailing sections this tool manages inside the issue file.""" + managed = [] + if self.desc.comments_mode == "section": + managed.append(self.desc.comments_heading) + if self.desc.rollup_mode == "section": + managed.append(self.desc.rollup_heading) + cut = len(body) + for heading in managed: + found = sections.find_section(body, heading, 2) + if found: + lines = body.split("\n") + cut = min(cut, len("\n".join(lines[: found[0]]))) + return body[:cut], body[cut:] + + def _title(self, meta: md.Metadata, body: str) -> str: + if self.desc.title_source == "key": + key = meta.find_key([self.desc.keys["title"]]) + value = meta.values.get(key) if key else None + if isinstance(value, str): + return value + m = re.search(r"^# (.+)$", body, re.MULTILINE) + return m.group(1).strip() if m else "" + + def _list(self, meta: md.Metadata, role: str) -> list[str]: + key = meta.find_key([self.desc.keys.get(role, "")]) if self.desc.keys.get(role) else None + value = meta.values.get(key) if key else None + if isinstance(value, list): + return list(value) + if isinstance(value, str) and value: + return [v.strip() for v in value.split(",") if v.strip()] + return [] + + def _scalar(self, meta: md.Metadata, role: str) -> str | None: + key = meta.find_key([self.desc.keys.get(role, "")]) if self.desc.keys.get(role) else None + value = meta.values.get(key) if key else None + if isinstance(value, list): + return ", ".join(value) or None + return value + + def _state(self, entry: Entry, meta: md.Metadata) -> tuple[str, str]: + st = self.desc.state + closed = {v.lower() for v in st.closed_values} + if st.mode == "directory": + raw = entry.state_dir + return ("closed" if raw.lower() in closed or raw == st.closed_dir else "open"), raw + raw = self._scalar(meta, "state") or st.open_value + return ("closed" if raw.lower() in closed else "open"), raw + + def url(self, path: Path) -> str: + rel = self.wc.rel(path) if self.wc.kind != "none" else path.as_posix() + return self.desc.url_template.format(path=rel) if self.desc.url_template else rel + + def _comments_text(self, entry: Entry, body: str) -> str: + if self.desc.comments_mode == "section": + found = sections.find_section(body, self.desc.comments_heading, 2) + if not found: + return "" + _, start, end = found + return "\n".join(body.split("\n")[start:end]) + path = self._comments_path(entry) + return self._read_file(path) if path.is_file() else "" + + def _comments_path(self, entry: Entry) -> Path: + if self.desc.layout == "folder-per-issue": + return entry.container / self.desc.comments_file + return entry.container.with_name(f"{entry.id}.{self.desc.comments_file}") + + def _rollup_path(self, entry: Entry) -> Path: + if self.desc.layout == "folder-per-issue": + return entry.container / self.desc.rollup_file + return entry.container.with_name(f"{entry.id}.{self.desc.rollup_file}") + + def _attachments_dir(self, entry: Entry) -> Path: + if self.desc.layout == "folder-per-issue": + return entry.container / self.desc.attachments_dir + return entry.container.parent / self.desc.attachments_dir / entry.id + + def summary(self, entry: Entry) -> dict[str, object]: + meta, body = self._read(entry) + state, _raw = self._state(entry, meta) + return { + "id": entry.id, + "title": self._title(meta, body), + "state": state, + "labels": self._list(meta, "labels"), + "updated": self._scalar(meta, "updated"), + "url": self.url(entry.path), + } + + def view(self, ident: str, with_comments: bool = False) -> dict[str, object]: + entry = self.resolve(ident) + meta, body = self._read(entry) + main, _ = self._split_managed(body) + state, raw = self._state(entry, meta) + out: dict[str, object] = { + "id": entry.id, + "title": self._title(meta, body), + "state": state, + "status": raw, + "labels": self._list(meta, "labels"), + "assignees": self._list(meta, "assignees"), + "milestone": self._scalar(meta, "milestone"), + "author": self._scalar(meta, "author"), + "created": self._scalar(meta, "created"), + "updated": self._scalar(meta, "updated"), + "body": main.strip("\n") + "\n", + "url": self.url(entry.path), + "path": str(entry.path), + } + if with_comments: + comments: list[dict[str, object]] = [ + {"id": c.id, "author": c.author, "created": c.created, "body": c.body} + for c in sections.parse_comments(self._comments_text(entry, body)) + ] + # On GitHub the rollup is a comment, and skills read markers and + # earlier decisions out of it; return it the same way, first. + rollup, _, _ = self._read_rollup(entry) + if rollup: + comments.insert( + 0, {"id": 0, "author": "", "created": "", "body": rollup.strip(), "kind": "rollup"} + ) + out["comments"] = comments + return out + + def list_issues( + self, state: str = "open", labels: list[str] | None = None, search: str = "", limit: int = 100 + ) -> list[dict[str, object]]: + wanted = {lb.lower() for lb in labels or []} + words = search.lower().split() + out = [] + for entry in sorted(self.entries(), key=lambda e: e.number, reverse=True): + item = self.summary(entry) + if state != "all" and item["state"] != state: + continue + have = {str(lb).lower() for lb in item["labels"]} # type: ignore[attr-defined] + if not wanted <= have: + continue + if words: + text = self._read_file(entry.path).lower() + if not all(word in text for word in words): + continue + out.append(item) + if len(out) >= limit: + break + return out + + def labels(self) -> list[str]: + seen: set[str] = set() + for entry in self.entries(): + meta, _ = self._read(entry) + seen.update(self._list(meta, "labels")) + return sorted(seen) + + def field_get(self, ident: str, heading: str) -> dict[str, object]: + entry = self.resolve(ident) + _, body = self._read(entry) + main, _ = self._split_managed(body) + return {"id": entry.id, "heading": heading, "value": sections.get_field(main, heading)} + + # ------------------------------------------------------------ writing + + def _set(self, meta: md.Metadata, role: str, value: md.Value) -> None: + key = self.desc.keys.get(role, "") + if not key: + return + meta.values[meta.find_key([key]) or key] = value + + def create(self, title: str, body: str, labels: list[str], author: str = "") -> dict[str, object]: + self.require_writable() + title = " ".join(title.split()) + if not title: + raise TrackerError("the title file is empty", 2) + with self._allocation_lock(): + return self._create_locked(title, body, labels, author) + + def _create_locked(self, title: str, body: str, labels: list[str], author: str) -> dict[str, object]: + # Numbering continues the prefix new reports get (``2026-`` next to an + # older ``2025-`` series starts from the 2026 series' own maximum). + number = max((e.number for e in self.entries() if e.prefix == self.desc.id.prefix), default=0) + 1 + st = self.desc.state + base = self.root / st.open_dir if st.mode == "directory" else self.root + meta = md.Metadata(style=self.desc.metadata, list_style=self.desc.list_style) + if self.desc.metadata == "none": + meta.style = "none" + if self.desc.title_source == "key": + self._set(meta, "title", title) + self._set(meta, "state", st.open_value) + self._set(meta, "labels", list(labels)) + self._set(meta, "assignees", []) + self._set(meta, "milestone", None) + if author: + self._set(meta, "author", author) + self._set(meta, "created", _today()) + self._set(meta, "updated", _today()) + text = self._clean(body).strip("\n") + "\n" + if self.desc.title_source == "h1": + text = f"# {title}\n\n{text}" + rendered = md.render(meta, text) + # Reserve the ID with an exclusive create: two writers that picked the + # same number cannot both succeed, and the loser takes the next one. + for _attempt in range(50): + ident = self.desc.id.render(number, slugify(title)) + if self.desc.layout == "folder-per-issue": + path = self._issue_file(base / ident, ident) + else: + path = base / f"{ident}.md" + try: + fd = self._open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, make_dirs=True) + except FileExistsError: + number += 1 + continue + with os.fdopen(fd, "w", encoding="utf-8") as handle: + handle.write(rendered) + return {"id": ident, "url": self.url(path), "path": str(path)} + raise TrackerError("could not reserve a new tracker ID after 50 attempts", 5) + + def edit( + self, + ident: str, + add_labels: list[str], + remove_labels: list[str], + add_assignees: list[str], + remove_assignees: list[str], + milestone: str | None, + clear_milestone: bool, + title: str | None, + body: str | None, + ) -> dict[str, object]: + self.require_writable() + entry = self.resolve(ident) + meta, current = self._read(entry) + for role, add, remove in ( + ("labels", add_labels, remove_labels), + ("assignees", add_assignees, remove_assignees), + ): + if add or remove: + values = [v for v in self._list(meta, role) if v.lower() not in {r.lower() for r in remove}] + values += [a for a in add if a.lower() not in {v.lower() for v in values}] + self._set(meta, role, values) + if clear_milestone: + self._set(meta, "milestone", None) + elif milestone is not None: + self._set(meta, "milestone", milestone) + if title is not None: + title = " ".join(title.split()) + if self.desc.title_source == "key": + self._set(meta, "title", title) + else: + new_h1 = f"# {title}" + current = re.sub(r"^# .+$", lambda _m: new_h1, current, count=1, flags=re.MULTILINE) + if body is not None: + body = self._clean(body) + _, managed = self._split_managed(current) + head = "" + if self.desc.title_source == "h1": + m = re.match(r"^# .+\n", current) + head = m.group(0) + "\n" if m and not body.lstrip().startswith("# ") else "" + current = head + body.strip("\n") + "\n" + ("\n" + managed.lstrip("\n") if managed else "") + self._write(entry, meta, current) + return {"id": entry.id, "url": self.url(entry.path)} + + def field_set(self, ident: str, heading: str, value: str) -> dict[str, object]: + self.require_writable() + entry = self.resolve(ident) + meta, body = self._read(entry) + main, managed = self._split_managed(body) + new_main, created = sections.set_field( + main, heading, self._clean(value), self.desc.field_heading_level + ) + self._write(entry, meta, new_main + ("\n" + managed.lstrip("\n") if managed else "")) + return {"id": entry.id, "heading": heading, "created": created} + + def comment(self, ident: str, body: str, author: str) -> dict[str, object]: + self.require_writable() + entry = self.resolve(ident) + meta, text = self._read(entry) + # Checked before anything is written: a sidecar comment must not land + # while the report's own update is then refused. + self._require_rewritable(entry, meta) + existing = self._comments_text(entry, text) + cid = sections.next_comment_id(existing) + level = 3 if self.desc.comments_mode == "section" else 2 + rendered = sections.render_comment(cid, author or "@unknown", _now(), body, level) + if self.desc.comments_mode == "section": + text = sections.append_to_section(text, self.desc.comments_heading, 2, rendered) + self._write(entry, meta, text) + else: + path = self._comments_path(entry) + prefix = ( + sections.close_fences(existing).rstrip("\n") + "\n\n" + if existing.strip() + else f"# Comments on {entry.id}\n\n" + ) + self._write_file(path, prefix + rendered) + self._write(entry, meta, text) + return {"id": entry.id, "comment_id": cid, "url": self.url(entry.path)} + + def set_state(self, ident: str, closed: bool, reason: str = "completed") -> dict[str, object]: + self.require_writable() + entry = self.resolve(ident) + meta, body = self._read(entry) + st = self.desc.state + value = ( + (st.not_planned_value if reason == "not-planned" else st.closed_value) + if closed + else st.open_value + ) + if st.mode == "field": + self._set(meta, "state", value) + self._write(entry, meta, body) + return {"id": entry.id, "state": "closed" if closed else "open"} + if self.desc.keys.get("state") and meta.find_key([self.desc.keys["state"]]): + self._set(meta, "state", value) + self._write(entry, meta, body) + target_dir = self.root / (st.closed_dir if closed else st.open_dir) + moves = [entry.container] + if self.desc.layout == "file-per-issue": + for extra in (self._comments_path(entry), self._rollup_path(entry), self._attachments_dir(entry)): + if extra.exists(): + moves.append(extra) + for src in moves: + rel = src.relative_to(entry.container.parent) + dst = target_dir / rel + self._contained(src) + self._contained(dst) + if src.resolve() != dst.resolve(): + self.wc.move(src, dst) + return {"id": entry.id, "state": "closed" if closed else "open"} + + def _rollup_name(self) -> str: + # Fixed, so a checkout folder named "Security Reports" cannot produce a + # marker the next append fails to recognise. + return "security-tracker" + + def _read_rollup(self, entry: Entry) -> tuple[str | None, md.Metadata, str]: + meta, body = self._read(entry) + if self.desc.rollup_mode == "section": + found = sections.find_section(body, self.desc.rollup_heading, 2) + if not found: + return None, meta, body + _, start, end = found + return "\n".join(body.split("\n")[start:end]).strip("\n"), meta, body + path = self._rollup_path(entry) + return (self._read_file(path) if path.is_file() else None), meta, body + + def _write_rollup(self, entry: Entry, meta: md.Metadata, body: str, rollup: str) -> None: + if self.desc.rollup_mode == "section": + found = sections.find_section(body, self.desc.rollup_heading, 2) + lines = body.split("\n") + if found: + _, start, end = found + body = "\n".join([*lines[:start], "", rollup.strip("\n"), "", *lines[end:]]) + else: + body = sections.append_to_section(body, self.desc.rollup_heading, 2, rollup) + self._write(entry, meta, body) + else: + self._write_file(self._rollup_path(entry), rollup) + self._write(entry, meta, body) + + def rollup_append(self, ident: str, action: str, body: str, user: str, now: str) -> dict[str, object]: + self.require_writable() + entry = self.resolve(ident) + existing, meta, text = self._read_rollup(entry) + self._require_rewritable(entry, meta) + entry_text = sections.render_entry(now[:10], user, action, body) + self._write_rollup( + entry, meta, text, sections.rollup_append(existing, self._rollup_name(), entry_text) + ) + return {"id": entry.id, "url": self.url(self._rollup_target(entry))} + + def rollup_amend_latest(self, ident: str, action: str, body: str) -> dict[str, object]: + self.require_writable() + entry = self.resolve(ident) + existing, meta, text = self._read_rollup(entry) + self._require_rewritable(entry, meta) + amended = sections.rollup_amend_latest(existing or "", action, body) + if amended is None: + raise refused( + f"the latest rollup entry on {entry.id} is not a {action!r} entry; refusing to amend" + ) + self._write_rollup(entry, meta, text, amended) + return {"id": entry.id, "url": self.url(self._rollup_target(entry))} + + def _rollup_target(self, entry: Entry) -> Path: + return entry.path if self.desc.rollup_mode == "section" else self._rollup_path(entry) + + def body(self, ident: str) -> str: + """The report body without metadata or managed sections: the `### Field` + markdown generate-cve-json reads on stdin.""" + entry = self.resolve(ident) + _, text = self._read(entry) + main, _ = self._split_managed(text) + level = self.desc.field_heading_level + if level != 3: + # generate-cve-json reads `### Field` sections only: shift the + # tracker's field level (and anything nested under it) to match. + lines = main.split("\n") + for idx, lvl, title in sections.heading_lines(lines): + if lvl >= level: + lines[idx] = "#" * min(6, 3 + lvl - level) + " " + title + main = "\n".join(lines) + return main.strip("\n") + "\n" + + def attach(self, ident: str, content: bytes, name: str) -> dict[str, object]: + self.require_writable() + entry = self.resolve(ident) + safe = Path(name).name + if safe in ("", ".", ".."): + raise TrackerError("invalid attachment name", 2) + target = self._attachments_dir(entry) / safe + fd = self._open(target, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, make_dirs=True) + with os.fdopen(fd, "wb") as handle: + handle.write(content) + return {"id": entry.id, "name": safe, "url": self.url(target)} diff --git a/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/vcs.py b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/vcs.py new file mode 100644 index 000000000..f058e40b2 --- /dev/null +++ b/plugins/magpie-vetted-ops/tools/local-tracker/src/magpie_local_tracker/vcs.py @@ -0,0 +1,390 @@ +# SPDX-License-Identifier: Apache-2.0 +# https://www.apache.org/licenses/LICENSE-2.0 +"""The working copy under the tracker: Git or SVN. + +Only three things are needed from it: what changed under the tracker root, +a history-preserving move (closing a report in a folder-per-state tracker), +and the commit-and-push that shares a session's work with the team. All of +it is scoped to the tracker root — nothing outside it is ever staged. +""" + +from __future__ import annotations + +import os +import re +import shutil +import subprocess +import tempfile +import urllib.parse +from dataclasses import dataclass, field +from pathlib import Path + + +class VcsError(RuntimeError): + pass + + +class VcsRefused(VcsError): + """A publish that would send something other than tracker changes.""" + + +# Every git call runs with the helpers git can execute switched off. These +# operations may run outside the sandbox (vetted-op-read / vetted-op-tracker), +# and the repository is chosen by a policy file; its own configuration must +# not be able to start a program there. +_GIT_SAFE = ( + "-c", "core.fsmonitor=false", + "-c", "core.hooksPath=/dev/null", + "-c", "core.sshCommand=ssh", + # No automatic gc or maintenance: gc.recentObjectsHook and friends are commands. + "-c", "gc.auto=0", + "-c", "maintenance.auto=false", + # Only network transports: a local-path or file:// remote would run + # receive-pack — and that repository's own hooks — on this machine. + "-c", "protocol.allow=never", + "-c", "protocol.https.allow=always", + "-c", "protocol.ssh.allow=always", +) # fmt: skip + +#: Push URL schemes publish accepts. Tests widen this; production cannot. +_PUSH_SCHEMES = ("https", "ssh") +_SCP_LIKE = re.compile(r"^(?:[A-Za-z0-9._-]+@)?[A-Za-z0-9.-]+\.[A-Za-z]{2,}:(?!/)") + + +def push_url_allowed(url: str) -> bool: + """An https:// or ssh:// URL with a host, or scp-style ``user@host.tld:path``.""" + url = url.strip() + if "://" in url: + parts = urllib.parse.urlsplit(url) + return parts.scheme.lower() in _PUSH_SCHEMES and bool(parts.hostname) + return "ssh" in _PUSH_SCHEMES and bool(_SCP_LIKE.match(url)) + + +# Repository-local configuration keys known not to run anything. Any other +# key set in the repository's own config (filter drivers, merge or diff +# drivers, aliases, include files, url rewrites, gpg.program, …) refuses the +# operation: a helper there could run a program outside the sandbox. +_SAFE_LOCAL_KEY = re.compile( + r"^(?:" + r"core\.(?:repositoryformatversion|filemode|bare|logallrefupdates|ignorecase|precomposeunicode" + r"|autocrlf|safecrlf|eol|symlinks|sparsecheckout|untrackedcache|splitindex|commitgraph|longpaths" + r"|quotepath|checkstat|trustctime)" + r"|extensions\.(?:objectformat|worktreeconfig|refstorage)" + r"|remote\.[^\s]+\.(?:url|pushurl|fetch|push|prune|tagopt|mirror|skipdefaultupdate|promisor)" + r"|branch\.[^\s]+\.(?:remote|merge|pushremote|rebase|description)" + r"|user\.(?:name|email|signingkey|useconfigonly)" + r"|push\.(?:default|autosetupremote|followtags)|pull\.(?:rebase|ff)|fetch\.prune" + r"|init\.defaultbranch|commit\.gpgsign|tag\.gpgsign|gpg\.format" + r"|index\.(?:version|threads|skiphash)|feature\.(?:manyfiles|experimental)" + r")$", + re.IGNORECASE, +) + + +#: The empty tree, per object format: attributes are read from it (see `_run`). +_EMPTY_TREE = { + "sha1": "4b825dc642cb6eb9a060e54bf8d69288fbee4904", + "sha256": "6ef19b41225c5369f1c104d45d8d85efa9b057b53b14b4b9b939dd74decc5321", +} + + +@dataclass +class WorkingCopy: + kind: str # "git", "svn", or "none" + top: Path + _empty_tree: str = field(default=_EMPTY_TREE["sha1"], repr=False) + + def _check_git_config(self) -> None: + """Refuse a repository whose own config sets anything outside the + known-safe keys (see `_SAFE_LOCAL_KEY`). Reading config runs nothing.""" + proc = subprocess.run( + ["git", "config", "--list", "--show-scope", "--null"], + cwd=self.top, + capture_output=True, + text=True, + check=False, + ) + if proc.returncode != 0: + raise VcsRefused(f"cannot read the repository's git config: {proc.stderr.strip()}") + risky = [] + sha256 = False + # With --show-scope --null git emits "scope\0key\nvalue\0" per entry. + fields = proc.stdout.split("\0") + for scope, entry in zip(fields[0::2], fields[1::2], strict=False): + key, _, value = entry.partition("\n") + if scope in ("local", "worktree") and not _SAFE_LOCAL_KEY.match(key): + risky.append(key) + if key.lower() == "extensions.objectformat" and value.strip().lower() == "sha256": + sha256 = True + self._empty_tree = _EMPTY_TREE["sha256" if sha256 else "sha1"] + if risky: + raise VcsRefused( + "the repository's own git config sets " + + ", ".join(sorted(set(risky))[:5]) + + ", which could run a program when git runs here outside the sandbox; " + "remove it from .git/config, or run the git step by hand" + ) + + def _run(self, *args: str, check: bool = True) -> str: + env = None + if args and args[0] == "git": + # Checked before *every* git command, not once: the window between + # the check and the command is as small as it can be made. + self._check_git_config() + args = ("git", *_GIT_SAFE, *args[1:]) + # And within that window, attributes come from the empty tree, not + # the working tree the agent can write: no `.gitattributes` there can + # switch on a filter, diff or merge driver whatever the config says + # by then. Git older than 2.42 ignores the variable. + env = {**os.environ, "GIT_ATTR_SOURCE": self._empty_tree} + try: + proc = subprocess.run(args, cwd=self.top, capture_output=True, text=True, check=False, env=env) + except FileNotFoundError as exc: + raise VcsError(f"{args[0]} not found on PATH") from exc + if check and proc.returncode != 0: + raise VcsError(f"{' '.join(args)} failed: {proc.stderr.strip() or proc.stdout.strip()}") + return proc.stdout + + def rel(self, path: Path) -> str: + return path.resolve().relative_to(self.top.resolve()).as_posix() + + # ------------------------------------------------------------ status + + def status(self, root: Path) -> list[dict[str, str]]: + if self.kind == "git": + out = self._run("git", "status", "--porcelain=v1", "--untracked-files=all", "--", self.rel(root)) + return [{"status": ln[:2].strip() or "?", "path": ln[3:]} for ln in out.splitlines() if ln] + if self.kind == "svn": + out = self._run("svn", "status", self.rel(root)) + return [ + {"status": ln[0], "path": ln[8:].strip()} for ln in out.splitlines() if ln and ln[0] != " " + ] + return [] + + def push_destination(self) -> str | None: + """The remote-tracking ref `git push` sends this branch to (`@{push}`), or None. + + Outgoing commits are measured against that ref, not against "every + remote": a commit already on `origin/feature` but not on `origin/main` + would still be sent when the branch pushes to `main`. + """ + if self.kind != "git": + return None + try: + ref = self._run("git", "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{push}").strip() + if not ref: + return None + self._run("git", "rev-parse", "--verify", "--quiet", ref) + except VcsError: + return None + return ref + + def outgoing(self) -> list[str] | None: + """Commits a push would send; None when the push destination is unknown.""" + if self.kind != "git": + return [] + dest = self.push_destination() + if dest is None: + return None + out = self._run("git", "rev-list", "HEAD", "--not", dest) + return [ln for ln in out.split() if ln] + + def push_target(self) -> tuple[str, str] | None: + """``(remote, remote ref)`` this branch pushes to, or None when unknown. + + The remote is git's own push resolution (``%(push:remotename)``), so a + remote whose name contains ``/`` is not split in the wrong place. The + ref is ``%(push:remoteref)`` when git fills it in; otherwise it is what + ``push.default`` would push to: the upstream branch for ``simple`` / + ``upstream`` (git's default), the same name for ``current`` / + ``matching``. + """ + if self.kind != "git": + return None + branch = self._run("git", "symbolic-ref", "-q", "HEAD", check=False).strip() + if not branch.startswith("refs/heads/"): + return None + out = self._run( + "git", "for-each-ref", "--format=%(push:remotename)%00%(push:remoteref)", branch, check=False + ).strip() + remote, _, ref = out.partition("\0") + if remote and not ref: + short = branch.removeprefix("refs/heads/") + mode = self._run("git", "config", "--get", "push.default", check=False).strip().lower() + if mode in ("", "simple", "upstream", "tracking"): + ref = self._run("git", "config", "--get", f"branch.{short}.merge", check=False).strip() + elif mode in ("current", "matching"): + ref = branch + return (remote, ref) if remote and ref.startswith("refs/heads/") else None + + def _push(self, target: tuple[str, str]) -> None: + """Push HEAD to exactly the checked ref: an explicit refspec, so neither + ``push.default=matching`` nor a remote's ``push`` refspecs nor + ``push.followTags`` can add other branches or tags to the push.""" + remote, ref = target + self._run("git", "push", "--no-verify", "--no-follow-tags", remote, f"HEAD:{ref}") + + def _check_outgoing_scoped(self, rel_root: str) -> tuple[list[str], tuple[str, str]]: + """Refuse a push that would also publish anything outside the tracker + folder — on a shared branch that is someone else's work — or whose + destination cannot be established. Returns the outgoing commits and the + ``(remote, ref)`` they were checked against, which is all a push sends.""" + commits = self.outgoing() + target = self.push_target() + if commits is None or target is None: + raise VcsRefused( + "cannot tell where `git push` would send this branch (no upstream or push destination); " + "set one (for example `git push -u origin ` once, by hand) before publishing" + ) + dest = self.push_destination() or "" + remote = target[0] + url = self._run("git", "remote", "get-url", "--push", remote).strip() + if not push_url_allowed(url): + raise VcsRefused( + f"refusing to push to {url!r}: publish pushes only to an https:// or ssh:// remote " + "(a local path or file:// remote would run that repository's hooks on this machine)" + ) + prefix = "" if rel_root in ("", ".") else rel_root.rstrip("/") + "/" + for sha in commits: + # A plain commit is compared with its parent. A merge is compared + # with --cc: only paths that differ from *every* parent, i.e. changes + # made in the merge itself. What a merge brings in from a parent is + # judged where it was made — that parent's commits are either + # already at the destination or outgoing and checked here too. + # -z: paths unquoted. + parents = self._run("git", "rev-list", "--parents", "-n", "1", sha).split()[1:] + mode = ["--cc"] if len(parents) > 1 else ["--root"] + out = self._run("git", "diff-tree", *mode, "-r", "--no-commit-id", "--name-only", "-z", sha) + paths = [p for p in out.split("\0") if p] + stray = [p for p in paths if prefix and not p.startswith(prefix)] + if stray: + raise VcsRefused( + f"the branch has unpushed commit {sha[:12]} touching files outside {rel_root}/ " + f"({', '.join(stray[:3])}); pushing would publish them too. Push or move that work " + "first, or keep the tracker on its own branch." + ) + # Per-commit checks cannot see a merge that keeps an *old* version of a + # file the destination has since changed (-s ours): that is a revert the + # push would publish. Compare the trees too. + tree = self._run("git", "diff", "--name-only", "-z", dest, "HEAD") + stray = [p for p in tree.split("\0") if p and prefix and not p.startswith(prefix)] + if stray: + raise VcsRefused( + f"pushing would change files outside {rel_root}/ at the destination " + f"({', '.join(stray[:3])}); refusing" + ) + return commits, target + + def unpushed(self) -> int | None: + """Commits not yet at the push destination; None when that is unknown. + + A publish whose commit succeeded but whose push failed leaves the tracker + clean yet unpublished; this is how `pending` and `publish` still see it. + """ + if self.kind != "git": + return 0 + commits = self.outgoing() + return None if commits is None else len(commits) + + def remotes(self) -> list[str]: + if self.kind == "git": + out = self._run("git", "remote", "-v", check=False) + return sorted({ln.split()[1] for ln in out.splitlines() if len(ln.split()) >= 2}) + if self.kind == "svn": + out = self._run("svn", "info", "--show-item", "url", check=False) + return [out.strip()] if out.strip() else [] + return [] + + # ------------------------------------------------------------ move + + def _tracked(self, path: Path) -> bool: + if self.kind == "git": + return bool(self._run("git", "ls-files", "--", self.rel(path), check=False).strip()) + if self.kind == "svn": + out = self._run("svn", "status", "--depth", "empty", self.rel(path), check=False) + return not out.startswith("?") + return False + + def move(self, src: Path, dst: Path) -> None: + dst.parent.mkdir(parents=True, exist_ok=True) + if self.kind == "git" and self._tracked(src): + self._run("git", "mv", self.rel(src), self.rel(dst)) + elif self.kind == "svn" and self._tracked(src): + self._run( + "svn", "add", "--parents", "--force", "--depth", "empty", self.rel(dst.parent), check=False + ) + self._run("svn", "mv", self.rel(src), self.rel(dst)) + else: + shutil.move(str(src), str(dst)) + + # ------------------------------------------------------------ publish + + def publish(self, root: Path, message: str, trailers: list[str], push: bool) -> dict[str, object]: + rel_root = self.rel(root) + if self.kind == "git": + if push: + self._check_outgoing_scoped(rel_root) + self._run("git", "add", "-A", "--", rel_root) + if not self._run("git", "diff", "--cached", "--name-only", "--", rel_root).strip(): + # An earlier publish may have committed without pushing; with + # --push, push it now (an unknown destination was refused above). + commits, target = self._check_outgoing_scoped(rel_root) if push else ([], ("", "")) + if push and commits: + self._push(target) + revision = self._run("git", "rev-parse", "HEAD").strip() + return { + "committed": False, + "pushed": True, + "revision": revision, + "note": "pushed earlier commits", + } + return {"committed": False, "pushed": False, "revision": None, "note": "nothing to commit"} + with tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False, encoding="utf-8") as fh: + fh.write(message) + msg_path = fh.name + args = ["git", "commit", "--no-verify", "-F", msg_path] + for trailer in trailers: + args += ["--trailer", trailer] + args += ["--", rel_root] + try: + self._run(*args) + finally: + Path(msg_path).unlink(missing_ok=True) + revision = self._run("git", "rev-parse", "HEAD").strip() + pushed = False + if push: + _, target = self._check_outgoing_scoped(rel_root) + self._push(target) + pushed = True + return {"committed": True, "pushed": pushed, "revision": revision} + if self.kind == "svn": + for entry in self.status(root): + if entry["status"] == "?": + self._run("svn", "add", "--parents", "--force", entry["path"]) + elif entry["status"] == "!": + self._run("svn", "rm", "--force", entry["path"]) + if not self.status(root): + return {"committed": False, "pushed": False, "revision": None, "note": "nothing to commit"} + full = message.rstrip("\n") + ("\n\n" + "\n".join(trailers) if trailers else "") + "\n" + with tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False, encoding="utf-8") as fh: + fh.write(full) + msg_path = fh.name + try: + out = self._run("svn", "commit", "-F", msg_path, rel_root) + finally: + Path(msg_path).unlink(missing_ok=True) + m = re.search(r"Committed revision (\d+)", out) + # An SVN commit is the publish: there is no separate push. + return {"committed": True, "pushed": True, "revision": m.group(1) if m else None} + raise VcsError("the tracker root is not inside a Git or SVN working copy") + + +def detect(root: Path) -> WorkingCopy: + """Find the working copy that contains ``root`` (nearest ``.git`` / ``.svn``).""" + start = root.resolve() + for candidate in [start, *start.parents]: + if (candidate / ".git").exists(): + return WorkingCopy("git", candidate) + if (candidate / ".svn").is_dir(): + return WorkingCopy("svn", candidate) + return WorkingCopy("none", start) diff --git a/plugins/magpie-vetted-ops/tools/vetted-ops/README.md b/plugins/magpie-vetted-ops/tools/vetted-ops/README.md index d72ac582a..a90504a01 100644 --- a/plugins/magpie-vetted-ops/tools/vetted-ops/README.md +++ b/plugins/magpie-vetted-ops/tools/vetted-ops/README.md @@ -506,24 +506,29 @@ of its `gh` builder, and the skills keep calling the same operations: ```toml [tracker] platform = "local-files" # github (default) | local-files | jira -project = "/abs/path/to/.apache-magpie/tools/local-tracker" root = "/abs/path/to/the/repo/security/reports" # the tracker folder # platform = "jira" -# project = "/abs/path/to/.apache-magpie/tools/jira" # jira_url = "https://issues.example.org/jira" # jira_project = "FOOSEC" # jira_security_level = "Security Team" +# jira_issue_type = "Bug" # default Bug # jira_deployment = "server" # server | cloud ``` The guarantees above carry over, because the translation lives in reviewed code ([`tracker_backends.py`](src/vetted_ops/tracker_backends.py)), not in the policy: -- **The program is fixed.** The policy supplies data — the backend's tool - directory, which must be the `tools/local-tracker` or - `tools/jira` directory the platform names, and its pinned - target — never a command. The argv is `uv run --project