From 3bedc6dc47e5200059993c0f515befbbd8d2b40e Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Wed, 21 Oct 2026 10:44:11 +0100 Subject: [PATCH 01/14] Agent-skills: Add a new all around skills for fuzzing improvement Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/README.md | 16 +- infra/experimental/agent-skills/USAGE.md | 210 ++++++++++++++++++ .../agent-skills/copy_to_global.sh | 93 ++++---- infra/experimental/agent-skills/helper.py | 3 + .../oss-fuzz-coverage-pipeline/SKILL.md | 151 +++++++++++++ .../references/worktree_delivery.md | 138 ++++++++++++ .../scripts/select_projects.sh | 172 ++++++++++++++ 7 files changed, 742 insertions(+), 41 deletions(-) create mode 100644 infra/experimental/agent-skills/USAGE.md create mode 100644 infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md create mode 100644 infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md create mode 100755 infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh diff --git a/infra/experimental/agent-skills/README.md b/infra/experimental/agent-skills/README.md index 0d59cb10cb0f..6b959a120d78 100644 --- a/infra/experimental/agent-skills/README.md +++ b/infra/experimental/agent-skills/README.md @@ -3,7 +3,7 @@ Skills and tooling that let an agent CLI (Gemini CLI or Claude Code) write, build, and extend OSS-Fuzz fuzzing integrations. The folder ships: -- Six skills that give an agent OSS-Fuzz-specific knowledge. +- Seven skills that give an agent OSS-Fuzz-specific knowledge. - `infra/experimental/agent-skills/helper.py`, a wrapper that launches agent sessions over one or more OSS-Fuzz projects in parallel. - `copy_to_global.sh`, an installer that places the skills where your @@ -12,6 +12,9 @@ build, and extend OSS-Fuzz fuzzing integrations. The folder ships: Before running anything in this folder, review the [threat model](#threat-model). This tooling is experimental and runs agents in unrestricted modes. +For the full walkthrough — install the skills, run the coverage pipeline, +verify results — see [USAGE.md](USAGE.md). + ## Contents | Item | Purpose | @@ -22,8 +25,9 @@ This tooling is experimental and runs agents in unrestricted modes. | `fuzzing-jvm-expert/` | Skill: fuzz JVM projects (Java/Kotlin/Scala) with Jazzer | | `fuzzing-python-expert/` | Skill: fuzz Python projects with Atheris | | `oss-fuzz-engineer/` | Skill: OSS-Fuzz infra workflows (integrate, fix, extend) | +| `oss-fuzz-coverage-pipeline/` | Skill: language-agnostic end-to-end coverage pipeline wrapping the language fuzzing skills — selects projects by glob/wildcard or all, auto-detects language from project.yaml, fixes builds, triages coverage, improves targets, stages paired worktree branches; ships `scripts/select_projects.sh` | | `infra/experimental/agent-skills/helper.py` | Driver that launches agent sessions per OSS-Fuzz project | -| `copy_to_global.sh` | Installs the skills into `~/.gemini/skills` or `~/.claude/skills` | +| `copy_to_global.sh` | Installs the skills into `~/.gemini/skills`, `~/.claude/skills`, or any skills directory you pass as an argument (e.g. Cline's `.cline/skills`) | See each skill's `SKILL.md` for the detailed guidance the agent receives. @@ -42,7 +46,13 @@ See each skill's `SKILL.md` for the detailed guidance the agent receives. ```bash # 1. Install the skills into your agent CLI. -./copy_to_global.sh gemini # or: ./copy_to_global.sh claude +./copy_to_global.sh # defaults to claude; or: ./copy_to_global.sh gemini +# Any other argument is used as the skills directory itself (any SKILL.md- +# compatible agent, e.g. Cline's workspace directory; relative paths resolve +# against your current directory): +./copy_to_global.sh /path/to/vscode-workspace/.cline/skills +# Multiple targets in one invocation are fine: +# ./copy_to_global.sh gemini claude ../.cline/skills # 2. Confirm the agent CLI is reachable. gemini --version # or: claude --version diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md new file mode 100644 index 000000000000..f74867f21505 --- /dev/null +++ b/infra/experimental/agent-skills/USAGE.md @@ -0,0 +1,210 @@ +# OSS-Fuzz agent skills: installation, usage and verification guide + +An end-to-end walkthrough: install the skills into your agent, run the coverage pipeline on OSS-Fuzz projects, and verify the results at every stage, from a single local run all the way up to production coverage numbers. + +Companion to [README.md](README.md), which covers the folder overview and the threat model. + +## 1. What you have + +| Piece | Role | +|---|---| +| Six knowledge skills | `oss-fuzz-engineer` (infrastructure) plus one fuzzing skill per language: `fuzzing-memory-unsafe-expert`, `fuzzing-go-expert`, `fuzzing-rust-expert`, `fuzzing-jvm-expert`, `fuzzing-python-expert` | +| `oss-fuzz-coverage-pipeline` | The orchestrator skill: project selection with upstream viability checks, build fixes, live coverage triage, harness/corpus/dictionary improvements, and delivery as worktree branch duo pairs validated through a temporary copy bridge. It auto-detects each project's language from `project.yaml` and activates the matching language skill. | +| `oss-fuzz-coverage-pipeline/scripts/select_projects.sh` | Deterministic project selector: globs, `--language`, `--broken`, `--limit`, `--exclude`. | +| `copy_to_global.sh` | Installer: `claude` (the default), `gemini`, or any skills directory you name. | +| `helper.py` | An optional headless batch driver for the `claude` and `gemini` CLIs only; it is not needed for interactive use. | + +The skills are plain `SKILL.md` markdown, so they are model-agnostic and portable to any skill-compatible host (Claude Code, Gemini CLI, or Cline in VS Code) with any model, including GLM via Z AI. + +### 1.1 The seventh skill is a wrapper around the other six + +`oss-fuzz-coverage-pipeline` is not a replacement for the six knowledge skills; it is an orchestrator that composes them. For every project it activates two companions: `oss-fuzz-engineer` (always, for infrastructure knowledge) plus exactly one language skill, picked automatically from the project's `project.yaml` `language` field: + +| `project.yaml` language | Sub-skill the wrapper activates | +|---|---| +| `c`, `c++`, or missing | `fuzzing-memory-unsafe-expert` | +| `go` | `fuzzing-go-expert` | +| `rust` | `fuzzing-rust-expert` | +| `jvm` | `fuzzing-jvm-expert` | +| `python` | `fuzzing-python-expert` | +| other (javascript, swift, ruby, ...) | none dedicated: `oss-fuzz-engineer` only, or scope the run with `--language` | + +The wrapper delegates all domain knowledge to those skills and their references: how to write harnesses in each language, how to drive `infra/helper.py`, how to read coverage reports and how to generate structured seeds. What the wrapper itself contributes is the process layer that none of the six contain: project selection and upstream viability checks, the phase discipline, the git guardrails, worktree duo pairs with the temporary copy bridge, the validation gates with the fix budget, the run manifest, rejection handling and the final summary. + +The six knowledge skills remain fully usable on their own; the wrapper simply composes them into one end-to-end delivery pipeline, so nothing in them was duplicated or superseded. + +## 2. Prerequisites + +- A skill-compatible agent host: **Cline** (VS Code), **Claude Code** or **Gemini CLI**. Any model works; nothing here is model-specific. +- **Docker**, because the pipeline builds projects through OSS-Fuzz's `infra/helper.py`. +- **Python 3** and **curl** (the selector's `--broken` mode fetches the public build status). +- This oss-fuzz checkout on disk. +- **Security**: this tooling builds and runs untrusted open-source code. Run it in a sandbox or VM on a trusted network (see the threat model in README.md). + +## 3. Installation + +Run from `infra/experimental/agent-skills/` inside your oss-fuzz checkout: + +```sh +cd /infra/experimental/agent-skills + +./copy_to_global.sh # Claude Code, installs to ~/.claude/skills (the default) +./copy_to_global.sh gemini # Gemini CLI, installs to ~/.gemini/skills +./copy_to_global.sh /path/to/vscode-workspace/.cline/skills # Cline (any model) +./copy_to_global.sh /any/skills/dir # any other SKILL.md-compatible agent +./copy_to_global.sh claude gemini DIR # several targets at once +``` + +Notes: + +- All seven skills are installed **as a set**, because they cross-reference each other. +- Existing skills of the same name in the target directory are **overwritten**. +- The installed copy is a **copy**, so after editing any skill here, re-run the installer to refresh it. + +## 4. Verify the installation + +Check per host, after starting a **fresh session** (skills are scanned at startup): + +- **Claude Code**: ask *"what skills do you have?"* and expect seven, including `oss-fuzz-coverage-pipeline`; typing `/` should offer `/oss-fuzz-coverage-pipeline`. +- **Gemini CLI**: `/skills list` (or `/skills reload` after reinstalling). +- **Cline**: bottom bar, scale icon, **Skills tab**: seven skills listed; typing `/` in chat offers the skill commands. + +Then smoke-test the selector (it works from any directory): + +```sh +/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh 'json*' +``` + +Expected output: project names with auto-detected languages, for example: + +``` +json c++ +json-c c++ +jsonparser go +jsonpickle python +json5format rust +... +Selected 18 project(s) matching: json* +``` + +## 5. Usage + +### 5.1 Select the project set + +```sh +SEL=/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts + +$SEL/select_projects.sh '*' # all 1,382 projects (any language) +$SEL/select_projects.sh --language c,c++ '*' # the ~600 C/C++ projects +$SEL/select_projects.sh 'lib*' 'json*' # glob subset +$SEL/select_projects.sh --broken --limit 20 '*' # currently broken builds first +$SEL/select_projects.sh --exclude 'curl,wget' --language c,c++ '*' +``` + +The recommended order at scale is **broken builds first** (quick wins that also unlock coverage data), followed by the lowest own-code coverage on security-sensitive surface. + +### 5.2 Kick off the pipeline (interactive, any host or model) + +Single project: either phrase it naturally and let the description auto-trigger the skill: + +> Use the oss-fuzz-coverage-pipeline skill on json-c: verify upstream viability from the Dockerfile, fix its builds, triage its live coverage on its own code, improve fuzz targets, corpora and dictionaries where security-sensitive, and deliver a worktree branch duo pair. Never commit, never push and do not stage by default. + +or force the skill explicitly with `/oss-fuzz-coverage-pipeline`. + +Multiple projects: + +> Use the oss-fuzz-coverage-pipeline skill. Phase 0: run scripts/select_projects.sh --broken --limit 5 'lib*', verify upstream viability for each, record the manifest, then run the full pipeline on each viable project, one at a time, resuming from the manifest between sessions. + +### 5.3 What the agent will do (Phases 0 to 5) + +0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. Record a run manifest at `build/coverage-pipeline/manifest.md`. +1. Fix builds through `infra/helper.py` only: the main fuzzer build fix is always mandatory; coverage and introspector build fixes are optional only when the failure cause is missing corpora or an OOM/timeout, with the exemption recorded. +2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. +3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. +4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. +5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, no crash within the first 60 seconds, no stale coverage in the final half); and a like-for-like coverage delta. After a fix budget of 3 rounds, pairs still failing are moved to the rejected area, never deleted. + +Crashes found along the way are treated as **findings** (documented with reproducers), not as failures to hide. + +### 5.4 The deliverables + +- The list of surviving worktree branch duo pairs, with paths and base commit SHAs recorded. +- `summary.md` in the marked working area: one markdown table row per duo pair (project, language, duo branches, worktree paths, summary of work, coverage delta, status). +- Skipped or unprocessable projects in the same table with the concrete reason. +- Rejected duo pairs moved into the rejected area and marked clearly, with the failure reason and what was attempted. +- Per-project reports, the manifest, and the next actions for you. + +### 5.5 Optional: headless batch (Claude Code and Gemini CLI only) + +```sh +python3 infra/experimental/agent-skills/helper.py show-prompt fix-builds htslib # dry run +python3 infra/experimental/agent-skills/helper.py fix-builds proj1 proj2 +python3 infra/experimental/agent-skills/helper.py fix-broken-builds --language c,c++ --limit 20 +python3 infra/experimental/agent-skills/helper.py run-task --task "..." proj1 +``` + +## 6. Verify the results + +### 6.1 Immediately: what to inspect after each project + +| Artifact | Where | What "good" looks like | +|---|---|---| +| Run manifest | `build/coverage-pipeline/manifest.md` (oss-fuzz root) | A row per project with language, status, duo branches, worktree paths, base SHAs and report link | +| Final summary | `/summary.md` | One table covering every selected project: delivered, rejected (moved to rejected area) and skipped, each with its reason where relevant | +| Duo pairs | `/worktrees/oss-fuzz/` and `worktrees/upstream/` | Both branches exist (`oss-fuzz--` and `-`), each holding plain working-tree changes | +| Git safety | both worktrees | Nothing staged (unless justified and recorded), nothing committed, nothing pushed, anywhere | +| Bridge | OSS-Fuzz worktree Dockerfile | Tagged `OSS-FUZZ-AGENT-BRIDGE:BEGIN/END` blocks plus the marked staging folder, kept intact for your manual removal later | +| Validation evidence | per-project report | The declared sanitizer/engine matrix built; per-fuzzer `check_build` passes; quality-gate run logs (120 s minimum, no fast crash, no staleness); a like-for-like coverage-delta table | + +Handy checks: + +```sh +# Working-tree changes present, nothing staged, in both worktrees: +git -C status --short +git -C status --short + +# Nothing accidentally committed on the branches: +git -C log --oneline main..HEAD # expect nothing + +# Rejected pairs moved, not deleted: +ls /rejected/ +``` + +### 6.2 Definition of done (per project, taken from the skill) + +- [ ] All mandatory builds green from the OSS-Fuzz worktree; per-fuzzer `check_build` passes +- [ ] Quality gate clean for every new or changed target: 120 s minimum, no fast crash, no staleness +- [ ] Coverage: documented gains on the targeted files and zero digression elsewhere +- [ ] Duo pair holds plain working-tree changes; nothing staged, committed or pushed +- [ ] Bridge tagged and kept; rejection (if any) moved to the rejected area; manifest row final + +### 6.3 Your review and merge loop (the human part) + +1. Read the summary table and the per-project reports; spot-check the diffs in both worktrees of each delivered pair. +2. Commit the **upstream branch first**, then the OSS-Fuzz branch (the bridge assumes the upstream content), and push both. +3. After the real upstream merge, remove the bridge blocks and the bridge staging folders manually (this is why they are tagged). +4. File crash **findings** as upstream issues; the reproducers are in the reports. +5. Move or remove the delivered worktrees as you see fit, revisit or discard the rejected pairs, and remove the pristine clones once the base SHAs are no longer needed. +6. Mark the manifest rows `delivered` (or leave the recorded rejected/skipped states). + +### 6.4 After merging: verify in production + +After your changes land and OSS-Fuzz production rebuilds (usually within a day): + +- **Build status** green for the project: `https://oss-fuzz-build-logs.storage.googleapis.com/status.json` +- **Coverage actually improved**: the canonical latest-report pointer is `https://storage.googleapis.com/oss-fuzz-coverage/latest_report_info/.json`; open its `summary.json` and compare own-code covered lines against the baseline recorded in your report. Per-target views live under `reports-by-target///linux/report.html`. +- **Reachability improved** (optional): the project profile at `https://introspector.oss-fuzz.com/project-profile?project=` should show fewer reachable-but-dark functions. + +## 7. Troubleshooting + +| Symptom | Fix | +|---|---| +| Skill does not auto-trigger | Invoke it explicitly: `/oss-fuzz-coverage-pipeline`, or "use the ... skill" | +| Skill not listed after install | Start a fresh session; on Gemini run `/skills reload`; confirm the install path is the one your host reads | +| Agent uses old skill behaviour | The installed copy is stale; re-run `copy_to_global.sh` | +| Project skipped as blocked(...) | That is by design: archived, inactive (24 months), moved and non-git upstreams always stop and notify; the reason is in the summary and manifest | +| Pair keeps failing the gates | The fix budget is 3 rounds; afterwards the pair is rejected and moved to the rejected area, ready for a later attempt | +| Selector `--broken` fails | It needs network access and curl; check that `status.json` is reachable | +| Concurrent builds clobber Docker images | Pipeline rule: one project (per project name) at a time | +| Agent tries to commit, push or stage | Interrupt it; the skill forbids all three (staging only when absolutely necessary); review the branch | +| Coverage looks worse locally | Like-for-like rule: never compare fresh local runs with production reports (see the skill references) | diff --git a/infra/experimental/agent-skills/copy_to_global.sh b/infra/experimental/agent-skills/copy_to_global.sh index e4eeb4b178c2..8e0b35983092 100755 --- a/infra/experimental/agent-skills/copy_to_global.sh +++ b/infra/experimental/agent-skills/copy_to_global.sh @@ -15,50 +15,67 @@ # ################################################################################ -# Copy the skills folder into the global skills directory - -# Accept user input. Should by default be "gemini" but can be overridden by the user. -SKILLS_DIR=${1:-"gemini"} +# Install the skills in this folder into an agent's skills directory. +# +# Usage: +# ./copy_to_global.sh [target ...] +# +# Targets: +# claude -> ~/.claude/skills (default when no target is given) +# gemini -> ~/.gemini/skills +# -> any other argument is used as the skills directory +# itself (created if missing; resolved relative to your +# current working directory). This works for any agent +# that reads SKILL.md-style skills, e.g. Cline's +# workspace directory: +# ./copy_to_global.sh /path/to/vscode-workspace/.cline/skills +# +# Multiple targets may be given in one invocation, e.g.: +# ./copy_to_global.sh gemini claude /abs/path/.cline/skills +# +# Skills are auto-discovered: every directory next to this script that +# contains a SKILL.md is copied. Existing skills of the same name in the +# target directory are overwritten. +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -# Gemini skills dir = ~/.gemini/skills -# Claude skills dir = ~/.claude/skills -if [ "$SKILLS_DIR" == "gemini" ]; then - GLOBAL_SKILLS_DIR="$HOME/.gemini/skills" -elif [ "$SKILLS_DIR" == "claude" ]; then - GLOBAL_SKILLS_DIR="$HOME/.claude/skills" -else - echo "Invalid skills directory specified. Use 'gemini' or 'claude'." +# Auto-discover the skills next to this script. +SKILLS=() +for entry in "$SCRIPT_DIR"/*/; do + if [ -f "${entry}SKILL.md" ]; then + SKILLS+=("$(basename "$entry")") + fi +done +if [ ${#SKILLS[@]} -eq 0 ]; then + echo "No skills found in $SCRIPT_DIR" >&2 exit 1 fi -# Log target skill directory -echo "Copying skills to global skills directory: $GLOBAL_SKILLS_DIR" - +# Resolve a target name to a skills directory. Unknown names are treated as +# directory paths so that new agents work without script changes. +resolve_target() { + case "$1" in + gemini) echo "$HOME/.gemini/skills" ;; + claude) echo "$HOME/.claude/skills" ;; + *) echo "$1" ;; + esac +} -# Now copy each of the skills from the local "skills" directory to the global skills directory -# Overwrite existing skills with the same name. -# Check if the global skills directory exists, if not create it -if [ ! -d "$GLOBAL_SKILLS_DIR" ]; then - mkdir -p "$GLOBAL_SKILLS_DIR" +# Default to claude when no target is given. +if [ $# -eq 0 ]; then + set -- claude fi -# Copy each skill from the local "skills" directory to the global skills directory -# Make sure we work from this scripts base folder and copy each of the skills -# fuzzing-memory-unsafe-expert -# fuzzing-go-expert -# fuzzing-rust-expert -# fuzzing-jvm-expert -# fuzzing-python-expert -# oss-fuzz-engineer -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -for skill in "fuzzing-memory-unsafe-expert" "fuzzing-go-expert" "fuzzing-rust-expert" "fuzzing-jvm-expert" "fuzzing-python-expert" "oss-fuzz-engineer"; do - abs_skill="$SCRIPT_DIR/$skill" - # Copy over the skill and replace any existing skill with the same name in the global skills directory - if [ -d "$GLOBAL_SKILLS_DIR/$skill" ]; then - rm -rf "$GLOBAL_SKILLS_DIR/$skill" - fi - - echo "Copying $abs_skill to $GLOBAL_SKILLS_DIR/" - cp -r "$abs_skill" "$GLOBAL_SKILLS_DIR/" +for target in "$@"; do + GLOBAL_SKILLS_DIR="$(resolve_target "$target")" + echo "Installing ${#SKILLS[@]} skills to: $GLOBAL_SKILLS_DIR" + mkdir -p "$GLOBAL_SKILLS_DIR" + for skill in "${SKILLS[@]}"; do + # Replace any existing skill with the same name in the target directory. + if [ -d "$GLOBAL_SKILLS_DIR/$skill" ]; then + rm -rf "$GLOBAL_SKILLS_DIR/$skill" + fi + echo " copying $skill" + cp -r "$SCRIPT_DIR/$skill" "$GLOBAL_SKILLS_DIR/" + done done diff --git a/infra/experimental/agent-skills/helper.py b/infra/experimental/agent-skills/helper.py index 0f76bdf343ad..4b488c3e7b48 100644 --- a/infra/experimental/agent-skills/helper.py +++ b/infra/experimental/agent-skills/helper.py @@ -73,6 +73,9 @@ ' - Rust → **fuzzing-rust-expert**\n' ' - Java/Kotlin/JVM → **fuzzing-jvm-expert**\n' ' - Python → **fuzzing-python-expert**\n' + ' - For end-to-end coverage work (fix builds, triage coverage, improve\n' + ' fuzz targets, stage paired worktree branches for human review),\n' + ' follow **oss-fuzz-coverage-pipeline**.\n' ' Activate all relevant skills at the start of your session.') EXPAND_SIZE_GUIDANCE = { diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md new file mode 100644 index 000000000000..40dca9b64ef3 --- /dev/null +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -0,0 +1,151 @@ +--- +name: oss-fuzz-coverage-pipeline +description: >- + Use this skill to bring a project that is already integrated into OSS-Fuzz (C/C++, Go, Rust, JVM or Python) to a well-fuzzed state: fix broken builds, triage live coverage, improve fuzz targets, corpora and dictionaries, and deliver the work as worktree branch duo pairs. Never commits, never pushes and by default never stages. Selects projects by name, glob pattern or all projects, verifies upstream viability from the Dockerfile, auto-detects the language from project.yaml and activates the matching language skill. Processes one project at a time via a run manifest. +--- + +# OSS-Fuzz coverage pipeline + +End-to-end pipeline for an already-integrated OSS-Fuzz project (any language with a dedicated fuzzing skill) whose fuzzing is broken or weak. The goal is that the project's own security-sensitive code is reachable and actually exercised by fuzz targets, and that the work is delivered as worktree branch duo pairs that a human can review, commit and push. + +This skill is a process wrapper. It defines what to do and in which order; it deliberately does not re-teach the underlying knowledge. While working through the phases you MUST also use: + +- **oss-fuzz-engineer**, always: OSS-Fuzz infrastructure, builds, `infra/helper.py`, project layout, public build status and coverage data. +- the **language-appropriate fuzzing skill**, auto-determined from each project's `project.yaml` `language` field: + +| `project.yaml` language | Sub-skill to activate | +|---|---| +| `c`, `c++`, or missing | **fuzzing-memory-unsafe-expert** | +| `go` | **fuzzing-go-expert** | +| `rust` | **fuzzing-rust-expert** | +| `jvm` | **fuzzing-jvm-expert** | +| `python` | **fuzzing-python-expert** | +| other (javascript, swift, ruby, ...) | none dedicated: rely on oss-fuzz-engineer, or scope the run with `--language` | + +plus their references (`code_coverage.md`, `structured_seed_generation.md`, `helper.md`, `oss_fuzz_project_structure.md`) where pointed to below. + +## Hard guardrails + +1. **Never `git commit` and never `git push`, in either repository: this is strictly forbidden.** Staging (`git add`) is also skipped by default. Leave changes as plain working-tree files (modified or untracked), and stage only when it is absolutely necessary for the task, recording why in the report. If a commit happens by accident, stop immediately and notify the user; never try to quietly undo it. +2. **Every build, check, run and coverage measurement goes through `python3 infra/helper.py`** inside the OSS-Fuzz Docker environment. Never validate with bare local toolchains (direct clang, cmake, cargo, gradle and the like). Reading and inspecting cloned source locally is fine. +3. The worktree duo pairs hold the only copies of the work. Do not run destructive git commands inside a worktree, and do not delete worktrees; rejected pairs are moved to the rejected area, never deleted. +4. Every claim of improvement must be backed by a measurement: build success, fuzzer run output or coverage numbers. No measurement, no claim. + +## Deliverables + +The result of a run, all inside the marked working area (see [references/worktree_delivery.md](references/worktree_delivery.md)): + +1. The list of surviving worktree branch duo pairs: OSS-Fuzz side `oss-fuzz--`, upstream side `-`, with their worktree paths and base commit SHAs recorded. +2. A final summary in markdown table form, one row per duo pair: project, language, duo branches, worktree paths, summary of work, coverage delta, status. +3. Skipped or unprocessable projects listed in the same summary with the concrete reason. +4. Rejected duo pairs (still failing after the fix budget) moved into the rejected area, kept intact, and marked clearly in the summary with the failure reason and what was attempted. + +## The pipeline + +Work the phases strictly in order, and conclude each phase in writing (manifest and final report) before starting the next. + +### Phase 0: Select projects and verify upstream viability + +Selection uses the deterministic selector bundled with this skill (`scripts/select_projects.sh` next to this SKILL.md); output is `projectlanguage`, one per line: + +```sh +scripts/select_projects.sh '*' # all projects (any language) +scripts/select_projects.sh 'lib*' 'json*' # glob subset +scripts/select_projects.sh --language c,c++ '*' # scope by language +scripts/select_projects.sh --broken --limit 20 '*' # broken builds first +scripts/select_projects.sh --exclude 'curl,wget' '*' +``` + +Before doing any real work on a selected project, verify the upstream is viable. **Never trust `main_repo` in project.yaml: it may be missing or wrong. Always read the project's Dockerfile to find what is actually cloned or downloaded** (a git clone, a downloaded tarball, or svn), and run the checks against that real location: + +- repository marked archived: stop and notify, manifest `blocked(upstream-archived)`; +- no commits for 24 months or more: stop and notify, `blocked(upstream-inactive)`; +- repository moved or abandoned (moved redirects, "we moved" banners, homepage pointing elsewhere while the integration still points at the dead location): stop and notify, `blocked(upstream-moved)`, noting the new location if it is discoverable, but never chase it automatically; +- source is not a git repository (tarball or svn): stop and notify, `blocked(upstream-not-git)`. + +Stopping is mandatory in every case; it is never agent judgement. Then create (or resume) the run manifest at `build/coverage-pipeline/manifest.md` in the OSS-Fuzz root: + +| project | language | status | duo branches | worktree paths | base SHAs | report | notes | +|---|---|---|---|---|---|---|---| + +Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. + +Multi-project discipline: process **one project at a time** (builds of the same project share Docker image names). With 1,300+ projects in OSS-Fuzz (~600 C/C++), prefer prioritised subsets over `*`: broken builds first (fast wins that also unlock coverage data), then the lowest own-code coverage on security-sensitive surface. + +### Phase 1: Builds, mandatory and optional fixes + +Get the builds into a state where analysis is possible, always via `infra/helper.py`: + +```sh +python3 infra/helper.py build_fuzzers +python3 infra/helper.py build_fuzzers --sanitizer coverage +python3 infra/helper.py build_fuzzers --sanitizer introspector +python3 infra/helper.py check_build +``` + +- The main fuzzer build fix is **always mandatory**: minimal, targeted changes only. +- Coverage and introspector build fixes are expected, but are **optional when the failure cause is missing corpora or an OOM/timeout**. Record the exemption with evidence, then proceed using the most recent successful historical report (Phase 2 date walk-back). +- `check_build` failures mean a harness is defective (it crashes immediately); fix that too (see the failure triage in the worktree reference). +- The full `helper.py introspector` chain is heavy; reserve it for when an actual Introspector report is needed (see `code_coverage.md` section 7). + +### Phase 2: Triage live coverage on the project's own code + +Canonical live coverage references (resolve `YYYYMMDD` with the walk-back rule below): + +- overall report: `https://storage.googleapis.com/oss-fuzz-coverage//reports//linux/report.html` +- per fuzz target: `https://storage.googleapis.com/oss-fuzz-coverage//reports-by-target///linux/report.html` (use it to attribute dark code to specific harnesses) +- parse the machine-readable `summary.json` siblings at the same paths; the HTML is the human view. +- reachability detail: `https://introspector.oss-fuzz.com/project-profile?project=` + +**Date walk-back**: coverage and introspector builds sometimes fail (cross-check the build status on introspector.oss-fuzz.com), so the latest date may have no report. Resolve the date by walking backwards day by day, at most 30 days, until a report exists, and record the date actually used. + +Produce a written target list; do not improvise targets later. + +1. Filter to the project's own code (paths under `/src//`), dropping system headers, vendored dependencies and test files; rank by dark covered-line counts, not percentages. +2. **Verify build scope before trusting any percentage**: a high percentage can be an artefact of a partial build. Compare the upstream source tree's modules against the files present in the report; flag any module that exists upstream but is absent from the build, and treat "well covered" claims as invalid until the scope is verified. Enabling missing modules is a valid Phase 3 action. +3. Cross-check Fuzz Introspector for **statically reached but dynamically unreached** functions: reachable from harnesses but never hit in the coverage report. These are prime targets (seeds, dictionaries and harness adjustment to convert static reachability into real coverage). +4. Classify each candidate security-sensitive or not; the point is attack surface, not raw percentage. In scope: file-format parsers and loaders, wire-protocol handlers, deserialisers, decompression and codec paths, custom string and number parsing on untrusted data, path and filename handling, IPC message handling, state machines driven by untrusted input, crypto primitives. Out of scope, usually: diagnostic logging and printf-style output, UI code, config writing, build tooling and test helpers, unless untrusted data reaches them. +5. Record per target: the metric that flagged it, whether it is seed-limited or harness-limited, and the expected bug class. + +### Phase 3: Act on the targets + +| Signal | Action | +|---|---| +| File dark because no harness reaches it | **New harness** (harness-limited) | +| Harness reaches it but the corpus never gets deep | **Seeds** (structured generation) and/or **dictionary** | +| Statically reached but dynamically unreached | **Seeds and dictionaries first**, then harness adjustment (state setup, enabling the path) | +| Path disabled or held fixed in the harness | **Fix the harness** (fuzz what the attacker controls) | +| Module missing from the build entirely | **Fix the build configuration** to compile it in | +| Build configuration hides code from fuzzing | **Fix build configuration** (flags, `$LIB_FUZZING_ENGINE` wiring) | + +Write and validate harnesses per the language-appropriate skill from the mapping at the top of this skill (threat model first, typed input as the sub-skill prescribes, dictionaries and seed corpora into `$OUT`, and study the project's own existing harnesses for conventions). Never target code that the production corpus already covers well. + +### Phase 4: Stage in worktree duo pairs + +Follow [references/worktree_delivery.md](references/worktree_delivery.md) precisely. Essentials: + +- A **duo pair is always created**, even when no OSS-Fuzz changes are needed, because all testing goes through the temporary copy bridge in the OSS-Fuzz side Dockerfile, and the main checkouts must stay clean. +- Branches: OSS-Fuzz side `oss-fuzz--`, upstream side `-`; the identical `` is what pairs them. +- The pristine main/master clones live under `upstream-clones/`, the worktrees under `worktrees/`, inside one marked working area whose concrete paths are reported as part of the result. +- The **temporary copy bridge** copies the changed upstream files over the cloned source at Docker build time, simulating a merged upstream without any commits: tagged BEGIN/END blocks in the Dockerfile plus a marked bridge staging folder inside the OSS-Fuzz worktree. Never remove bridges automatically; they are part of the delivered result. +- Leave changes as plain working-tree files: no staging by default, no commits, no pushes, ever. + +### Phase 5: Validate, reject or deliver + +Validate each duo pair independently and one at a time, always through `infra/helper.py`: + +1. **Build matrix**: read `sanitizers:` and `fuzzing_engines:` from `projects//project.yaml` and validate every declared configuration (for example `--sanitizer address`, `undefined`, `memory`, and engines such as `--engine afl` where declared). +2. **Per-fuzzer gate (mandatory)**: `build_fuzzers` succeeds and `check_build` passes for every new or updated fuzzer specifically, not just project-wide. +3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require no fast crash (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. A crash is either a finding (document it with a reproducer; it is a deliverable, not an obstacle) or a harness defect (fix it); never silence a crash without justification. +4. **Coverage delta, like-for-like** (same flags, same corpus source, before versus after): targeted files gain covered lines and no project-own file digresses. +5. **Fix budget of 3 rounds**: fix, retest, and at most three attempts per gate failure. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is **rejected**: move both worktrees into the rejected area with `git worktree move` (never delete), keep the branches, and record the failure reason, what was attempted and the new paths. +6. Deliver: finalise the per-project report and the manifest row, and add the row to the final summary table. + +## Definition of done + +- [ ] All mandatory builds green from the OSS-Fuzz worktree; per-fuzzer `check_build` passes. +- [ ] Quality gate clean for every new or changed target: 120 s minimum, no fast crash, no staleness. +- [ ] Coverage: documented gains on targeted files and zero digression on other project-own files (like-for-like measurement). +- [ ] Duo pairs hold plain working-tree changes; nothing staged (unless justified and recorded), nothing committed, nothing pushed. +- [ ] Bridges present and tagged; rejected pairs moved to the rejected area (not deleted); skipped projects recorded with reasons. +- [ ] Final summary table complete (delivered, rejected and skipped), per-project reports written, manifest rows final. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md new file mode 100644 index 000000000000..a8e632e50e77 --- /dev/null +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md @@ -0,0 +1,138 @@ +# Worktree-based delivery for the OSS-Fuzz coverage pipeline + +Companion reference for the `oss-fuzz-coverage-pipeline` skill: the marked working area, worktree duo pairs, the temporary copy bridge, the validation loop, rejection, pristine clone removal and the final handover. + +## The marked working area + +All artefacts live under one clearly marked working area, separate from the main OSS-Fuzz checkout and from any normal clone, with the pristine clones and the worktrees in different directories: + +``` +/ # for example ~/agent-coverage-work/ + upstream-clones// # pristine clones of the default branch (scaffolding only) + worktrees/oss-fuzz/oss-fuzz--/ + worktrees/upstream/-/ + rejected/ # rejected duo pairs moved here, kept intact + summary.md # the final summary table +``` + +The concrete paths, the branch names and the base commit SHAs are part of the result: record them in the manifest and the final summary. + +## Branch naming and duo pairs + +- Clone the upstream default branch (main or master) if a local clone does not already exist, and create worktree branches from it. Clones live under `upstream-clones/` and never under `worktrees/`. +- OSS-Fuzz side branch: `oss-fuzz--`. Upstream side branch: `-`. The `` (the topic) is identical on both sides and is what pairs them; for example `oss-fuzz-json-c-add-bcf-parser` paired with `json-c-add-bcf-parser`. +- **Always create the duo pair, even when no OSS-Fuzz changes are needed**: all testing goes through the temporary copy bridge, which lives in the OSS-Fuzz side Dockerfile, and the main checkouts must stay clean. + +```sh +# OSS-Fuzz side, run from the main OSS-Fuzz checkout: +git worktree add -b oss-fuzz-- \ + /worktrees/oss-fuzz/oss-fuzz-- + +# Upstream side, run from the pristine clone of the default branch: +git -C /upstream-clones/ worktree add -b - \ + /worktrees/upstream/- + +# Record the base each branch was created from: +git -C /upstream-clones/ rev-parse HEAD # goes into the manifest +``` + +## Never commit, never push, never stage by default + +Leave every change as plain working-tree files (modified or untracked). Commit and push are strictly forbidden. Staging is skipped unless it is absolutely necessary for the task, and any unavoidable staging is recorded with a reason in the report. If a commit happens by accident, stop and notify the user immediately; never try to quietly undo it. + +## The temporary copy bridge + +The project Dockerfile normally clones the upstream source at build time, so testing upstream changes would normally require a merge. Since commits are forbidden, simulate the merge at Docker build time instead: + +1. Copy the changed upstream files into a clearly marked bridge staging folder inside the OSS-Fuzz worktree, for example `projects//bridge//`, mirroring their upstream layout. The staging folder is part of the working-tree result. +2. Add a tagged block to the project Dockerfile, placed after the normal clone or download step, that copies the bridged files over the cloned source: + +```dockerfile +# OSS-FUZZ-AGENT-BRIDGE:BEGIN - +COPY bridge//fuzz/ $SRC//fuzz/ +# OSS-FUZZ-AGENT-BRIDGE:END - +``` + +3. Adjust the COPY source paths to whatever the Docker build context actually provides (verify on the first build; the project's existing `COPY build.sh $SRC/` lines show the convention), and make sure the destination matches where `build.sh` expects the source (`$SRC/`). +4. Never remove or comment out bridge blocks automatically: they are part of the delivered result and are removed manually by the user only after the real upstream merge has happened. + +Reading, cloning and inspecting upstream code locally is fine as usual; only validation builds must go through `python3 infra/helper.py`. + +## The validation loop (per duo pair) + +Run pairs one at a time (builds of the same project share Docker image names), from the OSS-Fuzz worktree root, always through `infra/helper.py`: + +```sh +# 1. Build the declared matrix from projects//project.yaml: +python3 infra/helper.py build_fuzzers --sanitizer address +python3 infra/helper.py build_fuzzers --sanitizer undefined # as declared +python3 infra/helper.py build_fuzzers --sanitizer coverage +# plus engines where declared, for example: --engine afl + +# 2. Mandatory per-fuzzer gate, for every new or updated fuzzer: +python3 infra/helper.py check_build + +# 3. Fuzzing-quality gate (thresholds below): +python3 infra/helper.py run_fuzzer -- \ + -max_total_time=120 -timeout=25 -rss_limit_mb=2560 +``` + +Quality thresholds, which apply on top of `check_build` passing, and to corpora, dictionaries and option changes as well as new harnesses: + +- minimum run 120 seconds per target, 300 seconds for corpus-heavy changes; +- a crash within the first 60 seconds is a fast-crash failure; +- coverage must keep increasing: no growth in the final half of the run counts as stale (compare the `cov:` figures between libFuzzer checkpoints in the run log); +- record the evidence (run logs and the coverage trend) in the per-project report. + +These libFuzzer-style flags apply to every OSS-Fuzz engine: Jazzer, Atheris and Go-native fuzzing all wrap libFuzzer. + +### Coverage delta, like-for-like + +1. Baseline before any change: build the coverage configuration in a clean state and run `python3 infra/helper.py coverage `; copy the resulting `summary.json` to a scratch directory outside the build tree. +2. Candidate: identical flags and identical corpus source in the duo-pair worktree; save its `summary.json` the same way. +3. Compare per-file `lines.covered` counts on project-own paths: targeted files must gain covered lines, and no other project-own file may lose them. With an identical corpus, locally covered lines can only come from your change, which is what makes small deltas trustworthy. +4. Never compare a fresh local number with a production report; local short-run coverage is always lower (see `code_coverage.md` section 7 and `structured_seed_generation.md`). + +### Failure triage, fix budget and rejection + +| Symptom | Likely cause | Action | +|---|---|---| +| Compile failure in the pair | Harness or build-configuration defect | Fix; counts against the budget | +| Crash immediately, even on trivial input | Harness defect | Fix the harness; re-run | +| Crash on mutated input only, reproducible | Possibly a real bug | Minimise, verify against latest upstream, document as a finding with a reproducer | +| Timeout | Pathologically slow path or loop | Bound input, stream with limits, or justify a higher limit in the report | +| OOM | Unbounded allocation from fuzzed lengths | Bound allocations in the harness | +| Leak report | Resource owned per iteration | Fix the harness to free or reuse; if the library itself leaks, document it | +| Stale or non-growing coverage | Weak corpus, weak harness or wrong target | Seeds, dictionaries or harness redesign | + +The fix budget is **3 rounds** per gate failure: fix, retest, and at most three attempts. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is rejected. Rejected pairs are **moved, never deleted**, so they can be attempted again later: + +```sh +git -C worktree move \ + /worktrees/oss-fuzz/oss-fuzz-- \ + /rejected/oss-fuzz-- + +git -C /upstream-clones/ worktree move \ + /worktrees/upstream/- \ + /rejected/- +``` + +Use `git worktree move` (not a plain `mv`) so the worktrees stay functional with their branch refs intact. The summary records the failure reason, what was attempted across the rounds, and the new rejected-area paths. + +## Pristine clone removal + +The `upstream-clones/` repositories are scaffolding: keep them while work is in progress, because branch creation, diffs and base tracking need them. Record each branch's base commit SHA in the manifest before making changes. Remove a clone only when removal is safe, normally at the very end after the user has committed the work or confirmed the plain files suffice; with nothing ever committed, deleting a clone leaves the worktree files intact but orphans their git links and branch refs, which is exactly why the base SHAs must already be recorded. If removal is not safe at handover, keep the clone and document the exact removal command instead. Never risk the work to satisfy the cleanup preference. + +## Final summary and handover + +Write `summary.md` at the working-area root with one markdown table covering everything that was selected: + +| project | language | status | duo branches | worktree paths | summary of work | coverage delta | reason | +|---|---|---|---|---|---|---|---| + +- status is `delivered`, `rejected (moved to rejected area)` or `skipped/blocked`; +- skipped rows carry the concrete reason, for example `blocked(upstream-archived)`; +- rejected rows carry the failure reason and what was attempted; +- delivered rows summarise the work (harnesses, corpora, dictionaries, build configuration, bridges) and the like-for-like coverage delta. + +The complete handover is: the surviving duo pairs in `worktrees/`, the rejected pairs in `rejected/`, `summary.md`, the per-project reports under `projects//` in the main checkout, the manifest, the recorded base SHAs, and the next actions for the user: review, commit the upstream branch first and then the OSS-Fuzz branch, remove the bridge blocks and staging folders, then move or remove worktrees and pristine clones as appropriate. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh new file mode 100755 index 000000000000..9ee867efc9de --- /dev/null +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh @@ -0,0 +1,172 @@ +#!/bin/bash -eu +# Select OSS-Fuzz projects for the coverage-pipeline skill. +# +# Usage: +# select_projects.sh [options] PATTERN [PATTERN ...] +# +# PATTERNs are shell globs matched against project directory names under +# projects/ — e.g. 'lib*', '*xml*', 'json-c', or '*' for all projects. +# +# Options: +# --broken only projects whose most recent public build +# failed (build regressions; needs network) +# --include-never-built with --broken: also include projects that have +# never built successfully (new/incomplete) +# --limit N output at most N projects (default: unlimited) +# --exclude GLOBS comma-separated name globs to drop, e.g. 'curl,*xml*' +# -h, --help show this help +# +# --language LIST comma-separated languages to keep, e.g. 'c,c++' +# or 'python' (default: all languages) +# +# Always applied: +# - project must exist locally under projects/ with a project.yaml +# - a missing language field counts as 'c++' (the legacy OSS-Fuzz default) +# +# Output: "projectlanguage" per line, sorted and de-duplicated. A +# summary line goes to stderr. Exits 1 when nothing matches. + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +OSS_FUZZ_ROOT="$(cd "$SCRIPT_DIR/../../../../.." && pwd)" +PROJECTS_DIR="$OSS_FUZZ_ROOT/projects" +STATUS_URL='https://oss-fuzz-build-logs.storage.googleapis.com/status.json' + +BROKEN=0 +NEVER_BUILT=0 +LIMIT=0 +EXCLUDE='' +LANGS='' +PATTERNS=() + +while [ $# -gt 0 ]; do + case "$1" in + --broken) BROKEN=1 ;; + --include-never-built) NEVER_BUILT=1 ;; + --limit) LIMIT="${2:?--limit needs a value}"; shift ;; + --exclude) EXCLUDE="${2:?--exclude needs a value}"; shift ;; + --language) LANGS="${2:?--language needs a value}"; shift ;; + -h|--help) + awk 'NR>1 && /^#/ {sub(/^# ?/, ""); print} NR>1 && !/^#/ {exit}' "$0" + exit 0 ;; + -*) echo "unknown option: $1 (see --help)" >&2; exit 2 ;; + *) PATTERNS+=("$1") ;; + esac + shift +done + +if [ ${#PATTERNS[@]} -eq 0 ]; then + echo "error: provide at least one pattern ('*' selects all)" >&2 + exit 2 +fi + +# Echo the project's normalized language (missing field = 'c++', the +# legacy OSS-Fuzz default). Returns 1 when project.yaml is missing. +project_language() { + local yaml="$PROJECTS_DIR/$1/project.yaml" lang + [ -f "$yaml" ] || return 1 + lang="$(sed -n 's/^language:[[:space:]]*//p' "$yaml" | head -1 \ + | tr -d "\"' \t\r" | tr '[:upper:]' '[:lower:]')" + [ -n "$lang" ] || lang='c++' + printf '%s\n' "$lang" +} + +# True when LANG passes the --language filter (empty filter = all). +lang_wanted() { + if [ -z "$LANGS" ]; then + return 0 + fi + local want + # Unquoted expansion on purpose: split the comma list on whitespace. + for want in ${LANGS//,/ }; do + if [ "$want" = "$1" ]; then + return 0 + fi + done + return 1 +} + +# True when NAME matches any of the globs passed after it. +matches_any() { + local name="$1" pat + shift + for pat in "$@"; do + # Unquoted on purpose: the argument is a glob pattern. + case "$name" in + $pat) return 0 ;; + esac + done + return 1 +} + +# --- Gather candidates: local dirs matching a pattern, C/C++ only. -------- +SELECTED=() +for dir in "$PROJECTS_DIR"/*/; do + name="$(basename "$dir")" + lang="$(project_language "$name")" || continue + matches_any "$name" "${PATTERNS[@]}" || continue + lang_wanted "$lang" || continue + SELECTED+=("$name|$lang") +done + +# --- Optional exclusions. --------------------------------------------------- +if [ -n "$EXCLUDE" ]; then + IFS=',' read -r -a EX_GLOBS <<< "$EXCLUDE" + FILTERED=() + for entry in "${SELECTED[@]}"; do + if ! matches_any "${entry%%|*}" "${EX_GLOBS[@]}"; then + FILTERED+=("$entry") + fi + done + SELECTED=("${FILTERED[@]+${FILTERED[@]}}") +fi + +# --- Optional broken-build filter (public build status). ------------------- +if [ "$BROKEN" -eq 1 ]; then + echo "Fetching build status from: $STATUS_URL" >&2 + BROKEN_LIST="$(curl -fsSL --max-time 60 "$STATUS_URL" | python3 -c ' +import json, sys +regressions_only = sys.argv[1] != "1" +status = json.load(sys.stdin) +for entry in status.get("projects", []): + name = entry.get("name") + history = entry.get("history") or [] + if not name or not history: + continue + latest = max(history, key=lambda h: h.get("finish_time", "")) + if latest.get("success", True): + continue + if regressions_only and not entry.get("last_successful_build"): + continue + print(name) +' "$NEVER_BUILT")" || { echo "error: could not fetch/parse build status" >&2; exit 1; } + FILTERED=() + for entry in "${SELECTED[@]}"; do + if printf '%s\n' "$BROKEN_LIST" | grep -qxF "${entry%%|*}"; then + FILTERED+=("$entry") + fi + done + SELECTED=("${FILTERED[@]+${FILTERED[@]}}") +fi + +# --- Sort, cap, emit. ------------------------------------------------------- +if [ ${#SELECTED[@]} -eq 0 ]; then + echo 'No projects matched.' >&2 + exit 1 +fi + +LIST="$(printf '%s\n' "${SELECTED[@]}" | sort -u)" +COUNT="$(printf '%s\n' "$LIST" | grep -c .)" +if [ "$LIMIT" -gt 0 ] && [ "$LIMIT" -lt "$COUNT" ]; then + LIST="$(printf '%s\n' "$LIST" | head -n "$LIMIT")" + COUNT="$LIMIT" +fi + +printf '%s\n' "$LIST" | sed 's/|/\t/' +SCOPE='project(s)' +if [ -n "$LANGS" ]; then + SCOPE="$LANGS project(s)" +fi +if [ "$BROKEN" -eq 1 ]; then + SCOPE="$SCOPE, broken builds only" +fi +echo "Selected $COUNT $SCOPE matching: ${PATTERNS[*]}" >&2 From 059a14dec232ed7927534a009727e16067adbd25 Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Wed, 21 Oct 2026 10:49:44 +0100 Subject: [PATCH 02/14] Add license Signed-off-by: Arthur Chan --- .../scripts/select_projects.sh | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh index 9ee867efc9de..4bb8a0ba2d0a 100755 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh @@ -1,4 +1,18 @@ #!/bin/bash -eu +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# # Select OSS-Fuzz projects for the coverage-pipeline skill. # # Usage: From 3456f491cc6227f82084a98613b44dfc3448f546 Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Wed, 21 Oct 2026 11:39:53 +0100 Subject: [PATCH 03/14] Update the skill on the deliverable and working effort Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/USAGE.md | 7 ++-- .../oss-fuzz-coverage-pipeline/SKILL.md | 22 ++++++++--- .../references/worktree_delivery.md | 38 ++++++++++++++++++- 3 files changed, 56 insertions(+), 11 deletions(-) diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md index f74867f21505..60ba3b6d98f8 100644 --- a/infra/experimental/agent-skills/USAGE.md +++ b/infra/experimental/agent-skills/USAGE.md @@ -121,14 +121,14 @@ Multiple projects: 1. Fix builds through `infra/helper.py` only: the main fuzzer build fix is always mandatory; coverage and introspector build fixes are optional only when the failure cause is missing corpora or an OOM/timeout, with the exemption recorded. 2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. 3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. -4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. +4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. A project may yield several pairs when worthwhile: fuzzer pairs, each carrying its harness plus that harness's corpora, dictionaries and options; one consolidated standalone corpus/dict/options pair (`-corpus-dict-options`); and, when the main build was broken, a `-build-fix` pair created first, whose changes later pairs copy in purely as a testing scaffold and remove again after conclusion. 5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, no crash within the first 60 seconds, no stale coverage in the final half); and a like-for-like coverage delta. After a fix budget of 3 rounds, pairs still failing are moved to the rejected area, never deleted. Crashes found along the way are treated as **findings** (documented with reproducers), not as failures to hide. ### 5.4 The deliverables -- The list of surviving worktree branch duo pairs, with paths and base commit SHAs recorded. +- The list of surviving worktree branch duo pairs (possibly several per project, grouped by the Phase 4 rules: fuzzer pairs with their own corpora, dictionaries and options, one standalone corpus pair, and a build-fix pair when needed), with paths and base commit SHAs recorded. - `summary.md` in the marked working area: one markdown table row per duo pair (project, language, duo branches, worktree paths, summary of work, coverage delta, status). - Skipped or unprocessable projects in the same table with the concrete reason. - Rejected duo pairs moved into the rejected area and marked clearly, with the failure reason and what was attempted. @@ -150,7 +150,7 @@ python3 infra/experimental/agent-skills/helper.py run-task --task "..." proj1 | Artifact | Where | What "good" looks like | |---|---|---| | Run manifest | `build/coverage-pipeline/manifest.md` (oss-fuzz root) | A row per project with language, status, duo branches, worktree paths, base SHAs and report link | -| Final summary | `/summary.md` | One table covering every selected project: delivered, rejected (moved to rejected area) and skipped, each with its reason where relevant | +| Final summary | `/summary.md` | One table covering every selected project and every pair: delivered, rejected (moved to rejected area) and skipped, each with its reason where relevant, and a depends-on column recording pairs tested with a copied build fix | | Duo pairs | `/worktrees/oss-fuzz/` and `worktrees/upstream/` | Both branches exist (`oss-fuzz--` and `-`), each holding plain working-tree changes | | Git safety | both worktrees | Nothing staged (unless justified and recorded), nothing committed, nothing pushed, anywhere | | Bridge | OSS-Fuzz worktree Dockerfile | Tagged `OSS-FUZZ-AGENT-BRIDGE:BEGIN/END` blocks plus the marked staging folder, kept intact for your manual removal later | @@ -204,6 +204,7 @@ After your changes land and OSS-Fuzz production rebuilds (usually within a day): | Agent uses old skill behaviour | The installed copy is stale; re-run `copy_to_global.sh` | | Project skipped as blocked(...) | That is by design: archived, inactive (24 months), moved and non-git upstreams always stop and notify; the reason is in the summary and manifest | | Pair keeps failing the gates | The fix budget is 3 rounds; afterwards the pair is rejected and moved to the rejected area, ready for a later attempt | +| Dependent pair retains some build-fix content | Entangled same-line edits; the residual fragment is flagged in the report and summary, and is safe because the fix branch commits first | | Selector `--broken` fails | It needs network access and curl; check that `status.json` is reachable | | Concurrent builds clobber Docker images | Pipeline rule: one project (per project name) at a time | | Agent tries to commit, push or stage | Interrupt it; the skill forbids all three (staging only when absolutely necessary); review the branch | diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index 40dca9b64ef3..6bd0bab419b9 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -35,8 +35,8 @@ plus their references (`code_coverage.md`, `structured_seed_generation.md`, `hel The result of a run, all inside the marked working area (see [references/worktree_delivery.md](references/worktree_delivery.md)): -1. The list of surviving worktree branch duo pairs: OSS-Fuzz side `oss-fuzz--`, upstream side `-`, with their worktree paths and base commit SHAs recorded. -2. A final summary in markdown table form, one row per duo pair: project, language, duo branches, worktree paths, summary of work, coverage delta, status. +1. The list of surviving worktree branch duo pairs: OSS-Fuzz side `oss-fuzz--`, upstream side `-`, with their worktree paths and base commit SHAs recorded. A project may contribute several pairs (fuzzer pairs, a standalone corpus pair, a build-fix pair), grouped per the Phase 4 rules. +2. A final summary in markdown table form, one row per duo pair: project, language, duo branches, depends-on, worktree paths, summary of work, coverage delta, status. 3. Skipped or unprocessable projects listed in the same summary with the concrete reason. 4. Rejected duo pairs (still failing after the fix budget) moved into the rejected area, kept intact, and marked clearly in the summary with the failure reason and what was attempted. @@ -68,7 +68,7 @@ Stopping is mandatory in every case; it is never agent judgement. Then create (o | project | language | status | duo branches | worktree paths | base SHAs | report | notes | |---|---|---|---|---|---|---|---| -Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. +Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. When a project yields several pairs, give each pair its own manifest row (same project, its own suffix and duo branches), with a `tested-with: -build-fix` note in the notes column when it depends on the build fix. Multi-project discipline: process **one project at a time** (builds of the same project share Docker image names). With 1,300+ projects in OSS-Fuzz (~600 C/C++), prefer prioritised subsets over `*`: broken builds first (fast wins that also unlock coverage data), then the lowest own-code coverage on security-sensitive surface. @@ -83,8 +83,8 @@ python3 infra/helper.py build_fuzzers --sanitizer introspector python3 infra/helper.py check_build ``` -- The main fuzzer build fix is **always mandatory**: minimal, targeted changes only. -- Coverage and introspector build fixes are expected, but are **optional when the failure cause is missing corpora or an OOM/timeout**. Record the exemption with evidence, then proceed using the most recent successful historical report (Phase 2 date walk-back). +- The main fuzzer build fix is **always mandatory**: minimal, targeted changes only. When the main build is broken, the fix becomes its own `-build-fix` duo pair, created and validated before any dependent pair (see Phase 4 and the worktree reference). +- Coverage and introspector build fixes are expected, but are **optional when the failure cause is missing corpora or an OOM/timeout**. Record the exemption with evidence, then proceed using the most recent successful historical report (Phase 2 date walk-back). These failures do not trigger the build-fix protocol. - `check_build` failures mean a harness is defective (it crashes immediately); fix that too (see the failure triage in the worktree reference). - The full `helper.py introspector` chain is heavy; reserve it for when an actual Introspector report is needed (see `code_coverage.md` section 7). @@ -130,6 +130,15 @@ Follow [references/worktree_delivery.md](references/worktree_delivery.md) precis - The **temporary copy bridge** copies the changed upstream files over the cloned source at Docker build time, simulating a merged upstream without any commits: tagged BEGIN/END blocks in the Dockerfile plus a marked bridge staging folder inside the OSS-Fuzz worktree. Never remove bridges automatically; they are part of the delivered result. - Leave changes as plain working-tree files: no staging by default, no commits, no pushes, ever. +**Grouping rules, one project and several pairs:** + +- A single delivered pair is the minimum bar, but when worthwhile, develop **multiple pairs per project**: lower-priority targets from the Phase 2 list that still carry worth, in your own judgement backed by the written rationale, are developed in the same run, each as its own pair. +- A pair that fixes or adds a fuzzer **carries that fuzzer's corpora, dictionaries and options in the same pair**: a harness never ships separately from its own seeds and settings. +- All remaining corpora, dictionaries and options changes with no fuzzer change are consolidated into a **single standalone pair** with the suffix `-corpus-dict-options`. +- A general build fix not tied to one fuzzer gets its own pair with the suffix `-build-fix`; build-configuration changes a specific fuzzer needs ride inside that fuzzer's pair. + +For a project whose main build was failing, the build-fix protocol in [references/worktree_delivery.md](references/worktree_delivery.md) applies: the fix pair is created and validated first, and every later pair copies the fix in purely as a testing scaffold, removing it again after conclusion. + ### Phase 5: Validate, reject or deliver Validate each duo pair independently and one at a time, always through `infra/helper.py`: @@ -139,7 +148,7 @@ Validate each duo pair independently and one at a time, always through `infra/he 3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require no fast crash (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. A crash is either a finding (document it with a reproducer; it is a deliverable, not an obstacle) or a harness defect (fix it); never silence a crash without justification. 4. **Coverage delta, like-for-like** (same flags, same corpus source, before versus after): targeted files gain covered lines and no project-own file digresses. 5. **Fix budget of 3 rounds**: fix, retest, and at most three attempts per gate failure. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is **rejected**: move both worktrees into the rejected area with `git worktree move` (never delete), keep the branches, and record the failure reason, what was attempted and the new paths. -6. Deliver: finalise the per-project report and the manifest row, and add the row to the final summary table. +6. Deliver: remove any copied build-fix changes and verify the remaining diff is purely this pair's own work (see the worktree reference), finalise the per-project report and the manifest row, and add the row to the final summary table. ## Definition of done @@ -147,5 +156,6 @@ Validate each duo pair independently and one at a time, always through `infra/he - [ ] Quality gate clean for every new or changed target: 120 s minimum, no fast crash, no staleness. - [ ] Coverage: documented gains on targeted files and zero digression on other project-own files (like-for-like measurement). - [ ] Duo pairs hold plain working-tree changes; nothing staged (unless justified and recorded), nothing committed, nothing pushed. +- [ ] Grouping respected: fuzzer pairs carry their own corpora, dictionaries and options; standalone corpus work sits in one pair; dependent pairs have their copied build-fix changes removed and verified. - [ ] Bridges present and tagged; rejected pairs moved to the rejected area (not deleted); skipped projects recorded with reasons. - [ ] Final summary table complete (delivered, rejected and skipped), per-project reports written, manifest rows final. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md index a8e632e50e77..0659312d6d2a 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md @@ -36,6 +36,39 @@ git -C /upstream-clones/ worktree add -b -/upstream-clones/ rev-parse HEAD # goes into the manifest ``` +## Pair grouping per project + +A project is delivered through one or more duo pairs: + +| Pair type | Contents | Count per project | +|---|---|---| +| Fuzzer pair | A fixed or new harness plus its own corpora, dictionaries and options, plus any build-configuration change that specific harness needs | Zero or more | +| Standalone corpus pair | All corpora, dictionaries and options changes with no fuzzer change | Zero or one, suffix `-corpus-dict-options` | +| Build-fix pair | A general build fix not tied to one fuzzer | Zero or one, suffix `-build-fix` | + +A single delivered pair is the minimum bar; develop multiple pairs when worthwhile. Worth is your own judgement, but it must be backed by the written Phase 2 rationale (dark covered lines, security sensitivity, reachability, expected impact), recorded per pair. Do not fragment trivially: corpus, dictionary and option tweaks that are not fuzzer-related always fold into the standalone pair. + +Each pair gets its own worktrees, bridge blocks, validation, fix budget and summary row. A project counts as delivered when at least one of its pairs is delivered; rejected pairs are still recorded individually. + +## Build-fix dependency protocol (main build failures only) + +This protocol applies when the **main fuzzer build** fails. Coverage or introspector build failures do not trigger it; they follow the exemption rule (missing corpora or OOM/timeout) plus the historical-report date walk-back. + +1. **Fix pair first**: create the `-build-fix` duo pair from master, fix and validate it. It is a deliverable in its own right. +2. **Dependent pairs copy the fix in for testing only**: every later fuzzer or corpora pair on the project is created from master as normal, then the build-fix changes are copied into its worktree(s) as a testing scaffold so the project builds and the gates can run. Record `tested-with: -build-fix` in the manifest and summary row; the user's commit order is the fix branch first. +3. **Remove the copied fix after conclusion**, whether the pair is delivered or rejected. The fix lives exclusively in the build-fix pair; a dependent pair's final diff contains only its own work. + +### Copying the fix in, and removing it selectively + +At copy time, record the file list and **snapshot the fix-affected files as copied**. Do the dependent work on top. At removal time, for each fix-affected file: + +- if the dependent work never touched the file, restore the branch-base version (the simple case); +- if it did (for example both edited the same Makefile), removal is surgical: compute the delta from the fix snapshot to the current file (exactly the dependent's own edits), restore the branch-base version, then re-apply that delta onto it. Result: base plus own work, fix gone. + +Verification after removal is mandatory: re-diff the worktree against the branch base and confirm only the dependent's own work remains. If an edit is genuinely entangled with the fix on the same lines and cannot be separated, preserve the dependent's functional intent, keep the minimal necessary fix fragment, and flag it explicitly in the report and summary (never keep fix content silently). This is safe in practice because the fix branch is committed first. + +Rejected dependent pairs: remove the copied fix changes first, then move the pair to the rejected area. + ## Never commit, never push, never stage by default Leave every change as plain working-tree files (modified or untracked). Commit and push are strictly forbidden. Staging is skipped unless it is absolutely necessary for the task, and any unavoidable staging is recorded with a reason in the report. If a commit happens by accident, stop and notify the user immediately; never try to quietly undo it. @@ -127,10 +160,11 @@ The `upstream-clones/` repositories are scaffolding: keep them while work is in Write `summary.md` at the working-area root with one markdown table covering everything that was selected: -| project | language | status | duo branches | worktree paths | summary of work | coverage delta | reason | -|---|---|---|---|---|---|---|---| +| project | language | status | duo branches | depends on | worktree paths | summary of work | coverage delta | reason | +|---|---|---|---|---|---|---|---|---| - status is `delivered`, `rejected (moved to rejected area)` or `skipped/blocked`; +- depends on records `tested-with: -build-fix` for pairs validated with a copied build fix; commit those fix branches first; - skipped rows carry the concrete reason, for example `blocked(upstream-archived)`; - rejected rows carry the failure reason and what was attempted; - delivered rows summarise the work (harnesses, corpora, dictionaries, build configuration, bridges) and the like-for-like coverage delta. From 5c6cd86edc0f5389ba4aa4c527fb2b9753d8e38e Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Wed, 21 Oct 2026 15:28:00 +0100 Subject: [PATCH 04/14] Update language and project matching Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/USAGE.md | 4 ++-- .../agent-skills/oss-fuzz-coverage-pipeline/SKILL.md | 8 +++++--- .../references/worktree_delivery.md | 3 ++- .../oss-fuzz-coverage-pipeline/scripts/select_projects.sh | 5 +++-- 4 files changed, 12 insertions(+), 8 deletions(-) diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md index 60ba3b6d98f8..1f9aebb687dd 100644 --- a/infra/experimental/agent-skills/USAGE.md +++ b/infra/experimental/agent-skills/USAGE.md @@ -117,9 +117,9 @@ Multiple projects: ### 5.3 What the agent will do (Phases 0 to 5) -0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. Record a run manifest at `build/coverage-pipeline/manifest.md`. +0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. The selector reports the language **declared** in project.yaml; the Dockerfile base image is never a language signal. Record a run manifest at `build/coverage-pipeline/manifest.md`. 1. Fix builds through `infra/helper.py` only: the main fuzzer build fix is always mandatory; coverage and introspector build fixes are optional only when the failure cause is missing corpora or an OOM/timeout, with the exemption recorded. -2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. +2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. 3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. 4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. A project may yield several pairs when worthwhile: fuzzer pairs, each carrying its harness plus that harness's corpora, dictionaries and options; one consolidated standalone corpus/dict/options pair (`-corpus-dict-options`); and, when the main build was broken, a `-build-fix` pair created first, whose changes later pairs copy in purely as a testing scaffold and remove again after conclusion. 5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, no crash within the first 60 seconds, no stale coverage in the final half); and a like-for-like coverage delta. After a fix budget of 3 rounds, pairs still failing are moved to the rejected area, never deleted. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index 6bd0bab419b9..01c0d0c5c97c 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -11,7 +11,7 @@ End-to-end pipeline for an already-integrated OSS-Fuzz project (any language wit This skill is a process wrapper. It defines what to do and in which order; it deliberately does not re-teach the underlying knowledge. While working through the phases you MUST also use: - **oss-fuzz-engineer**, always: OSS-Fuzz infrastructure, builds, `infra/helper.py`, project layout, public build status and coverage data. -- the **language-appropriate fuzzing skill**, auto-determined from each project's `project.yaml` `language` field: +- the **language-appropriate fuzzing skill**, auto-determined from each project's `project.yaml` `language` field, which is the sole source of truth for the language. The Dockerfile base image is toolchain compatibility only and is never a language signal: a C++ project with a Rust module may legitimately sit on the rust base image. | `project.yaml` language | Sub-skill to activate | |---|---| @@ -46,7 +46,7 @@ Work the phases strictly in order, and conclude each phase in writing (manifest ### Phase 0: Select projects and verify upstream viability -Selection uses the deterministic selector bundled with this skill (`scripts/select_projects.sh` next to this SKILL.md); output is `projectlanguage`, one per line: +Selection uses the deterministic selector bundled with this skill (`scripts/select_projects.sh` next to this SKILL.md); output is `projectlanguage`, one per line, where the language is the one **declared** in `project.yaml` (a missing field is the legacy `c++` default), not verified against the source, and the Dockerfile base image is never a language signal. Activate the matching sub-skill per the mapping above. ```sh scripts/select_projects.sh '*' # all projects (any language) @@ -68,7 +68,7 @@ Stopping is mandatory in every case; it is never agent judgement. Then create (o | project | language | status | duo branches | worktree paths | base SHAs | report | notes | |---|---|---|---|---|---|---|---| -Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. When a project yields several pairs, give each pair its own manifest row (same project, its own suffix and duo branches), with a `tested-with: -build-fix` note in the notes column when it depends on the build fix. +Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. When a project yields several pairs, give each pair its own manifest row (same project, its own suffix and duo branches), with a `tested-with: -build-fix` note in the notes column when it depends on the build fix. The language column starts as the declared language; after a mid-run language switch (see Phase 2), record it as `declared(X)/actual(Y)`. Multi-project discipline: process **one project at a time** (builds of the same project share Docker image names). With 1,300+ projects in OSS-Fuzz (~600 C/C++), prefer prioritised subsets over `*`: broken builds first (fast wins that also unlock coverage data), then the lowest own-code coverage on security-sensitive surface. @@ -101,6 +101,8 @@ Canonical live coverage references (resolve `YYYYMMDD` with the walk-back rule b Produce a written target list; do not improvise targets later. +**Verify the declared language while analysing**: while reading the source tree, build scripts and existing harnesses, check that the declared language (project.yaml) matches reality. If it does not, switch the active language sub-skill to the matching one and **continue in all cases**, even when the actual language falls outside the run's `--language` filter, recording the discrepancy: manifest `declared(X)/actual(Y)`, evidence in the report, actual language in the summary with the declared one noted. For mixed-language projects, the project-level language stays the corrected declared one, but a pair whose harness targets a module in a different language may activate that language's skill for that pair, noted in the pair's row. + 1. Filter to the project's own code (paths under `/src//`), dropping system headers, vendored dependencies and test files; rank by dark covered-line counts, not percentages. 2. **Verify build scope before trusting any percentage**: a high percentage can be an artefact of a partial build. Compare the upstream source tree's modules against the files present in the report; flag any module that exists upstream but is absent from the build, and treat "well covered" claims as invalid until the scope is verified. Enabling missing modules is a valid Phase 3 action. 3. Cross-check Fuzz Introspector for **statically reached but dynamically unreached** functions: reachable from harnesses but never hit in the coverage report. These are prime targets (seeds, dictionaries and harness adjustment to convert static reachability into real coverage). diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md index 0659312d6d2a..7b0571d7a80c 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md @@ -48,7 +48,7 @@ A project is delivered through one or more duo pairs: A single delivered pair is the minimum bar; develop multiple pairs when worthwhile. Worth is your own judgement, but it must be backed by the written Phase 2 rationale (dark covered lines, security sensitivity, reachability, expected impact), recorded per pair. Do not fragment trivially: corpus, dictionary and option tweaks that are not fuzzer-related always fold into the standalone pair. -Each pair gets its own worktrees, bridge blocks, validation, fix budget and summary row. A project counts as delivered when at least one of its pairs is delivered; rejected pairs are still recorded individually. +Each pair gets its own worktrees, bridge blocks, validation, fix budget and summary row. A project counts as delivered when at least one of its pairs is delivered; rejected pairs are still recorded individually. For mixed-language projects, a pair whose harness targets a module in a different language may activate that language's skill, noted in the pair's row. ## Build-fix dependency protocol (main build failures only) @@ -165,6 +165,7 @@ Write `summary.md` at the working-area root with one markdown table covering eve - status is `delivered`, `rejected (moved to rejected area)` or `skipped/blocked`; - depends on records `tested-with: -build-fix` for pairs validated with a copied build fix; commit those fix branches first; +- the language column shows the actual language used, with the declared one noted where they differ, for example `rust (declared c++)`; - skipped rows carry the concrete reason, for example `blocked(upstream-archived)`; - rejected rows carry the failure reason and what was attempted; - delivered rows summarise the work (harnesses, corpora, dictionaries, build configuration, bridges) and the like-for-like coverage delta. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh index 4bb8a0ba2d0a..a6ffef88c58b 100755 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts/select_projects.sh @@ -37,8 +37,9 @@ # - project must exist locally under projects/ with a project.yaml # - a missing language field counts as 'c++' (the legacy OSS-Fuzz default) # -# Output: "projectlanguage" per line, sorted and de-duplicated. A -# summary line goes to stderr. Exits 1 when nothing matches. +# Output: "projectlanguage" per line, sorted and de-duplicated, where +# language is the language DECLARED in project.yaml (not verified against +# the source). A summary line goes to stderr. Exits 1 when nothing matches. SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" OSS_FUZZ_ROOT="$(cd "$SCRIPT_DIR/../../../../.." && pwd)" From eb9ee778698819ab7cc6101ad123b9ea1e19aeed Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Wed, 21 Oct 2026 15:37:15 +0100 Subject: [PATCH 05/14] Update fresh checking rules Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/USAGE.md | 2 +- .../agent-skills/oss-fuzz-coverage-pipeline/SKILL.md | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md index 1f9aebb687dd..f9a9c5616a3e 100644 --- a/infra/experimental/agent-skills/USAGE.md +++ b/infra/experimental/agent-skills/USAGE.md @@ -117,7 +117,7 @@ Multiple projects: ### 5.3 What the agent will do (Phases 0 to 5) -0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. The selector reports the language **declared** in project.yaml; the Dockerfile base image is never a language signal. Record a run manifest at `build/coverage-pipeline/manifest.md`. +0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. The selector reports the language **declared** in project.yaml; the Dockerfile base image is never a language signal. All project data is reread fresh on every run, so a project whose project.yaml changed since an earlier run is treated according to the new values, never previous memory. Reruns after a previous batch of duo pairs likewise re-derive coverage and targets from fresh data, because the previous batch may have changed the picture. Record a run manifest at `build/coverage-pipeline/manifest.md`. 1. Fix builds through `infra/helper.py` only: the main fuzzer build fix is always mandatory; coverage and introspector build fixes are optional only when the failure cause is missing corpora or an OOM/timeout, with the exemption recorded. 2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. 3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index 01c0d0c5c97c..e676babacece 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -30,6 +30,7 @@ plus their references (`code_coverage.md`, `structured_seed_generation.md`, `hel 2. **Every build, check, run and coverage measurement goes through `python3 infra/helper.py`** inside the OSS-Fuzz Docker environment. Never validate with bare local toolchains (direct clang, cmake, cargo, gradle and the like). Reading and inspecting cloned source locally is fine. 3. The worktree duo pairs hold the only copies of the work. Do not run destructive git commands inside a worktree, and do not delete worktrees; rejected pairs are moved to the rejected area, never deleted. 4. Every claim of improvement must be backed by a measurement: build success, fuzzer run output or coverage numbers. No measurement, no claim. +5. **Fresh data on every run**: whenever this skill determines any project data, treat it as a new check and never rely on memory from previous runs. Reread `project.yaml` (language, sanitizers, engines, `main_repo`), the Dockerfile, the build status, the coverage report pointers and the upstream state on every run. If a project's language changed since an earlier run (for example C++ to Rust), the new run must see the new value and select the sub-skill accordingly. The manifest carries process state (what is done and pending), not world facts: when a fresh read contradicts something recorded earlier, the fresh read wins and the manifest row is corrected. This matters most on reruns: a project may have been updated by a previous batch of duo pairs (merged and since picked up by production), so earlier coverage data, target lists and assumptions no longer hold. A second run re-derives everything from fresh data and treats earlier findings, including which code was dark or already covered, as history rather than truth. ## Deliverables @@ -46,7 +47,7 @@ Work the phases strictly in order, and conclude each phase in writing (manifest ### Phase 0: Select projects and verify upstream viability -Selection uses the deterministic selector bundled with this skill (`scripts/select_projects.sh` next to this SKILL.md); output is `projectlanguage`, one per line, where the language is the one **declared** in `project.yaml` (a missing field is the legacy `c++` default), not verified against the source, and the Dockerfile base image is never a language signal. Activate the matching sub-skill per the mapping above. +Selection uses the deterministic selector bundled with this skill (`scripts/select_projects.sh` next to this SKILL.md); output is `projectlanguage`, one per line, where the language is the one **declared** in `project.yaml` (a missing field is the legacy `c++` default), not verified against the source, and the Dockerfile base image is never a language signal. Activate the matching sub-skill per the mapping above. Reread this data on every run, even for projects seen before: a project whose `project.yaml` changed since an earlier run (for example its language moving from C++ to Rust) is treated according to the fresh values, never the previous run's memory. ```sh scripts/select_projects.sh '*' # all projects (any language) @@ -68,7 +69,7 @@ Stopping is mandatory in every case; it is never agent judgement. Then create (o | project | language | status | duo branches | worktree paths | base SHAs | report | notes | |---|---|---|---|---|---|---|---| -Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. When a project yields several pairs, give each pair its own manifest row (same project, its own suffix and duo branches), with a `tested-with: -build-fix` note in the notes column when it depends on the build fix. The language column starts as the declared language; after a mid-run language switch (see Phase 2), record it as `declared(X)/actual(Y)`. +Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. When a project yields several pairs, give each pair its own manifest row (same project, its own suffix and duo branches), with a `tested-with: -build-fix` note in the notes column when it depends on the build fix. The language column starts as the declared language; after a mid-run language switch (see Phase 2), record it as `declared(X)/actual(Y)`. A rerun after a previous delivered batch starts fresh rows (or a new dated section) and treats earlier rows as history only; previously rejected pairs may be deliberately retried against the new state. Multi-project discipline: process **one project at a time** (builds of the same project share Docker image names). With 1,300+ projects in OSS-Fuzz (~600 C/C++), prefer prioritised subsets over `*`: broken builds first (fast wins that also unlock coverage data), then the lowest own-code coverage on security-sensitive surface. @@ -99,7 +100,7 @@ Canonical live coverage references (resolve `YYYYMMDD` with the walk-back rule b **Date walk-back**: coverage and introspector builds sometimes fail (cross-check the build status on introspector.oss-fuzz.com), so the latest date may have no report. Resolve the date by walking backwards day by day, at most 30 days, until a report exists, and record the date actually used. -Produce a written target list; do not improvise targets later. +Produce a written target list; do not improvise targets later. On a rerun of a project previously worked on, do not reuse the earlier target list: re-derive it from the fresh reports, because the previous batch of pairs may have shifted coverage (previously dark targets may now be covered and must be dropped; still-dark ones are re-attempted). **Verify the declared language while analysing**: while reading the source tree, build scripts and existing harnesses, check that the declared language (project.yaml) matches reality. If it does not, switch the active language sub-skill to the matching one and **continue in all cases**, even when the actual language falls outside the run's `--language` filter, recording the discrepancy: manifest `declared(X)/actual(Y)`, evidence in the report, actual language in the summary with the declared one noted. For mixed-language projects, the project-level language stays the corrected declared one, but a pair whose harness targets a module in a different language may activate that language's skill for that pair, noted in the pair's row. From 6788e6a89239c6253a0ef0c56d88e0eaedc04410 Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Thu, 22 Oct 2026 04:33:41 +0100 Subject: [PATCH 06/14] Add strict no modification rules for upstream code Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/USAGE.md | 3 ++- .../agent-skills/oss-fuzz-coverage-pipeline/SKILL.md | 8 +++++--- .../references/worktree_delivery.md | 2 ++ 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md index f9a9c5616a3e..597d713f1db4 100644 --- a/infra/experimental/agent-skills/USAGE.md +++ b/infra/experimental/agent-skills/USAGE.md @@ -120,7 +120,7 @@ Multiple projects: 0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. The selector reports the language **declared** in project.yaml; the Dockerfile base image is never a language signal. All project data is reread fresh on every run, so a project whose project.yaml changed since an earlier run is treated according to the new values, never previous memory. Reruns after a previous batch of duo pairs likewise re-derive coverage and targets from fresh data, because the previous batch may have changed the picture. Record a run manifest at `build/coverage-pipeline/manifest.md`. 1. Fix builds through `infra/helper.py` only: the main fuzzer build fix is always mandatory; coverage and introspector build fixes are optional only when the failure cause is missing corpora or an OOM/timeout, with the exemption recorded. 2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. -3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. +3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. It never modifies upstream product code: upstream changes are limited to fuzzers with their corpora, dictionaries and options, fuzz build configuration and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions; product-code bugs become findings. 4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. A project may yield several pairs when worthwhile: fuzzer pairs, each carrying its harness plus that harness's corpora, dictionaries and options; one consolidated standalone corpus/dict/options pair (`-corpus-dict-options`); and, when the main build was broken, a `-build-fix` pair created first, whose changes later pairs copy in purely as a testing scaffold and remove again after conclusion. 5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, no crash within the first 60 seconds, no stale coverage in the final half); and a like-for-like coverage delta. After a fix budget of 3 rounds, pairs still failing are moved to the rejected area, never deleted. @@ -204,6 +204,7 @@ After your changes land and OSS-Fuzz production rebuilds (usually within a day): | Agent uses old skill behaviour | The installed copy is stale; re-run `copy_to_global.sh` | | Project skipped as blocked(...) | That is by design: archived, inactive (24 months), moved and non-git upstreams always stop and notify; the reason is in the summary and manifest | | Pair keeps failing the gates | The fix budget is 3 rounds; afterwards the pair is rejected and moved to the rejected area, ready for a later attempt | +| Agent wants to patch upstream product code | Forbidden: upstream changes are limited to fuzzers, fuzz build configuration and macro-guarded additions; product-code bugs are findings | | Dependent pair retains some build-fix content | Entangled same-line edits; the residual fragment is flagged in the report and summary, and is safe because the fix branch commits first | | Selector `--broken` fails | It needs network access and curl; check that `status.json` is reachable | | Concurrent builds clobber Docker images | Pipeline rule: one project (per project name) at a time | diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index e676babacece..b011ea2a2ec5 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -31,6 +31,7 @@ plus their references (`code_coverage.md`, `structured_seed_generation.md`, `hel 3. The worktree duo pairs hold the only copies of the work. Do not run destructive git commands inside a worktree, and do not delete worktrees; rejected pairs are moved to the rejected area, never deleted. 4. Every claim of improvement must be backed by a measurement: build success, fuzzer run output or coverage numbers. No measurement, no claim. 5. **Fresh data on every run**: whenever this skill determines any project data, treat it as a new check and never rely on memory from previous runs. Reread `project.yaml` (language, sanitizers, engines, `main_repo`), the Dockerfile, the build status, the coverage report pointers and the upstream state on every run. If a project's language changed since an earlier run (for example C++ to Rust), the new run must see the new value and select the sub-skill accordingly. The manifest carries process state (what is done and pending), not world facts: when a fresh read contradicts something recorded earlier, the fresh read wins and the manifest row is corrected. This matters most on reruns: a project may have been updated by a previous batch of duo pairs (merged and since picked up by production), so earlier coverage data, target lists and assumptions no longer hold. A second run re-derives everything from fresh data and treats earlier findings, including which code was dark or already covered, as history rather than truth. +6. **Upstream product code is off-limits**: never modify the upstream project's own source code. Bugs in it are findings (documented, filed by the user), never fixes by the agent. The only permitted upstream changes are: fuzzers (harnesses and their corpora, dictionaries and options), fuzz build configuration (wiring the targets into the build), and additions guarded by the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro, which is the sole sanctioned product-code edit (the macro is set in OSS-Fuzz builds and is the standard way to make behaviour fuzz-friendly without affecting production). If product code gets modified by accident, stop immediately and notify, never quietly undo. The OSS-Fuzz repo side (Dockerfile, `build.sh`, `project.yaml`) is not affected by this rule. ## Deliverables @@ -84,7 +85,7 @@ python3 infra/helper.py build_fuzzers --sanitizer introspector python3 infra/helper.py check_build ``` -- The main fuzzer build fix is **always mandatory**: minimal, targeted changes only. When the main build is broken, the fix becomes its own `-build-fix` duo pair, created and validated before any dependent pair (see Phase 4 and the worktree reference). +- The main fuzzer build fix is **always mandatory**: minimal, targeted changes only. When the main build is broken, the fix becomes its own `-build-fix` duo pair, created and validated before any dependent pair (see Phase 4 and the worktree reference). An upstream-side build fix must itself stay within the allowed upstream categories (fuzzers, fuzz build configuration, the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro); if the build can only be repaired by changing product code, stop and notify `blocked(product-code-fix-needed)`. - Coverage and introspector build fixes are expected, but are **optional when the failure cause is missing corpora or an OOM/timeout**. Record the exemption with evidence, then proceed using the most recent successful historical report (Phase 2 date walk-back). These failures do not trigger the build-fix protocol. - `check_build` failures mean a harness is defective (it crashes immediately); fix that too (see the failure triage in the worktree reference). - The full `helper.py introspector` chain is heavy; reserve it for when an actual Introspector report is needed (see `code_coverage.md` section 7). @@ -121,7 +122,7 @@ Produce a written target list; do not improvise targets later. On a rerun of a p | Module missing from the build entirely | **Fix the build configuration** to compile it in | | Build configuration hides code from fuzzing | **Fix build configuration** (flags, `$LIB_FUZZING_ENGINE` wiring) | -Write and validate harnesses per the language-appropriate skill from the mapping at the top of this skill (threat model first, typed input as the sub-skill prescribes, dictionaries and seed corpora into `$OUT`, and study the project's own existing harnesses for conventions). Never target code that the production corpus already covers well. +Write and validate harnesses per the language-appropriate skill from the mapping at the top of this skill (threat model first, typed input as the sub-skill prescribes, dictionaries and seed corpora into `$OUT`, and study the project's own existing harnesses for conventions). Never target code that the production corpus already covers well. Fuzz-unfriendly behaviour inside product code (non-determinism, network, real-time limits) is handled by adding `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded code, never by editing the product code unguarded. ### Phase 4: Stage in worktree duo pairs @@ -148,7 +149,7 @@ Validate each duo pair independently and one at a time, always through `infra/he 1. **Build matrix**: read `sanitizers:` and `fuzzing_engines:` from `projects//project.yaml` and validate every declared configuration (for example `--sanitizer address`, `undefined`, `memory`, and engines such as `--engine afl` where declared). 2. **Per-fuzzer gate (mandatory)**: `build_fuzzers` succeeds and `check_build` passes for every new or updated fuzzer specifically, not just project-wide. -3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require no fast crash (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. A crash is either a finding (document it with a reproducer; it is a deliverable, not an obstacle) or a harness defect (fix it); never silence a crash without justification. +3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require no fast crash (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. A crash is either a finding (document it with a reproducer; it is a deliverable, not an obstacle) or a harness defect (fix it); never silence a crash without justification. A crash rooted in product code is always a finding, never a product-code patch. 4. **Coverage delta, like-for-like** (same flags, same corpus source, before versus after): targeted files gain covered lines and no project-own file digresses. 5. **Fix budget of 3 rounds**: fix, retest, and at most three attempts per gate failure. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is **rejected**: move both worktrees into the rejected area with `git worktree move` (never delete), keep the branches, and record the failure reason, what was attempted and the new paths. 6. Deliver: remove any copied build-fix changes and verify the remaining diff is purely this pair's own work (see the worktree reference), finalise the per-project report and the manifest row, and add the row to the final summary table. @@ -160,5 +161,6 @@ Validate each duo pair independently and one at a time, always through `infra/he - [ ] Coverage: documented gains on targeted files and zero digression on other project-own files (like-for-like measurement). - [ ] Duo pairs hold plain working-tree changes; nothing staged (unless justified and recorded), nothing committed, nothing pushed. - [ ] Grouping respected: fuzzer pairs carry their own corpora, dictionaries and options; standalone corpus work sits in one pair; dependent pairs have their copied build-fix changes removed and verified. +- [ ] Upstream diffs contain only allowed changes: fuzzers with their corpora, dictionaries and options, fuzz build configuration, and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions; no product code touched. - [ ] Bridges present and tagged; rejected pairs moved to the rejected area (not deleted); skipped projects recorded with reasons. - [ ] Final summary table complete (delivered, rejected and skipped), per-project reports written, manifest rows final. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md index 7b0571d7a80c..6422181ec1f9 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md @@ -48,6 +48,8 @@ A project is delivered through one or more duo pairs: A single delivered pair is the minimum bar; develop multiple pairs when worthwhile. Worth is your own judgement, but it must be backed by the written Phase 2 rationale (dark covered lines, security sensitivity, reachability, expected impact), recorded per pair. Do not fragment trivially: corpus, dictionary and option tweaks that are not fuzzer-related always fold into the standalone pair. +Upstream worktrees contain only allowed changes: fuzzers with their corpora, dictionaries and options, fuzz build configuration, and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions. Upstream product code is never modified; product-code bugs are findings. The OSS-Fuzz worktrees carry the integration changes (Dockerfile, `build.sh`, `project.yaml`) plus the bridge. + Each pair gets its own worktrees, bridge blocks, validation, fix budget and summary row. A project counts as delivered when at least one of its pairs is delivered; rejected pairs are still recorded individually. For mixed-language projects, a pair whose harness targets a module in a different language may activate that language's skill, noted in the pair's row. ## Build-fix dependency protocol (main build failures only) From a36c0faa13d8e7cd62d9b2eb1815d4b6be5a3a46 Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Thu, 22 Oct 2026 08:25:09 +0100 Subject: [PATCH 07/14] Update build check failing case Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/USAGE.md | 11 ++--- .../oss-fuzz-coverage-pipeline/SKILL.md | 11 ++--- .../references/worktree_delivery.md | 43 ++++++++++++++----- 3 files changed, 44 insertions(+), 21 deletions(-) diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md index 597d713f1db4..73a74c26d0a9 100644 --- a/infra/experimental/agent-skills/USAGE.md +++ b/infra/experimental/agent-skills/USAGE.md @@ -122,9 +122,9 @@ Multiple projects: 2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. 3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. It never modifies upstream product code: upstream changes are limited to fuzzers with their corpora, dictionaries and options, fuzz build configuration and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions; product-code bugs become findings. 4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. A project may yield several pairs when worthwhile: fuzzer pairs, each carrying its harness plus that harness's corpora, dictionaries and options; one consolidated standalone corpus/dict/options pair (`-corpus-dict-options`); and, when the main build was broken, a `-build-fix` pair created first, whose changes later pairs copy in purely as a testing scaffold and remove again after conclusion. -5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, no crash within the first 60 seconds, no stale coverage in the final half); and a like-for-like coverage delta. After a fix budget of 3 rounds, pairs still failing are moved to the rejected area, never deleted. +5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory, or failing solely due to a marked real finding); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, nothing fuzzer-attributable within the first 60 seconds, no stale coverage in the final half); and a like-for-like coverage delta. Every post-build crash, leak, OOM, timeout or assertion is classified first: real upstream findings keep the pair delivered with the discovery marked, while only fuzzer faults consume the fix budget of 3 rounds, after which the pair is moved to the rejected area, never deleted. Nothing is ever newly silenced, pre-existing suppressions are honoured unchanged, and a timeout from genuinely heavy legitimate processing may receive a measured, light increase delivered as fuzzer options. -Crashes found along the way are treated as **findings** (documented with reproducers), not as failures to hide. +Crashes, leaks, OOMs, timeouts and assertions are classified (fuzzer fault versus real finding); real findings are **documented with reproducers**, never hidden and never silenced. ### 5.4 The deliverables @@ -154,7 +154,7 @@ python3 infra/experimental/agent-skills/helper.py run-task --task "..." proj1 | Duo pairs | `/worktrees/oss-fuzz/` and `worktrees/upstream/` | Both branches exist (`oss-fuzz--` and `-`), each holding plain working-tree changes | | Git safety | both worktrees | Nothing staged (unless justified and recorded), nothing committed, nothing pushed, anywhere | | Bridge | OSS-Fuzz worktree Dockerfile | Tagged `OSS-FUZZ-AGENT-BRIDGE:BEGIN/END` blocks plus the marked staging folder, kept intact for your manual removal later | -| Validation evidence | per-project report | The declared sanitizer/engine matrix built; per-fuzzer `check_build` passes; quality-gate run logs (120 s minimum, no fast crash, no staleness); a like-for-like coverage-delta table | +| Validation evidence | per-project report | The declared sanitizer/engine matrix built; per-fuzzer `check_build` passes (or fails solely due to a marked real finding); quality-gate run logs (120 s minimum, nothing fuzzer-attributable, no staleness); a like-for-like coverage-delta table | Handy checks: @@ -172,8 +172,8 @@ ls /rejected/ ### 6.2 Definition of done (per project, taken from the skill) -- [ ] All mandatory builds green from the OSS-Fuzz worktree; per-fuzzer `check_build` passes -- [ ] Quality gate clean for every new or changed target: 120 s minimum, no fast crash, no staleness +- [ ] All mandatory builds green from the OSS-Fuzz worktree; per-fuzzer `check_build` passes (or fails solely due to a marked real finding) +- [ ] Quality gate clean for every new or changed target: 120 s minimum, nothing fuzzer-attributable, no staleness; real findings excepted and marked - [ ] Coverage: documented gains on the targeted files and zero digression elsewhere - [ ] Duo pair holds plain working-tree changes; nothing staged, committed or pushed - [ ] Bridge tagged and kept; rejection (if any) moved to the rejected area; manifest row final @@ -204,6 +204,7 @@ After your changes land and OSS-Fuzz production rebuilds (usually within a day): | Agent uses old skill behaviour | The installed copy is stale; re-run `copy_to_global.sh` | | Project skipped as blocked(...) | That is by design: archived, inactive (24 months), moved and non-git upstreams always stop and notify; the reason is in the summary and manifest | | Pair keeps failing the gates | The fix budget is 3 rounds; afterwards the pair is rejected and moved to the rejected area, ready for a later attempt | +| Fuzzer crashes but it looks like a genuine bug | Classify first: real findings keep the pair delivered and marked; only fuzzer faults use the fix budget; never silence leaks, OOMs or timeouts | | Agent wants to patch upstream product code | Forbidden: upstream changes are limited to fuzzers, fuzz build configuration and macro-guarded additions; product-code bugs are findings | | Dependent pair retains some build-fix content | Entangled same-line edits; the residual fragment is flagged in the report and summary, and is safe because the fix branch commits first | | Selector `--broken` fails | It needs network access and curl; check that `status.json` is reachable | diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index b011ea2a2ec5..4ea50dc670ae 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -32,6 +32,7 @@ plus their references (`code_coverage.md`, `structured_seed_generation.md`, `hel 4. Every claim of improvement must be backed by a measurement: build success, fuzzer run output or coverage numbers. No measurement, no claim. 5. **Fresh data on every run**: whenever this skill determines any project data, treat it as a new check and never rely on memory from previous runs. Reread `project.yaml` (language, sanitizers, engines, `main_repo`), the Dockerfile, the build status, the coverage report pointers and the upstream state on every run. If a project's language changed since an earlier run (for example C++ to Rust), the new run must see the new value and select the sub-skill accordingly. The manifest carries process state (what is done and pending), not world facts: when a fresh read contradicts something recorded earlier, the fresh read wins and the manifest row is corrected. This matters most on reruns: a project may have been updated by a previous batch of duo pairs (merged and since picked up by production), so earlier coverage data, target lists and assumptions no longer hold. A second run re-derives everything from fresh data and treats earlier findings, including which code was dark or already covered, as history rather than truth. 6. **Upstream product code is off-limits**: never modify the upstream project's own source code. Bugs in it are findings (documented, filed by the user), never fixes by the agent. The only permitted upstream changes are: fuzzers (harnesses and their corpora, dictionaries and options), fuzz build configuration (wiring the targets into the build), and additions guarded by the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro, which is the sole sanctioned product-code edit (the macro is set in OSS-Fuzz builds and is the standard way to make behaviour fuzz-friendly without affecting production). If product code gets modified by accident, stop immediately and notify, never quietly undo. The OSS-Fuzz repo side (Dockerfile, `build.sh`, `project.yaml`) is not affected by this rule. +7. **Classify every post-build crash; never silence**: a crash, memleak, OOM or timeout after a successful build, whether it surfaces in `check_build` or in the fuzzing runs, is never an automatic failure: classify it first (procedure in the worktree reference). Fuzzer faults consume the fix budget; real findings keep the pair delivered with the discovery marked and consume nothing. Never introduce new silencing (no leak-detection muting, no suppression flags or files, no raised limits to dodge a failure). Pre-existing suppressions in the existing integration or existing fuzzers are honoured unchanged: carried over for new and fixed fuzzers, neither removed nor widened, and only what they already cover is out of scope. ## Deliverables @@ -87,7 +88,7 @@ python3 infra/helper.py check_build - The main fuzzer build fix is **always mandatory**: minimal, targeted changes only. When the main build is broken, the fix becomes its own `-build-fix` duo pair, created and validated before any dependent pair (see Phase 4 and the worktree reference). An upstream-side build fix must itself stay within the allowed upstream categories (fuzzers, fuzz build configuration, the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro); if the build can only be repaired by changing product code, stop and notify `blocked(product-code-fix-needed)`. - Coverage and introspector build fixes are expected, but are **optional when the failure cause is missing corpora or an OOM/timeout**. Record the exemption with evidence, then proceed using the most recent successful historical report (Phase 2 date walk-back). These failures do not trigger the build-fix protocol. -- `check_build` failures mean a harness is defective (it crashes immediately); fix that too (see the failure triage in the worktree reference). +- A `check_build` failure caused by a crash is classified like any other post-build crash (see the worktree reference): it is a harness defect only when the crash is the fuzzer's fault; a crash classified as a real finding still passes this gate with the discovery marked. - The full `helper.py introspector` chain is heavy; reserve it for when an actual Introspector report is needed (see `code_coverage.md` section 7). ### Phase 2: Triage live coverage on the project's own code @@ -148,16 +149,16 @@ For a project whose main build was failing, the build-fix protocol in [reference Validate each duo pair independently and one at a time, always through `infra/helper.py`: 1. **Build matrix**: read `sanitizers:` and `fuzzing_engines:` from `projects//project.yaml` and validate every declared configuration (for example `--sanitizer address`, `undefined`, `memory`, and engines such as `--engine afl` where declared). -2. **Per-fuzzer gate (mandatory)**: `build_fuzzers` succeeds and `check_build` passes for every new or updated fuzzer specifically, not just project-wide. -3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require no fast crash (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. A crash is either a finding (document it with a reproducer; it is a deliverable, not an obstacle) or a harness defect (fix it); never silence a crash without justification. A crash rooted in product code is always a finding, never a product-code patch. +2. **Per-fuzzer gate (mandatory)**: `build_fuzzers` succeeds, and `check_build` passes for every new or updated fuzzer specifically, not just project-wide, **or its failure is solely attributable to a crash classified as a real finding** (documented with a reproducer and marked). +3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require **no crash, leak, OOM or timeout attributable to the fuzzer** (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. Every post-build crash goes through the classification procedure in the worktree reference: real findings keep the pair delivered with the discovery marked, fuzzer faults consume the fix budget, and a crash rooted in product code is always a finding, never a product-code patch. 4. **Coverage delta, like-for-like** (same flags, same corpus source, before versus after): targeted files gain covered lines and no project-own file digresses. 5. **Fix budget of 3 rounds**: fix, retest, and at most three attempts per gate failure. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is **rejected**: move both worktrees into the rejected area with `git worktree move` (never delete), keep the branches, and record the failure reason, what was attempted and the new paths. 6. Deliver: remove any copied build-fix changes and verify the remaining diff is purely this pair's own work (see the worktree reference), finalise the per-project report and the manifest row, and add the row to the final summary table. ## Definition of done -- [ ] All mandatory builds green from the OSS-Fuzz worktree; per-fuzzer `check_build` passes. -- [ ] Quality gate clean for every new or changed target: 120 s minimum, no fast crash, no staleness. +- [ ] All mandatory builds green from the OSS-Fuzz worktree; per-fuzzer `check_build` passes (or fails solely due to a marked real finding). +- [ ] Quality gate clean for every new or changed target: 120 s minimum, nothing fuzzer-attributable (crash, leak, OOM, timeout) within the first 60 seconds, no staleness; real findings excepted and marked. - [ ] Coverage: documented gains on targeted files and zero digression on other project-own files (like-for-like measurement). - [ ] Duo pairs hold plain working-tree changes; nothing staged (unless justified and recorded), nothing committed, nothing pushed. - [ ] Grouping respected: fuzzer pairs carry their own corpora, dictionaries and options; standalone corpus work sits in one pair; dependent pairs have their copied build-fix changes removed and verified. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md index 6422181ec1f9..1a7524ed5104 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md @@ -128,19 +128,40 @@ These libFuzzer-style flags apply to every OSS-Fuzz engine: Jazzer, Atheris and 3. Compare per-file `lines.covered` counts on project-own paths: targeted files must gain covered lines, and no other project-own file may lose them. With an identical corpus, locally covered lines can only come from your change, which is what makes small deltas trustworthy. 4. Never compare a fresh local number with a production report; local short-run coverage is always lower (see `code_coverage.md` section 7 and `structured_seed_generation.md`). -### Failure triage, fix budget and rejection +### Crash classification (every post-build crash, leak, OOM, timeout or assertion) -| Symptom | Likely cause | Action | +A crash after a successful build is never an automatic failure, whether it surfaces in `check_build` or in the fuzzing runs. Capture the output and artefact, reproduce it standalone (`run_fuzzer` with the artefact), then classify. + +**Fuzzer faults** (fix within the budget): + +| Symptom | Cause | Action | +|---|---|---| +| Crash on any input | Wrong fuzzer assumptions or an API violation | Fix the harness to match the API contract | +| Target mishandles the input | The normal flow requires legitimate input | Make the harness construct valid input (structured generation, seeds, validation before the call) | +| Assertion or abort | The assertion is a precondition legitimate callers guarantee, or it fires on documented unsupported input (accepted production behaviour) | Fix the harness or add input checking so such input is never produced | +| Leak report | Harness-owned resource per iteration | Fix the harness to free or reuse it | +| OOM | Unbounded allocation driven by fuzzed lengths in the harness | Bound the allocations | +| Timeout | Harness-induced pathological slowness from its own unbounded input | Bound input, stream with limits | + +**Real findings** (the pair stays delivered with the discovery marked; no budget consumed): + +| Symptom | Cause | Outcome | |---|---|---| -| Compile failure in the pair | Harness or build-configuration defect | Fix; counts against the budget | -| Crash immediately, even on trivial input | Harness defect | Fix the harness; re-run | -| Crash on mutated input only, reproducible | Possibly a real bug | Minimise, verify against latest upstream, document as a finding with a reproducer | -| Timeout | Pathologically slow path or loop | Bound input, stream with limits, or justify a higher limit in the report | -| OOM | Unbounded allocation from fuzzed lengths | Bound allocations in the harness | -| Leak report | Resource owned per iteration | Fix the harness to free or reuse; if the library itself leaks, document it | -| Stale or non-growing coverage | Weak corpus, weak harness or wrong target | Seeds, dictionaries or harness redesign | - -The fix budget is **3 rounds** per gate failure: fix, retest, and at most three attempts. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is rejected. Rejected pairs are **moved, never deleted**, so they can be attempted again later: +| Sanitizer crash (ASan, MSan, UBSan and related) | Genuine memory-safety or correctness bug in upstream code | Finding with a reproducer | +| Leak report | Upstream product code allocates on an exercised path and never frees | Finding; never mute detection | +| OOM | An upstream memleak accumulating across iterations | Finding; never silence it or raise the limit to dodge it | +| Timeout | Genuine hang, infinite loop or pathological blowup in upstream code on attacker-influenceable input | Finding | +| Assertion or abort | Unexpected data snuck past earlier checking and reached it, and reaching it is not accepted production behaviour | Finding | + +**Timeout exception for legitimately heavy processing**: when the timeout is genuine, heavy but legitimate upstream work, a light increase of the timeout is allowed, but only once it has been determined, by measuring the completion time under the raised limit, that the increase gives the process enough time to finish. Deliver the increase as fuzzer options (for example the `.options` file) and document the measurement in the report. If that determination cannot be made, no timeout change happens at all: classify normally. + +**Silencing rules**: never introduce new silencing, whether leak-detection muting (for example `-detect_leaks=0`), suppression files or annotations, or raised limits to dodge a failure. Pre-existing suppressions in the existing integration or existing fuzzers are honoured unchanged: carried over for new and fixed fuzzers, neither removed nor widened, noted in the report, and only what they already cover is out of scope. + +After classifying a real finding: mark it, quarantine the artefact, re-run to confirm the remainder is stable, and measure the coverage delta on the non-crashing run. + +### Fix budget and rejection + +The fix budget is **3 rounds** per gate failure caused by a fuzzer fault (including compile failures and staleness): fix, retest, and at most three attempts. A pair still failing after the third round is rejected. Rejected pairs are **moved, never deleted**, so they can be attempted again later: ```sh git -C worktree move \ From af9454f15efaa84ce10857e01f9f7974e34d110b Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Fri, 23 Oct 2026 04:34:29 +0100 Subject: [PATCH 08/14] Add multiple targets consideration and update README/USAGE Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/README.md | 9 +- infra/experimental/agent-skills/USAGE.md | 57 +++++++++--- .../fuzzing-coverage-targeting/SKILL.md | 90 +++++++++++++++++++ infra/experimental/agent-skills/helper.py | 3 +- .../oss-fuzz-coverage-pipeline/SKILL.md | 11 ++- 5 files changed, 151 insertions(+), 19 deletions(-) create mode 100644 infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md diff --git a/infra/experimental/agent-skills/README.md b/infra/experimental/agent-skills/README.md index 6b959a120d78..a30e0319949f 100644 --- a/infra/experimental/agent-skills/README.md +++ b/infra/experimental/agent-skills/README.md @@ -3,7 +3,7 @@ Skills and tooling that let an agent CLI (Gemini CLI or Claude Code) write, build, and extend OSS-Fuzz fuzzing integrations. The folder ships: -- Seven skills that give an agent OSS-Fuzz-specific knowledge. +- Eight skills that give an agent OSS-Fuzz-specific knowledge. - `infra/experimental/agent-skills/helper.py`, a wrapper that launches agent sessions over one or more OSS-Fuzz projects in parallel. - `copy_to_global.sh`, an installer that places the skills where your @@ -12,8 +12,8 @@ build, and extend OSS-Fuzz fuzzing integrations. The folder ships: Before running anything in this folder, review the [threat model](#threat-model). This tooling is experimental and runs agents in unrestricted modes. -For the full walkthrough — install the skills, run the coverage pipeline, -verify results — see [USAGE.md](USAGE.md). +For the full walkthrough (install the skills, run the coverage pipeline, +verify results) see [USAGE.md](USAGE.md). ## Contents @@ -25,7 +25,8 @@ verify results — see [USAGE.md](USAGE.md). | `fuzzing-jvm-expert/` | Skill: fuzz JVM projects (Java/Kotlin/Scala) with Jazzer | | `fuzzing-python-expert/` | Skill: fuzz Python projects with Atheris | | `oss-fuzz-engineer/` | Skill: OSS-Fuzz infra workflows (integrate, fix, extend) | -| `oss-fuzz-coverage-pipeline/` | Skill: language-agnostic end-to-end coverage pipeline wrapping the language fuzzing skills — selects projects by glob/wildcard or all, auto-detects language from project.yaml, fixes builds, triages coverage, improves targets, stages paired worktree branches; ships `scripts/select_projects.sh` | +| `oss-fuzz-coverage-pipeline/` | Skill: language-agnostic end-to-end coverage pipeline wrapping the language fuzzing skills (selects projects by glob/wildcard or all, auto-detects language from project.yaml, fixes builds, triages coverage, improves targets, stages paired worktree branches; ships `scripts/select_projects.sh`) | +| `fuzzing-coverage-targeting/` | Skill: which fuzz targets to create and what shape each harness takes (worth model, dark-region taxonomy, harness patterns, portfolio with overlap rules; default cap 5 pairs per project) | | `infra/experimental/agent-skills/helper.py` | Driver that launches agent sessions per OSS-Fuzz project | | `copy_to_global.sh` | Installs the skills into `~/.gemini/skills`, `~/.claude/skills`, or any skills directory you pass as an argument (e.g. Cline's `.cline/skills`) | diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md index 73a74c26d0a9..eeda887212f0 100644 --- a/infra/experimental/agent-skills/USAGE.md +++ b/infra/experimental/agent-skills/USAGE.md @@ -10,6 +10,7 @@ Companion to [README.md](README.md), which covers the folder overview and the th |---|---| | Six knowledge skills | `oss-fuzz-engineer` (infrastructure) plus one fuzzing skill per language: `fuzzing-memory-unsafe-expert`, `fuzzing-go-expert`, `fuzzing-rust-expert`, `fuzzing-jvm-expert`, `fuzzing-python-expert` | | `oss-fuzz-coverage-pipeline` | The orchestrator skill: project selection with upstream viability checks, build fixes, live coverage triage, harness/corpus/dictionary improvements, and delivery as worktree branch duo pairs validated through a temporary copy bridge. It auto-detects each project's language from `project.yaml` and activates the matching language skill. | +| `fuzzing-coverage-targeting` | The architect skill: which fuzz targets are worth creating and what shape each harness takes (worth model, taxonomy, patterns, portfolio and overlap rules); used by the pipeline in Phases 2, 3 and 5, and usable on its own | | `oss-fuzz-coverage-pipeline/scripts/select_projects.sh` | Deterministic project selector: globs, `--language`, `--broken`, `--limit`, `--exclude`. | | `copy_to_global.sh` | Installer: `claude` (the default), `gemini`, or any skills directory you name. | | `helper.py` | An optional headless batch driver for the `claude` and `gemini` CLIs only; it is not needed for interactive use. | @@ -31,7 +32,7 @@ The skills are plain `SKILL.md` markdown, so they are model-agnostic and portabl The wrapper delegates all domain knowledge to those skills and their references: how to write harnesses in each language, how to drive `infra/helper.py`, how to read coverage reports and how to generate structured seeds. What the wrapper itself contributes is the process layer that none of the six contain: project selection and upstream viability checks, the phase discipline, the git guardrails, worktree duo pairs with the temporary copy bridge, the validation gates with the fix budget, the run manifest, rejection handling and the final summary. -The six knowledge skills remain fully usable on their own; the wrapper simply composes them into one end-to-end delivery pipeline, so nothing in them was duplicated or superseded. +The six knowledge skills remain fully usable on their own; the wrapper simply composes them into one end-to-end delivery pipeline, so nothing in them was duplicated or superseded. The architect skill `fuzzing-coverage-targeting` joins the composition in Phases 2, 3 and 5 to rank worthy targets and shape harnesses (default cap 5 pairs per project, adjustable in your prompt); the wrapper remains the only boss. ## 2. Prerequisites @@ -89,7 +90,18 @@ Selected 18 project(s) matching: json* ## 5. Usage -### 5.1 Select the project set +There are two ways to run the pipeline. Both use the same installed skills, follow the same phases and produce the same deliverables; they differ only in who drives the agent. + +### 5.1 Two ways to run + +| | Approach A: interactive | Approach B: helper.py batch | +|---|---|---| +| Hosts | any skill-compatible host (Cline, Claude Code, Gemini CLI), any model | the `claude` or `gemini` CLI only | +| Style | conversational, you approve each step | headless, one prompt per project | +| Parallelism | one project at a time | parallel sessions, default 2, `-j` to change | +| Best for | pilots, first runs, learning the flow, tight review | batches of projects and repeated campaigns | + +### 5.2 Select the project set (shared by both approaches) ```sh SEL=/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/scripts @@ -103,7 +115,7 @@ $SEL/select_projects.sh --exclude 'curl,wget' --language c,c++ '*' The recommended order at scale is **broken builds first** (quick wins that also unlock coverage data), followed by the lowest own-code coverage on security-sensitive surface. -### 5.2 Kick off the pipeline (interactive, any host or model) +### 5.3 Approach A: run interactively (any host or model) Single project: either phrase it naturally and let the description auto-trigger the skill: @@ -115,18 +127,18 @@ Multiple projects: > Use the oss-fuzz-coverage-pipeline skill. Phase 0: run scripts/select_projects.sh --broken --limit 5 'lib*', verify upstream viability for each, record the manifest, then run the full pipeline on each viable project, one at a time, resuming from the manifest between sessions. -### 5.3 What the agent will do (Phases 0 to 5) +### 5.4 What the agent will do in either approach (Phases 0 to 5) 0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. The selector reports the language **declared** in project.yaml; the Dockerfile base image is never a language signal. All project data is reread fresh on every run, so a project whose project.yaml changed since an earlier run is treated according to the new values, never previous memory. Reruns after a previous batch of duo pairs likewise re-derive coverage and targets from fresh data, because the previous batch may have changed the picture. Record a run manifest at `build/coverage-pipeline/manifest.md`. 1. Fix builds through `infra/helper.py` only: the main fuzzer build fix is always mandatory; coverage and introspector build fixes are optional only when the failure cause is missing corpora or an OOM/timeout, with the exemption recorded. -2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. +2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. Target selection and harness shaping come from the architect skill `fuzzing-coverage-targeting`, producing a ranked multi-target portfolio with overlap-aware incremental worth (default cap 5 pairs per project, adjustable in your prompt). 3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. It never modifies upstream product code: upstream changes are limited to fuzzers with their corpora, dictionaries and options, fuzz build configuration and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions; product-code bugs become findings. 4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. A project may yield several pairs when worthwhile: fuzzer pairs, each carrying its harness plus that harness's corpora, dictionaries and options; one consolidated standalone corpus/dict/options pair (`-corpus-dict-options`); and, when the main build was broken, a `-build-fix` pair created first, whose changes later pairs copy in purely as a testing scaffold and remove again after conclusion. 5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory, or failing solely due to a marked real finding); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, nothing fuzzer-attributable within the first 60 seconds, no stale coverage in the final half); and a like-for-like coverage delta. Every post-build crash, leak, OOM, timeout or assertion is classified first: real upstream findings keep the pair delivered with the discovery marked, while only fuzzer faults consume the fix budget of 3 rounds, after which the pair is moved to the rejected area, never deleted. Nothing is ever newly silenced, pre-existing suppressions are honoured unchanged, and a timeout from genuinely heavy legitimate processing may receive a measured, light increase delivered as fuzzer options. Crashes, leaks, OOMs, timeouts and assertions are classified (fuzzer fault versus real finding); real findings are **documented with reproducers**, never hidden and never silenced. -### 5.4 The deliverables +### 5.5 The deliverables (either approach) - The list of surviving worktree branch duo pairs (possibly several per project, grouped by the Phase 4 rules: fuzzer pairs with their own corpora, dictionaries and options, one standalone corpus pair, and a build-fix pair when needed), with paths and base commit SHAs recorded. - `summary.md` in the marked working area: one markdown table row per duo pair (project, language, duo branches, worktree paths, summary of work, coverage delta, status). @@ -134,15 +146,38 @@ Crashes, leaks, OOMs, timeouts and assertions are classified (fuzzer fault versu - Rejected duo pairs moved into the rejected area and marked clearly, with the failure reason and what was attempted. - Per-project reports, the manifest, and the next actions for you. -### 5.5 Optional: headless batch (Claude Code and Gemini CLI only) +### 5.6 Approach B: run the same work with helper.py (headless, Claude Code and Gemini CLI) + +`helper.py` drives the same skills without an interactive session: it builds task-specific prompts that reference the skills (including `oss-fuzz-coverage-pipeline` and `fuzzing-coverage-targeting`), auto-detects `claude` or `gemini` on `PATH`, and launches one headless session per project, in parallel (default 2). The skills must be installed for the CLI it drives (`~/.claude/skills` or `~/.gemini/skills`; see section 3). + +Equivalent commands for the manual steps: + +| Manual step (Approach A) | helper.py equivalent | +|---|---| +| Preview what will be asked | `python3 infra/experimental/agent-skills/helper.py show-prompt ` | +| Full pipeline on named projects | `python3 infra/experimental/agent-skills/helper.py run-task --task "Use the oss-fuzz-coverage-pipeline skill end to end: verify upstream viability, fix builds, triage coverage via fuzzing-coverage-targeting, deliver worktree duo pairs; never commit or push" proj1 proj2` | +| Fix known-broken builds | `python3 infra/experimental/agent-skills/helper.py fix-builds proj1 proj2` | +| Discover and fix broken builds | `python3 infra/experimental/agent-skills/helper.py fix-broken-builds --language c,c++ --limit 20 -j 2` (add `--print-only` to preview the selection) | +| Expand coverage on a project | `python3 infra/experimental/agent-skills/helper.py expand-oss-fuzz-projects proj` | +| Onboard a new project | `python3 infra/experimental/agent-skills/helper.py integrate-project https://github.com/org/repo` | +| Clean local run artefacts | `python3 infra/experimental/agent-skills/helper.py clean proj` | + +A typical batch run: ```sh -python3 infra/experimental/agent-skills/helper.py show-prompt fix-builds htslib # dry run -python3 infra/experimental/agent-skills/helper.py fix-builds proj1 proj2 -python3 infra/experimental/agent-skills/helper.py fix-broken-builds --language c,c++ --limit 20 -python3 infra/experimental/agent-skills/helper.py run-task --task "..." proj1 +cd +# 1. Dry run: print the exact prompt that would be sent. +python3 infra/experimental/agent-skills/helper.py show-prompt run-task --task "Use the oss-fuzz-coverage-pipeline skill end to end" htslib +# 2. Run it across the batch. +python3 infra/experimental/agent-skills/helper.py run-task --task "Use the oss-fuzz-coverage-pipeline skill end to end" htslib json-c +# 3. Watch progress; each session logs here. +tail -f build/agent-logs/*.log ``` +Session output lands in `build/agent-logs/-.log`, and each agent writes a per-project report under `projects//`; verification afterwards is exactly section 6. + +Caveats: helper.py only drives the `claude` and `gemini` CLIs (Cline users stay with Approach A); sessions run with permissions skipped, so respect the threat model in README.md and use a sandbox; `fix-builds` and `expand-oss-fuzz-projects` use their own built-in workflows, which are narrower than the full pipeline (for the complete duo-pair protocol use `run-task` with an explicit task string naming the skill); and parallelism is bounded by Docker capacity, so keep `-j 2` unless your machine can take more concurrent builds. + ## 6. Verify the results ### 6.1 Immediately: what to inspect after each project diff --git a/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md b/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md new file mode 100644 index 000000000000..64f900276b8b --- /dev/null +++ b/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md @@ -0,0 +1,90 @@ +--- +name: fuzzing-coverage-targeting +description: + Use this skill to decide which fuzz targets a project needs and what shape each harness should take: rank dark reachable regions from coverage data into a diversified portfolio of worthy harnesses (any language), using a worth model, a dark-region taxonomy, harness pattern selection (parser, round-trip, property, stateful, option-matrix, component-API) and overlap-aware diminishing returns with a default cap of 5 pairs per project. Use alongside the language fuzzing skills (which say how to write a harness) and the OSS-Fuzz coverage pipeline (which runs the process). +--- + +# Fuzzing coverage targeting + +This skill is the architect for fuzz target selection: it answers **which** parts of a project are worth harnessing and **what shape** each harness should take. It does not write harnesses (that is the language skills' craft), it owns no process state (that is the coverage pipeline's job), and it issues no commands (that is oss-fuzz-engineer's territory). It is language-agnostic and works standalone whenever a task is about choosing or planning fuzz targets. + +## Inputs + +- Fresh coverage data: the project's `summary.json` and per-target reports (`reports-by-target`), filtered to the project's own code with build scope verified (a high percentage over a partial build is meaningless). +- Fuzz Introspector data: statically reachable but dynamically unreached functions, optimal targets, blockers. +- The project's source structure: modules, public APIs, state machines, configuration surfaces. +- The language expert's threat model, when composed by the pipeline: attacker-reachable surfaces are the top ranking weight among equals. + +## Worth model + +Score each dark region as **dark reachable lines × functional importance × bug-class diversity ÷ harness cost**: + +- **Dark reachable lines**: lines statically reachable from some harness entry but uncovered, counted as absolute lines, not percentages. +- **Functional importance**: core parsing, transformation, protocol logic and data handling rank high; diagnostics, debug printers, test helpers and build-time code are excluded outright. +- **Bug-class diversity**: memory safety (ASan/MSan/UBSan), logic and assertion violations, hangs and pathological blowups, and leaks. A region rich in several classes outranks a one-class region of equal size. +- **Harness cost**: API complexity, state setup, corpus construction effort. + +## Dark-region taxonomy and harness opportunity types + +| Dark region type | Typical shape | Example surfaces | +|---|---|---| +| Parser or decoder entry | bytes-in harness | file-format loaders, wire-protocol frames, deserialisers | +| Transformation or codec stage | round-trip or stage harness | encode/decode pairs, colour conversion, filters, re-serialisation | +| Stateful session or protocol object | stateful sequence harness | connection objects, protocol state machines, transactions | +| Internal component API | component harness | regex engines, expression evaluators, container and data-structure operations, maths kernels | +| Configuration or option surface | option-matrix harness | format variants, feature flags, mode switches | +| Encode or serialise direction | round-trip harness | writers and serializers (often dark because only the decode side gets fuzzed) | +| Text or language frontend | structured grammar-ish harness | compilers, template engines, query languages | +| Error and recovery paths | mutation of valid seeds | malformed variants of legitimate inputs | + +The taxonomy is the antidote to the "one parser pair per project" outcome: every module shape is a candidate, not only attacker-bytes entry points. + +## Harness pattern catalogue + +- **Bytes-into-parser** (classic): feed the buffer to the decode entry point; the language skills' craft sections cover it fully. +- **Round-trip**: encode a structured input, decode it back, assert the invariant (identity, or documented lossiness): + +```cpp +extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { + FuzzedDataProvider fdp(data, size); + auto obj = Thing::parse(fdp.ConsumeRemainingBytesAsString()); + std::string out = obj.serialize(); // dark encode direction + Thing round = Thing::parse(out); // decoder re-entry + if (obj != round) abort(); // invariant: stable round-trip + return 0; +} +``` + +- **Property or invariant / differential**: check a documented property of one implementation (cheap: commutativity, idempotence, bounds, canonical-form equality) or compare against a reference implementation (expensive but powerful). +- **Stateful sequence**: build a session object, then drive N API calls from the fuzzed bytes; required for protocol and connection logic that no single call exercises: + +```cpp +extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { + FuzzedDataProvider fdp(data, size); + Session s; // state machine under test + while (fdp.remaining_bytes() > 0) { + uint8_t op = fdp.ConsumeIntegralInRange(0, Session::kMaxOp); + std::vector arg = + fdp.ConsumeBytes(fdp.ConsumeIntegralInRange(0, 64)); + s.step(op, arg); // ignore documented errors + } + return 0; +} +``` + +- **Option or configuration matrix**: consume small values as option flags and mode switches before the payload so one target sweeps many compiled paths cheaply; pair with a dictionary of option names. +- **Component API**: call the internal module through its intended API with typed, bounded parameters; the language skills' legitimate-input requirement applies (construct valid objects; never feed raw bytes where a struct is expected). + +## Portfolio construction + +1. Enumerate dark regions from the inputs; classify each by the taxonomy; score each by the worth model. +2. Apply the security weighting: attacker-reachable surfaces rank first among equals; functional dark logic is fully worthy; diagnostics, logging, printf-style output, test helpers and build-time code stay excluded. +3. Produce a **ranked multi-target list** expected to yield several focused harnesses rather than one or two; a single-target list is a smell that the taxonomy was not applied. +4. Selection is iterative with diminishing returns, executed by the pipeline with its coverage ledger: after each accepted pair, re-rank the remaining targets by **incremental** worth (projected gains minus already-covered ground). A pair whose gains mostly overlap already-accepted coverage is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded. +5. **Default cap: at most 5 pairs per project** (fuzzer pairs plus the standalone corpus pair), unless the user raises or lowers the threshold in their prompt. Stop earlier when the remaining targets' projected incremental worth is marginal. + +## Non-goals + +- No harness writing: the language skills own craft and idioms. +- No process, sequencing, acceptance or state: the coverage pipeline owns the manifest, the ledger and every decision. +- No infrastructure commands: oss-fuzz-engineer owns `infra/helper.py` usage and report fetching. diff --git a/infra/experimental/agent-skills/helper.py b/infra/experimental/agent-skills/helper.py index 4b488c3e7b48..9787aa6605fd 100644 --- a/infra/experimental/agent-skills/helper.py +++ b/infra/experimental/agent-skills/helper.py @@ -75,7 +75,8 @@ ' - Python → **fuzzing-python-expert**\n' ' - For end-to-end coverage work (fix builds, triage coverage, improve\n' ' fuzz targets, stage paired worktree branches for human review),\n' - ' follow **oss-fuzz-coverage-pipeline**.\n' + ' follow **oss-fuzz-coverage-pipeline**, using **fuzzing-coverage-targeting**\n' + ' for target selection and portfolio rules.\n' ' Activate all relevant skills at the start of your session.') EXPAND_SIZE_GUIDANCE = { diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index 4ea50dc670ae..684b90f82904 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -22,6 +22,8 @@ This skill is a process wrapper. It defines what to do and in which order; it de | `python` | **fuzzing-python-expert** | | other (javascript, swift, ruby, ...) | none dedicated: rely on oss-fuzz-engineer, or scope the run with `--language` | +- **fuzzing-coverage-targeting**, in Phases 2, 3 and 5: which targets are worth harnessing and what shape each harness takes (worth model, dark-region taxonomy, harness patterns, portfolio cap and overlap rules). + plus their references (`code_coverage.md`, `structured_seed_generation.md`, `helper.md`, `oss_fuzz_project_structure.md`) where pointed to below. ## Hard guardrails @@ -102,7 +104,7 @@ Canonical live coverage references (resolve `YYYYMMDD` with the walk-back rule b **Date walk-back**: coverage and introspector builds sometimes fail (cross-check the build status on introspector.oss-fuzz.com), so the latest date may have no report. Resolve the date by walking backwards day by day, at most 30 days, until a report exists, and record the date actually used. -Produce a written target list; do not improvise targets later. On a rerun of a project previously worked on, do not reuse the earlier target list: re-derive it from the fresh reports, because the previous batch of pairs may have shifted coverage (previously dark targets may now be covered and must be dropped; still-dark ones are re-attempted). +Produce a written target list via **fuzzing-coverage-targeting** (worth model, dark-region taxonomy, ranked multi-target list) and initialise the run's coverage ledger from the per-target baseline reports (`reports-by-target`); do not improvise targets later. On a rerun of a project previously worked on, do not reuse the earlier target list: re-derive it from the fresh reports, because the previous batch of pairs may have shifted coverage (previously dark targets may now be covered and must be dropped; still-dark ones are re-attempted). **Verify the declared language while analysing**: while reading the source tree, build scripts and existing harnesses, check that the declared language (project.yaml) matches reality. If it does not, switch the active language sub-skill to the matching one and **continue in all cases**, even when the actual language falls outside the run's `--language` filter, recording the discrepancy: manifest `declared(X)/actual(Y)`, evidence in the report, actual language in the summary with the declared one noted. For mixed-language projects, the project-level language stays the corrected declared one, but a pair whose harness targets a module in a different language may activate that language's skill for that pair, noted in the pair's row. @@ -123,7 +125,7 @@ Produce a written target list; do not improvise targets later. On a rerun of a p | Module missing from the build entirely | **Fix the build configuration** to compile it in | | Build configuration hides code from fuzzing | **Fix build configuration** (flags, `$LIB_FUZZING_ENGINE` wiring) | -Write and validate harnesses per the language-appropriate skill from the mapping at the top of this skill (threat model first, typed input as the sub-skill prescribes, dictionaries and seed corpora into `$OUT`, and study the project's own existing harnesses for conventions). Never target code that the production corpus already covers well. Fuzz-unfriendly behaviour inside product code (non-determinism, network, real-time limits) is handled by adding `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded code, never by editing the product code unguarded. +Choose the harness shape per **fuzzing-coverage-targeting**'s pattern catalogue (round-trip, property, stateful sequence, option-matrix, component-API or classic parser), then write and validate it per the language-appropriate skill from the mapping at the top of this skill (threat model first, typed input as the sub-skill prescribes, dictionaries and seed corpora into `$OUT`, and study the project's own existing harnesses for conventions). Never target code that the production corpus already covers well. Fuzz-unfriendly behaviour inside product code (non-determinism, network, real-time limits) is handled by adding `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded code, never by editing the product code unguarded. ### Phase 4: Stage in worktree duo pairs @@ -151,10 +153,12 @@ Validate each duo pair independently and one at a time, always through `infra/he 1. **Build matrix**: read `sanitizers:` and `fuzzing_engines:` from `projects//project.yaml` and validate every declared configuration (for example `--sanitizer address`, `undefined`, `memory`, and engines such as `--engine afl` where declared). 2. **Per-fuzzer gate (mandatory)**: `build_fuzzers` succeeds, and `check_build` passes for every new or updated fuzzer specifically, not just project-wide, **or its failure is solely attributable to a crash classified as a real finding** (documented with a reproducer and marked). 3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require **no crash, leak, OOM or timeout attributable to the fuzzer** (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. Every post-build crash goes through the classification procedure in the worktree reference: real findings keep the pair delivered with the discovery marked, fuzzer faults consume the fix budget, and a crash rooted in product code is always a finding, never a product-code patch. -4. **Coverage delta, like-for-like** (same flags, same corpus source, before versus after): targeted files gain covered lines and no project-own file digresses. +4. **Coverage delta, like-for-like** (same flags, same corpus source, before versus after): targeted files gain covered lines and no project-own file digresses. Update the run's coverage ledger with the pair's measured coverage and compute its **incremental** contribution against what existing fuzzers and previously accepted pairs already cover: a pair whose gains mostly overlap is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded (rules from fuzzing-coverage-targeting). 5. **Fix budget of 3 rounds**: fix, retest, and at most three attempts per gate failure. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is **rejected**: move both worktrees into the rejected area with `git worktree move` (never delete), keep the branches, and record the failure reason, what was attempted and the new paths. 6. Deliver: remove any copied build-fix changes and verify the remaining diff is purely this pair's own work (see the worktree reference), finalise the per-project report and the manifest row, and add the row to the final summary table. +Portfolio cap: at most **5 pairs per project** by default (fuzzer pairs plus the standalone corpus pair); the user's prompt may raise or lower the threshold, and the run stops adding pairs earlier when the remaining incremental worth is marginal. + ## Definition of done - [ ] All mandatory builds green from the OSS-Fuzz worktree; per-fuzzer `check_build` passes (or fails solely due to a marked real finding). @@ -162,6 +166,7 @@ Validate each duo pair independently and one at a time, always through `infra/he - [ ] Coverage: documented gains on targeted files and zero digression on other project-own files (like-for-like measurement). - [ ] Duo pairs hold plain working-tree changes; nothing staged (unless justified and recorded), nothing committed, nothing pushed. - [ ] Grouping respected: fuzzer pairs carry their own corpora, dictionaries and options; standalone corpus work sits in one pair; dependent pairs have their copied build-fix changes removed and verified. +- [ ] Portfolio respected: ranked multi-target selection via fuzzing-coverage-targeting, incremental worth and overlap rule applied, cap honoured (default 5 pairs per project unless the user's prompt overrides it). - [ ] Upstream diffs contain only allowed changes: fuzzers with their corpora, dictionaries and options, fuzz build configuration, and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions; no product code touched. - [ ] Bridges present and tagged; rejected pairs moved to the rejected area (not deleted); skipped projects recorded with reasons. - [ ] Final summary table complete (delivered, rejected and skipped), per-project reports written, manifest rows final. From 08c976a5e7c15838dd957d2cd3477ebba503b8cd Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Fri, 23 Oct 2026 06:54:54 +0100 Subject: [PATCH 09/14] Fix syntax error Signed-off-by: Arthur Chan --- .../agent-skills/fuzzing-coverage-targeting/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md b/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md index 64f900276b8b..c991bb8a5dd8 100644 --- a/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md +++ b/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md @@ -1,6 +1,6 @@ --- name: fuzzing-coverage-targeting -description: +description: >- Use this skill to decide which fuzz targets a project needs and what shape each harness should take: rank dark reachable regions from coverage data into a diversified portfolio of worthy harnesses (any language), using a worth model, a dark-region taxonomy, harness pattern selection (parser, round-trip, property, stateful, option-matrix, component-API) and overlap-aware diminishing returns with a default cap of 5 pairs per project. Use alongside the language fuzzing skills (which say how to write a harness) and the OSS-Fuzz coverage pipeline (which runs the process). --- From 6b0ebaa7cf669445c374ff985bcefb09528a950a Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Fri, 23 Oct 2026 09:49:06 +0100 Subject: [PATCH 10/14] Update coverage and helper.py Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/README.md | 1 + infra/experimental/agent-skills/USAGE.md | 13 +- infra/experimental/agent-skills/helper.py | 224 +++++++++++++++++- .../oss-fuzz-coverage-pipeline/SKILL.md | 2 +- .../references/worktree_delivery.md | 18 +- 5 files changed, 232 insertions(+), 26 deletions(-) diff --git a/infra/experimental/agent-skills/README.md b/infra/experimental/agent-skills/README.md index a30e0319949f..b8e994ce9f67 100644 --- a/infra/experimental/agent-skills/README.md +++ b/infra/experimental/agent-skills/README.md @@ -95,6 +95,7 @@ full flag listings. A summary of the available subcommands: | `fix-builds` | Diagnose and fix broken project builds | `python infra/experimental/agent-skills/helper.py fix-builds htslib` | | `fix-broken-builds` | Auto-discover broken projects from the public build status and fix up to `--limit` of them (C/C++ by default) | `python infra/experimental/agent-skills/helper.py fix-broken-builds --limit 50` | | `run-task` | Run an arbitrary `--task` string per project | `python infra/experimental/agent-skills/helper.py run-task --task "Add a harness for the XML attribute parser" open62541` | +| `coverage-pipeline` | Run the full coverage pipeline (skills-driven, prompt generated automatically) over named or `--broken`-discovered projects; `--cap N` sets the pair cap | `python infra/experimental/agent-skills/helper.py coverage-pipeline --broken --limit 10` | | `add-chronos-support` | Add Chronos support to a project | `python infra/experimental/agent-skills/helper.py add-chronos-support json-c` | | `integrate-project` | Onboard a new project from a Git URL | `python infra/experimental/agent-skills/helper.py integrate-project https://github.com/org/repo` | | `clean` | Remove local artifacts from previous agent runs | `python infra/experimental/agent-skills/helper.py clean open62541` | diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md index eeda887212f0..f5a2cece8b9e 100644 --- a/infra/experimental/agent-skills/USAGE.md +++ b/infra/experimental/agent-skills/USAGE.md @@ -134,7 +134,7 @@ Multiple projects: 2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. Target selection and harness shaping come from the architect skill `fuzzing-coverage-targeting`, producing a ranked multi-target portfolio with overlap-aware incremental worth (default cap 5 pairs per project, adjustable in your prompt). 3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. It never modifies upstream product code: upstream changes are limited to fuzzers with their corpora, dictionaries and options, fuzz build configuration and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions; product-code bugs become findings. 4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. A project may yield several pairs when worthwhile: fuzzer pairs, each carrying its harness plus that harness's corpora, dictionaries and options; one consolidated standalone corpus/dict/options pair (`-corpus-dict-options`); and, when the main build was broken, a `-build-fix` pair created first, whose changes later pairs copy in purely as a testing scaffold and remove again after conclusion. -5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory, or failing solely due to a marked real finding); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, nothing fuzzer-attributable within the first 60 seconds, no stale coverage in the final half); and a like-for-like coverage delta. Every post-build crash, leak, OOM, timeout or assertion is classified first: real upstream findings keep the pair delivered with the discovery marked, while only fuzzer faults consume the fix budget of 3 rounds, after which the pair is moved to the rejected area, never deleted. Nothing is ever newly silenced, pre-existing suppressions are honoured unchanged, and a timeout from genuinely heavy legitimate processing may receive a measured, light increase delivered as fuzzer options. +5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory, or failing solely due to a marked real finding); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, nothing fuzzer-attributable within the first 60 seconds, no stale coverage in the final half); and a coverage gain read from the runs' libFuzzer `cov:` metric (fast path; a coverage build only as fallback). Every post-build crash, leak, OOM, timeout or assertion is classified first: real upstream findings keep the pair delivered with the discovery marked, while only fuzzer faults consume the fix budget of 3 rounds, after which the pair is moved to the rejected area, never deleted. Nothing is ever newly silenced, pre-existing suppressions are honoured unchanged, and a timeout from genuinely heavy legitimate processing may receive a measured, light increase delivered as fuzzer options. Crashes, leaks, OOMs, timeouts and assertions are classified (fuzzer fault versus real finding); real findings are **documented with reproducers**, never hidden and never silenced. @@ -155,7 +155,8 @@ Equivalent commands for the manual steps: | Manual step (Approach A) | helper.py equivalent | |---|---| | Preview what will be asked | `python3 infra/experimental/agent-skills/helper.py show-prompt ` | -| Full pipeline on named projects | `python3 infra/experimental/agent-skills/helper.py run-task --task "Use the oss-fuzz-coverage-pipeline skill end to end: verify upstream viability, fix builds, triage coverage via fuzzing-coverage-targeting, deliver worktree duo pairs; never commit or push" proj1 proj2` | +| Full pipeline on named projects | `python3 infra/experimental/agent-skills/helper.py coverage-pipeline proj1 proj2` (the prompt is generated automatically; `--cap N` overrides the pair cap) | +| Discover broken projects and run the full pipeline | `python3 infra/experimental/agent-skills/helper.py coverage-pipeline --broken --limit 10` | | Fix known-broken builds | `python3 infra/experimental/agent-skills/helper.py fix-builds proj1 proj2` | | Discover and fix broken builds | `python3 infra/experimental/agent-skills/helper.py fix-broken-builds --language c,c++ --limit 20 -j 2` (add `--print-only` to preview the selection) | | Expand coverage on a project | `python3 infra/experimental/agent-skills/helper.py expand-oss-fuzz-projects proj` | @@ -167,16 +168,16 @@ A typical batch run: ```sh cd # 1. Dry run: print the exact prompt that would be sent. -python3 infra/experimental/agent-skills/helper.py show-prompt run-task --task "Use the oss-fuzz-coverage-pipeline skill end to end" htslib +python3 infra/experimental/agent-skills/helper.py show-prompt --task coverage-pipeline htslib # 2. Run it across the batch. -python3 infra/experimental/agent-skills/helper.py run-task --task "Use the oss-fuzz-coverage-pipeline skill end to end" htslib json-c +python3 infra/experimental/agent-skills/helper.py coverage-pipeline --cap 5 htslib json-c # 3. Watch progress; each session logs here. tail -f build/agent-logs/*.log ``` Session output lands in `build/agent-logs/-.log`, and each agent writes a per-project report under `projects//`; verification afterwards is exactly section 6. -Caveats: helper.py only drives the `claude` and `gemini` CLIs (Cline users stay with Approach A); sessions run with permissions skipped, so respect the threat model in README.md and use a sandbox; `fix-builds` and `expand-oss-fuzz-projects` use their own built-in workflows, which are narrower than the full pipeline (for the complete duo-pair protocol use `run-task` with an explicit task string naming the skill); and parallelism is bounded by Docker capacity, so keep `-j 2` unless your machine can take more concurrent builds. +Caveats: helper.py only drives the `claude` and `gemini` CLIs (Cline users stay with Approach A); sessions run with permissions skipped, so respect the threat model in README.md and use a sandbox; `fix-builds` and `expand-oss-fuzz-projects` use their own built-in workflows, which are narrower than the full pipeline (`coverage-pipeline` is the full-protocol command, and `run-task` remains for free-form work); and parallelism is bounded by Docker capacity, so keep `-j 2` unless your machine can take more concurrent builds. ## 6. Verify the results @@ -189,7 +190,7 @@ Caveats: helper.py only drives the `claude` and `gemini` CLIs (Cline users stay | Duo pairs | `/worktrees/oss-fuzz/` and `worktrees/upstream/` | Both branches exist (`oss-fuzz--` and `-`), each holding plain working-tree changes | | Git safety | both worktrees | Nothing staged (unless justified and recorded), nothing committed, nothing pushed, anywhere | | Bridge | OSS-Fuzz worktree Dockerfile | Tagged `OSS-FUZZ-AGENT-BRIDGE:BEGIN/END` blocks plus the marked staging folder, kept intact for your manual removal later | -| Validation evidence | per-project report | The declared sanitizer/engine matrix built; per-fuzzer `check_build` passes (or fails solely due to a marked real finding); quality-gate run logs (120 s minimum, nothing fuzzer-attributable, no staleness); a like-for-like coverage-delta table | +| Validation evidence | per-project report | The declared sanitizer/engine matrix built; per-fuzzer `check_build` passes (or fails solely due to a marked real finding); quality-gate run logs (120 s minimum, nothing fuzzer-attributable, no staleness); fast-path `cov:` deltas (coverage build only as fallback) | Handy checks: diff --git a/infra/experimental/agent-skills/helper.py b/infra/experimental/agent-skills/helper.py index 9787aa6605fd..b67dbe6f867f 100644 --- a/infra/experimental/agent-skills/helper.py +++ b/infra/experimental/agent-skills/helper.py @@ -29,6 +29,9 @@ python infra/experimental/agent-skills/helper.py add-chronos-support \ open62541 json-c htslib + python infra/experimental/agent-skills/helper.py coverage-pipeline \ + open62541 json-c + This will launch parallel agent sessions to expand fuzzing coverage, fix broken builds, add Chronos support, or carry out an arbitrary task for each listed project, producing local changes and a per-project report. @@ -36,6 +39,7 @@ import argparse import concurrent.futures +import fnmatch import glob import json import os @@ -687,6 +691,52 @@ Once the work is done you should exit the process. Do NOT commit or push anything. Leave all changes locally for the security engineer to review. """) +PIPELINE_PROMPT_TEMPLATE = textwrap.dedent("""\ + You are an OSS-Fuzz engineer running the full coverage pipeline for the + **{project}** OSS-Fuzz project. + + ## Important: use your skills + + You MUST use the following skills throughout this task: + {fuzzing_skills_blurb} + + ## Objective + + Run the **oss-fuzz-coverage-pipeline** skill end to end for **{project}**: + verify upstream viability from the Dockerfile, fix the builds, triage live + coverage via **fuzzing-coverage-targeting** (ranked multi-target list and + coverage ledger), write or fix harnesses via the language-appropriate + skill, stage the work as worktree branch duo pairs with the temporary + copy bridge, and validate each pair through the crash classification + procedure. Portfolio cap: at most {cap} pairs for this project. + + ## Hard guardrails (from the skill; they are binding) + + - Never commit, never push; no staging by default. + - Never modify upstream product code (fuzzers, fuzz build configuration + and FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION-guarded additions only). + - Every build, check, run and measurement goes through + python3 infra/helper.py. + - Classify every post-build crash, leak, OOM and timeout; never + introduce new silencing. + - Re-read all project data fresh; trust nothing from previous runs. + - Validate each pair on the sanitizer/engine matrix from project.yaml, + and evaluate coverage from the fuzzer runs' libFuzzer cov: metric + (fast path), using a coverage build only as fallback. + + ## Working directory and outputs + + You are working from: `{oss_fuzz_root}` + The project directory is: `{oss_fuzz_root}/projects/{project}` + Write the per-project report to: `{report_path}` + Record the run manifest at + `{oss_fuzz_root}/build/coverage-pipeline/manifest.md` and the final + summary per the skill's worktree delivery reference. + + Once the work is done you should exit the process. Do NOT commit or push + anything. Leave all changes locally for the security engineer to review. + """) + EXPAND_SUMMARY_PROMPT_TEMPLATE = textwrap.dedent("""\ You are an OSS-Fuzz engineer tasked with writing a consolidated summary of a multi-round fuzzing expansion that was just completed for the @@ -992,16 +1042,19 @@ def build_prompt(task, task_description=None, round_num=1, total_rounds=1, - expansion_size='medium'): + expansion_size='medium', + pipeline_cap=5): """Build the agent prompt for a given task and project. Args: - task: One of 'expand', 'fix-build', 'free-task', 'add-chronos'. + task: One of 'expand', 'fix-build', 'free-task', 'add-chronos', + 'coverage-pipeline'. project: OSS-Fuzz project name. task_description: Free-form task description (free-task only). round_num: Current expansion round (1-based; expand only). total_rounds: Total number of rounds planned (expand only). expansion_size: 'small', 'medium', or 'large' (expand only). + pipeline_cap: Portfolio cap in pairs (coverage-pipeline only). """ if task == 'expand': if total_rounds > 1: @@ -1018,6 +1071,9 @@ def build_prompt(task, elif task == 'add-chronos': report_name = 'chronos_report.md' template = ADD_CHRONOS_PROMPT_TEMPLATE + elif task == 'coverage-pipeline': + report_name = 'coverage_pipeline_report.md' + template = PIPELINE_PROMPT_TEMPLATE else: raise ValueError(f'Unknown task: {task}') @@ -1028,6 +1084,8 @@ def build_prompt(task, report_path=report_path, fuzzing_skills_blurb=FUZZING_SKILLS_BLURB, ) + if task == 'coverage-pipeline': + fmt_kwargs['cap'] = pipeline_cap if task == 'expand': fmt_kwargs['expansion_size_guidance'] = EXPAND_SIZE_GUIDANCE.get( expansion_size, EXPAND_SIZE_GUIDANCE['medium']) @@ -1060,14 +1118,16 @@ def print_prompt(task, task_description=None, round_num=1, total_rounds=1, - expansion_size='medium'): + expansion_size='medium', + pipeline_cap=5): """Print the prompt that would be sent to an agent.""" prompt, _ = build_prompt(task, project, task_description=task_description, round_num=round_num, total_rounds=total_rounds, - expansion_size=expansion_size) + expansion_size=expansion_size, + pipeline_cap=pipeline_cap) label = f'{task}' if task == 'expand' and total_rounds > 1: label += f' round {round_num}/{total_rounds}' @@ -1082,7 +1142,8 @@ def launch_agent_session(agent_cli, task_description=None, round_num=1, total_rounds=1, - expansion_size='medium'): + expansion_size='medium', + pipeline_cap=5): """Launch an agent session for a single project. Returns a subprocess.Popen object. @@ -1092,7 +1153,8 @@ def launch_agent_session(agent_cli, task_description=task_description, round_num=round_num, total_rounds=total_rounds, - expansion_size=expansion_size) + expansion_size=expansion_size, + pipeline_cap=pipeline_cap) round_label = (f' round {round_num}/{total_rounds}' if task == 'expand' and total_rounds > 1 else '') @@ -1302,7 +1364,8 @@ def _run_single_session(agent_cli, task_description=None, round_num=1, total_rounds=1, - expansion_size='medium'): + expansion_size='medium', + pipeline_cap=5): """Launch an agent session and wait for it to complete. Returns a dict with the project name, return code, and log path. @@ -1313,7 +1376,8 @@ def _run_single_session(agent_cli, task_description=task_description, round_num=round_num, total_rounds=total_rounds, - expansion_size=expansion_size) + expansion_size=expansion_size, + pipeline_cap=pipeline_cap) if proc is None: return {'project': project, 'returncode': -1, 'log_path': None} @@ -1334,11 +1398,14 @@ def _run_sessions(task, args): _validate_projects(projects) task_description = getattr(args, 'task_description', None) - _, report_name = build_prompt(task, projects[0], task_description) + pipeline_cap = getattr(args, 'cap', 5) + _, report_name = build_prompt(task, projects[0], task_description, + pipeline_cap=pipeline_cap) if args.print_only: for project in projects: - print_prompt(task, project, task_description) + print_prompt(task, project, task_description, + pipeline_cap=pipeline_cap) return agent_cli = args.agent or find_agent_cli() @@ -1365,7 +1432,8 @@ def _run_sessions(task, args): with concurrent.futures.ThreadPoolExecutor( max_workers=max_parallel) as executor: futures = { - executor.submit(_run_single_session, agent_cli, task, project, task_description): + executor.submit(_run_single_session, agent_cli, task, project, + task_description, 1, 1, 'medium', pipeline_cap): project for project in projects } for future in concurrent.futures.as_completed(futures): @@ -1641,6 +1709,53 @@ def cmd_fix_broken_builds(args): _run_sessions('fix-build', args) +def cmd_coverage_pipeline(args): + """Handle the coverage-pipeline subcommand. + + Runs the full coverage pipeline (oss-fuzz-coverage-pipeline skill) over + the named projects, or over projects discovered with --broken. The prompt + is generated automatically; no task string is needed. + """ + projects = list(args.projects) + + if args.broken: + languages = [ + lang.strip() for lang in args.language.split(',') if lang.strip() + ] + if not languages: + print('[!] --language must name at least one language.', file=sys.stderr) + sys.exit(1) + limit = args.limit if args.limit else 50 + selected, total_matching = discover_broken_projects(languages, limit) + print(f'[*] Found {total_matching} broken project(s) ' + f'for language(s): {", ".join(languages)}') + if not selected: + print('[*] Nothing to run.') + return + if len(selected) < total_matching: + print(f'[*] Taking the first {len(selected)} (of {total_matching}); ' + f'raise --limit to do more.') + projects.extend(p for p in selected if p not in projects) + + if args.exclude: + patterns = [p.strip() for p in args.exclude.split(',') if p.strip()] + projects = [ + p for p in projects + if not any(fnmatch.fnmatch(p, pat) for pat in patterns) + ] + + if not projects: + print('[!] No projects to run on (name projects or use --broken).', + file=sys.stderr) + sys.exit(1) + + if args.limit and not args.broken: + projects = projects[:args.limit] + + args.projects = projects + _run_sessions('coverage-pipeline', args) + + def cmd_run_task(args): """Handle the run-task subcommand.""" _run_sessions('free-task', args) @@ -1680,7 +1795,8 @@ def cmd_show_prompt(args): task_description=task_description, round_num=round_num, total_rounds=total_rounds, - expansion_size=expansion_size) + expansion_size=expansion_size, + pipeline_cap=getattr(args, 'cap', 5)) def main(): @@ -1723,6 +1839,17 @@ def main(): # Preview which projects would be fixed (no agent launched): python %(prog)s fix-broken-builds --print-only + # Run the full coverage pipeline over two projects + # (the prompt is generated automatically): + python %(prog)s coverage-pipeline open62541 json-c + + # Auto-discover up to 10 broken C/C++ projects and run the + # full pipeline on each: + python %(prog)s coverage-pipeline --broken --limit 10 + + # Preview the generated pipeline prompt without an agent: + python %(prog)s coverage-pipeline --print-only open62541 + # Run a free-form task on one or more projects: python %(prog)s run-task \\ --task-description "Investigate why the XML parser harness \\ @@ -1921,6 +2048,76 @@ def main(): ) fix_broken_parser.set_defaults(func=cmd_fix_broken_builds) + # coverage-pipeline (full pipeline, prompt generated automatically) + pipeline_parser = subparsers.add_parser( + 'coverage-pipeline', + help='Run the full coverage pipeline (oss-fuzz-coverage-pipeline ' + 'skill) over named or auto-discovered projects; the prompt is ' + 'generated automatically.', + ) + pipeline_parser.add_argument( + 'projects', + nargs='*', + help='One or more OSS-Fuzz project names (or none, with --broken).', + ) + pipeline_parser.add_argument( + '--broken', + action='store_true', + help='Discover broken projects from the public OSS-Fuzz build status ' + '(same discovery as fix-broken-builds) instead of naming projects.', + ) + pipeline_parser.add_argument( + '--language', + default=DEFAULT_FIX_LANGUAGES, + metavar='LANGS', + help='Comma-separated project languages to include with --broken ' + f'(default: "{DEFAULT_FIX_LANGUAGES}").', + ) + pipeline_parser.add_argument( + '--limit', + type=int, + default=0, + metavar='N', + help='Maximum number of projects to run on (default: no limit for ' + 'named projects, 50 with --broken).', + ) + pipeline_parser.add_argument( + '--exclude', + default='', + metavar='GLOBS', + help='Comma-separated project-name globs to exclude.', + ) + pipeline_parser.add_argument( + '--cap', + type=int, + default=5, + metavar='N', + help='Portfolio cap: maximum pairs per project, passed into the ' + 'generated prompt (default: 5).', + ) + pipeline_parser.add_argument( + '--agent', + choices=SUPPORTED_AGENTS, + default=None, + help='Agent CLI to use (default: auto-detect).', + ) + pipeline_parser.add_argument( + '--print-only', + action='store_true', + help='Print the prompts (and the discovered project list) without ' + 'launching agent sessions.', + ) + pipeline_parser.add_argument( + '-j', + '--max-parallel', + type=int, + default=DEFAULT_MAX_PARALLEL, + metavar='N', + help='Maximum number of agent sessions to run in parallel ' + f'(default: {DEFAULT_MAX_PARALLEL}).', + ) + pipeline_parser.set_defaults(func=cmd_coverage_pipeline) + # run-task run_task_parser = subparsers.add_parser( 'run-task', @@ -2002,7 +2199,8 @@ def main(): help='OSS-Fuzz project names.') show_parser.add_argument( '--task', - choices=['expand', 'fix-build', 'free-task', 'add-chronos'], + choices=['expand', 'fix-build', 'free-task', 'add-chronos', + 'coverage-pipeline'], default='expand', help='Which task prompt to show (default: expand).', ) diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index 684b90f82904..d98631ca9702 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -153,7 +153,7 @@ Validate each duo pair independently and one at a time, always through `infra/he 1. **Build matrix**: read `sanitizers:` and `fuzzing_engines:` from `projects//project.yaml` and validate every declared configuration (for example `--sanitizer address`, `undefined`, `memory`, and engines such as `--engine afl` where declared). 2. **Per-fuzzer gate (mandatory)**: `build_fuzzers` succeeds, and `check_build` passes for every new or updated fuzzer specifically, not just project-wide, **or its failure is solely attributable to a crash classified as a real finding** (documented with a reproducer and marked). 3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require **no crash, leak, OOM or timeout attributable to the fuzzer** (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. Every post-build crash goes through the classification procedure in the worktree reference: real findings keep the pair delivered with the discovery marked, fuzzer faults consume the fix budget, and a crash rooted in product code is always a finding, never a product-code patch. -4. **Coverage delta, like-for-like** (same flags, same corpus source, before versus after): targeted files gain covered lines and no project-own file digresses. Update the run's coverage ledger with the pair's measured coverage and compute its **incremental** contribution against what existing fuzzers and previously accepted pairs already cover: a pair whose gains mostly overlap is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded (rules from fuzzing-coverage-targeting). +4. **Coverage gain, fast path**: evaluate coverage from the libFuzzer `cov:` metric of the very runs the quality gate already performs (candidate versus baseline, same corpus source and duration; for a brand-new fuzzer, its absolute `cov:` on the standard seeds, noted as an approximation). Avoid a separate coverage build; fall back to `--sanitizer coverage` and per-file comparison only for marginal overlap decisions, ambiguous or flat results, or per-file numbers the report must show (see the worktree reference). Update the run's coverage ledger with the measured numbers and compute the pair's **incremental** contribution against what existing fuzzers and previously accepted pairs already cover: a pair whose gains mostly overlap is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded (rules from fuzzing-coverage-targeting). 5. **Fix budget of 3 rounds**: fix, retest, and at most three attempts per gate failure. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is **rejected**: move both worktrees into the rejected area with `git worktree move` (never delete), keep the branches, and record the failure reason, what was attempted and the new paths. 6. Deliver: remove any copied build-fix changes and verify the remaining diff is purely this pair's own work (see the worktree reference), finalise the per-project report and the manifest row, and add the row to the final summary table. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md index 1a7524ed5104..509d217eea22 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md @@ -101,8 +101,8 @@ Run pairs one at a time (builds of the same project share Docker image names), f # 1. Build the declared matrix from projects//project.yaml: python3 infra/helper.py build_fuzzers --sanitizer address python3 infra/helper.py build_fuzzers --sanitizer undefined # as declared -python3 infra/helper.py build_fuzzers --sanitizer coverage # plus engines where declared, for example: --engine afl +# (a --sanitizer coverage build is only needed on the fallback path below) # 2. Mandatory per-fuzzer gate, for every new or updated fuzzer: python3 infra/helper.py check_build @@ -121,12 +121,18 @@ Quality thresholds, which apply on top of `check_build` passing, and to corpora, These libFuzzer-style flags apply to every OSS-Fuzz engine: Jazzer, Atheris and Go-native fuzzing all wrap libFuzzer. -### Coverage delta, like-for-like +### Coverage evaluation: fast path by default -1. Baseline before any change: build the coverage configuration in a clean state and run `python3 infra/helper.py coverage `; copy the resulting `summary.json` to a scratch directory outside the build tree. -2. Candidate: identical flags and identical corpus source in the duo-pair worktree; save its `summary.json` the same way. -3. Compare per-file `lines.covered` counts on project-own paths: targeted files must gain covered lines, and no other project-own file may lose them. With an identical corpus, locally covered lines can only come from your change, which is what makes small deltas trustworthy. -4. Never compare a fresh local number with a production report; local short-run coverage is always lower (see `code_coverage.md` section 7 and `structured_seed_generation.md`). +Do not build a separate coverage configuration for routine pair evaluation. Use the fuzzing runs the quality gate already performs: + +1. Baseline: run the existing fuzzers on the pre-change build for the standard duration with the standard corpus source, and record the final libFuzzer `cov:` value (with `ft:` and `corp:` as supporting signals). +2. Candidate: the same runs on the pair's build, same corpus and duration; record the final `cov:`. +3. The pair's fast-path gain is candidate `cov:` versus baseline `cov:`. For a brand-new fuzzer with no baseline binary, the recorded gain is its absolute `cov:` on the standard seeds, noted as an approximation. +4. Like-for-like still applies: same instrumentation configuration (the normal ASan build already carries SanitizerCoverage, which is what `cov:` reports), same corpus source, same duration. Never compare fresh local numbers with production reports. + +The `cov:` trend between libFuzzer checkpoints is also the staleness signal, so one set of runs serves both the quality gate and the coverage check. + +**Fallback to a coverage build**, only when the fast path cannot answer: marginal overlap decisions between pairs, ambiguous or flat `cov:` results, or per-file gains the final report must show. Then build with `--sanitizer coverage` and compare per-file `lines.covered` counts on project-own paths (baseline from a clean state versus candidate in the pair worktree, identical flags and corpus source; targeted files must gain and no project-own file may lose), and use those numbers for the ledger decision. The overlap ledger records per-fuzzer `cov:` numbers by default and falls back to per-file data only when a decision is marginal. ### Crash classification (every post-build crash, leak, OOM, timeout or assertion) From fa3ea8fcd236189790c0bb7461875d11c4abe302 Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Mon, 26 Oct 2026 04:46:54 +0000 Subject: [PATCH 11/14] Update with clearer rules and instruction constraints Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/README.md | 2 +- infra/experimental/agent-skills/USAGE.md | 18 +- .../fuzzing-coverage-targeting/SKILL.md | 5 +- infra/experimental/agent-skills/helper.py | 276 +++++++++++++++++- .../oss-fuzz-coverage-pipeline/SKILL.md | 16 +- .../references/worktree_delivery.md | 30 +- 6 files changed, 306 insertions(+), 41 deletions(-) diff --git a/infra/experimental/agent-skills/README.md b/infra/experimental/agent-skills/README.md index b8e994ce9f67..2440959d4aed 100644 --- a/infra/experimental/agent-skills/README.md +++ b/infra/experimental/agent-skills/README.md @@ -95,7 +95,7 @@ full flag listings. A summary of the available subcommands: | `fix-builds` | Diagnose and fix broken project builds | `python infra/experimental/agent-skills/helper.py fix-builds htslib` | | `fix-broken-builds` | Auto-discover broken projects from the public build status and fix up to `--limit` of them (C/C++ by default) | `python infra/experimental/agent-skills/helper.py fix-broken-builds --limit 50` | | `run-task` | Run an arbitrary `--task` string per project | `python infra/experimental/agent-skills/helper.py run-task --task "Add a harness for the XML attribute parser" open62541` | -| `coverage-pipeline` | Run the full coverage pipeline (skills-driven, prompt generated automatically) over named or `--broken`-discovered projects; `--cap N` sets the pair cap | `python infra/experimental/agent-skills/helper.py coverage-pipeline --broken --limit 10` | +| `coverage-pipeline` | Run the full coverage pipeline (skills-driven, prompt generated automatically) over named or `--broken`-discovered projects; `--cap N` sets the pair cap; relay continuation sessions via `--max-sessions-per-project` and `--per-project-minutes` | `python infra/experimental/agent-skills/helper.py coverage-pipeline --broken --limit 10` | | `add-chronos-support` | Add Chronos support to a project | `python infra/experimental/agent-skills/helper.py add-chronos-support json-c` | | `integrate-project` | Onboard a new project from a Git URL | `python infra/experimental/agent-skills/helper.py integrate-project https://github.com/org/repo` | | `clean` | Remove local artifacts from previous agent runs | `python infra/experimental/agent-skills/helper.py clean open62541` | diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md index f5a2cece8b9e..e0d213a064ec 100644 --- a/infra/experimental/agent-skills/USAGE.md +++ b/infra/experimental/agent-skills/USAGE.md @@ -32,7 +32,7 @@ The skills are plain `SKILL.md` markdown, so they are model-agnostic and portabl The wrapper delegates all domain knowledge to those skills and their references: how to write harnesses in each language, how to drive `infra/helper.py`, how to read coverage reports and how to generate structured seeds. What the wrapper itself contributes is the process layer that none of the six contain: project selection and upstream viability checks, the phase discipline, the git guardrails, worktree duo pairs with the temporary copy bridge, the validation gates with the fix budget, the run manifest, rejection handling and the final summary. -The six knowledge skills remain fully usable on their own; the wrapper simply composes them into one end-to-end delivery pipeline, so nothing in them was duplicated or superseded. The architect skill `fuzzing-coverage-targeting` joins the composition in Phases 2, 3 and 5 to rank worthy targets and shape harnesses (default cap 5 pairs per project, adjustable in your prompt); the wrapper remains the only boss. +The six knowledge skills remain fully usable on their own; the wrapper simply composes them into one end-to-end delivery pipeline, so nothing in them was duplicated or superseded. The architect skill `fuzzing-coverage-targeting` joins the composition in Phases 2, 3 and 5 to rank worthy targets and shape harnesses (default cap 5 pairs per project, adjustable in your prompt); the wrapper remains the only boss. On any conflict, the pipeline skill's guardrails and rules override sub-skill guidance. ## 2. Prerequisites @@ -127,28 +127,31 @@ Multiple projects: > Use the oss-fuzz-coverage-pipeline skill. Phase 0: run scripts/select_projects.sh --broken --limit 5 'lib*', verify upstream viability for each, record the manifest, then run the full pipeline on each viable project, one at a time, resuming from the manifest between sessions. +Continuing after a session ends (Approach A): check `build/coverage-pipeline/manifest/.md`; if it records no terminal `STOP:` line, tell the agent: "Continue the portfolio loop from the manifest for <project> until a terminal stop." + ### 5.4 What the agent will do in either approach (Phases 0 to 5) -0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. The selector reports the language **declared** in project.yaml; the Dockerfile base image is never a language signal. All project data is reread fresh on every run, so a project whose project.yaml changed since an earlier run is treated according to the new values, never previous memory. Reruns after a previous batch of duo pairs likewise re-derive coverage and targets from fresh data, because the previous batch may have changed the picture. Record a run manifest at `build/coverage-pipeline/manifest.md`. +0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. The selector reports the language **declared** in project.yaml; the Dockerfile base image is never a language signal. All project data is reread fresh on every run, so a project whose project.yaml changed since an earlier run is treated according to the new values, never previous memory. Reruns after a previous batch of duo pairs likewise re-derive coverage and targets from fresh data, because the previous batch may have changed the picture. Record the per-project manifest fragment at `build/coverage-pipeline/manifest/.md`. 1. Fix builds through `infra/helper.py` only: the main fuzzer build fix is always mandatory; coverage and introspector build fixes are optional only when the failure cause is missing corpora or an OOM/timeout, with the exemption recorded. -2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. Target selection and harness shaping come from the architect skill `fuzzing-coverage-targeting`, producing a ranked multi-target portfolio with overlap-aware incremental worth (default cap 5 pairs per project, adjustable in your prompt). +2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. Target selection and harness shaping come from the architect skill `fuzzing-coverage-targeting`, producing a ranked multi-target portfolio with overlap-aware incremental worth. Each project's ranked list is then worked top-down until worth is exhausted (documented with numbers), the cap is reached (default 5 pairs per project, adjustable in your prompt) or the budget truly runs out; budget-deferred targets keep their written rationales as "not yet attempted", and a project with genuinely no worthy targets is marked `no-worthy-targets` rather than given a fabricated pair. 3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. It never modifies upstream product code: upstream changes are limited to fuzzers with their corpora, dictionaries and options, fuzz build configuration and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions; product-code bugs become findings. 4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. A project may yield several pairs when worthwhile: fuzzer pairs, each carrying its harness plus that harness's corpora, dictionaries and options; one consolidated standalone corpus/dict/options pair (`-corpus-dict-options`); and, when the main build was broken, a `-build-fix` pair created first, whose changes later pairs copy in purely as a testing scaffold and remove again after conclusion. -5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory, or failing solely due to a marked real finding); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, nothing fuzzer-attributable within the first 60 seconds, no stale coverage in the final half); and a coverage gain read from the runs' libFuzzer `cov:` metric (fast path; a coverage build only as fallback). Every post-build crash, leak, OOM, timeout or assertion is classified first: real upstream findings keep the pair delivered with the discovery marked, while only fuzzer faults consume the fix budget of 3 rounds, after which the pair is moved to the rejected area, never deleted. Nothing is ever newly silenced, pre-existing suppressions are honoured unchanged, and a timeout from genuinely heavy legitimate processing may receive a measured, light increase delivered as fuzzer options. +5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory, or failing solely due to a marked real finding); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, nothing fuzzer-attributable within the first 60 seconds, no stale coverage in the final half); and per-fuzzer coverage growth (start versus end `cov:` of the run, with `ft:` and corpus size; a coverage build only as the per-file fallback). Every post-build crash, leak, OOM, timeout or assertion is classified first: real upstream findings keep the pair delivered with the discovery marked, while only fuzzer faults consume the fix budget of 3 rounds, after which the pair is moved to the rejected area, never deleted. Nothing is ever newly silenced, pre-existing suppressions are honoured unchanged, and a timeout from genuinely heavy legitimate processing may receive a measured, light increase delivered as fuzzer options. Crashes, leaks, OOMs, timeouts and assertions are classified (fuzzer fault versus real finding); real findings are **documented with reproducers**, never hidden and never silenced. ### 5.5 The deliverables (either approach) - The list of surviving worktree branch duo pairs (possibly several per project, grouped by the Phase 4 rules: fuzzer pairs with their own corpora, dictionaries and options, one standalone corpus pair, and a build-fix pair when needed), with paths and base commit SHAs recorded. -- `summary.md` in the marked working area: one markdown table row per duo pair (project, language, duo branches, worktree paths, summary of work, coverage delta, status). +- Per-project summary fragments merged by the driver into `build/coverage-pipeline/summary.md`: one markdown table row per duo pair (project, language, duo branches, worktree paths, summary of work, coverage delta, status), plus a remaining (deferred) column so incomplete portfolios are visible at a glance. - Skipped or unprocessable projects in the same table with the concrete reason. - Rejected duo pairs moved into the rejected area and marked clearly, with the failure reason and what was attempted. +- Budget-deferred entries ("pair N: worth rationale written, not yet attempted") and `no-worthy-targets` projects, each with their evidence. - Per-project reports, the manifest, and the next actions for you. ### 5.6 Approach B: run the same work with helper.py (headless, Claude Code and Gemini CLI) -`helper.py` drives the same skills without an interactive session: it builds task-specific prompts that reference the skills (including `oss-fuzz-coverage-pipeline` and `fuzzing-coverage-targeting`), auto-detects `claude` or `gemini` on `PATH`, and launches one headless session per project, in parallel (default 2). The skills must be installed for the CLI it drives (`~/.claude/skills` or `~/.gemini/skills`; see section 3). +`helper.py` drives the same skills without an interactive session: it builds task-specific prompts that reference the skills (including `oss-fuzz-coverage-pipeline` and `fuzzing-coverage-targeting`), auto-detects `claude` or `gemini` on `PATH`, and launches one headless session per project, in parallel (default 2). The skills must be installed for the CLI it drives (`~/.claude/skills` or `~/.gemini/skills`; see section 3). For the full pipeline, `coverage-pipeline` runs a **relay**: after each session it checks the project's manifest fragment (`build/coverage-pipeline/manifest/.md`) for a terminal `STOP:` line and launches continuation sessions (up to `--max-sessions-per-project`, default 3; `--per-project-minutes` optionally caps each project's wall clock), so one project's portfolio is never limited by a single session's context. It also stops launching sessions when free disk drops below a safe threshold, recording a clean budget-exhausted (disk) stop with deferred entries, and it merges the per-project manifest and summary fragments into combined views after the run. Equivalent commands for the manual steps: @@ -185,7 +188,7 @@ Caveats: helper.py only drives the `claude` and `gemini` CLIs (Cline users stay | Artifact | Where | What "good" looks like | |---|---|---| -| Run manifest | `build/coverage-pipeline/manifest.md` (oss-fuzz root) | A row per project with language, status, duo branches, worktree paths, base SHAs and report link | +| Run manifest | per-project fragments under `build/coverage-pipeline/manifest/`, merged into `manifest.md` by the driver | A row per project with language, status, duo branches, worktree paths, base SHAs and report link | | Final summary | `/summary.md` | One table covering every selected project and every pair: delivered, rejected (moved to rejected area) and skipped, each with its reason where relevant, and a depends-on column recording pairs tested with a copied build fix | | Duo pairs | `/worktrees/oss-fuzz/` and `worktrees/upstream/` | Both branches exist (`oss-fuzz--` and `-`), each holding plain working-tree changes | | Git safety | both worktrees | Nothing staged (unless justified and recorded), nothing committed, nothing pushed, anywhere | @@ -241,6 +244,7 @@ After your changes land and OSS-Fuzz production rebuilds (usually within a day): | Project skipped as blocked(...) | That is by design: archived, inactive (24 months), moved and non-git upstreams always stop and notify; the reason is in the summary and manifest | | Pair keeps failing the gates | The fix budget is 3 rounds; afterwards the pair is rejected and moved to the rejected area, ready for a later attempt | | Fuzzer crashes but it looks like a genuine bug | Classify first: real findings keep the pair delivered and marked; only fuzzer faults use the fix budget; never silence leaks, OOMs or timeouts | +| Run stops for disk | By design: the driver stops relays cleanly below the free-disk threshold; the manifest records budget-exhausted (disk) with deferred entries | | Agent wants to patch upstream product code | Forbidden: upstream changes are limited to fuzzers, fuzz build configuration and macro-guarded additions; product-code bugs are findings | | Dependent pair retains some build-fix content | Entangled same-line edits; the residual fragment is flagged in the report and summary, and is safe because the fix branch commits first | | Selector `--broken` fails | It needs network access and curl; check that `status.json` is reachable | diff --git a/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md b/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md index c991bb8a5dd8..914f70890163 100644 --- a/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md +++ b/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md @@ -79,9 +79,10 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { 1. Enumerate dark regions from the inputs; classify each by the taxonomy; score each by the worth model. 2. Apply the security weighting: attacker-reachable surfaces rank first among equals; functional dark logic is fully worthy; diagnostics, logging, printf-style output, test helpers and build-time code stay excluded. -3. Produce a **ranked multi-target list** expected to yield several focused harnesses rather than one or two; a single-target list is a smell that the taxonomy was not applied. +3. Produce a **ranked multi-target list** expected to yield several focused harnesses rather than one or two; a single-target list is a smell that the taxonomy was not applied. If honest triage finds nothing worthy, say so: zero worthy targets is a valid outcome, and inventing a trivial pair to hit a one-pair floor is forbidden. The declaration must be proven, not asserted: list the top rejected regions with their numbers and the reason each fails the worth model. 4. Selection is iterative with diminishing returns, executed by the pipeline with its coverage ledger: after each accepted pair, re-rank the remaining targets by **incremental** worth (projected gains minus already-covered ground). A pair whose gains mostly overlap already-accepted coverage is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded. -5. **Default cap: at most 5 pairs per project** (fuzzer pairs plus the standalone corpus pair), unless the user raises or lowers the threshold in their prompt. Stop earlier when the remaining targets' projected incremental worth is marginal. +5. **Default cap: at most 5 pairs per project** (fuzzer pairs plus the standalone corpus pair; a `-build-fix` pair does not count against the cap), unless the user raises or lowers the threshold in their prompt. +6. **Work the ranked list top-down until the rules are exhausted.** Worth is measured per project and never compared across projects: in a multi-project batch, a second pair for project A competes only with A's own remaining dark regions, never with project B's candidates, and spreading one pair per project is an anti-pattern. A single delivered pair is only the floor for a project's delivered status, never a stopping point. Stop a project's loop only on one of: **(a) worth exhaustion**, the remaining marginal worth is genuinely low, documented with numbers; **(b) the cap**; or **(c) budget exhaustion**, when time or build budget truly runs out. Budget bounds ordering and depth only, never worth: record un-attempted worthy targets as "pair N of project X: worth rationale written, not yet attempted", never as a worth decision, and never drop a target as unworthy for budget reasons. ## Non-goals diff --git a/infra/experimental/agent-skills/helper.py b/infra/experimental/agent-skills/helper.py index b67dbe6f867f..e3321e236ccb 100644 --- a/infra/experimental/agent-skills/helper.py +++ b/infra/experimental/agent-skills/helper.py @@ -41,6 +41,7 @@ import concurrent.futures import fnmatch import glob +import re import json import os import shutil @@ -53,6 +54,9 @@ DEFAULT_MAX_PARALLEL = 2 +# Stop launching pipeline sessions when free disk drops below this. +MIN_FREE_DISK_GB = 20 + # Public OSS-Fuzz build status. Used by fix-broken-builds to auto-discover # projects whose most recent build failed. BUILD_STATUS_URL = ( @@ -705,10 +709,19 @@ Run the **oss-fuzz-coverage-pipeline** skill end to end for **{project}**: verify upstream viability from the Dockerfile, fix the builds, triage live coverage via **fuzzing-coverage-targeting** (ranked multi-target list and - coverage ledger), write or fix harnesses via the language-appropriate - skill, stage the work as worktree branch duo pairs with the temporary - copy bridge, and validate each pair through the crash classification - procedure. Portfolio cap: at most {cap} pairs for this project. + coverage ledger), then work the ranked target list **top-down, pair by + pair**, writing or fixing harnesses via the language-appropriate skill, + staging each pair as a worktree branch duo with the temporary copy bridge, + and validating it through the crash classification procedure. + + **One delivered pair is NOT completion.** Portfolio cap: at most {cap} + pairs for this project (improvement pairs; a `-build-fix` pair does not + count against the cap). Continue the loop until (a) the project's + remaining marginal worth is genuinely low (document the numbers), (b) the + cap is reached, or (c) your session budget (context or time) truly runs + out. Persist the full ranked target list (candidate pairs, suffixes, + rationales) in the manifest BEFORE starting the first pair, so later + sessions can continue from it. ## Hard guardrails (from the skill; they are binding) @@ -723,15 +736,91 @@ - Validate each pair on the sanitizer/engine matrix from project.yaml, and evaluate coverage from the fuzzer runs' libFuzzer cov: metric (fast path), using a coverage build only as fallback. + - Watch your context and time budget: when nearing the limit, checkpoint + cleanly (persist progress and deferred entries) instead of wrapping up + shallowly; a later session continues from the manifest. + - Where sub-skill guidance conflicts with these guardrails or the + pipeline skill, the pipeline wins. + - If disk space runs low, stop cleanly: record a budget-exhausted (disk) + stop with deferred entries; never continue on insufficient disk. + + ## Exit protocol (mandatory) + + Before exiting, update the manifest for **{project}**: the pairs + delivered so far, the remaining ranked targets with their worth + rationales, and a terminal stop line in exactly this format: + + STOP: {project} : + + Only exit without a STOP line if you truly ran out of context; in that + case still record progress and deferred entries ("pair N: worth + rationale written, not yet attempted") so the next session continues. ## Working directory and outputs You are working from: `{oss_fuzz_root}` The project directory is: `{oss_fuzz_root}/projects/{project}` Write the per-project report to: `{report_path}` - Record the run manifest at - `{oss_fuzz_root}/build/coverage-pipeline/manifest.md` and the final - summary per the skill's worktree delivery reference. + Record the per-project manifest fragment at + `{oss_fuzz_root}/build/coverage-pipeline/manifest/{project}.md` (one + authoritative file per project) and the per-project summary fragment at + `{oss_fuzz_root}/build/coverage-pipeline/summary/{project}.md`; the + batch driver merges combined views after the run. + + Once the work is done you should exit the process. Do NOT commit or push + anything. Leave all changes locally for the security engineer to review. + """) + +PIPELINE_CONTINUATION_PROMPT_TEMPLATE = textwrap.dedent("""\ + You are an OSS-Fuzz engineer CONTINUING the coverage pipeline for the + **{project}** OSS-Fuzz project; a previous session already made progress. + + ## Important: use your skills + + You MUST use the following skills throughout this task: + {fuzzing_skills_blurb} + + ## Objective + + First read the per-project manifest fragment at + `{oss_fuzz_root}/build/coverage-pipeline/manifest/{project}.md` and the + existing + report(s) under `{oss_fuzz_root}/projects/{project}/` to see exactly + where the previous session stopped (worktree duo pairs already created, + ledger state, remaining ranked targets). Then CONTINUE the + **oss-fuzz-coverage-pipeline** portfolio loop for **{project}** from that + point: work the remaining ranked targets top-down, pair by pair, with the + same validation and guardrails as a first session. Portfolio cap: at most + {cap} pairs for this project in total across sessions (improvement + pairs; a `-build-fix` pair does not count against the cap). + + Re-read all project data fresh per the skill's rules, but do not redo + work already delivered: existing delivered pairs and their worktrees are + carried forward as recorded in the manifest. + + ## Hard guardrails + + Identical to a first session: never commit, never push, no staging by + default; upstream product code off-limits; everything through + python3 infra/helper.py; classify every post-build crash; never introduce + new silencing; the cov: fast path for coverage evaluation; and when your + context or time runs low, checkpoint cleanly instead of wrapping up. + + ## Exit protocol (mandatory) + + Before exiting, update the manifest for **{project}**: the pairs + delivered so far (all sessions), the remaining ranked targets with their + worth rationales, and a terminal stop line in exactly this format: + + STOP: {project} : + + ## Working directory and outputs + + You are working from: `{oss_fuzz_root}` + The project directory is: `{oss_fuzz_root}/projects/{project}` + Update the per-project report at: `{report_path}` + and the manifest fragment at + `{oss_fuzz_root}/build/coverage-pipeline/manifest/{project}.md`. Once the work is done you should exit the process. Do NOT commit or push anything. Leave all changes locally for the security engineer to review. @@ -1074,6 +1163,9 @@ def build_prompt(task, elif task == 'coverage-pipeline': report_name = 'coverage_pipeline_report.md' template = PIPELINE_PROMPT_TEMPLATE + elif task == 'coverage-pipeline-continue': + report_name = 'coverage_pipeline_report.md' + template = PIPELINE_CONTINUATION_PROMPT_TEMPLATE else: raise ValueError(f'Unknown task: {task}') @@ -1084,7 +1176,7 @@ def build_prompt(task, report_path=report_path, fuzzing_skills_blurb=FUZZING_SKILLS_BLURB, ) - if task == 'coverage-pipeline': + if task in ('coverage-pipeline', 'coverage-pipeline-continue'): fmt_kwargs['cap'] = pipeline_cap if task == 'expand': fmt_kwargs['expansion_size_guidance'] = EXPAND_SIZE_GUIDANCE.get( @@ -1709,6 +1801,99 @@ def cmd_fix_broken_builds(args): _run_sessions('fix-build', args) +def _pipeline_stop_reached(project): + """True if a terminal STOP line is recorded for the project. + + Checks the per-project manifest fragment first, then a legacy single-file + manifest. Matching is tolerant: markdown emphasis, case and colon spacing + may vary from the requested exact format. + """ + manifest_dir = os.path.join(OSS_FUZZ_ROOT, 'build', 'coverage-pipeline', + 'manifest') + candidates = [ + os.path.join(manifest_dir, f'{project}.md'), + os.path.join(OSS_FUZZ_ROOT, 'build', 'coverage-pipeline', + 'manifest.md'), + ] + pattern = re.compile( + rf'^\s*\**\s*stop\s*:\s*{re.escape(project)}\s*:', re.IGNORECASE) + for manifest in candidates: + if not os.path.isfile(manifest): + continue + try: + with open(manifest, encoding='utf-8') as f: + content = f.read() + except OSError: + continue + for line in content.splitlines(): + if pattern.match(line): + return True + return False + + +def _disk_space_ok(): + """True when there is enough free disk to keep launching sessions.""" + try: + usage = shutil.disk_usage(OSS_FUZZ_ROOT) + except OSError: + return True + return usage.free >= MIN_FREE_DISK_GB * 1024 ** 3 + + +def _collate_fragments(fragment_dir, merged_path, header): + """Merge per-project fragment files into one deterministic view.""" + if not os.path.isdir(fragment_dir): + return + names = sorted(n for n in os.listdir(fragment_dir) if n.endswith('.md')) + if not names: + return + os.makedirs(os.path.dirname(merged_path), exist_ok=True) + with open(merged_path, 'w', encoding='utf-8') as out: + out.write(header + '\n\n') + for name in names: + with open(os.path.join(fragment_dir, name), encoding='utf-8') as f: + out.write(f.read().rstrip() + '\n\n') + + +def _run_pipeline_relay(agent_cli, args, project): + """Run coverage-pipeline sessions for one project until a terminal stop. + + Sessions run sequentially per project (the relay); projects run in + parallel via the executor. A session that exits without recording a + terminal STOP line is treated as a checkpoint, and the next session + continues from the manifest, up to --max-sessions-per-project. + """ + max_sessions = args.max_sessions_per_project + deadline_minutes = args.per_project_minutes # 0 means no limit. + started = datetime.now() + for session_num in range(1, max_sessions + 1): + if not _disk_space_ok(): + print(f'[!] {project}: insufficient disk space; stopping the relay ' + 'cleanly. The session should record a budget-exhausted (disk) ' + 'stop with deferred entries if it runs at all.') + return {'project': project, 'returncode': 0, 'log_path': None} + task = ('coverage-pipeline' if session_num == 1 + else 'coverage-pipeline-continue') + print(f'[*] {project}: launching session {session_num}/{max_sessions} ' + f'({task})') + result = _run_single_session(agent_cli, task, project, None, 1, 1, + 'medium', args.cap) + if _pipeline_stop_reached(project): + print(f'[*] {project}: terminal stop recorded; relay complete.') + return result + if (result['returncode'] != 0 and session_num == 1): + print(f'[*] {project}: first session failed; not relaying.') + return result + if (deadline_minutes and + (datetime.now() - started).total_seconds() > deadline_minutes * 60): + print(f'[*] {project}: per-project time budget reached; stopping the ' + 'relay (remaining targets should be recorded as deferred).') + return result + print(f'[*] {project}: session cap reached without a terminal stop; ' + 'remaining targets should be recorded as deferred.') + return result + + def cmd_coverage_pipeline(args): """Handle the coverage-pipeline subcommand. @@ -1753,7 +1938,59 @@ def cmd_coverage_pipeline(args): projects = projects[:args.limit] args.projects = projects - _run_sessions('coverage-pipeline', args) + + if args.print_only: + for project in projects: + print_prompt('coverage-pipeline', project, pipeline_cap=args.cap) + print_prompt('coverage-pipeline-continue', project, + pipeline_cap=args.cap) + return + + agent_cli = args.agent or find_agent_cli() + if agent_cli is None: + print( + '[!] No supported agent CLI found on PATH ' + f'({", ".join(SUPPORTED_AGENTS)}).\n' + ' Install one or use --print-only to see the prompts.', + file=sys.stderr) + sys.exit(1) + + print(f'[*] Using agent CLI: {agent_cli}') + print(f'[*] Task: coverage-pipeline (relay, up to ' + f'{args.max_sessions_per_project} session(s) per project)') + print(f'[*] Projects: {", ".join(projects)}') + print(f'[*] Max parallel projects: {args.max_parallel}') + print() + + with concurrent.futures.ThreadPoolExecutor( + max_workers=args.max_parallel) as executor: + futures = { + executor.submit(_run_pipeline_relay, agent_cli, args, project): + project for project in projects + } + for future in concurrent.futures.as_completed(futures): + future.result() + + print('\n[*] All relays complete. Check the following for results:') + for project in projects: + report = os.path.join(OSS_FUZZ_ROOT, 'projects', project, + 'coverage_pipeline_report.md') + exists = 'EXISTS' if os.path.isfile(report) else 'MISSING' + print(f' - projects/{project}/coverage_pipeline_report.md [{exists}]') + + _collate_fragments( + os.path.join(OSS_FUZZ_ROOT, 'build', 'coverage-pipeline', 'manifest'), + os.path.join(OSS_FUZZ_ROOT, 'build', 'coverage-pipeline', 'manifest.md'), + '# Merged run manifest (generated from per-project fragments)') + _collate_fragments( + os.path.join(OSS_FUZZ_ROOT, 'build', 'coverage-pipeline', 'summary'), + os.path.join(OSS_FUZZ_ROOT, 'build', 'coverage-pipeline', 'summary.md'), + '# Merged run summary (generated from per-project fragments)') + print('[*] Merged manifest and summary views written under ' + 'build/coverage-pipeline/.') + + print('\n[*] Review local changes with:') + print(f' cd {OSS_FUZZ_ROOT} && git status && git worktree list') def cmd_run_task(args): @@ -1847,6 +2084,10 @@ def main(): # full pipeline on each: python %(prog)s coverage-pipeline --broken --limit 10 + # Relay: up to 5 sessions per project, capped at 4 hours each: + python %(prog)s coverage-pipeline --max-sessions-per-project 5 \\ + --per-project-minutes 240 open62541 + # Preview the generated pipeline prompt without an agent: python %(prog)s coverage-pipeline --print-only open62541 @@ -2095,6 +2336,23 @@ def main(): help='Portfolio cap: maximum pairs per project, passed into the ' 'generated prompt (default: 5).', ) + pipeline_parser.add_argument( + '--max-sessions-per-project', + type=int, + default=3, + metavar='N', + help='Relay: maximum agent sessions per project; a continuation ' + 'session is launched while the manifest shows no terminal STOP line ' + '(default: 3).', + ) + pipeline_parser.add_argument( + '--per-project-minutes', + type=int, + default=0, + metavar='N', + help='Relay: stop launching sessions for a project after this many ' + 'minutes of sessions (default: 0, no limit).', + ) pipeline_parser.add_argument( '--agent', choices=SUPPORTED_AGENTS, diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index d98631ca9702..76e31b1d6212 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -35,6 +35,7 @@ plus their references (`code_coverage.md`, `structured_seed_generation.md`, `hel 5. **Fresh data on every run**: whenever this skill determines any project data, treat it as a new check and never rely on memory from previous runs. Reread `project.yaml` (language, sanitizers, engines, `main_repo`), the Dockerfile, the build status, the coverage report pointers and the upstream state on every run. If a project's language changed since an earlier run (for example C++ to Rust), the new run must see the new value and select the sub-skill accordingly. The manifest carries process state (what is done and pending), not world facts: when a fresh read contradicts something recorded earlier, the fresh read wins and the manifest row is corrected. This matters most on reruns: a project may have been updated by a previous batch of duo pairs (merged and since picked up by production), so earlier coverage data, target lists and assumptions no longer hold. A second run re-derives everything from fresh data and treats earlier findings, including which code was dark or already covered, as history rather than truth. 6. **Upstream product code is off-limits**: never modify the upstream project's own source code. Bugs in it are findings (documented, filed by the user), never fixes by the agent. The only permitted upstream changes are: fuzzers (harnesses and their corpora, dictionaries and options), fuzz build configuration (wiring the targets into the build), and additions guarded by the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro, which is the sole sanctioned product-code edit (the macro is set in OSS-Fuzz builds and is the standard way to make behaviour fuzz-friendly without affecting production). If product code gets modified by accident, stop immediately and notify, never quietly undo. The OSS-Fuzz repo side (Dockerfile, `build.sh`, `project.yaml`) is not affected by this rule. 7. **Classify every post-build crash; never silence**: a crash, memleak, OOM or timeout after a successful build, whether it surfaces in `check_build` or in the fuzzing runs, is never an automatic failure: classify it first (procedure in the worktree reference). Fuzzer faults consume the fix budget; real findings keep the pair delivered with the discovery marked and consume nothing. Never introduce new silencing (no leak-detection muting, no suppression flags or files, no raised limits to dodge a failure). Pre-existing suppressions in the existing integration or existing fuzzers are honoured unchanged: carried over for new and fixed fuzzers, neither removed nor widened, and only what they already cover is out of scope. +8. **Precedence**: these guardrails and this skill's rules override any conflicting guidance in the sub-skills; the sub-skills supply craft and knowledge within these bounds. ## Deliverables @@ -68,14 +69,14 @@ Before doing any real work on a selected project, verify the upstream is viable. - repository moved or abandoned (moved redirects, "we moved" banners, homepage pointing elsewhere while the integration still points at the dead location): stop and notify, `blocked(upstream-moved)`, noting the new location if it is discoverable, but never chase it automatically; - source is not a git repository (tarball or svn): stop and notify, `blocked(upstream-not-git)`. -Stopping is mandatory in every case; it is never agent judgement. Then create (or resume) the run manifest at `build/coverage-pipeline/manifest.md` in the OSS-Fuzz root: +Stopping is mandatory in every case; it is never agent judgement. Then create (or resume) the per-project manifest fragment at `build/coverage-pipeline/manifest/.md` (one authoritative file per project; the batch driver merges a combined `manifest.md` view after the run; a legacy single-file manifest is history): | project | language | status | duo branches | worktree paths | base SHAs | report | notes | |---|---|---|---|---|---|---|---| -Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. When a project yields several pairs, give each pair its own manifest row (same project, its own suffix and duo branches), with a `tested-with: -build-fix` note in the notes column when it depends on the build fix. The language column starts as the declared language; after a mid-run language switch (see Phase 2), record it as `declared(X)/actual(Y)`. A rerun after a previous delivered batch starts fresh rows (or a new dated section) and treats earlier rows as history only; previously rejected pairs may be deliberately retried against the new state. +Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, `no-worthy-targets`, `deferred(budget)`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. When a project yields several pairs, give each pair its own manifest row (same project, its own suffix and duo branches), with a `tested-with: -build-fix` note in the notes column when it depends on the build fix. The language column starts as the declared language; after a mid-run language switch (see Phase 2), record it as `declared(X)/actual(Y)`. A rerun after a previous delivered batch starts fresh rows (or a new dated section) and treats earlier rows as history only; previously rejected pairs may be deliberately retried against the new state. Sessions exit via the exit protocol: before exiting, a session updates the manifest with the pairs delivered, the remaining ranked targets with rationales, and a terminal stop line `STOP: : `; running out of context means checkpointing progress and deferred entries, never a shallow wrap-up. Declaring `no-worthy-targets` requires genuine judgement and proof: a per-region worth analysis with the top rejected candidate regions listed, their numbers, and why each fails the worth model; it is never a quick early exit and is reviewed as seriously as a failure. -Multi-project discipline: process **one project at a time** (builds of the same project share Docker image names). With 1,300+ projects in OSS-Fuzz (~600 C/C++), prefer prioritised subsets over `*`: broken builds first (fast wins that also unlock coverage data), then the lowest own-code coverage on security-sensitive surface. +Multi-project discipline: **serialise only real `infra/helper.py` executions** (Docker builds, checks and runs: one project at a time, since image names collide). Analysis (triage, coverage study) and worktree/fuzzer generation may proceed in parallel across projects and pairs. With 1,300+ projects in OSS-Fuzz (~600 C/C++), prefer prioritised subsets over `*`: broken builds first (fast wins that also unlock coverage data), then the lowest own-code coverage on security-sensitive surface. ### Phase 1: Builds, mandatory and optional fixes @@ -104,7 +105,7 @@ Canonical live coverage references (resolve `YYYYMMDD` with the walk-back rule b **Date walk-back**: coverage and introspector builds sometimes fail (cross-check the build status on introspector.oss-fuzz.com), so the latest date may have no report. Resolve the date by walking backwards day by day, at most 30 days, until a report exists, and record the date actually used. -Produce a written target list via **fuzzing-coverage-targeting** (worth model, dark-region taxonomy, ranked multi-target list) and initialise the run's coverage ledger from the per-target baseline reports (`reports-by-target`); do not improvise targets later. On a rerun of a project previously worked on, do not reuse the earlier target list: re-derive it from the fresh reports, because the previous batch of pairs may have shifted coverage (previously dark targets may now be covered and must be dropped; still-dark ones are re-attempted). +Produce a written target list via **fuzzing-coverage-targeting** (worth model, dark-region taxonomy, ranked multi-target list) and initialise the run's coverage ledger from the per-target baseline reports (`reports-by-target`); do not improvise targets later. **Plan first, on disk**: persist the complete ranked list (every candidate pair with its suffix and worth rationale) into the manifest before any Phase 3 or 4 work, so continuation sessions and deferral recording always have the full picture. On a rerun of a project previously worked on, do not reuse the earlier target list: re-derive it from the fresh reports, because the previous batch of pairs may have shifted coverage (previously dark targets may now be covered and must be dropped; still-dark ones are re-attempted). **Verify the declared language while analysing**: while reading the source tree, build scripts and existing harnesses, check that the declared language (project.yaml) matches reality. If it does not, switch the active language sub-skill to the matching one and **continue in all cases**, even when the actual language falls outside the run's `--language` filter, recording the discrepancy: manifest `declared(X)/actual(Y)`, evidence in the report, actual language in the summary with the declared one noted. For mixed-language projects, the project-level language stays the corrected declared one, but a pair whose harness targets a module in a different language may activate that language's skill for that pair, noted in the pair's row. @@ -139,7 +140,7 @@ Follow [references/worktree_delivery.md](references/worktree_delivery.md) precis **Grouping rules, one project and several pairs:** -- A single delivered pair is the minimum bar, but when worthwhile, develop **multiple pairs per project**: lower-priority targets from the Phase 2 list that still carry worth, in your own judgement backed by the written rationale, are developed in the same run, each as its own pair. +- Work each project's ranked target list **top-down until the rules are exhausted**: multiple pairs per project are the norm whenever worth remains, up to the cap. A single delivered pair is the floor for the project's delivered status, never a stopping point, and worth is measured per project, never compared across projects (one pair per project is a batch anti-pattern). Stop only on worth exhaustion (remaining marginal worth genuinely low, documented with numbers), the 5-pair cap (default, prompt-adjustable; improvement pairs, with a `-build-fix` pair not counting against it), or genuine budget exhaustion; budget-deferred targets are recorded as "pair N of project X: worth rationale written, not yet attempted", never as worth decisions. A project with genuinely no worthy targets is marked `no-worthy-targets` with its triage evidence; never invent a trivial pair to meet the floor. - A pair that fixes or adds a fuzzer **carries that fuzzer's corpora, dictionaries and options in the same pair**: a harness never ships separately from its own seeds and settings. - All remaining corpora, dictionaries and options changes with no fuzzer change are consolidated into a **single standalone pair** with the suffix `-corpus-dict-options`. - A general build fix not tied to one fuzzer gets its own pair with the suffix `-build-fix`; build-configuration changes a specific fuzzer needs ride inside that fuzzer's pair. @@ -153,11 +154,12 @@ Validate each duo pair independently and one at a time, always through `infra/he 1. **Build matrix**: read `sanitizers:` and `fuzzing_engines:` from `projects//project.yaml` and validate every declared configuration (for example `--sanitizer address`, `undefined`, `memory`, and engines such as `--engine afl` where declared). 2. **Per-fuzzer gate (mandatory)**: `build_fuzzers` succeeds, and `check_build` passes for every new or updated fuzzer specifically, not just project-wide, **or its failure is solely attributable to a crash classified as a real finding** (documented with a reproducer and marked). 3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require **no crash, leak, OOM or timeout attributable to the fuzzer** (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. Every post-build crash goes through the classification procedure in the worktree reference: real findings keep the pair delivered with the discovery marked, fuzzer faults consume the fix budget, and a crash rooted in product code is always a finding, never a product-code patch. -4. **Coverage gain, fast path**: evaluate coverage from the libFuzzer `cov:` metric of the very runs the quality gate already performs (candidate versus baseline, same corpus source and duration; for a brand-new fuzzer, its absolute `cov:` on the standard seeds, noted as an approximation). Avoid a separate coverage build; fall back to `--sanitizer coverage` and per-file comparison only for marginal overlap decisions, ambiguous or flat results, or per-file numbers the report must show (see the worktree reference). Update the run's coverage ledger with the measured numbers and compute the pair's **incremental** contribution against what existing fuzzers and previously accepted pairs already cover: a pair whose gains mostly overlap is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded (rules from fuzzing-coverage-targeting). +4. **Coverage growth, per fuzzer (fast path)**: for every fuzzer created or modified, compare the start and end `cov:` of its own quality-gate run: coverage must increase over the run, with `ft:` (features) and corpus size as supporting increase signals. No baseline build, no cross-build comparison and no separate coverage build are needed; a `--sanitizer coverage` build remains only as the per-file fallback for file-level numbers the report must show or marginal overlap decisions (see the worktree reference). Update the run's coverage ledger with the measured numbers and compute the pair's **incremental** contribution against what existing fuzzers and previously accepted pairs already cover: a pair whose gains mostly overlap is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded (rules from fuzzing-coverage-targeting). 5. **Fix budget of 3 rounds**: fix, retest, and at most three attempts per gate failure. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is **rejected**: move both worktrees into the rejected area with `git worktree move` (never delete), keep the branches, and record the failure reason, what was attempted and the new paths. 6. Deliver: remove any copied build-fix changes and verify the remaining diff is purely this pair's own work (see the worktree reference), finalise the per-project report and the manifest row, and add the row to the final summary table. +7. **Exit protocol and relay**: before a session exits, update the manifest (pairs delivered, remaining ranked targets with rationales, terminal `STOP:` line per the format in Phase 0); when context or time runs low, checkpoint cleanly rather than wrapping up shallowly, because the batch driver relaunches continuation sessions until a terminal stop or the session cap. -Portfolio cap: at most **5 pairs per project** by default (fuzzer pairs plus the standalone corpus pair); the user's prompt may raise or lower the threshold, and the run stops adding pairs earlier when the remaining incremental worth is marginal. +Portfolio loop: work each project's ranked list top-down to the **cap of 5 pairs per project** by default (prompt-adjustable; improvement pairs, with a `-build-fix` pair not counting against it), stopping earlier only on documented worth exhaustion (remaining marginal worth genuinely low, with numbers), and recording genuine budget exhaustion as deferred entries ("pair N of project X: worth rationale written, not yet attempted"), never as worth decisions. Disk discipline: check available disk before builds; when low, stop cleanly with a `budget-exhausted (disk)` stop and deferred entries rather than continuing. ## Definition of done diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md index 509d217eea22..1cb54c0a7b6b 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md @@ -46,12 +46,16 @@ A project is delivered through one or more duo pairs: | Standalone corpus pair | All corpora, dictionaries and options changes with no fuzzer change | Zero or one, suffix `-corpus-dict-options` | | Build-fix pair | A general build fix not tied to one fuzzer | Zero or one, suffix `-build-fix` | -A single delivered pair is the minimum bar; develop multiple pairs when worthwhile. Worth is your own judgement, but it must be backed by the written Phase 2 rationale (dark covered lines, security sensitivity, reachability, expected impact), recorded per pair. Do not fragment trivially: corpus, dictionary and option tweaks that are not fuzzer-related always fold into the standalone pair. +Work the ranked target list top-down per project until the rules are exhausted; a single delivered pair is the floor for delivered status, never a stopping point, and worth is never compared across projects. Worth is your own judgement, but it must be backed by the written Phase 2 rationale (dark covered lines, security sensitivity, reachability, expected impact), recorded per pair. Stop a project only on documented worth exhaustion (with numbers), the 5-pair cap, or genuine budget exhaustion, recorded as deferred entries ("pair N of project X: worth rationale written, not yet attempted"), never as worth decisions. A project with no worthy targets is marked `no-worthy-targets` with its triage evidence; never fabricate a minimal pair. The declaration must be proven, not asserted: list the top rejected candidate regions with their numbers and the reason each fails the worth model, and treat it as seriously as a failure exit. Do not fragment trivially: corpus, dictionary and option tweaks that are not fuzzer-related always fold into the standalone pair. Upstream worktrees contain only allowed changes: fuzzers with their corpora, dictionaries and options, fuzz build configuration, and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions. Upstream product code is never modified; product-code bugs are findings. The OSS-Fuzz worktrees carry the integration changes (Dockerfile, `build.sh`, `project.yaml`) plus the bridge. Each pair gets its own worktrees, bridge blocks, validation, fix budget and summary row. A project counts as delivered when at least one of its pairs is delivered; rejected pairs are still recorded individually. For mixed-language projects, a pair whose harness targets a module in a different language may activate that language's skill, noted in the pair's row. +## Build reuse across pairs of the same project + +Every pair must be **runnable solo**: a complete, self-contained OSS-Fuzz build from its own worktrees, never depending on artifacts from another pair. Shared build artifacts across pairs are forbidden. The only permitted saving is Docker cache discipline: build a project's pairs back-to-back, with no `--clean` and no cache-busting between pairs of the same project (clean between projects instead), and keep bridge COPY blocks as late as possible in the Dockerfile so the shared layers (environment setup, clone) stay cached. + ## Build-fix dependency protocol (main build failures only) This protocol applies when the **main fuzzer build** fails. Coverage or introspector build failures do not trigger it; they follow the exemption rule (missing corpora or OOM/timeout) plus the historical-report date walk-back. @@ -95,7 +99,7 @@ Reading, cloning and inspecting upstream code locally is fine as usual; only val ## The validation loop (per duo pair) -Run pairs one at a time (builds of the same project share Docker image names), from the OSS-Fuzz worktree root, always through `infra/helper.py`: +Serialise only the real `infra/helper.py` executions (one pair's builds, checks and runs at a time: Docker image names collide); analysis and generation work may proceed in parallel. Commands run from the OSS-Fuzz worktree root, always through `infra/helper.py`: ```sh # 1. Build the declared matrix from projects//project.yaml: @@ -121,18 +125,13 @@ Quality thresholds, which apply on top of `check_build` passing, and to corpora, These libFuzzer-style flags apply to every OSS-Fuzz engine: Jazzer, Atheris and Go-native fuzzing all wrap libFuzzer. -### Coverage evaluation: fast path by default - -Do not build a separate coverage configuration for routine pair evaluation. Use the fuzzing runs the quality gate already performs: +### Coverage evaluation: per-fuzzer growth by default -1. Baseline: run the existing fuzzers on the pre-change build for the standard duration with the standard corpus source, and record the final libFuzzer `cov:` value (with `ft:` and `corp:` as supporting signals). -2. Candidate: the same runs on the pair's build, same corpus and duration; record the final `cov:`. -3. The pair's fast-path gain is candidate `cov:` versus baseline `cov:`. For a brand-new fuzzer with no baseline binary, the recorded gain is its absolute `cov:` on the standard seeds, noted as an approximation. -4. Like-for-like still applies: same instrumentation configuration (the normal ASan build already carries SanitizerCoverage, which is what `cov:` reports), same corpus source, same duration. Never compare fresh local numbers with production reports. +For every fuzzer created or modified, the fast path needs no baseline build and no cross-build comparison: record the libFuzzer `cov:` at the start and at the end of its own quality-gate run, and require coverage to increase over the run, with `ft:` (features) and corpus size (`corp:`) as supporting increase signals. This is the same signal as the staleness rule, so one run serves both checks. -The `cov:` trend between libFuzzer checkpoints is also the staleness signal, so one set of runs serves both the quality gate and the coverage check. +**Fallback to a coverage build**, only when file-level numbers are needed: per-file gains the final report must show, or a marginal overlap decision between pairs. Then build with `--sanitizer coverage` and compare per-file `lines.covered` counts on project-own paths (baseline from a clean state versus candidate in the pair worktree, identical flags and corpus source; targeted files must gain and no project-own file may lose), and use those numbers for the ledger decision. -**Fallback to a coverage build**, only when the fast path cannot answer: marginal overlap decisions between pairs, ambiguous or flat `cov:` results, or per-file gains the final report must show. Then build with `--sanitizer coverage` and compare per-file `lines.covered` counts on project-own paths (baseline from a clean state versus candidate in the pair worktree, identical flags and corpus source; targeted files must gain and no project-own file may lose), and use those numbers for the ledger decision. The overlap ledger records per-fuzzer `cov:` numbers by default and falls back to per-file data only when a decision is marginal. +Like-for-like still applies to the fallback: identical instrumentation configuration, corpus source and duration; never compare fresh local numbers with production reports. The overlap ledger records per-fuzzer growth numbers by default and falls back to per-file data only when a decision is marginal. ### Crash classification (every post-build crash, leak, OOM, timeout or assertion) @@ -187,12 +186,13 @@ The `upstream-clones/` repositories are scaffolding: keep them while work is in ## Final summary and handover -Write `summary.md` at the working-area root with one markdown table covering everything that was selected: +Write a per-project summary fragment at `build/coverage-pipeline/summary/.md` containing the project's row(s); the batch driver deterministically merges all fragments into `build/coverage-pipeline/summary.md` after the run. The merged table uses one header and covers everything that was selected: -| project | language | status | duo branches | depends on | worktree paths | summary of work | coverage delta | reason | -|---|---|---|---|---|---|---|---|---| +| project | language | status | duo branches | depends on | worktree paths | summary of work | coverage delta | remaining (deferred) | reason | +|---|---|---|---|---|---|---|---|---|---| -- status is `delivered`, `rejected (moved to rejected area)` or `skipped/blocked`; +- status is `delivered` (with any `deferred(budget)` entries listed alongside as "pair N: worth rationale written, not yet attempted"), `rejected (moved to rejected area)`, `no-worthy-targets`, or `skipped/blocked`; +- deferred entries and no-worthy-targets rows carry their evidence, so a later run picks them up first; - depends on records `tested-with: -build-fix` for pairs validated with a copied build fix; commit those fix branches first; - the language column shows the actual language used, with the declared one noted where they differ, for example `rust (declared c++)`; - skipped rows carry the concrete reason, for example `blocked(upstream-archived)`; From f98950678e17b3cc28f0feb1902668a4d1724d45 Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Thu, 29 Oct 2026 11:27:31 +0000 Subject: [PATCH 12/14] Add coverage improvement obligation --- infra/experimental/agent-skills/helper.py | 41 +++++++++---------- .../oss-fuzz-coverage-pipeline/SKILL.md | 14 +++++++ 2 files changed, 34 insertions(+), 21 deletions(-) diff --git a/infra/experimental/agent-skills/helper.py b/infra/experimental/agent-skills/helper.py index e3321e236ccb..c53236bfba43 100644 --- a/infra/experimental/agent-skills/helper.py +++ b/infra/experimental/agent-skills/helper.py @@ -1491,13 +1491,14 @@ def _run_sessions(task, args): task_description = getattr(args, 'task_description', None) pipeline_cap = getattr(args, 'cap', 5) - _, report_name = build_prompt(task, projects[0], task_description, - pipeline_cap=pipeline_cap) + _, report_name = build_prompt(task, + projects[0], + task_description, + pipeline_cap=pipeline_cap) if args.print_only: for project in projects: - print_prompt(task, project, task_description, - pipeline_cap=pipeline_cap) + print_prompt(task, project, task_description, pipeline_cap=pipeline_cap) return agent_cli = args.agent or find_agent_cli() @@ -1524,8 +1525,7 @@ def _run_sessions(task, args): with concurrent.futures.ThreadPoolExecutor( max_workers=max_parallel) as executor: futures = { - executor.submit(_run_single_session, agent_cli, task, project, - task_description, 1, 1, 'medium', pipeline_cap): + executor.submit(_run_single_session, agent_cli, task, project, task_description, 1, 1, 'medium', pipeline_cap): project for project in projects } for future in concurrent.futures.as_completed(futures): @@ -1812,11 +1812,10 @@ def _pipeline_stop_reached(project): 'manifest') candidates = [ os.path.join(manifest_dir, f'{project}.md'), - os.path.join(OSS_FUZZ_ROOT, 'build', 'coverage-pipeline', - 'manifest.md'), + os.path.join(OSS_FUZZ_ROOT, 'build', 'coverage-pipeline', 'manifest.md'), ] - pattern = re.compile( - rf'^\s*\**\s*stop\s*:\s*{re.escape(project)}\s*:', re.IGNORECASE) + pattern = re.compile(rf'^\s*\**\s*stop\s*:\s*{re.escape(project)}\s*:', + re.IGNORECASE) for manifest in candidates: if not os.path.isfile(manifest): continue @@ -1837,7 +1836,7 @@ def _disk_space_ok(): usage = shutil.disk_usage(OSS_FUZZ_ROOT) except OSError: return True - return usage.free >= MIN_FREE_DISK_GB * 1024 ** 3 + return usage.free >= MIN_FREE_DISK_GB * 1024**3 def _collate_fragments(fragment_dir, merged_path, header): @@ -1872,12 +1871,12 @@ def _run_pipeline_relay(agent_cli, args, project): 'cleanly. The session should record a budget-exhausted (disk) ' 'stop with deferred entries if it runs at all.') return {'project': project, 'returncode': 0, 'log_path': None} - task = ('coverage-pipeline' if session_num == 1 - else 'coverage-pipeline-continue') + task = ('coverage-pipeline' + if session_num == 1 else 'coverage-pipeline-continue') print(f'[*] {project}: launching session {session_num}/{max_sessions} ' f'({task})') - result = _run_single_session(agent_cli, task, project, None, 1, 1, - 'medium', args.cap) + result = _run_single_session(agent_cli, task, project, None, 1, 1, 'medium', + args.cap) if _pipeline_stop_reached(project): print(f'[*] {project}: terminal stop recorded; relay complete.') return result @@ -1942,8 +1941,7 @@ def cmd_coverage_pipeline(args): if args.print_only: for project in projects: print_prompt('coverage-pipeline', project, pipeline_cap=args.cap) - print_prompt('coverage-pipeline-continue', project, - pipeline_cap=args.cap) + print_prompt('coverage-pipeline-continue', project, pipeline_cap=args.cap) return agent_cli = args.agent or find_agent_cli() @@ -1965,8 +1963,8 @@ def cmd_coverage_pipeline(args): with concurrent.futures.ThreadPoolExecutor( max_workers=args.max_parallel) as executor: futures = { - executor.submit(_run_pipeline_relay, agent_cli, args, project): - project for project in projects + executor.submit(_run_pipeline_relay, agent_cli, args, project): project + for project in projects } for future in concurrent.futures.as_completed(futures): future.result() @@ -2457,8 +2455,9 @@ def main(): help='OSS-Fuzz project names.') show_parser.add_argument( '--task', - choices=['expand', 'fix-build', 'free-task', 'add-chronos', - 'coverage-pipeline'], + choices=[ + 'expand', 'fix-build', 'free-task', 'add-chronos', 'coverage-pipeline' + ], default='expand', help='Which task prompt to show (default: expand).', ) diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index 76e31b1d6212..86f77912041e 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -114,6 +114,17 @@ Produce a written target list via **fuzzing-coverage-targeting** (worth model, d 3. Cross-check Fuzz Introspector for **statically reached but dynamically unreached** functions: reachable from harnesses but never hit in the coverage report. These are prime targets (seeds, dictionaries and harness adjustment to convert static reachability into real coverage). 4. Classify each candidate security-sensitive or not; the point is attack surface, not raw percentage. In scope: file-format parsers and loaders, wire-protocol handlers, deserialisers, decompression and codec paths, custom string and number parsing on untrusted data, path and filename handling, IPC message handling, state machines driven by untrusted input, crypto primitives. Out of scope, usually: diagnostic logging and printf-style output, UI code, config writing, build tooling and test helpers, unless untrusted data reaches them. 5. Record per target: the metric that flagged it, whether it is seed-limited or harness-limited, and the expected bug class. +### Phase 2.5: Coverage improvement obligation (mandatory when coverage is low) + +After live coverage triage, apply this gate: + +- If the project's own-code line coverage is **below 15%**, the skill **must** produce at least one deliverable improvement pair before the project can be marked `delivered`. +- The obligation is recorded in the manifest as `coverage-obligation: ` and the target coverage goal. +- This overrides any "already viable" early exit. The agent is required to: + - Propose and implement at least one new fuzzer targeting a dark security-sensitive area, **or** + - Significantly expand the corpus + dictionary for existing fuzzers with measurable coverage gain (documented before/after `summary.json` or `reports-by-target`). +- Only after this obligation is satisfied (and validated via the quality gate in Phase 5) may the project be marked `delivered`. +- Projects already above 15% own-code coverage may exit early if no other work is needed. ### Phase 3: Act on the targets @@ -131,6 +142,9 @@ Choose the harness shape per **fuzzing-coverage-targeting**'s pattern catalogue ### Phase 4: Stage in worktree duo pairs Follow [references/worktree_delivery.md](references/worktree_delivery.md) precisely. Essentials: +- All remaining corpora, dictionaries and options changes with no fuzzer change are consolidated into a **single standalone pair** with the suffix `-corpus-dict-options`. +- A general build fix not tied to one fuzzer gets its own pair with the suffix `-build-fix`; build-configuration changes a specific fuzzer needs ride inside that fuzzer's pair. +- When the coverage obligation in Phase 2.5 is active, the primary deliverable type becomes **`-coverage-improvement`**. This pair may contain new fuzz targets, expanded corpora + dictionaries, and minor fuzz build configuration changes. It is the only pair type allowed to satisfy the coverage obligation. - A **duo pair is always created**, even when no OSS-Fuzz changes are needed, because all testing goes through the temporary copy bridge in the OSS-Fuzz side Dockerfile, and the main checkouts must stay clean. - Branches: OSS-Fuzz side `oss-fuzz--`, upstream side `-`; the identical `` is what pairs them. From 73a2c2f8db4876d685a2e0daa86be3fbd3c58ff9 Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Tue, 3 Nov 2026 10:19:20 +0000 Subject: [PATCH 13/14] Update heavy processing rules Signed-off-by: Arthur Chan --- .../oss-fuzz-coverage-pipeline/SKILL.md | 21 +++++++++++++++---- .../references/worktree_delivery.md | 4 +++- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index 86f77912041e..df0b2eeaa961 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -24,7 +24,7 @@ This skill is a process wrapper. It defines what to do and in which order; it de - **fuzzing-coverage-targeting**, in Phases 2, 3 and 5: which targets are worth harnessing and what shape each harness takes (worth model, dark-region taxonomy, harness patterns, portfolio cap and overlap rules). -plus their references (`code_coverage.md`, `structured_seed_generation.md`, `helper.md`, `oss_fuzz_project_structure.md`) where pointed to below. +plus their references (`code_coverage.md`, `structured_seed_generation.md`, `helper.md`, `oss_fuzz_project_structure.md`) where pointed to below — loaded on demand, targeted to the section a phase needs, never as bulk pre-reading (see Context discipline). ## Hard guardrails @@ -37,6 +37,18 @@ plus their references (`code_coverage.md`, `structured_seed_generation.md`, `hel 7. **Classify every post-build crash; never silence**: a crash, memleak, OOM or timeout after a successful build, whether it surfaces in `check_build` or in the fuzzing runs, is never an automatic failure: classify it first (procedure in the worktree reference). Fuzzer faults consume the fix budget; real findings keep the pair delivered with the discovery marked and consume nothing. Never introduce new silencing (no leak-detection muting, no suppression flags or files, no raised limits to dodge a failure). Pre-existing suppressions in the existing integration or existing fuzzers are honoured unchanged: carried over for new and fixed fuzzers, neither removed nor widened, and only what they already cover is out of scope. 8. **Precedence**: these guardrails and this skill's rules override any conflicting guidance in the sub-skills; the sub-skills supply craft and knowledge within these bounds. +## Context discipline (every phase) + +The pipeline is long and build-heavy; context is a budget like disk or time, and running out of it loses work. These rules are mandatory in every phase, and they matter most exactly when pressure tempts the opposite. + +1. **Load knowledge lazily and targeted.** Sub-skills and references are read on demand, for the phase that needs them: a bounded search or a line-ranged read of the operative section, never a bulk pre-read of every skill, reference and source file before the first deliverable exists. +2. **Big data never crosses the chat.** Coverage reports, `summary.json`, `status.json`, introspector pages and logs are downloaded to disk (`curl -fsSL -o `, or `infra/helper.py ... > `) and reduced with a local `python3`/`jq` one-liner that prints only the numbers needed (`scripts/select_projects.sh` shows the pattern). Never pull raw JSON, HTML or a full log through a fetch tool, and never `cat` a large file into context. +3. **Bound every output.** Cap every search and read: `grep -m`, `head -c`/`tail -c`, explicit line ranges. No unbounded dumps (limitless `grep -B`/`-A`, full-file `sed -n` prints, full log tails), and a file already on disk is never re-read "to be sure". +4. **Background long builds; poll, never wait.** Builds and gate runs run in the background with output redirected to a log file, polled with one short bounded command (`tail -c 300 `) at short intervals. A multi-minute foreground `sleep` burns a full result slot. A failure costs one bounded extraction of the error region (first `error:` or sanitizer summary plus a few lines), never a whole-log re-read. +5. **Small, anchored, verified edits.** Keep every edit under ~4000 characters; take anchors from a fresh small read of the exact region immediately before editing, and verify the result with a bounded grep right after. Never do string surgery on code with `sed` (one unescaped metacharacter silently mangles a file and costs a whole debug cycle); use a small `python3 -c` replacement or rewrite the small file whole. +6. **Checkpoint after every completed step, not just every phase.** The manifest row, plan and coverage ledger are updated immediately after each step (pair staged, fix applied, build or gate result, decision taken), so an eviction or a fresh session recovers with one small manifest read plus bounded verification (`git diff --stat`, `grep -c`), never by reconstructing state from memory or mass re-reads. +7. **Evidence lives on disk, not in context.** Sanitizer reports, run logs and coverage numbers are saved under the per-project report area and referenced by path; context keeps only the decisive lines (verdict, crash type, top frames, `cov:` numbers). + ## Deliverables The result of a run, all inside the marked working area (see [references/worktree_delivery.md](references/worktree_delivery.md)): @@ -74,7 +86,7 @@ Stopping is mandatory in every case; it is never agent judgement. Then create (o | project | language | status | duo branches | worktree paths | base SHAs | report | notes | |---|---|---|---|---|---|---|---| -Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, `no-worthy-targets`, `deferred(budget)`, or `blocked()`. Update the row at the end of every phase; a fresh session resumes work by reading the manifest. When a project yields several pairs, give each pair its own manifest row (same project, its own suffix and duo branches), with a `tested-with: -build-fix` note in the notes column when it depends on the build fix. The language column starts as the declared language; after a mid-run language switch (see Phase 2), record it as `declared(X)/actual(Y)`. A rerun after a previous delivered batch starts fresh rows (or a new dated section) and treats earlier rows as history only; previously rejected pairs may be deliberately retried against the new state. Sessions exit via the exit protocol: before exiting, a session updates the manifest with the pairs delivered, the remaining ranked targets with rationales, and a terminal stop line `STOP: : `; running out of context means checkpointing progress and deferred entries, never a shallow wrap-up. Declaring `no-worthy-targets` requires genuine judgement and proof: a per-region worth analysis with the top rejected candidate regions listed, their numbers, and why each fails the worth model; it is never a quick early exit and is reviewed as seriously as a failure. +Statuses: `selected`, `viable`, `builds-green`, `triaged`, `improved`, `validated`, `delivered`, `rejected`, `no-worthy-targets`, `deferred(budget)`, or `blocked()`. Update the row at the end of every phase and after every completed step (pair staged, fix applied, build or gate result, decision taken), so an eviction mid-phase cannot orphan work; a fresh or post-eviction session resumes by reading the manifest fragment plus bounded verification (`git diff --stat`, a couple of targeted greps), never by re-reading big files to re-establish ground truth. When a project yields several pairs, give each pair its own manifest row (same project, its own suffix and duo branches), with a `tested-with: -build-fix` note in the notes column when it depends on the build fix. The language column starts as the declared language; after a mid-run language switch (see Phase 2), record it as `declared(X)/actual(Y)`. A rerun after a previous delivered batch starts fresh rows (or a new dated section) and treats earlier rows as history only; previously rejected pairs may be deliberately retried against the new state. Sessions exit via the exit protocol: before exiting, a session updates the manifest with the pairs delivered, the remaining ranked targets with rationales, and a terminal stop line `STOP: : `; running out of context means checkpointing progress and deferred entries, never a shallow wrap-up. Declaring `no-worthy-targets` requires genuine judgement and proof: a per-region worth analysis with the top rejected candidate regions listed, their numbers, and why each fails the worth model; it is never a quick early exit and is reviewed as seriously as a failure. Multi-project discipline: **serialise only real `infra/helper.py` executions** (Docker builds, checks and runs: one project at a time, since image names collide). Analysis (triage, coverage study) and worktree/fuzzer generation may proceed in parallel across projects and pairs. With 1,300+ projects in OSS-Fuzz (~600 C/C++), prefer prioritised subsets over `*`: broken builds first (fast wins that also unlock coverage data), then the lowest own-code coverage on security-sensitive surface. @@ -89,6 +101,7 @@ python3 infra/helper.py build_fuzzers --sanitizer introspector python3 infra/helper.py check_build ``` +- **Run long builds in the background**: redirect output to a log file and poll with one short bounded command (`tail -c 300 `); never hold a slot with a multi-minute foreground `sleep`, and never re-read a whole build log. On failure, extract only the error region (first `error:` or sanitizer summary plus a few lines) into the report before fixing. - The main fuzzer build fix is **always mandatory**: minimal, targeted changes only. When the main build is broken, the fix becomes its own `-build-fix` duo pair, created and validated before any dependent pair (see Phase 4 and the worktree reference). An upstream-side build fix must itself stay within the allowed upstream categories (fuzzers, fuzz build configuration, the `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` macro); if the build can only be repaired by changing product code, stop and notify `blocked(product-code-fix-needed)`. - Coverage and introspector build fixes are expected, but are **optional when the failure cause is missing corpora or an OOM/timeout**. Record the exemption with evidence, then proceed using the most recent successful historical report (Phase 2 date walk-back). These failures do not trigger the build-fix protocol. - A `check_build` failure caused by a crash is classified like any other post-build crash (see the worktree reference): it is a harness defect only when the crash is the fuzzer's fault; a crash classified as a real finding still passes this gate with the discovery marked. @@ -100,7 +113,7 @@ Canonical live coverage references (resolve `YYYYMMDD` with the walk-back rule b - overall report: `https://storage.googleapis.com/oss-fuzz-coverage//reports//linux/report.html` - per fuzz target: `https://storage.googleapis.com/oss-fuzz-coverage//reports-by-target///linux/report.html` (use it to attribute dark code to specific harnesses) -- parse the machine-readable `summary.json` siblings at the same paths; the HTML is the human view. +- parse the machine-readable `summary.json` siblings at the same paths; the HTML is the human view. Both are downloaded with `curl -fsSL -o ` and reduced to numbers by a local `python3` one-liner — never pulled raw through a fetch/chat tool or `cat`ed into context (a single report can be megabytes; `scripts/select_projects.sh` shows the pattern). - reachability detail: `https://introspector.oss-fuzz.com/project-profile?project=` **Date walk-back**: coverage and introspector builds sometimes fail (cross-check the build status on introspector.oss-fuzz.com), so the latest date may have no report. Resolve the date by walking backwards day by day, at most 30 days, until a report exists, and record the date actually used. @@ -167,7 +180,7 @@ Validate each duo pair independently and one at a time, always through `infra/he 1. **Build matrix**: read `sanitizers:` and `fuzzing_engines:` from `projects//project.yaml` and validate every declared configuration (for example `--sanitizer address`, `undefined`, `memory`, and engines such as `--engine afl` where declared). 2. **Per-fuzzer gate (mandatory)**: `build_fuzzers` succeeds, and `check_build` passes for every new or updated fuzzer specifically, not just project-wide, **or its failure is solely attributable to a crash classified as a real finding** (documented with a reproducer and marked). -3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require **no crash, leak, OOM or timeout attributable to the fuzzer** (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. Every post-build crash goes through the classification procedure in the worktree reference: real findings keep the pair delivered with the discovery marked, fuzzer faults consume the fix budget, and a crash rooted in product code is always a finding, never a product-code patch. +3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require **no crash, leak, OOM or timeout attributable to the fuzzer** (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. Every post-build crash goes through the classification procedure in the worktree reference: real findings keep the pair delivered with the discovery marked, fuzzer faults consume the fix budget, and a crash rooted in product code is always a finding, never a product-code patch. Sanitizer output is saved to disk in the per-project report area and only the decisive lines enter context; gate retries never re-read a whole report. 4. **Coverage growth, per fuzzer (fast path)**: for every fuzzer created or modified, compare the start and end `cov:` of its own quality-gate run: coverage must increase over the run, with `ft:` (features) and corpus size as supporting increase signals. No baseline build, no cross-build comparison and no separate coverage build are needed; a `--sanitizer coverage` build remains only as the per-file fallback for file-level numbers the report must show or marginal overlap decisions (see the worktree reference). Update the run's coverage ledger with the measured numbers and compute the pair's **incremental** contribution against what existing fuzzers and previously accepted pairs already cover: a pair whose gains mostly overlap is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded (rules from fuzzing-coverage-targeting). 5. **Fix budget of 3 rounds**: fix, retest, and at most three attempts per gate failure. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is **rejected**: move both worktrees into the rejected area with `git worktree move` (never delete), keep the branches, and record the failure reason, what was attempted and the new paths. 6. Deliver: remove any copied build-fix changes and verify the remaining diff is purely this pair's own work (see the worktree reference), finalise the per-project report and the manifest row, and add the row to the final summary table. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md index 1cb54c0a7b6b..d593a341a028 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md @@ -116,6 +116,8 @@ python3 infra/helper.py run_fuzzer -- \ -max_total_time=120 -timeout=25 -rss_limit_mb=2560 ``` +Run each command in the background with its output redirected to a log file and poll with one short bounded command (`tail -c 300 `); never hold a tool slot with a multi-minute `sleep`, and never re-read whole build or run logs. A failure costs one bounded extraction of the error region; a passing gate run costs only its `cov:`/`ft:`/`corp:` lines. + Quality thresholds, which apply on top of `check_build` passing, and to corpora, dictionaries and option changes as well as new harnesses: - minimum run 120 seconds per target, 300 seconds for corpus-heavy changes; @@ -135,7 +137,7 @@ Like-for-like still applies to the fallback: identical instrumentation configura ### Crash classification (every post-build crash, leak, OOM, timeout or assertion) -A crash after a successful build is never an automatic failure, whether it surfaces in `check_build` or in the fuzzing runs. Capture the output and artefact, reproduce it standalone (`run_fuzzer` with the artefact), then classify. +A crash after a successful build is never an automatic failure, whether it surfaces in `check_build` or in the fuzzing runs. Capture the output and artefact, reproduce it standalone (`run_fuzzer` with the artefact), then classify. Save the full sanitizer report and the artefact path under the per-project report area; context keeps only the decisive frames (crash type, top stack frame, allocated-by line for leaks), so fix rounds never re-read a whole report. **Fuzzer faults** (fix within the budget): From 381d63edd024dda24bf9d3d038266c5b6bc8b582 Mon Sep 17 00:00:00 2001 From: Arthur Chan Date: Tue, 10 Nov 2026 05:33:51 +0000 Subject: [PATCH 14/14] Update corpus/dict and coverage rules Signed-off-by: Arthur Chan --- infra/experimental/agent-skills/USAGE.md | 8 ++-- .../fuzzing-coverage-targeting/SKILL.md | 14 ++++--- infra/experimental/agent-skills/helper.py | 19 ++++++++- .../oss-fuzz-coverage-pipeline/SKILL.md | 42 +++++++++---------- .../references/worktree_delivery.md | 7 +++- .../references/structured_seed_generation.md | 16 +++++++ 6 files changed, 71 insertions(+), 35 deletions(-) diff --git a/infra/experimental/agent-skills/USAGE.md b/infra/experimental/agent-skills/USAGE.md index e0d213a064ec..fc9ef132f4e3 100644 --- a/infra/experimental/agent-skills/USAGE.md +++ b/infra/experimental/agent-skills/USAGE.md @@ -9,7 +9,7 @@ Companion to [README.md](README.md), which covers the folder overview and the th | Piece | Role | |---|---| | Six knowledge skills | `oss-fuzz-engineer` (infrastructure) plus one fuzzing skill per language: `fuzzing-memory-unsafe-expert`, `fuzzing-go-expert`, `fuzzing-rust-expert`, `fuzzing-jvm-expert`, `fuzzing-python-expert` | -| `oss-fuzz-coverage-pipeline` | The orchestrator skill: project selection with upstream viability checks, build fixes, live coverage triage, harness/corpus/dictionary improvements, and delivery as worktree branch duo pairs validated through a temporary copy bridge. It auto-detects each project's language from `project.yaml` and activates the matching language skill. | +| `oss-fuzz-coverage-pipeline` | The orchestrator skill: project selection with upstream viability checks, build fixes, live coverage triage, harness improvements (corpus/dictionary work only where a proven gate justifies it), and delivery as worktree branch duo pairs validated through a temporary copy bridge. It auto-detects each project's language from `project.yaml` and activates the matching language skill. | | `fuzzing-coverage-targeting` | The architect skill: which fuzz targets are worth creating and what shape each harness takes (worth model, taxonomy, patterns, portfolio and overlap rules); used by the pipeline in Phases 2, 3 and 5, and usable on its own | | `oss-fuzz-coverage-pipeline/scripts/select_projects.sh` | Deterministic project selector: globs, `--language`, `--broken`, `--limit`, `--exclude`. | | `copy_to_global.sh` | Installer: `claude` (the default), `gemini`, or any skills directory you name. | @@ -133,10 +133,10 @@ Continuing after a session ends (Approach A): check `build/coverage-pipeline/man 0. Select the projects with the selector, then verify upstream viability by reading the **Dockerfile** (never trusting `main_repo`): archived, inactive for 24 months or more, moved, or not a git repository all mean stop and notify `blocked(...)`. The selector reports the language **declared** in project.yaml; the Dockerfile base image is never a language signal. All project data is reread fresh on every run, so a project whose project.yaml changed since an earlier run is treated according to the new values, never previous memory. Reruns after a previous batch of duo pairs likewise re-derive coverage and targets from fresh data, because the previous batch may have changed the picture. Record the per-project manifest fragment at `build/coverage-pipeline/manifest/.md`. 1. Fix builds through `infra/helper.py` only: the main fuzzer build fix is always mandatory; coverage and introspector build fixes are optional only when the failure cause is missing corpora or an OOM/timeout, with the exemption recorded. -2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. Target selection and harness shaping come from the architect skill `fuzzing-coverage-targeting`, producing a ranked multi-target portfolio with overlap-aware incremental worth. Each project's ranked list is then worked top-down until worth is exhausted (documented with numbers), the cap is reached (default 5 pairs per project, adjustable in your prompt) or the budget truly runs out; budget-deferred targets keep their written rationales as "not yet attempted", and a project with genuinely no worthy targets is marked `no-worthy-targets` rather than given a fabricated pair. +2. Triage live coverage using the canonical report URLs (overall `reports//linux/report.html` and per target `reports-by-target///linux/report.html`), walking the date back up to 30 days when coverage builds failed, plus the Introspector project profile. It verifies build scope before trusting percentages, treats statically-reached-but-dynamically-unreached functions as prime targets, and filters to security-sensitive attack surface. It also verifies the declared language against reality and switches the language sub-skill if project.yaml is mislabelled, continuing in all cases and recording the discrepancy; a per-pair override applies for mixed-language projects. Target selection and harness shaping come from the architect skill `fuzzing-coverage-targeting`, producing a ranked multi-target portfolio with overlap-aware incremental worth. Each project's ranked list is then worked top-down until worth is exhausted (documented with numbers), the cap is reached (default 5 pairs per project, adjustable in your prompt) or the budget truly runs out; budget-deferred targets keep their written rationales as "not yet attempted", and a project with genuinely no worthy targets is marked `no-worthy-targets` rather than given a fabricated pair; coverage percentage is a signal, never a threshold (named projects are processed whatever their coverage), and only project-own code is in scope — project-maintained versions of third-party code count, unmodified vendored dependencies do not. 3. Act: new or updated harnesses, structured seed corpora, dictionaries, enabling missing modules and build-configuration fixes, via the language-appropriate skill. It never modifies upstream product code: upstream changes are limited to fuzzers with their corpora, dictionaries and options, fuzz build configuration and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions; product-code bugs become findings. -4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. A project may yield several pairs when worthwhile: fuzzer pairs, each carrying its harness plus that harness's corpora, dictionaries and options; one consolidated standalone corpus/dict/options pair (`-corpus-dict-options`); and, when the main build was broken, a `-build-fix` pair created first, whose changes later pairs copy in purely as a testing scaffold and remove again after conclusion. -5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory, or failing solely due to a marked real finding); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, nothing fuzzer-attributable within the first 60 seconds, no stale coverage in the final half); and per-fuzzer coverage growth (start versus end `cov:` of the run, with `ft:` and corpus size; a coverage build only as the per-file fallback). Every post-build crash, leak, OOM, timeout or assertion is classified first: real upstream findings keep the pair delivered with the discovery marked, while only fuzzer faults consume the fix budget of 3 rounds, after which the pair is moved to the rejected area, never deleted. Nothing is ever newly silenced, pre-existing suppressions are honoured unchanged, and a timeout from genuinely heavy legitimate processing may receive a measured, light increase delivered as fuzzer options. +4. Stage a **worktree branch duo pair** (OSS-Fuzz side `oss-fuzz--`, upstream side `-`, always both, even with no OSS-Fuzz changes), created from pristine default-branch clones inside one marked working area, with a tagged **temporary copy bridge** in the Dockerfile simulating the merged upstream. Changes stay as plain working-tree files: no staging by default, never a commit, never a push. A project may yield several pairs when worthwhile: fuzzer pairs, each carrying its harness plus that harness's corpora, dictionaries and options; one consolidated standalone corpus/dict/options pair (`-corpus-dict-options`, staged only when that work passes the seed-limited gate analysis and shows measured gain — worthless leftovers are dropped); and, when the main build was broken, a `-build-fix` pair created first, whose changes later pairs copy in purely as a testing scaffold and remove again after conclusion. +5. Validate each pair through `infra/helper.py`: the full sanitizer and engine matrix declared in `project.yaml`; `check_build` per new or updated fuzzer (mandatory, or failing solely due to a marked real finding); a fuzzing-quality gate (120 seconds minimum, 300 for corpus-heavy changes, nothing fuzzer-attributable within the first 60 seconds, no stale coverage in the final half; corpus/dict/options-only changes additionally pass the value gate: documented mutation-resistant gates, multi-byte dictionary tokens, per-seed gain on the targeted dark files, and a gain over plain unseeded fuzzing); and per-fuzzer coverage growth (start versus end `cov:` of the run, with `ft:` and corpus size; a coverage build only as the per-file fallback). Every post-build crash, leak, OOM, timeout or assertion is classified first: real upstream findings keep the pair delivered with the discovery marked, while only fuzzer faults consume the fix budget of 3 rounds, after which the pair is moved to the rejected area, never deleted. Nothing is ever newly silenced, pre-existing suppressions are honoured unchanged, and a timeout from genuinely heavy legitimate processing may receive a measured, light increase delivered as fuzzer options. Crashes, leaks, OOMs, timeouts and assertions are classified (fuzzer fault versus real finding); real findings are **documented with reproducers**, never hidden and never silenced. diff --git a/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md b/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md index 914f70890163..2224743a2d28 100644 --- a/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md +++ b/infra/experimental/agent-skills/fuzzing-coverage-targeting/SKILL.md @@ -10,7 +10,7 @@ This skill is the architect for fuzz target selection: it answers **which** part ## Inputs -- Fresh coverage data: the project's `summary.json` and per-target reports (`reports-by-target`), filtered to the project's own code with build scope verified (a high percentage over a partial build is meaningless). +- Fresh coverage data: the project's `summary.json` and per-target reports (`reports-by-target`), filtered to the project's own code with build scope verified (a high percentage over a partial build is meaningless). Own code includes versions of third-party code the project maintains in-tree (its fork or locally patched copies); vendored dependencies kept unmodified from their upstream are not targets. - Fuzz Introspector data: statically reachable but dynamically unreached functions, optimal targets, blockers. - The project's source structure: modules, public APIs, state machines, configuration surfaces. - The language expert's threat model, when composed by the pipeline: attacker-reachable surfaces are the top ranking weight among equals. @@ -20,10 +20,12 @@ This skill is the architect for fuzz target selection: it answers **which** part Score each dark region as **dark reachable lines × functional importance × bug-class diversity ÷ harness cost**: - **Dark reachable lines**: lines statically reachable from some harness entry but uncovered, counted as absolute lines, not percentages. -- **Functional importance**: core parsing, transformation, protocol logic and data handling rank high; diagnostics, debug printers, test helpers and build-time code are excluded outright. +- **Functional importance**: core parsing, transformation, protocol logic and data handling rank high; diagnostics, debug printers, test helpers, build-time code and unmodified vendored third-party dependencies are excluded outright. - **Bug-class diversity**: memory safety (ASan/MSan/UBSan), logic and assertion violations, hangs and pathological blowups, and leaks. A region rich in several classes outranks a one-class region of equal size. - **Harness cost**: API complexity, state setup, corpus construction effort. +A region plain fuzzing already reaches within a normal run carries **zero incremental worth for corpus-only work**: seeds and dictionaries pay only where a mutation-resistant gate (multi-byte magic numbers, checksums, length fields, required keywords) holds the mutator out. + ## Dark-region taxonomy and harness opportunity types | Dark region type | Typical shape | Example surfaces | @@ -72,16 +74,16 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { } ``` -- **Option or configuration matrix**: consume small values as option flags and mode switches before the payload so one target sweeps many compiled paths cheaply; pair with a dictionary of option names. +- **Option or configuration matrix**: consume small values as option flags and mode switches before the payload so one target sweeps many compiled paths cheaply; pair with a dictionary of option names (multi-byte tokens traceable to the code; never single characters). - **Component API**: call the internal module through its intended API with typed, bounded parameters; the language skills' legitimate-input requirement applies (construct valid objects; never feed raw bytes where a struct is expected). ## Portfolio construction 1. Enumerate dark regions from the inputs; classify each by the taxonomy; score each by the worth model. 2. Apply the security weighting: attacker-reachable surfaces rank first among equals; functional dark logic is fully worthy; diagnostics, logging, printf-style output, test helpers and build-time code stay excluded. -3. Produce a **ranked multi-target list** expected to yield several focused harnesses rather than one or two; a single-target list is a smell that the taxonomy was not applied. If honest triage finds nothing worthy, say so: zero worthy targets is a valid outcome, and inventing a trivial pair to hit a one-pair floor is forbidden. The declaration must be proven, not asserted: list the top rejected regions with their numbers and the reason each fails the worth model. -4. Selection is iterative with diminishing returns, executed by the pipeline with its coverage ledger: after each accepted pair, re-rank the remaining targets by **incremental** worth (projected gains minus already-covered ground). A pair whose gains mostly overlap already-accepted coverage is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded. -5. **Default cap: at most 5 pairs per project** (fuzzer pairs plus the standalone corpus pair; a `-build-fix` pair does not count against the cap), unless the user raises or lowers the threshold in their prompt. +3. Produce a **ranked multi-target list** expected to yield several focused harnesses rather than one or two; a single-target list is a smell that the taxonomy was not applied. If honest triage finds nothing worthy, say so: zero worthy targets is a valid outcome — including for an already well-fuzzed project — and inventing a trivial pair to hit a one-pair floor is forbidden. The declaration must be proven, not asserted: list the top rejected regions with their numbers and the reason each fails the worth model. +4. Selection is iterative with diminishing returns, executed by the pipeline with its coverage ledger: after each accepted pair, re-rank the remaining targets by **incremental** worth (projected gains minus already-covered ground). A pair whose gains mostly overlap already-accepted coverage is redesigned toward different dark regions, folded into standalone corpus work — only when the region is provably seed-limited by a mutation-resistant gate — or dropped, with the numbers recorded. +5. **Default cap: at most 5 pairs per project** (fuzzer pairs plus the standalone corpus pair; a `-build-fix` pair does not count against the cap), unless the user raises or lowers the threshold in their prompt. The standalone corpus pair is exceptional — justified only by a proven seed-limited gate — and is never staged as portfolio filler. 6. **Work the ranked list top-down until the rules are exhausted.** Worth is measured per project and never compared across projects: in a multi-project batch, a second pair for project A competes only with A's own remaining dark regions, never with project B's candidates, and spreading one pair per project is an anti-pattern. A single delivered pair is only the floor for a project's delivered status, never a stopping point. Stop a project's loop only on one of: **(a) worth exhaustion**, the remaining marginal worth is genuinely low, documented with numbers; **(b) the cap**; or **(c) budget exhaustion**, when time or build budget truly runs out. Budget bounds ordering and depth only, never worth: record un-attempted worthy targets as "pair N of project X: worth rationale written, not yet attempted", never as a worth decision, and never drop a target as unworthy for budget reasons. ## Non-goals diff --git a/infra/experimental/agent-skills/helper.py b/infra/experimental/agent-skills/helper.py index c53236bfba43..29e3373641bd 100644 --- a/infra/experimental/agent-skills/helper.py +++ b/infra/experimental/agent-skills/helper.py @@ -736,6 +736,20 @@ - Validate each pair on the sanitizer/engine matrix from project.yaml, and evaluate coverage from the fuzzer runs' libFuzzer cov: metric (fast path), using a coverage build only as fallback. + - Coverage percentage is never a pass/fail threshold: deliver only + worthy pairs (measured coverage/effectiveness gain), and mark a + project no-worthy-targets with proof when no worthy improvement + remains — never stage unworthy work just to deliver something. + - Corpus/dictionary/options-only work is exceptional, never a default + deliverable: stage a -corpus-dict-options pair only with a documented + gate analysis (magic numbers, checksums or keyword gates that mutation + cannot cross, cited file:line) and measured gain; single-character + dictionary tokens and corpora for small, trivially-reachable APIs are + zero-value — skip them and record why. + - Only the project's own code is a target: third-party code is out of + scope, unless the project maintains its own version of it in-tree + (forks or locally patched copies count; unmodified vendored + dependencies do not). - Watch your context and time budget: when nearing the limit, checkpoint cleanly (persist progress and deferred entries) instead of wrapping up shallowly; a later session continues from the manifest. @@ -803,7 +817,10 @@ Identical to a first session: never commit, never push, no staging by default; upstream product code off-limits; everything through python3 infra/helper.py; classify every post-build crash; never introduce - new silencing; the cov: fast path for coverage evaluation; and when your + new silencing; the cov: fast path for coverage evaluation; coverage + percentage is never a threshold — worthy pairs or an honest + no-worthy-targets; corpus/dictionary/options-only pairs stay exceptional + (documented gate analysis or skip); project-own code only; and when your context or time runs low, checkpoint cleanly instead of wrapping up. ## Exit protocol (mandatory) diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md index df0b2eeaa961..79c495029493 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/SKILL.md @@ -1,7 +1,7 @@ --- name: oss-fuzz-coverage-pipeline description: >- - Use this skill to bring a project that is already integrated into OSS-Fuzz (C/C++, Go, Rust, JVM or Python) to a well-fuzzed state: fix broken builds, triage live coverage, improve fuzz targets, corpora and dictionaries, and deliver the work as worktree branch duo pairs. Never commits, never pushes and by default never stages. Selects projects by name, glob pattern or all projects, verifies upstream viability from the Dockerfile, auto-detects the language from project.yaml and activates the matching language skill. Processes one project at a time via a run manifest. + Use this skill to bring a project that is already integrated into OSS-Fuzz (C/C++, Go, Rust, JVM or Python) to a well-fuzzed state: fix broken builds, triage live coverage, improve fuzz targets, and add corpora and dictionaries only where a proven mutation-resistant gate justifies them, then deliver the work as worktree branch duo pairs. Never commits, never pushes and by default never stages. Selects projects by name, glob pattern or all projects, verifies upstream viability from the Dockerfile, auto-detects the language from project.yaml and activates the matching language skill. Processes one project at a time via a run manifest. --- # OSS-Fuzz coverage pipeline @@ -122,30 +122,28 @@ Produce a written target list via **fuzzing-coverage-targeting** (worth model, d **Verify the declared language while analysing**: while reading the source tree, build scripts and existing harnesses, check that the declared language (project.yaml) matches reality. If it does not, switch the active language sub-skill to the matching one and **continue in all cases**, even when the actual language falls outside the run's `--language` filter, recording the discrepancy: manifest `declared(X)/actual(Y)`, evidence in the report, actual language in the summary with the declared one noted. For mixed-language projects, the project-level language stays the corrected declared one, but a pair whose harness targets a module in a different language may activate that language's skill for that pair, noted in the pair's row. -1. Filter to the project's own code (paths under `/src//`), dropping system headers, vendored dependencies and test files; rank by dark covered-line counts, not percentages. +1. Filter to the project's own code (paths under `/src//`), dropping system headers and test files; rank by dark covered-line counts, not percentages. **Third-party code is not a target, unless the project maintains its own version of it**: an in-tree copy the project develops or patches (its own fork, locally applied patches, project-specific commits) counts as own code, while a vendored dependency kept unmodified in sync with its upstream does not. 2. **Verify build scope before trusting any percentage**: a high percentage can be an artefact of a partial build. Compare the upstream source tree's modules against the files present in the report; flag any module that exists upstream but is absent from the build, and treat "well covered" claims as invalid until the scope is verified. Enabling missing modules is a valid Phase 3 action. -3. Cross-check Fuzz Introspector for **statically reached but dynamically unreached** functions: reachable from harnesses but never hit in the coverage report. These are prime targets (seeds, dictionaries and harness adjustment to convert static reachability into real coverage). +3. Cross-check Fuzz Introspector for **statically reached but dynamically unreached** functions: reachable from harnesses but never hit in the coverage report. These are prime targets, but determine the cause before acting: a mutation-resistant gate justifies seeds or a dictionary; a harness or state limitation calls for harness adjustment. 4. Classify each candidate security-sensitive or not; the point is attack surface, not raw percentage. In scope: file-format parsers and loaders, wire-protocol handlers, deserialisers, decompression and codec paths, custom string and number parsing on untrusted data, path and filename handling, IPC message handling, state machines driven by untrusted input, crypto primitives. Out of scope, usually: diagnostic logging and printf-style output, UI code, config writing, build tooling and test helpers, unless untrusted data reaches them. -5. Record per target: the metric that flagged it, whether it is seed-limited or harness-limited, and the expected bug class. -### Phase 2.5: Coverage improvement obligation (mandatory when coverage is low) +5. Record per target: the metric that flagged it, whether it is seed-limited or harness-limited — for any seed- or dictionary-driven candidate, the specific mutation-resistant gates named with file:line — and the expected bug class. +### Phase 2.5: Worth-based improvement obligation -After live coverage triage, apply this gate: +After live coverage triage, apply this gate. Coverage percentage is a triage signal, never a pass/fail threshold: a run over named projects must process each of them whatever its coverage level (selection-based runs merely prioritise low-coverage candidates), and a low number raises the scrutiny of the triage without forcing work by itself. -- If the project's own-code line coverage is **below 15%**, the skill **must** produce at least one deliverable improvement pair before the project can be marked `delivered`. -- The obligation is recorded in the manifest as `coverage-obligation: ` and the target coverage goal. -- This overrides any "already viable" early exit. The agent is required to: - - Propose and implement at least one new fuzzer targeting a dark security-sensitive area, **or** - - Significantly expand the corpus + dictionary for existing fuzzers with measurable coverage gain (documented before/after `summary.json` or `reports-by-target`). -- Only after this obligation is satisfied (and validated via the quality gate in Phase 5) may the project be marked `delivered`. -- Projects already above 15% own-code coverage may exit early if no other work is needed. +- The obligation: every delivered pair must be **worthy** — projected to measurably improve the fuzzing coverage or effectiveness of the project's own security-sensitive code (worth model from fuzzing-coverage-targeting) — and its gain must be measured in Phase 5. The worth decision is recorded in the manifest. +- Where dark security-sensitive regions remain, deliver worthy pairs: a new or fixed fuzzer targeting them, **or** corpus + dictionary expansion only when a documented **gate analysis** proves the dark code is seed-limited (mutation-resistant gates: magic numbers, checksums, length fields, required keywords — cited with file:line) and the measured before/after gain (documented `summary.json` or `reports-by-target` numbers) is material. +- If the triage finds no worthy improvement — typically a project whose own code is already well fuzzed, with remaining dark regions limited to diagnostics, unmodified vendored third-party code or unreachable code — mark the project `no-worthy-targets` with the proof (top rejected regions, their numbers, why each fails the worth model). That is an accepted, honest outcome, in either direction: never imagine or stage unworthy work just to have something to deliver, and a high coverage number is never by itself an early exit — the worth analysis still runs and is recorded. +- Corpus/dictionary/options-only work is never mandatory and never filler: for small or trivially-reachable APIs (plain fuzzing crosses the entry checks within the quality-gate window), it has zero value and must be skipped with the evidence recorded. ### Phase 3: Act on the targets | Signal | Action | |---|---| | File dark because no harness reaches it | **New harness** (harness-limited) | -| Harness reaches it but the corpus never gets deep | **Seeds** (structured generation) and/or **dictionary** | -| Statically reached but dynamically unreached | **Seeds and dictionaries first**, then harness adjustment (state setup, enabling the path) | +| Harness reaches it but the corpus never gets deep | **Seeds** (structured generation) and/or **dictionary** — only after naming the mutation-resistant gate that blocks the mutator (file:line) | +| Statically reached but dynamically unreached | Determine the cause first: a mutation-resistant gate → **seeds/dictionary**; a harness or state limitation → **fix the harness** (state setup, enabling the path) | +| Dark code reachable within seconds by plain fuzzing (simple API, no gates) | **No corpus work**: seeds add nothing there; harness work or nothing | | Path disabled or held fixed in the harness | **Fix the harness** (fuzz what the attacker controls) | | Module missing from the build entirely | **Fix the build configuration** to compile it in | | Build configuration hides code from fuzzing | **Fix build configuration** (flags, `$LIB_FUZZING_ENGINE` wiring) | @@ -155,9 +153,9 @@ Choose the harness shape per **fuzzing-coverage-targeting**'s pattern catalogue ### Phase 4: Stage in worktree duo pairs Follow [references/worktree_delivery.md](references/worktree_delivery.md) precisely. Essentials: -- All remaining corpora, dictionaries and options changes with no fuzzer change are consolidated into a **single standalone pair** with the suffix `-corpus-dict-options`. +- Leftover corpora, dictionaries and options changes with no fuzzer change fold into a **single standalone pair** with the suffix `-corpus-dict-options` only when they pass the seed-limited gate analysis and show measured gain; otherwise they are dropped with recorded numbers. - A general build fix not tied to one fuzzer gets its own pair with the suffix `-build-fix`; build-configuration changes a specific fuzzer needs ride inside that fuzzer's pair. -- When the coverage obligation in Phase 2.5 is active, the primary deliverable type becomes **`-coverage-improvement`**. This pair may contain new fuzz targets, expanded corpora + dictionaries, and minor fuzz build configuration changes. It is the only pair type allowed to satisfy the coverage obligation. +- When Phase 2.5 identifies a worthy improvement, the primary deliverable type is **`-coverage-improvement`**: new fuzz targets, or — only with the documented gate analysis and measured gain — expanded corpora + dictionaries, plus the minor fuzz build configuration changes they need. It is the only pair type that discharges a worth obligation on its own. - A **duo pair is always created**, even when no OSS-Fuzz changes are needed, because all testing goes through the temporary copy bridge in the OSS-Fuzz side Dockerfile, and the main checkouts must stay clean. - Branches: OSS-Fuzz side `oss-fuzz--`, upstream side `-`; the identical `` is what pairs them. @@ -169,7 +167,7 @@ Follow [references/worktree_delivery.md](references/worktree_delivery.md) precis - Work each project's ranked target list **top-down until the rules are exhausted**: multiple pairs per project are the norm whenever worth remains, up to the cap. A single delivered pair is the floor for the project's delivered status, never a stopping point, and worth is measured per project, never compared across projects (one pair per project is a batch anti-pattern). Stop only on worth exhaustion (remaining marginal worth genuinely low, documented with numbers), the 5-pair cap (default, prompt-adjustable; improvement pairs, with a `-build-fix` pair not counting against it), or genuine budget exhaustion; budget-deferred targets are recorded as "pair N of project X: worth rationale written, not yet attempted", never as worth decisions. A project with genuinely no worthy targets is marked `no-worthy-targets` with its triage evidence; never invent a trivial pair to meet the floor. - A pair that fixes or adds a fuzzer **carries that fuzzer's corpora, dictionaries and options in the same pair**: a harness never ships separately from its own seeds and settings. -- All remaining corpora, dictionaries and options changes with no fuzzer change are consolidated into a **single standalone pair** with the suffix `-corpus-dict-options`. +- Leftover corpora, dictionaries and options changes with no fuzzer change fold into a **single standalone pair** with the suffix `-corpus-dict-options` **only** when they pass the seed-limited gate analysis and show measured gain; leftovers that fail it — single-character dictionary tokens, corpora for trivially-reachable APIs — are dropped with recorded numbers, never folded into a deliverable pair for its own sake. - A general build fix not tied to one fuzzer gets its own pair with the suffix `-build-fix`; build-configuration changes a specific fuzzer needs ride inside that fuzzer's pair. For a project whose main build was failing, the build-fix protocol in [references/worktree_delivery.md](references/worktree_delivery.md) applies: the fix pair is created and validated first, and every later pair copies the fix in purely as a testing scaffold, removing it again after conclusion. @@ -180,13 +178,13 @@ Validate each duo pair independently and one at a time, always through `infra/he 1. **Build matrix**: read `sanitizers:` and `fuzzing_engines:` from `projects//project.yaml` and validate every declared configuration (for example `--sanitizer address`, `undefined`, `memory`, and engines such as `--engine afl` where declared). 2. **Per-fuzzer gate (mandatory)**: `build_fuzzers` succeeds, and `check_build` passes for every new or updated fuzzer specifically, not just project-wide, **or its failure is solely attributable to a crash classified as a real finding** (documented with a reproducer and marked). -3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require **no crash, leak, OOM or timeout attributable to the fuzzer** (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. Every post-build crash goes through the classification procedure in the worktree reference: real findings keep the pair delivered with the discovery marked, fuzzer faults consume the fix budget, and a crash rooted in product code is always a finding, never a product-code patch. Sanitizer output is saved to disk in the per-project report area and only the decisive lines enter context; gate retries never re-read a whole report. -4. **Coverage growth, per fuzzer (fast path)**: for every fuzzer created or modified, compare the start and end `cov:` of its own quality-gate run: coverage must increase over the run, with `ft:` (features) and corpus size as supporting increase signals. No baseline build, no cross-build comparison and no separate coverage build are needed; a `--sanitizer coverage` build remains only as the per-file fallback for file-level numbers the report must show or marginal overlap decisions (see the worktree reference). Update the run's coverage ledger with the measured numbers and compute the pair's **incremental** contribution against what existing fuzzers and previously accepted pairs already cover: a pair whose gains mostly overlap is redesigned toward different dark regions, folded into standalone corpus work, or dropped, with the numbers recorded (rules from fuzzing-coverage-targeting). +3. **Fuzzing-quality gate (beyond check_build)**: for every new or updated fuzzer, corpus, dictionary or option change, actually fuzz and require **no crash, leak, OOM or timeout attributable to the fuzzer** (nothing within the first 60 seconds) and coverage that keeps increasing rather than going stale (no growth in the final half of the run counts as stale). Minimum run 120 seconds per target, 300 seconds for corpus-heavy changes. Every post-build crash goes through the classification procedure in the worktree reference: real findings keep the pair delivered with the discovery marked, fuzzer faults consume the fix budget, and a crash rooted in product code is always a finding, never a product-code patch. Sanitizer output is saved to disk in the per-project report area and only the decisive lines enter context; gate retries never re-read a whole report. Corpus-, dictionary- and options-only changes additionally require their **value gate**: the gate analysis on file (what blocks progress, cited file:line, why coverage-guided mutation cannot cross it cheaply); dictionary tokens that are multi-byte and traceable to source constructs (format tags, magic words, keywords) — a dictionary dominated by single-character tokens fails the pair outright; every seed validated to parse, pass the entry checks and measurably move the targeted dark file's covered-line count (per-file fallback numbers when the global `cov:` trend hides it); and the zero-value test — if the unseeded run reaches the same code within the 120 s/300 s window, the pair is rejected as worthless with the numbers recorded. +4. **Coverage growth, per fuzzer (fast path)**: for every fuzzer created or modified, compare the start and end `cov:` of its own quality-gate run: coverage must increase over the run, with `ft:` (features) and corpus size as supporting increase signals. No baseline build, no cross-build comparison and no separate coverage build are needed; a `--sanitizer coverage` build remains only as the per-file fallback for file-level numbers the report must show or marginal overlap decisions (see the worktree reference). Update the run's coverage ledger with the measured numbers and compute the pair's **incremental** contribution against what existing fuzzers and previously accepted pairs already cover: a pair whose gains mostly overlap is redesigned toward different dark regions, folded into standalone corpus work (only when the region is provably seed-limited by a mutation-resistant gate), or dropped, with the numbers recorded (rules from fuzzing-coverage-targeting). 5. **Fix budget of 3 rounds**: fix, retest, and at most three attempts per gate failure. A pair still failing after the third round (compile failure, staleness, early crash or any other gate) is **rejected**: move both worktrees into the rejected area with `git worktree move` (never delete), keep the branches, and record the failure reason, what was attempted and the new paths. 6. Deliver: remove any copied build-fix changes and verify the remaining diff is purely this pair's own work (see the worktree reference), finalise the per-project report and the manifest row, and add the row to the final summary table. 7. **Exit protocol and relay**: before a session exits, update the manifest (pairs delivered, remaining ranked targets with rationales, terminal `STOP:` line per the format in Phase 0); when context or time runs low, checkpoint cleanly rather than wrapping up shallowly, because the batch driver relaunches continuation sessions until a terminal stop or the session cap. -Portfolio loop: work each project's ranked list top-down to the **cap of 5 pairs per project** by default (prompt-adjustable; improvement pairs, with a `-build-fix` pair not counting against it), stopping earlier only on documented worth exhaustion (remaining marginal worth genuinely low, with numbers), and recording genuine budget exhaustion as deferred entries ("pair N of project X: worth rationale written, not yet attempted"), never as worth decisions. Disk discipline: check available disk before builds; when low, stop cleanly with a `budget-exhausted (disk)` stop and deferred entries rather than continuing. +Portfolio loop: work each project's ranked list top-down to the **cap of 5 pairs per project** by default (prompt-adjustable; improvement pairs, with a `-build-fix` pair not counting against it), stopping earlier only on documented worth exhaustion (remaining marginal worth genuinely low, with numbers), and recording genuine budget exhaustion as deferred entries ("pair N of project X: worth rationale written, not yet attempted"), never as worth decisions. Corpus/dict-only pairs are never portfolio filler: documented worth exhaustion is always preferred over a corpus pair staged just to have something to deliver. Disk discipline: check available disk before builds; when low, stop cleanly with a `budget-exhausted (disk)` stop and deferred entries rather than continuing. ## Definition of done @@ -194,7 +192,7 @@ Portfolio loop: work each project's ranked list top-down to the **cap of 5 pairs - [ ] Quality gate clean for every new or changed target: 120 s minimum, nothing fuzzer-attributable (crash, leak, OOM, timeout) within the first 60 seconds, no staleness; real findings excepted and marked. - [ ] Coverage: documented gains on targeted files and zero digression on other project-own files (like-for-like measurement). - [ ] Duo pairs hold plain working-tree changes; nothing staged (unless justified and recorded), nothing committed, nothing pushed. -- [ ] Grouping respected: fuzzer pairs carry their own corpora, dictionaries and options; standalone corpus work sits in one pair; dependent pairs have their copied build-fix changes removed and verified. +- [ ] Grouping respected: fuzzer pairs carry their own corpora, dictionaries and options; a standalone corpus pair exists only with a documented gate analysis and measured gain (no single-character dictionaries, no corpora for trivially-reachable APIs); dependent pairs have their copied build-fix changes removed and verified. - [ ] Portfolio respected: ranked multi-target selection via fuzzing-coverage-targeting, incremental worth and overlap rule applied, cap honoured (default 5 pairs per project unless the user's prompt overrides it). - [ ] Upstream diffs contain only allowed changes: fuzzers with their corpora, dictionaries and options, fuzz build configuration, and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions; no product code touched. - [ ] Bridges present and tagged; rejected pairs moved to the rejected area (not deleted); skipped projects recorded with reasons. diff --git a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md index d593a341a028..1f55ff61c808 100644 --- a/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md +++ b/infra/experimental/agent-skills/oss-fuzz-coverage-pipeline/references/worktree_delivery.md @@ -43,10 +43,11 @@ A project is delivered through one or more duo pairs: | Pair type | Contents | Count per project | |---|---|---| | Fuzzer pair | A fixed or new harness plus its own corpora, dictionaries and options, plus any build-configuration change that specific harness needs | Zero or more | -| Standalone corpus pair | All corpora, dictionaries and options changes with no fuzzer change | Zero or one, suffix `-corpus-dict-options` | +| Coverage-improvement pair | The pair discharging a Phase 2.5 worth obligation: new fuzz targets, or gate-justified corpus/dictionary expansion, plus the fuzz build configuration they need | Zero or more, suffix `-coverage-improvement` | +| Standalone corpus pair | Leftover corpora, dictionaries and options changes with no fuzzer change — staged only when the seed-limited gate analysis passes and the gain is measured | Zero or one, suffix `-corpus-dict-options` | | Build-fix pair | A general build fix not tied to one fuzzer | Zero or one, suffix `-build-fix` | -Work the ranked target list top-down per project until the rules are exhausted; a single delivered pair is the floor for delivered status, never a stopping point, and worth is never compared across projects. Worth is your own judgement, but it must be backed by the written Phase 2 rationale (dark covered lines, security sensitivity, reachability, expected impact), recorded per pair. Stop a project only on documented worth exhaustion (with numbers), the 5-pair cap, or genuine budget exhaustion, recorded as deferred entries ("pair N of project X: worth rationale written, not yet attempted"), never as worth decisions. A project with no worthy targets is marked `no-worthy-targets` with its triage evidence; never fabricate a minimal pair. The declaration must be proven, not asserted: list the top rejected candidate regions with their numbers and the reason each fails the worth model, and treat it as seriously as a failure exit. Do not fragment trivially: corpus, dictionary and option tweaks that are not fuzzer-related always fold into the standalone pair. +Work the ranked target list top-down per project until the rules are exhausted; a single delivered pair is the floor for delivered status, never a stopping point, and worth is never compared across projects. Worth is your own judgement, but it must be backed by the written Phase 2 rationale (dark covered lines, security sensitivity, reachability, expected impact), recorded per pair. Stop a project only on documented worth exhaustion (with numbers), the 5-pair cap, or genuine budget exhaustion, recorded as deferred entries ("pair N of project X: worth rationale written, not yet attempted"), never as worth decisions. A project with no worthy targets is marked `no-worthy-targets` with its triage evidence; never fabricate a minimal pair, and this includes an already well-fuzzed project: a high coverage number never forces work, and admitting that fuzzing can no longer be worthily improved — with the proof — is an accepted outcome. The declaration must be proven, not asserted: list the top rejected candidate regions with their numbers and the reason each fails the worth model, and treat it as seriously as a failure exit. Do not fragment trivially: corpus, dictionary and option tweaks that are not fuzzer-related fold into the standalone pair when they pass the seed-limited gate analysis; worthless leftovers are dropped with recorded numbers, never folded in just to have something to deliver. Upstream worktrees contain only allowed changes: fuzzers with their corpora, dictionaries and options, fuzz build configuration, and `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION`-guarded additions. Upstream product code is never modified; product-code bugs are findings. The OSS-Fuzz worktrees carry the integration changes (Dockerfile, `build.sh`, `project.yaml`) plus the bridge. @@ -125,6 +126,8 @@ Quality thresholds, which apply on top of `check_build` passing, and to corpora, - coverage must keep increasing: no growth in the final half of the run counts as stale (compare the `cov:` figures between libFuzzer checkpoints in the run log); - record the evidence (run logs and the coverage trend) in the per-project report. +Corpus-, dictionary- and options-only changes must additionally pass the value gate (Phase 5): documented mutation-resistant gates cited with file:line, multi-byte dictionary tokens traceable to source constructs (no single-character tokens), per-seed validated gain on the targeted dark files, and a demonstrated gain over what plain fuzzing reaches unseeded within the run window — otherwise the change is dropped as zero-value, with the numbers recorded. + These libFuzzer-style flags apply to every OSS-Fuzz engine: Jazzer, Atheris and Go-native fuzzing all wrap libFuzzer. ### Coverage evaluation: per-fuzzer growth by default diff --git a/infra/experimental/agent-skills/oss-fuzz-engineer/references/structured_seed_generation.md b/infra/experimental/agent-skills/oss-fuzz-engineer/references/structured_seed_generation.md index f424cf9b634e..47b3cf17b18a 100644 --- a/infra/experimental/agent-skills/oss-fuzz-engineer/references/structured_seed_generation.md +++ b/infra/experimental/agent-skills/oss-fuzz-engineer/references/structured_seed_generation.md @@ -154,6 +154,20 @@ A generator is a natural place to also emit libFuzzer dictionaries the mutator synthesize tokens it would rarely discover byte-by-byte. VLC emits both seeds and `dictionaries/*.dict` from the same script. +## When seeds and dictionaries have ~zero value + +- **Entry checks plain fuzzing crosses within seconds** — small APIs, + one- or two-byte comparisons, a single length check: coverage-guided + mutation gets through on its own, so seeds and dictionaries add nothing. + Confirm the gate is genuinely mutation-resistant (multi-byte magic, + checksums, table lookups) before generating anything. +- **Single-character dictionary tokens** are worthless by definition: the + mutator discovers single bytes trivially. Filter them out; a dictionary + dominated by them means the gate analysis failed and no dictionary is + warranted. +- **Already well-covered or harness-limited code** (see below): no seed can + help; record that instead of generating anyway. + ## Seed-limited vs harness-limited code Before generating seeds, confirm the dark code is actually reachable by an @@ -199,6 +213,8 @@ generation, but the seeds you built remain a valuable starting corpus for it. - [ ] Targets chosen from `summary.json` (reachable, low coverage), not intuition. - [ ] Confirmed the dark code is seed-limited, not harness-limited. +- [ ] Gates confirmed genuinely mutation-resistant (not trivially fuzzable), so the seeds are not zero-value. +- [ ] Dictionary tokens are multi-byte and traceable to source constructs; no single-character tokens. - [ ] Generator is deterministic and emits small, minimal-but-valid seeds. - [ ] Each seed validated: it parses *and* moves the intended file's coverage. - [ ] Seeds appended to existing corpora (never replaced); script copied in via Dockerfile.