Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions infra/experimental/agent-skills/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Skills and tooling that let an agent CLI (Gemini CLI or Claude Code) write,
build, and extend OSS-Fuzz fuzzing integrations. The folder ships:

- Six skills that give an agent OSS-Fuzz-specific knowledge.
- Eight skills that give an agent OSS-Fuzz-specific knowledge.
- `infra/experimental/agent-skills/helper.py`, a wrapper that launches agent
sessions over one or more OSS-Fuzz projects in parallel.
- `copy_to_global.sh`, an installer that places the skills where your
Expand All @@ -12,6 +12,9 @@ build, and extend OSS-Fuzz fuzzing integrations. The folder ships:
Before running anything in this folder, review the [threat model](#threat-model).
This tooling is experimental and runs agents in unrestricted modes.

For the full walkthrough (install the skills, run the coverage pipeline,
verify results) see [USAGE.md](USAGE.md).

## Contents

| Item | Purpose |
Expand All @@ -22,8 +25,10 @@ This tooling is experimental and runs agents in unrestricted modes.
| `fuzzing-jvm-expert/` | Skill: fuzz JVM projects (Java/Kotlin/Scala) with Jazzer |
| `fuzzing-python-expert/` | Skill: fuzz Python projects with Atheris |
| `oss-fuzz-engineer/` | Skill: OSS-Fuzz infra workflows (integrate, fix, extend) |
| `oss-fuzz-coverage-pipeline/` | Skill: language-agnostic end-to-end coverage pipeline wrapping the language fuzzing skills (selects projects by glob/wildcard or all, auto-detects language from project.yaml, fixes builds, triages coverage, improves targets, stages paired worktree branches; ships `scripts/select_projects.sh`) |
| `fuzzing-coverage-targeting/` | Skill: which fuzz targets to create and what shape each harness takes (worth model, dark-region taxonomy, harness patterns, portfolio with overlap rules; default cap 5 pairs per project) |
| `infra/experimental/agent-skills/helper.py` | Driver that launches agent sessions per OSS-Fuzz project |
| `copy_to_global.sh` | Installs the skills into `~/.gemini/skills` or `~/.claude/skills` |
| `copy_to_global.sh` | Installs the skills into `~/.gemini/skills`, `~/.claude/skills`, or any skills directory you pass as an argument (e.g. Cline's `.cline/skills`) |

See each skill's `SKILL.md` for the detailed guidance the agent receives.

Expand All @@ -42,7 +47,13 @@ See each skill's `SKILL.md` for the detailed guidance the agent receives.

```bash
# 1. Install the skills into your agent CLI.
./copy_to_global.sh gemini # or: ./copy_to_global.sh claude
./copy_to_global.sh # defaults to claude; or: ./copy_to_global.sh gemini
# Any other argument is used as the skills directory itself (any SKILL.md-
# compatible agent, e.g. Cline's workspace directory; relative paths resolve
# against your current directory):
./copy_to_global.sh /path/to/vscode-workspace/.cline/skills
# Multiple targets in one invocation are fine:
# ./copy_to_global.sh gemini claude ../.cline/skills

# 2. Confirm the agent CLI is reachable.
gemini --version # or: claude --version
Expand Down Expand Up @@ -84,6 +95,7 @@ full flag listings. A summary of the available subcommands:
| `fix-builds` | Diagnose and fix broken project builds | `python infra/experimental/agent-skills/helper.py fix-builds htslib` |
| `fix-broken-builds` | Auto-discover broken projects from the public build status and fix up to `--limit` of them (C/C++ by default) | `python infra/experimental/agent-skills/helper.py fix-broken-builds --limit 50` |
| `run-task` | Run an arbitrary `--task` string per project | `python infra/experimental/agent-skills/helper.py run-task --task "Add a harness for the XML attribute parser" open62541` |
| `coverage-pipeline` | Run the full coverage pipeline (skills-driven, prompt generated automatically) over named or `--broken`-discovered projects; `--cap N` sets the pair cap; relay continuation sessions via `--max-sessions-per-project` and `--per-project-minutes` | `python infra/experimental/agent-skills/helper.py coverage-pipeline --broken --limit 10` |
| `add-chronos-support` | Add Chronos support to a project | `python infra/experimental/agent-skills/helper.py add-chronos-support json-c` |
| `integrate-project` | Onboard a new project from a Git URL | `python infra/experimental/agent-skills/helper.py integrate-project https://github.com/org/repo` |
| `clean` | Remove local artifacts from previous agent runs | `python infra/experimental/agent-skills/helper.py clean open62541` |
Expand Down
253 changes: 253 additions & 0 deletions infra/experimental/agent-skills/USAGE.md

Large diffs are not rendered by default.

93 changes: 55 additions & 38 deletions infra/experimental/agent-skills/copy_to_global.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,50 +15,67 @@
#
################################################################################

# Copy the skills folder into the global skills directory

# Accept user input. Should by default be "gemini" but can be overridden by the user.
SKILLS_DIR=${1:-"gemini"}
# Install the skills in this folder into an agent's skills directory.
#
# Usage:
# ./copy_to_global.sh [target ...]
#
# Targets:
# claude -> ~/.claude/skills (default when no target is given)
# gemini -> ~/.gemini/skills
# <directory> -> any other argument is used as the skills directory
# itself (created if missing; resolved relative to your
# current working directory). This works for any agent
# that reads SKILL.md-style skills, e.g. Cline's
# workspace directory:
# ./copy_to_global.sh /path/to/vscode-workspace/.cline/skills
#
# Multiple targets may be given in one invocation, e.g.:
# ./copy_to_global.sh gemini claude /abs/path/.cline/skills
#
# Skills are auto-discovered: every directory next to this script that
# contains a SKILL.md is copied. Existing skills of the same name in the
# target directory are overwritten.

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Gemini skills dir = ~/.gemini/skills
# Claude skills dir = ~/.claude/skills
if [ "$SKILLS_DIR" == "gemini" ]; then
GLOBAL_SKILLS_DIR="$HOME/.gemini/skills"
elif [ "$SKILLS_DIR" == "claude" ]; then
GLOBAL_SKILLS_DIR="$HOME/.claude/skills"
else
echo "Invalid skills directory specified. Use 'gemini' or 'claude'."
# Auto-discover the skills next to this script.
SKILLS=()
for entry in "$SCRIPT_DIR"/*/; do
if [ -f "${entry}SKILL.md" ]; then
SKILLS+=("$(basename "$entry")")
fi
done
if [ ${#SKILLS[@]} -eq 0 ]; then
echo "No skills found in $SCRIPT_DIR" >&2
exit 1
fi

# Log target skill directory
echo "Copying skills to global skills directory: $GLOBAL_SKILLS_DIR"

# Resolve a target name to a skills directory. Unknown names are treated as
# directory paths so that new agents work without script changes.
resolve_target() {
case "$1" in
gemini) echo "$HOME/.gemini/skills" ;;
claude) echo "$HOME/.claude/skills" ;;
*) echo "$1" ;;
esac
}

# Now copy each of the skills from the local "skills" directory to the global skills directory
# Overwrite existing skills with the same name.
# Check if the global skills directory exists, if not create it
if [ ! -d "$GLOBAL_SKILLS_DIR" ]; then
mkdir -p "$GLOBAL_SKILLS_DIR"
# Default to claude when no target is given.
if [ $# -eq 0 ]; then
set -- claude
fi

# Copy each skill from the local "skills" directory to the global skills directory
# Make sure we work from this scripts base folder and copy each of the skills
# fuzzing-memory-unsafe-expert
# fuzzing-go-expert
# fuzzing-rust-expert
# fuzzing-jvm-expert
# fuzzing-python-expert
# oss-fuzz-engineer
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
for skill in "fuzzing-memory-unsafe-expert" "fuzzing-go-expert" "fuzzing-rust-expert" "fuzzing-jvm-expert" "fuzzing-python-expert" "oss-fuzz-engineer"; do
abs_skill="$SCRIPT_DIR/$skill"
# Copy over the skill and replace any existing skill with the same name in the global skills directory
if [ -d "$GLOBAL_SKILLS_DIR/$skill" ]; then
rm -rf "$GLOBAL_SKILLS_DIR/$skill"
fi

echo "Copying $abs_skill to $GLOBAL_SKILLS_DIR/"
cp -r "$abs_skill" "$GLOBAL_SKILLS_DIR/"
for target in "$@"; do
GLOBAL_SKILLS_DIR="$(resolve_target "$target")"
echo "Installing ${#SKILLS[@]} skills to: $GLOBAL_SKILLS_DIR"
mkdir -p "$GLOBAL_SKILLS_DIR"
for skill in "${SKILLS[@]}"; do
# Replace any existing skill with the same name in the target directory.
if [ -d "$GLOBAL_SKILLS_DIR/$skill" ]; then
rm -rf "$GLOBAL_SKILLS_DIR/$skill"
fi
echo " copying $skill"
cp -r "$SCRIPT_DIR/$skill" "$GLOBAL_SKILLS_DIR/"
done
done
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
name: fuzzing-coverage-targeting
description: >-
Use this skill to decide which fuzz targets a project needs and what shape each harness should take: rank dark reachable regions from coverage data into a diversified portfolio of worthy harnesses (any language), using a worth model, a dark-region taxonomy, harness pattern selection (parser, round-trip, property, stateful, option-matrix, component-API) and overlap-aware diminishing returns with a default cap of 5 pairs per project. Use alongside the language fuzzing skills (which say how to write a harness) and the OSS-Fuzz coverage pipeline (which runs the process).
---

# Fuzzing coverage targeting

This skill is the architect for fuzz target selection: it answers **which** parts of a project are worth harnessing and **what shape** each harness should take. It does not write harnesses (that is the language skills' craft), it owns no process state (that is the coverage pipeline's job), and it issues no commands (that is oss-fuzz-engineer's territory). It is language-agnostic and works standalone whenever a task is about choosing or planning fuzz targets.

## Inputs

- Fresh coverage data: the project's `summary.json` and per-target reports (`reports-by-target`), filtered to the project's own code with build scope verified (a high percentage over a partial build is meaningless). Own code includes versions of third-party code the project maintains in-tree (its fork or locally patched copies); vendored dependencies kept unmodified from their upstream are not targets.
- Fuzz Introspector data: statically reachable but dynamically unreached functions, optimal targets, blockers.
- The project's source structure: modules, public APIs, state machines, configuration surfaces.
- The language expert's threat model, when composed by the pipeline: attacker-reachable surfaces are the top ranking weight among equals.

## Worth model

Score each dark region as **dark reachable lines × functional importance × bug-class diversity ÷ harness cost**:

- **Dark reachable lines**: lines statically reachable from some harness entry but uncovered, counted as absolute lines, not percentages.
- **Functional importance**: core parsing, transformation, protocol logic and data handling rank high; diagnostics, debug printers, test helpers, build-time code and unmodified vendored third-party dependencies are excluded outright.
- **Bug-class diversity**: memory safety (ASan/MSan/UBSan), logic and assertion violations, hangs and pathological blowups, and leaks. A region rich in several classes outranks a one-class region of equal size.
- **Harness cost**: API complexity, state setup, corpus construction effort.

A region plain fuzzing already reaches within a normal run carries **zero incremental worth for corpus-only work**: seeds and dictionaries pay only where a mutation-resistant gate (multi-byte magic numbers, checksums, length fields, required keywords) holds the mutator out.

## Dark-region taxonomy and harness opportunity types

| Dark region type | Typical shape | Example surfaces |
|---|---|---|
| Parser or decoder entry | bytes-in harness | file-format loaders, wire-protocol frames, deserialisers |
| Transformation or codec stage | round-trip or stage harness | encode/decode pairs, colour conversion, filters, re-serialisation |
| Stateful session or protocol object | stateful sequence harness | connection objects, protocol state machines, transactions |
| Internal component API | component harness | regex engines, expression evaluators, container and data-structure operations, maths kernels |
| Configuration or option surface | option-matrix harness | format variants, feature flags, mode switches |
| Encode or serialise direction | round-trip harness | writers and serializers (often dark because only the decode side gets fuzzed) |
| Text or language frontend | structured grammar-ish harness | compilers, template engines, query languages |
| Error and recovery paths | mutation of valid seeds | malformed variants of legitimate inputs |

The taxonomy is the antidote to the "one parser pair per project" outcome: every module shape is a candidate, not only attacker-bytes entry points.

## Harness pattern catalogue

- **Bytes-into-parser** (classic): feed the buffer to the decode entry point; the language skills' craft sections cover it fully.
- **Round-trip**: encode a structured input, decode it back, assert the invariant (identity, or documented lossiness):

```cpp
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
FuzzedDataProvider fdp(data, size);
auto obj = Thing::parse(fdp.ConsumeRemainingBytesAsString());
std::string out = obj.serialize(); // dark encode direction
Thing round = Thing::parse(out); // decoder re-entry
if (obj != round) abort(); // invariant: stable round-trip
return 0;
}
```

- **Property or invariant / differential**: check a documented property of one implementation (cheap: commutativity, idempotence, bounds, canonical-form equality) or compare against a reference implementation (expensive but powerful).
- **Stateful sequence**: build a session object, then drive N API calls from the fuzzed bytes; required for protocol and connection logic that no single call exercises:

```cpp
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
FuzzedDataProvider fdp(data, size);
Session s; // state machine under test
while (fdp.remaining_bytes() > 0) {
uint8_t op = fdp.ConsumeIntegralInRange<uint8_t>(0, Session::kMaxOp);
std::vector<uint8_t> arg =
fdp.ConsumeBytes<uint8_t>(fdp.ConsumeIntegralInRange<int>(0, 64));
s.step(op, arg); // ignore documented errors
}
return 0;
}
```

- **Option or configuration matrix**: consume small values as option flags and mode switches before the payload so one target sweeps many compiled paths cheaply; pair with a dictionary of option names (multi-byte tokens traceable to the code; never single characters).
- **Component API**: call the internal module through its intended API with typed, bounded parameters; the language skills' legitimate-input requirement applies (construct valid objects; never feed raw bytes where a struct is expected).

## Portfolio construction

1. Enumerate dark regions from the inputs; classify each by the taxonomy; score each by the worth model.
2. Apply the security weighting: attacker-reachable surfaces rank first among equals; functional dark logic is fully worthy; diagnostics, logging, printf-style output, test helpers and build-time code stay excluded.
3. Produce a **ranked multi-target list** expected to yield several focused harnesses rather than one or two; a single-target list is a smell that the taxonomy was not applied. If honest triage finds nothing worthy, say so: zero worthy targets is a valid outcome — including for an already well-fuzzed project — and inventing a trivial pair to hit a one-pair floor is forbidden. The declaration must be proven, not asserted: list the top rejected regions with their numbers and the reason each fails the worth model.
4. Selection is iterative with diminishing returns, executed by the pipeline with its coverage ledger: after each accepted pair, re-rank the remaining targets by **incremental** worth (projected gains minus already-covered ground). A pair whose gains mostly overlap already-accepted coverage is redesigned toward different dark regions, folded into standalone corpus work — only when the region is provably seed-limited by a mutation-resistant gate — or dropped, with the numbers recorded.
5. **Default cap: at most 5 pairs per project** (fuzzer pairs plus the standalone corpus pair; a `-build-fix` pair does not count against the cap), unless the user raises or lowers the threshold in their prompt. The standalone corpus pair is exceptional — justified only by a proven seed-limited gate — and is never staged as portfolio filler.
6. **Work the ranked list top-down until the rules are exhausted.** Worth is measured per project and never compared across projects: in a multi-project batch, a second pair for project A competes only with A's own remaining dark regions, never with project B's candidates, and spreading one pair per project is an anti-pattern. A single delivered pair is only the floor for a project's delivered status, never a stopping point. Stop a project's loop only on one of: **(a) worth exhaustion**, the remaining marginal worth is genuinely low, documented with numbers; **(b) the cap**; or **(c) budget exhaustion**, when time or build budget truly runs out. Budget bounds ordering and depth only, never worth: record un-attempted worthy targets as "pair N of project X: worth rationale written, not yet attempted", never as a worth decision, and never drop a target as unworthy for budget reasons.

## Non-goals

- No harness writing: the language skills own craft and idioms.
- No process, sequencing, acceptance or state: the coverage pipeline owns the manifest, the ledger and every decision.
- No infrastructure commands: oss-fuzz-engineer owns `infra/helper.py` usage and report fetching.
Loading
Loading