Report unsafe agent behavior or evaluation-isolation bypasses privately when public disclosure could expose users or systems. Security fixes are applied to the latest revision on the main branch.
Report a vulnerability · Trust boundaries · Evaluation safeguards
GitHub private vulnerability reporting is currently disabled for this repository. Arrange a private channel with a maintainer before sharing sensitive details. If you cannot reach a maintainer directly, open an issue requesting a private contact route without describing the vulnerability or attaching reproduction payloads.
Caution
Never include live credentials, private repository contents, or production customer data in a report. Public documentation typos or non-security defects belong in a standard GitHub issue.
Once a private reporting channel is established, provide the minimum evidence needed to reproduce the issue:
| Field | Required detail |
|---|---|
| Affected surface | File path, section heading, and commit SHA or version (1.1.0). |
| Reproduction | Minimal prompt and sanitized fixture files. |
| Harness environment | Coding agent CLI/IDE, enabled tools, sandbox mode, and permission flags. |
| Observed behavior | The unsafe recommendation, instruction-injection path, or isolation breach with a sanitized trace. |
| Impact | Violated boundary and affected data, filesystem path, or external system. |
| Surface | Security responsibility |
|---|---|
Distributed skill (skills/system-design-compass/) |
Pure Markdown guidance (SKILL.md and 14 reference files) plus static UI metadata (agents/openai.yaml). Contains no executable scripts, pre-approved allowed-tools, credentials, or runtime network calls. |
| Agent harness | Enforces tool execution policies, filesystem sandboxing, network access, and user approval gates. Markdown instructions are advisory context, not an OS access-control boundary. |
| User request | Supplies explicit authorization for code changes or external actions. Skill recommendations never expand user-granted permissions. |
Repository evaluation tools (tools/*.py) |
Local development scripts that launch agent CLIs (codex or agy) inside temporary directories; never packaged or installed with the skill. |
Running tools/evaluate_skill.py or tools/evaluate_agy.py executes model turns against fixture files. The evaluation harness enforces multiple defense-in-depth checks, alongside explicit boundaries that operators must understand:
- Fixture path sanitization (
safe_path): Before writing case fixtures into a temporary workspace,tools/evaluate_skill.pyrejects any path that escapes the workspace root, touches.codex,.agents, or.git, or attempts to overwriteAGENTS.md,AGENTS.override.md,SKILL.md, or theskill/directory. - Per-invocation catalog isolation:
evaluate_skill.pydisables host skill discovery, plugins, apps, hooks, memories, andAGENTS.mdper command without mutating global user configuration. Because CLI flags cannot block direct filesystem reads of home-directory skills, command traces (including escaped Windows paths) are audited and any foreign skill read is markedcontaminated. - Workspace mutation detection: A cryptographic SHA-256 digest (
digest(workspace)) is computed before and after each subject run. Any unexpected file creation or edit marks the runworkspace-mutatedand excludes it from grading. - Tool-free blind grading: Grader invocations treat subject answers strictly as untrusted data (
inline_context) with tools, skills, plugins, and MCP servers disabled (tools: []inevaluate_agy.py). - Sandbox selection:
read-onlyis the default sandbox inevaluate_skill.py. When Windows environments block shell reads inread-onlymode, operators may pass--sandbox danger-full-accessonly for trusted authored fixtures. Workspace digests cannot prevent or detect side effects outside the temporary directory.
See the Comparative evaluation protocol for complete isolation rules, status codes, and trace-auditing procedures.