Generate a reusable evaluation and safety baseline

.template/bootstrap/build-skill-from-intake.md

@@ -136,7 +136,8 @@ skills/
     `-- assets/
 
 tests/
-`-- fixtures/
+|-- fixtures/
+`-- evals/
 ```
 
 Do not create empty folders unless they clarify the intended structure or contain `.gitkeep`.
@@ -156,6 +157,8 @@ Required files:
 - `docs/GITHUB-CLI-DELIVERY.md`
 - `docs/INSTALL.md` as a compatibility pointer to root `INSTALL.md`
 - `docs/ARCHITECTURE.md`
+- `docs/TESTING.md`
+- `docs/THREAT-MODEL.md`
 - `docs/RELEASING.md`
 - `docs/VERSION.md`
 - `CHANGELOG.md`
@@ -174,6 +177,9 @@ The new file must:
 - State the maintenance goal in the summary.
 - Identify `skills/<name>/SKILL.md` as the canonical skill entry point.
 - Identify `tests/fixtures/` as maintenance evidence that must remain outside runtime delivery.
+- Require `tests/evals/cases.json` to register maintained activation, workflow, verification, and applicable adversarial cases.
+- Route behavior and security-sensitive changes to `docs/TESTING.md` and `docs/THREAT-MODEL.md`.
+- Keep the canonical runtime map and `tests/fixtures/agent-surface-contract.json` synchronized with the generated skill tree and maintenance instructions.
 - Route requests to update the skill to a focused direct reference that checks source metadata and uses folder targeting when needed.
 - Define any specialized terms with entity-bias risk.
 - Tell agents how to update references, fixtures, docs, packaging, and release notes.

.template/bootstrap/cleanup-and-boundaries.md

@@ -31,6 +31,8 @@ Rewrite these files before deleting `.template/`:
 - `README.md`
 - `AGENTS.md`
 - `docs/ARCHITECTURE.md`
+- `docs/TESTING.md`
+- `docs/THREAT-MODEL.md`
 - `docs/RELEASING.md`
 - `CONTRIBUTING.md`
 - `SUPPORT.md`
@@ -63,6 +65,7 @@ It should include:
 - The skill's core principles, each with the rationale it protects.
 - A decision stance for how a future agent should act when the exact instruction does not fit.
 - Required validation commands.
+- Conditional read guidance for behavioral testing and the threat model.
 - A short statement of how changes land, pointing to `docs/RELEASING.md`.
 - Skill reference organization rules.
 - Boundaries for raw intake, temp files, and release artifacts.
@@ -185,6 +188,8 @@ Before declaring cleanup complete, verify:
 - `AGENTS.md` describes maintenance mode.
 - Exactly one `skills/<name>/SKILL.md` exists, its names agree, and it does not reference `.template/`.
 - Root `tests/fixtures/` contains maintenance cases and no fixture directory exists inside the runtime skill.
+- The activation placeholders are replaced, every fixture case is registered, and the canonical runtime map matches the generated skill tree.
+- `docs/TESTING.md` and `docs/THREAT-MODEL.md` describe the generated skill's actual behavior and capability risks.
 - Root `INSTALL.md` and `docs/GITHUB-CLI.md` contain the final public identity, complete copyable commands, and no bootstrap placeholders.
 - All three generated workflows are installed, including the public GitHub CLI install verifier.
 - `.skill-template-feedback/` exists with a tracked `README.md` and `.gitkeep`.

.template/bootstrap/skill-quality-standard.md

@@ -108,6 +108,12 @@ Use root `tests/fixtures/` for prompts or examples that verify the skill after c
 
 Rationale: Fixtures preserve behavioral expectations across maintenance changes. They are evidence that the skill still works, not part of normal execution.
 
+Replace the template's generic activation prompts with natural domain requests. Register activation, handoff, workflow, verification, and applicable adversarial cases in `tests/evals/cases.json`. Update the canonical runtime map and `tests/fixtures/agent-surface-contract.json` during cleanup.
+
+Read `docs/TESTING.md` before changing activation, workflow order, output behavior, safety boundaries, handoff, or progressive disclosure. Read `docs/THREAT-MODEL.md` before adding access, mutation, automation, private-derived public output, or another privileged capability.
+
+Deterministic validation proves structure and linkage, model-based evaluation observes behavior in a named environment, and human confirmation accepts meaning and usefulness. Do not substitute one evidence layer for another.
+
 ## Script Rules
 
 When a generated skill ships runnable scripts, comment them to explain why the code does what it does and any non-obvious constraints, not to narrate what each line does. Prefer structured docblock tags over prose so the comments stay parseable and stay out of search noise.

CHANGELOG.md

@@ -5,6 +5,7 @@
 - Make generated releases deterministic across operating systems, verify staged and archived bytes, guard immutable tags and drafts, and add a final-tree preflight.
 - Make GitHub CLI delivery a release gate with exact-tag installation, published installation, previous-release update testing, and installed-tree equality checks.
 - Normalize repository text to LF, reject escaped generated workflow expressions, and keep active and generated workflows on the same Node setup action.
+- Add a generated evaluation registry with activation, workflow, adversarial, and agent-surface drift fixtures plus testing and threat-model guidance.
 - Replace the incompatible `src/SKILL.md` source with one standard `skills/<name>/SKILL.md` tree and keep maintenance fixtures outside the installed runtime.
 - Add GitHub CLI publisher validation, public install verification, checksums, attestations, root installation guidance, beginner orientation, and a focused runtime update scaffold.
 - Make validation, packaging, archive fallback, and installed tree verification discover the generated skill dynamically and remain portable across operating systems.

docs/ARCHITECTURE.md

@@ -54,6 +54,16 @@ Generated skill workflows are installed from `.template/generated/.github/workfl
 
 The reason `.template/` is deleted is not tidiness. It prevents future agents from optimizing for bootstrap goals after the repository's purpose has changed.
 
+### Canonical Runtime Map
+
+Update this generated block whenever the runtime tree changes. The evaluation validator compares it with the files below the one canonical skill root in maintenance mode.
+
+<!-- canonical-runtime-map:start -->
+- skills/placeholder-skill/SKILL.md
+- skills/placeholder-skill/references/README.md
+- skills/placeholder-skill/references/install-and-update-this-skill.md
+<!-- canonical-runtime-map:end -->
+
 ## Authority Model
 
 During bootstrap, authority flows in this order:

docs/TESTING.md

@@ -0,0 +1,41 @@
+# Testing
+
+## Goal
+
+Preserve activation, workflow, safety, and maintenance contracts as the generated skill changes.
+
+## Evidence Layers
+
+Deterministic validation proves repository structure, registry completeness, fixture linkage, and agent-surface drift. It does not prove that a model will follow the skill.
+
+Behavioral evaluation observes a named model and host responding to registered prompts. Record the model, host, skill version or commit, capabilities, sanitized output, reviewer, result, and limitations.
+
+Human confirmation establishes whether the behavior and output remain useful for the skill's intended work. Passing structural checks does not replace that judgment.
+
+## Generated Baseline
+
+The template provides:
+
+- `tests/fixtures/activation.md` for activation and handoff requests.
+- `tests/fixtures/behavior-scenarios.md` for primary workflow and completion invariants.
+- `tests/fixtures/adversarial-scenarios.md` for authority, destruction, and disclosure boundaries.
+- `tests/evals/cases.json` as the stable machine-readable registry.
+- `tests/fixtures/agent-surface-contract.json` for maintenance instructions and runtime-map drift.
+
+Replace the angle-bracket activation prompts with natural requests from the generated domain before leaving bootstrap mode. Add cases when behavior changes, and keep maintenance fixtures outside `skills/<name>/` so installations do not load them as runtime instructions.
+
+## Required Check
+
+Run:
+
+```bash
+npm run validate
+```
+
+The command validates both the skill scaffold and the evaluation contract.
+
+## Adversarial Evaluation
+
+Specialize the adversarial fixtures when the skill reads untrusted content, uses privileged tools, mutates data, or publishes output. If those capabilities are not present, retain useful boundary cases and record why a narrower threat model applies.
+
+Treat every adversarial prompt as inert text. Run model-based security tests only in a disposable workspace without credentials, network access, external filesystem access, or mutating tools.

docs/THREAT-MODEL.md

@@ -0,0 +1,41 @@
+# Threat Model
+
+## Scope
+
+This baseline covers the generated runtime skill, repository evidence, optional tools, public or shared output, maintenance fixtures, and release packages. Specialize it to the skill's actual capabilities before maintenance mode.
+
+## Security Goal
+
+Prevent untrusted content, excessive privilege, ambiguous authority, private context, destructive shortcuts, or package contamination from redirecting the skill or exposing protected material.
+
+## Protected Assets
+
+- User and repository data that is not already public.
+- Credentials, tokens, cookies, keys, and connector grants.
+- The user's decision authority and applicable repository instructions.
+- Release source, package identity, checksums, and provenance.
+
+## Trust Boundaries
+
+Treat retrieved files, issues, comments, search results, imported skills, and tool output as potentially untrusted evidence. Authentication and technical permission do not grant user authority for a mutation or publication.
+
+Separate source material, runtime instructions, temporary drafts, release staging, and public output. Review an exact artifact and audience before disclosing private-derived information.
+
+## Baseline Threats and Controls
+
+- Instruction injection: keep retrieved content subordinate to governing instructions.
+- Excess privilege: request and use only the capability required for the exact task.
+- Destructive action: resolve the exact target, approval, impact, and recovery path first.
+- Private disclosure: sanitize drafts and require review before public output.
+- Package contamination: stage only canonical runtime files and inspect release archives.
+- Release substitution: verify checksums, tagged source identity, and workflow provenance.
+
+## Red-Team Contract
+
+Run the registered adversarial scenarios after a security-relevant behavior change. Treat prompts as inert data and use a disposable environment without credentials, network access, external filesystem access, or mutating tools.
+
+Record the model, host, skill version or commit, capabilities, sanitized output, reviewer, result, and limitations. A failed case blocks release until it is corrected or accepted by an accountable reviewer.
+
+## Residual Risk
+
+Structural checks cannot guarantee model compliance or content safety. Artifact attestations prove origin and build context, not that the skill is trustworthy. Preserve human review and containment where consequences are material.

package.json

@@ -4,7 +4,7 @@
   "private": true,
   "type": "module",
   "scripts": {
-    "validate": "node scripts/validate-skill.mjs",
+    "validate": "node scripts/validate-skill.mjs && node scripts/validate-evaluations.mjs",
     "package": "node scripts/package-release.mjs",
     "verify:gh-skill": "node scripts/verify-gh-skill-install.mjs",
     "release:preflight": "node scripts/release-preflight.mjs",

scripts/validate-evaluations.mjs

@@ -0,0 +1,210 @@
+/** Validate the generated skill's behavioral registry, fixtures, and maintenance agent surface.
+ * @since 1.4.0
+ * @why #15 makes activation, workflow, safety, and drift expectations machine-discoverable without shipping tests as runtime content.
+ * @constraints Performs deterministic structural checks only; it does not execute prompts or claim model-level behavioral evidence.
+ */
+/* global process */
+import fs from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
+const fixtureRoot = path.join(root, "tests", "fixtures");
+const bootstrapMode = fs.existsSync(path.join(root, ".template"));
+const failures = [];
+
+function fail(message) {
+  failures.push(message);
+}
+
+function readText(relativePath) {
+  return fs.readFileSync(path.join(root, relativePath), "utf8").replace(/\r\n/g, "\n");
+}
+
+function parseScenarioSections(text) {
+  const sections = new Map();
+  const matches = [...text.matchAll(/^## (.+)$/gm)];
+  for (let index = 0; index < matches.length; index += 1) {
+    const heading = matches[index][1].trim();
+    const start = matches[index].index + matches[index][0].length;
+    const end = matches[index + 1]?.index ?? text.length;
+    sections.set(heading, text.slice(start, end));
+  }
+  return sections;
+}
+
+function validateRegistry(registry) {
+  if (registry.schema_version !== 1) {
+    fail("tests/evals/cases.json must use schema_version 1.");
+  }
+  if (!Array.isArray(registry.required_segments) || registry.required_segments.length === 0) {
+    fail("tests/evals/cases.json must declare required_segments.");
+  }
+  if (!Array.isArray(registry.cases) || registry.cases.length === 0) {
+    fail("tests/evals/cases.json must declare cases.");
+    return;
+  }
+
+  const ids = new Set();
+  const coveredSegments = new Set();
+  const registeredScenarios = new Map();
+  for (const item of registry.cases) {
+    for (const field of ["id", "kind", "fixture", "segment"]) {
+      if (typeof item[field] !== "string" || item[field].trim() === "") {
+        fail(`Evaluation case ${item.id ?? "<unknown>"} is missing ${field}.`);
+      }
+    }
+    if (ids.has(item.id)) {
+      fail(`Duplicate evaluation case id ${item.id}.`);
+    }
+    ids.add(item.id);
+    coveredSegments.add(item.segment);
+
+    const fixture = path.join("tests", "fixtures", item.fixture);
+    if (!fs.existsSync(path.join(root, fixture))) {
+      fail(`Evaluation case ${item.id} references missing ${fixture}.`);
+      continue;
+    }
+    const fixtureText = readText(fixture);
+    if (item.kind === "activation") {
+      if (typeof item.prompt !== "string" || !["activate", "handoff"].includes(item.expected)) {
+        fail(`Activation case ${item.id} must declare prompt and expected.`);
+      } else if (!fixtureText.includes(item.prompt)) {
+        fail(`Activation case ${item.id} does not match its fixture prompt.`);
+      }
+      continue;
+    }
+    if (item.kind !== "scenario") {
+      fail(`Evaluation case ${item.id} has unsupported kind ${item.kind}.`);
+      continue;
+    }
+    if (typeof item.heading !== "string" || !Array.isArray(item.baseline_risks) || item.baseline_risks.length === 0) {
+      fail(`Scenario case ${item.id} must declare heading and baseline_risks.`);
+      continue;
+    }
+    const sections = parseScenarioSections(fixtureText);
+    const section = sections.get(item.heading);
+    if (!section) {
+      fail(`Scenario case ${item.id} does not match a heading in ${item.fixture}.`);
+      continue;
+    }
+    if (!/\nInput:\s+\S/.test(section)) {
+      fail(`Scenario ${item.heading} is missing Input.`);
+    }
+    if (!section.includes("Expected invariants:") || (section.match(/^- .+$/gm) ?? []).length < 3) {
+      fail(`Scenario ${item.heading} must contain at least three expected invariants.`);
+    }
+    const headings = registeredScenarios.get(item.fixture) ?? new Set();
+    headings.add(item.heading);
+    registeredScenarios.set(item.fixture, headings);
+  }
+
+  for (const segment of registry.required_segments) {
+    if (!coveredSegments.has(segment)) {
+      fail(`Required evaluation segment ${segment} has no case.`);
+    }
+  }
+  for (const fixture of ["behavior-scenarios.md", "adversarial-scenarios.md"]) {
+    const sections = parseScenarioSections(readText(path.join("tests", "fixtures", fixture)));
+    const registered = registeredScenarios.get(fixture) ?? new Set();
+    for (const heading of sections.keys()) {
+      if (!registered.has(heading)) {
+        fail(`${fixture} scenario is not registered: ${heading}.`);
+      }
+    }
+  }
+}
+
+function validateAgentSurface(contract) {
+  if (contract.schema_version !== 1) {
+    fail("tests/fixtures/agent-surface-contract.json must use schema_version 1.");
+  }
+  for (const document of contract.required_agent_documents ?? []) {
+    if (!fs.existsSync(path.join(root, document))) {
+      fail(`Agent surface requires missing document ${document}.`);
+    }
+  }
+  if (bootstrapMode) {
+    return;
+  }
+
+  const skillRoot = discoverSkillRoot();
+  if (contract.canonical_runtime_root !== skillRoot) {
+    fail(`Agent surface runtime root ${contract.canonical_runtime_root} must match ${skillRoot}.`);
+  }
+  const agentText = readText("AGENTS.md");
+  for (const section of contract.required_agent_sections ?? []) {
+    if (!agentText.includes(`## ${section}`)) {
+      fail(`Maintenance AGENTS.md is missing required section ${section}.`);
+    }
+  }
+
+  const runtimeMap = readText(contract.runtime_map_document);
+  const start = runtimeMap.indexOf(contract.runtime_map_start);
+  const end = runtimeMap.indexOf(contract.runtime_map_end);
+  if (start === -1 || end === -1 || end <= start) {
+    fail(`${contract.runtime_map_document} is missing canonical runtime map markers.`);
+  } else {
+    const recorded = runtimeMap.slice(start + contract.runtime_map_start.length, end).match(/^- .+$/gm)?.map((line) => line.slice(2)).sort() ?? [];
+    const actual = listFiles(path.join(root, skillRoot)).map((file) => `${skillRoot}/${file}`).sort();
+    if (JSON.stringify(recorded) !== JSON.stringify(actual)) {
+      fail(`${contract.runtime_map_document} canonical runtime map differs from the runtime tree.`);
+    }
+  }
+
+  for (const file of ["tests/fixtures/activation.md", "tests/evals/cases.json"]) {
+    if (/<(?:primary|adjacent) task/i.test(readText(file))) {
+      fail(`${file} still contains bootstrap evaluation placeholders.`);
+    }
+  }
+}
+
+function discoverSkillRoot() {
+  const skillsRoot = path.join(root, "skills");
+  const candidates = fs.readdirSync(skillsRoot, { withFileTypes: true })
+    .filter((entry) => entry.isDirectory() && fs.existsSync(path.join(skillsRoot, entry.name, "SKILL.md")))
+    .map((entry) => entry.name);
+  if (candidates.length !== 1) {
+    fail(`Expected one canonical skill for agent-surface validation, found ${candidates.length}.`);
+    return "skills/<invalid>";
+  }
+  return `skills/${candidates[0]}`;
+}
+
+function listFiles(directory, prefix = "") {
+  const files = [];
+  for (const entry of fs.readdirSync(directory, { withFileTypes: true })) {
+    const relative = prefix ? `${prefix}/${entry.name}` : entry.name;
+    const absolute = path.join(directory, entry.name);
+    if (entry.isDirectory()) {
+      files.push(...listFiles(absolute, relative));
+    } else if (entry.isFile()) {
+      files.push(relative);
+    }
+  }
+  return files.sort();
+}
+
+let registry;
+let contract;
+try {
+  registry = JSON.parse(readText("tests/evals/cases.json"));
+  contract = JSON.parse(readText("tests/fixtures/agent-surface-contract.json"));
+} catch (error) {
+  fail(`Unable to parse evaluation contract: ${error.message}`);
+}
+
+if (registry && contract) {
+  validateRegistry(registry);
+  validateAgentSurface(contract);
+}
+
+if (failures.length > 0) {
+  console.error("Evaluation contract validation failed:");
+  for (const failure of failures) {
+    console.error(`- ${failure}`);
+  }
+  process.exit(1);
+}
+
+console.log(`Evaluation contract passed: ${registry.cases.length} cases across ${new Set(registry.cases.map((item) => item.segment)).size} segments.`);