examples(tau2_custom/blackbox): type the seed's structured tool params

[aviweit]

Closes #692

What changed

6 files, +372/-10.

• The three seed scripts now declare the benchmark's own types: the FlightType,
  CabinClass and Insurance literals and the FlightInfo, Passenger and Payment
  models, copied verbatim from the pinned build's data_model.py. Parameter names,
  docstrings and the env_* call bodies are untouched.
• Each script carries the classes its own signature needs. The store derives a tool's
  schema from that one module and resolves no import from a sibling script, so a shared
  types module would silently yield a flat schema. The duplication is deliberate.
• seed_capability/reference/data_model.py (new) — the benchmark's data model,
  byte-identical to the pinned build's and to the direct arm's copy, declared as
  capability_sources so it is materialized into the optimizer's
  ./guidance/sources/data_model.py. Previously capability_sources: [], so the blackbox
  optimizer had no focused data model while the direct one did.
• protected_paths gains reference/*.
• optimizer/INSTRUCTIONS.md rule 2 rewritten: the signature and docstring together
  are the schema; the seed's signatures are frozen; a new tool uses plain types by default;
  a new tool with a genuinely structured parameter copies the real class from the reference
  file, with its dependencies and without any tau2.* import or DB class; and the four
  constraints whose breach yields a flat schema with no error (same file, one-line def,
  no self-referential model, no missing dependency).

Verification

• The published schema for book_reservation and update_reservation_flights is
  identical to what the benchmark sends its in-process agent, after resolving that
  side's $refs: same properties, enums, nested fields, descriptions and required lists.
  That is the point — the two arms now advertise one tool surface.
• All 14 scripts parse to exactly one tool each; the three typed ones yield enums and
  nested item schemas, the other 11 are unchanged.
• ./guidance/sources/data_model.py placement confirmed against the direct arm's run dirs,
  where every work/cand_*/guidance/sources/data_model.py exists and is byte-identical to
  the benchmark's file.
• Spec parses; num_trials, max_iterations and the gate settings are untouched.

Dependency

Against the pinned store (0.2.1) this change is a no-op — not a degradation. That
store derives a tool's params from the docstring and the signature's top-level annotation,
where CabinClass collapses to the same string as str did and List[FlightInfo | dict]
to the same array as list did. Verified on all three tools by running 0.2.1's code path:
the stored params, the description and the returns come out identical to the current
main versions, so the published schema, the generated handler and therefore the score stay
exactly as they are today.

The annotations only start to matter against a store that derives the schema from them. That
fix lives on aviweit/skillberry-store @ fix/vmcp-rich-tool-schema and is not upstream
yet; the blackbox_env.py pin that selects it is a separate cap-evolve branch, deliberately
not in this PR. So this can merge on its own, with no behaviour change until the store moves.

Known gaps, not addressed here

• No drift guard. The data model now exists in three copies (the benchmark's, each
  arm's reference/), plus class blocks inside the three scripts, and the optimizer edits
  those scripts every iteration. A deleted class makes the derivation raise, which is
  swallowed, and the schema silently falls back to flat. One test should assert the copies
  match and that exactly those three tools derive a non-flat schema. Note
  direct/seed_capability/tools/tools.py claims it is "Generated by
  examples/tau2_custom/direct/seed_from_tau2.py" — that script does not exist.
• List[X | dict] is kept for fidelity with the benchmark, though it publishes "an X
  or any object", which weakens the constraint it exists to express.
• A tool importing pydantic pays a per-call pip install in the docker executor, whose
  failure is silently ignored. The three seed tools are affected; this is the reason the new
  rule defaults new tools to plain types.

🤖 Generated with Claude Code

[skillberry-bot]

🏷️ Automatic Labeling

I've analyzed this pull request and added the following labels:

• enhancement - documentation - algorithm - benchmark-smoke-tau2 - enhancement - documentation - algorithm - benchmark-smoke-tau2 - priority-p2

These labels were selected based on the PR title, description, and changed files. If you believe any labels are incorrect or missing, feel free to adjust them manually.

[aviweit]

First paired direct-vs-blackbox run with the typed seed

Headline: the two arms measure the same thing, which is the objective. A store-delivered
tool surface should score like the in-process one; in this run it does. The seeds are
identical task for task, and the sealed test numbers differ by 0.47σ — no material
difference at the resolution this setup can resolve.

Both arms, same 7 tasks (split_ids.json: train = val = test), 5 trials, 3 iterations,
hill-climb, gate_k_se: 0.2.

	agent	user simulator	gateway
direct	aws/claude-sonnet-5	aws/claude-sonnet-5	https://ete-litellm.ai-models.vpc.res.ibm.com
blackbox	ibm/skillberry-local → proxy → aws/claude-sonnet-5	aws/claude-sonnet-5 (straight to the gateway)	same

The blackbox arm ran against a store that does derive the schema from the annotations, so
the typed seed was live on both sides: the published inputSchema for
update_reservation_flights carried cabin's enum of business / economy / basic_economy
(confirmed in the proxy's own log during this run's last iteration). What it did not have
is this PR's optimizer-side half — its deployed optimizer/INSTRUCTIONS.md predates the
rule-2 rewrite, and capability_sources was still [], so its optimizer had no data model.

	seed on val	best val	sealed test	seed on test
direct	0.514	0.714 (cand_0002)	0.543	0.514
blackbox	0.514	0.514 (none accepted)	0.429	0.429

1. The two sealed numbers are indistinguishable. 0.543 vs 0.429 is 0.114 apart with an SE
of the difference of 0.241, i.e. 0.47σ — well inside noise, so this run shows no material
difference between delivering the capability in-process and delivering it through the store
and proxy. Neither arm improved on its own seed either: direct +0.029 on test, blackbox
exactly 0, since all three of its candidates regressed and were correctly rejected.

2. Delivery parity is the solid finding, and it is what this PR is for. The two seeds
scored 0.514 task for task identically on val — t8=1.00 t10=1.00 t20=0.80 t23=0.00 t35=0.00 t42=0.80 t44=0.00 in both arms, not merely the same mean. With the seed typed and
the store deriving the schema from those annotations, routing tools through the store and
proxy costs nothing against calling them in-process. Equivalence is the target for this
delivery path, and it only becomes reachable once both sides advertise the same tool
surface — which is what the typed seed provides.

The one asymmetry left is on the optimizer side, not the delivery side: direct's optimizer
found a real gain (t44, 0.00 → 1.00 on val, holding 0.80 on test) while blackbox's three
candidates all regressed. Its deployed instructions predate the rule-2 rewrite and it had no
data model, which this PR changes.

3. Everything else is below the measurement floor. 7 tasks × 5 trials gives ±0.20 per-task
noise — proved within the direct run itself, where the byte-identical seed moved t10
1.00 → 0.80 and t42 0.80 → 1.00 between val and test. That is why cand_0002's 0.714 on val
fell to 0.543 on test: nearly all of it was t23 drawing 4/5 and then 1/5. The gate accepted
it while stating it could only resolve effects above 41 points
(Δ̄=+0.1714 > 0.2·SE=0.0413 (SE=0.2067, resolvable effect size 2·SE=0.4134)), which is what
gate_k_se: 0.2 permits; at k=1 it would have been rejected. More trials, k_se: 1.0 and a
genuinely disjoint test split are needed before any of these numbers carry weight.

One detail worth recording: no pass anywhere came from an error masquerading as success. The
only passes without a write call are on t10, in both the seed and the candidate, and that is
a task needing no DB change.