fix(tests): repoint the model-options vacuity guard at the Python mock

[cezara98t]

Problem

list_model_options scored 0.91 in the 2026-08-04 nightly with every substantive check green. Only its vacuity guard failed:

path: "mocks/uip"
includes: ['>> "$(dirname "$0")/calls.log"']

That asserts the sh mock's redirect. #2375 (Aug 3) rewrote mocks/uip in Python and deleted that line, so the criterion can no longer pass. Weights are 1.5+1.5+3.0+1.0+3.0+1.0 = 11.0; losing the 1.0 gives 10.0/11.0 = 0.909 — exactly the observed score.

This task (#2361) landed Jul 30, four days before the rewrite. #2375 updated the mock README — which even documents this trap ("It broke once already") — but missed the task.

Fix

Assert the Python mock's FLAT_LOG assignment instead. calls.log is the sink the guard above grades, so this names the coupling directly. The comment now points at the mock so the next refactor catches it.

Not a blanket rename. mock_template_ambiguous and mock_template_taxonomy are still sh mocks and legitimately keep the old redirect — both are overlays listed second, so they shadow the Python mock in the two tasks that use them.

Verification

• Asserted string is byte-exact present in _shared/mock_template/mocks/uip
• YAML parses; total weight unchanged at 11.0, so the task returns to 1.00
• No local coder-eval run: it is not installed in my environment. This PR touches a task YAML, so the smoke workflow will execute the task itself — that is the real passing-run signal here.

🤖 Generated with Claude Code

[github-actions]

Claude finished @cezara98t's task in 1m 26s —— View job

---

Summary

Repoints the vacuity guard in list_model_options.yaml from the deleted sh mock's >> "$(dirname "$0")/calls.log" redirect to the Python mock's FLAT_LOG = os.path.join(MOCK_DIR, "calls.log") assignment. Fixes the 0.91 score caused by #2375's mock rewrite.

Change-by-Change Review

1. tests/tasks/uipath-ixp/smoke/list_model_options.yaml

Severity: OK

The change is correct and well-scoped:

• Asserted string verified. FLAT_LOG = os.path.join(MOCK_DIR, "calls.log") is byte-exact at line 43 of tests/tasks/uipath-ixp/_shared/mock_template/mocks/uip. The file_contains criterion will pass.
• Weight unchanged. Total remains 11.0 (1.5 + 1.5 + 3.0 + 1.0 + 3.0 + 1.0). No scoring side effects.
• YAML valid. Structure is well-formed; all other criteria are untouched.
• Comment is valuable. The expanded comment (lines 85–90) names the coupling explicitly and points maintainers at the mock file for future refactors — addresses the exact drift that caused this bug.
• Overlay mocks unaffected. mock_template_ambiguous and mock_template_taxonomy are still sh scripts and still use the old redirect pattern. No other .yaml task references the old sh pattern for this mock — confirmed via grep.

What's Missing

Nothing identified.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill frontmatter changed
E2E Tests	OK	Task YAML fix is correct; total weight preserved
Skill Body	N/A	No skill body changed
References & Assets	N/A	No references changed
Repo Hygiene	OK	Scoped change, no secrets, no cross-skill deps

Issues for Manual Review

None found.

Conclusion

Clean, well-motivated fix. The asserted string is byte-exact in the Python mock, weight totals are preserved, and the expanded comment should prevent recurrence. Approve.

---

[github-actions]

Claude finished @cezara98t's task in 2m 49s —— View job

---

Coder-eval task lint (advisory)

1 task YAML changed; verdict counts: 0 Critical, 1 High, 0 Medium, 0 Low, 0 OK (but see note — the High is only the passing-run claim, not a task-design issue).

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

❌ High — PR body does not claim the changed task has been run and passed. The author writes:

"No local coder-eval run: it is not installed in my environment. This PR touches a task YAML, so the smoke workflow will execute the task itself — that is the real passing-run signal here."

The reasoning is sound (mechanical string replacement, YAML parses, weights unchanged at 11.0, and CI will validate), but the rubric requires a past-tense claim. Please edit the PR description after CI passes to add a line like: Ran skill-ixp-list-model-options-smoke in CI and it passed.

Per-task lint

tests/tasks/uipath-ixp/smoke/list_model_options.yaml — verdict: OK

The fix is correct and well-scoped:

• Self-report anti-pattern: Not triggered — model_options.md carries ground-truth model identifiers the agent retrieves from skill knowledge, not a self-report of actions performed.
• Prompt over-specification: Clean — prompt states the goal ("List available model options"), no procedure leak.
• Meaningful coverage: Strong — 6 criteria (skill_triggered + file_exists + ground-truth content check + actionable-context check + negative control via call log + vacuity guard). Weight total 11.0, unchanged.
• Could pass for the wrong reason: No — the three snake_case identifiers (gemini_2_5_flash, gemini_2_5_pro, gpt_4o_2024_05_13) are skill-specific knowledge; the negative list-models exclusion prevents the common trap.
• Near-duplicate: Not duplicated — compared against pick_capable_model (capability-based mutation), change_model_variant (colloquial→canonical mutation), query_model_and_preprocessing (read current config), list_projects, list_documents. All materially distinct.
• Validate-only flow tests: N/A (not a flow skill).
• CLI verb reachability: No command_executed criteria — axis does not apply.
• Redundant uip CLI / Run-limit fields: Clean.

I verified the asserted string FLAT_LOG = os.path.join(MOCK_DIR, "calls.log") is byte-exact present at line 43 of tests/tasks/uipath-ixp/_shared/mock_template/mocks/uip.

Within-PR duplicates

• No duplicate clusters detected (only 1 task changed).

Conclusion

• ⚠ The task itself passes all rubric axes (verdict: OK). Only the evidence-of-passing-run convention is missing — advisory, not blocking merge. Once CI greens, consider editing the PR body to record the passing run.

[cezara98t]

Closing — superseded by planned harness-level support for a structured JSONL invocation log, not because the diagnosis changed.

Recorded so the finding isn't lost with the PR. In the 2026-08-04 nightly, list_model_options scores 0.91 with every substantive check green. Its last criterion asserts the sh mock's redirect:

path: "mocks/uip"
includes: ['>> "$(dirname "$0")/calls.log"']   # weight 1.0

#2375 rewrote mocks/uip in Python and deleted that line, so the criterion cannot pass: 10.0/11.0 = 0.909. It exists as a non-vacuity partner for the 3.0 projects list-models negative guard, using the fallback the mock README prescribes when a correct run makes no uip call — as this task's does.

For whoever migrates this task to the structured log: delete that criterion, do not port it. Once the harness itself fails a negative assertion against a missing log, and generates the recorder rather than each suite hand-writing one, there is no mock source text left to assert against. Translating the sentinel into a jsonl-era equivalent would carry the anti-pattern into the new mechanism — the negative guard should simply become an invocation assertion with a zero-count bound.

Also relevant to that migration: mock_template_ambiguous and mock_template_taxonomy are still sh mocks that never gained the JSONL sink, and both are overlays listed second, so they shadow the Python mock in rename_ambiguous_entity and move_field_between_groups. Tasks overlaying them record nothing structured today.

Until that migration lands, this task stays at 0.91 — a known, deterministic false negative, not a skill regression.