Measure agentic-actions-auditor against zizmor, add a workflow and evals

[mkzung]

agentic-actions-auditor documents nine vectors, but nothing said how often they occur or whether zizmor already covers them, and zizmor is not mentioned anywhere in the plugin. This measures that first, then converts the skill to a workflow and adds an eval suite shaped by the answer.

What the corpus says

60 workflows invoking claude-code-action, run-gemini-cli, codex-action or ai-inference, each pinned by content SHA. zizmor covers the general surface heavily: unpinned-uses 87 findings across 46 files, artipacked 49 across 43, template-injection 12 across 4, and nothing at all in 8 of the 60.

Of the six statically decidable vectors, F occurs in 16 workflows, B in 2, H in 1, and A, D and I in none.

The two sets are disjoint. Every template-injection finding points at a run: step and none at a prompt field. Neither of the two workflows where a github.event expression reaches an agent prompt drew that rule; one drew nothing at all, the other only unpinned-uses and github-app. That is the case for the plugin existing, and it is now measured rather than asserted.

corpus/ is gitignored and no repository is named anywhere. Committing a third party's exploitable workflow as a fixture would publish a list of live targets, so every fixture here is written by hand.

What is in the PR

workflows/audit-agentic.js runs discovery, then four sweeps in parallel by vector family, then the report. Three of its four exits are refusals. The one that matters is no-agent: the skill's own When NOT to Use puts a workflow with no AI action out of scope, so the run stops and points at general tooling rather than producing a findings report organised by agentic vectors about a workflow that has no agent.

tests/vectors.py carries a signature per decidable vector, each paired with a fixture that must match and a safe twin that must not. C, E and G get none on purpose: deciding them needs the meaning of a prompt or the behaviour of a later step. measure.py refuses to print counts if that self-test fails, and workflow_smoke.mjs drives the workflow with stubbed agents to check every exit is reachable and that no sweep family goes missing or fails silently.

Two things to weigh rather than take on trust

The F signature reported zero until it was keyed on claude_args --allowedTools instead of an allowed_tools: input, which claude-code-action dropped after v0.0.32 and which the vector-f reference never named. Corrected, F is the most common vector in the corpus. The case ordering follows the corrected number; before the fix I had it backwards.

The five cases do not discriminate, and the problem is the premise. claude plugin eval is in early access here, so instead I ran each fixture in isolation against a fresh agent with no reference material:

Fixture	Unaided arm
01, expression in prompt	found it, and called it prompt injection rather than shell injection
02, env-var intermediary	traced the flow and said the indirection stops template injection but not prompt injection
built to separate the arms on F	wrote unprompted that a prefix allowlist is permeable to command substitution
agent hidden in a composite action	followed the uses: reference, found the agent and the injection

Four fixtures, four no-plugin passes. So the nine vectors are not scarce knowledge for a current model, and neither rewriting the fixtures nor hiding the agent behind another file changes that.

evals/README.md sets out what is left: recall across a directory of many workflows rather than depth on one file, which is about stamina rather than knowledge and is the one hypothesis the four failures above do not settle; consistent labelling so findings can be counted; and the refusal, which the workflow enforces structurally rather than asking the model for. I would rather have your opinion on that before rebuilding the cases, since #229 deleted a case for the same reason.

One small thing the pilot turned up: the unaided arm pointed out that secrets is unavailable inside composite actions, so an anthropic_api_key: ${{ env.SOMETHING }} there silently resolves to empty. That matches GitHub's documentation and cross-file-resolution.md does not mention it, though it covers resolving and input tracing. Happy to add it in a separate PR if useful.

make self-test, make lint and make validate pass. make python-tests fails the same way on a clean checkout, in constant-time-analysis. Note that validate_reference_links skips any path with evals in it, so validate does not actually cover that directory.

Written with Claude Code, reviewed and verified by me.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.