Skip to content

test(arsenal): lock the advertised tool count to the real callable surface#41

Open
Pazificateur69 wants to merge 1 commit into
elder-plinius:mainfrom
Pazificateur69:test/arsenal-count-honesty
Open

test(arsenal): lock the advertised tool count to the real callable surface#41
Pazificateur69 wants to merge 1 commit into
elder-plinius:mainfrom
Pazificateur69:test/arsenal-count-honesty

Conversation

@Pazificateur69

Copy link
Copy Markdown

What

The arsenal advertises "83 tools" and verify-claims defends it by counting id: / name: source lines — a source-line count that would keep passing even if an entry became uncallable (e.g. bloodhound is counted but import_only, never a generic callable tool).

Change

Adds src/__tests__/arsenal-count-honesty.test.ts that ties the number to the real registered arrays (TOOL_ADAPTERS + BUILTIN_TOOLS + EXTERNAL_TOOLS === 83) and pins the invariant that makes the count honest:

an adapter is off the generic callable surface (adapterToCustomTool → null) iff it is explicitly gated by execution mode (catalog_only / import_only).

So a tool can be counted-but-not-generically-callable only on purposemetasploit / hydra (catalog_only, reachable solely via the approval-gated post-ex drivers) and bloodhound (import_only) — never through a silent DEFAULT / mint slip. Also asserts isMintable agrees with actual callability, the gated set is exactly {metasploit, hydra, bloodhound}, and every minted adapter has a real handler + a unique name. Sibling of no-phantom-tools / stub-honesty.

Pure test — no behavior change.

Review standard (all green locally)

npm run typecheck · npm test · npm run doctor · npm run test:gate · npm run smoke

…rface

The arsenal advertises "83 tools" and verify-claims defends it by counting
`id:`/`name:` source lines — a source-line count that would keep passing even
if an entry became uncallable. Add a guard that ties the number to the real
registered arrays (TOOL_ADAPTERS + BUILTIN_TOOLS + EXTERNAL_TOOLS === 83) and
pins the invariant that makes the count honest:

  an adapter is off the generic callable surface (adapterToCustomTool → null)
  IFF it is explicitly gated by execution mode (catalog_only / import_only).

So a tool can be counted-but-not-generically-callable only on purpose —
metasploit / hydra (catalog_only, reachable solely via the approval-gated
post-ex drivers) and bloodhound (import_only) — never through a silent DEFAULT
or mint slip. Also asserts isMintable agrees with actual callability, the gated
set is exactly {metasploit, hydra, bloodhound}, and every minted adapter has a
real handler and a unique name. Sibling of no-phantom-tools / stub-honesty.
@jmagly

jmagly commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

The invariant tests here (mintable ⇔ callable, no silently-dead / wrongly-gated) are a genuinely good guard. But the two hardcoded pins are already stale against current main, so this reds CI on merge:

  • expect(total).toBe(83)main is now 102 (67 adapters + 31 builtins + 4 externals).
  • gated set ['bloodhound','hydra','metasploit']main now has 5: +frida, +pacu (both catalog_only).

Fix: rebase and update both constants — or better, derive the count and gated set from the array lengths / a flag predicate so they don't need hand-updating every time a tool is added (which is the brittleness this test is otherwise trying to prevent).

@jmagly

jmagly commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Please rebase this branch onto current main and reduce it to the advertised honesty test.

This should be a tiny test-focused PR, but the branch currently carries unrelated stale-base deletions of benchmark corpora, scripts, provenance docs, and tests.

Suggested agent repair path:

  1. Start from current main.
  2. Reapply only the advertised tool-count honesty test.
  3. Restore all unrelated files.
  4. Run the focused test command plus any relevant typecheck/static check.
  5. Confirm git diff --name-status upstream/main...HEAD shows only the intended test change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants