Skip to content

feat(nvidia-skill-finder): use the Hermes built-in catalog under Hermes - #820

Merged
mosheabr merged 19 commits into
mainfrom
jdudash/skill-finder-hermes-catalog
Nov 10, 2026
Merged

mosheabr merged 19 commits into
mainfrom
jdudash/skill-finder-hermes-catalog

Conversation

@jasonnvidia

@jasonnvidia jasonnvidia commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Teaches nvidia-skill-finder to use Hermes Agent's built-in skills catalog instead of npx, and bumps nvidia-skills to 1.5.0. This addresses review feedback on NousResearch/hermes-agent#146865, which flagged that the skill runs an unpinned npx command and installs with --global --yes in a way that bypasses Hermes's security scan and is not profile-aware.

What changed

  • SKILL.md keeps the npx path first and unbroken as the default, with Hermes as a single pointer in each of the lookup and install sections.
  • references/hermes-catalog.md (new) carries the whole Hermes flow: search, the single-keyword strategy, the full-catalog fallback, install, and identifier rules. This follows the progressive-disclosure pattern the skill already uses for references/taxonomy-routing.md.
  • plugins/nvidia-skills/README.md drops "Hermes support is coming soon" and documents what the finder runs per harness.
  • Version 1.4.1 → 1.5.0.

Two findings worth flagging

Identifier casing determines trust. hermes skills install nvidia/skills/<name> (lowercase) resolves to a community-trust skills.sh mirror with a placeholder description, while NVIDIA/skills/<name> resolves to the trusted first-party source. Segment count does not matter — NVIDIA/skills/<name> and NVIDIA/skills/skills/<name> both resolve to the same trusted skill with the same scan hash. The skill now specifies the uppercase form and tells the model to copy the Identifier from search output verbatim.

hermes skills search matches substrings, not tokens.

Testing

See comments below for updated testing details

@jasonnvidia
jasonnvidia force-pushed the jdudash/skill-finder-hermes-catalog branch from 156190d to 9109ad1 Compare November 4, 2026 19:54
@jasonnvidia

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

@mosheabr

mosheabr commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Hey Jason, nice work on this. I tested it on Hermes 0.22.7 and the flow works well: it picked up the skill on its own, used the Hermes search instead of npx, and gave the uppercase NVIDIA/ install commands. The casing thing checks out too: lowercase really does come back as community trust.

One thing I hit: hermes skills search only returns 25 results by default. jetson has 39 matches, so jetson-quick-start got cut off, and when I asked how to set up an Orin Nano, it never suggested it. Adding --limit 100 to the search command in hermes-catalog.md (both copies) should fix it.

Small one: maybe mention that --source nvidia needs Hermes 0.22+. My older install rejected it.

Happy to approve once that's in and re-signed.

jasonnvidia added a commit that referenced this pull request Nov 5, 2026
…ult 25

hermes skills search returns 25 results by default, which silently truncates
the broad single-keyword queries this guidance asks for: jetson matches 39
skills, so jetson-quick-start was cut off and never recommended. Pass
--limit 100 on every search.

Also note that --source nvidia requires Hermes 0.22 or newer, and give the
fallback for older installs.

Reported by teknium1 on #820.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
@mosheabr

mosheabr commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Thanks Jason, the --limit fix works. I re-ran it on Hermes 0.22.7: jetson now returns all 39 results, including jetson-quick-start, where the default capped it at 25. The fix commit landed after the last signing, though, so skill.oms.sig no longer matches references/hermes-catalog.md and Content Integrity is failing. Could you comment /nvskills-ci to re-sign? Once that's green, I'll approve.

jasonnvidia and others added 14 commits November 5, 2026 16:19
The finder told every agent to run `npx skills add nvidia/skills --list`
for catalog lookup and `npx skills add ... --global --yes` for install.
On Hermes that bypasses the built-in catalog and the skill security scan,
and `--global` writes the default profile's skills directory regardless of
the active profile.

Branch both the catalog lookup and the install on the agent the skill is
running in. Under Hermes use `hermes skills search <query> --source nvidia`
and `hermes skills install NVIDIA/skills/<skill-name>`; every other agent
keeps the existing `npx` path. Identifier casing is significant --
`NVIDIA/...` resolves to the trusted first-party source while `nvidia/...`
resolves to a community-trust mirror -- so the skill is told to copy the
identifier from search output verbatim.

Hermes search matches every word in the query, so note that one or two
keywords work where a natural-language phrase returns nothing.

Also update the plugin README, which still said Hermes support was coming
soon, and release the plugin as 1.5.0.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
`hermes skills search` requires every word in the query to match, so a
descriptive phrase usually returns nothing: `jetson pinmux` returns no
rows even though `jetson-customize-pinmux` exists, and `OpenUSD` does not
return `omniverse-usd-performance-tuning`. Across ten realistic lookups
only six found the right skill. That is a real regression against the
`npx` path, where the model sees all 431 names and descriptions and does
the matching itself.

Searching one broad product, family, or lane keyword at a time recovers
every one of those misses, so require single-keyword queries and tell the
model to run two or three of them and choose from the full descriptions
`--json` returns rather than trusting the first row.

Add a full-catalog fallback for when those searches still come up empty,
so a narrow search term cannot hide a skill that exists. `hermes skills
browse` is not used for this: it truncates both names and descriptions and
costs more context than the catalog listing it would replace.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
…ence

The Hermes branch had grown to roughly 50 inline lines, most of it recovery
detail for the substring-match behaviour of hermes skills search. Keep the
correctness-critical parts inline -- the two commands, the NVIDIA/ vs nvidia/
casing rule, and the guard against running hermes outside Hermes -- and move
the recall tactics, full-catalog fallback, and identifier confirmation to
references/hermes-catalog.md, loaded only when searches come up empty.

This follows the progressive-disclosure pattern the skill already uses for
references/taxonomy-routing.md.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
…ermes

Put the npx commands back first and unbroken so the common path reads as the
default, and reduce the Hermes branch to one bullet in each of the lookup and
install sections pointing at references/hermes-catalog.md. The reference now
carries the whole Hermes flow rather than only the recovery tactics.

Reorders the plugin README the same way.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Keep 'When No Skills are Found' exactly as it was on main; the Hermes variant
of the ecosystem search does not need to be called out there.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
generate_versions.py tracks the last commit touching each skill directory, so
editing the skill leaves versions.json stale.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
…ult 25

hermes skills search returns 25 results by default, which silently truncates
the broad single-keyword queries this guidance asks for: jetson matches 39
skills, so jetson-quick-start was cut off and never recommended. Pass
--limit 100 on every search.

Also note that --source nvidia requires Hermes 0.22 or newer, and give the
fallback for older installs.

Reported by teknium1 on #820.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Signed-off-by: Jason Dudash <jdudash@nvidia.com>
… searching

Hermes search matches the query as a plain case-insensitive substring over
each entry's joined fields, so it misses phrases and truncates broad keywords
at the default limit of 25. Working around that took ~8 searches per question,
which cost more context than reading the whole catalog and could still miss
the right skill.

Read the 437-entry catalog listing once and choose from it, then confirm the
identifier with a single narrow search before installing through Hermes, which
keeps the security scan and profile awareness. Across the 22-question product
sweep this cut searches from 176 to 21 and transcripts from 250KB to 46KB at
equal accuracy.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Signed-off-by: Jason Dudash <jdudash@nvidia.com>
@jasonnvidia
jasonnvidia force-pushed the jdudash/skill-finder-hermes-catalog branch from 645d745 to f73b929 Compare November 5, 2026 21:20
@jasonnvidia

Copy link
Copy Markdown
Collaborator Author

Thanks Moshe. Your --limit catch turned out to be more than a one-line fix, so I want to flag up front that I changed the approach rather than just patching it.

What your finding led to. The 25-result cap was a symptom. Hermes search matches the query as a plain case-insensitive substring over each entry's joined fields, so it also misses phrases entirely — jetson pinmux returns nothing even though jetson-customize-pinmux exists. Working around both problems meant telling the model to run several broad single-keyword searches, so I measured what that actually costs:

approach complete? context
full catalog listing (name + truncated description) yes 14.7k tokens
~8 hermes skills search --json --limit 100 calls no, recall gaps ~39k tokens
npx skills add nvidia/skills --list yes 47k tokens

Searching was the most expensive option and the only incomplete one. Since the whole point of the Hermes branch was avoiding npx's context cost, that undercut the rationale for it.

What it does now. Read the 437-entry catalog listing once and pick from it, then confirm the chosen skill's identifier with one narrow search before installing through Hermes:

curl -sL .../metadata.json | jq -r '.skills[] | "\(.name): \(.description[0:110])"'
hermes skills search <skill-name> --source nvidia --json --limit 100
hermes skills install <identifier>

Install still goes through Hermes, so the security scan and profile awareness are unchanged. Your --limit 100 is still on the confirmation search, and the 0.22+ note for --source is still in.

Results — same 22 product questions, forced activation, Hermes 0.22.5 on a clean box:

metric search-first pull-first
named an acceptable skill 22/22 21/22
hermes skills search calls 176 21
used npx (must be 0) 0 0
lowercase nvidia/ identifiers 0 0
total transcript 250 KB 46 KB

Orin Nano / jetson-quick-start now resolves from the listing instead of depending on a search happening to hit it. I also re-tested Claude and Codex: both still use the npx path, neither touches hermes.

Caveats:

  • The one miss was a transient Hermes skill-load failure (skill_view returned "not found"), not a routing error — the model then read my test's "read-only" instruction as forbidding it from enabling the plugin. Rerunning the identical prompt gave the right answer. Worth knowing it happens.
  • The listing needs jq; there's a python3 fallback in the reference for boxes without it.
  • It reads from raw.githubusercontent.com, the same host as the existing fallback catalog sources.

Ask: could you independently verify this before approving? My numbers come from my own harness and my own choice of 22 questions, so they're worth confirming against a different setup. Most useful checks would be your Orin Nano prompt that previously missed jetson-quick-start, a few product areas I didn't cover, and a run on 0.22.7 since I tested on 0.22.5. I'm specifically interested in whether the model reliably reads the full listing rather than reaching for search out of habit, and whether the install identifiers stay uppercase.

@jasonnvidia

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
The python3 -c one-liner was broken two ways. Backslash escapes inside an
f-string expression are only legal from Python 3.12, so it raised SyntaxError on
macOS's system 3.9 and Ubuntu 22.04's 3.10. Separately, Hermes's approval layer
classifies inline interpreter scripts as "script execution via -e/-c flag" and
refuses to run them at all.

Use python3 -m json.tool piped through grep and cut instead, which works
unchanged on 3.9 and clears the same approval check, and tell the model not to
reach for python3 -c.

Thanks to @mosheabr for catching the version incompatibility in review.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
@jasonnvidia

jasonnvidia commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

Thanks Moshe — per your Slack thread on Python version bug.
Good catch, and it was worse than you reported. Pushed a fix in cc8c779.

It fails on every Python version, not just 3.11 and older. You're right about the PEP 701 rule, but this particular one-liner never worked anywhere. Because the snippet is wrapped in shell single quotes, the \" escapes reach Python literally, so 3.12+ trips on them too — just with a different error:

Python 3.9.6    SyntaxError: f-string expression part cannot include a backslash
Python 3.14.7   SyntaxError: unexpected character after line continuation character

So the fallback was dead on arrival. Thanks for pulling the thread.

There was a second problem with it too. Hermes's approval layer classifies inline interpreter scripts as "script execution via -e/-c flag" and refuses to run them, so even a correctly-quoted python3 -c would have been blocked. Rather than fix the quoting, I dropped the -c form entirely:

curl -sL https://raw.githubusercontent.com/NVIDIA/skills/main/.github/scripts/marketplace/metadata.json \
  | python3 -m json.tool | grep -E '"(name|description)":' | cut -c1-130

Verified byte-identical output (898 lines) on macOS's system 3.9.6 and on 3.14.7. json.tool, grep and cut need no escaping and clear the same approval check. The reference now also tells the model explicitly not to reach for python3 -c, so it doesn't reinvent the broken form.

Please don't take my word for the behaviour claims — the approval-layer block and the catalog-pull flow are both worth an independent run on your side before this merges.

On the red check: it isn't from this change. The last signing commit (4313e49) rewrote BENCHMARK.md with the 1.6.8 evaluator numbers, but benchmarks.json still carries the 1.6.6 ones, so aggregate_benchmarks.py --check fails. versions.json is stale for the same reason. I deliberately did not hand-commit either — they're the regen bot's files, and the re-sign below will just re-stale them. Happy to push a regen after the re-sign if you'd rather see the PR fully green before merging; otherwise the post-merge bot picks it up.

/nvskills-ci

@jasonnvidia

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

svc-nvskills-signing and others added 3 commits November 10, 2026 13:32
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
@mosheabr
mosheabr merged commit 817b875 into main Nov 10, 2026
9 checks passed
mosheabr pushed a commit that referenced this pull request Nov 10, 2026
…ult 25

hermes skills search returns 25 results by default, which silently truncates
the broad single-keyword queries this guidance asks for: jetson matches 39
skills, so jetson-quick-start was cut off and never recommended. Pass
--limit 100 on every search.

Also note that --source nvidia requires Hermes 0.22 or newer, and give the
fallback for older installs.

Reported by teknium1 on #820.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
@mosheabr
mosheabr deleted the jdudash/skill-finder-hermes-catalog branch November 10, 2026 15:43
jasonnvidia added a commit to jasonnvidia/hermes-agent that referenced this pull request Nov 10, 2026
NVIDIA/skills#820 landed on main, so repin to the commit that attaches the
NVSkills signatures for it. The plugin at the old pin still reached for a
python3 -c JSON fallback that Hermes's approval layer refuses as "script
execution via -e/-c flag" (and that was a SyntaxError on Python 3.11 and
older); the new pin replaces it with a json.tool pipeline.

Also bumps the cosmetic version label to match plugin.json (1.5.0) and moves
the banner URL to the same commit.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
teknium1 pushed a commit to NousResearch/hermes-agent that referenced this pull request Nov 11, 2026
NVIDIA/skills#820 landed on main, so repin to the commit that attaches the
NVSkills signatures for it. The plugin at the old pin still reached for a
python3 -c JSON fallback that Hermes's approval layer refuses as "script
execution via -e/-c flag" (and that was a SyntaxError on Python 3.11 and
older); the new pin replaces it with a json.tool pipeline.

Also bumps the cosmetic version label to match plugin.json (1.5.0) and moves
the banner URL to the same commit.

Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants