Skip to content

NVIDIA/skills nim sync fixes and signing #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ohadmo merged 4 commits into from
Oct 21, 2026
Merged

NVIDIA/skills nim sync fixes and signing #57

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
51efb04
skill card
ohadmo Oct 20, 2026
1c07594
Fix public production API
ohadmo Oct 20, 2026
c02a045
resolve validation findings and configure eval credentials
ohadmo Oct 20, 2026
25ae7f9
Attach NVSkills validation signatures
svc-nvskills-signing Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ flow as well. Skills are also discoverable by partner harnesses directly from th
| **OpenFold2** | Monomer protein structure prediction (NIM). | [`openfold2-nim`](nim-skills/openfold2-nim) |
| **OpenFold3** | Biomolecular complex structure prediction (NIM). | [`openfold3-nim`](nim-skills/openfold3-nim) |
| **ProteinMPNN** | Inverse folding / sequence design for a target backbone (NIM). | [`proteinmpnn-nim`](nim-skills/proteinmpnn-nim) |
| **RFD4-Proteina** | Joint protein structure and sequence design, conditioned generation, and reward-guided search through internal staging or a local release-candidate NIM. | [`rfd4-proteina-nim`](nim-skills/rfd4-proteina-nim) |
| **RFD4-Proteina** | Joint protein structure and sequence design, conditioned generation, and reward-guided search through the NVIDIA hosted API or public NIM container. | [`rfd4-proteina-nim`](nim-skills/rfd4-proteina-nim) |
| **RFdiffusion** | De novo protein backbone and binder design (NIM). | [`rfdiffusion-nim`](nim-skills/rfdiffusion-nim) |
| **RFD4-Proteina (open model)** | Native model setup, annotated-CIF authoring, generation, search, evaluation, campaign tracking, and delivery. Requires a separate RFProteina model checkout. | [`rfd4-proteina-setup`](open-models-skills/rfd4-proteina/rfd4-proteina-setup), [`rfd4-proteina-annotated-cif`](open-models-skills/rfd4-proteina/rfd4-proteina-annotated-cif), [`rfd4-proteina-generate`](open-models-skills/rfd4-proteina/rfd4-proteina-generate), [`rfd4-proteina-search`](open-models-skills/rfd4-proteina/rfd4-proteina-search), [`rfd4-proteina-evaluate`](open-models-skills/rfd4-proteina/rfd4-proteina-evaluate), [`rfd4-proteina-autoresearch`](open-models-skills/rfd4-proteina/rfd4-proteina-autoresearch), [`rfd4-proteina-commitment-curves`](open-models-skills/rfd4-proteina/rfd4-proteina-commitment-curves), [`rfd4-proteina-campaign`](open-models-skills/rfd4-proteina/rfd4-proteina-campaign), [`rfd4-proteina-package`](open-models-skills/rfd4-proteina/rfd4-proteina-package) |
| **Proteina-Complexa** | Protein binder design for protein and small molecule targets. Combines a pretrained flow-based generative model (built on La-Proteina) with inference-time optimization for high-quality binder generation. | [`complexa-setup`](open-models-skills/proteina-complexa/complexa-setup), [`complexa-target`](open-models-skills/proteina-complexa/complexa-target), [`complexa-design`](open-models-skills/proteina-complexa/complexa-design), [`complexa-sweep`](open-models-skills/proteina-complexa/complexa-sweep), [`complexa-evaluate-pdbs`](open-models-skills/proteina-complexa/complexa-evaluate-pdbs) |
Expand Down Expand Up @@ -90,13 +90,16 @@ OpenFold3 and the MSA-Search to OpenFold3 workflow are pending validation. Their
current hosted runs timed out and should not yet be used for pass/fail or model
comparison.

The RFD4-Proteina NIM skill uses a separate five-case dataset covering internal staging,
local deployment instructions, offline preparation, and result interpretation.
Its imported [NV-CARPS benchmark](nim-skills/rfd4-proteina-nim/BENCHMARK.md) and
[validation evidence](nim-skills/rfd4-proteina-nim/evals/validation-results.json)
retain the source evaluation results. See the
[RFD4 bundle notes](nim-skills/README.md#rfd4-proteina) for provenance,
access requirements, and offline client tests.
The RFD4-Proteina NIM skill targets the
[released product](https://build.nvidia.com/ipd/rfd4-proteina), with a five-case
dataset covering authenticated hosted inference, public container instructions,
offline preparation, and result interpretation. Its
[live agent benchmark](nim-skills/rfd4-proteina-nim/BENCHMARK.md) and a new
service-account signature are pending for this revision;
[offline validation](nim-skills/rfd4-proteina-nim/evals/validation-results.json)
does not establish production inference success. See the
[RFD4 bundle notes](nim-skills/README.md#rfd4-proteina) for public provenance,
access requirements, and client tests.

- **Evo 2**

Expand Down
37 changes: 19 additions & 18 deletions nim-skills/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,33 +16,34 @@ Use the top-level README for setup and agent-discovery guidance.
## RFD4-Proteina

[`rfd4-proteina-nim`](rfd4-proteina-nim/SKILL.md) supports joint protein structure
and sequence design, conditioning, and reward-guided search through an internal
staging endpoint or a local release-candidate deployment. It requires access to
the selected service; the bundle retains its internal visibility metadata and
prerelease version boundaries. Its Python client uses the standard library.

The complete bundle was imported unchanged from `bionemo/rfd4-skill`, branch
`omosafi/nvcarps-signing`, commit
`3dc70a052c6449cd582f8d771cb97bc122f43862`. That commit was authored by
`nvcarps-service-account` and added the
[`BENCHMARK.md`](rfd4-proteina-nim/BENCHMARK.md) and
[`skill.oms.sig`](rfd4-proteina-nim/skill.oms.sig) artifacts.
The [live validation evidence](rfd4-proteina-nim/evals/validation-results.json),
request fixtures, references, tests, and UI metadata are included as well.
Source repository paths and evaluation dates in those files are historical
provenance. Changes to signed content require a new evaluation and signature.
and sequence design, conditioning, and reward-guided search through the released
[NVIDIA API](https://build.nvidia.com/ipd/rfd4-proteina) or a self-hosted public
NIM container. Hosted inference uses `NVIDIA_API_KEY` (`NGC_API_KEY` fallback);
the local client sends no inference credential. The Python client uses only
the standard library.

The bundle's endpoint, authentication, deployment instructions, and PD-1 request
are grounded in public product documentation. See the
[source notes](rfd4-proteina-nim/references/sources.md),
[fixture manifest](rfd4-proteina-nim/assets/source-manifest.json), and
[skill card](rfd4-proteina-nim/skill-card.md).

The previous evaluation/signature no longer applies to these modified files.
[`BENCHMARK.md`](rfd4-proteina-nim/BENCHMARK.md) marks live evaluation as pending;
a new service-account signature must be generated after evaluation. The
[validation record](rfd4-proteina-nim/evals/validation-results.json) describes
offline checks without claiming production inference or local GPU deployment.

The canonical toolkit copy is `nim-skills/rfd4-proteina-nim/`; the generated
plugin copy is `skills/bionemo-agent-toolkit/skills/rfd4-proteina-nim/`.
Both retain the original signed files. Run the client tests from the toolkit
root:
Keep both copies in sync. Run the client tests from the toolkit root:

```bash
python -B -m unittest discover -s nim-skills/rfd4-proteina-nim/tests -v
```

These tests use synthetic responses and a temporary loopback HTTP service.
They do not perform live staging inference.
They do not perform live production inference.

For native RFProteina model workflows, see the separate
[open-model skill collection](../open-models-skills/rfd4-proteina/README.md).
144 changes: 30 additions & 114 deletions nim-skills/rfd4-proteina-nim/BENCHMARK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,125 +1,41 @@
# Skill Benchmark: rfd4-proteina-nim

> ✅ **Overall verdict: PASS — Recommended for publication**
> **Overall verdict: NOT EVALUATED — public-service revision requires a new run**

## Publication Recommendation
## Scope

Recommended for publication based on the completed evaluation evidence in this report.
This revision targets the released RFD4-Proteina NIM at
`https://health.api.nvidia.com/v1/biology/ipd/rfd4-proteina/generative`, with
Bearer authentication, and the public `nvcr.io/nim/ipd/rfd4-proteina` container.
Its endpoint, authentication, examples, documentation, and evaluation tasks have
changed. Earlier scores and signatures do not attest to these files.

## Evaluation Metadata
## Available checks

- Skill: `rfd4-proteina-nim`
- Evaluation date: 2026-10-13
- Evaluator version: `1.5.6`
- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)
- Tasks: 5 evaluation tasks (5 positive)
- Dataset digest: `sha256:f0c2a36cb6f8e1c81678477d5b8b7fed33996d1a6b7f31320272009e6c953a36` (skill-evaluator-dataset-snapshot/1)
- Attempts per task: 1
- Environment: `k8s-sandbox`
- Tier 2 evidence: required for publication
- Tier 3 evidence: required for publication
[Client tests](tests/test_design.py) exercise request preparation, hosted
credential handling, local requests, redirects, pending/error responses, and
artifact extraction using synthetic responses and temporary loopback servers.
[Validation results](evals/validation-results.json) record the checks performed
for this revision. They are not a live agent benchmark or proof of production
inference success.

Each task attempt ran in its own isolated sandbox pod.
## Evaluation dataset

## What This Report Answers
Five task definitions are in [evals/evals.json](evals/evals.json): hosted binder
execution, local deployment instructions, offline search preparation,
condition-spec preparation, and pending/oracle handling. The separate trigger
set has nine positive and six negative queries.

The three-tier evaluation checks whether the skill:
Hosted execution requires a valid NVIDIA API key made available to the agent
through the evaluation environment's secret mechanism and access to the product.
Offline cases require neither credentials nor a live NIM. A missing credential
or failed service request must remain a reported blocker, not a fabricated run.

- is safe to use;
- produces correct answers;
- is discovered and activated when needed;
- helps the agent complete the user's goal and expected workflow; and
- avoids wasted skill and tool usage.
## Results

## Results at a Glance

| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
|---|---:|---:|
| Overall | Not available | 84.1% — baseline ran, but no comparable score was available; uplift unavailable |
| Security | Not available | 80.0% → 80.0% (±0.0 points) |
| Correctness | Not available | 32.0% → 84.0% (+52.0 points) |
| Discoverability | Not available | 89.0% — baseline ran, but no comparable score was available; uplift unavailable |
| Effectiveness | Not available | 35.0% → 76.5% (+41.5 points) |
| Efficiency | Not available | 91.2% — baseline ran, but no comparable score was available; uplift unavailable |

**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.

Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.

## Token Usage

Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.

| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |
|---|---|---:|---:|---:|---:|---|
| claude-code | All cases | 2,333,177 | 2,287,391 | +45,786 | +2.00% | skill 5/5; base 5/5 |
| claude-code | condition-spec | 647,875 | 475,447 | +172,428 | +36.27% | skill 1/1; base 1/1 |
| claude-code | local-rc-instructions | 105,080 | 347,202 | -242,122 | -69.74% | skill 1/1; base 1/1 |
| claude-code | offline-search | 783,943 | 737,343 | +46,600 | +6.32% | skill 1/1; base 1/1 |
| claude-code | oracle-and-pending | 63,527 | 347,543 | -284,016 | -81.72% | skill 1/1; base 1/1 |
| claude-code | staging-enzyme | 732,752 | 379,856 | +352,896 | +92.90% | skill 1/1; base 1/1 |
| codex | All cases | 624,263 | 3,021,821 | -2,397,558 | -79.34% | skill 5/5; base 5/5 |
| codex | condition-spec | 66,307 | 90,244 | -23,937 | -26.52% | skill 1/1; base 1/1 |
| codex | local-rc-instructions | 48,769 | 76,050 | -27,281 | -35.87% | skill 1/1; base 1/1 |
| codex | offline-search | 249,759 | 374,076 | -124,317 | -33.23% | skill 1/1; base 1/1 |
| codex | oracle-and-pending | 29,820 | 105,430 | -75,610 | -71.72% | skill 1/1; base 1/1 |
| codex | staging-enzyme | 229,608 | 2,376,021 | -2,146,413 | -90.34% | skill 1/1; base 1/1 |
| ALL AGENTS | Dataset aggregate | 2,957,440 | 5,309,212 | -2,351,772 | -44.30% | skill 10/10; base 10/10 |

Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated.

## Tier Status

| Tier | Purpose | Status | Evidence |
|---|---|---|---|
| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 4 finding(s) |
| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) |
| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 5 task(s) |

## Findings and Observations

<details>
<summary>Show detailed findings and successful checks</summary>

- **MEDIUM** PII/phone_numbers: International phone number (`assets/requests/25_secondary_structure_conditioning.json:8`)
- **MEDIUM** QUALITY/quality_correctness: Instructions don't mention 'run_script' (`skills/rfd4-proteina-nim/SKILL.md`)
- **MEDIUM** SECURITY/Whitespace Padding (P9): Prompt Injection: U+0020 x121 (`assets/requests/16_ppi_binder.json:8`)
- **LOW** SCRIPT_LINT/magic_numbers: design.py contains magic numbers (`skills/rfd4-proteina-nim/scripts/design.py`)

</details>

## Scoring Methodology

<details>
<summary>Show dimension definitions, source signals, and thresholds</summary>

| Dimension | Question | Scored signals |
|---|---|---|
| Security | Is it safe to use? | `security` (100%) |
| Correctness | Is the answer correct? | `accuracy` (100%) |
| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) |

- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.

Signals present in this run:

- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
- `accuracy` (Accuracy): final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).

</details>

## Freshness

Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.
No live-agent scores, production inference results, or local GPU deployment
results are claimed for this revision. Regenerate the benchmark with the
published evaluation tooling and obtain a new service-account signature after
the final content passes evaluation. Catalog onboarding remains pending until
that evidence is attached; an offline test result must not be relabeled PASS
for Tier 3.
Loading
Loading