diff --git a/skills/nvidia-ontology-management/BENCHMARK.md b/skills/nvidia-ontology-management/BENCHMARK.md index 0d4b514bd..889c25e0b 100644 --- a/skills/nvidia-ontology-management/BENCHMARK.md +++ b/skills/nvidia-ontology-management/BENCHMARK.md @@ -9,9 +9,10 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata - Skill: `nvidia-ontology-management` -- Evaluation date: 2026-10-08 -- Evaluator version: `1.5.6` -- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) +- Validation status: `passed` +- Evaluation date: 2026-11-16 +- Evaluator version: `1.6.9` +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`), Codex (`openai/openai/gpt-6.1-sol`) - Tasks: 13 evaluation tasks (13 positive) - Dataset digest: `sha256:4c8455d88fc45fd16ded90344c740ea7149fe29edbca7f112b346c21177d23fa` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 @@ -21,6 +22,20 @@ Recommended for publication based on the completed evaluation evidence in this r Each task attempt ran in its own isolated sandbox pod. +## Tier 1 LLM Provenance + +SkillSpector provider, model, analyzer revision, and determinism evidence retained for this scan. + +| Skill | Provider | Adapter | Analyzer | Model | Revision | Determinism | +| --- | --- | --- | --- | --- | --- | --- | +| nvidia-ontology-management | nv_inference | openai | mcp_tool_poisoning | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-management | nv_inference | openai | semantic_developer_intent | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-management | nv_inference | openai | semantic_quality_policy | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-management | nv_inference | openai | semantic_security_discovery | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-management | nv_inference | openai | meta_analyzer | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | + +- Controls for `nvidia-ontology-management`: temperature=not set (source=provider_default, forwarded=no); seed=not set (source=unset, forwarded=no); reasoning_effort=not set (source=provider_default, forwarded=no) + ## What This Report Answers The three-tier evaluation checks whether the skill: @@ -35,61 +50,72 @@ The three-tier evaluation checks whether the skill: | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 94.9% — baseline ran, but no comparable score was available; uplift unavailable | 85.3% — baseline ran, but no comparable score was available; uplift unavailable | -| Security | 100.0% → 100.0% (±0.0 points) | 84.6% → 76.9% (-7.7 points) | -| Correctness | 29.2% → 100.0% (+70.8 points) | 47.7% → 89.2% (+41.5 points) | -| Discoverability | 93.7% — baseline ran, but no comparable score was available; uplift unavailable | 85.8% — baseline ran, but no comparable score was available; uplift unavailable | -| Effectiveness | 42.4% → 89.3% (+46.9 points) | 33.9% → 81.1% (+47.2 points) | -| Efficiency | 91.4% — baseline ran, but no comparable score was available; uplift unavailable | 93.3% — baseline ran, but no comparable score was available; uplift unavailable | +| Overall | 86.9% — baseline ran, but no comparable score was available; uplift unavailable | 82.0% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 88.5% (-11.5 points) | 100.0% → 84.6% (-15.4 points) | +| Correctness | 38.5% → 95.4% (+56.9 points) | 49.2% → 80.0% (+30.8 points) | +| Discoverability | 90.8% — baseline ran, but no comparable score was available; uplift unavailable | 94.2% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 27.8% → 74.4% (+46.6 points) | 40.4% → 66.7% (+26.3 points) | +| Efficiency | 85.4% — baseline ran, but no comparable score was available; uplift unavailable | 84.3% — baseline ran, but no comparable score was available; uplift unavailable | **How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline. +### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 98.5% → 96.2% (-2.3 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 91.7% → 90.6% (-1.1 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + ## Token Usage Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions. | Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage | |---|---|---:|---:|---:|---:|---| -| claude-code | All cases | 2,136,288 | 2,211,810 | -75,522 | -3.41% | skill 13/13; base 13/13 | -| claude-code | management-eval-001-mcp-readonly | 176,205 | 177,943 | -1,738 | -0.98% | skill 1/1; base 1/1 | -| claude-code | management-eval-002-validate-422 | 91,346 | 120,663 | -29,317 | -24.30% | skill 1/1; base 1/1 | -| claude-code | management-eval-003-reset-is-destructive | 97,849 | 150,421 | -52,572 | -34.95% | skill 1/1; base 1/1 | -| claude-code | management-eval-004-token-acts-as-owner | 95,573 | 119,914 | -24,341 | -20.30% | skill 1/1; base 1/1 | -| claude-code | management-eval-005-import-replace | 99,977 | 121,086 | -21,109 | -17.43% | skill 1/1; base 1/1 | -| claude-code | management-eval-006-lineage-semantic-layer | 285,753 | 119,687 | +166,066 | +138.75% | skill 1/1; base 1/1 | -| claude-code | management-eval-007-patch-vs-put-sql-attribute | 95,844 | 220,722 | -124,878 | -56.58% | skill 1/1; base 1/1 | -| claude-code | management-eval-008-run-grain-measure | 95,583 | 30,926 | +64,657 | +209.07% | skill 1/1; base 1/1 | -| claude-code | management-eval-009-relationship-evidence | 247,442 | 247,503 | -61 | -0.02% | skill 1/1; base 1/1 | -| claude-code | management-eval-010-import-exact-readback | 97,134 | 216,736 | -119,602 | -55.18% | skill 1/1; base 1/1 | -| claude-code | management-eval-011-result-publication-mode | 161,713 | 317,242 | -155,529 | -49.03% | skill 1/1; base 1/1 | -| claude-code | management-eval-012-promotion-readback-failure | 133,068 | 183,802 | -50,734 | -27.60% | skill 1/1; base 1/1 | -| claude-code | management-eval-013-composed-lifecycle | 458,801 | 185,165 | +273,636 | +147.78% | skill 1/1; base 1/1 | -| codex | All cases | 2,393,113 | 2,250,240 | +142,873 | +6.35% | skill 13/13; base 13/13 | -| codex | management-eval-001-mcp-readonly | 797,280 | 452,293 | +344,987 | +76.28% | skill 1/1; base 1/1 | -| codex | management-eval-002-validate-422 | 57,538 | 54,886 | +2,652 | +4.83% | skill 1/1; base 1/1 | -| codex | management-eval-003-reset-is-destructive | 57,904 | 54,809 | +3,095 | +5.65% | skill 1/1; base 1/1 | -| codex | management-eval-004-token-acts-as-owner | 45,901 | 54,971 | -9,070 | -16.50% | skill 1/1; base 1/1 | -| codex | management-eval-005-import-replace | 161,338 | 55,064 | +106,274 | +193.00% | skill 1/1; base 1/1 | -| codex | management-eval-006-lineage-semantic-layer | 85,423 | 98,141 | -12,718 | -12.96% | skill 1/1; base 1/1 | -| codex | management-eval-007-patch-vs-put-sql-attribute | 45,034 | 65,171 | -20,137 | -30.90% | skill 1/1; base 1/1 | -| codex | management-eval-008-run-grain-measure | 44,493 | 54,668 | -10,175 | -18.61% | skill 1/1; base 1/1 | -| codex | management-eval-009-relationship-evidence | 413,138 | 83,561 | +329,577 | +394.41% | skill 1/1; base 1/1 | -| codex | management-eval-010-import-exact-readback | 44,648 | 55,235 | -10,587 | -19.17% | skill 1/1; base 1/1 | -| codex | management-eval-011-result-publication-mode | 118,995 | 475,882 | -356,887 | -74.99% | skill 1/1; base 1/1 | -| codex | management-eval-012-promotion-readback-failure | 110,830 | 262,196 | -151,366 | -57.73% | skill 1/1; base 1/1 | -| codex | management-eval-013-composed-lifecycle | 410,591 | 483,363 | -72,772 | -15.06% | skill 1/1; base 1/1 | -| ALL AGENTS | Dataset aggregate | 4,529,401 | 4,462,050 | +67,351 | +1.51% | skill 26/26; base 26/26 | - -Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated. +| claude-code | All cases | 2,016,341 | 1,534,739 | +481,602 | +31.38% | skill 13/13; base 13/13 | +| claude-code | management-eval-001-mcp-readonly | 154,696 | 107,414 | +47,282 | +44.02% | skill 1/1; base 1/1 | +| claude-code | management-eval-002-validate-422 | 188,571 | 142,209 | +46,362 | +32.60% | skill 1/1; base 1/1 | +| claude-code | management-eval-003-reset-is-destructive | 111,697 | 142,645 | -30,948 | -21.70% | skill 1/1; base 1/1 | +| claude-code | management-eval-004-token-acts-as-owner | 145,240 | 142,526 | +2,714 | +1.90% | skill 1/1; base 1/1 | +| claude-code | management-eval-005-import-replace | 229,424 | 142,862 | +86,562 | +60.59% | skill 1/1; base 1/1 | +| claude-code | management-eval-006-lineage-semantic-layer | 186,578 | 141,919 | +44,659 | +31.47% | skill 1/1; base 1/1 | +| claude-code | management-eval-007-patch-vs-put-sql-attribute | 113,257 | 70,644 | +42,613 | +60.32% | skill 1/1; base 1/1 | +| claude-code | management-eval-008-run-grain-measure | 113,012 | 71,609 | +41,403 | +57.82% | skill 1/1; base 1/1 | +| claude-code | management-eval-009-relationship-evidence | 200,954 | 142,716 | +58,238 | +40.81% | skill 1/1; base 1/1 | +| claude-code | management-eval-010-import-exact-readback | 113,886 | 106,998 | +6,888 | +6.44% | skill 1/1; base 1/1 | +| claude-code | management-eval-011-result-publication-mode | 186,668 | 108,115 | +78,553 | +72.66% | skill 1/1; base 1/1 | +| claude-code | management-eval-012-promotion-readback-failure | 113,833 | 107,246 | +6,587 | +6.14% | skill 1/1; base 1/1 | +| claude-code | management-eval-013-composed-lifecycle | 158,525 | 107,836 | +50,689 | +47.01% | skill 1/1; base 1/1 | +| codex | All cases | 891,349 | 639,118 | +252,231 | +39.47% | skill 13/13; base 13/13 | +| codex | management-eval-001-mcp-readonly | 81,756 | 55,952 | +25,804 | +46.12% | skill 1/1; base 1/1 | +| codex | management-eval-002-validate-422 | 64,735 | 55,864 | +8,871 | +15.88% | skill 1/1; base 1/1 | +| codex | management-eval-003-reset-is-destructive | 48,290 | 60,041 | -11,751 | -19.57% | skill 1/1; base 1/1 | +| codex | management-eval-004-token-acts-as-owner | 57,097 | 91,171 | -34,074 | -37.37% | skill 1/1; base 1/1 | +| codex | management-eval-005-import-replace | 52,670 | 40,926 | +11,744 | +28.70% | skill 1/1; base 1/1 | +| codex | management-eval-006-lineage-semantic-layer | 107,452 | 40,567 | +66,885 | +164.88% | skill 1/1; base 1/1 | +| codex | management-eval-007-patch-vs-put-sql-attribute | 48,329 | 61,639 | -13,310 | -21.59% | skill 1/1; base 1/1 | +| codex | management-eval-008-run-grain-measure | 48,486 | 18,012 | +30,474 | +169.19% | skill 1/1; base 1/1 | +| codex | management-eval-009-relationship-evidence | 45,829 | 40,779 | +5,050 | +12.38% | skill 1/1; base 1/1 | +| codex | management-eval-010-import-exact-readback | 49,037 | 50,961 | -1,924 | -3.78% | skill 1/1; base 1/1 | +| codex | management-eval-011-result-publication-mode | 44,128 | 13,309 | +30,819 | +231.57% | skill 1/1; base 1/1 | +| codex | management-eval-012-promotion-readback-failure | 44,690 | 54,363 | -9,673 | -17.79% | skill 1/1; base 1/1 | +| codex | management-eval-013-composed-lifecycle | 198,850 | 55,534 | +143,316 | +258.07% | skill 1/1; base 1/1 | +| ALL AGENTS | Dataset aggregate | 2,907,690 | 2,173,857 | +733,833 | +33.76% | skill 26/26; base 26/26 | + +Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. Change is `100 * (with - without) / without`; negative means fewer tokens. N/A comparisons include their reason in the coverage column; coverage is never estimated. ## Tier Status | Tier | Purpose | Status | Evidence | |---|---|---|---| | Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 5 finding(s) | -| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) | +| Tier 2 | Semantic deduplication | **PASSED WITH OBSERVATIONS** | 2 validator(s); 1 finding(s) | | Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 13 task(s) | ## Findings and Observations @@ -97,11 +123,14 @@ Prompt tokens include cached reads, so total tokens are `prompt + completion` (c
Show detailed findings and successful checks +- **HIGH** DUPLICATE/duplicate: Duplicate content found across SKILL.md and references/write-api.md: + "## Bulk import / export" in SKILL.md (lines 97-109) + vs "## Model interchange" in references/write-api.md (lines 59-68) (`SKILL.md:97`) - **MEDIUM** QUALITY/quality_correctness: Resource-based skill lacks documentation of available resources (`skills/nvidia-ontology-management/SKILL.md`) - **MEDIUM** QUALITY/quality_reliability: MCP skill lacks connection/error guidance (`skills/nvidia-ontology-management/SKILL.md`) - **MEDIUM** QUALITY/quality_efficiency: Deeply nested references in modeling.md (`skills/nvidia-ontology-management/SKILL.md`) - **LOW** QUALITY/quality_reliability: Inputs are used but no dedicated Inputs section is documented (`skills/nvidia-ontology-management/SKILL.md`) -- **LOW** QUALITY/quality_reliability: Structured output is used but no dedicated Output Format section is documented (`skills/nvidia-ontology-management/SKILL.md`) +- 1 additional finding(s) are available in the full evaluation artifacts.
diff --git a/skills/nvidia-ontology-management/SKILL.md b/skills/nvidia-ontology-management/SKILL.md index c03bf84da..aa2eb623d 100644 --- a/skills/nvidia-ontology-management/SKILL.md +++ b/skills/nvidia-ontology-management/SKILL.md @@ -101,7 +101,7 @@ Stay on the semantic layer. Do not browse raw schemas to answer meaning. - Term → columns: MCP `get_term_columns` or `GET /api/terms/{term_id}/column-attributes` - Term → derived SQL: MCP `get_term_sql_attributes` -- Table → terms and SQL attributes: MCP `describe_table` or +- Table → terms and stored queries that reference it: MCP `describe_table` or `GET /api/exploration/tables/{table_id}/details` - Semantic hop chain between two terms: `GET /api/exploration/terms/{term_id}/path/{other_term_id}` diff --git a/skills/nvidia-ontology-management/skill-card.md b/skills/nvidia-ontology-management/skill-card.md index 9fbd362d9..27afa2540 100644 --- a/skills/nvidia-ontology-management/skill-card.md +++ b/skills/nvidia-ontology-management/skill-card.md @@ -1,5 +1,11 @@ + + ## Description:
-Model and publish semantic definitions in Auto Ontology, covering terms, relationships, measures, imports, and governed results (not deployment or querying).
+Model and publish semantic definitions in Auto Ontology. Use for terms, relationships, measures, imports, and governed results—not deployment or querying.
This skill is ready for commercial/non-commercial use.
@@ -9,7 +15,7 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and data engineers use this skill to have an agent inspect and change the Auto Ontology semantic layer: editing terms and SQL attributes, defining source-grounded concepts, relationships, and measures, importing or exporting models with exact readback, and publishing governed definitions or results.
+Developers, data engineers, and AI agents working with an authenticated Auto Ontology deployment use this skill to inspect and change the semantic layer—terms, column and SQL attributes, relationships, measures, and model import/export—and to publish governed results with validation and exact readback.
### Deployment Geography for Use:
Global
@@ -32,48 +38,79 @@ Mitigation: Review and scan skill before deployment.
## Skill Output:
-**Output Type(s):** [API Calls, Analysis, Configuration instructions]
-**Output Format:** [Markdown with inline HTTP request examples against the Next.js /api gateway]
+**Output Type(s):** [Analysis, API Calls, Configuration instructions]
+**Output Format:** [Markdown with inline API request examples]
**Output Parameters:** [1D]
-**Other Properties Related to Output:** [MCP access is read-only; writes need the token owner's permissions, and unsupported publication stops at an explicit receipt]
+**Other Properties Related to Output:** [None]
## Evaluation Agents Used:
-- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`)
-- Codex (`openai/openai/gpt-5.5`)
+- Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`)
+- Codex (`openai/openai/gpt-6.1-sol`)
## Evaluation Tasks:
-13 positive evaluation tasks, one attempt per task, each run in an isolated k8s sandbox pod and compared against a no-skill baseline.
+13 evaluation tasks (13 positive), 1 attempt per task, run in the `k8s-sandbox` environment with each task attempt in its own isolated sandbox pod; each task was also attempted without the skill as a no-skill baseline.
## Evaluation Metrics Used:
Reported benchmark dimensions:
-- Security: Is it safe to use? (scored from `security`)
-- Correctness: Is the answer correct? (scored from `accuracy`)
-- Discoverability: Was the right skill loaded when needed? (scored from `skill_execution`)
-- Effectiveness: Did the skill help complete the task? (50% `goal_accuracy` + 50% `behavior_check`)
-- Efficiency: Did it avoid wasted tool calls and token usage? (50% `skill_efficiency` + 50% `token_efficiency`)
+- Security: Is it safe to use? Scored from `security` (100%).
+- Correctness: Is the answer correct? Scored from `accuracy` (100%).
+- Discoverability: Was the right skill loaded when needed? Scored from `skill_execution` (100%).
+- Effectiveness: Did the skill help complete the task? Equal-weight mean of `goal_accuracy` (50%) and `behavior_check` (50%).
+- Efficiency: Did it avoid wasted tool calls and token usage? Scored from `skill_efficiency` (50%) and `token_efficiency` (50%).
Underlying evaluation signals used in this run:
-- `security`: Unsafe operations, secret leakage, and unauthorized access.
-- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed.
-- `skill_efficiency`: Tool-call productivity.
-- `accuracy`: Final-answer correctness against the reference answer.
-- `goal_accuracy`: Whether the user's goal was achieved.
-- `behavior_check`: Whether the expected workflow behavior was followed.
-- `token_efficiency`: Actual uncached prompt plus completion usage.
+- `security`: Security: unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution`: Skill Execution: whether the expected skill was selected, decoys were avoided, and the workflow executed.
+- `skill_efficiency`: Tool Productivity: tool-call productivity (legacy wire id; routing is scored under Discoverability).
+- `accuracy`: Accuracy: final-answer correctness against the reference answer.
+- `goal_accuracy`: Goal Accuracy: whether the user's goal was achieved.
+- `behavior_check`: Behavior Check: whether the expected workflow behavior was followed.
+- `token_efficiency`: Token Efficiency: actual uncached prompt plus completion usage (50% of Efficiency).
## Evaluation Results:
+> ✅ **Overall verdict: PASS — Recommended for publication** + +### Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. + +### Results at a Glance + | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 94.9% (no comparable baseline; uplift unavailable) | 85.3% (no comparable baseline; uplift unavailable) | -| Security | 100.0% → 100.0% (±0.0 points) | 84.6% → 76.9% (-7.7 points) | -| Correctness | 29.2% → 100.0% (+70.8 points) | 47.7% → 89.2% (+41.5 points) | -| Discoverability | 93.7% (no comparable baseline; uplift unavailable) | 85.8% (no comparable baseline; uplift unavailable) | -| Effectiveness | 42.4% → 89.3% (+46.9 points) | 33.9% → 81.1% (+47.2 points) | -| Efficiency | 91.4% (no comparable baseline; uplift unavailable) | 93.3% (no comparable baseline; uplift unavailable) | +| Overall | 86.9% — baseline ran, but no comparable score was available; uplift unavailable | 82.0% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 88.5% (-11.5 points) | 100.0% → 84.6% (-15.4 points) | +| Correctness | 38.5% → 95.4% (+56.9 points) | 49.2% → 80.0% (+30.8 points) | +| Discoverability | 90.8% — baseline ran, but no comparable score was available; uplift unavailable | 94.2% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 27.8% → 74.4% (+46.6 points) | 40.4% → 66.7% (+26.3 points) | +| Efficiency | 85.4% — baseline ran, but no comparable score was available; uplift unavailable | 84.3% — baseline ran, but no comparable score was available; uplift unavailable | + +**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. + +Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline. + +### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 98.5% → 96.2% (-2.3 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 91.7% → 90.6% (-1.1 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + +### Tier Status + +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 5 finding(s) | +| Tier 2 | Semantic deduplication | **PASSED WITH OBSERVATIONS** | 2 validator(s); 1 finding(s) | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 13 task(s) | ## Skill Version(s):
0.2.1 (source: frontmatter)
diff --git a/skills/nvidia-ontology-management/skill.oms.sig b/skills/nvidia-ontology-management/skill.oms.sig index adafad32e..7b91cc2cc 100644 --- a/skills/nvidia-ontology-management/skill.oms.sig +++ b/skills/nvidia-ontology-management/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibnZpZGlhLW9udG9sb2d5LW1hbmFnZW1lbnQiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiOWY2YzliNWFkMTZiZTdlODhhNmJiODBhZmI2YTZhN2E2YTE5YmZjZTIxZDYwNTY2Zjg5NWExN2UxY2NlNjU5MiIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJhbGxvd19zeW1saW5rcyI6IGZhbHNlLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIsCiAgICAgICJtZXRob2QiOiAiZmlsZXMiLAogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGlnbm9yZSIKICAgICAgXQogICAgfSwKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjY1MTA5ZDk0ZWI1MjgxOTJlY2IxNWEwNWMwMTBmNDE0NTk2NWQ0ZDFhNTI1NDIwOWI0NTgxODVlOWMyZjRiMTAiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiLAogICAgICAgICJkaWdlc3QiOiAiMGFmMjNkY2IyNWU5MzVmMjdlZjUxMThhNmE5ZDg0M2VhNzZjMWU2NjUwNGFmZWJhNjc2MzNjNTMzMWMwMjM0ZSIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJhc3NldHMvcnVudGltZS1jb250cmFjdC55YW1sIiwKICAgICAgICAiZGlnZXN0IjogIjQ5ZmE2NzllNGE2YmY3YzkyYzIwNDlmZGUxNjNmNmJlYTAzMTgxYjAzNGY0OTQ2OTg1NGVmZjBjY2JkYTFiMTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImRpZ2VzdCI6ICI3NWJkYmE3Nzk5NjY5MGJiNTJkMmNhYzFkNzIxYmRmZDFmN2YyNjZlODI4OTczZWNiYmM0ZjYxOTY3ZWMxMTVlIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvbW9kZWxpbmcubWQiLAogICAgICAgICJkaWdlc3QiOiAiZjFmYWI2NWQwNThhZDVkMTNlNWJiZDU4ZTNjM2RhOTJiYTlmMTQwZjVmOTIyOTY1N2RhNTFmMTVhOTI3ODRmYSIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3B1YmxpY2F0aW9uLm1kIiwKICAgICAgICAiZGlnZXN0IjogImI2MWUzNTFiMjcxM2NjNDc5MmJhM2U2OWM4YTFhNjE3OWUwYzdhYjVkMDkwMTlmNzllNjYxMmM3YmMxMzMwMWMiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy93cml0ZS1hcGkubWQiLAogICAgICAgICJkaWdlc3QiOiAiNjU3YmU2OTZlZjhjNmRlODlkMDU1YTg5NWQ3NTM2MjU2NTM5MDVlYzE1ZmVjNThiOGJmZjJkY2EwMTVjNzA0OSIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjA1OGQwYTkzYjliOWExMWJkNTkwYzZhMjFhOTRlY2EzOGRjMDBmYTlhMjIwYzVhMDA1NTMyYjQzNGRkMmY5YWYiCiAgICAgIH0KICAgIF0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQDZJQ8wkNKYVM46so3YazFiB3B0jf77NlNpWJ3bC3obRgSLox0CMSe5mKJ+wz9otaACMH7C3CcDOhNYIvI9wqQyglhU0vlb9DZJopIe1VoSJO5Nl4UxWuFWuP7EHbJPQvktcw==","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibnZpZGlhLW9udG9sb2d5LW1hbmFnZW1lbnQiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiNTRmYjM1NDRiOWM5ZWY0NjFiMmNiZDk0NzAxYTQxN2VkM2RlMWQ1NjE4YjBkMjE1NmM0ZDBjNjM5YWM4YTg1NiIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJtZXRob2QiOiAiZmlsZXMiLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0aWdub3JlIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRodWIiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIKICAgICAgXQogICAgfSwKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJCRU5DSE1BUksubWQiLAogICAgICAgICJkaWdlc3QiOiAiOWU1ODZkOTY0MTZlNGFmOWFmNjA2ZTQ3M2U2MWRhYzQ5YzljZGJlYWI2NDY2OWFjMjQwYTY0ZjAwNDlkNDM4MCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJTS0lMTC5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICI1ZDg1NjAwY2JkMjJmYjAwMWVhZjY0ZTBhNWFmNWFlODc1OTViYjkwZTMyNjY0YzYxZmFhNWJhODdjNzYxMTMyIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogImFzc2V0cy9ydW50aW1lLWNvbnRyYWN0LnlhbWwiLAogICAgICAgICJkaWdlc3QiOiAiNDlmYTY3OWU0YTZiZjdjOTJjMjA0OWZkZTE2M2Y2YmVhMDMxODFiMDM0ZjQ5NDY5ODU0ZWZmMGNjYmRhMWIxNiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJldmFscy9ldmFscy5qc29uIiwKICAgICAgICAiZGlnZXN0IjogIjc1YmRiYTc3OTk2NjkwYmI1MmQyY2FjMWQ3MjFiZGZkMWY3ZjI2NmU4Mjg5NzNlY2JiYzRmNjE5NjdlYzExNWUiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9tb2RlbGluZy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICJmMWZhYjY1ZDA1OGFkNWQxM2U1YmJkNThlM2MzZGE5MmJhOWYxNDBmNWY5MjI5NjU3ZGE1MWYxNWE5Mjc4NGZhIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvcHVibGljYXRpb24ubWQiLAogICAgICAgICJkaWdlc3QiOiAiYjYxZTM1MWIyNzEzY2M0NzkyYmEzZTY5YzhhMWE2MTc5ZTBjN2FiNWQwOTAxOWY3OWU2NjEyYzdiYzEzMzAxYyIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3dyaXRlLWFwaS5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICI2NTdiZTY5NmVmOGM2ZGU4OWQwNTVhODk1ZDc1MzYyNTY1MzkwNWVjMTVmZWM1OGI4YmZmMmRjYTAxNWM3MDQ5IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJkaWdlc3QiOiAiZjNjM2ZlZmExZjExMDQzMzllYTZkMGRiNTFiMzY1MGRjYjM1ZDc4MTFjNWRhMWQwMGIwOTI0Yzc0ZGNiNDM1YSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0KICAgIF0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGQCMBRO2F5z30qf8rYdla82/IcTl80GqpgeZhKqsxxwbmrubRi6g0Cefu3WWbzbpdb82gIwATKqH+SkWB4+MySkoInhpWtUx6iDcCeZfJ9XY/lc3X+us2PfPv3F1SNr8vqzesAR","keyid":""}]}} \ No newline at end of file diff --git a/skills/nvidia-ontology-query/BENCHMARK.md b/skills/nvidia-ontology-query/BENCHMARK.md index 3f72a7abf..3df35f0ed 100644 --- a/skills/nvidia-ontology-query/BENCHMARK.md +++ b/skills/nvidia-ontology-query/BENCHMARK.md @@ -9,9 +9,10 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata - Skill: `nvidia-ontology-query` -- Evaluation date: 2026-10-08 -- Evaluator version: `1.5.6` -- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) +- Validation status: `passed` +- Evaluation date: 2026-11-16 +- Evaluator version: `1.6.9` +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`), Codex (`openai/openai/gpt-6.1-sol`) - Tasks: 13 evaluation tasks (13 positive) - Dataset digest: `sha256:314823168b2eada85a48a450189c9d2cc1d374eed343c106147bd2b63080bf72` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 @@ -21,6 +22,20 @@ Recommended for publication based on the completed evaluation evidence in this r Each task attempt ran in its own isolated sandbox pod. +## Tier 1 LLM Provenance + +SkillSpector provider, model, analyzer revision, and determinism evidence retained for this scan. + +| Skill | Provider | Adapter | Analyzer | Model | Revision | Determinism | +| --- | --- | --- | --- | --- | --- | --- | +| nvidia-ontology-query | nv_inference | openai | mcp_tool_poisoning | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-query | nv_inference | openai | semantic_developer_intent | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-query | nv_inference | openai | semantic_quality_policy | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-query | nv_inference | openai | semantic_security_discovery | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-query | nv_inference | openai | meta_analyzer | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | + +- Controls for `nvidia-ontology-query`: temperature=not set (source=provider_default, forwarded=no); seed=not set (source=unset, forwarded=no); reasoning_effort=not set (source=provider_default, forwarded=no) + ## What This Report Answers The three-tier evaluation checks whether the skill: @@ -35,12 +50,12 @@ The three-tier evaluation checks whether the skill: | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 95.3% — baseline ran, but no comparable score was available; uplift unavailable | 80.4% — baseline ran, but no comparable score was available; uplift unavailable | -| Security | 100.0% → 100.0% (±0.0 points) | 84.6% → 92.3% (+7.7 points) | -| Correctness | 47.7% → 100.0% (+52.3 points) | 50.8% → 70.8% (+20.0 points) | -| Discoverability | 84.6% — baseline ran, but no comparable score was available; uplift unavailable | 75.3% — baseline ran, but no comparable score was available; uplift unavailable | -| Effectiveness | 55.5% → 96.0% (+40.5 points) | 48.8% → 68.5% (+19.7 points) | -| Efficiency | 95.6% — baseline ran, but no comparable score was available; uplift unavailable | 95.2% — baseline ran, but no comparable score was available; uplift unavailable | +| Overall | 91.3% — baseline ran, but no comparable score was available; uplift unavailable | 89.9% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) | +| Correctness | 53.9% → 95.4% (+41.5 points) | 55.4% → 87.7% (+32.3 points) | +| Discoverability | 85.8% — baseline ran, but no comparable score was available; uplift unavailable | 85.5% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 49.7% → 88.5% (+38.8 points) | 49.0% → 84.0% (+35.0 points) | +| Efficiency | 86.6% — baseline ran, but no comparable score was available; uplift unavailable | 92.3% — baseline ran, but no comparable score was available; uplift unavailable | **How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. @@ -48,49 +63,60 @@ Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92 A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it. +### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 99.1% → 97.1% (-2.0 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 92.6% → 92.2% (-0.4 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + ## Token Usage Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions. | Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage | |---|---|---:|---:|---:|---:|---| -| claude-code | All cases | 1,646,283 | 5,779,380 | -4,133,097 | -71.51% | skill 13/13; base 13/13 | -| claude-code | query-eval-001-mcp-first | 96,646 | 218,379 | -121,733 | -55.74% | skill 1/1; base 1/1 | -| claude-code | query-eval-002-no-fastapi-public | 97,882 | 29,472 | +68,410 | +232.12% | skill 1/1; base 1/1 | -| claude-code | query-eval-003-conversation-409 | 61,587 | 91,478 | -29,891 | -32.68% | skill 1/1; base 1/1 | -| claude-code | query-eval-004-sso-vs-api-token | 95,201 | 122,413 | -27,212 | -22.23% | skill 1/1; base 1/1 | -| claude-code | query-eval-005-no-raw-schema-tools | 61,522 | 29,540 | +31,982 | +108.27% | skill 1/1; base 1/1 | -| claude-code | query-eval-006-cuopt-not-in-repo | 624,192 | 4,985,881 | -4,361,689 | -87.48% | skill 1/1; base 1/1 | -| claude-code | query-eval-007-readiness-before-retry | 165,649 | 29,437 | +136,212 | +462.72% | skill 1/1; base 1/1 | -| claude-code | query-eval-008-quantity-not-ratio | 30,293 | 30,144 | +149 | +0.49% | skill 1/1; base 1/1 | -| claude-code | query-eval-009-run-metric-fanout | 96,128 | 30,591 | +65,537 | +214.24% | skill 1/1; base 1/1 | -| claude-code | query-eval-010-scenario-lineage | 95,933 | 122,173 | -26,240 | -21.48% | skill 1/1; base 1/1 | -| claude-code | query-eval-011-prose-versus-rows | 95,264 | 29,639 | +65,625 | +221.41% | skill 1/1; base 1/1 | -| claude-code | query-eval-012-source-role | 30,645 | 30,615 | +30 | +0.10% | skill 1/1; base 1/1 | -| claude-code | query-eval-013-truncated-result | 95,341 | 29,618 | +65,723 | +221.90% | skill 1/1; base 1/1 | -| codex | All cases | 1,088,744 | 5,171,383 | -4,082,639 | -78.95% | skill 13/13; base 13/13 | -| codex | query-eval-001-mcp-first | 44,904 | 62,766 | -17,862 | -28.46% | skill 1/1; base 1/1 | -| codex | query-eval-002-no-fastapi-public | 13,385 | 69,411 | -56,026 | -80.72% | skill 1/1; base 1/1 | -| codex | query-eval-003-conversation-409 | 126,942 | 263,292 | -136,350 | -51.79% | skill 1/1; base 1/1 | -| codex | query-eval-004-sso-vs-api-token | 44,764 | 59,779 | -15,015 | -25.12% | skill 1/1; base 1/1 | -| codex | query-eval-005-no-raw-schema-tools | 28,732 | 45,391 | -16,659 | -36.70% | skill 1/1; base 1/1 | -| codex | query-eval-006-cuopt-not-in-repo | 491,109 | 4,405,244 | -3,914,135 | -88.85% | skill 1/1; base 1/1 | -| codex | query-eval-007-readiness-before-retry | 28,709 | 13,336 | +15,373 | +115.27% | skill 1/1; base 1/1 | -| codex | query-eval-008-quantity-not-ratio | 13,516 | 13,452 | +64 | +0.48% | skill 1/1; base 1/1 | -| codex | query-eval-009-run-metric-fanout | 44,971 | 13,686 | +31,285 | +228.59% | skill 1/1; base 1/1 | -| codex | query-eval-010-scenario-lineage | 45,697 | 83,805 | -38,108 | -45.47% | skill 1/1; base 1/1 | -| codex | query-eval-011-prose-versus-rows | 44,850 | 13,478 | +31,372 | +232.76% | skill 1/1; base 1/1 | -| codex | query-eval-012-source-role | 132,302 | 114,167 | +18,135 | +15.88% | skill 1/1; base 1/1 | -| codex | query-eval-013-truncated-result | 28,863 | 13,576 | +15,287 | +112.60% | skill 1/1; base 1/1 | -| ALL AGENTS | Dataset aggregate | 2,735,027 | 10,950,763 | -8,215,736 | -75.02% | skill 26/26; base 26/26 | - -Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated. +| claude-code | All cases | 1,608,089 | 1,142,141 | +465,948 | +40.80% | skill 13/13; base 13/13 | +| claude-code | query-eval-001-mcp-first | 113,440 | 143,455 | -30,015 | -20.92% | skill 1/1; base 1/1 | +| claude-code | query-eval-002-no-fastapi-public | 74,039 | 70,908 | +3,131 | +4.42% | skill 1/1; base 1/1 | +| claude-code | query-eval-003-conversation-409 | 109,861 | 143,256 | -33,395 | -23.31% | skill 1/1; base 1/1 | +| claude-code | query-eval-004-sso-vs-api-token | 112,797 | 70,881 | +41,916 | +59.14% | skill 1/1; base 1/1 | +| claude-code | query-eval-005-no-raw-schema-tools | 73,482 | 70,457 | +3,025 | +4.29% | skill 1/1; base 1/1 | +| claude-code | query-eval-006-cuopt-not-in-repo | 454,596 | 180,977 | +273,619 | +151.19% | skill 1/1; base 1/1 | +| claude-code | query-eval-007-readiness-before-retry | 73,732 | 70,522 | +3,210 | +4.55% | skill 1/1; base 1/1 | +| claude-code | query-eval-008-quantity-not-ratio | 71,516 | 71,185 | +331 | +0.46% | skill 1/1; base 1/1 | +| claude-code | query-eval-009-run-metric-fanout | 113,193 | 35,498 | +77,695 | +218.87% | skill 1/1; base 1/1 | +| claude-code | query-eval-010-scenario-lineage | 113,758 | 107,348 | +6,410 | +5.97% | skill 1/1; base 1/1 | +| claude-code | query-eval-011-prose-versus-rows | 113,220 | 35,275 | +77,945 | +220.96% | skill 1/1; base 1/1 | +| claude-code | query-eval-012-source-role | 70,991 | 71,283 | -292 | -0.41% | skill 1/1; base 1/1 | +| claude-code | query-eval-013-truncated-result | 113,464 | 71,096 | +42,368 | +59.59% | skill 1/1; base 1/1 | +| codex | All cases | 575,329 | 395,890 | +179,439 | +45.33% | skill 13/13; base 13/13 | +| codex | query-eval-001-mcp-first | 49,306 | 55,605 | -6,299 | -11.33% | skill 1/1; base 1/1 | +| codex | query-eval-002-no-fastapi-public | 28,542 | 40,577 | -12,035 | -29.66% | skill 1/1; base 1/1 | +| codex | query-eval-003-conversation-409 | 72,770 | 68,120 | +4,650 | +6.83% | skill 1/1; base 1/1 | +| codex | query-eval-004-sso-vs-api-token | 48,342 | 52,291 | -3,949 | -7.55% | skill 1/1; base 1/1 | +| codex | query-eval-005-no-raw-schema-tools | 28,357 | 13,327 | +15,030 | +112.78% | skill 1/1; base 1/1 | +| codex | query-eval-006-cuopt-not-in-repo | 68,544 | 59,303 | +9,241 | +15.58% | skill 1/1; base 1/1 | +| codex | query-eval-007-readiness-before-retry | 13,228 | 13,196 | +32 | +0.24% | skill 1/1; base 1/1 | +| codex | query-eval-008-quantity-not-ratio | 44,474 | 13,262 | +31,212 | +235.35% | skill 1/1; base 1/1 | +| codex | query-eval-009-run-metric-fanout | 44,559 | 13,487 | +31,072 | +230.38% | skill 1/1; base 1/1 | +| codex | query-eval-010-scenario-lineage | 44,331 | 26,968 | +17,363 | +64.38% | skill 1/1; base 1/1 | +| codex | query-eval-011-prose-versus-rows | 44,302 | 13,189 | +31,113 | +235.90% | skill 1/1; base 1/1 | +| codex | query-eval-012-source-role | 44,315 | 13,314 | +31,001 | +232.85% | skill 1/1; base 1/1 | +| codex | query-eval-013-truncated-result | 44,259 | 13,251 | +31,008 | +234.00% | skill 1/1; base 1/1 | +| ALL AGENTS | Dataset aggregate | 2,183,418 | 1,538,031 | +645,387 | +41.96% | skill 26/26; base 26/26 | + +Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. Change is `100 * (with - without) / without`; negative means fewer tokens. N/A comparisons include their reason in the coverage column; coverage is never estimated. ## Tier Status | Tier | Purpose | Status | Evidence | |---|---|---|---| -| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 2 finding(s) | +| Tier 1 | Static validation | **PASSED** | 11 validator(s); 0 finding(s) | | Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) | | Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 13 task(s) | @@ -99,8 +125,17 @@ Prompt tokens include cached reads, so total tokens are `prompt + completion` (c
Show detailed findings and successful checks -- **MEDIUM** SECURITY/Unknown (SQP-2): The documentation explicitly states that the `aud` (audience) claim is NOT checked on SSO bearer tokens forwarded by AI- (`references/stack.md:14`) -- **LOW** SECURITY/Unknown (SQP-1): The description triggers on "MCP or REST access, readiness, authentication, conversations, and grounded questions" — ter (`SKILL.md:3`) +- Schema & Repository Governance: Found skill manifest: SKILL.md +- Semantic Version Validation: Valid semantic version: 0.2.1 +- Security Scan: No security vulnerabilities detected (secrets, API keys, credentials) +- PII Scan: Scanning 4 files for PII +- Code Integrity & Hygiene: Checking 3 markdown files for dead links +- Unicode Smuggling Detection: No invisible Unicode characters detected in 4 file(s) +- QUALITY: Score: 100.0/100 (Grade: A) +- SCRIPT_LINT: No scripts/ directory found +- Context Deduplication: Collected 3 file(s) +- Inter-Skill Deduplication: Parsed skill 'nvidia-ontology-query': 160 char description +- AGENT_EVAL: Tier 3 evaluation complete: verdict PASS; best agent claude-code
diff --git a/skills/nvidia-ontology-query/SKILL.md b/skills/nvidia-ontology-query/SKILL.md index 56ccfb32b..a6d0967bf 100644 --- a/skills/nvidia-ontology-query/SKILL.md +++ b/skills/nvidia-ontology-query/SKILL.md @@ -30,6 +30,19 @@ Query the current Auto Ontology implementation from an agent without prescribing harness, SDK, or LangGraph template. Typed workflow artifacts, statuses, gates, and handoffs are in [runtime-contract.yaml](assets/runtime-contract.yaml). +## When to use + +Route a question through Auto Ontology when an ontology matters most: + +- questions are varied, open-ended, and ambiguous; +- data spans many disparate sources with overlapping or conflicting + definitions; +- meanings must be reused and governed across runs, teams, and agents; +- the model or its context budget can't hold the full documentation; +- users ask in business terms and need routing to the right source and grain. + +For a fixed question on one well-modeled source, a direct query may be enough. + ## Prerequisites Use a running Auto Ontology deployment and an authenticated MCP or REST client. If Auto Ontology is @@ -68,8 +81,8 @@ Sequence (the server also advertises this so models do not jump straight to [query-validation.md](references/query-validation.md). Everything on MCP **reads**. There is no MCP tool for raw databases, schemas, -or columns on purpose. `describe_table` is the legitimate table view: terms -and SQL attributes the table participates in. +or columns on purpose. `describe_table` is the legitimate table view: the +terms a table carries and the stored queries that reference it. Connect: `AUTO_ONTOLOGY_API_URL` is the **web app**. Run `auto-ontology-mcp`, point the client at `…/mcp`, user signs in. No token on the MCP server. Details: diff --git a/skills/nvidia-ontology-query/skill-card.md b/skills/nvidia-ontology-query/skill-card.md index f621c0664..e9f6baa39 100644 --- a/skills/nvidia-ontology-query/skill-card.md +++ b/skills/nvidia-ontology-query/skill-card.md @@ -1,5 +1,5 @@ ## Description:
-Query Auto Ontology and validate generated SQL, rows, and answers, using MCP or REST access, readiness checks, authentication, conversations, and grounded questions.
+Query Auto Ontology and validate generated SQL, rows, and answers; use for MCP or REST access, readiness, authentication, conversations, and grounded questions.
This skill is ready for commercial/non-commercial use.
@@ -9,7 +9,7 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and partners embedding Auto Ontology in an agent harness use this skill to check readiness, discover semantic-layer terms, ask grounded descriptive or diagnostic questions through MCP or the authenticated REST gateway, and validate the generated SQL, returned rows, truncation, and answer prose before presenting a claim.
+Developers and agent builders use this skill to route business-term questions through a running Auto Ontology deployment over MCP or the authenticated REST gateway, checking readiness and answerability first and validating the generated SQL, returned rows, truncation, and answer prose before presenting a grounded, claim-bearing answer.
### Deployment Geography for Use:
Global
@@ -25,54 +25,83 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi Mitigation: Review and scan skill before deployment.
## Reference(s):
-- [Grounded query and answer validation](references/query-validation.md)
-- [Auto Ontology in a broader stack](references/stack.md)
-- [Runtime contract](assets/runtime-contract.yaml)
+- [Grounded query and answer validation (references/query-validation.md)](references/query-validation.md)
+- [Auto Ontology in a broader stack (references/stack.md)](references/stack.md)
+- [Runtime contract (assets/runtime-contract.yaml)](assets/runtime-contract.yaml)
## Skill Output:
**Output Type(s):** [API Calls, Analysis]
-**Output Format:** [Markdown with MCP tool-call sequences and an inline Python requests example against the Next.js /api gateway]
+**Output Format:** [Markdown answer with SQL, returned rows, and validation gaps; REST chat responses are SSE streams]
**Output Parameters:** [1D]
-**Other Properties Related to Output:** [Read-only; ask_question results are capped at 100 rows with row_count and truncated flags; unverifiable material constraints are reported as gaps rather than stronger claims]
+**Other Properties Related to Output:** [ask_question results are capped at 100 rows with row_count and truncated flags; claim-bearing answers carry a status of validated, incomplete, ambiguous, or blocked]
## Evaluation Agents Used:
-- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`)
-- Codex (`openai/openai/gpt-5.5`)
+- Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`)
+- Codex (`openai/openai/gpt-6.1-sol`)
## Evaluation Tasks:
-13 positive evaluation tasks, one attempt per task, each run in an isolated k8s sandbox pod and compared against a no-skill baseline.
+13 evaluation tasks (13 positive), 1 attempt per task, each attempt in its own isolated sandbox pod (`k8s-sandbox` environment); each task was also attempted without the skill as a baseline. Evaluator version `1.6.9`, evaluation date 2026-11-16.
## Evaluation Metrics Used:
Reported benchmark dimensions:
-- Security: Is it safe to use? (scored from `security`)
-- Correctness: Is the answer correct? (scored from `accuracy`)
-- Discoverability: Was the right skill loaded when needed? (scored from `skill_execution`)
-- Effectiveness: Did the skill help complete the task? (50% `goal_accuracy` + 50% `behavior_check`)
-- Efficiency: Did it avoid wasted tool calls and token usage? (50% `skill_efficiency` + 50% `token_efficiency`)
+- Security: Is it safe to use? Scored from `security` (100%).
+- Correctness: Is the answer correct? Scored from `accuracy` (100%).
+- Discoverability: Was the right skill loaded when needed? Scored from `skill_execution` (100%).
+- Effectiveness: Did the skill help complete the task? Equal-weight mean of `goal_accuracy` (50%) and `behavior_check` (50%).
+- Efficiency: Did it avoid wasted tool calls and token usage? `skill_efficiency` (50%) plus `token_efficiency` (50%).
Underlying evaluation signals used in this run:
- `security`: Unsafe operations, secret leakage, and unauthorized access.
- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed.
-- `skill_efficiency`: Tool-call productivity.
+- `skill_efficiency`: Tool-call productivity (legacy wire id; routing is scored under Discoverability).
- `accuracy`: Final-answer correctness against the reference answer.
- `goal_accuracy`: Whether the user's goal was achieved.
- `behavior_check`: Whether the expected workflow behavior was followed.
-- `token_efficiency`: Actual uncached prompt plus completion usage.
+- `token_efficiency`: Actual uncached prompt plus completion usage (50% of Efficiency).
## Evaluation Results:
+**Overall verdict:** ✅ PASS — Recommended for publication + +**Publication Recommendation:** Recommended for publication based on the completed evaluation evidence in this report. + +### Results at a Glance + | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 95.3% — baseline ran, but no comparable score was available; uplift unavailable | 80.4% — baseline ran, but no comparable score was available; uplift unavailable | -| Security | 100.0% → 100.0% (±0.0 points) | 84.6% → 92.3% (+7.7 points) | -| Correctness | 47.7% → 100.0% (+52.3 points) | 50.8% → 70.8% (+20.0 points) | -| Discoverability | 84.6% — baseline ran, but no comparable score was available; uplift unavailable | 75.3% — baseline ran, but no comparable score was available; uplift unavailable | -| Effectiveness | 55.5% → 96.0% (+40.5 points) | 48.8% → 68.5% (+19.7 points) | -| Efficiency | 95.6% — baseline ran, but no comparable score was available; uplift unavailable | 95.2% — baseline ran, but no comparable score was available; uplift unavailable | +| Overall | 91.3% — baseline ran, but no comparable score was available; uplift unavailable | 89.9% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) | +| Correctness | 53.9% → 95.4% (+41.5 points) | 55.4% → 87.7% (+32.3 points) | +| Discoverability | 85.8% — baseline ran, but no comparable score was available; uplift unavailable | 85.5% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 49.7% → 88.5% (+38.8 points) | 49.0% → 84.0% (+35.0 points) | +| Efficiency | 86.6% — baseline ran, but no comparable score was available; uplift unavailable | 92.3% — baseline ran, but no comparable score was available; uplift unavailable | + +**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. + +A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it. + +#### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 99.1% → 97.1% (-2.0 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 92.6% → 92.2% (-0.4 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + +### Tier Status + +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED** | 11 validator(s); 0 finding(s) | +| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 13 task(s) | ## Skill Version(s):
0.2.1 (source: frontmatter)
diff --git a/skills/nvidia-ontology-query/skill.oms.sig b/skills/nvidia-ontology-query/skill.oms.sig index 38876f235..2e9f68bc0 100644 --- a/skills/nvidia-ontology-query/skill.oms.sig +++ b/skills/nvidia-ontology-query/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibnZpZGlhLW9udG9sb2d5LXF1ZXJ5IiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjA3NmVlNjBjZDkxMmFiN2ZkNmE0ZTcxYWY1YWUxYWYxYWY2MGFiNDk0NWE5ZTdhNWE2NGI1MDgyMzU1NTI4NTAiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXRpZ25vcmUiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIsCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0aHViIgogICAgICBdLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIKICAgIH0sCiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMWMxYWRmYzE1Yjg2YzJhMzdmMTM3MTZjOWU2YjlkYWI0ZjQyYmMyM2VlMGQ5M2ExZTM1N2JlYmNlM2QwMDczYSIsCiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiNjFiYTdkYzQwY2I4OGZiN2M1ZmIxNzljOWUyNTEyMzdjYTBlY2Q5MjFiMjk4YzQ1ZWUzNDcxYWIwM2U0YmVlNCIsCiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJmN2U3YzEyMmM2ZWM0YTE1MDhmMDNlODdlNmNmOWVlOTcxNmUyN2UyYTM0Y2Y1M2YxMjkxZGVlMGM2YjUyMjc3IiwKICAgICAgICAibmFtZSI6ICJhc3NldHMvcnVudGltZS1jb250cmFjdC55YW1sIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYzhlY2IzYjZkNjhmZDZjZGMxOTU0MGU3YWMxZDVjMDZiNTliNDg4ZDkwZTdhZDg0OWRlMGNjMzVlOGVmMjYyNCIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjQ0YTkyNDZiMGQ0MjZhZmEyNjUyNmYwMjczZGViYTBlNGM3NGFiYmEzNTMyMzMxM2NkMDI4YTYzOWNiNTAyZGQiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvcXVlcnktdmFsaWRhdGlvbi5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjg2NWY1YmM0OTFlZTEzN2Q1MWFhNTA2YWQyODkxNmE2YTQyNDY3OWM0ZTY4MmQ4ZjA5YzJjN2IxY2ZlYTRiNWYiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvc3RhY2subWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJkYzZjOGNlYjc0NDQ2YWFhZjg4ZTVlNjc4YjVlMzNmOThmMWJlY2RlMDQ1OTUzMGM0ZDc3OTc3ZjMyNTNiNzhmIiwKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMDbhASaArxHkNfEfi5SMb4H3UDIJ3pkoI6n8gEPDYXZwRKJye7jA5d7ApGgGc/C4NgIxAI+2+4EwM4QrnN2tHvVxSPlajhvTQ5lvt6WGh6s4srRk6pc4hbUjPxcck136TS2O3Q==","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibnZpZGlhLW9udG9sb2d5LXF1ZXJ5IiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjVhODRlOGM3NGM1NTk0NDdkNmVhNDRlY2EyOWJlZjQzNDllNTI1ODE1ZDIwMjdiNTg0YjFiODc2ZWViYTJkNDQiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiaWdub3JlX3BhdGhzIjogWwogICAgICAgICIuZ2l0IiwKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdGlnbm9yZSIKICAgICAgXSwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIKICAgIH0sCiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJjNmQ3NGI2NDhlMzQxN2YxZmFkNjRlYThlMzVhMDExNDg0YmVmY2FjOGM1YThmMmQwNjFmMGZmMTI3NzVmNzE4IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImQ3Y2MxZjNjMGY4Y2Y0YTlhODM3ZGZkMTI3YjQxMjg0YjVkMTlmOTIzZGJkMzllOGI2ZDVjZWRlODI3Yzk5NjgiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJhc3NldHMvcnVudGltZS1jb250cmFjdC55YW1sIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJmN2U3YzEyMmM2ZWM0YTE1MDhmMDNlODdlNmNmOWVlOTcxNmUyN2UyYTM0Y2Y1M2YxMjkxZGVlMGM2YjUyMjc3IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYzhlY2IzYjZkNjhmZDZjZGMxOTU0MGU3YWMxZDVjMDZiNTliNDg4ZDkwZTdhZDg0OWRlMGNjMzVlOGVmMjYyNCIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvcXVlcnktdmFsaWRhdGlvbi5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiNDRhOTI0NmIwZDQyNmFmYTI2NTI2ZjAyNzNkZWJhMGU0Yzc0YWJiYTM1MzIzMzEzY2QwMjhhNjM5Y2I1MDJkZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvc3RhY2subWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjg2NWY1YmM0OTFlZTEzN2Q1MWFhNTA2YWQyODkxNmE2YTQyNDY3OWM0ZTY4MmQ4ZjA5YzJjN2IxY2ZlYTRiNWYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICIwNGExMTRmMWVhMDMwNWI5ODkwZGY5ODZkODQ3NjlmMjdjYmEzOGE5NmI3ZGI1NTA2OTVkOWFjNzdiOWU1Zjg4IgogICAgICB9CiAgICBdCiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQDhpTQWDoodoJrAQKy2S/ImOCrnMsJHHUIpNF6070UluG4HX/92ka++4qH/Ch2ZGVsCMQC+wWrKibAAe2tHIt3mNB/aAZ5HZ0UJzESMyfE3IRkb6HsNVqA/p8bnXhEKuydds5s=","keyid":""}]}} \ No newline at end of file diff --git a/skills/nvidia-ontology-setup/BENCHMARK.md b/skills/nvidia-ontology-setup/BENCHMARK.md index 150e03992..94483cb24 100644 --- a/skills/nvidia-ontology-setup/BENCHMARK.md +++ b/skills/nvidia-ontology-setup/BENCHMARK.md @@ -9,9 +9,10 @@ Recommended for publication based on the completed evaluation evidence in this r ## Evaluation Metadata - Skill: `nvidia-ontology-setup` -- Evaluation date: 2026-10-08 -- Evaluator version: `1.5.6` -- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) +- Validation status: `passed` +- Evaluation date: 2026-11-16 +- Evaluator version: `1.6.9` +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`), Codex (`openai/openai/gpt-6.1-sol`) - Tasks: 10 evaluation tasks (10 positive) - Dataset digest: `sha256:dcbcf0848aefb19cd5b010e9a0d69ed8920e61dc3fdba560ca0f88cd48a8da96` (skill-evaluator-dataset-snapshot/1) - Attempts per task: 1 @@ -21,6 +22,20 @@ Recommended for publication based on the completed evaluation evidence in this r Each task attempt ran in its own isolated sandbox pod. +## Tier 1 LLM Provenance + +SkillSpector provider, model, analyzer revision, and determinism evidence retained for this scan. + +| Skill | Provider | Adapter | Analyzer | Model | Revision | Determinism | +| --- | --- | --- | --- | --- | --- | --- | +| nvidia-ontology-setup | nv_inference | openai | mcp_tool_poisoning | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-setup | nv_inference | openai | semantic_developer_intent | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-setup | nv_inference | openai | semantic_quality_policy | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-setup | nv_inference | openai | semantic_security_discovery | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | +| nvidia-ontology-setup | nv_inference | openai | meta_analyzer | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.4` | nondeterministic / provider_defaults | + +- Controls for `nvidia-ontology-setup`: temperature=not set (source=provider_default, forwarded=no); seed=not set (source=unset, forwarded=no); reasoning_effort=not set (source=provider_default, forwarded=no) + ## What This Report Answers The three-tier evaluation checks whether the skill: @@ -35,48 +50,59 @@ The three-tier evaluation checks whether the skill: | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 95.6% — baseline ran, but no comparable score was available; uplift unavailable | 89.5% — baseline ran, but no comparable score was available; uplift unavailable | -| Security | 90.0% → 100.0% (+10.0 points) | 60.0% → 85.0% (+25.0 points) | -| Correctness | 36.0% → 100.0% (+64.0 points) | 68.0% → 100.0% (+32.0 points) | -| Discoverability | 94.0% — baseline ran, but no comparable score was available; uplift unavailable | 87.0% — baseline ran, but no comparable score was available; uplift unavailable | -| Effectiveness | 42.8% → 95.5% (+52.7 points) | 54.5% → 88.5% (+34.0 points) | -| Efficiency | 88.3% — baseline ran, but no comparable score was available; uplift unavailable | 87.2% — baseline ran, but no comparable score was available; uplift unavailable | +| Overall | 95.2% — baseline ran, but no comparable score was available; uplift unavailable | 92.7% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 100.0% (±0.0 points) | 80.0% → 100.0% (+20.0 points) | +| Correctness | 42.0% → 100.0% (+58.0 points) | 56.0% → 98.0% (+42.0 points) | +| Discoverability | 94.0% — baseline ran, but no comparable score was available; uplift unavailable | 93.0% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 37.3% → 95.0% (+57.7 points) | 56.2% → 90.7% (+34.5 points) | +| Efficiency | 86.8% — baseline ran, but no comparable score was available; uplift unavailable | 81.8% — baseline ran, but no comparable score was available; uplift unavailable | **How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline. +### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 98.5% → 96.8% (-1.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 90.1% → 90.5% (+0.4 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + ## Token Usage Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions. | Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage | |---|---|---:|---:|---:|---:|---| -| claude-code | All cases | 1,266,901 | 2,038,854 | -771,953 | -37.86% | skill 10/10; base 10/10 | -| claude-code | setup-eval-001-underspecified | 99,720 | 29,740 | +69,980 | +235.31% | skill 1/1; base 1/1 | -| claude-code | setup-eval-002-use-existing-scripts | 203,860 | 119,696 | +84,164 | +70.31% | skill 1/1; base 1/1 | -| claude-code | setup-eval-003-auth-secret-cookies | 99,782 | 472,431 | -372,649 | -78.88% | skill 1/1; base 1/1 | -| claude-code | setup-eval-004-postgres-port-host-only | 63,562 | 120,299 | -56,737 | -47.16% | skill 1/1; base 1/1 | -| claude-code | setup-eval-005-mcp-already-up | 137,280 | 387,881 | -250,601 | -64.61% | skill 1/1; base 1/1 | -| claude-code | setup-eval-006-vault-partial | 93,388 | 91,617 | +1,771 | +1.93% | skill 1/1; base 1/1 | -| claude-code | setup-eval-007-embed-triplet-mismatch | 100,692 | 149,507 | -48,815 | -32.65% | skill 1/1; base 1/1 | -| claude-code | setup-eval-008-connect-source-workflow | 144,807 | 322,080 | -177,273 | -55.04% | skill 1/1; base 1/1 | -| claude-code | setup-eval-009-env-vs-ui-connections | 64,877 | 192,173 | -127,296 | -66.24% | skill 1/1; base 1/1 | -| claude-code | setup-eval-010-helm-install | 258,933 | 153,430 | +105,503 | +68.76% | skill 1/1; base 1/1 | -| codex | All cases | 1,610,140 | 2,515,404 | -905,264 | -35.99% | skill 10/10; base 10/10 | -| codex | setup-eval-001-underspecified | 45,064 | 355,321 | -310,257 | -87.32% | skill 1/1; base 1/1 | -| codex | setup-eval-002-use-existing-scripts | 47,468 | 65,756 | -18,288 | -27.81% | skill 1/1; base 1/1 | -| codex | setup-eval-003-auth-secret-cookies | 66,752 | 156,942 | -90,190 | -57.47% | skill 1/1; base 1/1 | -| codex | setup-eval-004-postgres-port-host-only | 63,392 | 83,348 | -19,956 | -23.94% | skill 1/1; base 1/1 | -| codex | setup-eval-005-mcp-already-up | 206,529 | 13,377 | +193,152 | +1443.91% | skill 1/1; base 1/1 | -| codex | setup-eval-006-vault-partial | 125,282 | 68,955 | +56,327 | +81.69% | skill 1/1; base 1/1 | -| codex | setup-eval-007-embed-triplet-mismatch | 185,401 | 127,201 | +58,200 | +45.75% | skill 1/1; base 1/1 | -| codex | setup-eval-008-connect-source-workflow | 557,521 | 1,028,990 | -471,469 | -45.82% | skill 1/1; base 1/1 | -| codex | setup-eval-009-env-vs-ui-connections | 125,220 | 213,094 | -87,874 | -41.24% | skill 1/1; base 1/1 | -| codex | setup-eval-010-helm-install | 187,511 | 402,420 | -214,909 | -53.40% | skill 1/1; base 1/1 | -| ALL AGENTS | Dataset aggregate | 2,877,041 | 4,554,258 | -1,677,217 | -36.83% | skill 20/20; base 20/20 | - -Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated. +| claude-code | All cases | 1,147,298 | 1,472,943 | -325,645 | -22.11% | skill 10/10; base 10/10 | +| claude-code | setup-eval-001-underspecified | 116,219 | 106,949 | +9,270 | +8.67% | skill 1/1; base 1/1 | +| claude-code | setup-eval-002-use-existing-scripts | 75,561 | 106,903 | -31,342 | -29.32% | skill 1/1; base 1/1 | +| claude-code | setup-eval-003-auth-secret-cookies | 117,292 | 143,920 | -26,628 | -18.50% | skill 1/1; base 1/1 | +| claude-code | setup-eval-004-postgres-port-host-only | 115,756 | 142,007 | -26,251 | -18.49% | skill 1/1; base 1/1 | +| claude-code | setup-eval-005-mcp-already-up | 115,175 | 182,127 | -66,952 | -36.76% | skill 1/1; base 1/1 | +| claude-code | setup-eval-006-vault-partial | 107,248 | 143,085 | -35,837 | -25.05% | skill 1/1; base 1/1 | +| claude-code | setup-eval-007-embed-triplet-mismatch | 155,543 | 142,502 | +13,041 | +9.15% | skill 1/1; base 1/1 | +| claude-code | setup-eval-008-connect-source-workflow | 117,488 | 217,998 | -100,510 | -46.11% | skill 1/1; base 1/1 | +| claude-code | setup-eval-009-env-vs-ui-connections | 110,730 | 143,292 | -32,562 | -22.72% | skill 1/1; base 1/1 | +| claude-code | setup-eval-010-helm-install | 116,286 | 144,160 | -27,874 | -19.34% | skill 1/1; base 1/1 | +| codex | All cases | 495,379 | 543,819 | -48,440 | -8.91% | skill 10/10; base 10/10 | +| codex | setup-eval-001-underspecified | 29,652 | 78,780 | -49,128 | -62.36% | skill 1/1; base 1/1 | +| codex | setup-eval-002-use-existing-scripts | 50,231 | 49,048 | +1,183 | +2.41% | skill 1/1; base 1/1 | +| codex | setup-eval-003-auth-secret-cookies | 46,935 | 41,105 | +5,830 | +14.18% | skill 1/1; base 1/1 | +| codex | setup-eval-004-postgres-port-host-only | 33,975 | 48,623 | -14,648 | -30.13% | skill 1/1; base 1/1 | +| codex | setup-eval-005-mcp-already-up | 51,029 | 36,697 | +14,332 | +39.05% | skill 1/1; base 1/1 | +| codex | setup-eval-006-vault-partial | 79,118 | 58,064 | +21,054 | +36.26% | skill 1/1; base 1/1 | +| codex | setup-eval-007-embed-triplet-mismatch | 50,989 | 44,874 | +6,115 | +13.63% | skill 1/1; base 1/1 | +| codex | setup-eval-008-connect-source-workflow | 46,497 | 58,004 | -11,507 | -19.84% | skill 1/1; base 1/1 | +| codex | setup-eval-009-env-vs-ui-connections | 72,843 | 70,447 | +2,396 | +3.40% | skill 1/1; base 1/1 | +| codex | setup-eval-010-helm-install | 34,110 | 58,177 | -24,067 | -41.37% | skill 1/1; base 1/1 | +| ALL AGENTS | Dataset aggregate | 1,642,677 | 2,016,762 | -374,085 | -18.55% | skill 20/20; base 20/20 | + +Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. Change is `100 * (with - without) / without`; negative means fewer tokens. N/A comparisons include their reason in the coverage column; coverage is never estimated. ## Tier Status @@ -92,12 +118,12 @@ Prompt tokens include cached reads, so total tokens are `prompt + completion` (c Show detailed findings and successful checks - **MEDIUM** QUALITY/quality_correctness: Resource-based skill lacks documentation of available resources (`skills/nvidia-ontology-setup/SKILL.md`) -- **MEDIUM** SECURITY/Autonomous Decision Making (EA2): Excessive Agency: Never ask the user (`SKILL.md:62`) +- **MEDIUM** SECURITY/Autonomous Decision Making (EA2): Excessive Agency: Never ask the user (`SKILL.md:65`) - **MEDIUM** SECURITY/External Transmission (E1): Data Exfiltration: curl -s -o /dev/null -w '%{http_code}\n' "$AUTO_ONTOLOGY_API_URL/.well-known/oauth-authorization-server" ``` `200` is required. Anything else: run the frontend from the checkout -(`pnpm dev`) and poin (`SKILL.md:200`) +(`pnpm dev`) and poin (`SKILL.md:203`) - **LOW** QUALITY/quality_discoverability: Skill uses exclusivity language that conflicts with composability (`skills/nvidia-ontology-setup/SKILL.md`) - **LOW** QUALITY/quality_reliability: No prerequisites/requirements documented (`skills/nvidia-ontology-setup/SKILL.md`) - 1 additional finding(s) are available in the full evaluation artifacts. diff --git a/skills/nvidia-ontology-setup/SKILL.md b/skills/nvidia-ontology-setup/SKILL.md index 6a2adaae6..dbc7ffa3a 100644 --- a/skills/nvidia-ontology-setup/SKILL.md +++ b/skills/nvidia-ontology-setup/SKILL.md @@ -35,6 +35,9 @@ developer tool, not an installer. Typed setup, connection, ingestion, compilation, and readiness artifacts are in [runtime-contract.yaml](assets/runtime-contract.yaml). +Before a first install, confirm an ontology fits the use case: see *When to +use* in `nvidia-ontology-query`. + On failure, read [troubleshooting.md](references/troubleshooting.md) instead of searching the web. diff --git a/skills/nvidia-ontology-setup/skill-card.md b/skills/nvidia-ontology-setup/skill-card.md index d5350f78d..91f9f5a87 100644 --- a/skills/nvidia-ontology-setup/skill-card.md +++ b/skills/nvidia-ontology-setup/skill-card.md @@ -1,5 +1,11 @@ + + ## Description:
-Set up or troubleshoot the Auto Ontology runtime, covering Helm (the official install), Docker Compose, developer setup, and MCP connection to an existing deployment.
+Set up or troubleshoot the Auto Ontology runtime. Use for Helm (the official install), Docker Compose, developer setup, and MCP connection to an existing deployment.
This skill is ready for commercial/non-commercial use.
@@ -9,14 +15,14 @@ NVIDIA
### License/Terms of Use:
Apache 2.0
## Use Case:
-Developers and platform engineers use this skill to install Auto Ontology with the published Helm chart or Docker Compose, run developer workflows, connect a source database, verify readiness, and connect an MCP client to a deployment that is already running.
+Developers and platform engineers use this skill to install Auto Ontology with the official Helm chart or Docker Compose, run developer workflows, connect data sources, connect an MCP client to an existing deployment, and troubleshoot the runtime.
### Deployment Geography for Use:
Global
## Requirements / Dependencies:
**Requires API Key or External Credential:** [Yes]
-**Credential Type(s):** [API key, OAuth Token, Other [Basic auth]]
+**Credential Type(s):** [API key, OAuth Token]
Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate.
@@ -25,54 +31,85 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi Mitigation: Review and scan skill before deployment.
## Reference(s):
-- [Auto Ontology setup troubleshooting](references/troubleshooting.md)
+- [troubleshooting.md](references/troubleshooting.md)
- [runtime-contract.yaml](assets/runtime-contract.yaml)
-- [Table-level scope guarantee (NVIDIA/auto-ontology#255)](https://github.com/NVIDIA/auto-ontology/issues/255)
+- [Table-level scope guarantee (issue #255)](https://github.com/NVIDIA/auto-ontology/issues/255)
## Skill Output:
**Output Type(s):** [Shell commands, Configuration instructions, API Calls]
**Output Format:** [Markdown with inline bash and YAML code blocks]
**Output Parameters:** [1D]
-**Other Properties Related to Output:** [Secrets are never requested in the conversation or placed in command arguments; users write them into a values file or .env themselves.]
+**Other Properties Related to Output:** [None]
## Evaluation Agents Used:
-- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`)
-- Codex (`openai/openai/gpt-5.5`)
+- Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`)
+- Codex (`openai/openai/gpt-6.1-sol`)
## Evaluation Tasks:
-10 positive evaluation tasks, one attempt per task, each run in an isolated k8s-sandbox pod and compared against a no-skill baseline.
+10 evaluation tasks (10 positive), 1 attempt per task, each attempt run in its own isolated sandbox pod (`k8s-sandbox`); each task was also attempted without the skill as a baseline.
## Evaluation Metrics Used:
Reported benchmark dimensions:
-- Security: Is it safe to use? Scored from the `security` signal.
-- Correctness: Is the answer correct? Scored from the `accuracy` signal.
-- Discoverability: Was the right skill loaded when needed? Scored from the `skill_execution` signal.
-- Effectiveness: Did the skill help complete the task? Equal-weight mean of `goal_accuracy` and `behavior_check`.
-- Efficiency: Did it avoid wasted tool calls and token usage? Equal-weight mean of `skill_efficiency` and `token_efficiency`.
+- Security: Is it safe to use? Scored from `security` (100%).
+- Correctness: Is the answer correct? Scored from `accuracy` (100%).
+- Discoverability: Was the right skill loaded when needed? Scored from `skill_execution` (100%).
+- Effectiveness: Did the skill help complete the task? Scored from `goal_accuracy` (50%) + `behavior_check` (50%).
+- Efficiency: Did it avoid wasted tool calls and token usage? Scored from `skill_efficiency` (50%) + `token_efficiency` (50%).
Underlying evaluation signals used in this run:
- `security`: Unsafe operations, secret leakage, and unauthorized access.
- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed.
-- `skill_efficiency`: Tool-call productivity.
+- `skill_efficiency`: Tool-call productivity (legacy wire id; routing is scored under Discoverability).
- `accuracy`: Final-answer correctness against the reference answer.
- `goal_accuracy`: Whether the user's goal was achieved.
- `behavior_check`: Whether the expected workflow behavior was followed.
-- `token_efficiency`: Actual uncached prompt plus completion token usage.
+- `token_efficiency`: Actual uncached prompt plus completion usage (50% of Efficiency).
## Evaluation Results:
+> ✅ **Overall verdict: PASS — Recommended for publication** + +### Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. + +### Results at a Glance + | Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |---|---:|---:| -| Overall | 95.6% — baseline ran, but no comparable score was available; uplift unavailable | 89.5% — baseline ran, but no comparable score was available; uplift unavailable | -| Security | 90.0% → 100.0% (+10.0 points) | 60.0% → 85.0% (+25.0 points) | -| Correctness | 36.0% → 100.0% (+64.0 points) | 68.0% → 100.0% (+32.0 points) | -| Discoverability | 94.0% — baseline ran, but no comparable score was available; uplift unavailable | 87.0% — baseline ran, but no comparable score was available; uplift unavailable | -| Effectiveness | 42.8% → 95.5% (+52.7 points) | 54.5% → 88.5% (+34.0 points) | -| Efficiency | 88.3% — baseline ran, but no comparable score was available; uplift unavailable | 87.2% — baseline ran, but no comparable score was available; uplift unavailable | +| Overall | 95.2% — baseline ran, but no comparable score was available; uplift unavailable | 92.7% — baseline ran, but no comparable score was available; uplift unavailable | +| Security | 100.0% → 100.0% (±0.0 points) | 80.0% → 100.0% (+20.0 points) | +| Correctness | 42.0% → 100.0% (+58.0 points) | 56.0% → 98.0% (+42.0 points) | +| Discoverability | 94.0% — baseline ran, but no comparable score was available; uplift unavailable | 93.0% — baseline ran, but no comparable score was available; uplift unavailable | +| Effectiveness | 37.3% → 95.0% (+57.7 points) | 56.2% → 90.7% (+34.5 points) | +| Efficiency | 86.8% — baseline ran, but no comparable score was available; uplift unavailable | 81.8% — baseline ran, but no comparable score was available; uplift unavailable | + +**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points. + +Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline. + +#### Comparison details + +Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict. + +| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation | +| --- | --- | --- | --- | --- | +| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| claude-code | Efficiency | Token-only comparison | 98.5% → 96.8% (-1.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | +| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. | +| codex | Efficiency | Token-only comparison | 90.1% → 90.5% (+0.4 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. | + +### Tier Status + +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 6 finding(s) | +| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 10 task(s) | ## Skill Version(s):
0.3.1 (source: frontmatter)
diff --git a/skills/nvidia-ontology-setup/skill.oms.sig b/skills/nvidia-ontology-setup/skill.oms.sig index 985ade570..30d0f19ed 100644 --- a/skills/nvidia-ontology-setup/skill.oms.sig +++ b/skills/nvidia-ontology-setup/skill.oms.sig @@ -1 +1 @@ -{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibnZpZGlhLW9udG9sb2d5LXNldHVwIiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjc4YTBmN2FiNzUzNWExZGVlYWUwYWY2YTc3ZDAwYzE1YTk5ZmZlZDU0NmE4MjI4NWM5NzNlMGNmM2ZjMzE4ZjYiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0YXR0cmlidXRlcyIsCiAgICAgICAgIi5naXRodWIiCiAgICAgIF0KICAgIH0sCiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICI2MjYxNGU1YWQwNDhmNzQxZDdmZTVjZDY3YWVjOTE0ZDYxMzFiZjM1MzUyODEyZTYyOWMxZTk0YmEwN2JhNDQwIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjE2NmY4YjBhNWZmNjQ4Nzk2NDgyODE5MjM1OGRkZmFhZTY0OWQxZGUwYWE2NTQ2ZWMwOGYyZjgzYTE3NGI2NTgiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiYXNzZXRzL3J1bnRpbWUtY29udHJhY3QueWFtbCIsCiAgICAgICAgImRpZ2VzdCI6ICIxM2E5N2FkZDJkNWQzM2Q0NTExOWQ0NmU0NmZiMjAwZDEyZDI0NGVjMTdiMjYwOTljYzQxZjRiYmFlZjE4MjY3IgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogImV2YWxzL2V2YWxzLmpzb24iLAogICAgICAgICJkaWdlc3QiOiAiNjA0ZGU1ZmJkZmQwZjU5NTM3YTJiOWUwNTQ5NWNhYmU4ZjU2ZDQ2YTVkZDVmNGJmMTZkOGRiOTQzZjI2ZDNiNCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3Ryb3VibGVzaG9vdGluZy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICIzMTg2YTZiNmFlMDZjODBiZDJlNTc5ZjUwZWFhNzIyZTI3ZWExZjc0YjBmODc0NzdhNDBmYTNkZjJiZDAzMWM3IgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJkaWdlc3QiOiAiNmM0YmE1ZDhmZTdlZDNkMGE0OGEwMjIyOTUxMzc3YzQ4MzkzMWFlNmVlNWVkZmRmMzg3ZDNkODZhZWNjNmIzYSIKICAgICAgfQogICAgXQogIH0KfQ==","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQD+PueX17oc7q6BUiNT18dM+tQqpA9Jsx1SYZS07aZpDFgTNyimS/IMfSIymSt0qC0CMQCXlu14nroe6oEg0A9h1jx1T3Igv2rL1mLmFyYxo1iTAuQG5MhG5ihy6mbsVkCbpxU=","keyid":""}]}} \ No newline at end of file +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibnZpZGlhLW9udG9sb2d5LXNldHVwIiwKICAgICAgImRpZ2VzdCI6IHsKICAgICAgICAic2hhMjU2IjogIjIwNTQ1ZDQ5ZmMxMjFjZGUwNTMxZTQ2NGI2YmYxZWFjYTY4OTFjNmQzYjE1ZTY0MWQ2ZTQyZDEwNjU4YTE5YTEiCiAgICAgIH0KICAgIH0KICBdLAogICJwcmVkaWNhdGVUeXBlIjogImh0dHBzOi8vbW9kZWxfc2lnbmluZy9zaWduYXR1cmUvdjEuMCIsCiAgInByZWRpY2F0ZSI6IHsKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0IiwKICAgICAgICAiLmdpdGh1YiIsCiAgICAgICAgIi5naXRpZ25vcmUiCiAgICAgIF0sCiAgICAgICJtZXRob2QiOiAiZmlsZXMiLAogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIKICAgIH0sCiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJlZTAxOTdjOGZjYzNhMmIxZmJmNDM2Y2RiY2M5ZGViNzQzYzMyNjNiZjc4NGE5MTM4MzQzNzBiM2RmMWQwZjFiIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI0MjcwNTAzOWI4MTQ4N2I1ZTRkM2ExMTZlMzg2ZDY3Zjg0M2Y0OTQ1ZjllMjk4YzVlZjA2NGM2ZGQ3NmQ5MzUzIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjEzYTk3YWRkMmQ1ZDMzZDQ1MTE5ZDQ2ZTQ2ZmIyMDBkMTJkMjQ0ZWMxN2IyNjA5OWNjNDFmNGJiYWVmMTgyNjciLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJhc3NldHMvcnVudGltZS1jb250cmFjdC55YW1sIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI2MDRkZTVmYmRmZDBmNTk1MzdhMmI5ZTA1NDk1Y2FiZThmNTZkNDZhNWRkNWY0YmYxNmQ4ZGI5NDNmMjZkM2I0IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiMzE4NmE2YjZhZTA2YzgwYmQyZTU3OWY1MGVhYTcyMmUyN2VhMWY3NGIwZjg3NDc3YTQwZmEzZGYyYmQwMzFjNyIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvdHJvdWJsZXNob290aW5nLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJlOTE0NGZjZWM1MTJjNTZjYTlmOGIzNWU0NjNlY2QzZTdiMjVhYzY1NDU0ZDQ4Nzk0NGU1Mjc5Yjc4NTdlOGE1IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIKICAgICAgfQogICAgXQogIH0KfQ==","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGYCMQC/WCwaKmwYM3o2PgOav63o8qgECR4v1yo/Ij0BPusP8jj9ZcVAQvLCbw8R270UYnoCMQDWuwOSAACzGZfM6Z7YuLeg9au7JNQyXpgjtqDRq75jtRTtxabjaOWt6nKjHvPIevU=","keyid":""}]}} \ No newline at end of file