Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 40 additions & 46 deletions skills/cuopt-numerical-optimization-api/BENCHMARK.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
# Skill Benchmark: cuopt-numerical-optimization-api

> ❌ **Overall verdict: FAIL — Publication blocked**
> ✅ **Overall verdict: PASS — Recommended for publication**

The skill should be reviewed before publication. Address the blocking findings below, then rerun Skill Evaluator.
## Publication Recommendation

Recommended for publication based on the completed evaluation evidence in this report.

## Evaluation Metadata

- Skill: `cuopt-numerical-optimization-api`
- Validation status: `failed`
- Validation status: `passed`
- Evaluation date: 2026-11-10
- Evaluator version: `1.6.9`
- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`), Codex (`openai/openai/gpt-6.1-sol`)
Expand Down Expand Up @@ -48,12 +50,12 @@ The three-tier evaluation checks whether the skill:

| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
|---|---:|---:|
| Overall | 94.8% — baseline ran, but no comparable score was available; uplift unavailable | 86.0% — baseline ran, but no comparable score was available; uplift unavailable |
| Overall | 95.2% — baseline ran, but no comparable score was available; uplift unavailable | 86.6% — baseline ran, but no comparable score was available; uplift unavailable |
| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |
| Correctness | 86.7% → 100.0% (+13.3 points) | 91.1% → 95.6% (+4.5 points) |
| Discoverability | 100.0% — baseline ran, but no comparable score was available; uplift unavailable | 93.3% — baseline ran, but no comparable score was available; uplift unavailable |
| Effectiveness | 71.8% → 91.4% (+19.6 points) | 68.4% → 66.8% (-1.6 points) |
| Efficiency | 82.7% — baseline ran, but no comparable score was available; uplift unavailable | 74.3% — baseline ran, but no comparable score was available; uplift unavailable |
| Discoverability | 100.0% — baseline ran, but no comparable score was available; uplift unavailable | 92.8% — baseline ran, but no comparable score was available; uplift unavailable |
| Effectiveness | 65.1% → 91.4% (+26.3 points) | 56.2% → 74.1% (+17.9 points) |
| Efficiency | 84.6% — baseline ran, but no comparable score was available; uplift unavailable | 70.7% — baseline ran, but no comparable score was available; uplift unavailable |

**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.

Expand All @@ -66,69 +68,61 @@ Shared-source comparisons use the same signals and weights on both conditions. T
| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation |
| --- | --- | --- | --- | --- |
| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. |
| claude-code | Efficiency | Token-only comparison | 97.1% → 96.9% (-0.2 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |
| claude-code | Efficiency | Token-only comparison | 98.7% → 97.0% (-1.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |
| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. |
| codex | Efficiency | Token-only comparison | 88.4% → 88.7% (+0.3 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |
| codex | Efficiency | Token-only comparison | 88.3% → 88.3% (±0.0 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |

## Token Usage

Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.

| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |
|---|---|---:|---:|---:|---:|---|
| claude-code | All cases | 1,126,889 | 538,496 | +588,393 | +109.27% | skill 9/9; base 9/9 |
| claude-code | numopt-c-eval-001-milp-api-call-sequence | 194,772 | 72,094 | +122,678 | +170.16% | skill 1/1; base 1/1 |
| claude-code | numopt-c-eval-002-parameter-function-wrong-name | 113,539 | 71,561 | +41,978 | +58.66% | skill 1/1; base 1/1 |
| claude-code | numopt-c-eval-003-csr-constraint-matrix | 111,249 | 35,807 | +75,442 | +210.69% | skill 1/1; base 1/1 |
| claude-code | numopt-c-eval-004-qp-restrictions | 111,084 | 71,256 | +39,828 | +55.89% | skill 1/1; base 1/1 |
| claude-code | numopt-cli-eval-001-mps-sections-and-cli-command | 112,368 | 71,971 | +40,397 | +56.13% | skill 1/1; base 1/1 |
| claude-code | numopt-py-eval-001-lp-api-call-sequence | 112,297 | 36,069 | +76,228 | +211.34% | skill 1/1; base 1/1 |
| claude-code | numopt-py-eval-002-status-case-sensitivity | 148,808 | 35,748 | +113,060 | +316.27% | skill 1/1; base 1/1 |
| claude-code | numopt-py-eval-003-integer-vs-continuous-workers | 110,792 | 36,027 | +74,765 | +207.52% | skill 1/1; base 1/1 |
| claude-code | numopt-py-eval-004-qp-maximize-workaround | 111,980 | 107,963 | +4,017 | +3.72% | skill 1/1; base 1/1 |
| codex | All cases | 555,489 | 238,435 | +317,054 | +132.97% | skill 9/9; base 9/9 |
| codex | numopt-c-eval-001-milp-api-call-sequence | 48,492 | 26,160 | +22,332 | +85.37% | skill 1/1; base 1/1 |
| codex | numopt-c-eval-002-parameter-function-wrong-name | 60,990 | 26,007 | +34,983 | +134.51% | skill 1/1; base 1/1 |
| codex | numopt-c-eval-003-csr-constraint-matrix | 73,056 | 18,010 | +55,046 | +305.64% | skill 1/1; base 1/1 |
| codex | numopt-c-eval-004-qp-restrictions | 70,129 | 46,929 | +23,200 | +49.44% | skill 1/1; base 1/1 |
| codex | numopt-cli-eval-001-mps-sections-and-cli-command | 48,040 | 26,067 | +21,973 | +84.29% | skill 1/1; base 1/1 |
| codex | numopt-py-eval-001-lp-api-call-sequence | 48,447 | 21,924 | +26,523 | +120.98% | skill 1/1; base 1/1 |
| codex | numopt-py-eval-002-status-case-sensitivity | 61,889 | 21,427 | +40,462 | +188.84% | skill 1/1; base 1/1 |
| codex | numopt-py-eval-003-integer-vs-continuous-workers | 62,331 | 21,781 | +40,550 | +186.17% | skill 1/1; base 1/1 |
| codex | numopt-py-eval-004-qp-maximize-workaround | 82,115 | 30,130 | +51,985 | +172.54% | skill 1/1; base 1/1 |
| ALL AGENTS | Dataset aggregate | 1,682,378 | 776,931 | +905,447 | +116.54% | skill 18/18; base 18/18 |
| claude-code | All cases | 1,121,154 | 501,132 | +620,022 | +123.72% | skill 9/9; base 9/9 |
| claude-code | numopt-c-eval-001-milp-api-call-sequence | 112,671 | 71,924 | +40,747 | +56.65% | skill 1/1; base 1/1 |
| claude-code | numopt-c-eval-002-parameter-function-wrong-name | 149,126 | 71,816 | +77,310 | +107.65% | skill 1/1; base 1/1 |
| claude-code | numopt-c-eval-003-csr-constraint-matrix | 150,215 | 35,779 | +114,436 | +319.84% | skill 1/1; base 1/1 |
| claude-code | numopt-c-eval-004-qp-restrictions | 111,370 | 71,217 | +40,153 | +56.38% | skill 1/1; base 1/1 |
| claude-code | numopt-cli-eval-001-mps-sections-and-cli-command | 112,014 | 71,445 | +40,569 | +56.78% | skill 1/1; base 1/1 |
| claude-code | numopt-py-eval-001-lp-api-call-sequence | 111,558 | 35,759 | +75,799 | +211.97% | skill 1/1; base 1/1 |
| claude-code | numopt-py-eval-002-status-case-sensitivity | 111,038 | 35,707 | +75,331 | +210.97% | skill 1/1; base 1/1 |
| claude-code | numopt-py-eval-003-integer-vs-continuous-workers | 110,549 | 71,226 | +39,323 | +55.21% | skill 1/1; base 1/1 |
| claude-code | numopt-py-eval-004-qp-maximize-workaround | 152,613 | 36,259 | +116,354 | +320.90% | skill 1/1; base 1/1 |
| codex | All cases | 535,086 | 242,605 | +292,481 | +120.56% | skill 9/9; base 9/9 |
| codex | numopt-c-eval-001-milp-api-call-sequence | 52,596 | 25,850 | +26,746 | +103.47% | skill 1/1; base 1/1 |
| codex | numopt-c-eval-002-parameter-function-wrong-name | 52,572 | 21,721 | +30,851 | +142.03% | skill 1/1; base 1/1 |
| codex | numopt-c-eval-003-csr-constraint-matrix | 48,334 | 17,912 | +30,422 | +169.84% | skill 1/1; base 1/1 |
| codex | numopt-c-eval-004-qp-restrictions | 73,110 | 47,107 | +26,003 | +55.20% | skill 1/1; base 1/1 |
| codex | numopt-cli-eval-001-mps-sections-and-cli-command | 52,456 | 25,905 | +26,551 | +102.49% | skill 1/1; base 1/1 |
| codex | numopt-py-eval-001-lp-api-call-sequence | 48,187 | 26,163 | +22,024 | +84.18% | skill 1/1; base 1/1 |
| codex | numopt-py-eval-002-status-case-sensitivity | 66,940 | 21,741 | +45,199 | +207.90% | skill 1/1; base 1/1 |
| codex | numopt-py-eval-003-integer-vs-continuous-workers | 62,236 | 17,540 | +44,696 | +254.82% | skill 1/1; base 1/1 |
| codex | numopt-py-eval-004-qp-maximize-workaround | 78,655 | 38,666 | +39,989 | +103.42% | skill 1/1; base 1/1 |
| ALL AGENTS | Dataset aggregate | 1,656,240 | 743,737 | +912,503 | +122.69% | skill 18/18; base 18/18 |

Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. Change is `100 * (with - without) / without`; negative means fewer tokens. N/A comparisons include their reason in the coverage column; coverage is never estimated.

## Tier Status

| Tier | Purpose | Status | Evidence |
|---|---|---|---|
| Tier 1 | Static validation | **FAILED** | 11 validator(s); 10 finding(s) |
| Tier 2 | Semantic deduplication | **PASSED WITH OBSERVATIONS** | 2 validator(s); 3 finding(s) |
| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 10 finding(s) |
| Tier 2 | Semantic deduplication | **PASSED WITH OBSERVATIONS** | 2 validator(s); 1 finding(s) |
| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 9 task(s) |

## Findings and Observations

<details>
<summary>Show detailed findings and successful checks</summary>

- **HIGH** DUPLICATE/duplicate: Duplicate content found across assets/python/README.md and references/c_api.md and references/cli_api.md and references/python_api.md:
"# Assets — reference models" in assets/python/README.md (lines 1-17)
vs "## Reference Models" in references/c_api.md (lines 132-141)
vs "## Reference Models" in references/cli_api.md (lines 64-70)
vs "## Reference Models" in references/python_api.md (lines 137-149) (`assets/python/README.md:1`)
- **HIGH** DUPLICATE/duplicate: Duplicate content found across SKILL.md and references/qp_examples.md:
"## QP Rules (all interfaces)" in SKILL.md (lines 51-57)
vs "## Critical Reminders" in references/qp_examples.md (lines 180-188) (`SKILL.md:51`)
- **HIGH** DUPLICATE/duplicate: Duplicate content found across assets/python/portfolio/README.md and assets/python/portfolio/model.py and references/python_api.md and references/qp_examples.md:
"# Portfolio optimization (QP)" in assets/python/portfolio/README.md (lines 1-7)
vs "(module docstring)" in assets/python/portfolio/model.py (lines 1-3)
vs "## QP Example (beta — MINIMIZE only)" in references/python_api.md (lines 51-73)
vs "## Portfolio Optimization" in references/qp_examples.md (lines 3-19) (`assets/python/portfolio/README.md:1`)
- **MEDIUM** DUPLICATE/duplicate: Duplicate content found across assets/python/mps_solver/README.md and references/cli_api.md:
"## MPS File Format" in assets/python/mps_solver/README.md (lines 15-38)
vs "## MPS Format (required sections, in order)" in references/cli_api.md (lines 44-54) (`assets/python/mps_solver/README.md:15`)
- **MEDIUM** QUALITY/quality_efficiency: Deeply nested references in c_api.md (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- **MEDIUM** SCHEMA/frontmatter_field_placement: Root field 'version' is ignored; use 'metadata.version' (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- 8 additional finding(s) are available in the full evaluation artifacts.
- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- 6 additional finding(s) are available in the full evaluation artifacts.

</details>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,4 @@ gcc -I${INCLUDE_PATH} -L${LIB_PATH} -o lp_simple lp_simple.c -lcuopt
LD_LIBRARY_PATH=${LIB_PATH}:$LD_LIBRARY_PATH ./lp_simple
```

**See also:** [references/examples.md](../../references/examples.md) for parameter constants and more examples.
**See also:** [references/c_api.md](../../../references/c_api.md) for parameter constants and more examples.
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ gcc -I${INCLUDE_PATH} -L${LIB_PATH} -o lp_duals lp_duals.c -lcuopt
LD_LIBRARY_PATH=${LIB_PATH}:$LD_LIBRARY_PATH ./lp_duals
```

**See also:** [references/examples.md](../../references/examples.md) for full parameter reference.
**See also:** [references/c_api.md](../../../references/c_api.md) for full parameter reference.
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ gcc -I${INCLUDE_PATH} -L${LIB_PATH} -o milp_simple milp_simple.c -lcuopt
LD_LIBRARY_PATH=${LIB_PATH}:$LD_LIBRARY_PATH ./milp_simple
```

**See also:** [references/examples.md](../../references/examples.md) for full parameter reference.
**See also:** [references/c_api.md](../../../references/c_api.md) for full parameter reference.
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ gcc -I${INCLUDE_PATH} -L${LIB_PATH} -o milp_production milp_production.c -lcuopt
LD_LIBRARY_PATH=${LIB_PATH}:$LD_LIBRARY_PATH ./milp_production
```

**See also:** [references/examples.md](../../references/examples.md) for parameters and MIP options.
**See also:** [references/c_api.md](../../../references/c_api.md) for parameters and MIP options.
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ LD_LIBRARY_PATH=${LIB_PATH}:$LD_LIBRARY_PATH ./mps_solver data/sample.mps

**Data:** `data/sample.mps` is a small LP (two variables, two constraints). Use any MPS file path as the first argument.

**See also:** [references/examples.md](../../references/examples.md); repo example `docs/cuopt/source/cuopt-c/lp-qp-milp/examples/mps_file_example.c`.
**See also:** [references/c_api.md](../../../references/c_api.md); repo example `docs/cuopt/source/cuopt-c/lp-qp-milp/examples/mps_file_example.c`.
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ LP, MILP, and QP reference implementations. Use as reference when building new a
| least_squares | QP |
| maximization_workaround | QP |

**Run:** From each subdir, `python model.py`. QP is **beta** and supports **MINIMIZE** only. See [references/qp_examples.md](../references/qp_examples.md) for additional QP examples.
**Run:** From each subdir, `python model.py`. QP is **beta** and supports **MINIMIZE** only. See [references/qp_examples.md](../../references/qp_examples.md) for additional QP examples.
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
# Portfolio optimization (QP)

Minimize portfolio variance (risk) subject to fully invested (sum x = 1) and minimum return. Three assets; Q must be PSD.

**Run:** `python model.py`

**Note:** QP is beta; objective must be MINIMIZE.
Run: `python model.py`. Problem and constraints are described in `model.py`; see [references/qp_examples.md](../../../references/qp_examples.md).
Original file line number Diff line number Diff line change
Expand Up @@ -131,11 +131,4 @@ See [assets/c/README.md](../assets/c/README.md) for the conda-env include/librar

## Reference Models

| Model | Type | Location |
|-------|------|----------|
| Simple LP | LP | [assets/c/lp_basic/](../assets/c/lp_basic/) |
| Dual values | LP | [assets/c/lp_duals/](../assets/c/lp_duals/) |
| PDLP warmstart | LP | [assets/c/lp_warmstart/](../assets/c/lp_warmstart/) |
| Integer variable | MILP | [assets/c/milp_basic/](../assets/c/milp_basic/) |
| Production planning | MILP | [assets/c/milp_production_planning/](../assets/c/milp_production_planning/) |
| MPS file solver | LP/MILP | [assets/c/mps_solver/](../assets/c/mps_solver/) |
See [assets/c/README.md](../assets/c/README.md).
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,4 @@ Quadratic objectives are **MINIMIZE only** (for maximization, negate the objecti

## Reference Models

| Model | Type | Location |
|-------|------|----------|
| Minimal LP | LP | [assets/cli/lp_simple/](../assets/cli/lp_simple/) |
| Production planning | LP | [assets/cli/lp_production/](../assets/cli/lp_production/) |
| Facility location | MILP | [assets/cli/milp_facility/](../assets/cli/milp_facility/) |
See [assets/cli/README.md](../assets/cli/README.md).
Original file line number Diff line number Diff line change
Expand Up @@ -48,28 +48,9 @@ if problem.Status.name in ["Optimal", "FeasibleFound"]:
print(f"Open: {open_facility.getValue() > 0.5}, Production: {production.getValue()}")
```

## QP Example (beta — MINIMIZE only)
## QP Examples (beta — MINIMIZE only)

```python
problem = Problem("Portfolio")
x1 = problem.addVariable(lb=0, ub=1, vtype=CONTINUOUS, name="stock_a")
x2 = problem.addVariable(lb=0, ub=1, vtype=CONTINUOUS, name="stock_b")
x3 = problem.addVariable(lb=0, ub=1, vtype=CONTINUOUS, name="stock_c")

problem.setObjective(
0.04*x1*x1 + 0.02*x2*x2 + 0.01*x3*x3
+ 0.02*x1*x2 + 0.01*x1*x3 + 0.016*x2*x3,
sense=MINIMIZE,
)
problem.addConstraint(x1 + x2 + x3 == 1, name="budget")
problem.addConstraint(0.12*x1 + 0.08*x2 + 0.05*x3 >= 0.08, name="min_return")

problem.solve(SolverSettings())
if problem.Status.name in ["Optimal", "PrimalFeasible"]:
print(f"Variance: {problem.ObjValue}")
```

See [qp_examples.md](qp_examples.md) for least-squares, maximization workaround, and covariance matrix expansion.
See [qp_examples.md](qp_examples.md) for portfolio, least-squares, maximization workaround, and covariance matrix expansion.

## CRITICAL: Status Values Use PascalCase

Expand Down Expand Up @@ -136,14 +117,4 @@ problem.addConstraint(expr <= 100)

## Reference Models

| Model | Type | Location |
|-------|------|----------|
| Minimal LP | LP | [assets/python/lp_basic/](../assets/python/lp_basic/) |
| Dual values | LP | [assets/python/lp_duals/](../assets/python/lp_duals/) |
| PDLP warmstart | LP | [assets/python/lp_warmstart/](../assets/python/lp_warmstart/) |
| Integer variables | MILP | [assets/python/milp_basic/](../assets/python/milp_basic/) |
| Production planning | MILP | [assets/python/milp_production_planning/](../assets/python/milp_production_planning/) |
| Portfolio variance | QP | [assets/python/portfolio/](../assets/python/portfolio/) |
| Least squares | QP | [assets/python/least_squares/](../assets/python/least_squares/) |
| Maximization workaround | QP | [assets/python/maximization_workaround/](../assets/python/maximization_workaround/) |
| MPS file solver | LP/MILP | [assets/python/mps_solver/](../assets/python/mps_solver/) |
See [assets/python/README.md](../assets/python/README.md).
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -177,12 +177,9 @@ objective = (
)
```

## Critical Reminders
## Reminders

1. **MINIMIZE only** - solver rejects MAXIMIZE for QP
2. **Convexity** - Q should be positive semi-definite
3. **Beta status** - API may change in future versions
4. **Status checking** - use PascalCase: `"Optimal"` not `"OPTIMAL"`
See the QP rules in [SKILL.md](../SKILL.md#qp-rules-all-interfaces); status strings are PascalCase (`"Optimal"`).

---

Expand Down
Loading
Loading