feat(tier3): upgrade Harbor to 0.22.0

[chrisknvidia]

Closes #79

Summary

• Upgrade the pinned and locked Tier 3 backend from Harbor 0.13.2 to the current stable release, Harbor 0.22.0.
• Adapt command construction, secure Docker and trusted-local execution, installed-agent lifecycles, task staging, multi-step rewards, result collection, attempt accounting, and unified reports to Harbor 0.22 contracts.
• Keep native task selectors, logical dataset IDs, task display names, and structural attempt ordinals separate and fail closed on ambiguous identity or incomplete results.
• Preserve paired-baseline isolation by rejecting skill-controlled setup, healthchecks, prior trajectories, step workdirs/tests, verifier controls, and collect hooks where they could contaminate standard grading; retain Harbor-native ownership in custom_only mode.
• Harden diagnostics and artifacts against credential/userinfo leakage, bound report/error payloads, validate finite timeout values, and make mixed metric contracts and unavailable telemetry explicit.
• Bring local OpenCode NVIDIA Build execution into parity with Harbor 0.22's instruction/error lifecycle.

Harbor v0.22.0 is the latest upstream release as of 2026-08-26: https://github.com/harbor-framework/harbor/releases/tag/v0.22.0

Verification through ca3e731

The full local suites, packaging gates, and real Harbor runs below were collected on implementation snapshot 8b0110f. Follow-ups a783c2a and ca3e731 are limited to a behavior-preserving helper rename plus persistence assertion and a portable test invocation; focused local regressions and the complete exact-head GitHub matrix cover those diffs.

Automated

• Python 3.12 full suite: 6378 passed, 19 skipped, 9 deselected in 594.56s.
• Isolated Python 3.13 full suite with all extras and editable source: 6378 passed, 19 skipped, 9 deselected in 578.52s.
• Real-process integration slice for Docker sidecar cleanup, redaction, interrupted lifecycle, and local streaming: 7 passed in 52.58s.
• Report-focused suites: 350 passed, 4 skipped; direct report regression slice: 11 passed.
• Packaging/boundary suite: 241 passed in 39.56s; public benchmark gate passed.
• Wheel and sdist build plus strict Twine checks passed. Isolated wheel smoke installs passed on Python 3.12.12 and 3.13.11, including CLI, doctor, packaged templates, and harbor.__version__ == "0.22.0".
• Ruff lint passed; all 28 changed Python files pass Ruff format checking; git diff --check passed. The repository-wide format check still lists 18 files, all byte-identical to origin/main.
• Bandit high-severity scan: zero findings. pip-audit: no known vulnerabilities (the unpublished local skillevaluator package is skipped). Path-aware Gitleaks scans passed for the tracked diff and all 30 branch commits.
• Fern docs validation: 0 errors, 1 warning; redirect validation was unavailable without Fern authentication.
• Post-review CodeQL follow-up: the name-based sensitive-data heuristic was removed by renaming an integer-only attempt helper from trusted to structural; persistence coverage now also proves that all runner-internal underscore metadata is omitted from reward.json. Focused identity tests: 3 passed; long stop-on-pass merge path: 1 passed; broader collector suites: 176 passed.
• Windows Tier 2 exposed a CI-only harness bug: Windows resolved bash to the WSL launcher instead of a POSIX shell. The isolation test now invokes the staged evaluator with sys.executable -I on every OS while separately asserting the generated shell contract; the full isolation/judge slice passed with 459 passed, 2 skipped.
• All 30 commits have DCO sign-offs.
• Exact-head GitHub validation passed all 15 checks: Python 3.12/3.13, Windows Tier 2 (1292 passed, 30 skipped), native-Windows fail-closed, macOS Tier 2/Tier 3, RHEL 8, packaging, dependency review, DCO, Gitleaks, the Security CodeQL job, and the separate GitHub CodeQL gate. Alert fix: upgrade langchain-core for CVE-2026-45134 #10 is fixed by fresh analysis with zero results; it was not dismissed or suppressed.

Real Harbor 0.22 integration paths

1. Native Harbor BYOT / Docker / Codex / custom-only / skip baseline

   • Real Docker agent execution, step-local verifier, collection, retained jobs, and report completed in 220.5s.
   • Deliberately distinct selector (selector-attempt2), logical entry ID (logical-attempt9), and display name (publisher/display-attempt7) remained distinct.
   • The single step and custom reward scored overall=1.0 and native_quality=1.0; the staged task contained only the intended step-local test overlay.

2. Generated dataset / Docker / Codex / runtime setup / unified reports

   • Blocking top-level validation completed with exit 0 in 4m38s using custom_only and --skip-baseline.
   • A non-credential runtime_env value reached the verifier, and two pre-agent setup commands persisted an independently checked artifact before agent execution.
   • Custom metrics canary_protocol=1.0 and runtime_setup=1.0; 1/1 expected/scored attempt; no baseline or lift was synthesized.
   • JSON, HTML, and Markdown reports were generated and inspected. JSON records Tier 3 succeeded, overall 1.0, and both custom metrics; HTML contains both evaluator cards; Markdown records the overall passed validation and completed Tier 3 stage. The Tier 3 verdict is correctly neutral because this was custom-only, skip-baseline evidence with no lift.

3. Native Harbor BYOT / local / OpenCode / NVIDIA Build / custom-only

   • Required macOS Seatbelt sandbox, network explicitly enabled, and ambient agent-key inheritance disabled.
   • Real OpenCode 1.2.26 execution, multi-step ATIF conversion, step-local verification, collection, and report completed with exit 0 in 84.6s.
   • 1/1 expected/scored attempt; overall=1.0, native_quality=1.0; logical attempt accounting correctly recorded actual attempt 1 despite attempt-like selector/ID text.
   • ATIF v1.7 retained 9 steps and real totals of 110,717 input, 84,480 cached, and 3,568 output tokens. No matching local process or container remained after completion.

4. Generated dataset / Docker / Codex / paired baseline / standard + custom grading

   • Docker runtime preflight passed on a real model request.
   • After making the negative-control prompt terminate deterministically, controlled paired runs completed both agent arms and the with-skill arm scored all standard metrics plus its custom metric.
   • The external standard accuracy judge returned malformed non-JSON for the baseline, including once with the stronger model selected through the new verifier-only judge override. SkillEvaluator correctly treated the reward as unscoreable and failed closed, so current-head paired lift is not claimed as fully scored.

Exact provider-key scans found no configured provider credential in retained real-run artifacts. Temporary E2E fixtures/results were removed from the worktree after inspection; no fixture or generated runtime token is included in this branch.

Review

• Independent Review Guru passes found no remaining P1/P2 implementation defect.
• Review covered native identity and attempt spoofing, baseline contamination, standard/custom verifier boundaries, local and Docker lifecycle cleanup, option interactions, finite timeouts, error/truncation accounting, mixed-contract report truth, step/subagent trajectories, token semantics, secret/userinfo redaction, and regression behavior.
• A separate issue-to-diff audit confirmed the implementation covers issue [FEA]: Upgrade Tier 3 Harbor backend from 0.13.2 to 0.20+ #79's Harbor 0.22 compatibility surfaces and identified the runtime-setup and Markdown-report proof gaps; both are now covered by the successful generated Docker run above.
• Two independent follow-up reviews confirmed the CodeQL finding was a function-name heuristic rather than persisted sensitive data and recommended the behavior-preserving rename plus persistence-boundary assertion now included in a783c2a.

Remaining verification gaps

• Managed Harbor cloud backends were not exercised because no isolated approved target and credentials were available; the configured Kubernetes context is shared.
• Native Windows execution was not available locally. Windows behavior remains fail-closed by design and awaits exact-head CI confirmation.
• The current-head paired default-plus-custom path is not a fully scored lift success because the external baseline accuracy judge emitted malformed JSON as described above. Harbor execution, artifact collection, and fail-closed reward handling completed, but a publishable baseline score was not produced.
• Docker/Codex did not expose numeric Harbor token totals; reports correctly leave that telemetry unavailable. The local/OpenCode run above provides real token-metric coverage.

Draft status

This PR remains intentionally draft. Exact-head remote CI passed all 15 checks on ca3e731. Do not mark ready or merge until maintainer review is complete.

[rng1995]

One blocking merge-integration issue remains. Exact-head CI and DCO are green, but the collector conflict with current main must be resolved semantically before this can merge.

[rng1995]

[P1] Preserve the exception-type gate from current main\n\nThis scans plain agent transcripts whenever any root/step exception has a message. After merging current main, a verifier/healthcheck/task failure plus harmless agent prose such as 401 Unauthorized is therefore reported as an agent runtime failure, reintroducing the bug fixed by #84. I reproduced that on this head. Please rebase and enable text-log scanning only when at least one root/step exception_type is in _AGENT_RUNTIME_EXCEPTION_TYPES or _UNCONDITIONAL_AGENT_RUNTIME_EXCEPTION_TYPES; keep the JSON type=error and trajectory paths unchanged.

[rng1995]

@chrisknvidia Gentle ping when you have a chance: the remaining review thread is still waiting for an update or response. Please take a look when you can.