Restore pinned artifact downloads and propagate host URL limits

[nirmangupta]

Summary

Restore pinned HTTPS system_prompt_url and OPA bundle_url consumers and
propagate host URL limits through bundled dispatchers and Python, Node, and FFI
constructors. The inspected ACS baseline has bounded manifest extends fetching
but lacks these artifact consumers, so this is restoration plus limits
propagation, not constructor-only wiring.

Related: #21, #63.

Behavior and compatibility

• Reuse bounded HTTPS fetching with SHA256/SRI verification, byte limits,
  timeouts, and redirect limits. Reject malformed or conflicting sources before
  dispatch, including for hosts using custom callbacks.
• Preserve existing constructors and default limits. Artifact GET budgets do not
  change inference POST settings or custom callbacks' I/O.
• Stage remote OPA bundles in unique temporary directories and clean up after
  use, with bounded retries and a diagnostic if cleanup fails.
• Keep Regorus as the default when enabled; it rejects remote bundle archives.
  OPA remains opt-in. SDK manifest loaders' default URL budgets and top-level
  manifest URL loading are outside this change.

Validation

The public FFI regression
hook_constructor_passes_url_limits_to_default_annotators_only was copied to
baseline fa80590476ab629c255ab10b54ddd8d96e7348ce. It compiled and failed the
zero-timeout assertion with an HTTP connection error. It passes with this change.

Local Windows ARM64 results:

Command or suite	Result
cargo test --workspace --all-features --locked -- --test-threads=1	413 passed
cargo test -p agent-control-spec --no-default-features --features opa,default-dispatchers --locked --lib	123 passed
Python host-hook suite	32 passed
Node host-hook suite	21 passed
cargo check --workspace --all-features --locked	Passed
cargo check --manifest-path sdk/python/Cargo.toml --locked	Passed
Rust formatting/Clippy, Ruff, actionlint	Passed

The full Rust runs preceded the final scope-only trims (crate-private fetch
helpers and removal of redundant SDK parsing assertions). Affected artifact
tests, SDK host-hook suites, backend-specific compilation checks, and the
workspace/Python compilation checks passed after those trims.

OPA runs used 1.20.2 with AGENT_CONTROL_REQUIRE_OPA=1; removing OPA from PATH
correctly failed rather than skipped. CI installs checksum-pinned OPA and adds
the OPA-only configuration, which all-features testing does not exercise.

Pending before ready for review

• Linux/Unix execution and the remaining required CI jobs. The Unix-only OPA
  lifecycle case has not been executed locally.
• The initial parallel Rust run hit the unchanged
  rego_dispatcher_recovers_after_a_timed_out_evaluation test's five-second
  control timeout. It passed alone and in the full serial run; the parallel run
  is not claimed green.

A backend-free --no-default-features check also exposed a baseline compilation
error: Manifest calls rego_adapter_data_paths, whose definition is gated by
rego/opa. That unrelated issue is unchanged.

[MohammadHaroonAbuomar]

Thanks for this contribution, and for the unusually careful PR description. The claims in it check out. I ran the branch on Linux x86_64:

• cargo test -p agent-control-spec --all-features --locked --lib: 227 passed, with a real opa on PATH, so the Unix-only staged-bundle lifecycle test you could not run locally passes here too.
• cargo test -p agent-control-spec --no-default-features --features opa,default-dispatchers --locked --lib with AGENT_CONTROL_REQUIRE_OPA=1: 124 passed.
• Clippy with -D warnings and workspace cargo fmt --check: clean (one formatting nit in the Python crate, noted inline).
• The pinned OPA checksum in CI matches the upstream opa_linux_amd64_static.sha256 for v1.20.2.
• The --no-default-features build error you flagged reproduces identically on the base commit, so it is pre-existing, as you said.

The security shape looks right:

• A pin is required (exactly one of sha256 or integrity) before any network access, the digest is verified over the full body, and each failure path (mismatch, oversize, timeout, redirect budget, downgrade redirect, non-2xx, bad UTF-8) fails closed with a deterministic reason code and a test that pins it.
• Testing against a real loopback TLS listener with production certificate verification intact, instead of mocking the transport, is the right call. The production_tls_does_not_trust_test_ca guard is a nice touch.

Two items before this is ready:

1. The commit needs a Signed-off-by trailer (git commit --amend -s); every human commit in this repo carries one.
2. CI has not run yet (a first contribution needs workflow approval); once the branch is out of draft the required jobs can run.

One framing note for other readers: "restore" can read as a regression fix, but these consumers never existed in this repo's history; they come from the pre-extraction engine, as #63 says. The PR body already explains this, so no change needed.

Inline comments cover the rest. None are blockers.

[MohammadHaroonAbuomar]

This fetches the prompt on every dispatch, so each annotated intercept pays a network round trip and the artifact host becomes an availability dependency at evaluation time. The pin makes the content immutable, so a digest-keyed in-memory cache would be sound and would remove both costs. Fine as a follow-up; the fail-closed behavior here is correct.

[MohammadHaroonAbuomar]

Same per-evaluation refetch as the prompt path: each OPA evaluation downloads, stages, and deletes the bundle. The staging and cleanup design (unique 0o700 directory, create_new, bounded retries, Drop guard) is solid. A digest-keyed cache of the verified bytes would cut the per-evaluation network and disk churn; follow-up material, not a blocker.

[MohammadHaroonAbuomar]

cargo fmt --manifest-path sdk/python/Cargo.toml --check rewraps this import (trailing comma inside a single-line brace list). The root-workspace cargo fmt --all does not cover this crate, so CI will not catch it.

[MohammadHaroonAbuomar]

This parallel run now includes the loopback tests with delayed responses, plus real OPA. You already saw the pre-existing rego_dispatcher_recovers_after_a_timed_out_evaluation five-second control timeout trip once under parallel load. Worth watching this job for that flake; if it trips, raising that test's control timeout is probably the fix rather than serializing the suite.

[MohammadHaroonAbuomar]

Tiny nit: this label is derived from require_success while the neighboring messages use context. Deriving it from context keeps the two from drifting if another caller shows up.