Repository navigation
Conversation
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope. This change belongs in this repo. OHE-3693 (In Progress, Medium) asks for named templates, Enterprise org policy and the conversation API. This is the first review from this account, and the PR has no earlier review threads.
What checks out. Templates are now named by their durable name, with the image kept separately. Old image-based IDs still resolve, preferring the operator default. The fix that uses the resolved image for SDK-mismatch guidance looks right. A bad SANDBOX_TEMPLATE_ORG_CONFIG now stops startup. Explicit sandbox_spec_id choices are checked server-side against the authenticated org. At 730df57, 19 CI checks passed and 1 was skipped. Locally, the focused suites passed against PostgreSQL: 535 tests across test_sandbox_template_policy, test_dynamic_remote_sandbox_spec_service, test_remote_sandbox_service, test_live_status_app_conversation_service, test_sandbox_spec_service and test_docker_sandbox_service.
Blocking: the org policy is not applied to integration-started conversations (inline). The policy only reaches get_sandbox_spec_service through AuthUserContext. GitHub/GitLab/Slack/Jira/Bitbucket/Azure DevOps conversations start under ResolverUserContext, which keeps the new base method that returns None. Those launches get the unscoped spec service: they can use a saved default that is outside the org's allowlist, they fall back to the global operator default rather than the org's default_template_id, and grouped-sandbox reuse stays on. I checked this directly. With a policy set for org X, get_org_template_policy(X) returns a policy, but ResolverUserContext(..., resolver_org_id=X).get_sandbox_template_policy() returns None. OHE-3693 and docs/sandbox-template-selector.md both promise that a configured policy stays enforced even when the picker is hidden.
Merge readiness (not code defects):
- GitHub reports the branch as
dirty. A trial merge withmain(fbb9addcd) conflicts inlive_status_app_conversation_service.py,auth_user_context.py,user_context.pyandtest_remote_sandbox_service.py. All four are on the paths this review covers, so the resolved code needs another look. - The PR body says the branch has "only the template feature", but the branch also adds the org policy (
SANDBOX_TEMPLATE_ORG_CONFIG), theENABLE_SANDBOX_SELECTORflag,GET /api/v1/sandbox-specs/selectionandsandbox_spec_idonPOST /api/v1/app-conversations. Please update the description so human reviewers see the full API and config surface. - The PR template asks for screenshots of the Settings label change. The body says this was verified, but the capture is only in a local bundle and is not attached.
- The PR is still a draft.
🔄 CHANGES REQUESTED
…nish-948 # Conflicts: # frontend/src/routes/app-settings.tsx # integrations/resolver_context.py # openhands/app_server/app_conversation/live_status_app_conversation_service.py # openhands/app_server/user/auth_user_context.py # openhands/app_server/user/user_context.py # tests/unit/app_server/test_remote_sandbox_service.py
…id contract Co-authored-by: openhands <openhands@all-hands.dev>
ScopedSandboxSpecService.allow_sandbox_reuse unconditionally disabled default grouping for any org with a template policy, even when the picker is hidden and the user never made an explicit choice. Re-enable default grouping for policy-scoped orgs, but add strict_spec_eligibility so a same-org running sandbox is only reused when its spec is still in the org's allowed template_ids; unlike the legacy best-effort preference for unscoped orgs (which falls back to a stale-image sandbox when nothing current is left), a revoked spec is never reintroduced as a last resort for policy-scoped orgs. An explicit sandbox_spec_id on the request still bypasses grouping entirely, as before. Attaching directly to an existing sandbox_id keeps its own ownership/credential_org_id check, unaffected by this change. Co-authored-by: openhands <openhands@all-hands.dev>
…y custom sandbox A failed agent-server create on an existing sandbox read `sandbox_spec.image`, where `sandbox_spec` came from `resolve_existing_sandbox_spec`. That method's docstring says only `working_dir` is safe to read from it: when the sandbox's own spec was removed from the catalog, it returns the *current default* spec as a best-effort stand-in. Reading `.image` off that stand-in silently reported someone else's image as "the actual image" on the sandbox -- masking the openhands-sdk mismatch hint when the fallback resolved to the current bundled image, and attributing an unrelated image to the sandbox otherwise. Resolve the sandbox's own spec explicitly (`get_sandbox_spec`) for this hint instead. When it is still in the catalog, trust its image. When it is gone, only a legacy image-backed `sandbox_spec_id` (pre-dating named specs, where the id literally is the image) is still a reliable image reference; a removed *named* spec's original image is unknown, so neither the name nor the fallback default's image is reported. Co-authored-by: openhands <openhands@all-hands.dev>
…g default; simplify grouping/error-hint per review Finding #3: ScopedSandboxSpecService.resolve_existing_sandbox_spec inherited the base fallback to get_default_sandbox_spec(), which here is policy-gated and raises when the org's configured default template is itself unavailable. That gate exists to fail closed on *new* launches; it must not also block reading working_dir for a sandbox that already exists (e.g. on archive/resume). Override it to fall back to the delegate's own, unwrapped default instead, so a missing/misconfigured org default can't break access to sandboxes that started before the misconfiguration. Review follow-ups on the prior commits in this branch: - allow_sandbox_reuse (base and ScopedSandboxSpecService) always returned True; remove the property and the caller branch that checked it. - _filter_sandboxes_with_known_spec and _filter_sandboxes_with_allowed_spec had identical memoize/iterate/exception-handling logic, differing only in which resolver method they called. Merge into _filter_sandboxes_with_eligible_spec, choosing get_start_sandbox_spec vs get_sandbox_spec once based on strict_spec_eligibility; the strict-vs-best-effort fallback behavior at the call site is unchanged. - The conversation-create error hint no longer makes a second catalog call (which could itself fail and replace the real create error). It reuses the sandbox_spec already resolved above, trusting sandbox_spec.image only when sandbox_spec.id matches the sandbox's own sandbox_spec_id (i.e. resolve_existing_sandbox_spec found the real spec rather than substituting its fallback default); otherwise it falls back to the stored id only when that id is itself a recognized legacy image reference. Narrowed looks_like_image_reference's docstring to note it only recognizes the agent-server's own <repo-path>:<tag> shape, not image references in general (a bare name or name:tag is also a valid image reference). Co-authored-by: openhands <openhands@all-hands.dev>
… the create-failure hint
Per independent PostgreSQL test run and Opus follow-up:
- test_start_app_conversation_reports_agent_server_error_body failed
with "Mock object has no attribute 'id'": pydantic model fields
aren't in dir(SandboxSpecInfo), so Mock(spec=SandboxSpecInfo) rejects
access to .id even though the field exists on real instances. Give
the shared _arrange_start_app_conversation fixture a real
SandboxSpecInfo (id matching the sandbox's own sandbox_spec_id, the
common non-fallback case) instead of a bare Mock. No production
getattr/defensive change; the attribute is always present on a real
SandboxSpecInfo.
- The create-failure image hint now also trusts sandbox_spec.image
when it's an exact alias match for the stored sandbox_spec_id (a
catalog can resolve a legacy short image name to a spec whose
canonical .id differs), not only when .id matches outright. No extra
lookup. Added an 'alias' case to the parametrized regression test.
- Reworded the three user-facing SandboxError messages to say
"Sandbox Spec" consistently and give a concrete next step:
unavailable/disallowed spec -> "choose a Sandbox Spec available to
your organization"; missing org default -> "ask an org admin to
update its default Sandbox Spec". The hidden-picker message ("Sandbox
Spec selection is not enabled...") stays as a plain feature message.
- Trimmed the module docstring and the resolve_existing_sandbox_spec
override comment down to a line or two each.
Co-authored-by: openhands <openhands@all-hands.dev>
Review posted for
|
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
This review covers what has changed since my review of 730df57.
Resolved
- Integration-started conversations now get the org policy when the integration resolves an org.
ResolverUserContextusesresolver_org_id, and a test covers it. I resolved that thread. One case is still open; see the new finding below. - The branch merges cleanly with
mainagain, and the PR body now describes the full API and config surface.
Still open (blocking): live evidence for the Settings UI change. The Default Sandbox Spec dropdown in frontend/src/routes/app-settings.tsx now shows display_name instead of the raw ID. It also keeps a saved ID that is no longer in the catalog. The repository's PR template asks for video or screenshots for UI changes. The body describes a browser run, but no capture is attached. Please attach a screenshot or recording of Settings → Application in the running app. It should show the dropdown with friendly names, and that saving stores the stable spec ID. Unit tests and the mock handlers do not replace this.
New
- Blocking: integration runs on an unclaimed repository, or with no repository, still skip the policy. Details are inline on
integrations/resolver_context.py. - Non-blocking:
docs/sandbox-template-selector.mdcontradicts the current grouping behavior. Details are inline.
Checks. All 25 GitHub checks for bf6ae12 passed (one was skipped). Locally, the 7 focused suites (test_sandbox_template_policy, test_dynamic_remote_sandbox_spec_service, test_remote_sandbox_service, test_live_status_app_conversation_service, test_sandbox_org_secret_binding, test_docker_sandbox_service, test_sandbox_spec_service) passed against PostgreSQL: 834 passed, 1 skipped. The PR is still a draft.
🔄 CHANGES REQUESTED
LLM profile: unknown · Model: openhands/claude-opus-5-5
…policy ResolverUserContext.get_sandbox_template_policy() only consulted resolver_org_id, which is never set for no-repository and unclaimed-repository integration launches. That silently went unscoped while get_credential_org_id() (which already decides sandbox ownership for the same launches) falls back to the authenticated effective org. Reuse get_credential_org_id() for the policy lookup so ownership and policy agree on the same org for every routing path, keeping explicit resolver-org precedence and the unscoped fallback when there truly is no org. Extend the real DB-backed resolver routing fixture (test_resolver_creation_and_credential_lookup) to assert the enforced policy matches the stored credential org for every source/auth-scope/ backend combination, parametrize test_integration_resolver_enforces_org_policy_when_starting_sandboxes for explicit-resolver-org vs authenticated-org-fallback routing, and keep one small still-unscoped-when-no-org-at-all unit test. Correct docs/sandbox-template-selector.md: note the Runtime API #773 dependency/migration ordering before enabling the picker, and fix the grouping description (explicit choice starts fresh; default launches can still reuse an eligible same-user, same-org sandbox, not none at all). Co-authored-by: openhands <openhands@all-hands.dev>
|
✅ PR Artifacts Cleaned Up The |
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
This review covers what has changed since my review of bf6ae12. All three findings from that review are resolved, and I resolved both inline threads.
- Integration runs with an unclaimed repository or no repository: fixed.
ResolverUserContext.get_sandbox_template_policynow looks up the policy withget_credential_org_id(self). That is the same org that ends up owning the sandbox:resolver_org_idwhen the integration sets it, otherwise the authenticated org. The new tests cover all three cases: an explicit resolver org, the fallback to the authenticated org (including the DB-backed check intest_resolver_creation_and_credential_lookup), and no org at all, which stays unscoped.ResolverUserContextdoes not defineget_effective_org_iditself, so the lookup uses thesaas_user_authfallback as intended. I found no other side effects. - Docs on grouping: fixed.
docs/sandbox-template-selector.mdnow matches the code. A default launch can join a sandbox owned by the same user and organization if its spec is still allowed. An explicitsandbox_spec_idstarts a new sandbox. - Settings evidence: provided. The PR body now has a screenshot of the PR's Settings → Application screen running in a browser. The Default Sandbox dropdown shows the friendly names. The README next to it records the save sending
default_sandbox_spec_id: "v1_large"(HTTP 200) and reading the same ID back. The backend in that capture is mocked, which is the screenshot methodAGENTS.mddocuments for PR QA. This UI change only affects how the frontend rendersdisplay_nameand which value it saves. The screenshot is pinned to this commit, so the link keeps working after the.pr/directory is removed on approval.
Checks. All GitHub checks for 8d3dd0a passed (cleanup-on-approval was skipped). Locally, the focused sandbox, policy and resolver suites passed against PostgreSQL, now including tests/unit/integrations/test_resolver_context.py: 857 passed, 1 skipped.
The PR is still a draft. That is for the author to change and is not a code finding.
✅ APPROVED
HUMAN:
AGENT:
Why
An image reference cannot distinguish Sandbox Specs that share an image and have different resource settings. Operators need to make those environments available as separate choices.
Summary
sandbox_spec_idonPOST /api/v1/app-conversations. Serve available choices throughGET /api/v1/sandbox-specs/selection. The default-off flag controls the picker. The configured allowlist stays in effect when the picker is hidden.warm_runtime_config_namewith the existing catalog-derived image field for Runtime API compatibility. Operators define image and resource settings in the provider configuration before launch.Example and scope
An organization can offer a larger Monorepo Sandbox Spec alongside its smaller default. An operator configures each spec with an image and CPU and memory requests and limits. Users can select the monorepo spec for a conversation while smaller jobs continue to use the default. Two specs can also share an image and use different resource limits.
This change enables selection from the configured catalog. Automatic assignment by repository or workload and per-spec disk sizing are separate follow-ups. Initial rollout is Helm only.
Rollout and compatibility
Upgrade Runtime API #773 and apply its migration before deploying this Enterprise change. Older Runtime API versions accept the request field but do not apply named Sandbox Spec resource settings. Deploy Helm #1459 for the operator configuration and Canvas #18789 for the picker. Set each organization’s allowed IDs and default through
SANDBOX_TEMPLATE_ORG_CONFIG, then enable the picker withENABLE_SANDBOX_SELECTOR.Before enabling a named-spec policy, review stored image-based preferences and replace them with the intended allowed named IDs. An image can match several specs. Existing sandboxes retain their environment.
Enterprise #1448 lets an Agent Profile reference a spec through
sandbox.spec_id. The combined launch path must check the profile reference against the same organization policy as other choices. PRD-197 tracks how to report a stale saved profile reference.How to Test
Validation
The integration-policy follow-up is
8d3dd0a4b2f315d4617d00cb635283ff7e950ecb. Its focused backend run on local PostgreSQL reported 844 passed, 1 skipped. Regression coverage includes integration starts with no repository organization and runs with an explicit resolver organization.01d2131b23c0384a5d4ddb9fd44fc6ce46fcc1a1and SDK 1.63.1.Settings evidence
A browser ran Settings → Application with synthetic data. The picker displayed Large (8 GB). Saving sent
default_sandbox_spec_id: "v1_large", received HTTP 200, and read back the same ID.Save and read-back evidence
Issue Number
OHE-3693
Type
Enterprise server image for this PR: