Skip to content

feat: select predefined Sandbox Specs with organization policy (OHE-3693) - #948

Draft
rajshah4 wants to merge 19 commits into
mainfrom
codex/ohe-3693-named-sandbox-templates
Draft

rajshah4 wants to merge 19 commits into
mainfrom
codex/ohe-3693-named-sandbox-templates

Conversation

@rajshah4

@rajshah4 rajshah4 commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

  • A human has tested these changes.

AGENT:

Why

An image reference cannot distinguish Sandbox Specs that share an image and have different resource settings. Operators need to make those environments available as separate choices.

Summary

  • Use the name of a predefined Runtime API configuration as the stable Sandbox Spec ID. Show the friendly display name in Settings.
  • Use the effective organization’s allowlist and default for new sandbox launches, including integration runs with an unclaimed repository or no repository. Policy also applies with the picker hidden. An allowed explicit choice starts a new sandbox. With grouping enabled, a default launch can reuse a sandbox owned by the same user and organization if its spec is still allowed.
  • Accept an optional sandbox_spec_id on POST /api/v1/app-conversations. Serve available choices through GET /api/v1/sandbox-specs/selection. The default-off flag controls the picker. The configured allowlist stays in effect when the picker is hidden.
  • Forward the selected name as warm_runtime_config_name with the existing catalog-derived image field for Runtime API compatibility. Operators define image and resource settings in the provider configuration before launch.
  • Preserve metadata lookup for existing sandboxes if their spec disappears or the organization default becomes unavailable. Report the actual custom image in SDK compatibility errors when it is known.

Example and scope

An organization can offer a larger Monorepo Sandbox Spec alongside its smaller default. An operator configures each spec with an image and CPU and memory requests and limits. Users can select the monorepo spec for a conversation while smaller jobs continue to use the default. Two specs can also share an image and use different resource limits.

This change enables selection from the configured catalog. Automatic assignment by repository or workload and per-spec disk sizing are separate follow-ups. Initial rollout is Helm only.

Rollout and compatibility

Upgrade Runtime API #773 and apply its migration before deploying this Enterprise change. Older Runtime API versions accept the request field but do not apply named Sandbox Spec resource settings. Deploy Helm #1459 for the operator configuration and Canvas #18789 for the picker. Set each organization’s allowed IDs and default through SANDBOX_TEMPLATE_ORG_CONFIG, then enable the picker with ENABLE_SANDBOX_SELECTOR.

Before enabling a named-spec policy, review stored image-based preferences and replace them with the intended allowed named IDs. An image can match several specs. Existing sandboxes retain their environment.

Enterprise #1448 lets an Agent Profile reference a spec through sandbox.spec_id. The combined launch path must check the profile reference against the same organization policy as other choices. PRD-197 tracks how to report a stale saved profile reference.

How to Test

  1. Configure two named Sandbox Specs that share an image and have different resource limits.
  2. Allow both for one organization. Select the larger spec and verify that Runtime API uses the selected configuration.
  3. Hide the picker and verify the same policy applies to default launches and integration-started conversations.
  4. Use Application Settings to choose a friendly name and verify the saved value is the stable ID.
  5. With grouping enabled, verify a default launch can reuse an eligible same-organization sandbox. Remove its spec from the allowlist and verify a new conversation does not group into it. Try selecting the removed spec explicitly and confirm that the error explains how to choose an available spec.

Validation

The integration-policy follow-up is 8d3dd0a4b2f315d4617d00cb635283ff7e950ecb. Its focused backend run on local PostgreSQL reported 844 passed, 1 skipped. Regression coverage includes integration starts with no repository organization and runs with an explicit resolver organization.

  • The prior EKS provisioning run used Enterprise application commit 01d2131b23c0384a5d4ddb9fd44fc6ce46fcc1a1 and SDK 1.63.1.
  • A local test-only integration with feat: add organization default Agent Profiles with Sandbox Spec references (OHE-4186) #1448 passed 63 tests, including explicit Sandbox Spec selection and a revoked saved profile reference with the picker visible and hidden.
  • The frontend source is identical to the revision that passed 18 Settings tests with one skip and a production build.
  • Runtime API fix(budgets): call a disabled per-user budget Unlimited (OHE-3214) #773 passed 28 focused tests for named configuration and warm claims. On EKS, both live V1 launches reached READY with SDK 1.63.1. The explicit spec used the Go agent-server image at 2 CPU and 8 GiB, and the default used stock Python at 1 CPU and 3 GiB. The running image digests and cgroup limits matched both configurations. The disposable release used runc.

Settings evidence

A browser ran Settings → Application with synthetic data. The picker displayed Large (8 GB). Saving sent default_sandbox_spec_id: "v1_large", received HTTP 200, and read back the same ID.

Default Sandbox Spec picker

Save and read-back evidence

Issue Number

OHE-3693

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Enterprise server image for this PR:

ghcr.io/openhands/enterprise-server:sha-e4db3b7

@github-actions github-actions Bot added the type: feat A new feature label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations
  resolver_context.py
  openhands/app_server
  config.py 600-601
  openhands/app_server/app_conversation
  app_conversation_models.py
  live_status_app_conversation_service.py
  openhands/app_server/sandbox
  dynamic_remote_sandbox_spec_service.py
  remote_sandbox_service.py
  sandbox_spec_models.py
  sandbox_spec_router.py
  sandbox_spec_service.py 50, 64, 70
  sandbox_template_policy.py 93, 103
  openhands/app_server/user
  auth_user_context.py 241
  user_context.py 138
  server/services
  sandbox_template_policy.py
Project Total  

This report was generated by python-coverage-comment-action

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope. This change belongs in this repo. OHE-3693 (In Progress, Medium) asks for named templates, Enterprise org policy and the conversation API. This is the first review from this account, and the PR has no earlier review threads.

What checks out. Templates are now named by their durable name, with the image kept separately. Old image-based IDs still resolve, preferring the operator default. The fix that uses the resolved image for SDK-mismatch guidance looks right. A bad SANDBOX_TEMPLATE_ORG_CONFIG now stops startup. Explicit sandbox_spec_id choices are checked server-side against the authenticated org. At 730df57, 19 CI checks passed and 1 was skipped. Locally, the focused suites passed against PostgreSQL: 535 tests across test_sandbox_template_policy, test_dynamic_remote_sandbox_spec_service, test_remote_sandbox_service, test_live_status_app_conversation_service, test_sandbox_spec_service and test_docker_sandbox_service.

Blocking: the org policy is not applied to integration-started conversations (inline). The policy only reaches get_sandbox_spec_service through AuthUserContext. GitHub/GitLab/Slack/Jira/Bitbucket/Azure DevOps conversations start under ResolverUserContext, which keeps the new base method that returns None. Those launches get the unscoped spec service: they can use a saved default that is outside the org's allowlist, they fall back to the global operator default rather than the org's default_template_id, and grouped-sandbox reuse stays on. I checked this directly. With a policy set for org X, get_org_template_policy(X) returns a policy, but ResolverUserContext(..., resolver_org_id=X).get_sandbox_template_policy() returns None. OHE-3693 and docs/sandbox-template-selector.md both promise that a configured policy stays enforced even when the picker is hidden.

Merge readiness (not code defects):

  • GitHub reports the branch as dirty. A trial merge with main (fbb9addcd) conflicts in live_status_app_conversation_service.py, auth_user_context.py, user_context.py and test_remote_sandbox_service.py. All four are on the paths this review covers, so the resolved code needs another look.
  • The PR body says the branch has "only the template feature", but the branch also adds the org policy (SANDBOX_TEMPLATE_ORG_CONFIG), the ENABLE_SANDBOX_SELECTOR flag, GET /api/v1/sandbox-specs/selection and sandbox_spec_id on POST /api/v1/app-conversations. Please update the description so human reviewers see the full API and config surface.
  • The PR template asks for screenshots of the Settings label change. The body says this was verified, but the capture is only in a local bundle and is not attached.
  • The PR is still a draft.

🔄 CHANGES REQUESTED

openhands-agent and others added 7 commits November 20, 2026 19:12
…nish-948

# Conflicts:
#	frontend/src/routes/app-settings.tsx
#	integrations/resolver_context.py
#	openhands/app_server/app_conversation/live_status_app_conversation_service.py
#	openhands/app_server/user/auth_user_context.py
#	openhands/app_server/user/user_context.py
#	tests/unit/app_server/test_remote_sandbox_service.py
…id contract

Co-authored-by: openhands <openhands@all-hands.dev>
ScopedSandboxSpecService.allow_sandbox_reuse unconditionally disabled
default grouping for any org with a template policy, even when the
picker is hidden and the user never made an explicit choice. Re-enable
default grouping for policy-scoped orgs, but add
strict_spec_eligibility so a same-org running sandbox is only reused
when its spec is still in the org's allowed template_ids; unlike the
legacy best-effort preference for unscoped orgs (which falls back to a
stale-image sandbox when nothing current is left), a revoked spec is
never reintroduced as a last resort for policy-scoped orgs.

An explicit sandbox_spec_id on the request still bypasses grouping
entirely, as before. Attaching directly to an existing sandbox_id
keeps its own ownership/credential_org_id check, unaffected by this
change.

Co-authored-by: openhands <openhands@all-hands.dev>
…y custom sandbox

A failed agent-server create on an existing sandbox read
`sandbox_spec.image`, where `sandbox_spec` came from
`resolve_existing_sandbox_spec`. That method's docstring says only
`working_dir` is safe to read from it: when the sandbox's own spec was
removed from the catalog, it returns the *current default* spec as a
best-effort stand-in. Reading `.image` off that stand-in silently
reported someone else's image as "the actual image" on the sandbox --
masking the openhands-sdk mismatch hint when the fallback resolved to
the current bundled image, and attributing an unrelated image to the
sandbox otherwise.

Resolve the sandbox's own spec explicitly (`get_sandbox_spec`) for this
hint instead. When it is still in the catalog, trust its image. When it
is gone, only a legacy image-backed `sandbox_spec_id` (pre-dating named
specs, where the id literally is the image) is still a reliable image
reference; a removed *named* spec's original image is unknown, so
neither the name nor the fallback default's image is reported.

Co-authored-by: openhands <openhands@all-hands.dev>
…g default; simplify grouping/error-hint per review

Finding #3: ScopedSandboxSpecService.resolve_existing_sandbox_spec
inherited the base fallback to get_default_sandbox_spec(), which here
is policy-gated and raises when the org's configured default template
is itself unavailable. That gate exists to fail closed on *new*
launches; it must not also block reading working_dir for a sandbox
that already exists (e.g. on archive/resume). Override it to fall back
to the delegate's own, unwrapped default instead, so a
missing/misconfigured org default can't break access to sandboxes that
started before the misconfiguration.

Review follow-ups on the prior commits in this branch:
- allow_sandbox_reuse (base and ScopedSandboxSpecService) always
  returned True; remove the property and the caller branch that
  checked it.
- _filter_sandboxes_with_known_spec and
  _filter_sandboxes_with_allowed_spec had identical
  memoize/iterate/exception-handling logic, differing only in which
  resolver method they called. Merge into
  _filter_sandboxes_with_eligible_spec, choosing get_start_sandbox_spec
  vs get_sandbox_spec once based on strict_spec_eligibility; the
  strict-vs-best-effort fallback behavior at the call site is
  unchanged.
- The conversation-create error hint no longer makes a second catalog
  call (which could itself fail and replace the real create error).
  It reuses the sandbox_spec already resolved above, trusting
  sandbox_spec.image only when sandbox_spec.id matches the sandbox's
  own sandbox_spec_id (i.e. resolve_existing_sandbox_spec found the
  real spec rather than substituting its fallback default); otherwise
  it falls back to the stored id only when that id is itself a
  recognized legacy image reference. Narrowed
  looks_like_image_reference's docstring to note it only recognizes
  the agent-server's own <repo-path>:<tag> shape, not image references
  in general (a bare name or name:tag is also a valid image reference).

Co-authored-by: openhands <openhands@all-hands.dev>
… the create-failure hint

Per independent PostgreSQL test run and Opus follow-up:

- test_start_app_conversation_reports_agent_server_error_body failed
  with "Mock object has no attribute 'id'": pydantic model fields
  aren't in dir(SandboxSpecInfo), so Mock(spec=SandboxSpecInfo) rejects
  access to .id even though the field exists on real instances. Give
  the shared _arrange_start_app_conversation fixture a real
  SandboxSpecInfo (id matching the sandbox's own sandbox_spec_id, the
  common non-fallback case) instead of a bare Mock. No production
  getattr/defensive change; the attribute is always present on a real
  SandboxSpecInfo.

- The create-failure image hint now also trusts sandbox_spec.image
  when it's an exact alias match for the stored sandbox_spec_id (a
  catalog can resolve a legacy short image name to a spec whose
  canonical .id differs), not only when .id matches outright. No extra
  lookup. Added an 'alias' case to the parametrized regression test.

- Reworded the three user-facing SandboxError messages to say
  "Sandbox Spec" consistently and give a concrete next step:
  unavailable/disallowed spec -> "choose a Sandbox Spec available to
  your organization"; missing org default -> "ask an org admin to
  update its default Sandbox Spec". The hidden-picker message ("Sandbox
  Spec selection is not enabled...") stays as a plain feature message.

- Trimmed the module docstring and the resolve_existing_sandbox_spec
  override comment down to a line or two each.

Co-authored-by: openhands <openhands@all-hands.dev>
@rajshah4 rajshah4 changed the title feat: select named sandbox images and sizes (OHE-3693) feat: select predefined Sandbox Specs with organization policy (OHE-3693) Nov 21, 2026
@all-hands-bot

all-hands-bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Review posted for 8d3dd0a4b2f3

Verdict: ✅ APPROVED. See the review.

This is an automated status.

@all-hands-bot all-hands-bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

This review covers what has changed since my review of 730df57.

Resolved

  • Integration-started conversations now get the org policy when the integration resolves an org. ResolverUserContext uses resolver_org_id, and a test covers it. I resolved that thread. One case is still open; see the new finding below.
  • The branch merges cleanly with main again, and the PR body now describes the full API and config surface.

Still open (blocking): live evidence for the Settings UI change. The Default Sandbox Spec dropdown in frontend/src/routes/app-settings.tsx now shows display_name instead of the raw ID. It also keeps a saved ID that is no longer in the catalog. The repository's PR template asks for video or screenshots for UI changes. The body describes a browser run, but no capture is attached. Please attach a screenshot or recording of Settings → Application in the running app. It should show the dropdown with friendly names, and that saving stores the stable spec ID. Unit tests and the mock handlers do not replace this.

New

  • Blocking: integration runs on an unclaimed repository, or with no repository, still skip the policy. Details are inline on integrations/resolver_context.py.
  • Non-blocking: docs/sandbox-template-selector.md contradicts the current grouping behavior. Details are inline.

Checks. All 25 GitHub checks for bf6ae12 passed (one was skipped). Locally, the 7 focused suites (test_sandbox_template_policy, test_dynamic_remote_sandbox_spec_service, test_remote_sandbox_service, test_live_status_app_conversation_service, test_sandbox_org_secret_binding, test_docker_sandbox_service, test_sandbox_spec_service) passed against PostgreSQL: 834 passed, 1 skipped. The PR is still a draft.

🔄 CHANGES REQUESTED

LLM profile: unknown · Model: openhands/claude-opus-5-5

Comment thread integrations/resolver_context.py Resolved
Comment thread docs/sandbox-template-selector.md Resolved
openhands-agent and others added 2 commits November 21, 2026 12:52
…policy

ResolverUserContext.get_sandbox_template_policy() only consulted
resolver_org_id, which is never set for no-repository and
unclaimed-repository integration launches. That silently went
unscoped while get_credential_org_id() (which already decides sandbox
ownership for the same launches) falls back to the authenticated
effective org. Reuse get_credential_org_id() for the policy lookup so
ownership and policy agree on the same org for every routing path,
keeping explicit resolver-org precedence and the unscoped fallback
when there truly is no org.

Extend the real DB-backed resolver routing fixture
(test_resolver_creation_and_credential_lookup) to assert the enforced
policy matches the stored credential org for every source/auth-scope/
backend combination, parametrize
test_integration_resolver_enforces_org_policy_when_starting_sandboxes
for explicit-resolver-org vs authenticated-org-fallback routing, and
keep one small still-unscoped-when-no-org-at-all unit test.

Correct docs/sandbox-template-selector.md: note the Runtime API #773
dependency/migration ordering before enabling the picker, and fix the
grouping description (explicit choice starts fresh; default launches
can still reuse an eligible same-user, same-org sandbox, not none at
all).

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

✅ PR Artifacts Cleaned Up

The .pr/ directory has been automatically removed.

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

This review covers what has changed since my review of bf6ae12. All three findings from that review are resolved, and I resolved both inline threads.

  • Integration runs with an unclaimed repository or no repository: fixed. ResolverUserContext.get_sandbox_template_policy now looks up the policy with get_credential_org_id(self). That is the same org that ends up owning the sandbox: resolver_org_id when the integration sets it, otherwise the authenticated org. The new tests cover all three cases: an explicit resolver org, the fallback to the authenticated org (including the DB-backed check in test_resolver_creation_and_credential_lookup), and no org at all, which stays unscoped. ResolverUserContext does not define get_effective_org_id itself, so the lookup uses the saas_user_auth fallback as intended. I found no other side effects.
  • Docs on grouping: fixed. docs/sandbox-template-selector.md now matches the code. A default launch can join a sandbox owned by the same user and organization if its spec is still allowed. An explicit sandbox_spec_id starts a new sandbox.
  • Settings evidence: provided. The PR body now has a screenshot of the PR's Settings → Application screen running in a browser. The Default Sandbox dropdown shows the friendly names. The README next to it records the save sending default_sandbox_spec_id: "v1_large" (HTTP 200) and reading the same ID back. The backend in that capture is mocked, which is the screenshot method AGENTS.md documents for PR QA. This UI change only affects how the frontend renders display_name and which value it saves. The screenshot is pinned to this commit, so the link keeps working after the .pr/ directory is removed on approval.

Checks. All GitHub checks for 8d3dd0a passed (cleanup-on-approval was skipped). Locally, the focused sandbox, policy and resolver suites passed against PostgreSQL, now including tests/unit/integrations/test_resolver_context.py: 857 passed, 1 skipped.

The PR is still a draft. That is for the author to change and is not a code finding.

✅ APPROVED

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants