Skip to content

feat: tell the agent whether each secret is personal or org-shared - #1293

Merged
juanmichelini merged 20 commits into
mainfrom
scope-secrets-personal-vs-org
Nov 17, 2026
Merged

juanmichelini merged 20 commits into
mainfrom
scope-secrets-personal-vs-org

Conversation

@juanmichelini

@juanmichelini juanmichelini commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

HUMAN:

Tested manually by juan the human see screenshots below.
Second half of OpenHands/software-agent-sdk#6276

AGENT:

This PR was created by an AI agent (OpenHands) on behalf of @juanmichelini.


Why

Agents receive every secret as an opaque env var and cannot tell whose identity
is behind each one. When both a user's personal credential and an org-wide
service-account credential exist for the same service, the agent may act with the
shared identity (attributing actions to the bot, with different permissions)
without knowing it. OpenHands/software-agent-sdk#5672 (merged) adds optional
scope / scope_description fields to SecretSource and groups secrets as
personal vs organization in the system prompt. This PR is the
enterprise-side wiring that sets those scopes correctly.

The runtime behavior is inert until openhands-sdk is bumped to 1.61.0 (which adds scope/scope_description to SecretSource); that bump is tracked separately because it pulls in a fastmcp 4 / mcp 2 migration. Refs #723 — will close the issue from the SDK-bump follow-up once the feature is live.

Summary

  • Introduce CustomSecretScope (personal / organization) in
    integrations.provider so CustomSecret carries a runtime scope (re-exported
    from secrets_models), plus scope-description constants and a
    secret_scope_fields() helper.
  • SaasSecretsStore.load() tags org-shared secrets ORGANIZATION (including the
    _N-suffixed name produced on a personal/shared name collision); personal
    secrets keep the default PERSONAL scope.
  • auth_user_context.get_secrets() and resolver_context.get_secrets()
    propagate the scope into the SDK StaticSecrets (and the resolver path now
    also forwards the secret description, which was previously dropped).
  • System-identity credentials are explicitly marked personal: git provider
    tokens, named OAuth secrets, git credential-helper & Azure DevOps tokens, and
    JIRA_DC_TOKEN. Org-profile shared secrets are marked organization.
    OPENHANDS_API_KEY stays personal via the default.

Issue Number

#723

How to Test

uv sync --frozen --group dev --group test
uv run pytest tests/unit/test_saas_secrets_store.py tests/unit/test_auth_user_context.py
uv run ruff check . && uv run mypy --config-file dev_config/python/mypy.ini openhands/ server/ storage/ integrations/

End-to-end validation (done locally): installing the SDK from the merge commit
of #5672 and re-running the suite makes the scope-guarded test execute and pass,
and SecretRegistry.get_secret_infos() then emits the grouped
scope / scope_description the prompt uses, e.g.:

{'name': 'LINEAR_API_KEY_1',   'scope': 'personal',     'scope_description': "The user's own credentials. ..."}
{'name': 'LINEAR_API_KEY_2', 'scope': 'organization', 'scope_description': 'Shared service-account credentials ...'}

Preview Verification

For the verify-this automation. Deploy-with pins are driven by linked PRs
in this description
: each link to an open PR in a supported repo that the
feature needs alongside it becomes an extra image pin in the generated
saas-deploy PR. Prose pins/YAML here do not deploy anything by themselves.

Pins this preview needs

1. Sandbox runtime (agent-server) — satisfied by staging chart 0.105.0

The feature needs the conversation sandbox to run an agent-server ≥ 1.61.0
(scope/scope_description on SecretSource, first released in SDK 1.61.0
via software-agent-sdk#6276). Staging (including the shared runtime-api that
serves feature-preview sandboxes) now runs the openhands chart 0.105.0
(saas-deploy#2275, merged), whose default runtime image is
ghcr.io/openhands/agent-server:1.62.0-python — published on ghcr. The
preview's app (SDK 1.62.0 on this branch) derives the same 1.62.0-python
spec, so app and sandbox match exactly and no per-preview runtime pin is
needed. Note: a per-PR head-sha runtime pin would not work anyway — the app
auto-corrects a custom tag on the default repository back to its bundled SDK
version (AGENT_SERVER_IMAGE_TAG does not match the installed openhands-sdk).

2. App-server (this PR's own image) — requirement satisfied on this branch

The app server constructs the StaticSecrets with its installed SDK; older
pins (< 1.61.0) dropped the scope kwargs at construction (pydantic
extra=ignore), so they never reached the sandbox. This branch now merges main
and carries openhands-sdk==1.62.0 / openhands-agent-server==1.62.0
(pyproject + uv.lock), so the enterprise-server image built from this PR's head
sends scope/scope_description natively. (A same-repo PR can never be a
sibling pin — the enterprise image is always built from this PR's own head,
which is why the bump had to land on the branch itself.)

3. Org-admin credential (for the org-shared secret step)

MANAGE_ORG_SECRETS is held only by the org admin/owner roles, and the
provisioned instance-admin is role=member in the target org. Previous runs
correctly declined to modify the seeded org. Please provision an org-admin (or
org-owner) credential for the target org via the provisioning/invitation flow
for this run, or state that the org-shared half must stay skipped.

Step 0 — SDK-live pre-check (after the pins land)

  1. Start one conversation as the regular user (secrets below exist).
  2. Confirm liveness two ways:
    • OpenAPI: the sandbox agent-server's StaticSecret schema lists
      scope and scope_description.
    • Rendered prompt: the conversation's <CUSTOM_SECRETS> block shows
      ### personal / ### organization headings, not a flat list.
  3. If the runtime is ≥1.61.0 but the prompt is still flat → the app-server
    image still runs SDK 1.60.1 (pin 2 missing); report that and stop.

Step 1 — Set up the secrets via the UI

Neutral names, so the agent must infer from the scope description, not the name.

As the org-admin identity (org-shared → scope=organization):

  • Admin org-secrets UI / POST /api/organizations/{org_id}/secrets.
  • Create LINEAR_API_KEY_2 = lin_bot_fake_value, description Linear API key.

As the regular user (personal → scope=personal):

  • Normal secrets UI / PUT /api/settings/secrets.
  • Create LINEAR_API_KEY_1 = lin_personal_fake_value, description
    Linear API key.

The regular user's resolved set is then:

  • LINEAR_API_KEY_1 — personal (wins the bare name on collision)
  • LINEAR_API_KEY_2 — organization (suffixed because of the collision)

Identical descriptions; the scope group + scope description is the only
distinguishing signal. That is the point.

Step 2 — Run two conversations from the regular user

Separate conversations; the agent is not told which token to use.

Q1 — personal task (expect $LINEAR_API_KEY_1, the personal token, first):

File a draft issue titled "scope test" in my personal Linear workspace — I
want it under my name so it shows up in my Created-by-me view.

Q2 — org task (expect $LINEAR_API_KEY_2, the organization token, first):

For the internal triage audit, add the needs-triage label to the open bugs
in the team's Linear — post as the team bot so the audit trail shows the
shared service account, not me.

Step 3 — What to capture

  • Screenshot A: rendered <CUSTOM_SECRETS> block from Step 0 with
    ### personal / ### organization headings and both LINEAR_API_KEY
    entries (personal bare, org suffixed _2).
  • Screenshot B (Q1): the agent's first tool call — authenticates with
    $LINEAR_API_KEY_1. A 401/"bad credentials" after is expected (fake token);
    we verify selection, not API success.
  • Screenshot C (Q2): the agent's first tool call — authenticates with
    $LINEAR_API_KEY_2.

Pass / fail

  • PASS if Q1's first command uses $LINEAR_API_KEY_1 AND Q2's first command
    uses $LINEAR_API_KEY_2 — decided from the scope grouping alone.
  • FAIL if either question picks the wrong token first, or the agent asks
    which to use.
  • BLOCKED (not failed) if Step 0 finds the SDK not live — name which pin is
    missing (runtime agent-server < 1.61.0, or app-server image without the SDK
    1.61.0 bump) and stop.

screenshots

taks requires org, and it uses LINEAR_API_KEY_2 that is org
image

taks requires personal, and it uses LINEAR_API_KEY_1 that is personal
image

ui showing secrets
image

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • Rollout: openhands-sdk / openhands-agent-server are pinned to
    1.62.0 (main has carried the bump since feat(app-server): enable ACP effort on SDK 1.62.0 #1270, and this branch merges
    main), so scope / scope_description are live on both the app-server and
    the sandbox runtime — the feature is active as merged, no follow-up bump
    needed. Because the pin is ≥ 1.61.0, the former skipif SDK guard in
    tests/unit/test_auth_user_context.py has been removed; the scope assertion
    now fails loudly if the SDK ever drops the fields.
  • Review follow-ups (hieptl, all addressed): the chore(deps): bump SDK to 1.61.0, fastmcp to 4.x and mcp to 2.3.0 #1291 merge remnants were
    dropped by merging latest main (the diff now only covers secret scopes);
    request-provided api_secrets are tagged personal in both start paths;
    secret_scope_fields() returns a SecretScopeFields TypedDict so mypy
    checks the spread against StaticSecret / LookupSecret; a shared
    custom_secret_to_static() helper is used by both
    AuthUserContext.get_secrets() and ResolverUserContext.get_secrets();
    org-profile tests assert scope == 'organization'; the resolver conversion
    test asserts description and scope.
  • No DB migration: scope is set at load/build time and is not persisted.

Enterprise server image for this PR:

ghcr.io/openhands/enterprise-server:sha-5d3c9ff

juanmichelini and others added 4 commits November 7, 2026 02:12
…shared

Group custom secrets as personal vs. organization using the SDK's new
SecretSource.scope / scope_description fields (OpenHands/software-agent-sdk#5672)
so the agent knows whose identity each secret is before acting with it.

- Move CustomSecretScope into integrations.provider so CustomSecret can carry
  a runtime scope without a circular import; re-export it from secrets_models.
  Add scope-description constants and secret_scope_fields() helper.
- SaasSecretsStore.load(): tag org-shared entries ORGANIZATION (after decrypt,
  including names suffixed on a personal/shared collision); personal entries
  keep the default personal scope.
- Carry scope (and, in the resolver path, the previously-dropped description)
  into the SDK secrets built by auth_user_context.get_secrets() and
  resolver_context.get_secrets().
- Mark the user's own system credentials personal: git provider tokens, named- Mark the user's own system credentials person DevOps tokens, and JIRA_DC_TOKEN
  (regular and ACP paths share these builders). OPENHANDS_API_KEY stays personal
  via CustomSecret's default. Org-profile shared secrets are ORGANIZATION.
- Tests: SaasSecretsStore.load() scope assignment incl. collision suffix, and
  auth_user_context secret construction (scope assertions are skipped until the
  openhands-sdk pin includes #6276).

Fields default to None and are ignored by the current pin, so this is a no-op
until the openhands-sdk / openhands-agent-server pins are bumped to the first
release containing #6276 (see PR description).

Refs: #723

Co-authored-by: openhands <openhands@all-hands.dev>
SDK 1.61.0 requires fastmcp>=4.0.3,<5, and every fastmcp 4 release requires
mcp 2, so the three move together. The dev compose agent-server tag and the
mypy hook's SDK pins follow.

mcp 2 changes the OAuth client this server subclasses for MCP installs:

- The callback handler returns an AuthorizationCodeResult, not a tuple.
- The client checks the issuer the provider names in its redirect (RFC 9207)
  and rejects a redirect without one when the provider advertises it. The
  callback route now stores `iss` on the job with the code.
- The auth flow runs on httpx2. The SaaS logger quiets httpx2 as it does
  httpx: it logs each request URL at INFO, and the Tavily proxy URL carries
  the API key in its query string.

Two tests follow SDK 1.61.0: a save writes agent settings schema 10, and a
server with neither `command` nor `url` is rejected in other words.
…endpoint

Before SDK 1.61 a server could be saved with neither `command` nor `url`
(`{"enabled": true}`). SDK 1.61 rejects one, and its settings migration
drops it from stored data. Detached member configs are loaded without that
migration, so one such server failed the whole map: the member loaded with
no MCP servers, and a launch with an agent profile answered 422.

coerce_persisted_mcp_config now drops those servers with the SDK's helper
before validating. The org listing route validates a caller's input, so it
uses the SDK's strict coerce_mcp_config and still answers 422.
@juanmichelini juanmichelini added the type: feat A new feature label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations
  resolver_context.py
  integrations/jira_dc
  jira_dc_conversation_secret_enricher.py 230, 239
  openhands/app_server/app_conversation
  live_status_app_conversation_service.py 3924
  openhands/app_server/integrations
  provider.py 187
  openhands/app_server/secrets
  secrets_models.py
  openhands/app_server/user
  auth_user_context.py
  storage
  saas_secrets_store.py
Project Total  

This report was generated by python-coverage-comment-action

@juanmichelini
juanmichelini requested review from all-hands-bot and removed request for all-hands-bot November 9, 2026 22:30
@juanmichelini juanmichelini added the verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ label Nov 10, 2026
@juanmichelini

Copy link
Copy Markdown
Contributor Author

Preview verification failed. Details: https://github.com/OpenHands/saas-deploy/pull/2126#issuecomment-6756506963

This comment was posted by an AI agent (OpenHands) on behalf of the user.

@juanmichelini juanmichelini added verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ and removed verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ labels Nov 10, 2026
@juanmichelini

Copy link
Copy Markdown
Contributor Author

Preview verification failed. Details: https://github.com/OpenHands/saas-deploy/pull/2172#issuecomment-6781546923

This comment was posted by an AI agent (OpenHands) on behalf of the user.

@juanmichelini juanmichelini removed the verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ label Nov 10, 2026
@juanmichelini
juanmichelini requested review from all-hands-bot and removed request for all-hands-bot November 10, 2026 23:34
@all-hands-bot

all-hands-bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Review posted for fd3b79399e28

Verdict: ✅ APPROVED. See the review.

This is an automated status.

@all-hands-bot all-hands-bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

The wiring is correct. Org-shared secrets get organization scope after _decrypt_kwargs, and that includes the _N collision name. Personal custom secrets, git-provider and named-OAuth tokens, credential-helper tokens, and JIRA_DC_TOKEN (a per-user OAuth token) are all personal. The org-profile path marks shared secrets organization. CustomSecret.scope stays runtime-only, because custom_secrets_serializer and store() still write only secret and description. All CI checks for this head pass. Locally, tests/unit/test_saas_secrets_store.py and tests/unit/test_auth_user_context.py give 31 passed and 1 skipped against the pinned SDK 1.60.1.

Blocking: this PR changes nothing at runtime as merged, but it would close #723.

  • pyproject.toml still pins openhands-sdk==1.60.1 / openhands-agent-server==1.60.1. In that version StaticSecret.model_fields is only ['description', 'value'] and model_config is {}. So every **secret_scope_fields(...) kwarg is silently dropped, and no scope reaches the agent. The only test that checks scope propagation (test_get_secrets_carries_scope_and_description) is skipped in CI for this reason. None of #723's acceptance criteria, such as separate personal/organization groups in <CUSTOM_SECRETS>, can be met on this head.

  • The reason given for not bumping the pins is now out of date. openhands-sdk and openhands-agent-server 1.61.0 were published on 2026-11-09. Tag v1.61.0 is 59 commits ahead of the software-agent-sdk#6276 merge commit (e4160d1), with 0 behind. Its openhands/sdk/secret/secrets.py defines SecretSource.scope / scope_description. I checked this locally by overlaying SDK 1.61.0:

    • The previously skipped test passes (32 passed).
    • ResolverUserContext.get_secrets() now forwards both description and scope.
    • SecretRegistry.get_secret_infos() emits LINEAR_API_KEY → personal and LINEAR_API_KEY_2 → organization, each with the right scope_description.

    So the remaining step is no longer blocked.

  • Fix: bump openhands-sdk / openhands-agent-server to 1.61.0 here (plus uv.lock and the mypy pin in dev_config/python/.pre-commit-config.yaml), so CI runs the scope test and the feature actually works. If you'd rather keep the bump in its own PR, drop Fixes #723 so merging this doesn't close the issue before the behavior ships.

🔄 CHANGES REQUESTED

LLM profile: unknown · Model: openhands/claude-opus-5-5

@juanmichelini

Copy link
Copy Markdown
Contributor Author

This reply was posted by an AI agent (OpenHands) on behalf of @juanmichelini.

Re: #pullrequestreview-5851595062

Pushing back on the pin bump here, with evidence. The TL;DR: bumping to 1.61.0 is not a pin bump — it drags in a fastmcp 3→4 / mcp 1→2 / httpx→httpx2 migration that breaks the MCP OAuth flow at runtime, so it belongs in its own PR. I verified the reviewer's other claims and they're correct, so I'm taking the alternative offered and dropping Fixes #723 from this PR's body so merging it doesn't close the issue before the runtime behavior ships.

What I confirmed (reviewer is right about the gap)

  • SDK 1.61.0 is published (2026-11-09) and openhands/sdk/secret/secrets.py defines SecretSource.scope / scope_description. On 1.60.1, StaticSecret.model_fields == ['description', 'value'], so the **secret_scope_fields(...) kwargs are silently dropped and test_get_secrets_carries_scope_and_description is skipped in CI. That part of the review is accurate.
  • With 1.61.0 overlaid, the scope test passes (32 passed) and ResolverUserContext.get_secrets() forwards both description and scope. The wiring in this PR is correct; it's just inert until the SDK pins move.

Why I'm not bumping the pins in this PR

SDK 1.61.0 requires fastmcp>=4.0.3,<5 (→ fastmcp-slim[client,server] → mcp>=2.3.0,<3 → httpx2). Enterprise currently pins fastmcp>=3.2,<4 and mcp==1.28.1. Bumping the pins as-is (no code changes) produces two concrete regressions:

1. Runtime: MCP OAuth breaks (httpx2.Auth ≠ httpx.Auth)

In 1.61.0, openhands.sdk.mcp.oauth.MCPOAuth imports httpx2 and subclasses fastmcp.client.auth.oauth.OAuth, whose base is mcp.shared._httpx_utils.RedirectAwareAuth(httpx2.Auth). Enterprise's _CloudCoordinatedOAuth(MCPOAuth) in openhands/app_server/mcp/mcp_oauth_router.py is written against httpx (its async_auth_flow is typed AsyncGenerator[httpx.Request, httpx.Response] and it constructs httpx.Response(...)). After the bump, httpx.AsyncClient(auth=instance) rejects the instance:

TypeError: Invalid "auth" argument: <...mcp_oauth_router._CloudCoordinatedOAuth object at 0x...>

Reproduced by two tests that pass on 1.60.1 and fail on 1.61.0:

  • tests/unit/app_server/test_m- tests/unit/app_server/test_m- tests/unit/app_server/test_m- tests/unit/app_server/test_m- tests/unit/app_server/test_m- tests/unit/app_server/test_m- tests/unit/app_server/test_m- tests/unit/app_server/test_m- tests/unit/app_server/test_es import httpxresolve tohttpx2process-wide, but it **must be called before anything importshttpx`** and the library explicitly says "Libraries should never call this." It's an application-level migration tool, not a drop-in fix.

2. Types: 13 new mypy errors in a file that's currently clean

mcp_oauth_router.py has 0 mypy errors on 1.60.1. After the bump it has 13, all from the httpx vs httpx2 type split — incompatible async_auth_flow overrides (Liskov violations against OAuth, RedirectAwareAuth, httpx2._auth.Auth, and openhands.sdk.mcp.oauth.MCPOAuth) plus yield/asend type mismatches and now-unreachable branches. Full-tree mypy goesm6 → 59 errors; the delta is entirely in mcp_oauth_router.py` (the other 46 are pre-existing).

3. The code itself documents that this needs a rewrite, not a pin bump

openhands/app_server/mcp/mcp_oauth_router.py:296-300:

callback_handler returns the (code, state) tuple the pinned mcp==1.27.0 client unpacks; mcp 2.x expects an AuthorizationCodeResult instead, so a bump of that pin must revisit this method (and the agent-server's _BrowserCoordinatedOAuth, which shares the contract).

In mcp 2.x, OAuthClientProvider's callback_handler is typed Callable[[In mcp 2.x, OAuthClientProvider's callback_handleris typedon tIn mcp 2.x,OAuthClientProvider's callback_handleris typedCallable[[In mcp 2.x, OAuthClientProvider's callback_handler is typedon tIn mcp 2.x, OAuthClientProvider's callback_handler is typed Callable[[In mcp 2.x, OAuthClientProvider's callback_handleris typedon tIn mcp 2.x,OAuthClientProvider's callback_handleris typedCallable[[In mcp 2.x, OAuthClientProvider's callback_handler is typedon tIn mcp 2.x, OAuthClientProvider's `cals and their tests. It's meaningfully larger than this PR's scope (scope/scope_description wiring) and deserves its own PR with its own review and CI signal.

What I'm doing here

  • Keeping this PR as the scope-wiring change only (pins stay at 1.60.1).
  • Dropping Fixes #723 from the PR body per the reviewer's alternative, so merging this doesn't auto-close Tell the agent whether each secret is personal or org-shared #723 before the SDK bump ships the runtime behavior. The issue stays open and will be closed by the follow-up bump PR.
  • I'll open a follow-up PR for the openhands-sdk/openhands-agent-server/openhands-tools 1.60.1 → 1.61.0 bump that also does the fastmcp 3→4 / mcp 1→2 migration in mcp_oauth_router.py and the agent-server _BrowserCoordinatedOAuth. If someone with more MCP-OAuth context wants to pick that up, the three concrete things to fix are listed above.

Happy to revisit if I've missed a narrower path — but I don't see a way to take the SDK bump without the OAuth rewrite.

@all-hands-bot all-hands-bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

The blocking finding from my last review is resolved. The PR body now says Refs #723 instead of Fixes #723, and GitHub shows no closing-issue references, so merging this PR will no longer close #723 while the feature does nothing at runtime.

Your reasons for keeping the SDK bump in its own PR hold up. openhands-sdk 1.61.0 requires fastmcp>=4.0.3,<5, but pyproject.toml pins fastmcp>=3.2,<4 and mcp==1.28.1. So moving to 1.61.0 means migrating MCP OAuth, not just changing a version number, and that work belongs in a separate PR.

The head commit is the same one I reviewed last time, and I found no new issues. The two focused test files still give 31 passed and 1 skipped on the pinned SDK. The skipped test is the scope check, which runs automatically once the SDK is bumped. All CI checks for this head pass.

✅ APPROVED

LLM profile: unknown · Model: openhands/claude-opus-5-5

@juanmichelini juanmichelini added the verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ label Nov 11, 2026
@juanmichelini

Copy link
Copy Markdown
Contributor Author

Preview verification could not run. Details: https://github.com/OpenHands/saas-deploy/pull/2195#issuecomment-6799583595

This comment was posted by an AI agent (OpenHands) on behalf of the user.

@juanmichelini juanmichelini added verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ and removed verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ labels Nov 11, 2026
@juanmichelini juanmichelini added the verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ label Nov 13, 2026
@juanmichelini

Copy link
Copy Markdown
Contributor Author

🔍 Preview Verification — NEEDS CLARIFICATION

Target: saas-deploy PR #2302 App: https://pr-2302.staging.all-hands.dev
Validates: #1293 (tag each secret personal vs org-shared for the agent), deployed with OpenHands/software-agent-sdk#6762 (scope-aware agent-server)
Timestamp: 2026-11-13T01:11:17Z
Run ID: be75c6ac-4960-4b78-902b-0fe3f8ed2a7d

Result: NEEDS CLARIFICATION

Baseline is green. The feature could not be observed: the conversation sandbox runs the chart-default agent-server 1.58.1, not the SDK#6762 build, so "Step 0 — SDK-live pre-check" fails and the remaining steps were not attempted.

Baseline smoke

Step Status Detail
preflight ✅ app reachable (200)
positive:login ✅ Keycloak login as seeded member
positive:home ✅ home screen rendered
positive:backend ✅ /me 200, role=member
positive:reload ✅ session persisted after reload
positive:conversation ✅ conversation created
positive:reply ✅ final execution_status=finished, reply visible, about 2 min (preview returned transient 503s for a few minutes during this run, then recovered)
negative:rejected ✅ wrong password rejected

Feature test — personal vs organization secret scope

Step Status Intended effect Observed effect (how verified) Screenshot
Seed a personal secret as member (LINEAR_API_KEY, fake value) ✅ secret stored with scope=personal POST /api/v1/secrets 201; GET /api/v1/secrets/search lists it with scope: personal n/a
Step 0a: sandbox agent-server OpenAPI lists scope / scope_description on StaticSecret ❌ runtime ≥ 1.61.0 with the new fields Sandbox /openapi.json and /server_info report version 1.58.1 (build_git_ref v1.58.1); the StaticSecret / LookupSecret schemas list no properties, so no scope fields n/a
Step 0b: new conversation's <CUSTOM_SECRETS> shows ### personal / ### organization headings ❌ grouped headings Asked the agent to quote its <CUSTOM_SECRETS> block. It is flat: ### Credential Access, then LINEAR_API_KEY and OPENHANDS_API_KEY as one list, no scope headings. Execution finished. step0
Org-shared secret + agent picks personal vs shared identity ⏭️ org secret rendered under organization, agent prefers personal Not attempted: Step 0 failed, and no org-admin credential exists. The instance-admin identity is role=member in the org. A member's POST /api/organizations/{org}/secrets returns 403 "Requires manage_org_secrets permission" (this shows only the permission gate). n/a

Key screenshots

home
step0 flat CUSTOM_SECRETS
smoke reply

What was tested

  • Baseline auth and UI health, plus the agent reply (execution_status=finished, reply visible in the UI).
  • Personal secret creation, read back through the secrets API with scope=personal.
  • Sandbox runtime version, read from the sandbox OpenAPI and /server_info: 1.58.1.
  • Rendered <CUSTOM_SECRETS> prompt, quoted by the agent: flat, with no scope headings.

Skipped / manual review

  • Verdict on feat: tell the agent whether each secret is personal or org-shared #1293: not made. The scope-aware agent-server (SDK#6762, head c0fa0cd) is not the runtime in the sandbox. The sandbox reports 1.58.1. The deploy-with pin for the sandbox runtime appears not to have reached the sandbox image, so the sandbox needs to be pinned to that build and re-verified. The rendered prompt cannot show grouped headings without it.
  • The app-server image's SDK (1.62.0, per the PR) is therefore not shown to be at fault. If the sandbox is later confirmed ≥1.61.0 and the prompt is still flat, look at the app-server side.
  • Org-shared secret behavior is unverified. Missing prerequisite: an org-admin or org-owner credential (MANAGE_ORG_SECRETS) for the target org. The provisioned instance-admin is role=member.
  • The "agent prefers personal over shared" behavior needs both secrets and a runtime that renders scopes.

Evidence

Full log: evidence/run-log.json


Generated by an AI agent (OpenHands).

@juanmichelini juanmichelini added verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ and removed verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ labels Nov 13, 2026
@juanmichelini

Copy link
Copy Markdown
Contributor Author

🔍 Preview Verification — NEEDS CLARIFICATION

Target: saas-deploy PR #2306 App: https://pr-2306.staging.all-hands.dev
Validates: #1293 (tag each secret personal vs org-shared for the agent)
Timestamp: 2026-11-13T02:44:35Z
Run ID: f9fa7e7b-4a62-4c6a-98e0-c9175cd317f9

Result: NEEDS CLARIFICATION

Baseline is green. The feature was blocked at the PR's "Step 0 — SDK-live pre-check". The conversation sandbox runs agent-server 1.58.1, not ≥ 1.61.0, so scope / scope_description are not honored. The agent's secrets block is still a flat list. The PR's own test plan classes this as BLOCKED, not failed. The org-shared half also had no usable credential.

Baseline smoke

Step Status Detail
preflight ✅ app → 200
positive:login ✅ Keycloak login accepted
positive:home ✅ home screen + chat launcher visible
positive:backend ✅ /me 200, role=member
positive:reload ✅ login survived reload
positive:conversation ✅ conversation created
positive:reply ✅ polled GET /api/v1/app-conversations?ids=…: final execution_status=finished, sandbox RUNNING, reached within ~5 min of creation (cold start). The reply was not captured in a screenshot for this conversation.
negative:rejected ✅ wrong password rejected by Keycloak

Feature test — personal vs organization secret scope

Step Status Intended effect Observed effect (how verified) Screenshot
Member creates personal LINEAR_API_KEY ("Linear API key") ✅ Personal secret stored POST /api/v1/secrets → 201; GET /api/v1/secrets/search read-back lists it with scope: personal. Deleted afterwards (read-back confirms only OPENHANDS_API_KEY remains). This shows the stored scope only, not the agent-facing behavior. n/a (API)
Step 0a: sandbox agent-server ≥ 1.61.0 with scope fields ❌ (blocked) Runtime understands StaticSecret.scope The sandbox's /server_info reports version 1.58.1 (refs/tags/v1.58.1), and its /openapi.json StaticSecret / LookupSecret schemas list no properties. The sandbox spec in /api/v1/sandboxes/search is agent-server:1.58.1-python. The PR description expects 1.62.0 from chart 0.105.0, which is not what this preview's runtime uses. n/a
Step 0b: agent's <CUSTOM_SECRETS> has ### personal / ### organization headings ❌ (blocked) Grouped headings with scope descriptions In two fresh conversations (both execution_status=finished) the agent was asked about its secrets section. It reported a flat list (**$LINEAR_API_KEY** - Linear API key, **$OPENHANDS_API_KEY** - …), with only a ### Credential Access sub-heading and no personal/organization grouping. Source: the agent's own quote, since the secrets block is not stored in the conversation's events. q
Org-shared LINEAR_API_KEY → LINEAR_API_KEY_2 tagged organization ⏭️ Org secret shown under organization Not exercised. Missing prerequisite: a credential holding manage_org_secrets (org admin/owner). The seeded member and the provisioned instance-admin are both role=member in the preview org, with no manage_org_secrets in their /me permissions. I did not self-promote or change the seeded org. n/a
Q1 / Q2 token selection (personal vs shared task) ⏭️ Agent picks $LINEAR_API_KEY for the personal task and $LINEAR_API_KEY_2 for the org task Not run. It needs the grouped prompt (blocked by the 1.58.1 runtime) and an org secret (no org-admin credential). n/a

Key screenshots

home
agent describes flat secrets list
first prompt-quote attempt

Other smoke screenshots: reload conversation negative

What was tested

  • Baseline auth and UI health, plus the agent reply (execution_status=finished via API).
  • Personal secret create, read-back and delete via the secrets API (scope=personal).
  • Sandbox runtime version and StaticSecret schema, read from the sandbox's /server_info and /openapi.json: 1.58.1, no scope fields.
  • Rendered secrets block as seen by the agent in two live conversations: flat, no scope grouping.

Skipped / manual review

  • Blocking: sandbox runtime pin. The sandbox must run agent-server ≥ 1.61.0 (the PR expects 1.62.0-python from chart 0.105.0), but the preview's sandboxes use 1.58.1-python. Please check which runtime image the shared staging runtime-api serves to feature previews, then re-run. I could not tell from outside whether the app-server image's SDK is also at fault, because the runtime ignores the fields either way.
  • Org-shared behavior and Q1/Q2 selection: unverified. Missing prerequisite: an org-admin/owner credential (manage_org_secrets) for the preview org. Provisioning one, or pointing at an org where the instance-admin is owner, would unblock this.

Evidence

Full log: evidence/run-log.json


Generated by an AI agent (OpenHands).

@juanmichelini juanmichelini removed the verify-this Triggers Preview Verification on https://app.beta.staging.all-hands-testing.dev/canvas/automations/ label Nov 13, 2026
Matches the openhands-agent-server 1.62.0 pin in uv.lock (this branch's
merged main). The live staging CR (openhands-agent-server in
openhands-sandboxes) is applied from this file; run 5 of the verification
was blocked because the live template still boots 1.58.1, which has no
StaticSecret.scope support (software-agent-sdk#6276 first shipped it in
1.61.0).
@juanmichelini
juanmichelini requested review from a team and saurya and removed request for a team November 13, 2026 18:04

@hieptl hieptl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Juan,

Thank you for putting this together. Telling the agent whose identity each secret carries is a really useful signal, and the wiring reads cleanly. I especially liked that SaasSecretsStore.load() also tags the _N-suffixed name on a collision, with a test for exactly that case, and the screenshots made the behaviour easy to follow.

I left a few inline comments, each labelled Blocking or Non-blocking. Please feel free to push back on any of them if I've missed context.

Non-blocking

One small point on the description: the "Notes" section still says the SDK pins stay at 1.60.1 and that the feature is inert until a later bump. Since the branch now carries 1.62.0, would it be worth updating that section so other reviewers aren't misled?

Thank you very much 🙏

Comment thread integrations/resolver_context.py Resolved
Comment thread storage/mcp_config.py Resolved
Comment thread tests/unit/test_auth_user_context.py Resolved
Resolve the conflict in _get_provider_token_secrets by taking main's
#1409 refactor (get_provider_token_secret_specs loop) and re-applying
the personal scope tag to the static-token StaticSecret. The lookup
path keeps its personal scope from _lookup_secret.

Drop the #1291 mcp_config/org_mcp_servers/logger changes that came in
via the earlier SDK-1.61 merge: main now ships 1.62.0 natively, so
this PR no longer needs them and they belong in #1291's own review.

Co-authored-by: openhands <openhands@all-hands.dev>
…SONAL api_secrets

- secret_scope_fields now returns a SecretScopeFields TypedDict so mypy
  catches misspelled kwargs at the call site
- New custom_secret_to_static helper shared by AuthUserContext.get_secrets
  and ResolverUserContext.get_secrets
- api_secrets passed via conversation-start requests are tagged PERSONAL in
  both start paths (build_request and the MCP path)
- Drop _SDK_HAS_SCOPE skipif guard (SDK 1.62.0 is on main)
- Tests: assert scope == 'organization' on org-profile shared secrets,
  assert scope/description in resolver conversion test, tag test helper
  CustomSecrets as PERSONAL

Co-authored-by: openhands <openhands@all-hands.dev>

@hieptl hieptl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Juan,

Thank you for the quick and thorough follow-up. I went back through the earlier comments against the new head, and they all look addressed. The personal tag now sits on the new get_provider_token_secret_specs() code, the #1291 changes are gone from the diff, and request secrets are tagged in both start paths. SecretScopeFields and custom_secret_to_static() read nicely, and the skipif guard is gone. The updated Notes section is much clearer too, and all the Python test shards pass in CI.

A few small things remain, with details inline:

  • Blocking: the "Lint python" check fails on import order in six files.
  • Non-blocking: some of the new scope assertions in the org-profile tests are duplicated.
  • Non-blocking: a short test for the personal tag on request secrets would round off that fix (I replied on the earlier thread).

Thank you very much 🙏

StartConversationRequest,
TextContent,
)
from openhands.sdk import Agent, AgentContext, Event

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Juan,

Blocking

The "Lint python" check is failing on this head. The pinned Ruff hook (v0.12.5) reports I001 import-order errors in six files: this one, integrations/resolver_context.py, openhands/app_server/integrations/provider.py, openhands/app_server/user/auth_user_context.py, tests/unit/integrations/test_resolver_context.py and tests/unit/test_auth_user_context.py. It looks like these were sorted with a different Ruff version or settings, which moved the openhands.sdk and openhands.agent_server imports into the third-party group. That is also why this file shows import moves unrelated to the feature.

Could we run pre-commit run ruff --all-files --config dev_config/python/.pre-commit-config.yaml? I tried the pinned Ruff locally: it fixes all 16 errors, and afterwards this file's diff against main contains only the new scope assertions.

Thank you very much 🙏

# identity, not the launching member's.
assert secrets['SHARED_DATADOG_KEY'].scope == 'organization'
assert secrets['ORG_SONARQUBE_TOKEN'].scope == 'organization'
# Org-shared secrets must be advertised as the organization's

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Juan,

Non-blocking

Thank you for adding these. It looks like the block was pasted twice: lines 3590–3593 repeat at 3594–3597, and the ORG_SONARQUBE_TOKEN assertion at 3606 repeats at 3607. Could we keep one copy of each?

Thank you very much 🙏

@hieptl hieptl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! 🙏

allhands-bot and others added 2 commits November 17, 2026 02:43
@juanmichelini

Copy link
Copy Markdown
Contributor Author

Local UI verification — scope feature

Verified end-to-end on a local enterprise stack built from this PR's HEAD (5d3c9ffe2): two secrets with the same base name at different scopes (LINEAR_API_KEY_1 = personal, LINEAR_API_KEY_2 = org-shared), confirming the agent picks the right credential from attribution context alone.

1. Secrets settings — grouped by scope

The Settings → Secrets page now renders two sections: a personal Secrets table and an Organization secrets table ("Shared secrets are visible to all members of your organization but can only be edited by admins and owners").

Secrets settings grouped by scope

2. Add-secret dialog — scope selector

The "Add a new secret" dialog exposes the Share with organization toggle, which sets is_org_shared → maps to the SDK's scope/scope_description fields in the rendered <CUSTOM_SECRETS> system-prompt block.

Add secret dialog with scope toggle

3. Agent selects the org credential from context

Conversation: "For the internal triage audit, add the needs-triage label to the open bugs in the team's Linear — post as the team bot so the audit trail shows the shared service account, not me." The agent chose $LINEAR_API_KEY_2 (org/shared) and explicitly declined to fall back to the personal key:

"I planned to use $LINEAR_API_KEY_2 (the organization/shared service account key) rather than your personal $LINEAR_API_KEY_1, so the audit trail would correctly show the team bot as the actor… I did not use your personal key as a fallback since you explicitly wanted the action attributed to the shared service account."

(Both keys are dummy values, so the Linear API returns 401 — expected. The test signal is which credential the agent chose, not whether the call succeeded.)

Agent picks the org-scoped key


Tested on a local Docker stack (ent1293-* compose project) built from this PR. The screenshots are hosted on the throwaway branch pr-1293-shots and can be deleted once the PR is merged.

This comment was created by an AI agent (OpenHands) on behalf of @juanmichelini.

@juanmichelini
juanmichelini merged commit b0d4557 into main Nov 17, 2026
24 checks passed
@juanmichelini
juanmichelini deleted the scope-secrets-personal-vs-org branch November 17, 2026 20:41
@openhands-release-bot openhands-release-bot Bot added the released: 1.95.0 Shipped in 1.95.0 label Nov 19, 2026
@openhands-release-bot

Copy link
Copy Markdown
Contributor

🚀 Released in 1.95.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

released: 1.95.0 Shipped in 1.95.0 type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants