Activating an LLM profile via /api/profiles/{name}/activate doesn't update AgentProfile.llm_profile_ref, allowing the two to drift silently

[VascoSch92]

Bug Description

POST /api/profiles/{name}/activate (the LLM-profile activation endpoint) updates
agent_settings.llm and active_profile atomically. POST /api/agent-profiles/{id}/activate (the AgentProfile activation endpoint) is, by
contract, pointer-only and must never touch agent_settings
(ActivateAgentProfileResponse.agent_settings_applied is hardcoded False for
exactly this reason). Nothing reconciles the two: activating an LLM profile never
updates any AgentProfile.llm_profile_ref that pointed at the previously-active
profile, so a seeded/well-known default AgentProfile can end up permanently
referencing a stale or deleted LLM profile while the system's actual active LLM
profile has moved on.

Root Cause

# openhands-agent-server/openhands/agent_server/profiles_router.py
def apply_profile(settings: PersistedSettings) -> PersistedSettings:
    settings.agent_settings = settings.agent_settings.model_copy(update={"llm": llm})
    settings.active_profile = name
    return settings

versus

# openhands-agent-server/openhands/agent_server/agent_profiles_router.py
"""
...Activation here is pointer-only — unlike the LLM ``/activate`` it must
**not** write ``agent_settings`` (the creation-time-only contract).
"""
...
class ActivateAgentProfileResponse(BaseModel):
    ...
    agent_settings_applied: bool = False  # Always False: activation is pointer-only by contract.

resolve_agent_profile() (the only code path that actually re-resolves
llm_profile_ref into a real LLM config) is invoked from
conversation_service.py's _resolve_agent_from_profile, which only runs when a
conversation is launched by profile id. The much more common path — a client
fetching GET /api/settings and forwarding agent_settings to POST /api/conversations (the pattern used by this repo's own agent-server automations,
and by the local/well-known default AgentProfile launch path per #16193 in
OpenHands/OpenHands) — never touches resolve_agent_profile and therefore never
notices that llm_profile_ref is stale.

Steps to Reproduce

1. Save two LLM profiles, e.g. POST /api/profiles/profile-a and POST /api/profiles/profile-b.
2. POST /api/profiles/profile-a/activate — active_profile is now profile-a.
3. Confirm (or create) an AgentProfile named default whose llm_profile_ref
   is profile-a (this is the seeded default in a fresh instance).
4. POST /api/profiles/profile-b/activate — active_profile is now profile-b,
   agent_settings.llm now reflects profile-b.
5. GET /api/agent-profiles/default — llm_profile_ref still reads profile-a.
6. DELETE /api/profiles/profile-a — succeeds (no FK violation, since nothing
   currently guards default's specific well-known status the way named profiles
   are guarded elsewhere).
7. GET /api/agent-profiles/default — llm_profile_ref now names a profile
   that no longer exists, with no error, warning, or repair anywhere in the
   response.

Actual Impact Observed

Reproduced this drift on a live deployment: the active LLM profile and the
default AgentProfile's llm_profile_ref had silently diverged (two different
named profiles, two different underlying models), with no error surfaced anywhere
in GET /api/settings or GET /api/agent-profiles. This is the same root cause
independently reported (and only partially fixed — frontend gate logic only, per
the shipped PR) in OpenHands/OpenHands#16193, whose own "Suggested fix" section
explicitly flagged this as an unaddressed follow-up:

Keeping the seeded default.llm_profile_ref synchronized when the active LLM
profile changes may also reduce stale state, but the frontend gate should still
agree with the launch semantics.

See also #3841 (LLM/AgentProfile identity asymmetry — related but distinct: that
issue is about rename-safety, this one is about activation-time sync) and #4314
(a different, now-fixed desync path between PATCH /api/settings and
agent_settings.llm — same category of bug, different trigger).

Possible Fix Directions

Not prescribing one — this is a real design decision:

• (a) When an LLM profile is deactivated (a different one is activated) or
  deleted, cascade-update any AgentProfile.llm_profile_ref that pointed at it —
  at minimum for the seeded default profile, which is already special-cased
  elsewhere (see [Bug]: Composer blocks local default AgentProfile despite a valid active LLM profile OpenHands#16193/#16200) as "the enriched baseline that
  launches through live agent_settings."
• (b) Expose a computed staleness flag on GET /api/agent-profiles/{id} (e.g.
  llm_profile_ref_matches_active: bool) so callers can detect drift without
  reimplementing the diff themselves.
• (c) Auto-/materialize (or an equivalent dry-run resolve) as part of the
  GET /api/agent-profiles/{id} response so staleness is visible without a
  separate call.
• (d) Guard DELETE /api/profiles/{name} for default's current
  llm_profile_ref the same way it already guards other referenced profiles
  (the FK check in delete_llm_profile exists — worth checking why it didn't
  catch this).

[AzeelSajjad]

Opened #4372 (draft) with a fix for this.

Approach

Took directions (a) and (b), both narrowed, and skipped (c); (d) turned out to need no change (below).

(a) Cascade — restricted to the seeded profile, and gated. A new store-agnostic FK helper sync_seed_llm_ref joins the existing find_referrers / cascade_rename / delete_llm_profile / rename_llm_profile family in profile_refs.py. On activation it repoints only the profile named SEED_PROFILE_NAME, and only when its llm_profile_ref currently equals the outgoing profile, or is the dangling soft-ref a fresh instance is born with (#3933).

The gate is the point. Repointing every profile whose ref matched the outgoing one would silently rewrite a profile a user deliberately pinned that merely happened to coincide with the active profile. The seeded default is a mirror of live settings, not a user pin — so it follows, and nothing else is touched.

(b) Staleness flag. llm_profile_ref_matches_active on both GET /api/agent-profiles and GET /api/agent-profiles/{name}. Tri-state bool | None, where None means "not applicable/unknown" (ACP profile, or nothing active) and never "stale". Additive with a None default, so existing clients are unaffected. This is what surfaces drift the cascade deliberately declines to repair.

(c) skipped — auto-materialize inside GET adds per-profile disk I/O and changes the response shape for a diagnostic.

On direction (d)

The FK guard already works. delete_llm_profile holds the agent-store lock across scan-then-delete and raises ProfileReferenced. Verified against origin/main with a live server — the delete in repro step 6 returns:

HTTP 409
{"detail":"LLM profile is referenced by 1 agent profile(s): default"}

So step 6 → 7 (a llm_profile_ref naming a profile that no longer exists) isn't reachable through that path on current main; it only succeeds when the agent-profile store was never seeded, since seeding is lazy on GET /api/agent-profiles. No code change was needed, but the PR adds a regression test pinning the guard so it can't silently regress.

The drift itself (steps 4–5) reproduces exactly as reported — confirmed end-to-end on origin/main @ 973c3513: after activating profile-b, default.llm_profile_ref still read profile-a while active_profile was profile-b, with no error surfaced. Before/after transcripts from live servers are in the PR.

One gap left open deliberately

The seeded profile can be born in three states; this fixes two:

State	At seed time	default.llm_profile_ref	Repaired
A	an LLM profile was already active	that profile's name	yes
B	no active profile, no real LLM config	"default", dangling	yes
C	no active profile but a real LLM config	"default", resolves	no

State C is the legacy/migration path: _seed_default_llm_profile mints an LLM profile named default mirroring live settings, so the ref resolves — and a resolving ref is indistinguishable from a deliberate pin. revision doesn't work as a discriminator either: save_profile_preserving_identity bumps it only on overwrite, not on create, and it defaults to 0, so a user-authored default on first save looks identical to the seeded one. Repairing state C would risk exactly the clobber the design prevents, so it's documented in the helper's docstring rather than silently narrow — and llm_profile_ref_matches_active reports False for it, so it's visible.

Happy to revisit that trade-off if you'd prefer state C repaired too — it's a judgment call, not a technical blocker.

@VascoSch92 — this issue is assigned to you, so apologies if you were already partway into it; happy to close the PR if you'd rather carry it yourself.