Skip to content

fix latency, add dynamic model selector - #31

Open
Jehu Gray (lowelljehu) wants to merge 1 commit into
microsoft:mainfrom
lowelljehu:main
Open

fix latency, add dynamic model selector #31
Jehu Gray (lowelljehu) wants to merge 1 commit into
microsoft:mainfrom
lowelljehu:main

Conversation

@lowelljehu

Copy link
Copy Markdown

Latency:

  • Cap reasoning_effort for gpt-5/o-series models on raw chat-completions call sites (contextual_agent, semantic_translator tile-selection) to avoid unnecessary reasoning-token overhead on simple prompts.
  • Attempt reasoning_effort at Agent Service agent-creation time for reasoning models (analyst_agent), with a safe fallback since the installed azure-ai-agents SDK does not support the parameter.

Model selector:

  • AnalystAgent now honors the frontend's selected model end-to-end via a new _ensure_model()/_create_agent_capped() mechanism that recreates the Agent Service agent definition on model change while preserving threads (thread_id is not tied to a specific agent_id).
  • ClarifierAgent (greeting/identity short-circuit) now accepts a per-call model override instead of using a fixed singleton deployment, and no longer unconditionally sends reasoning_effort to non-reasoning models.
  • Fixed a pre-existing bug where ClarifierAgent used AZURE_AI_PROJECT_ENDPOINT (an Agent Service project endpoint) instead of AZURE_OPENAI_ENDPOINT, causing a silently-swallowed 401 on every greeting.
  • Added GET /api/models: discovers real Azure OpenAI deployments via the ARM management API (with a 5-minute in-memory cache and a static fallback list), so the dropdown reflects whatever models are actually deployed in a given tenant instead of a hardcoded list.
  • ModelSelector.tsx now fetches from /api/models instead of guessing availability from the overall /api/health status.

…scovery

Latency:
- Cap reasoning_effort for gpt-5/o-series models on raw chat-completions
  call sites (contextual_agent, semantic_translator tile-selection) to
  avoid unnecessary reasoning-token overhead on simple prompts.
- Attempt reasoning_effort at Agent Service agent-creation time for
  reasoning models (analyst_agent), with a safe fallback since the
  installed azure-ai-agents SDK does not support the parameter.

Model selector:
- AnalystAgent now honors the frontend's selected model end-to-end via
  a new _ensure_model()/_create_agent_capped() mechanism that recreates
  the Agent Service agent definition on model change while preserving
  threads (thread_id is not tied to a specific agent_id).
- ClarifierAgent (greeting/identity short-circuit) now accepts a
  per-call model override instead of using a fixed singleton deployment,
  and no longer unconditionally sends reasoning_effort to non-reasoning
  models.
- Fixed a pre-existing bug where ClarifierAgent used
  AZURE_AI_PROJECT_ENDPOINT (an Agent Service project endpoint) instead
  of AZURE_OPENAI_ENDPOINT, causing a silently-swallowed 401 on every
  greeting.
- Added GET /api/models: discovers real Azure OpenAI deployments via the
  ARM management API (with a 5-minute in-memory cache and a static
  fallback list), so the dropdown reflects whatever models are actually
  deployed in a given tenant instead of a hardcoded list.
- ModelSelector.tsx now fetches from /api/models instead of guessing
  availability from the overall /api/health status.
@lowelljehu

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="Microsoft"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant