Skip to content

fix: 2 agent bugs and improve test coverage#351

Open
altf4-games wants to merge 3 commits into
microsoft:devfrom
altf4-games:fix/bugs-and-test-coverage
Open

fix: 2 agent bugs and improve test coverage#351
altf4-games wants to merge 3 commits into
microsoft:devfrom
altf4-games:fix/bugs-and-test-coverage

Conversation

@altf4-games
Copy link
Copy Markdown

@altf4-games altf4-games commented May 20, 2026

What this PR does

This PR adds test coverage for four previously untested backend modules and fixes two runtime bugs.


Bug Fixes

1. Wrong input key sent to LLM — agents/agent_sort_data.py

The input JSON sent to the model used the key "value" but every example in the system prompt uses "values". The LLM was receiving inconsistent field names, which could cause unpredictable outputs.

2. Whitespace-only language code produces garbled prompt — agents/agent_language.py

build_language_instruction(" ") would produce a broken instruction with an empty display name because " " is truthy, so the or DEFAULT_LANGUAGE fallback never fired. The fix is to strip whitespace before the fallback check.


New Test Coverage

Added comprehensive unit test suites covering previously untested modules:

File Tests What's covered
test_semantic_types.py 208 All 7 classification helpers, full VL type map (43 types), name inference, prompt generator
test_agent_language.py 43 English/non-English/whitespace/unknown inputs, full vs compact mode, extra rules, marker injection
test_client_utils.py 31 Model name prefixing, Ollama URL normalisation, image block stripping, error detection, from_config
test_sort_data_agent.py 10 Input key regression guard, unicode serialisation, response parsing, dialog structure

Minor cleanup

  • client_utils.py: get_completion had a copy-pasted docstring from __init__ saying it "returns a client"
  • area.ts, line.ts: JSDoc comments said group() was "marked TODO" but it was already implemented in both files

@zhb-ai
Copy link
Copy Markdown
Collaborator

zhb-ai commented May 20, 2026

Thanks for the thorough PR! Regarding item 1 (Knowledge store depth):

This has already been fixed on the dev branch — _MAX_DEPTH["experiences"] is set to 2 with the comment # one sub-dir: "category/file.md", and the related tests are passing. You might have been looking at main which hadn't picked up that change yet. So you can drop that hunk from this PR.

@zhb-ai
Copy link
Copy Markdown
Collaborator

zhb-ai commented May 20, 2026

Item 2 (Auth-tier params in saved config): This is an intentional design choice rather than a bug. _default_params holds connector-level shared configuration (e.g., a service-account username for admin-pinned connectors), and per-user credentials are stored separately in an encrypted, identity-isolated vault. The include_auth_tier=False behavior allows the frontend to pre-populate the username field for UX convenience, while truly sensitive values (passwords, tokens) are always stripped. That said, I can see the argument for stricter isolation — let's discuss the trade-offs in a separate design thread before changing this behavior.

Item 3 (Wrong input key "value" → "values"): Confirmed — this is a real typo on dev as well. The system prompt examples all use "values" but the runtime code sends "value". Happy to accept this fix.

Item 4 (Whitespace-only language code): Confirmed — still present on dev. Extremely unlikely to trigger in practice, but the fix is correct and risk-free. Happy to accept.

Summary: Please rebase onto dev, drop item 1, drop item 2 (pending design discussion), and keep items 3 and 4. I'll merge once those adjustments are made. Thanks again for the thorough work!

@altf4-games altf4-games force-pushed the fix/bugs-and-test-coverage branch from d0036e0 to 9e77eec Compare May 20, 2026 16:43
@altf4-games altf4-games changed the base branch from main to dev May 20, 2026 16:47
@altf4-games altf4-games changed the title fix: 4 bugs + test coverage for previously untested modules fix: 2 agent bugs and improve test coverage May 20, 2026
@altf4-games
Copy link
Copy Markdown
Author

Hi @zhb-ai, thanks for the review!
I've updated the PR to incorporate your feedback:

  • Dropped the knowledge store depth restriction change (since it's already resolved in dev).
  • Dropped the auth-tier parameter stripping change from data_connector.py.
  • Reverted client_utils.py to match the fully consolidated litellm approach from upstream.
  • Adjusted the new test suites to align perfectly with the upstream model prefixing logic.
    All backend tests (1,800+) are running cleanly. Ready for another look whenever you have a moment!

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes two backend agent runtime issues (SortDataAgent input schema + language-code whitespace handling) and adds unit test coverage for previously untested agent/util modules, alongside small documentation comment corrections in GoFish chart templates.

Changes:

  • Fix SortDataAgent LLM input payload to use the "values" key (matching the system prompt examples).
  • Fix language instruction building to treat whitespace-only language codes as empty and fall back to the default language.
  • Add new backend unit test suites for agent language handling, semantic type helpers, client utilities, and sort-data agent behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/backend/agents/test_sort_data_agent.py Adds SortDataAgent tests for input construction, JSON extraction, and candidate metadata.
tests/backend/agents/test_semantic_types.py Adds comprehensive tests for semantic type constants, helper classifiers, VL mapping, inference, and prompt generation.
tests/backend/agents/test_client_utils.py Adds tests for Client model prefixing, Ollama base normalization, image stripping, error detection, and from_config.
tests/backend/agents/test_agent_language.py Adds tests for language instruction building/injection (including whitespace and unknown codes).
src/lib/agents-chart/gofish/templates/line.ts Updates JSDoc to reflect implemented multi-series behavior.
src/lib/agents-chart/gofish/templates/area.ts Updates JSDoc to reflect implemented multi-series behavior.
py-src/data_formulator/agents/agent_sort_data.py Fixes input payload key from value to values.
py-src/data_formulator/agents/agent_language.py Fixes whitespace-only language handling by stripping before fallback.

assert '"values"' in user_content, (
f"Expected key 'values' in user content but got:\n{user_content}"
)
assert '"value"' not in user_content.split('"values"')[0] or True # key 'value' must not appear as a standalone key
@@ -114,7 +114,7 @@ def build_language_instruction(language: str, *, mode: str = "full") -> str:

Returns ``""`` when *language* is ``"en"`` (or empty / unrecognised).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants