Repository navigation
fix(chat): embed curated-view schema statically to stop first-attempt table-name guessing - #29
Merged
Badri-Rajendran merged 1 commit intoSep 7, 2026
Conversation
…s guessing table names Live-tested after PR #26/#27: every one of 4 real questions still got validator_rejected on its first execute_read_only_sql attempt (guessed account_balance/transactions/positions/tax_lots instead of the real v_customer_balance/v_transaction_history/v_holdings/v_tax_lots). PR #27's retry-on-rejection fix recovered 3 of 4 on a second attempt, but "balance" failed twice in a row: it found the right table on retry, then guessed a column ("balance") that doesn't exist on v_customer_balance (raw signed postings needing sum(amount_money)) -- that column only exists on v_published_snapshot, the as-published figure for a period, not the live one. Root cause: get_database_schema is a tool the model can skip, and its own tool-call order for the balance question showed both SQL attempts happening before it ever called that tool. Fixed by embedding the exact 8 view/column list directly in _SYSTEM_PROMPT_TEMPLATE -- always in context from turn one, no longer dependent on the model choosing to call a tool before it guesses -- with a one-line note resolving the balance/v_published_snapshot ambiguity. get_database_schema stays available as a tool (harmless, still useful if the schema ever drifts from the prompt) and rule 8's retry guidance stays as a fallback. Verified live: all 4 previously-failing questions, including balance, now succeed on their first execute_read_only_sql call with zero rejections. New tests/unit/test_chat_orchestration_service.py: a tripwire (every curated view name appears in the prompt) plus a regression guard (the view/column listing itself never lists customer_id as a column, the exact confusion PR #26 already fixed once in describe_schema()'s dynamic output) -- not a content-wording pin, matching this codebase's existing "script-driven, not content-driven" test convention for the fake agent adapter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Live-tested after PR #26/#27 landed: every one of 4 real chat questions still got
validator_rejectedon the model's firstexecute_read_only_sqlattempt.Diagnosis (verified live)
The model guessed
account_balance,transactions,positions,tax_lotsinstead of the realview names
v_customer_balance,v_transaction_history,v_holdings,v_tax_lots. PR #27'sretry-on-rejection fix recovered 3 of 4 on a second attempt — but "balance" failed twice in a
row: it found the right table on retry, then guessed a column (
balance) that doesn't exist onv_customer_balance(raw signed postings needingsum(amount_money)for a total). That columnonly exists on
v_published_snapshot— the as-published figure for a period, a different thingentirely. Root cause:
get_database_schemais a tool the model can choose to skip, and its owntool-call order for the balance question showed both SQL attempts happening before it ever
called that tool.
Fix
Embed the exact 8 curated view/column list directly in
_SYSTEM_PROMPT_TEMPLATE— always incontext from turn one, no longer dependent on the model choosing to call a tool before it guesses
— with a one-line note resolving the
balance/v_published_snapshotambiguity. Keptget_database_schemaas a tool (harmless, still useful if the schema ever drifts from thehardcoded prompt) and rule 8's retry-on-rejection guidance as a fallback safety net.
Validation evidence
tests/unit/test_chat_orchestration_service.py: a tripwire asserting every curated viewname appears in the prompt, and a regression guard that the view/column listing itself never
lists
customer_idas a column (the exact confusion PR fix(chat): stop rejecting compound WHERE clauses; stop inviting a fabricated customer_id #26 already removed once fromdescribe_schema()'s dynamic output). Not a content-wording pin — matches this codebase'sexisting "script-driven, not content-driven" convention (
FakeAgentAdapterdeletessystem_prompt).uv run pytest tests/unit/test_chat_orchestration_service.py tests/unit/test_sql_tool_validator.py tests/integration/test_chat_security_perimeter.py tests/api/test_chat.py— 62 passed.uv run ruff check,uv run mypy --strict app,uv run lint-imports— clean, zero new baseline noise.funded/approved customer, and re-asked the exact 4 previously-failing questions. All 4 now
succeed on the first
execute_read_only_sqlcall with zero rejections, including "balance"(previously the one that failed twice) — correctly answers "$1,000.00 (LIVE figure)".
Security / DB / API impact
None. Prompt content only — no new endpoint, no wire-shape change, no change to the ADR 19 safety
perimeter (views, grants, validator allow-list all unchanged).