Make every Bot say where its answer came from

[davidmckayv]

Closes #142.

What happened

Q: A customer made 12 cash deposits just under the reporting threshold. What is our obligation?

Risk Analyst answered at length and with confidence: aggregate for CTR purposes, do not tip off the customer, escalate to AML/BSA, "the institution generally must file a SAR", "$5,000 or more", "file within 30 calendar days of initial detection", retain documentation "for 5 years".

The audit trail for that turn:

03:03:28  channel.routed   chosen=risk-analyst

One row. No tool call, no source, and nothing saying the answer came from the model rather than from anything this deployment can reach.

Several of those numbers may well be right. That is the problem — a confident, plausible, unsourced answer is indistinguishable from a confident, plausible, wrong one, on a question about whether to file, against what threshold, inside what deadline.

Why it belongs in the deployment, not in a package

One package's knowledge Bot already had a rule against exactly this, in its YAML. The Bot whose entire subject is regulatory obligation did not, because a remote-ag-ui agent gets its role_description and nothing else — in this case the single sentence "Investigate policies, transaction monitoring, and control evidence."

That asymmetry is the bug, and the issue names the fix: a rule this important sitting in one agent's YAML is a rule that will be missing from the next agent somebody adds. So it lives in the shared prompt, and reaches both paths — the built-in prompt, and the standing role message that is a remote Bot's only instruction.

It is unconditional, unlike the tool and computer guidance beside it. A Bot with no tools and no computer needs it most: it has nothing to read, so everything it says comes from its own knowledge.

What it does not do

It does not tell Bots to go and find a source.

An earlier attempt at this did, and Bots went hunting the open web and looped on a government 404 page — worse than the problem it was fixing. That attempt never left a branch, and the guidance now says so explicitly:

This is not an instruction to go looking. If nothing you can reach covers the question, answer as well as you can and mark it plainly as unverified. Do not go hunting the open web for something to cite, and do not keep retrying a page that is not giving you one.

An unsourced answer that says it is unsourced is honest. A search that never ends is a Bot that never answers.

Driven in Chrome, same question

Rebuilt the Bot container, restarted through scripts/start.sh, asked the identical question through the @ picker. The answer now opens:

Source checked: I searched Google Drive for internal AML/cash reporting policy terms, but found no matching policy or procedure to cite. The answer below is therefore from general AML knowledge and is not yet checked against your organisation's own policy or current local regulation.

And the trail behind it is no longer one row:

04:27:20  mcp.call_succeeded  google-drive/search_files  risk-analyst
04:27:17  mcp.call_succeeded  google-drive/search_files  risk-analyst
04:27:13  channel.routed      risk-analyst

So "Source checked" is a fact the audit trail confirms, not a claim the model made about itself. It checked what it could reach, twice, stopped, and answered.

The body changed too, and this is the part that matters most. Where the original asserted $5,000, 30 calendar days and 5 years, this one says "Likely obligation", "may need to be aggregated depending on the rule and timing", and "In many regimes, suspected structuring is reportable even if no single deposit reaches the cash reporting threshold". The invented figures and deadlines — the parts somebody would have acted on — are gone, without the answer becoming useless.

Also

docs/configuration.md now says that role_description is the whole brief for a remote-ag-ui agent. It is not obvious sitting beside a built-in agent that gets a prompt, and it is the reason this Bot had no rule to begin with.

Tests

Four added, two of them proven to fail with the wiring removed:

• a built-in Bot carries the rule even holding nothing at all
• a remote Bot carries it in the only instruction it ever gets
• it does not send the Bot hunting for a source
• it names the answers that must not be stated without one: a threshold, a deadline, a filing obligation, a figure

Three existing exact-prompt assertions were updated to include it, referencing the exported constant rather than restating the prose, so they stay exact without pinning the wording twice.