skills(search): empty or blocked search results are not a license to invent URLs

[ankitranjan7]

Why this issue exists

This came from the local:blocked-source-research eval. The task required a protected-page read and an independently fetched explanatory source.

Source reports:

• /Users/ankitranjan/Work/agentrhq/evals/results/run-20260820T085734Z/report.md
• /Users/ankitranjan/Work/agentrhq/evals/results/run-20260820T103952Z/report.md

Current Webcmd state

As of origin/main 22974697bb3fe784bad5ef5a208f2d6ee5a272a8 (webcmd-v0.7.4-2-g2297469):

• smart-search already says to extract useful result URLs from a fetched search page, then fetch target pages with webcmd web fetch.
• smart-search already says snippets and result titles are discovery only, not evidence.
• smart-search already says a page with zero usable result URLs is a failed search, not proof of no results, and the agent should move to the next engine.
• webcmd-usage describes webcmd plugin search as marketplace/plugin discovery and install-source lookup.
• Current CLI output does not prevent an agent from using webcmd plugin search <research topic> as a fake web search.
• Current CLI/skill surfaces do not enforce source provenance after blocked or empty search results.

Observed agent behavior

The Webcmd blocked-page path mostly worked:

• A direct fetch hit 403.
• webcmd web fetch --url ... -f json returned structured FETCH_BLOCKED.
• The agent created one browser session after the block.
• The live page title/heading were read through the browser.
• The session was closed after the task.

The source-evidence path failed in one run:

• DuckDuckGo HTML returned a 302/nginx response.
• Bing was fetched multiple times and returned generic TLS results, not JA3/impersonation evidence.
• The agent ran webcmd plugin search tls fingerprint / webcmd plugin search ja3 as if plugin catalog search were web search.
• After no fetched search result contained the desired source, the agent guessed known URLs: first a Salesforce blog slug that 404ed, then github.com/salesforce/ja3.
• The final JSON summarized a source URL that was not discovered from fetched search results.

A related evidence failure appeared in the later run:

• The source URL itself was correctly fetched from DuckDuckGo results.
• But product details were copied from /ecommerce/ after the protected page body only showed You bypassed the Cloudflare challenge! :D and no products.

The shared behavior is filling an evidence gap with plausible outside knowledge.

What went wrong

The agent treated "search did not find the source" as a prompt to recall or invent a likely URL.

It also confused two different CLI surfaces:

• webcmd plugin search finds installable Webcmd plugins.
• It is not a general web search engine.

The bad path is:

1. Source needed.
2. Search engine fetch is blocked, empty, redirected, or generic.
3. Agent tries plugin search as a replacement web search.
4. Plugin search returns no plugins.
5. Agent guesses a canonical source URL from memory.
6. Final answer looks sourced, but provenance is broken.

Why this matters

This is a trust boundary. If the source URL was not discovered from fetched search results and then fetched itself, the agent cannot claim it as evidence.

The CLI can make catalog-vs-web-search clearer, but it cannot fully prevent an agent from inventing a remembered URL. Some instruction or guardrail is probably necessary here.

Solution directions

The preferred solution should make source provenance explicit without adding lots of case-specific skill text.

Possible directions:

1. Skill/search policy: state that webcmd plugin search is catalog discovery only, not web search.
2. Evidence rule: a source may be summarized only after its URL appeared in fetched search results and that URL was fetched.
3. Failure behavior: if allowed search retries produce no usable result URLs, report the source gap instead of guessing.
4. CLI copy/help: make webcmd plugin search output/help label itself as plugin catalog discovery so weak agents do not mistake it for web search.
5. Optional product idea: structured search/fetch helpers could carry provenance, but that may be larger than this issue needs.

Open questions for senior review

• Should this be handled as a minimal smart-search skill rule, or should plugin-search output become clearer first?
• Do we need a provenance field/convention for fetched sources, or is that overkill?
• Should the same issue also cover invented page details, or should source-URL invention stay separate?

Acceptance criteria

• webcmd plugin search is not used as web search for research topics.
• Empty, blocked, or generic search results lead to another allowed engine once, then a reported gap.
• A summarized source URL must have appeared in fetched search results and must itself be fetched.
• The agent does not fill missing page/source evidence from memory.

Non-goals

• No change to webcmd web fetch TLS escalation here.
• No search-engine-specific workaround unless the general flow still fails.
• No fixture-specific wording.