[mayuriphad]

Fixes #101

Problem

fetch_web_page_executor passed every response body to trafilatura.extract(..., output_format="markdown"), whatever the Content-Type. trafilatura only extracts HTML and returns None for JSON, CSV, XML and plain text. The executor turned that into "" and returned success=True with <body></body>, so an agent couldn't read API responses or data files and wasn't told that anything went wrong.

Fix

• _fetch_with_redirects now returns the final httpx.Response instead of only its bytes, so the executor can see the headers.
• The executor branches on the media type:
  • text types (text/* other than HTML, application/json, application/xml, +json/+xml suffixes, CSV, NDJSON, YAML, JavaScript) are returned as response.text, truncated to MAX_LENGTH_WEB_FETCH_HTML as before;
  • text/html, application/xhtml+xml, or a response with no Content-Type still go through trafilatura, as now;
  • anything else (for example application/pdf) returns success=False with Unsupported content type: <type>.
• The tool description says text responses are returned as is.

Tests

• New parametrized test_web_fetch_returns_text_responses_as_is (JSON, a +json vendor type with charset, CSV, plain text, XML), test_web_fetch_extracts_html_responses and test_web_fetch_reports_unsupported_content_types. The text and unsupported-type cases fail on main.
• The html_response test helper now sends Content-Type: text/html. Before, httpx.Response(text=...) defaulted it to text/plain, which didn't match a real HTML page.
• tests/test_web_fetch.py and tests/test_web_search.py: 61 passed locally (Python 3.12, trafilatura 2.3.1). ruff format --check and ruff check are clean on both files.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.