Skip to content

feat(sdk): improve read_url error diagnostics and html table extraction - #1122

Closed
nilaymallikk wants to merge 8801 commits into
CodebuffAI:mainfrom
nilaymallikk:feature/read-url-diagnostics
Closed

feat(sdk): improve read_url error diagnostics and html table extraction#1122
nilaymallikk wants to merge 8801 commits into
CodebuffAI:mainfrom
nilaymallikk:feature/read-url-diagnostics

Conversation

@nilaymallikk

Copy link
Copy Markdown

Summary

Closes #1121.

Enhances the read_url tool in @codebuff/sdk and common tool schemas:

  • Structured Error Diagnostics: Adds typed error classifications (errorCode such as HTTP_ERROR, BLOCKED_ADDRESS, INVALID_URL, TOO_MANY_REDIRECTS, UNSUPPORTED_CONTENT_TYPE, RESPONSE_TOO_LARGE, TIMEOUT, ABORTED) and forwards HTTP status codes (e.g. 404, 429, 500) so caller agents can differentiate retryable rate-limits from permanent errors.
  • HTML Table Extraction: Added formatHtmlTables pre-processor during HTML readable text extraction to transform <table> elements into structured GitHub-Flavored Markdown tables instead of flattening cells into unformatted lines.

Scope

All changes are strictly scoped within:

  • sdk/src/tools/read-url.ts
  • sdk/src/__tests__/read-url.test.ts
  • common/src/tools/params/tool/read-url.ts

Verification

  • Passed unit tests: bun test sdk/src/__tests__/read-url.test.ts (21 pass)
  • Passed full SDK test suite: bun test sdk/src/__tests__/ (470 pass)
  • Passed CI build and packaging: bun run ci
  • Passed binary smoke test: bun cli/scripts/smoke-binary.ts cli/bin/freebuff

codebuff public sync bot added 30 commits August 19, 2026 08:31
Source: CodebuffAI/freebuff-private@e76be31e6b3ffe894ec6fb6fe0289ecce9fd4457
Source: CodebuffAI/freebuff-private@06c76ce5a6e375830b676c2473d7a15c27e4c9f1
Source: CodebuffAI/freebuff-private@228c49cc80b37907fa8506aa8e1fb4d817179493
Source: CodebuffAI/freebuff-private@38d88e049957238b9aba84f7dd25f973fdee5087
Source: CodebuffAI/freebuff-private@3b89b37ffac9f705298099c1a92cc12ff86439b1
Source: CodebuffAI/freebuff-private@6a6c2560a0d89ff9d4fb3b3b3dcf459e718aa015
Source: CodebuffAI/freebuff-private@6bbe32b95effc6c81479ffe1be3854bbf2b6a175
Source: CodebuffAI/freebuff-private@9c49375348733a5bdc7b13b5e5c71b6aac3c150c
Source: CodebuffAI/freebuff-private@18d9e898bfc0e44ac24017f980f81033123c2f83
Source: CodebuffAI/freebuff-private@67af829808816a20e117fd57c654b471e6857eae
Source: CodebuffAI/freebuff-private@3285b31b770fd5b07e854d993ac8e87e5e679f8b
Source: CodebuffAI/freebuff-private@79bfc42ece2b9e4cc4a88140e23479e5809da816
Source: CodebuffAI/freebuff-private@65df59b85e220b8f962fe01cdbe7d4e18824e320
Source: CodebuffAI/freebuff-private@e6b914d9b664add769727e29d0984c69e8717e74
Source: CodebuffAI/freebuff-private@1781f579a7d28f6609f75540529b58bd5ee8208f
Source: CodebuffAI/freebuff-private@d266b9ad26aa4d539a01097c6be544afada6b7e6
Source: CodebuffAI/freebuff-private@a518571bda930dd9f6d3c513f405f879d9474585
Source: CodebuffAI/freebuff-private@ff3322e5cc03fc4f33796f8f49fce51744b6ecb0
Source: CodebuffAI/freebuff-private@b96808549f5ba06fbac82369b419fef88bc6c6ae
Source: CodebuffAI/freebuff-private@2cf94fb38519b4fdc2caa80ce073ec2e423b2ddd
Source: CodebuffAI/freebuff-private@a3a71b3b96bd25ce2f4258704267af6ff646f9ec
Source: CodebuffAI/freebuff-private@0693087e2e254602d44514d9c44099410dda8323
Source: CodebuffAI/freebuff-private@efdf10960936cf0f87f8141191fb76b55706ed85
Source: CodebuffAI/freebuff-private@ad6bc9adfa02f0b967424df70a6b1202bcec285d
Source: CodebuffAI/freebuff-private@c71eca2672511e34e48c6e36344b060af95f0ee6
Source: CodebuffAI/freebuff-private@edac5d5b4121e342607f803b277b2a20eed859a0
Source: CodebuffAI/freebuff-private@05417f841e0359ece3e80876fe19ba5f25044f51
Source: CodebuffAI/freebuff-private@7f756a3d3d5ae9237b007a038334de6f143daff1
Source: CodebuffAI/freebuff-private@665569ba8321ae5a453633dd0693029e3414bf93
Source: CodebuffAI/freebuff-private@050b9f6fd4658f2589d2b4498e99298e49ccbb81
codebuff public sync bot and others added 24 commits August 24, 2026 20:18
Source: CodebuffAI/freebuff-private@2610acaadc5b7bcf8eded2d362c33b422ac1c69c
Source: CodebuffAI/freebuff-private@a186b4d3778cc5fac6907d345479f1bae22ae999
Source: CodebuffAI/freebuff-private@5faa5f6218239ee702b36f72bdf1c420c44d8389
Source: CodebuffAI/freebuff-private@fbd4ee3918974c2ced6550233174c3cbc19eaecc
Source: CodebuffAI/freebuff-private@7a5444e862dfd2b9d5b590c057719a662a7df4c7
Source: CodebuffAI/freebuff-private@241f8bfa8a8a97867900b2f71754638efa2cbe86
Source: CodebuffAI/freebuff-private@cfc7760af31f29f1acc7153a7e9fcb48d35f730b
Source: CodebuffAI/freebuff-private@6965ea40dbf54a44c932ca4df63b0dd6377bfec5
Source: CodebuffAI/freebuff-private@ecf4fa2e98fcefcec1a96cbd596b5b16bccd97e3
Source: CodebuffAI/freebuff-private@ce96ca2455d9c6a027d6e41048e8f2520f78754f
Source: CodebuffAI/freebuff-private@890653e2610c57c22298d532cae0129efb26f862
Source: CodebuffAI/freebuff-private@d60d93125f90cdca18e1fc97819b91997081a97f
Source: CodebuffAI/freebuff-private@70ba2a57ab1e48c9f799107675dfb6029dc6d54f
Source: CodebuffAI/freebuff-private@9c2be6595fa98153b9ec0474caed5ec3a58ec116
Source: CodebuffAI/freebuff-private@c1db5271ba86e1c89c3c9fc0c6e6a2a857d3b5dd
Source: CodebuffAI/freebuff-private@2846f67777f75ee23da1e3a3047176d66953d906
Source: CodebuffAI/freebuff-private@f5329a11c8431d6f5c98581c2bfc67fb2d1829ba
Source: CodebuffAI/freebuff-private@e319a9e3de31d8c4cd1d0a9285dfd5360eebb7a1
Source: CodebuffAI/freebuff-private@9cbbba39efda2071b8325bf2779b39170e35348c
Source: CodebuffAI/freebuff-private@8470da01520ea7c273efb6eca943fd37fb265189
Source: CodebuffAI/freebuff-private@884e8a41dd02bb5c0be09d8da8e465c994a6c6a4
Source: CodebuffAI/freebuff-private@b550c712e828f5211de07c5699a643fe0fa8f22e
Source: CodebuffAI/freebuff-private@ddfa6c89a27176e4e1afef0bd597077698e3d9e3
@codebuff-team

ghost commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Good instinct and correctly scoped to sdk/common (not touching any forbidden paths). The errorCode/status additions to read_url are backward-compatible (optional fields) and well-tested for the HTTP_ERROR/UNSUPPORTED_CONTENT_TYPE/INVALID_URL/BLOCKED_ADDRESS/TOO_MANY_REDIRECTS paths.

Two things need fixing before this is portable:

  1. Table cell escaping is missing. In formatHtmlTables (read-url.ts ~L200-235), cell text is joined with | but never escapes literal | characters that may appear in the extracted text (e.g. a table cell containing a|b). That will silently corrupt the resulting markdown table structure. Needs cell.replace(/\|/g, '\|') or similar before joining.

  2. Error classification via string matching is fragile. isInvalid (L449-452) and isTooLarge (L534-537) key off substrings of thrown error messages ('Only http:// and https:// URLs are supported', 'Response is too large', 'Response exceeded') rather than typed errors from assertUrlAllowed/fetch. If those message strings ever change, the errorCode silently degrades to the wrong classification with no test to catch it. Prefer having assertUrlAllowed throw a typed error (e.g. {code, message}) rather than matching text.

  3. Test coverage gap: you added TIMEOUT, ABORTED, and RESPONSE_TOO_LARGE to the error-code union and wrote classification logic for them, but there's no test exercising any of those three paths — exactly the paths most likely to regress given point 2.

Also worth a nested-table caveat: the regex-based <table>...</table> matcher will misbehave on nested tables (rare in practice, but worth a comment/guard).

Address the pipe-escaping bug and add tests for the untested error codes and this becomes straightforward to port.

@codebuff-team codebuff-team added bot:triaged Classified by the community triage bot pr:needs-work Right idea, not mergeable as written labels Aug 25, 2026
@victorxheng

ghost commented Aug 31, 2026

Copy link
Copy Markdown

Apologies — this PR was auto-closed by GitHub when we force-pushed a history rewrite of this repository (repository maintenance; every commit SHA changed). That was not a judgment on this PR, and GitHub does not allow us to reopen it because the commits it was based on no longer exist in the new history.

If you'd like to continue with this change: rebase your branch onto the new main (or recreate it from a fresh clone) and open a new PR — feel free to link back to this one for context, and we'll pick up the review there.

Sorry for the churn, and thanks for contributing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:triaged Classified by the community triage bot pr:needs-work Right idea, not mergeable as written

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve error diagnostics and tabular content extraction in read_url tool

3 participants