Skip to content

fix(query_engine): treat Unicode whitespace as whitespace and report non-ASCII errors - #1816

Merged
fengelniederhammer merged 5 commits into
mainfrom
1791-queries-can-fail-with-non-ascii-characters-outside-string-literals
Nov 18, 2026
Merged

fengelniederhammer merged 5 commits into
mainfrom
1791-queries-can-fail-with-non-ascii-characters-outside-string-literals

Conversation

@fengelniederhammer

@fengelniederhammer fengelniederhammer commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

resolves #1791

Summary

Queries with a non-breaking space (or other Unicode whitespace) outside string literals no longer fail to parse, and non-ASCII / invalid-UTF-8 input now produces clear error messages instead of a garbled 400 response body.

  • Unicode whitespace — the lexer skips the full Unicode White_Space set (nbsp, em space, ideographic space, NEL, etc.), not just ASCII space/tab/newline. A leading UTF-8 BOM is ignored.
  • Clear errors — non-ASCII outside string literals is rejected with Unexpected character U+XXXX; invalid UTF-8 with Invalid UTF-8 byte 0xNN. Both are plain ASCII, so the 400 body stays valid UTF-8. Invalid UTF-8 is rejected everywhere, including inside string literals and comments.
  • Decoding — bytes ≥ 0x80 are decoded on demand via simdutf; the ASCII path is unchanged. Columns now count code points, not bytes. Identifiers remain ASCII-only.

PR Checklist

  • All necessary documentation has been adapted or there is an issue to do so.
  • The implemented feature is covered by an appropriate test.

@fengelniederhammer
fengelniederhammer requested a balanced review from Copilot November 10, 2026 19:55
@fengelniederhammer fengelniederhammer changed the title fix: unicode whitespace, non-ASCII lexer errors fix(query_engine): treat Unicode whitespace as whitespace and report non-ASCII errors clearly Nov 10, 2026
@fengelniederhammer fengelniederhammer changed the title fix(query_engine): treat Unicode whitespace as whitespace and report non-ASCII errors clearly fix(query_engine): treat Unicode whitespace as whitespace and report non-ASCII errors Nov 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the Unicode lexer changes.

No open issues 🎉

What changed in this PR

Updates the SaneQL lexer to handle Unicode whitespace and produce readable errors for unsupported characters and invalid UTF-8, addressing #1791.

Changes:

  • Accepts Unicode whitespace and ignores a leading UTF-8 byte order mark.
  • Validates non-ASCII input, reports ASCII-safe errors, and counts columns by code point.
  • Adds lexer and query-level regression tests and documents UTF-8 requirements.
File Description
src/​rhydb/​query_engine/​saneql/​lexer.test.cpp Tests Unicode whitespace, UTF-8 validation, errors, and column tracking.
src/​rhydb/​query_engine/​saneql/​lexer.h Declares code-point decoding and whitespace helpers.
src/​rhydb/​query_engine/​saneql/​lexer.cpp Implements Unicode handling, BOM skipping, and clear diagnostics.
src/​rhydb/​query_engine/​saneql/​lexer_unicode.test.cpp Adds query-level tests for Unicode input and invalid UTF-8.
documentation/​query_documentation.md Documents UTF-8, leading BOM, and Unicode whitespace support.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The review identifies no active issues with the Unicode input-handling changes.

No open issues 🎉

Review effort: Balanced

@fengelniederhammer
fengelniederhammer marked this pull request as ready for review November 11, 2026 08:17
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

This is a preview of the changelog of the next release. If this branch is not up-to-date with the current main branch, the changelog may not be accurate. Rebase your branch on the main branch to get the most accurate changelog.

Note that this might contain changes that are on main, but not yet released.

Changelog:

0.17.6 (2026-11-18)

Features

  • query_engine: add type casts at query execution time (#1799) (f7acdf0)

Bug Fixes

  • query_engine: do not decompress every row before applying a limit (#1833) (d3c06a2)
  • query_engine: evaluate out-of-bound .at() expressions to null for null inputs (#1839) (511d2ee)

@taepper

taepper commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Do we know why we did not get a proper error message before? The result was not even valid json

Comment thread src/rhydb/query_engine/saneql/lexer.h Resolved
Comment thread src/rhydb/query_engine/saneql/lexer.h Resolved
Comment thread src/rhydb/query_engine/saneql/lexer.cpp Resolved
Comment thread src/rhydb/query_engine/saneql/lexer.cpp Resolved
Comment thread src/rhydb/query_engine/saneql/lexer.cpp Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the Unicode whitespace and UTF-8 lexer changes.

No open issues 🎉

Review effort: Balanced

@taepper taepper left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@fengelniederhammer
fengelniederhammer merged commit edc1c02 into main Nov 18, 2026
37 of 38 checks passed
@fengelniederhammer
fengelniederhammer deleted the 1791-queries-can-fail-with-non-ascii-characters-outside-string-literals branch November 18, 2026 09:13
taepper pushed a commit that referenced this pull request Nov 18, 2026
🤖 I have created a release *beep* *boop*
---


## [0.17.6](v0.17.5...v0.17.6)
(2026-11-18)


### Features

* **query_engine:** add type casts at query execution time
([#1799](#1799))
([f7acdf0](f7acdf0))


### Bug Fixes

* **query_engine:** do not decompress every row before applying a limit
([#1833](#1833))
([d3c06a2](d3c06a2))
* **query_engine:** evaluate out-of-bound `.at()` expressions to null
for null inputs ([#1839](#1839))
([511d2ee](511d2ee))
* **query_engine:** limit the nesting of queries
([#1828](#1828))
([599d3a4](599d3a4))
* **query_engine:** treat Unicode whitespace as whitespace and report
non-ASCII errors ([#1816](#1816))
([edc1c02](edc1c02))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Queries can fail with non-ascii characters outside string literals

3 participants