Skip to content

A tree-less Thrift IDL lane in the code index - #760

Open
rahulsjha wants to merge 18 commits into
ClickHouse:mainfrom
rahulsjha:feat/thrift-lane
Open

rahulsjha wants to merge 18 commits into
ClickHouse:mainfrom
rahulsjha:feat/thrift-lane

Conversation

@rahulsjha

@rahulsjha rahulsjha commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes: #624

Changelog category (leave one):

  • New Feature

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

The code index reads Thrift IDL files: structs, unions, exceptions, enums, typedefs, services and their functions become symbols, field and argument types become type-use edges, a service's extends becomes an inherits edge, and include "x.thrift" becomes an include.

Why

.thrift was in _CODE_TEXT_EXTENSIONS, so Thrift files were chunked and searchable but code.symbol, code.callers and code.outline knew nothing in them.

Scope and risk

A tree-less lane, following docs/development/adding-language-support.md and shaped like lang_proto.py:

  • server/code/domain/extract/lang_thrift.py: parse_thrift(path, source) -> ParsedFile, reference_sites, SITE_RULES, CHUNK_OPTIONS, wanted_paths / resolve_imports.
    • Symbols: struct / union / exception → struct; enum → enum; typedef T X → type; service → interface; each service function → function <mod>::Service.f whose signature is the function as written (Machine get(1: MachineId id) throws(1: shared.NotFound not_found)). A token scanner with one-pass brace matching; undotted root-file scopes retain .thrift so the shared script-name gate accepts them. Comments (//, #, /* */) and strings are skipped, so a definition inside a comment is not a symbol.
    • Sites: a field's, argument's, return's or throws entry's named type (field_type → uses_type, to struct / enum / type), and a service's extends (inherits, to an interface), bound by lane_units.bind_sites (same file 1.0, repository-unique 0.6, ambiguous → nothing). list<> / set<> / map<> are read through; base types (i32, string, binary, …) name nothing; an include-qualified shared.User binds by its last part.
    • Includes: include "shared.thrift" → the including file's directory and its nearest ancestors, then the root (lane_units.ancestor_candidates), settled against the namespace's live paths.
  • Registration: ".thrift": "thrift" in LANG_BY_EXTENSION, removed from _CODE_TEXT_EXTENSIONS; _parse_thrift in _PARSERS, thrift in LANE_UNIT_LANGS and _lane_module; "thrift": SCRIPT in name_scope; thrift in search_scope.
  • Extractor stamp: lang_thrift.py is listed in NON_EXTRACTOR_MODULES and no stamped module changes, so existing deployments re-chunk nothing. A namespace that already indexed Thrift as text gains the rows with loom code backfill --paths-file of its .thrift files.
  • Cost: an unclosed string or block comment runs to the end once; a struct header only accepts { (or xsd_all {) after its name, and an unfinished typedef skips what it read, so repeated unclosed headers cost their own length.

Compatibility, rollout and rollback

  • Wire: none (new rows for .thrift files only).
  • Operator steps: none required; a backfill of .thrift files gives an already-indexed namespace its Thrift symbols and edges.
  • Rollback: reverting returns .thrift to plain text; the Thrift rows written meanwhile stay until the files are re-indexed.

Tests

  • tests/unit/test_code_lang_thrift.py (21 test definitions): kinds and qualified names; spans and signatures; sites through containers and include-qualified names, no base types, nothing from a comment; includes deduplicated and resolved nearest first; unterminated input; registration (detection, LANE_UNIT_LANGS, _lane_module); binding (same file, repository-unique, extends → inherits, throws, builtins bind to nothing); three linear-time cases (assert_linear). Additional regressions cover root-file cross-file binding through the production name gate while excluding native candidates, function and type annotations, annotated aliases and return signatures, opaque native type hints, signed and leading-dot defaults and field IDs, and deeply nested attributes in source order.
  • tests/fixtures/code_lanes/thrift/acme/ (synthetic, listed in NOTICE.md): machines.thrift includes shared.thrift and its service extends BaseService. Type names don't reuse the proto fixture's (Instance), which would make proto's repository-wide binding ambiguous. expected.json regenerated with python -m tests.code_lanes.corpus --write; its diff is only the two Thrift files (160 → 165 edge keys, 468 → 485 total edges: machines.thrift gains 1 include, 1 inherits and 15 uses_type edges).
  • SAMPLES in tests/unit/lane_walks.py gains thrift, plus a thrift-includes walk, for test_code_regex_cost.py.

Historical contributor-reported checks in Linux (python:3.12-bookworm, uv sync --frozen --all-extras); these were not rerun during maintainer continuation:

pytest tests/unit/test_code_lang_thrift.py test_code_extractor_stamp.py test_code_lanes_corpus.py
       test_code_lanes_manifest.py test_code_regex_cost.py                                   967 passed
all tests/unit/test_code_*.py + test_lang_*.py, main vs branch back to back in one container:
  main:   1 failed, 5152 passed (cpp-deep-includes, timing)
  branch: timing-related failures were reported, including lang_hcl._REF_RE; their causes were not established here
ruff format / ruff check / mypy lang_thrift.py                                               clean

the lane on the fixture: symbols with spans and signatures, sites, include resolution · test logs and the expected.json diff

Root-file scopes retain .thrift when otherwise undotted, so they pass the existing script-name gate without admitting native candidates. An explicit stack traverses nested attributes in source order using the precomputed brace map. Function and type annotations, native type hints and complete numeric constants are consumed without creating metadata dependencies or hiding the following field. The unused token-peek helper is removed. Shared name-class rules, extraction stamps and cost thresholds are unchanged.

The changed parser and regression module pass Ruff, formatting and whitespace checks. Local execution was refused by admission before launch; it supplies no local RED or GREEN result. Hosted unit part 1 on 74f72ee90c passes all 36 Thrift cases, with no failures, skips or reruns. The parser and test bytes are identical here, but full CI and fresh whole-head reviews must qualify this new main composition.

Main includes the isolated erasure-fixture lease-table repair from #1076. This feature uses that exact fixture and changes no erasure setup or assertions. The standalone fixture regression remains in #1077; its source is preserved outside this feature diff. Current main's documentation is retained.

The historical 7a6e02a5ea rollup had 52 successes, two Pages publication skips and one Vercel team-authorization failure; it was not all green. Those results do not qualify this head. The authorization refusal is preserved; no account change, alternate authorization route or unchanged-head rerun was attempted.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): the contributor implemented the lane and reported the historical checks above. Codex preserved the reviewed source, composed current main and the test prerequisite, and arranged fresh independent reviews and hosted CI.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • make format-check, make lint, make typecheck and make test-unit pass; the integration suite passes if server behaviour changed. (ruff format/check, mypy on the module, and the code-lane tests above were run; the full make test-unit was not)
  • Generated files are regenerated in this PR, not by hand. (expected.json regenerated; no other generated file changes)
  • A new migration is additive and numbered after the latest on main; no merged migration is edited. (n/a)
  • A behaviour change to any file a plugin ships bumps both plugin manifests' version. (n/a)
  • The frontend builds and its tests pass, if frontend/ changed. (n/a)
  • Docs under docs/ describe the system as it now is. (code-index overview language table; code intelligence by language; adding-language-support lane list)
  • Repository text has no dates, people, host names, internal namespaces, ticket or decision numbers.
  • Commits are signed off (git commit -s), as CONTRIBUTING.md asks. Existing contributor history contains unsigned commits; their sign-offs remain unresolved.

lang_thrift.py reads structs, unions, exceptions, enums, typedefs, services
and service functions with a token scanner in the shape of lang_proto.py.
Field, argument, return and throws types (through list/set/map) and a
service's extends are reference sites bound by lane_units.bind_sites; base
types bind to nothing. Includes resolve against the including file's
directory and its ancestors. .thrift leaves the plain-text list.

Closes ClickHouse#624

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 18:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

rahulsjha and others added 2 commits November 18, 2026 05:27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
@zlareb1 zlareb1 self-assigned this Nov 18, 2026

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented-out Thrift includes become real dependency edges. The inline example reproduces the error on this head. All 212 focused lane, extractor-stamp, manifest and inventory tests pass, but they do not cover includes inside block comments.

@zlareb1

zlareb1 commented Nov 19, 2026

Copy link
Copy Markdown
Member

This head has merge conflicts with main in docs/development/adding-language-support.md, server/code/app/index/ingest_parse.py, tests/code_lanes/corpus.py and tests/fixtures/code_lanes/NOTICE.md. Please rebase and resolve those conflicts so CI can validate the combined changes. GitHub reports mergeable: CONFLICTING; a local merge-tree check confirms these four files.

# Conflicts:
#	docs/development/adding-language-support.md
#	server/code/app/index/ingest_parse.py
#	tests/code_lanes/corpus.py
#	tests/fixtures/code_lanes/NOTICE.md
An include inside a block comment was read by the raw-source regex and became a
dependency edge. Includes now come from the token stream, which drops comments.
@rahulsjha

Copy link
Copy Markdown
Contributor Author

Thanks @zlareb1. Both are fixed:

  • Includes in comments (e78d117). parse() now reads an include from the token stream: an include token followed by a closed string literal. The comment-aware tokenizer has already dropped comments by then. Your example now returns includes == []. The raw-source _INCLUDE_RE is gone. test_includes_inside_comments_are_not_includes covers a block comment, //, #, an include followed by a comment on the same line, and an unclosed literal.
  • Conflicts (26ab218). I merged main (4cee575) into the branch. All four conflicts came from the Lua lane landing in the same lists, and I kept both entries in each: LANE_UNIT_LANGS and _lane_module in ingest_parse.py, the three lane lists and _lane_module in tests/code_lanes/corpus.py, the NOTICE row, and the docs table, which I re-formatted with prettier 3.9.9.

In a Linux container, the Thrift, Lua, extractor-stamp, manifest, corpus, known-gaps and regex-cost tests give 991 passed and 1 failed. The failure is test_a_lane_walk_costs_linear_time[cpp-deep-includes], the C++ lane, which this PR doesn't touch. My machine was running another container at the time, and the walk hit the test's 60 s deadline. Timed back to back, the 4N walk is the same on both trees: 2.97 s on main and 2.92 s on this branch. A cold run on main also took 81 s, so the failure comes from the load, not from this change.

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commented-include finding is fixed. A separate valid field form still loses dependency edges: the parser reads field types only after a numeric ID. The focused parser, corpus, stamp, manifest and inventory checks pass; the inline reproduction is missing from that coverage.

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional-field-ID finding remains on this head. The parser is unchanged: struct User { 1: string id }\nstruct Envelope { optional User user } still yields no type sites, while adding 1: before optional yields the expected User site. All 215 focused lane, stamp, manifest and inventory tests pass; they still omit this valid field form.

The IDL grammar makes FieldID optional in struct fields, argument lists and
throws lists. _fields now reads each field as [ID:] [req] Type [&] name
[= value] [xsd options] [(annotations)], so an ID-less field's type is a
reference site, and default values and annotations are skipped.

Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>

# Conflicts:
#	docs/development/adding-language-support.md
#	server/code/app/index/ingest_parse.py
#	server/code/domain/extract/lang_detect.py
#	server/code/domain/extract/lang_text.py
#	tests/code_lanes/corpus.py
…ric ID

Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional-field-ID finding is fixed, including argument and throws types, defaults and annotations. No further blocking findings after checking parser registration, include resolution, binding, corpus output and extractor-stamp isolation. All 231 focused lane, corpus, known-gap, stamp, manifest and Thrift cost checks pass on Linux.

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A remaining plain-text classification assertion fails in CI and locally on this head. The new lane needs the corresponding test inventory update before merge; the earlier field-ID fix remains verified.

A .thrift file now goes to the Thrift lane, so assert that the ingest
dispatcher picks the lane and its service symbols instead.

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plain-text classification finding is fixed: the regression checks that the production dispatcher selects the Thrift lane and its service and function symbols. No further blocking findings after checking registration, binding, includes and extractor-stamp isolation. All 220 focused parser, dispatcher, manifest and stamp checks pass. The broader local corpus and cost checks stop at macOS resource-limit failures; Linux CI covers those checks.

@zlareb1

zlareb1 commented Nov 20, 2026

Copy link
Copy Markdown
Member

check CI failures

@vercel

vercel Bot commented Nov 20, 2026

Copy link
Copy Markdown

@rahulsjha is attempting to deploy a commit to the ClickHouse Team on Vercel.

A member of the Team first needs to authorize it.

@rahulsjha

Copy link
Copy Markdown
Contributor Author

Thanks @zlareb1, I looked at both failures.

  • ci / pytest unit part 5: 7900 passed, 1 failed: test_code_regex_cost.py::test_a_lane_walk_costs_linear_time[setup-cfg-entry-points-of-one-module]. That is the setup.cfg lane's timing check (2.79 s against its limit on the runner). This PR doesn't touch the setup.cfg lane or the regex cost test, so I read it as runner timing.
  • docker-smoke / Cell trigger end-to-end acceptance: the fixture reports Refused in its measure phase (fixture.py:920, monitor). This PR doesn't touch the cell trigger, and the same job fails the same way on other open PRs.

I've merged the latest main into the branch so CI reruns on a current base.

Fenced memory inserts require a lease table in the selected schema. Give each fixture database its own Keeper root while preserving the replicated content checks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 20, 2026 11:03pm UTC

Request Review

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking source findings. Checked parser registration, type metadata, root-file binding, signed defaults, annotations, nested attributes, includes and extractor-stamp isolation. All 258 focused Linux checks and four corpus checks pass. One earlier timing sample exceeded its bound; a repeat passes, and an isolated CPU probe scales from 0.010 s at 82 KB to 0.694 s at 5.25 MB across three fourfold input increases. Hosted CI remains a merge gate.

The fixture repair is on main. Its standalone regression remains in ClickHouse#1077; the feature diff retains its original tests.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

@zlareb1 zlareb1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings. Checked Thrift tokenization, field and return types, metadata, binding, include resolution, registration and extractor-stamp isolation against the current base. All 258 focused parser, dispatcher, manifest, known-gap and corpus checks pass on Linux. The Cell acceptance workflow remains a merge gate.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a tree-less Thrift IDL lane: the code index reads .thrift files as text only

3 participants