Repository navigation
Conversation
lang_thrift.py reads structs, unions, exceptions, enums, typedefs, services and service functions with a token scanner in the shape of lang_proto.py. Field, argument, return and throws types (through list/set/map) and a service's extends are reference sites bound by lane_units.bind_sites; base types bind to nothing. Includes resolve against the including file's directory and its ancestors. .thrift leaves the plain-text list. Closes ClickHouse#624 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
zlareb1
left a comment
There was a problem hiding this comment.
Commented-out Thrift includes become real dependency edges. The inline example reproduces the error on this head. All 212 focused lane, extractor-stamp, manifest and inventory tests pass, but they do not cover includes inside block comments.
|
This head has merge conflicts with |
# Conflicts: # docs/development/adding-language-support.md # server/code/app/index/ingest_parse.py # tests/code_lanes/corpus.py # tests/fixtures/code_lanes/NOTICE.md
An include inside a block comment was read by the raw-source regex and became a dependency edge. Includes now come from the token stream, which drops comments.
|
Thanks @zlareb1. Both are fixed:
In a Linux container, the Thrift, Lua, extractor-stamp, manifest, corpus, known-gaps and regex-cost tests give 991 passed and 1 failed. The failure is |
zlareb1
left a comment
There was a problem hiding this comment.
The commented-include finding is fixed. A separate valid field form still loses dependency edges: the parser reads field types only after a numeric ID. The focused parser, corpus, stamp, manifest and inventory checks pass; the inline reproduction is missing from that coverage.
zlareb1
left a comment
There was a problem hiding this comment.
The optional-field-ID finding remains on this head. The parser is unchanged: struct User { 1: string id }\nstruct Envelope { optional User user } still yields no type sites, while adding 1: before optional yields the expected User site. All 215 focused lane, stamp, manifest and inventory tests pass; they still omit this valid field form.
The IDL grammar makes FieldID optional in struct fields, argument lists and throws lists. _fields now reads each field as [ID:] [req] Type [&] name [= value] [xsd options] [(annotations)], so an ID-less field's type is a reference site, and default values and annotations are skipped. Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com> # Conflicts: # docs/development/adding-language-support.md # server/code/app/index/ingest_parse.py # server/code/domain/extract/lang_detect.py # server/code/domain/extract/lang_text.py # tests/code_lanes/corpus.py
…ric ID Signed-off-by: Rahul <132360698+rahulsjha@users.noreply.github.com>
zlareb1
left a comment
There was a problem hiding this comment.
The optional-field-ID finding is fixed, including argument and throws types, defaults and annotations. No further blocking findings after checking parser registration, include resolution, binding, corpus output and extractor-stamp isolation. All 231 focused lane, corpus, known-gap, stamp, manifest and Thrift cost checks pass on Linux.
zlareb1
left a comment
There was a problem hiding this comment.
A remaining plain-text classification assertion fails in CI and locally on this head. The new lane needs the corresponding test inventory update before merge; the earlier field-ID fix remains verified.
A .thrift file now goes to the Thrift lane, so assert that the ingest dispatcher picks the lane and its service symbols instead.
zlareb1
left a comment
There was a problem hiding this comment.
The plain-text classification finding is fixed: the regression checks that the production dispatcher selects the Thrift lane and its service and function symbols. No further blocking findings after checking registration, binding, includes and extractor-stamp isolation. All 220 focused parser, dispatcher, manifest and stamp checks pass. The broader local corpus and cost checks stop at macOS resource-limit failures; Linux CI covers those checks.
|
check CI failures |
|
@rahulsjha is attempting to deploy a commit to the ClickHouse Team on Vercel. A member of the Team first needs to authorize it. |
|
Thanks @zlareb1, I looked at both failures.
I've merged the latest main into the branch so CI reruns on a current base. |
Fenced memory inserts require a lease table in the selected schema. Give each fixture database its own Keeper root while preserving the replicated content checks. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
zlareb1
left a comment
There was a problem hiding this comment.
No blocking source findings. Checked parser registration, type metadata, root-file binding, signed defaults, annotations, nested attributes, includes and extractor-stamp isolation. All 258 focused Linux checks and four corpus checks pass. One earlier timing sample exceeded its bound; a repeat passes, and an isolated CPU probe scales from 0.010 s at 82 KB to 0.694 s at 5.25 MB across three fourfold input increases. Hosted CI remains a merge gate.
The fixture repair is on main. Its standalone regression remains in ClickHouse#1077; the feature diff retains its original tests. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
zlareb1
left a comment
There was a problem hiding this comment.
No blocking findings. Checked Thrift tokenization, field and return types, metadata, binding, include resolution, registration and extractor-stamp isolation against the current base. All 258 focused parser, dispatcher, manifest, known-gap and corpus checks pass on Linux. The Cell acceptance workflow remains a merge gate.
Closes: #624
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
The code index reads Thrift IDL files: structs, unions, exceptions, enums, typedefs, services and their functions become symbols, field and argument types become type-use edges, a service's
extendsbecomes an inherits edge, andinclude "x.thrift"becomes an include.Why
.thriftwas in_CODE_TEXT_EXTENSIONS, so Thrift files were chunked and searchable butcode.symbol,code.callersandcode.outlineknew nothing in them.Scope and risk
A tree-less lane, following
docs/development/adding-language-support.mdand shaped likelang_proto.py:server/code/domain/extract/lang_thrift.py:parse_thrift(path, source) -> ParsedFile,reference_sites,SITE_RULES,CHUNK_OPTIONS,wanted_paths/resolve_imports.struct/union/exception→struct;enum→enum;typedef T X→type;service→interface; each service function →function<mod>::Service.fwhose signature is the function as written (Machine get(1: MachineId id) throws(1: shared.NotFound not_found)). A token scanner with one-pass brace matching; undotted root-file scopes retain.thriftso the shared script-name gate accepts them. Comments (//,#,/* */) and strings are skipped, so a definition inside a comment is not a symbol.throwsentry's named type (field_type→uses_type, to struct / enum / type), and a service'sextends(inherits, to an interface), bound bylane_units.bind_sites(same file 1.0, repository-unique 0.6, ambiguous → nothing).list<>/set<>/map<>are read through; base types (i32,string,binary, …) name nothing; an include-qualifiedshared.Userbinds by its last part.include "shared.thrift"→ the including file's directory and its nearest ancestors, then the root (lane_units.ancestor_candidates), settled against the namespace's live paths.".thrift": "thrift"inLANG_BY_EXTENSION, removed from_CODE_TEXT_EXTENSIONS;_parse_thriftin_PARSERS,thriftinLANE_UNIT_LANGSand_lane_module;"thrift": SCRIPTinname_scope;thriftinsearch_scope.lang_thrift.pyis listed inNON_EXTRACTOR_MODULESand no stamped module changes, so existing deployments re-chunk nothing. A namespace that already indexed Thrift as text gains the rows withloom code backfill --paths-fileof its.thriftfiles.{(orxsd_all {) after its name, and an unfinishedtypedefskips what it read, so repeated unclosed headers cost their own length.Compatibility, rollout and rollback
.thriftfiles only)..thriftfiles gives an already-indexed namespace its Thrift symbols and edges..thriftto plain text; the Thrift rows written meanwhile stay until the files are re-indexed.Tests
tests/unit/test_code_lang_thrift.py(21 test definitions): kinds and qualified names; spans and signatures; sites through containers and include-qualified names, no base types, nothing from a comment; includes deduplicated and resolved nearest first; unterminated input; registration (detection,LANE_UNIT_LANGS,_lane_module); binding (same file, repository-unique,extends→ inherits,throws, builtins bind to nothing); three linear-time cases (assert_linear). Additional regressions cover root-file cross-file binding through the production name gate while excluding native candidates, function and type annotations, annotated aliases and return signatures, opaque native type hints, signed and leading-dot defaults and field IDs, and deeply nested attributes in source order.tests/fixtures/code_lanes/thrift/acme/(synthetic, listed inNOTICE.md):machines.thriftincludesshared.thriftand its service extendsBaseService. Type names don't reuse the proto fixture's (Instance), which would make proto's repository-wide binding ambiguous.expected.jsonregenerated withpython -m tests.code_lanes.corpus --write; its diff is only the two Thrift files (160 → 165 edge keys, 468 → 485 total edges:machines.thriftgains 1 include, 1 inherits and 15 uses_type edges).SAMPLESintests/unit/lane_walks.pygainsthrift, plus athrift-includeswalk, fortest_code_regex_cost.py.Historical contributor-reported checks in Linux (
python:3.12-bookworm,uv sync --frozen --all-extras); these were not rerun during maintainer continuation:the lane on the fixture: symbols with spans and signatures, sites, include resolution · test logs and the expected.json diff
Root-file scopes retain
.thriftwhen otherwise undotted, so they pass the existing script-name gate without admitting native candidates. An explicit stack traverses nested attributes in source order using the precomputed brace map. Function and type annotations, native type hints and complete numeric constants are consumed without creating metadata dependencies or hiding the following field. The unused token-peek helper is removed. Shared name-class rules, extraction stamps and cost thresholds are unchanged.The changed parser and regression module pass Ruff, formatting and whitespace checks. Local execution was refused by admission before launch; it supplies no local RED or GREEN result. Hosted unit part 1 on
74f72ee90cpasses all 36 Thrift cases, with no failures, skips or reruns. The parser and test bytes are identical here, but full CI and fresh whole-head reviews must qualify this new main composition.Main includes the isolated erasure-fixture lease-table repair from #1076. This feature uses that exact fixture and changes no erasure setup or assertions. The standalone fixture regression remains in #1077; its source is preserved outside this feature diff. Current main's documentation is retained.
The historical
7a6e02a5earollup had 52 successes, two Pages publication skips and one Vercel team-authorization failure; it was not all green. Those results do not qualify this head. The authorization refusal is preserved; no account change, alternate authorization route or unchanged-head rerun was attempted.AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (an agent wrote most of the diff): the contributor implemented the lane and reported the historical checks above. Codex preserved the reviewed source, composed current main and the test prerequisite, and arranged fresh independent reviews and hosted CI.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist
make format-check,make lint,make typecheckandmake test-unitpass; the integration suite passes if server behaviour changed. (ruff format/check, mypy on the module, and the code-lane tests above were run; the fullmake test-unitwas not)expected.jsonregenerated; no other generated file changes)main; no merged migration is edited. (n/a)frontend/changed. (n/a)docs/describe the system as it now is. (code-index overview language table; code intelligence by language; adding-language-support lane list)git commit -s), as CONTRIBUTING.md asks. Existing contributor history contains unsigned commits; their sign-offs remain unresolved.