@@ -16,3 +16,229 @@ To debug the parser script:
1616
17172 . Open VSCode and choose "Debug: Attach to Node process" from the command palette, and
1818 choose the process that looks like the parser-wrapper.
19+
20+ Building and running
21+ --------------------
22+
23+ The wrapper uses the exact ` typescript@7.1.0-dev.20261015.1 ` package for both
24+ compilation/type checking and the native runtime. Its integrity and those of all
25+ platform-specific optional dependencies are recorded in ` package-lock.json ` .
26+
27+ ``` sh
28+ npm ci
29+ npm run build
30+ npm test
31+ node build/main.js --version
32+ ```
33+
34+ Node.js 18.15.0, the existing Bazel toolchain version, supports the wrapper and
35+ native API; no Node.js upgrade is required. Build scripts invoke TypeScript's
36+ ` lib/tsc.js ` launcher explicitly because Node.js 18.15.0 cannot load its
37+ extensionless ESM ` bin/tsc ` entry point. The installed wrapper must include its
38+ ` package.json ` to retain the ESM package boundary.
39+
40+ The existing TSLint rules require the legacy JavaScript compiler API, not the
41+ native TypeScript 7 API. If compatible TSLint tooling is not already installed,
42+ run it separately without replacing the wrapper's build/runtime dependency:
43+
44+ ``` sh
45+ npm exec --yes --package=tslint@6.1.3 --package=typescript@5.9.2 -- tslint --project .
46+ ```
47+
48+ ` SEMMLE_TYPESCRIPT_HOME ` still overrides the runtime dependency. It must point to
49+ the installed TypeScript package directory (containing ` package.json ` ), with the
50+ native ` typescript/sync ` and ` typescript/ast ` entry points and matching executable.
51+ A legacy 5.x module or a directory from which the bundled package merely happens
52+ to be resolvable is not a valid override.
53+
54+ Distribution packaging
55+ ----------------------
56+
57+ The Bazel build produces two archives: ` :typescript ` (` javascript.zip ` ) contains
58+ the compiled ESM wrapper and JavaScript API; ` :native ` (` native.zip ` ) contains the
59+ matching native packages for the target CodeQL distribution:
60+
61+ | Distribution | Native packages |
62+ | --- | --- |
63+ | ` linux64 ` | ` @typescript/typescript-linux-x64 ` |
64+ | ` linux-arm64 ` | ` @typescript/typescript-linux-arm64 ` |
65+ | ` osx64 ` | ` @typescript/typescript-darwin-arm64 ` and ` @typescript/typescript-darwin-x64 ` |
66+ | ` win64 ` | ` @typescript/typescript-win32-x64 ` |
67+
68+ The build installs the selected packages explicitly with ` --force --no-save ` ,
69+ using the pinned versions and integrities in the lockfile. Merely passing
70+ ` --force ` to ` npm ci ` still filters optional dependencies by the build host.
71+ ` package-runtime.mjs ` verifies the native package versions and marks executables
72+ as ` 0755 ` before ZIP creation. The common CodeQL archive contains the wrapper;
73+ the architecture archive installs native packages into
74+ ` tools/typescript-parser-wrapper/node_modules/@typescript ` . Both archives must
75+ be installed, with executable permissions preserved. No separate TypeScript
76+ installation is needed.
77+
78+ Run the installed-package smoke test with:
79+
80+ ``` sh
81+ bazel test //javascript/extractor/lib/typescript:package-test
82+ ```
83+
84+ It checks all distribution selections, mismatched runtime rejection, common/arch
85+ ZIP separation and executable metadata, then installs the merged archives with
86+ the production ` ripunzip ` and parses TS, TSX, and malformed source through the
87+ installed JSON protocol. It runs on the host platform and uses Bazel's Node.js
88+ runtime. The Java extractor test suite also installs both ZIPs and restores
89+ executable permissions using their central-directory metadata.
90+
91+ All four distribution runtime ZIPs have been cross-assembled and inspected.
92+ Installed parsing has been verified with Node.js 18.15.0 on macOS arm64 and on
93+ macOS x64 through Rosetta. Linux x64, Linux arm64, and Windows x64 still require
94+ execution on matching hosts; cross-assembling their ZIPs is not execution coverage.
95+
96+ Native lifecycle and memory
97+ ---------------------------
98+
99+ Each file owns a fresh native API/process. AST projection and JSON serialization
100+ finish before the program is disposed and the API is closed, including on
101+ failure. No native process or compiler cache is retained between files, during
102+ stdout backpressure, or across reset/quit. This deliberately favors bounded
103+ resource lifetime over reusing a compiler process; the measured cost is described
104+ below.
105+
106+ ` SEMMLE_TYPESCRIPT_MEMORY_THRESHOLD ` remains a positive megabyte budget, defaulting
107+ to 1000 when the Java extractor does not supply it. At native-process creation,
108+ the wrapper subtracts Node's current RSS (not just its V8 heap) from that budget
109+ and passes the remaining bytes, with a minimum of one, as the child's
110+ ` GOMEMLIMIT ` . The parent's original ` GOMEMLIMIT ` is restored after spawning.
111+ This is a Go garbage-collector soft limit, not a hard combined RSS ceiling:
112+ Node allocations during extraction, native non-Go allocations, and oversized
113+ individual files can exceed the budget. Closing the native process after every
114+ file prevents native caches and heap capacity from accumulating across files.
115+ Node RSS exceeding the budget produces a warning on stderr.
116+
117+ Native stderr is inherited separately from the JSON stdout protocol. Syntax
118+ diagnostics remain successful AST responses; native, I/O, and protocol failures
119+ are reported on stderr and terminate the wrapper with a nonzero exit status.
120+ Reset, quit, EOF, and failure invalidate queued prefetch work; quit also closes
121+ stdin without requiring the sender to close its pipe.
122+
123+ TypeScript 7.1 parser spike
124+ --------------------------
125+
126+ The ` ts7 ` directory contains an isolated parser compatibility probe using the
127+ published npm ` typescript@next ` package. The snapshot is pinned to
128+ ` 7.1.0-dev.20261015.1 ` in ` ts7/package.json ` , with package integrity recorded in
129+ ` ts7/package-lock.json ` .
130+
131+ From this directory, run:
132+
133+ ``` sh
134+ cd ts7
135+ npm ci --ignore-scripts
136+ npm test
137+ ```
138+
139+ Only Node.js and npm are needed; no source build or Go toolchain is required.
140+ The probe has its own installation, including TypeScript 5.9.2 for comparison.
141+ TypeScript's optional dependencies supply the host's matching native executable.
142+ To update the snapshot, verify that ` next ` still targets 7.1 and run
143+ ` npm install --save-exact typescript@next ` in ` ts7 ` , then rerun the probe and
144+ review the manifest and lockfile. The probe has only been run on macOS arm64;
145+ distribution packaging and its smoke test are described above.
146+
147+ If a minimum-release-age policy blocks the nightly, obtain approval for a
148+ command-local exception for ` typescript ` and ` @typescript/* ` rather than changing
149+ saved npm configuration.
150+
151+ The probe compares TS/TSX AST traversal and UTF-16 source positions, syntax-error
152+ messages and offsets, and scanner output with 5.9. It also exercises standalone
153+ source-file leases, program/snapshot disposal, repeated parsing, and API shutdown.
154+ For syntax diagnostics, use a single-file program with ` noResolve ` and ` noLib ` ;
155+ standalone source-file leases do not expose the old ` parseDiagnostics ` field.
156+ Compiler-option enums such as ` JsxEmit.Preserve ` must be passed as numbers, not
157+ tsconfig-style strings. Diagnostic ` text ` and ` pos ` must be mapped to the old
158+ ` messageText ` and ` start ` fields. AST properties must be explicitly projected
159+ before JSON serialization because the new nodes expose them through getters.
160+
161+ Wrapper source port
162+ -------------------
163+
164+ The wrapper targets the native ` typescript/sync ` and ` typescript/ast ` entry
165+ points, using the same pinned dependency in the production ESM build.
166+ The probe command above also runs
167+ ` wrapper.test.mjs ` , which transpiles the wrapper into an isolated ESM directory
168+ using the probe's 5.9 compiler and exercises the actual JSON protocol with the
169+ pinned 7.1 runtime.
170+
171+ AST nodes are explicitly projected into plain objects before their native
172+ program is disposed. The projection preserves the Java converter's property
173+ names, including optional member tokens, type-parameter defaults, import
174+ attributes, and namespace flags. EOF metadata retains the ` EndOfFileToken `
175+ spelling; the native ` TypeOfExpression ` , ` TypeOfKeyword ` , ` InstanceOfKeyword ` ,
176+ and ` KeyOfKeyword ` names already match the Java converter.
177+
178+ ` npm test ` in the wrapper directory instead exercises the compiled production
179+ output, copied with its ESM package boundary into a temporary installation. It
180+ covers repeated parses, native process disposal, the child memory budget,
181+ reset/quit/EOF, cancelled prefetch, runtime overrides, and fatal native failures.
182+
183+ Compatibility and regression coverage
184+ ------------------------------------
185+
186+ Native interface ` extends ` and class ` implements ` elements are type references.
187+ The wrapper projects them into the expression-shaped heritage nodes consumed by
188+ the Java converter, preserving generic arguments and qualified names.
189+
190+ The native parser still produces ASTs for legacy ` assert { ... } ` import
191+ assertions, but adds diagnostic 2880. The wrapper excludes only that diagnostic
192+ to preserve previously accepted syntax; genuine syntax diagnostics are unchanged.
193+ Import-attribute keys and values are now explicitly serialized. Their absence
194+ from the old wrapper's JSON was a bug: the Java converter expects ` key ` /` value ` ,
195+ but the old filter exposed neither correctly. The corresponding TRAP expectation
196+ was regenerated; existing QL expectations did not need changes.
197+
198+ Rescanning preserves token text and offsets while retaining the original
199+ pre-rescan token kinds on the wire. In particular, regex and template-tail tokens
200+ keep the classifications expected by existing TRAP tests.
201+
202+ Source-phase imports (` import source wasm from "./module.wasm" ` ) are accepted.
203+ Their declaration, imported path, and variable bindings use the existing import
204+ representation; the schema does not yet distinguish the source phase. The new
205+ ` TypeScript/SourceImports ` QL test covers these bindings.
206+
207+ Validation on October 19, 2026 passed the pinned-compiler build/type check,
208+ existing TSLint rules, seven compiled-output tests, 23 probe/compatibility tests,
209+ 500 Java tests, 68 TypeScript QL tests, the syntax-error query test, both extraction
210+ diagnostic integration tests, and the installed-package smoke test. The
211+ internal-error integration test deliberately terminates the native child rather
212+ than relying on the old parser's stack overflow.
213+
214+ For QL tests, install the newly built extractor and explicitly select it; otherwise
215+ the CLI may silently exercise an older installed extractor:
216+
217+ ``` sh
218+ # From the repository root, with the Java extractor build dependencies available:
219+ bazel run //javascript:install
220+ codeql resolve languages --search-path=javascript/extractor-pack:. --format=json
221+ codeql test run -j4 --search-path=javascript/extractor-pack:. \
222+ javascript/ql/test/library-tests/TypeScript \
223+ javascript/ql/test/query-tests/LanguageFeatures/SyntaxError
224+ ```
225+
226+ Performance comparison
227+ ----------------------
228+
229+ On macOS arm64 with Node.js 18.15.0, a comparison against the pre-migration
230+ wrapper from ` 1971c368e1d ` compiled with TypeScript 5.9.2 gave these medians over three
231+ runs:
232+
233+ | Workload | 5.9.2 time | 7.1 time | 5.9.2 combined RSS | 7.1 combined RSS |
234+ | --- | --- | --- | --- | --- |
235+ | 103 TypeScript fixtures (33,589 bytes) | 0.237 s | 1.349 s | 124.1 MiB | 113.9 MiB |
236+ | Five parses of TypeScript's 588,128-byte declaration file | 0.991 s | 1.116 s | 269.3 MiB | 249.1 MiB |
237+
238+ Timing covers prepare-files, extraction, serialization, and protocol responses
239+ after the metadata handshake, not Java conversion or database import. Combined
240+ RSS is the maximum sampled sum of Node and its native descendants, sampled every
241+ 5 ms; short-lived peaks may be missed. The per-file native process costs about
242+ 5.7x on tiny files and 1.13x on the declaration workload, while measured memory is
243+ slightly lower. The bounded lifecycle is retained deliberately; this migration
244+ does not claim a speedup.
0 commit comments