Repository navigation
feat!: v1, the new parser as the package root - #125
Merged
Merged
Conversation
Projects each engine's output into canonical records keyed by tree path, so engines that number events differently still compare. Joins the two sides like a merge of sorted lists, so a node only one side has is reported once. Times each log from its bytes in a fresh process, warm and cold, with retained heap. Writes results.jsonl and a Markdown report by size band.
--baseline=<results.jsonl> adds a section to the report. It states each engine's median per-log change in warm time, cold time and retained memory, against its own figures in the earlier run, over the logs both runs timed.
…uilds An end-to-end test on the corpus, in Node and Chromium, found bytes faster and never larger than a string. ADR 0005 supersedes the input choice in 0004.
One hand-written entry per event type (299) in src/next/catalog. It gives the public type info (EVENT_TYPES and eventType(), with a shape and a kind per type), the engine's rules per type, and a lazy text rule. The new parser imports nothing from the rest of src/. Catalog.test.ts fails when an entry drifts from the event database. Type ids are append-only. The build leaves src/next out until it is exported.
Add EVENT_LINES: for every event type, the Salesforce description and the name, format and meaning of each `|` field. Real logs confirm 118 layouts; the rest follow the Salesforce docs and are marked unobserved. EventTypeInfo exposes them as description, fields and observed.
Each ENTRIES entry lists its fields, and text rules name the fields they read; names resolve to positions once, at load, and the compiler rejects an unknown name. The docs for each type and field move to TSDoc on the type-only EventFields, so no bundle carries them. hasLineNumber now follows from a first field named `line`. Five rules read fields no real line states. VALIDATION_PASS, VF_EVALUATE_FORMULA_END and BULK_DML_RETRY no longer give text; WF_EMAIL_ALERT and WF_APPROVAL_SUBMITTER drop their empty parts.
Text rules and rewrites move from GRAMMAR into a private table behind eventText, so the engine cannot build text in another order. GRAMMAR keeps the scanner's facts, acceptsText among them. Fields becomes a cursor valid only during one call; its continuation() returns the continuation lines joined by '\n', called at most once and only for types that take text. Rules see LineFields, without continuation().
Two engines can differ in their types and names. The facts projection maps each result into one canonical shape: tree shape, times, counts, limits, issues and truncation. Text, cpuType and suffix are left out. --projection=facts|full selects it, and facts is the default. The diff key records which projection a run used.
The engine reads line numbers and counts on every line, so Fields gets lineNumber(i) and int(i, prefix): no text is decoded for them. Null means the field is absent or empty. NaN means the field states something else; the engine turns it into null and a parsing error. fieldsContract.ts is the suite every Fields adapter passes. fieldsOf runs it now, and the byte cursor will too. The name-to-position resolver moves out of textOf, so the fact rules can use it.
Source holds the two reads that differ between the node and browser builds (ADR 0005): line ends and UTF-8 text. NodeSource uses Buffer.indexOf and toString. BrowserSource finds line ends four bytes at a time, and builds short ASCII text without TextDecoder. ByteFields is the Fields cursor over the bytes. One cursor serves every line, finds field offsets only as far as a read needs, and reads line numbers and counts without decoding. typeIdAt finds a type id from the name's bytes. Both sources pass the Fields contract.
The catalog now states the facts a line adds to the tree: the counter it adds one to, the field that holds its rows or heap bytes, and how its namespace is found. Scan-time work that no declaration states is a named hook. grammarOf resolves each field name to its position at load, so the engine reads flat integers, and a name the entry lacks throws. lineField replaces hasLineNumber.
Store keeps one row per event in parallel typed-array columns, in log order (ADR 0002). Counts and heap figures go in sparse pools, because most rows have none. Columns start from the source size, grow by half and are trimmed when the parse ends, which also indexes the rows by type. StringTable interns byte ranges, so a column holds a number, and decodes each value once, on first read.
LogBuilder reads the log's bytes once and writes each event to the store. It keeps today's tree rules and today's order of side effects: line-number matching, unwinding, discontinuity, the maximum-size cut, next-line exits, a new execution, the package end and the VF leaf. The tree is built without recursion, so a deep log cannot overflow the stack. A matched exit line gets no row; it folds into its frame. Continuation lines extend the last event's byte range. The cursor drops the CRs, empty lines, markers and event-like lines inside that range, as today. Issues are deduplicated and ordered as today. A bad timestamp or line number is now a parsing error. Today the parse throws.
The builder reads each event's namespace from its line's bytes, by today's rule for its type, and interns it. An event that states none takes its frame's. A method entry takes the namespace its exit line states, also when that line unwinds it. Today's 'default' is no namespace, and blocks the frame's. When the next event is read, the last one's text is complete, so the hooks run then, as today: a limit block becomes a snapshot, a limit exception an error issue, and a fatal error a fatal issue. src/next/limits.ts copies today's limit text parsing and derivation, because src/next imports nothing from the rest of src/. StringTable.lookup finds a value without adding it.
Each row gets the counts, rows and heap figures its own line states. SOQL and SOSL frames take their rows from the line that closes or unwinds them, and the running heap is clamped at 0, as today. After the build, today's passes run in today's order: the log's times, the rollups (sums, the heap peak by max, heap self from leaves only, duration self by subtraction) and the flow residuals. Rows are in prefix order, so one pass down from the last id adds each subtree before its root. The package merge runs while the tree is built: an entry in the namespace of the run before it extends that run and gets no row. Known differences: every package entry's duration is its exit less its start; today some stay at 0. An issue after a merged entry points at the entry it merged into.
bytes/ascii.ts holds the ASCII bytes the engine tests for, with names that
say which bracket they are: OPEN meant '(' in the builder and '[' in the
cursor. It also holds the byte-range searches that namespaces.ts and the
cursor each kept a copy of. tags.ts holds the one tagOf, which four files
defined.
Output identical on 36 logs. Bundled A/B over 10 alternating rounds:
median -0.5% to +1.9%, against an A/A floor of +0.1% to +2.3%.
timestampClose, timestampIn, isTypeName, firstEventLine and lineText read only bytes, so they are functions beside the text-level line rules, with no builder state. Output identical on 36 logs. Bundled A/B over 10 rounds: median -1.4% to +1.4%.
LogMarkers holds what the log states about itself apart from its events: the issues, the parsing errors and the limit blocks. It runs on lines that are no event and no text, and on event types with a hook. The builder keeps the scan, the tree and the continuation fast path, and passes the last event's place to each call, so the order of side effects stays. builder.ts goes from 1,014 lines to 769. Output identical on 36 logs. Bundled A/B over 12 rounds: median -0.8% to +0.9%.
RollupView guarded each read with a NONE test, then called countsOf or heapOf, which give a row a pool slot when it has none. The guard alone kept the read safe. countIndex and heapIndex read the slot once and give -1 for a row with no slot, so the view needs no guard. Evidence: - pnpm run ci: 580 passed, 6 skipped. - Projection of 36 logs, node and browser engines: identical. - Read of every rollup and heap getter on a 50 MB log, 4 alternating rounds: min 6.13-6.19 ms before, 3.14-3.24 ms after.
Evidence: pnpm run ci: 580 passed, 6 skipped.
LimitValue.limit is null when the log states no ceiling, so a caller can tell a stated 0 from a missing ceiling. percentUsed stays null for both. Combined figures take the highest stated ceiling, and ignore null. BREAKING CHANGE: LimitValue.limit is number | null. Evidence: - The updated GovernorLimits and Heap tests failed on the old code (2 failed), and pass on the new code. - A new assertion: a stated 'Number of callouts: 0 out of 0' gives limit 0, not null. - pnpm run ci: 580 passed, 6 skipped.
AGENTS.md asks for a const beside each public union. KIND, SHAPE, CPU_TYPE, ISSUE_TYPE, PARSE_PHASE and TRUNCATION_KIND were missing. Each union type now derives from its const, as CATEGORY does. ParsePhase and TruncationKind are new named types for unions that were inline. DEBUG_CATEGORY_TOKEN maps each debug category to its log header token, the inverse of DEBUG_CATEGORY. The MCP server needs it for its labels (certinia/debug-log-analyzer-mcp#170). Evidence: - PublicApi.test.ts, updated first, failed on the old surface (2 failed, 3 type errors), and passes now. - A Catalog test checks each category maps back to its token. - pnpm run ci: 580 passed, 6 skipped.
The limit, user and debug level objects are deep-frozen at runtime, but their types let a caller write to them. They are readonly now: LimitValue, Limits, LimitSnapshot, NamespaceLimits, GovernorLimits, UserInfo, LogTimezone and DebugLevelSetting. snapshots is a readonly array and byNamespace a ReadonlyMap. On EventOf<T>, field(name) takes only the names EventFields[T] lists. Each shape keeps its own member, so isFrame still narrows. Evidence: - EventText.test.ts gains a @ts-expect-error on a wrong field name, and a check that isFrame narrows exitStamp to a number. On the old type, pnpm typecheck failed: Unused '@ts-expect-error' directive. - Emitted code of header.ts, limits.ts and events.ts, types and comments stripped: identical to the parent commit. The same check reports a difference against a commit that changed limits.ts. - tsc --extendedDiagnostics: 50,919 instantiations (control 49,058), check time 0.119 s (control 0.126 s). - pnpm run ci: 580 passed, 6 skipped.
AGENTS.md now names bytes/ascii.ts, bytes/lines.ts, tags.ts, engine/tables.ts and engine/markers.ts. Two comments named a plan step and a research brief that are not in the repository. The v0 comments stay: each one gives the reason for a rule that keeps the 0.x output. The ADRs stay as written: they are dated records. Evidence: pnpm run ci: 580 passed, 6 skipped.
With the legacy parser gone, the harness has one engine, and 'next' named it against a parser that no longer exists. --engines takes current. A shared formatter module was considered and dropped: the copies are two one-line ms and mb helpers, and median is already shared from large.ts. Evidence: - pnpm run ci: 580 passed, 6 skipped. - pnpm run compare on the 8 MB log, --engines=current,current --projection=full: 1 of 1 identical.
RollupView guarded each read with a NONE test, then called countsOf or heapOf, which give a row a pool slot when it has none. The guard alone kept the read safe. countIndex and heapIndex read the slot once and give -1 for a row with no slot, so the view needs no guard. Evidence: - pnpm run ci: 862 passed, 6 skipped. - Read of every rollup and heap getter on a 50 MB log, 4 alternating rounds against 07d4fba: min 5.95-6.00 ms before, 3.05-3.08 ms after. - The parse path is not changed.
# Conflicts: # src/store/store.ts
lcottercertinia
pushed a commit
that referenced
this pull request
Oct 9, 2026
## 📝 Overview Adds a new, experimental parser at `@apexdevtools/apex-log-parser/next`. It replaces today's parser in v1. Today's `parse` does not change. Against v0.2.0, on synthetic logs of 8 to 100 MB: 6–9× faster, and the parsed log keeps 8× less memory. ## 🛠️ Changes - `src/next/`: a standalone parser. It scans bytes into typed-array columns, builds the tree in one pass and reads events as views. It imports nothing from the rest of `src/`. - Async `parse` with 5 ms time slices, `signal` and `onProgress`. It reads a string, bytes, a `Blob`, a `Response`, a `ReadableStream` or an async iterable. - Optional `worker` option, and `toBuffers` / `fromBuffers` to move a parsed log between threads. - Node and browser builds through package `exports` (`./next`, `./next/worker`). - `benchmarks/`: every benchmark moved here, plus `compare`, `bench:views`, `bench:async` and a deopt test. - ADRs 0001–0005 and the `CONTEXT.md` glossary. ## 🧩 Type of change - [ ] 🐛 Bug fix — fixes an issue - [x] ✨ Feature — adds new functionality - [ ] ♻️ Refactor — no behaviour change - [x] ⚡ Performance - [x] 📝 Docs - [ ] 🔧 Chore — tooling, CI, deps - [ ] 💥 Breaking change ## 🔗 Related issues Related to #34, #35, #37 and #71. #125 closes them, once the new parser is the package root. ## ✅ Checklist - [x] Tests added/updated - [x] `pnpm run ci` passes locally - [x] Changeset added (`pnpm changeset`) if this affects the published package - [x] Docs updated if behaviour or the public API changed
This was referenced Oct 9, 2026
lcottercertinia
pushed a commit
that referenced
this pull request
Oct 10, 2026
## 📝 Overview CodSpeed runs in simulation mode only. It counts instructions with V8's `--no-opt`, so CI never measures TurboFan-optimised code, and its figures do not match local wall time. This adds a walltime job on a CodSpeed macro runner: real time, with the full JIT. It runs only on manual dispatch. Pull requests keep simulation mode as the gate. ## 🛠️ Changes - `walltime` job on `workflow_dispatch` only — macro runner minutes are billed (600 a month on the free plan), and a pull request would run fork code on the runner. - Runner `codspeed-macro-x64-ryzen-9950x-ubuntu-24-04` — wall time on a shared runner is too noisy. - Docs state what simulation does not measure, and why walltime is manual. Before the first walltime run, an org admin must allow public repositories in the macro runner group. Without that, the job queues until it times out. Memory mode was tried here and removed. It tracks `malloc`, not the V8 heap, so it shows none of today's parser's objects. It moves to #125, where the v1 parser keeps its store in typed arrays that memory mode does track. ## 🧩 Type of change - [ ] 🐛 Bug fix — fixes an issue - [ ] ✨ Feature — adds new functionality - [ ] ♻️ Refactor — no behaviour change - [ ] ⚡ Performance - [ ] 📝 Docs - [x] 🔧 Chore — tooling, CI, deps - [ ] 💥 Breaking change ## 🔗 Related issues N/A ## ✅ Checklist - [ ] Tests added/updated - [x] `pnpm run ci` passes locally - [ ] Changeset added (`pnpm changeset`) if this affects the published package - [x] Docs updated if behaviour or the public API changed
Resolve the bench files for v1: parse.bench.ts builds with the engine under the parse_<size> names, and large.ts keeps the heap gate. Rename the logs in legacy-v0.3.json so --baseline matches them. Turn CodSpeed memory mode back on: v1's store is in typed arrays, which it tracks.
lcottercertinia
marked this pull request as ready for review
October 15, 2026 16:25
lcottercertinia
previously approved these changes
Oct 15, 2026
lcottercertinia
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Overview
v1: the new parser becomes the package root, and the legacy parser is removed. This PR is stacked on #124. It merges after the v0.3 release.
🛠️ Changes
src/nexttosrc/. The exports are now.(node build for thenodecondition, else browser) and./worker../nextand./next/workerare removed.bench:largefigures as the baseline (benchmarks/baselines/legacy-v0.3.json).Performance against v0.3, on logs of 8 to 100 MB: 5–8× faster, with 6× less memory. A 100 MB log parses in 182 ms instead of 1,215 ms, and holds 141 MB instead of 812 MB.
Consumer check: debug-log-analyzer migrated locally (2548/2548 tests, build), and checked by hand in the Extension Development Host. A 100 MB log parses in about 220 ms in the webview.
🧩 Type of change
🔗 Related issues
Stacked on #124.
Closes #34:
ofTypegives typed per-type indexes, andeventsan iterative walk of every event.Closes #35: every type has a
kind, and SOSL has its own kind apart from SOQL.Closes #37: the new parser is the package root, and the legacy parser is removed.
Closes #71: the catalog names every event's fields, and
EventFieldsdocuments each one.Closes #123:
details.caughton each throw (#129), andflowErrorswith theFLOW_ELEMENT_ERRORtext change (#128). Both merge into this branch first.Closes #127:
DEBUG_CATEGORY_TOKENis exported from the package root.✅ Checklist
pnpm run cipasses locallypnpm changeset) if this affects the published package