Add semantic highlighting with LSP semantic tokens

[Seddryck]

Goal

Add LSP semantic tokens for Expressif so editors can highlight language constructs according to their syntactic/semantic role rather than relying only on a TextMate grammar or generic lexical patterns.

This is the fifth feature in the initial LSP roadmap and should come after diagnostics, function completion, function hover, and signature help. At that point the server already has reliable parsing, source-position mapping, and enough language knowledge to classify important constructs consistently.

Why semantic tokens

Expressif contains constructs that are visually similar as text but represent different language concepts. Server-side semantic classification can distinguish these using the parsed syntax tree.

Useful initial categories include:

@variable
function(...)
.field
$n
"literal"
123
|  |>

A TextMate grammar can provide useful baseline syntax coloring, but semantic tokens let the language server classify constructs using actual syntax rather than regular-expression matching.

Scope

Implement LSP semantic token support that:

• advertises a semantic-token legend and server capability;
• walks or queries the Expressif.Syntax representation of the current document;
• emits semantic tokens for a deliberately small, stable set of Expressif constructs;
• maps source spans accurately to LSP line/character positions;
• supports multiline documents correctly;
• emits deterministic, non-overlapping token ranges accepted by standard LSP clients;
• updates tokens as the document changes;
• remains editor-agnostic.

The first version should prioritize correct classification and source ranges over an exhaustive token taxonomy.

Initial token categories

The implementation should cover at least the following language concepts when they can be identified reliably from Expressif.Syntax:

Variables

@myVariable

Classify variable references independently from ordinary identifiers or literals.

Functions

text-to-lower
add
record

Classify the function identifier itself, not its complete call range.

Where a function can be resolved through the shared function metadata introduced by #16, resolution may be used to enrich classification, but semantic highlighting must not require runtime evaluation.

Field access

.name
.address.city

The field/member portion should be distinguishable from functions and variables.

Positional/contextual references

Examples include constructs such as:

$n
$n-1
$0
$1

Only classify forms that are actually represented and distinguishable by the current Expressif grammar.

Literals

At minimum distinguish obvious literal categories supported by the syntax model, such as strings and numeric values. Additional categories such as booleans, null/empty values, dates, arrays, records, or tuples may be added when they map cleanly to standard LSP token types.

Operators

Operators such as pipeline forms should be classified where the LSP/client token model makes that useful:

|
|>

Do not create a custom token type for every punctuation mark merely because it appears in the grammar.

Semantic token legend

Prefer standard LSP semantic token types whenever they describe the Expressif concept adequately, for example variable, function, property, string, number, and operator.

Introduce custom token types only when an Expressif-specific distinction has clear editor value and cannot be represented reasonably with standard types.

The legend should be stable because clients interpret token indices according to the capability advertised by the server.

Token modifiers should be kept minimal in the first version. Do not invent modifiers for distinctions that the server cannot determine reliably.

Syntax-driven classification

Semantic token classification must be derived from Expressif.Syntax nodes/tokens and their source spans.

Do not implement a parallel lexer or regular-expression scanner inside the language server solely for highlighting. If the syntax tree does not expose enough information to classify an important construct accurately, improve or consume the syntax representation rather than duplicating the grammar.

Conceptually:

DocumentStore
     │
     ▼
SyntaxService / Expressif.Syntax
     │
     ▼
syntax nodes + source spans
     │
     ▼
SemanticTokenService
     │
     ▼
SemanticTokensHandler

As with the existing architecture, OmniSharp/LSP protocol types should remain at the handler boundary where practical.

Full vs incremental tokens

A full-document semantic-token request is sufficient for the initial implementation unless incremental/delta support is already straightforward with the selected LSP library.

Correctness is more important than introducing token-delta caching prematurely.

If delta support is not implemented, the capability negotiation should advertise only the modes actually supported.

Interaction with syntax highlighting

Semantic tokens are intended to complement, not replace, editor lexical highlighting.

The VS Code extension from #13 may continue to provide a TextMate grammar for baseline coloring and immediate highlighting before the server responds. Semantic tokens should add syntax-aware distinctions on top of that without embedding Expressif classification logic in the extension.

The language server remains the source of semantic classification.

Range and encoding requirements

Semantic token ranges must correctly handle:

• tokens on the first and later lines;
• several tokens on the same line;
• multiline expressions;
• Unicode text before a classified token, according to LSP character-position rules;
• adjacent constructs;
• document edits that move tokens to different lines/columns.

Generated tokens must be sorted according to LSP semantic-token encoding rules and must not overlap unless the protocol/client explicitly supports the chosen representation.

Testing

Tests should cover representative expressions combining several token types, for example:

@customer | .name | text-to-upper

and more structured expressions containing nested calls, contextual references, and literals.

Tests should assert semantic categories and source ranges rather than client-specific colors. The language server defines meaning; the editor/theme decides visual appearance.

Negative tests should verify that punctuation or literal content resembling language constructs is not misclassified. For example, "@foo | add" should remain a string rather than producing nested variable/function/operator tokens.

Acceptance criteria

• The language server advertises standard LSP semantic-token capability with a documented legend.
• Full-document semantic-token requests are supported.
• Variables are classified from parsed Expressif syntax.
• Function identifiers are classified independently from their arguments/call ranges.
• Field/property access is classified when represented by the syntax model.
• Supported contextual/positional references are classified distinctly and consistently.
• String and numeric literals are classified at minimum.
• Relevant pipeline/operators are classified where supported by the chosen legend.
• Semantic classification is derived from Expressif.Syntax; no duplicate highlighting lexer/grammar is introduced in the server.
• Source ranges and LSP delta encoding are correct for multiline documents and multiple tokens per line.
• Token output is deterministic, sorted, and non-overlapping.
• Editing a document produces semantic tokens corresponding to the latest synchronized text.
• Tests assert token kinds and ranges for representative mixed expressions and negative literal cases.
• The implementation remains editor-agnostic; VS Code-specific color choices are not part of the language server.

Out of scope

• Theme definitions or choosing actual colors.
• Replacing the VS Code TextMate grammar.
• Go-to-definition, references, rename, or workspace symbols; these remain deferred by the roadmap.
• Full type inference or data-flow analysis.
• Highlighting based on runtime values.
• Exhaustive classification of every punctuation/token in the grammar.
• Incremental semantic-token delta support unless it falls out naturally from the implementation.