Skip to content

Add semantic tokens support - #136

Open
msujew wants to merge 6 commits into
mainfrom
msujew/semantic-tokens
Open

msujew wants to merge 6 commits into
mainfrom
msujew/semantic-tokens

Conversation

@msujew

@msujew msujew commented Aug 6, 2026

Copy link
Copy Markdown
Member

Closes #95

Provides the infrastructure to implement semantic tokens support for any language. Comes with:

  • Testing infrastructure using AssertSemanticTokens.
  • Options to easily provide custom token types/modifier via ExtendableSemanticTokensLegendProvider.
  • Composable semantic tokens data via SemanticTokensBuilder.
  • Simple semantic token highlighting strategies via TokenHighlightingStrategy and TokenBasedSemanticTokensProvider.
  • A semantic token implementation for the grammar language.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite Current: d971cab Previous: d803417 Ratio
BenchmarkWorkspaceCycle (typefox.dev/fastbelt/examples/statemachine) - MB/s 13 MB/s 5.21 MB/s 2.50

This comment was automatically generated by workflow using github-action-benchmark.

@msujew
msujew force-pushed the msujew/semantic-tokens branch from 97f005c to d971cab Compare August 6, 2026 12:55
@ssmifi ssmifi mentioned this pull request Aug 7, 2026

@Lotes Lotes left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have found 2 things.

Comment thread test/doc_fixture_lsp.go
Comment thread test/doc_fixture_lsp.go
@msujew
msujew force-pushed the msujew/semantic-tokens branch from d971cab to 0e72789 Compare August 19, 2026 13:44
@msujew
msujew requested a review from Lotes August 24, 2026 11:27

@Lotes Lotes left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Looks good. Just some little questions about the legend provider.

Comment thread internal/grammar/semantic_tokens_test.go

@sailingKieler sailingKieler left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @msujew for this contribution, too, now I made it through it.

The review done by Claude brought interesting issues up, and I have a few requests for more documentation, see below.

Comment thread internal/grammar/semantic_tokens_test.go
Comment thread server/semantic_tokens_legend.go
Comment thread server/semantic_tokens_builder.go Outdated
Comment thread server/semantic_tokens_provider.go
Comment thread server/server.go
Comment thread server/semantic_tokens_builder.go Outdated
Comment thread server/semantic_tokens_legend.go
Comment thread server/semantic_tokens_provider.go Outdated
Comment thread server/semantic_tokens_builder.go
Comment thread server/semantic_tokens_legend.go Outdated
@msujew
msujew force-pushed the msujew/semantic-tokens branch 3 times, most recently from b409f4d to eb49559 Compare September 1, 2026 11:06
@msujew
msujew force-pushed the msujew/semantic-tokens branch from eb49559 to b2f2426 Compare September 8, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[LSP] Semantic token provider

3 participants