Implement multi-language support by msujew · Pull Request #130 · TypeFox/fastbelt

msujew · 2026-07-23T14:52:40Z

Closes #96

Does as the title says, similar to how it is outlined in the issue:

Adds a fastbelt/cmd package adopters can use to write their own CLIs to generate Fastbelt code.
Adds a LanguageSelector service that is used to determine during runtime what language a URI/document belongs to.
Implements a switch-case based selector in the parser and lexer to circumvent the need for a separate service instance per language.
Adds a multilang example that shows that this is working as expected (including parser, lexer and completion tests).

FYI, while this has a Plugins field in the API, plugins aren't actually supported in this PR. This will need a follow-up.

cdietrich · 2026-07-24T07:56:39Z

wondering about the common lexical scope and an option to have a lexer per entry rule
#131

github-actions

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite	Current: `40d792c`	Previous: `d803417`	Ratio
`BenchmarkWorkspaceCycle (typefox.dev/fastbelt/examples/statemachine) - MB/s`	`13.16` MB/s	`5.21` MB/s	`2.53`

This comment was automatically generated by workflow using github-action-benchmark.

msujew · 2026-07-27T14:26:48Z

@cdietrich Thanks for the input. I plan to finish implementing this once #127 lands, since it refactors a lot of the lexer infrastructure.

ssmifi

Looks solid, thanks. I have a couple of questions to get a better understanding before merging and some documentation seems to be stale.

ssmifi · 2026-08-07T16:07:48Z

@@ -55,7 +55,7 @@ func runGenerateCLI(opts generateOptions) error {
 	}

 	sc := grammar.CreateServices()


Nice separation.
Question: The following lines are conceptionally doing for one file what parseAndMerge and reportDiagnostic do generically now. I was wondering if we can de-duplicate some code here and use come common helper?

ssmifi · 2026-08-10T09:09:23Z

+
+type DocumentMatcher func(uri URI) bool
+
+// DocumentFilter matches a document by LSP language id and/or a glob over the


DocumentSelector?

ssmifi · 2026-08-10T09:13:04Z

+	"typefox.dev/fastbelt/util/service"
+)
+
+type DocumentMatcher func(uri URI) bool


As this is an exported symbol, it should probably get a doc, even if self-explanatory.

Lotes · 2026-08-14T09:02:32Z

+// documents.
+type Language struct {
+	Entry      string
+	LanguageID string


Add a lexer mode, after #127 is merged.

Or combine it within the syntax of the entry rule:

entry XRule on mode YMode: ...

Omitting on mode YMode falls back to the default mode.

Agree with @Lotes.

However, this idea mixes concepts that don't belong together:

entry XRule on mode YMode: ...

I don't support this proposal.

Yes, I would keep the entry lexer mode local to the Language struct value.

Lotes · 2026-08-14T09:05:26Z

@@ -0,0 +1,15 @@
+grammar MultilangModel


I thought this PR is also about multi file support.
But yes, this might be a smaller step towards this direction.

Lotes · 2026-08-14T09:12:50Z

+	DocStateLocalSymbols // 0x0010
 	// DocStateLinked marks that cross-references were linked.
-	DocStateLinked // 0x0010
+	DocStateLinked // 0x0020


Why the comment "//0x0001" ...? Who needs it?
The only reason in my eyes: to not forget the flag nature of this constant group.

Alternative: Rename the constants to reflect the flag nature.

DocStateFlagLinked

DocStateBitMaskLinked

sailingKieler

Thanks @msujew for this effort! Great step forward.

Actually, I doubt the necessity and benefit for the additional language build customization for the multi language aspect. Other concerns might justify that.
I left some comments inline.

The other aspect is the documentation of impact of multiple entry rules on the lexer.
That is something I didn't expect in the first place, and others (newbies) probably also don't. See also below.

sailingKieler · 2026-08-28T08:50:03Z

+
+// LanguageSelector resolves a document URI to the index of the owning language
+// (into the configured languages), or -1 if none match.
+type LanguageSelector interface {


On the first impression this felt a bit over engineered to me.
After thinking about it for a moment or tow, I came to the conclusion that this mainly bridges the lack of multi-bindings in our dependency container, here, multiple DocumentSelectors, correct?

I would appreciate a corresponding hint on that here in the docs.

sailingKieler · 2026-08-28T09:44:06Z

+	}
+	if err := ctx.Build(); err != nil {
+		log.Fatalf("multilang generation failed: %v", err)
+	}


Is this route really needed for the multi lange setup?

Why not having it configured manually in the scaffold provided services.go, as you did in this language impl? Coding agents will be able to do that shortly.

Hence, I'm favor of skipping the generation in service_generator.go.

sailingKieler · 2026-08-28T09:50:46Z

@@ -55,7 +55,7 @@ func runGenerateCLI(opts generateOptions) error {
 	}

 	sc := grammar.CreateServices()


sailingKieler · 2026-08-28T09:59:41Z

+				n.AppendLine(")")
+			})
+			n.AppendLine("}")
+		}


I don't see the need/benefit of this, see also my remark in languages/multilang/gen/main.go.

Having it codified in services.go based on an initial template should be fine IMHO.
The single rule validation in cmd/fastbelt/generator.go could be suppressed by a cmd line switch that is to be added in the // go:generate ... config.

sailingKieler · 2026-08-28T10:04:33Z

+// documents.
+type Language struct {
+	Entry      string
+	LanguageID string


Agree with @Lotes.

However, this idea mixes concepts that don't belong together:

entry XRule on mode YMode: ...

I don't support this proposal.

sailingKieler · 2026-08-28T10:14:29Z

+						}
+					}
+				})
+				nn.AppendLine("},")


uuuhhh this is sophisticated.

This should be documented very prominently, specifically the fact parser (entry) rule names/ids are used in the lexer setup.

sailingKieler · 2026-08-28T10:24:12Z

+// Grammars with multiple configured languages use [NewMultiLanguageLexer]
+// instead, passing one token type list per language; at lex time the document
+// is routed to its language's token set via [core.LanguageSelector], mirroring
+// the generated parser's entry rule dispatch.


I lexer_gen.go I asked for prominent documentation of this fact.
Okay, it is already stated here but I didn't get the consequences of this fairly compact statement.

I strongly advocate to extend it, explicit state that reachable token sets from each entry rules are determined.
That this way ambiguities among the entire token definitions set are tolerated as long the sets reachable from the entry rules a unambiguous (enough).
And that this is also beneficial performance wise (less dispatches)

sailingKieler · 2026-08-28T10:26:41Z

Let's adhere the common pattern of naming this file doc.go.

//go:generate go run ./gen

can go to services.go, like in the other languages.

msujew added 2 commits July 27, 2026 15:59

Implement multi-language support

03c238d

Rebase

9f6d652

msujew force-pushed the msujew/multi-language-support branch from 78988e1 to 9f6d652 Compare July 27, 2026 14:14

github-actions Bot reviewed Jul 27, 2026

View reviewed changes

Fix scaffolding

18be4d2

msujew added 2 commits July 27, 2026 17:14

Remove DocumentParser service

25a7226

Support multi-language lexers

292cbfb

msujew mentioned this pull request Aug 3, 2026

Add per-language lexers to eliminate cross-language keyword pollution #131

Closed

Add completion tests

227ab88

msujew marked this pull request as ready for review August 5, 2026 14:00

msujew mentioned this pull request Aug 6, 2026

[LSP] Listen to file system events #118

Open

ssmifi reviewed Aug 10, 2026

View reviewed changes

Review comments

40d792c

Lotes reviewed Aug 10, 2026

View reviewed changes

Comment thread examples/arithmetics/services.go

msujew mentioned this pull request Aug 11, 2026

[CLI] Support type lowering in the generated symbol containers #145

Open

Lotes reviewed Aug 14, 2026

View reviewed changes

sailingKieler reviewed Aug 28, 2026

View reviewed changes

sailingKieler mentioned this pull request Aug 28, 2026

Support infix rules #134

Merged

		@@ -55,7 +55,7 @@ func runGenerateCLI(opts generateOptions) error {
		}

		sc := grammar.CreateServices()


		type DocumentMatcher func(uri URI) bool

		// DocumentFilter matches a document by LSP language id and/or a glob over the

Uh oh!

Uh oh!

Conversation

msujew commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cdietrich commented Jul 24, 2026

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

⚠️ Performance Alert ⚠️

Uh oh!

msujew commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssmifi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sailingKieler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

msujew commented Jul 23, 2026 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

msujew commented Jul 27, 2026 •

edited

Loading