Introduce token modes

[Lotes]

Closes #50

• added explicit token modes (lexer modes)
• if none is given, one is create implicitly
• if not given, it collects all keywords, token declarations and token groups from the grammar
• token modes can have following members
  • own token declarations
  • token refs to top-level token declaration
  • own token groups
  • keywords
  • keyword selection using a RegExp
• token commands were also introduced
  • push(name) to push a token mode onto the token mode stack
  • pop to pop the topmost token mode
  • mode(name) to set the topmost token mode
• I also extended token declarations to be keywords as well
• a lot of changes happened in the lexer generator
  • I tidied up the code organization a lot, I hope you like it as well!
  • token types are split from group type and token commands
    • it made sense during the journey in order to write a command/type differently in different modes... we can discuss if it still makes sense
  • "keywords" I renamed conceptually to KeywordSelector
• validations around token commands were created
• a lot of tests were added by AI, they caught bugs and show the expectations when using token modes

[msujew]

Nice! Can you resolve the merge conflict so we can check the performance in the GitHub CI? Locally, it seems to look good. Only a 2-3% loss on the statemachine example, which is acceptable.

[Lotes]

@msujew I discovered a problem with my main grammar.

The problem arises when I create a token mode and start to create token declarations within this token mode. As soon as I complete the word "token", the language server seems to end up in a endless loop. It does not react anymore. After completing the declaration and restarting the LS, it works. So the problem is during editing incomplete grammars :-/...

token mode default {
  toke //complete to "token" to see the problem
}

The rule TokenModeMember: TokenDeclUsage | TokenGroupUsage | TokenUsage | KeywordUsage | KeywordSelector is actually having some alternatives with a common (nested) prefix. I hoped that the LL(STAR) is handling this ^^*. Or maybe it is a different problem?

If I would refactor the grammar to have no common prefix in the subrules of TokenModeMember the problem might disappear and the performance would increase.
But it would also make the grammar more difficult I think.

Before I refactor it would be good to discuss whether these members have a good syntax or whether we should alternate it (which could remove the common prefix automatically).

[Lotes]

Ideas...

Some quick fixes or code actions come into my mind:

1. inline a token rule that points to a keyword
2. and the opposite: extract all keywords to a common token rule

Or

3. a hover tooltip over a keyword selector which keywords are included

[msujew]

I discovered a problem with my main grammar.

Does this require some setup? I could not reproduce this:

But it would also make the grammar more difficult I think.

I would keep it as it is for now 👍

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite	Current: 1ee2bcd	Previous: d803417	Ratio
BenchmarkWorkspaceCycle (typefox.dev/fastbelt/examples/statemachine) - MB/s	11.92 MB/s	5.21 MB/s	2.29

This comment was automatically generated by workflow using github-action-benchmark.

[Lotes]

I discovered a problem with my main grammar.

Does this require some setup? I could not reproduce this:

No, it should fail out of the box. I can also complete this line as you started. But afterwards toke gets underlined and the LS is frozen. I cannot jump to definition or anything else.
I could not find out how to debug this yet.

EDIT:

I found out how to debug. Some kind of endless-loop in entered in this case.

EDIT:

Fixed the endless loop and another issue with the help of @msujew

[Lotes]

Added some findings and also questions from my side.

[msujew]

Really solid contribution. Thanks a lot!

I noticed another few missing validations that we should add as part of this PR:

• Duplicate keyword/token definitions in the same token mode should result in an error. I.e. you can currently write token mode default { "x" "x" }.
• Similarly Token references should be unique in the same token mode. Right know, you can write something like token mode default { hidden WS comment WS } without a validation error appearing.
• When a token is declared, but not referenced in any token mode, the diagnostic appears on the first usage site. I would expect it on the token declaration itself. Also, similar to the keyword issue, this should be an error.
• A non-default token mode without a pop command should show a diagnostic.

[msujew]

Looking closer at some of the logic, I found some issues with some specific constellations. Works much better in general now, though!

[sailingKieler]

Good job @Lotes,
did a medium deep dive into this PR, Claude (Sonnet) did a much deeper one (the long headlined comments).

I like the tests you contributed a lot. However, esp. in the util tests I had trouble to get what the tested behavior is supposed to be.

Please add some docs, I left some remarks.
If the remarks provided by Claude don't make sense, feel free to ignore them.
(Maybe it's worth a comment, why they're invalid?)

[sailingKieler]

Thanks @Lotes for your updates.
I have a few more remark & questions for the sake of making things clear and precise.

[msujew]

I let AI review the PR again. I believe some issues slipped in after rebasing on top of the infix rule changes. Also, some minor other issues have been found as well.

[msujew]

Mode-local tokens cannot be referenced from another mode or a top-level group. Only this scope hoists TokenDecls nested under TokenMode > TokenDeclUsage. The generated ScopeTokenUsageTokenRef and ScopeTokenGroupTokenRefs (linker_gen.go:60-64) use the default chain, which only exports root children.

Repro: token mode S { token STR: /[a-z]+/ "" -> pop }andtoken mode O { STR "~" -> pop }(ortoken group G { STR }): Could not resolve reference to 'STR'and generation aborts, whileValue=STRin a parser rule links fine andcheckUniqueRuleNames` already treats the name as grammar-global.

Fix: register mode-level TokenDecl/TokenGroup under the grammar root in DescribeLocal (then this hand-walk and the Members() re-walks in the generator can go), or override the two token-ref scopes the same way.