Implement remaining minor functional gaps with TS implementation#41
Merged
Implement remaining minor functional gaps with TS implementation#41
Conversation
Port comprehensive tests from TS test suite (comment, number, string, text, value, implicit list, extension, property dive, auto-close, syntax errors, NaN). Fix 7 bugs discovered: - Remove preprocessEscapes from Parse() (only needed for TSV test data) - Enable escape processing in backtick strings (matching TS behavior) - Fix matchString unterminated detection for lone quote chars - Fix matchNumber treating comment starters (#, //) as text continuation - Add bad token (TinBD) panic in lexer for unterminated strings/comments - Fix token consumption to only occur when an alt matches (enables syntax error detection for }/]/: at top level) - Fix Deep() array merge to recursively merge elements at same index - Add NaN/Infinity as value keywords matching TS defaults - Add post-parse unconsumed token check for syntax errors Document 5 language platform mismatches between Go and TypeScript: array named properties, undefined vs null, non-string input, error details, and custom configuration. All 52 tests pass (22 non-TSV + 26 parser TSV + 4 utility TSV). https://claude.ai/code/session_01FUyByRfJWUvyPkFoLZ5z6H
- Add JsonicError type with Code, Detail, Pos, Row, Col, Src fields - Change Parse() and Parser.Start() to return (any, error) - Lexer stores errors on Lex.Err and returns ZZ tokens to gracefully wind down the parser instead of panicking - Update all tests to use (any, error) pattern and expectParseError helper - All 52 tests pass with zero panics in the library code https://claude.ai/code/session_01FUyByRfJWUvyPkFoLZ5z6H
- Error() now outputs [jsonic/<code>]: <message> header matching TS format - Add errsite() to generate source extract with line numbers and ^ carets - Error messages match TS defaults: "unexpected character(s): <src>", "unterminated string: <src>", "unterminated comment: <src>" - Add makeJsonicError() helper that builds Detail from error templates - Pass full source through to errors for context extraction - Replace TestPlatformMismatch_ErrorDetails with comprehensive TestErrorFormat https://claude.ai/code/session_01FUyByRfJWUvyPkFoLZ5z6H
Introduces the Options/Make API (options.go) matching TypeScript's Jsonic.make() pattern. Adds lex enable/disable flags, configurable value definitions, map/list/safe options, and rule start configuration to LexConfig. Extends Parser with MaxMul and ErrorMessages fields for per-instance customization. https://claude.ai/code/session_01FUyByRfJWUvyPkFoLZ5z6H
Implements the TypeScript plugin system in Go with idiomatic APIs: - Plugin type: func(j *Jsonic, opts map[string]any) - Use(): Register and invoke plugins with optional config, supports chaining - Token(): Register custom fixed tokens with dynamic Tin allocation - Rule(): Modify or create grammar rules (add alternates, actions, conditions) - AddMatcher(): Add custom lexer matchers with priority ordering - Config()/RSM(): Direct access to parser internals for advanced plugins Infrastructure changes: - Per-instance FixedTokens and TinNames maps on LexConfig (no global mutation) - CustomMatchers slice on LexConfig integrated into nextRaw() dispatch - Cursor() accessor on Lex for custom matcher position management - Plugin state (tinByName, nameByTin, nextTin) on Jsonic struct Includes 24 tests covering plugin invocation, token registration, rule modification, custom matchers, priority ordering, instance isolation, and composite plugin workflows. https://claude.ai/code/session_01FUyByRfJWUvyPkFoLZ5z6H
Multi-character fixed tokens: - matchFixed() now tries longest match first via FixedSorted - SortFixedTokens() rebuilds sorted list after token registration - Text matching stops at multi-char fixed tokens Ender system: - EnderChars on LexConfig, wired from Options.Ender - matchText() stops at ender characters Custom escape mappings: - EscapeMap on LexConfig, wired from StringOptions.Escape - matchString() checks custom map before built-in escapes Config-aware text boundaries: - isTextContinuation replaced with Lex.isTextChar() method - Checks config's FixedTokens, EnderChars, and StringChars Subscriptions: - LexSub/RuleSub types, Sub() method on Jsonic - LexSubs fire in ParseAlts after each token read - RuleSubs fire in parser loop after each rule step - Context carries LexSubs/RuleSubs for dispatch Instance derivation: - Derive() creates child inheriting parent's config, tokens, matchers, ender chars, escape map, plugins, and subscriptions - Child modifications don't affect parent Dynamic options: - SetOptions() merges options, rebuilds config/grammar, preserves per-instance state, re-applies plugins Rule exclude: - Exclude() removes grammar alternates by group tag - filterAlts helper for tag-based alternate filtering Parse metadata: - ParseMeta() passes metadata to Context.Meta - Accessible in rule actions/conditions via ctx.Meta - Parser.StartMeta() supports meta + subscriptions 41 tests total (17 new) covering all features. https://claude.ai/code/session_01FUyByRfJWUvyPkFoLZ5z6H
Add all remaining minor feature gaps to complete TypeScript parity: - Empty source handling (Lex.Empty, Lex.EmptyResult options) - Custom parser.start function override (Parser.Start option) - Error hints system (Hint option, displayed in error output) - Config modify callbacks (ConfigModify option) - TokenSet() method for named token group lookup (IGNORE, VAL, KEY) - LexCheck callbacks for all 7 built-in matchers (pre-matcher interceptors) - RuleSpec helper methods (Clear, AddOpen, AddClose, PrependOpen, PrependClose, AddBO, AddAO, AddBC, AddAC) - Rule exclude from options (Rule.Exclude) - Debug plugin with Describe() and trace mode 28 new tests added (69 total), all passing. https://claude.ai/code/session_01FUyByRfJWUvyPkFoLZ5z6H
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.