chore(release): ship a matching lindera.yml in every release archive - #38
Draft
Saeris wants to merge 2 commits into
Draft
chore(release): ship a matching lindera.yml in every release archive#38Saeris wants to merge 2 commits into
lindera.yml in every release archive#38Saeris wants to merge 2 commits into
Conversation
…ion >= 2 `lindera_fts5_tokenizer_init` could never load: two bugs sit on the init path, and the second is unreachable until the first is fixed. `bind_fts5_pointer` passed the pointer's VALUE rather than its address. `target` is `&mut *mut FTS5API`, so `target.cast()` auto-derefs the reference and hands `sqlite3_bind_pointer` the inner (null) pointer instead of `&pRet` as SQLite's documented `fts5_api_from_db` requires. Nothing reports an error — bind returns SQLITE_OK and step returns SQLITE_ROW — because nothing is malformed; fts5() simply has nowhere to write, so `p_fts5_api` stays null and init fails at SQLITE_INTERNAL. `ensure_fts5_api_version` then required exactly 2. SQLite 3.47.0 raised `fts5_api.iVersion` to 3 (adding fts5_tokenizer_v2), and the change is purely additive: `xCreateTokenizer`, the only entry point used here, is unchanged and keeps its position. An equality test rejects every SQLite released since, so it now accepts any version at or above the minimum. Verified against SQLite 3.53.0 with --features=embed-ipadic: the extension loads and all five CJK probes match (日本語, 辞書, 図書館, 食べる, コーヒー) — none of which the stock unicode61 tokenizer finds.
The release archives contain only the shared library, so anyone using a prebuilt extension has to know that LINDERA_CONFIG_PATH is required, then find a configuration for it in a source tree they never cloned. resources/lindera.yml is also IPADIC-specific: it names embedded://ipadic and filters on IPADIC part-of-speech tags. Five of the six release variants embed a different dictionary, so it is the wrong configuration for them. Add a configuration per embedded dictionary and give each build matrix entry a `config` field naming its own, so every archive carries the one matching the dictionary it embedded. It is staged as a plain `lindera.yml` inside the archive, since the variant is already in the archive's filename and a fixed name keeps the setup identical everywhere. The new configurations mirror resources/lindera.yml, adjusted per dictionary: - unidic: japanese_stop_tags is restricted to the top-level tags IPADIC and UniDic agree on. The IPADIC config's deeper tags (助詞,格助詞,一般) subdivide differently in UniDic and would silently match nothing. - ko-dic: korean_stop_tags with EP/EF/JKG, the Korean counterparts of the 助動詞/助詞 the IPADIC config drops. japanese_iteration_mark is dropped. - cc-cedict, jieba: segmentation plus unicode_normalize only. Lindera ships no Chinese-specific filters, and the Japanese and Korean stop-tag filters key on tag vocabularies these dictionaries do not use. embed-cjk keeps resources/lindera.yml: it embeds three dictionaries but a configuration selects one, and IPADIC is the existing default. Also document the prebuilt path in the README, which until now covered only building from source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the FTS5 fix (#37), split out so it can be reviewed on its own. It addresses a secondary issue mentioned in #34. I'll leave it as a draft until the base FTS5 fix branch is merged, after which I'll rebase this on main in case any changes get added during review.
The problem
The release archives contain only the shared library.
LINDERA_CONFIG_PATHis required for the tokenizer to load, so a user of a prebuilt extension has to discover that requirement, then go find a configuration somewhere undocumented.Also,
resources/lindera.ymlis also IPADIC-specific, it namesembedded://ipadicand filters on IPADIC part-of-speech tags. Five of the six release variants embed a different dictionary, so copying it across is the wrong configuration for all of them.The fix
This adds a configuration per embedded dictionary and gives each build matrix entry a
configfield naming its own, so every archive carries the configuration matching the dictionary it actually embedded.It is staged as a plain
lindera.ymlinside the archive: the variant is already in the archive's filename, and a fixed name keeps the setup instructions identical for every download.embed-ipadic,embed-cjkresources/lindera.yml(existing)embed-unidicresources/lindera-unidic.ymlembed-ko-dicresources/lindera-ko-dic.ymlembed-cc-cedictresources/lindera-cc-cedict.ymlembed-jiebaresources/lindera-jieba.ymlembed-cjkembeds three dictionaries but a configuration selects one, so it keeps the existing IPADIC default.How the new configurations were derived
Each mirrors
resources/lindera.yml, adjusted only where the dictionary requires it. Every filter name and tag value was checked againstlindera-analysis-5.0.1rather than inferred:japanese_stop_tagsis restricted to the top-level tags IPADIC and UniDic agree on (助詞,助動詞,記号). The IPADIC config's deeper tags (助詞,格助詞,一般) subdivide differently in UniDic, so carrying them over would silently match nothing.korean_stop_tagswithEP/EF/JKG, the Korean counterparts of the助動詞/助詞the IPADIC config drops.japanese_iteration_markis dropped as inapplicable.unicode_normalizeonly. Lindera ships no Chinese-specific filters, and the Japanese and Korean stop-tag filters key on tag vocabularies these dictionaries do not use. The language-neutral filters (stop_words,length,lowercase) remain available to anyone who wants them.remove_diacritical_markkeepsjapanese: falsein the Japanese configs, as in the existing one. Stripping dakuten would collapse か/が and は/ば/ぱ into a single token.Also
The README documented only the build-from-source path. Added a short prebuilt-extension section and a feature→configuration table.
Verification
embedded://dictionary URIs (ipadic,unidic,ko-dic,cc-cedict,jieba) against the lindera 5.0.1 sources.unicode_normalize,japanese_iteration_mark,japanese_stop_tags,japanese_katakana_stem,korean_stop_tags,remove_diacritical_mark) against the*_FILTER_NAMEconstants inlindera-analysis-5.0.1, and theEP/EF/JKGvalues against that crate's own ko-dic fixtures.Compress-Archive -Path <dll>,lindera.ymlplaces both entries flat at the archive root, matchingzip --junk-paths, so the layout is identical across all three platforms.