Skip to content

Fix extension initialization: bind the address of the fts5_api out-pointer, and accept iVersion >= 2 - #37

Open
Saeris wants to merge 1 commit into
lindera:mainfrom
Saeris:fix/fts5-api-pointer-and-version
Open

Fix extension initialization: bind the address of the fts5_api out-pointer, and accept iVersion >= 2#37
Saeris wants to merge 1 commit into
lindera:mainfrom
Saeris:fix/fts5-api-pointer-and-version

Conversation

@Saeris

@Saeris Saeris commented Aug 6, 2026

Copy link
Copy Markdown

Fixes #34.

The extension could not be loaded at all in the LTS version of node:sqlite because lindera_fts5_tokenizer_init always returned an error. There are two bugs on that path, and both have to be fixed for the extension to initialize; the second is only reachable once the first is fixed.

1. bind_fts5_pointer passed the pointer's value, not its address

SQLite's documented way to obtain the FTS5 API is fts5_api_from_db, from FTS5 §7 Extending FTS5:

fts5_api *pRet = 0;
sqlite3_bind_pointer(pStmt, 1, (void*)&pRet, "fts5_api_ptr", NULL);
sqlite3_step(pStmt);

Note &pRet — fts5() writes the api pointer into that variable.

target here is &mut *mut FTS5API, so target.cast::<c_void>() auto-derefs the reference and passes the inner *mut FTS5API (i.e. pRet's own value, which is null at that moment) instead of &pRet. sqlite3_bind_pointer returns SQLITE_OK and sqlite3_step returns SQLITE_ROW, because nothing is technically wrong; fts5() simply has nowhere to write. p_fts5_api stays null and init fails at ok_or(SQLITE_INTERNAL).

Instrumented, before the fix:

bind rc=0  step rc=100 (SQLITE_ROW)  p_fts5_api null? true

The fix binds (&raw mut *target), matching the C pattern.

2. ensure_fts5_api_version required exactly 2

Once the pointer arrives, init fails at the next gate. SQLite 3.47.0 (2024-10-21) raised fts5_api.iVersion from 2 to 3, adding fts5_tokenizer_v2 with xCreateTokenizer_v2 / xFindTokenizer_v2 (3.47.0 changelog). ext/fts5/fts5.h now reads:

struct fts5_api {
  int iVersion;                   /* Currently always set to 3 */

The bump is purely additive: xCreateTokenizer (the only entry point this extension uses) is unchanged and keeps its position in the struct, with the v2 members appended after xCreateFunction. So,, an exact-match test rejects every SQLite released in the last ~21 months for no reason. Changed to >=, which still rejects a genuinely older API that would be missing members we rely on.

Verification

Built --release --features=embed-ipadic and loaded into SQLite 3.53.0 via Node's built-in node:sqlite:

LOAD: OK

  hit   MATCH "日本語"   -> 1 row(s)
  hit   MATCH "辞書"     -> 1 row(s)
  hit   MATCH "図書館"   -> 1 row(s)
  hit   MATCH "食べる"   -> 1 row(s)
  hit   MATCH "コーヒー" -> 1 row(s)

5/5 matched

Every one of those scores 0 with the stock unicode61 tokenizer, so this exercises real Lindera segmentation end to end. cargo test --release --features=embed-ipadic passes (2 tests + 1 doctest), and cargo fmt --check is clean. cargo clippy reports one pre-existing collapsible_if warning in Drop for PreparedStatement, untouched by this change.

Re-verified after rebasing onto main at #36 (lindera 2.0.1 → 5.0.1): builds clean, same 5/5, tests and fmt --check still pass.

…ion >= 2

`lindera_fts5_tokenizer_init` could never load: two bugs sit on the init
path, and the second is unreachable until the first is fixed.

`bind_fts5_pointer` passed the pointer's VALUE rather than its address.
`target` is `&mut *mut FTS5API`, so `target.cast()` auto-derefs the
reference and hands `sqlite3_bind_pointer` the inner (null) pointer
instead of `&pRet` as SQLite's documented `fts5_api_from_db` requires.
Nothing reports an error — bind returns SQLITE_OK and step returns
SQLITE_ROW — because nothing is malformed; fts5() simply has nowhere to
write, so `p_fts5_api` stays null and init fails at SQLITE_INTERNAL.

`ensure_fts5_api_version` then required exactly 2. SQLite 3.47.0 raised
`fts5_api.iVersion` to 3 (adding fts5_tokenizer_v2), and the change is
purely additive: `xCreateTokenizer`, the only entry point used here, is
unchanged and keeps its position. An equality test rejects every SQLite
released since, so it now accepts any version at or above the minimum.

Verified against SQLite 3.53.0 with --features=embed-ipadic: the
extension loads and all five CJK probes match (日本語, 辞書, 図書館,
食べる, コーヒー) — none of which the stock unicode61 tokenizer finds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extension fails to load on SQLite >= 3.47: fts5_api.iVersion is 3, but init requires exactly 2

1 participant