contract(unicharset): UNICHARSET property, script & other_case accessors (3 byte-parity leaves)#556
Merged
Merged
Conversation
|
Warning Review limit reached
More reviews will be available in 46 minutes and 1 second. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthrough
ChangesUniCharSet Properties, Script, and Other-Case Parity
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/board/LATEST_STATE.md:
- Around line 116-120: The markdown file has blank lines within blockquote
sections that violate the MD028 linting rule. Remove the blank lines appearing
at lines 116, 118, and 120 within the blockquote to keep all the `>` prefixed
lines contiguous. If a separator between entries is needed, place it outside the
blockquote section rather than inserting blank lines within the blockquote
itself.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 7fd1372c-f454-4fe9-8fa2-790fe5fb33fb
📒 Files selected for processing (5)
.claude/board/EPIPHANIES.md.claude/board/LATEST_STATE.mdcrates/lance-graph-contract/examples/unicharset_dump.rscrates/lance-graph-contract/src/class_view.rscrates/lance-graph-contract/src/unicharset.rs
Comment on lines
+116
to
+120
|
|
||
| > **2026-06-20 — ADDED (D-UNICHARSET-SCRIPT, the script-table leaf)**: `lance_graph_contract::unicharset::UniCharSet` gained `get_script(id) -> i32` / `get_script_table_size()` / `script_from_script_id(sid) -> Option<&str>` / `script_of(id) -> Option<&str>` / `dump_script()`, backed by new `script_ids: Vec<i32>` + an interned `scripts: Vec<String>`. The first leaf to transcode an **interning side-table** (`add_script`, `unicharset.cpp:1063`): `null_script` "NULL" seeded at sid 0 (the `unichar_insert` set_script, `unicharset.cpp:680` → `null_sid_ == 0`), real scripts intern from 1 in id order. Script name = token after the optional bbox/stats CSV (mixed-tier safe). Out-of-range → `null_sid_` 0 (`unicharset.h:681`). **Byte-identical 112/112** vs tesseract's own `get_script` on real `eng.lstm-unicharset` (self-validating oracle `script` mode; table `["NULL","Common","Latin"]`). Additive, zero-dep, behaviour-preserving on the bijection. +4 tests (19 unicharset total). Consumed by `tesseract-core::CharSet::{get_script,script_of}`. EPIPHANIES `E-CPP-PARITY-4`; fourth leaf of `PROBE-OGAR-ADAPTER-UNICHARSET`. Branch `claude/happy-hamilton-0azlw4`. | ||
|
|
||
| > **2026-06-20 — ADDED (D-UNICHARSET-PROPS, the property-accessor leaf)**: `lance_graph_contract::unicharset::UniCharSet` gained the character-category surface `get_isalpha` / `get_islower` / `get_isupper` / `get_isdigit` / `get_ispunctuation` / `get_isngram` + `dump_properties()`, backed by a new `props: Vec<u8>` parsed from the per-line hex bitmask (`unicharset.cpp:824`; masked to `ISALPHA=0x1 ISLOWER=0x2 ISUPPER=0x4 ISDIGIT=0x8 ISPUNCTUATION=0x10`). Accessors mirror the C++ inline guard (`unicharset.h:497+`): out-of-range id → `false` (`INVALID_UNICHAR_ID`); `get_isngram` is always-false on the plain-table load path (`unicharset.cpp:893`). **Byte-identical 112/112** vs tesseract's own `get_is*` on real `eng.lstm-unicharset` (self-validating oracle: bijection half cross-checks the 5.5.0-header/5.3.4-lib layout, then the property half diffs 0). Additive, zero-dep, behaviour-preserving on the existing id↔unichar bijection (lenient default-0 for a missing/!hex token). +5 tests (15 unicharset total). Consumed by `tesseract-core::CharSet::get_is*`. EPIPHANIES `E-CPP-PARITY-3`; the third leaf of `PROBE-OGAR-ADAPTER-UNICHARSET` (after D-UNICHARSET-1 + D-UNICHAR-1). Branch `claude/happy-hamilton-0azlw4`. | ||
|
|
There was a problem hiding this comment.
Remove blank blockquote separators to satisfy MD028.
Line 116, Line 118, and Line 120 introduce blank lines inside a blockquote, which trips markdownlint (MD028). Keep contiguous > lines (or convert separators outside the quote) to avoid lint noise.
Suggested doc-only fix
->
> **2026-06-20 — ADDED (D-UNICHARSET-SCRIPT, the script-table leaf)**: ...
->
> **2026-06-20 — ADDED (D-UNICHARSET-PROPS, the property-accessor leaf)**: ...
-> 📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| > **2026-06-20 — ADDED (D-UNICHARSET-SCRIPT, the script-table leaf)**: `lance_graph_contract::unicharset::UniCharSet` gained `get_script(id) -> i32` / `get_script_table_size()` / `script_from_script_id(sid) -> Option<&str>` / `script_of(id) -> Option<&str>` / `dump_script()`, backed by new `script_ids: Vec<i32>` + an interned `scripts: Vec<String>`. The first leaf to transcode an **interning side-table** (`add_script`, `unicharset.cpp:1063`): `null_script` "NULL" seeded at sid 0 (the `unichar_insert` set_script, `unicharset.cpp:680` → `null_sid_ == 0`), real scripts intern from 1 in id order. Script name = token after the optional bbox/stats CSV (mixed-tier safe). Out-of-range → `null_sid_` 0 (`unicharset.h:681`). **Byte-identical 112/112** vs tesseract's own `get_script` on real `eng.lstm-unicharset` (self-validating oracle `script` mode; table `["NULL","Common","Latin"]`). Additive, zero-dep, behaviour-preserving on the bijection. +4 tests (19 unicharset total). Consumed by `tesseract-core::CharSet::{get_script,script_of}`. EPIPHANIES `E-CPP-PARITY-4`; fourth leaf of `PROBE-OGAR-ADAPTER-UNICHARSET`. Branch `claude/happy-hamilton-0azlw4`. | |
| > **2026-06-20 — ADDED (D-UNICHARSET-PROPS, the property-accessor leaf)**: `lance_graph_contract::unicharset::UniCharSet` gained the character-category surface `get_isalpha` / `get_islower` / `get_isupper` / `get_isdigit` / `get_ispunctuation` / `get_isngram` + `dump_properties()`, backed by a new `props: Vec<u8>` parsed from the per-line hex bitmask (`unicharset.cpp:824`; masked to `ISALPHA=0x1 ISLOWER=0x2 ISUPPER=0x4 ISDIGIT=0x8 ISPUNCTUATION=0x10`). Accessors mirror the C++ inline guard (`unicharset.h:497+`): out-of-range id → `false` (`INVALID_UNICHAR_ID`); `get_isngram` is always-false on the plain-table load path (`unicharset.cpp:893`). **Byte-identical 112/112** vs tesseract's own `get_is*` on real `eng.lstm-unicharset` (self-validating oracle: bijection half cross-checks the 5.5.0-header/5.3.4-lib layout, then the property half diffs 0). Additive, zero-dep, behaviour-preserving on the existing id↔unichar bijection (lenient default-0 for a missing/!hex token). +5 tests (15 unicharset total). Consumed by `tesseract-core::CharSet::get_is*`. EPIPHANIES `E-CPP-PARITY-3`; the third leaf of `PROBE-OGAR-ADAPTER-UNICHARSET` (after D-UNICHARSET-1 + D-UNICHAR-1). Branch `claude/happy-hamilton-0azlw4`. | |
| > **2026-06-20 — ADDED (D-UNICHARSET-SCRIPT, the script-table leaf)**: `lance_graph_contract::unicharset::UniCharSet` gained `get_script(id) -> i32` / `get_script_table_size()` / `script_from_script_id(sid) -> Option<&str>` / `script_of(id) -> Option<&str>` / `dump_script()`, backed by new `script_ids: Vec<i32>` + an interned `scripts: Vec<String>`. The first leaf to transcode an **interning side-table** (`add_script`, `unicharset.cpp:1063`): `null_script` "NULL" seeded at sid 0 (the `unichar_insert` set_script, `unicharset.cpp:680` → `null_sid_ == 0`), real scripts intern from 1 in id order. Script name = token after the optional bbox/stats CSV (mixed-tier safe). Out-of-range → `null_sid_` 0 (`unicharset.h:681`). **Byte-identical 112/112** vs tesseract's own `get_script` on real `eng.lstm-unicharset` (self-validating oracle `script` mode; table `["NULL","Common","Latin"]`). Additive, zero-dep, behaviour-preserving on the bijection. +4 tests (19 unicharset total). Consumed by `tesseract-core::CharSet::{get_script,script_of}`. EPIPHANIES `E-CPP-PARITY-4`; fourth leaf of `PROBE-OGAR-ADAPTER-UNICHARSET`. Branch `claude/happy-hamilton-0azlw4`. | |
| > **2026-06-20 — ADDED (D-UNICHARSET-PROPS, the property-accessor leaf)**: `lance_graph_contract::unicharset::UniCharSet` gained the character-category surface `get_isalpha` / `get_islower` / `get_isupper` / `get_isdigit` / `get_ispunctuation` / `get_isngram` + `dump_properties()`, backed by a new `props: Vec<u8>` parsed from the per-line hex bitmask (`unicharset.cpp:824`; masked to `ISALPHA=0x1 ISLOWER=0x2 ISUPPER=0x4 ISDIGIT=0x8 ISPUNCTUATION=0x10`). Accessors mirror the C++ inline guard (`unicharset.h:497+`): out-of-range id → `false` (`INVALID_UNICHAR_ID`); `get_isngram` is always-false on the plain-table load path (`unicharset.cpp:893`). **Byte-identical 112/112** vs tesseract's own `get_is*` on real `eng.lstm-unicharset` (self-validating oracle: bijection half cross-checks the 5.5.0-header/5.3.4-lib layout, then the property half diffs 0). Additive, zero-dep, behaviour-preserving on the existing id↔unichar bijection (lenient default-0 for a missing/!hex token). +5 tests (15 unicharset total). Consumed by `tesseract-core::CharSet::get_is*`. EPIPHANIES `E-CPP-PARITY-3`; the third leaf of `PROBE-OGAR-ADAPTER-UNICHARSET` (after D-UNICHARSET-1 + D-UNICHAR-1). Branch `claude/happy-hamilton-0azlw4`. | |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 116-116: Blank line inside blockquote
(MD028, no-blanks-blockquote)
[warning] 118-118: Blank line inside blockquote
(MD028, no-blanks-blockquote)
[warning] 120-120: Blank line inside blockquote
(MD028, no-blanks-blockquote)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/board/LATEST_STATE.md around lines 116 - 120, The markdown file has
blank lines within blockquote sections that violate the MD028 linting rule.
Remove the blank lines appearing at lines 116, 118, and 120 within the
blockquote to keep all the `>` prefixed lines contiguous. If a separator between
entries is needed, place it outside the blockquote section rather than inserting
blank lines within the blockquote itself.
Source: Linters/SAST tools
…arity 112/112
Add get_is{alpha,lower,upper,digit,punctuation} + get_isngram + dump_properties
to UniCharSet, parsing the per-line hex property bitmask (unicharset.cpp:824) into
a props: Vec<u8> and masking each flag exactly as set_is*(id, properties & MASK)
(unicharset.cpp:888-892). Accessors mirror the C++ inline guard (unicharset.h:497+):
an out-of-range id -> false (INVALID_UNICHAR_ID); is_ngram is always-false on the
plain-table load path (unicharset.cpp:893).
Byte-identical 112/112 vs tesseract's own get_is* on real eng.lstm-unicharset via a
self-validating oracle: one harness dumps both the id<->unichar bijection (proven
112/112, E-CPP-PARITY-1) and the new properties; the bijection diffing 0 proves the
5.5.0-header / 5.3.4-lib layout is sound, making the property diff (also 0) trustworthy
despite the version skew. Third leaf through PROBE-OGAR-ADAPTER-UNICHARSET.
- +5 unicharset tests (15 total); clippy -D warnings + fmt clean (scoped -p)
- examples/unicharset_dump.rs gains a `properties` mode (reproduces the parity diff,
matching the committed-harness pattern of the sibling leaves)
- incidental: rustfmt 1.9.0 normalized 2 pre-existing test-assert wraps in class_view.rs
- board: EPIPHANIES E-CPP-PARITY-3; LATEST_STATE branch-work + D-UNICHARSET-PROPS
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1
…112/112 Add get_script + get_script_table_size + script_from_script_id + script_of + dump_script to UniCharSet, backed by an interned scripts: Vec<String> and a parallel script_ids: Vec<i32>. The first leaf to transcode an interning side-table: add_script (unicharset.cpp:1063, insertion-order dedup) with null_script "NULL" seeded at sid 0 by the first unichar_insert (unicharset.cpp:680, so null_sid_ == 0 always); real scripts intern from 1 in first-seen id order. get_script mirrors unicharset.h:681 (out-of-range -> 0). The script name is the token immediately after the optional bbox/stats CSV (token[3] if token[2] has a comma, else token[2]) -- covers every column tier present on real eng data (id 0 is tier-5 no-CSV, others tier-1 CSV) without the full multi-tier parser, which the remaining columns will need (next leaf). Byte-identical 112/112 vs tesseract's own get_script on real eng.lstm-unicharset via the self-validating oracle (script mode; table = [NULL, Common, Latin] confirmed empirically before writing the Rust). Fourth leaf of PROBE-OGAR-ADAPTER-UNICHARSET. - +4 unicharset tests (19 total); clippy -D warnings + fmt clean (scoped -p) - examples/unicharset_dump.rs gains a `script` mode (reproduces the parity diff) - board: EPIPHANIES E-CPP-PARITY-4; LATEST_STATE branch-work + D-UNICHARSET-SCRIPT Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1
…te-parity 112/112
Add get_other_case + dump_other_case to UniCharSet, backed by other_cases:
Vec<i32>. other_case is the case-paired unichar id ('C' -> 'c'), parsed as the
token right after the script and clamped at load exactly as unicharset.cpp:901:
a parsed value not less than size -- and the absent-column default
(unicharset.cpp:813, = size) -- folds to the id itself. get_other_case mirrors
unicharset.h:703 (out-of-range id -> INVALID_UNICHAR_ID -1).
Byte-identical 112/112 vs tesseract's own get_other_case on real
eng.lstm-unicharset (self-validating oracle, other_case mode; 60/112 self, 52
real pairs e.g. C->c). This is the last field cleanly reachable by token-offset;
direction/mirror/bbox sit after other_case and need the full multi-tier column
parser (next leaf). Fifth leaf of PROBE-OGAR-ADAPTER-UNICHARSET.
- +4 unicharset tests (23 total); clippy -D warnings + fmt clean (scoped -p)
- examples/unicharset_dump.rs gains an `other_case` mode (reproduces the diff)
- board: EPIPHANIES E-CPP-PARITY-5; LATEST_STATE branch-work + D-UNICHARSET-OTHERCASE
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1
AdaWorldAPI
force-pushed
the
claude/happy-hamilton-0azlw4
branch
from
2f87370 to
c8cb95c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends
lance-graph-contract::unicharset::UniCharSet(the OGAR Core's TesseractUNICHARSETcontent-store tier) with three transcoded accessor families, each diffed byte-identical 112/112 against the real libtesseractUNICHARSETparser oneng.lstm-unicharset.Leaves (PROBE-OGAR-ADAPTER-UNICHARSET, continued)
7952b38unicharset.cpp:824,888-892get_isalpha/islower/isupper/isdigit/ispunctuation,get_isngram,dump_propertiescb719fbunicharset.cpp:680,1063get_script,get_script_table_size,script_from_script_id,script_of,dump_script2f87370other_case(case pair)unicharset.cpp:901,unicharset.h:703get_other_case,dump_other_caseHow parity was proven (self-validating oracle)
The in-env libtesseract is 5.3.4 with no dev headers; the source tree is 5.5.0 — mixing them is an ABI hazard. The oracle therefore dumps both the id↔unichar bijection (a proven 112/112 reference) and the new field: the bijection half diffing 0 confirms the object layout is sound before the new field's diff is trusted. Each diff is reproducible from the committed
examples/unicharset_dump.rs {properties|script|other_case}.For the script and
other_caseleaves the oracle's output defined the spec (e.g."NULL"is seeded at script-id 0 byunichar_insert, not by file order;other_case >= sizefolds to self) — read first, then transcoded to match (measure-before-assert).Scope / safety
-D warnings+ fmt clean.EPIPHANIES.mdE-CPP-PARITY-3/4/5;LATEST_STATE.mdbranch-work + inventory each commit.other_caseis the lastUNICHARSETfield reachable by simple token-offset;direction/mirror/bbox/stats need the multi-tier column parser (next, larger leaf). The consumer side (tesseract-core surfacing these throughCharSet) is the companion tesseract-rs PR.🤖 Generated with Claude Code
https://claude.ai/code/session_016b33swuXE23hKtqxsHu9p1
Generated by Claude Code
Summary by CodeRabbit
Documentation
New Features
Tests