Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ jobs:
with:
components: rustfmt, clippy
- run: cargo fmt --all --check
- run: python3 scripts/check-release-versions.py
- run: cargo clippy --workspace --all-targets -- -D warnings
- run: cargo doc --workspace --no-deps --locked
env:
Expand Down Expand Up @@ -84,8 +85,8 @@ jobs:
- uses: actions/setup-python@v6
with:
python-version: "3.12"
# CPU-only development package: no ML dependencies or model files.
- run: python -m pip install setuptools==78.1.0
# Installed CPU/mock contracts only; no weights, PyTorch or inference.
- run: python -m pip install setuptools==78.1.0 numpy==2.5.3
- run: python -m pip install --no-deps --no-build-isolation ./packages/tokenfold-select
- run: python -I -m unittest discover -s "${{ github.workspace }}/packages/tokenfold-select/tests" -v
working-directory: ${{ runner.temp }}
Expand Down
28 changes: 25 additions & 3 deletions .github/workflows/publish-packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
version:
description: Existing GitHub release version to publish
required: true
default: 0.5.1
default: 0.5.2

permissions:
contents: read
Expand Down Expand Up @@ -35,6 +35,7 @@ jobs:
npm_version=$(node -p "require('./packages/tokenfold/package.json').version")
test "$VERSION" = "$cargo_version"
test "$VERSION" = "$npm_version"
python3 scripts/check-release-versions.py
test "v$VERSION" = "$(git describe --tags --exact-match)"
- run: cargo publish -p tokenfold-core --dry-run --locked
- name: Validate registry packages from the locked source tree
Expand Down Expand Up @@ -143,8 +144,29 @@ jobs:
name: python-sdist
path: dist/*

select-python:
needs: validate
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
ref: v${{ inputs.version }}
- uses: actions/setup-python@v6
with:
python-version: "3.12"
- run: python -m pip install build twine numpy==2.5.3
- run: python -m build --outdir dist-select packages/tokenfold-select
- run: python -m twine check dist-select/*
- run: python -m pip install --no-deps dist-select/*.whl
- run: python -I -m unittest discover -s "${{ github.workspace }}/packages/tokenfold-select/tests" -v
working-directory: ${{ runner.temp }}
- uses: actions/upload-artifact@v7
with:
name: python-select
path: dist-select/*

publish-python:
needs: [publish-crates, python-wheels, python-sdist]
needs: [publish-crates, python-wheels, python-sdist, select-python]
runs-on: ubuntu-latest
permissions:
id-token: write
Expand Down Expand Up @@ -245,7 +267,7 @@ jobs:
! readelf -l "$binary" | grep -q INTERP

release-ready:
needs: [validate, python-wheels, python-sdist, npm-smoke]
needs: [validate, python-wheels, python-sdist, select-python, npm-smoke]
runs-on: ubuntu-latest
steps:
- run: echo "All registry artifacts passed their pre-publication checks."
Expand Down
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,23 @@

## [Unreleased]

## [0.5.2] - 2026-10-17

- Align Cargo, Python binding, npm and five platform packages at 0.5.2; add the
optional `tokenfold-select` wheel/sdist to the existing publication gate.
- Default the optional Select runtime worker to the pinned Select v2 FP32 ONNX
CPU backend. Add integrity-only inspection and explicit approval-config setup;
preserve the legacy PyTorch development backend as an explicit option.
No weights or automatic downloads are bundled. Core defaults and experimental
Select/approval/fallback boundaries remain unchanged. ONNX task quality and
cold/warm performance are not yet qualified.
- Prepare the separate Select v2 model distribution in both ONNX and Safetensors
formats, with custom-loader instructions; retain ONNX as the worker default.
- Update the offline quickstart notebook for 0.5.2, with a disabled-by-default
Select v2 ONNX example pinned to the published model commit and Safetensors
guidance; keep legacy v1 results explicitly separate from v2 evidence.
- Check registry-version consistency in CI and before publication.

- Explicitly release the persistent observation ledger's exclusive lock when its
owner drops, so a briefly inherited Unix descriptor cannot delay restart.
Keep live-owner refusal and snapshot safety unchanged; add a Unix regression.
Expand Down
5 changes: 5 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@ same ones.

Release commits go through a PR like everything else.

Also align `packages/tokenfold-select/pyproject.toml` and its package version;
`python scripts/check-release-versions.py` checks the registry version set. The
optional Select wheel/sdist is published alongside the `tokenfold` binding.
Model weights remain a separate, explicitly approved Hugging Face release.

```sh
git switch main && git pull
git switch -c release/v0.4.1
Expand Down
20 changes: 10 additions & 10 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ members = [
]

[workspace.package]
version = "0.5.1"
version = "0.5.2"
edition = "2024"
rust-version = "1.98"
license = "Apache-2.0"
Expand Down
52 changes: 49 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,53 @@ Tokenfold solves a different problem first: repeated keys, values, schemas, rows

**When structure ends, rank what matters.**

Tokenfold Select is an Apache-2.0 LoRA adapter on
**Select v2 is the default scorer backend in the optional `tokenfold-select`
0.5.2 package: local FP32 ONNX inference, original text, whole-group ranking.**
Provision the separate pinned model artifact, generate explicit runtime approval,
and invoke `tokenfold --experimental select`. Core remains model-free and
deterministic; installing or upgrading Tokenfold does not silently load a model
or enable lossy selection. See the [Select v2 setup and limits](packages/tokenfold-select/README.md).
The separate model bundle also carries Safetensors weights and a custom reference
loader; ONNX remains the CLI worker default. V2 is an experimental preview:
[Both formats are published under Apache-2.0](https://huggingface.co/snchimata/tokenfold-select-v2).
Synthetic CPU parity is verified, but task-quality and cold/warm performance
qualification are still open. The [example notebook](examples/quickstart.ipynb#10-tokenfold-select-v2-optional-local-ranking)
keeps model execution off until you explicitly enable it.

<details>
<summary><strong>Python: try Tokenfold Select v2 (ONNX)</strong></summary>

Install `tokenfold-select[onnx]==0.5.2` and `huggingface_hub==1.33.0`, then download
only the ONNX profile at the pinned release commit:

```python
from pathlib import Path
from huggingface_hub import snapshot_download
from tokenfold_select import OnnxScorer

root = Path(snapshot_download(
"snchimata/tokenfold-select-v2",
revision="78736d9afea819322f212d6d53eedcf68ac7a39b",
allow_patterns=["model.onnx", "model.onnx.data", "onnx-manifest.json",
"tokenizer/*", "runtime/*", "LICENSE", "NOTICE.md"],
)).resolve()
scorer = OnnxScorer(root, development=True)
response = scorer.score({
"schema_version": 1, "model_revision": scorer.revision,
"query": "Which delivery city?",
"groups": [{"id": "delivery", "text": "Delivery city: Paris"}],
})
print(response) # scores only; your controller owns protection and allocation
```

For Safetensors, use the model card's custom loader rather than a stock
Transformers classification class. Neither format silently changes Core behavior.

</details>

### Tokenfold Select v1 (legacy model)

The separately published Select v1 is an Apache-2.0 LoRA adapter on
`ibm-granite/granite-embedding-reranker-english-r2`. It scores candidate spans
against a query; your allocator applies the token budget and force-keeps
required content. It is a separately distributed offering. Core remains
Expand All @@ -352,7 +398,7 @@ model-free and deterministic and neither loads nor invokes Select.
| Output | Compressed payload + exact receipt | Ranking logits |

<details>
<summary><strong>Python: load the model and score spans</strong></summary>
<summary><strong>Legacy v1 Python: load the LoRA adapter and score spans</strong></summary>

```python
from pathlib import Path
Expand Down Expand Up @@ -393,7 +439,7 @@ def score(query: str, spans: list[str]) -> list[float]:

</details>

### Tokenfold Select benchmarks
### Tokenfold Select v1 benchmarks (not v2 evidence)

[Tokenfold Select][tokenfold-select] is Tokenfold's query-aware model for choosing
what fills a tight context window after structural compression ends.
Expand Down
2 changes: 1 addition & 1 deletion crates/tokenfold-adapters/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,6 @@ description = "Explicit, model-free provider observation adapters for Tokenfold.
readme = "README.md"

[dependencies]
tokenfold-core = { version = "0.5.1", path = "../tokenfold-core" }
tokenfold-core = { version = "0.5.2", path = "../tokenfold-core" }
serde = { version = "1", features = ["derive"] }
serde_json = { version = "1", features = ["preserve_order"] }
10 changes: 5 additions & 5 deletions crates/tokenfold-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,11 @@ name = "tokenfold"
path = "src/main.rs"

[dependencies]
tokenfold-adapters = { version = "0.5.1", path = "../tokenfold-adapters" }
tokenfold-core = { version = "0.5.1", path = "../tokenfold-core" }
tokenfold-output = { version = "0.5.1", path = "../tokenfold-output" }
tokenfold-learn = { version = "0.5.1", path = "../tokenfold-learn" }
tokenfold-rag = { version = "0.5.1", path = "../tokenfold-rag" }
tokenfold-adapters = { version = "0.5.2", path = "../tokenfold-adapters" }
tokenfold-core = { version = "0.5.2", path = "../tokenfold-core" }
tokenfold-output = { version = "0.5.2", path = "../tokenfold-output" }
tokenfold-learn = { version = "0.5.2", path = "../tokenfold-learn" }
tokenfold-rag = { version = "0.5.2", path = "../tokenfold-rag" }
clap = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
Expand Down
2 changes: 1 addition & 1 deletion crates/tokenfold-learn/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@ keywords = ["llm", "tokens", "compression", "policy", "token-optimization"]
categories = ["compression"]

[dependencies]
tokenfold-core = { version = "0.5.1", path = "../tokenfold-core" }
tokenfold-core = { version = "0.5.2", path = "../tokenfold-core" }
2 changes: 1 addition & 1 deletion crates/tokenfold-output/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@ keywords = ["llm", "tokens", "compression", "reports", "token-optimization"]
categories = ["compression"]

[dependencies]
tokenfold-core = { version = "0.5.1", path = "../tokenfold-core" }
tokenfold-core = { version = "0.5.2", path = "../tokenfold-core" }
2 changes: 1 addition & 1 deletion crates/tokenfold-py/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ name = "tokenfold_py"
crate-type = ["cdylib", "rlib"]

[dependencies]
tokenfold-core = { version = "0.5.1", path = "../tokenfold-core" }
tokenfold-core = { version = "0.5.2", path = "../tokenfold-core" }
serde_json = { workspace = true }
pyo3 = { version = "0.29.0", features = ["abi3-py39"] }

Expand Down
2 changes: 1 addition & 1 deletion crates/tokenfold-rag/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ description = "Bounded BM25 search over authorized Tokenfold evidence."
readme = "README.md"

[dependencies]
tokenfold-core = { version = "0.5.1", path = "../tokenfold-core" }
tokenfold-core = { version = "0.5.2", path = "../tokenfold-core" }
Loading
Loading