Skip to content

build(deps): bump tokenizers from 0.22.2 to 0.23.2 in /skills/remove-ai-marks/scripts - #41

Merged
elkaix merged 2 commits into
mainfrom
dependabot/pip/skills/remove-ai-marks/scripts/tokenizers-0.23.2
Sep 11, 2026
Merged

build(deps): bump tokenizers from 0.22.2 to 0.23.2 in /skills/remove-ai-marks/scripts#41
elkaix merged 2 commits into
mainfrom
dependabot/pip/skills/remove-ai-marks/scripts/tokenizers-0.23.2

Conversation

@dependabot

@dependabot dependabot Bot commented on behalf of github Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Bumps tokenizers from 0.22.2 to 0.23.2.

Release notes

Sourced from tokenizers's releases.

v0.23.2

This is the last v0 release, we are moving to v1!!

More details coming soon 👀

What's Changed

New Contributors

Full Changelog: huggingface/tokenizers@v0.23.1...v0.23.2

Release v0.23.1

TL;DR

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without free-threaded support). 0.23.1 is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular and free-threaded 3.14t), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.

There is no functional 0.23.0 published — we tag 0.23.1 directly so users don't accidentally pull a never-shipped version.

🚨 Breaking changes

  • Drop Python 3.9 (#1952) — requires-python = ">=3.10"; 3.9 users stay on 0.22.x.
  • add_tokens normalizes content at insertion (#1995) — re-saved tokenizer.json may differ in the added_tokens block. Existing files load unchanged.
  • Type stubs are precise (#1928, #1997) — methods that returned Any now return real types; mypy --strict may surface previously-hidden errors. Stub layout also moved from tokenizers/<sub>/__init__.pyi to tokenizers/<sub>.pyi. This breaks the surface of some of the processors like RobertaProcessign's __init__ .
  • 3.14t-only: setters/getters return PyResult<T> because of Arc<RwLock<Tokenizer>>; a poisoned lock surfaces as PyException instead of a panic.

... (truncated)

Commits
  • 88a4498 add lock
  • e4ea65f real release
  • 7c5964b add lock
  • bc405ad push rc0
  • d582781 Add a ParityBpeTrainer example and list it in the trainers API docs (#2217)
  • 447890f fix(python): pin the ruff rule set so a ruff release can't redden main (#2292)
  • 68b3ab7 chore(node): take node security alerts from 33 to 0 (#2287)
  • 74ef81f ci: pin every action to one SHA, drop stale audit suppressions (#2291)
  • b26727f chore(node): drop 8 unused devDependencies (#2286)
  • 7c7cfd8 chore: remove stale examples (kills 50% of dependabot traffic) (#2285)
  • Additional commits viewable in compare view

@dependabot dependabot Bot added dependencies Pull requests that update a dependency file python Pull requests that update python code labels Sep 7, 2026
@dependabot
dependabot Bot requested a review from elkaix as a code owner September 7, 2026 09:39
@dependabot dependabot Bot added dependencies Pull requests that update a dependency file python Pull requests that update python code labels Sep 7, 2026
@dependabot dependabot Bot mentioned this pull request Sep 7, 2026
Closed
@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot
dependabot Bot force-pushed the dependabot/pip/skills/remove-ai-marks/scripts/tokenizers-0.23.2 branch from a4733f4 to 75e5c10 Compare September 10, 2026 19:27
@coderabbitai

coderabbitai Bot commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: bba583b7-15cc-4675-b4fe-ca2341810a5e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot
dependabot Bot force-pushed the dependabot/pip/skills/remove-ai-marks/scripts/tokenizers-0.23.2 branch from 75e5c10 to 5b1b0d4 Compare September 10, 2026 20:38
@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot @github

dependabot Bot commented on behalf of github Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Looks like this PR is already up-to-date with main! If you'd still like to recreate it from scratch, overwriting any edits, you can request @dependabot recreate.

@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot @github

dependabot Bot commented on behalf of github Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Looks like this PR is already up-to-date with main! If you'd still like to recreate it from scratch, overwriting any edits, you can request @dependabot recreate.

@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot @github

dependabot Bot commented on behalf of github Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Looks like this PR is already up-to-date with main! If you'd still like to recreate it from scratch, overwriting any edits, you can request @dependabot recreate.

@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot @github

dependabot Bot commented on behalf of github Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Looks like this PR is already up-to-date with main! If you'd still like to recreate it from scratch, overwriting any edits, you can request @dependabot recreate.

@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot @github

dependabot Bot commented on behalf of github Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Looks like this PR is already up-to-date with main! If you'd still like to recreate it from scratch, overwriting any edits, you can request @dependabot recreate.

@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot @github

dependabot Bot commented on behalf of github Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Looks like this PR is already up-to-date with main! If you'd still like to recreate it from scratch, overwriting any edits, you can request @dependabot recreate.

@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot @github

dependabot Bot commented on behalf of github Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Looks like this PR is already up-to-date with main! If you'd still like to recreate it from scratch, overwriting any edits, you can request @dependabot recreate.

@elkaix

elkaix commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@dependabot rebase

@dependabot @github

dependabot Bot commented on behalf of github Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Looks like this PR is already up-to-date with main! If you'd still like to recreate it from scratch, overwriting any edits, you can request @dependabot recreate.

@dependabot
dependabot Bot force-pushed the dependabot/pip/skills/remove-ai-marks/scripts/tokenizers-0.23.2 branch 2 times, most recently from 2ec8918 to d8db815 Compare September 10, 2026 23:25
@dependabot
build(deps): bump tokenizers in /skills/remove-ai-marks/scripts
903206b 
Bumps [tokenizers](https://github.com/huggingface/tokenizers) from 0.22.2 to 0.23.2.
- [Release notes](https://github.com/huggingface/tokenizers/releases)
- [Changelog](https://github.com/huggingface/tokenizers/blob/main/RELEASE.md)
- [Commits](huggingface/tokenizers@v0.22.2...v0.23.2)

---
updated-dependencies:
- dependency-name: tokenizers
  dependency-version: 0.23.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
@dependabot
dependabot Bot force-pushed the dependabot/pip/skills/remove-ai-marks/scripts/tokenizers-0.23.2 branch from d8db815 to 903206b Compare September 10, 2026 23:30
@elkaix
elkaix merged commit d66250b into main Sep 11, 2026
13 of 15 checks passed
@dependabot
dependabot Bot deleted the dependabot/pip/skills/remove-ai-marks/scripts/tokenizers-0.23.2 branch September 11, 2026 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@elkaix elkaix Awaiting requested review from elkaix

Assignees

No one assigned

Labels

dependencies Pull requests that update a dependency file python Pull requests that update python code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@elkaix