From d82f58177dd68701774d8011bafbcd43138e112b Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Sun, 25 Oct 2026 16:15:00 +0900 Subject: [PATCH 1/2] Retire the upstream sync: sync-upstream.sh and make-upstream-pr-branch.sh refuse to run ArcadeData/arcadedb deleted bindings/python in #10615 (commit 1e0507a522, 2026-10-23, 143 files), so running sync-upstream.sh would merge that deletion into this fork. Decision "Stop syncing" (2026-10-25): nothing needs the engine tree synced any more, since wheels take their jars from the release image and upstream engine PRs branch from upstream/main. - sync-upstream.sh exits 1 with a message naming #10615 and the decision; only --help still prints (the old help, with a retired banner). - make-upstream-pr-branch.sh exits 1 the same way. - AGENTS.md and bindings/python/AGENTS.md: the sync routine is retired by the decision of 2026-10-25; the release routine stays under "ask first". - sync-upstream.md and upstream-bindings-pr.md are marked retired with the reason and a pointer to ArcadeData/arcadedb-python; testing an engine fix without a sync is documented in sync-upstream.md. - release.md, upstream-issues.md, contributing.md, CAMPAIGN.md and the mkdocs nav no longer tell anyone to run the sync. Co-Authored-By: Claude Sonnet 5.5 --- AGENTS.md | 4 +- benchmarks/experiments/CAMPAIGN.md | 2 +- bindings/python/AGENTS.md | 4 +- .../python/docs/development/contributing.md | 4 +- bindings/python/docs/development/release.md | 30 +++--- .../python/docs/development/sync-upstream.md | 92 +++++-------------- .../docs/development/upstream-bindings-pr.md | 74 ++------------- .../docs/development/upstream-issues.md | 12 +-- bindings/python/mkdocs.yml | 4 +- make-upstream-pr-branch.sh | 18 ++++ sync-upstream.sh | 37 +++++++- 11 files changed, 119 insertions(+), 162 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 88679a08248..7c0547434cc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,6 +7,6 @@ Our work is in two places, each with its own rules: - `bindings/python/`: the Python bindings. Read `bindings/python/AGENTS.md` before changing anything there. - `benchmarks/`: the benchmark harness and its published results. Read `benchmarks/AGENTS.md` (and its `README.md`) before changing anything there. -The Java engine under `engine/` and the per-language `e2e-*` suites belong to upstream. Report a cause there instead of changing it here: file an issue, and send a pull request only for a low-hanging fix (`bindings/python/docs/development/upstream-issues.md`). The periodic `bindings/python` update to upstream is a separate routine (`bindings/python/docs/development/upstream-bindings-pr.md`). +The Java engine under `engine/` and the per-language `e2e-*` suites belong to upstream. Report a cause there instead of changing it here: file an issue, and send a pull request only for a low-hanging fix (`bindings/python/docs/development/upstream-issues.md`). The periodic `bindings/python` update to upstream is retired: upstream deleted `bindings/python` in ArcadeData/arcadedb#10615 (2026-10-23) and the bindings continue in ArcadeData/arcadedb-python (`bindings/python/docs/development/upstream-bindings-pr.md`). -Planned changes, not final and not yet public: the Python bindings may move to their own repository and PyPI project, and the benchmarks to `humemai/dbbench`, in mid-November 2026, after the last measuring pass and before the paper is submitted. Until then do not rename or move this repository, and do not change the release or upstream-sync routines without asking. The record is `.notes/bench/DECISIONS.md` #182 and #190. +Planned changes, not final and not yet public: the Python bindings may move to their own repository and PyPI project, and the benchmarks to `humemai/dbbench`, in mid-November 2026, after the last measuring pass and before the paper is submitted. Until then do not rename or move this repository, and do not change the release routine without asking. The upstream-sync routine (`sync-upstream.sh` and `make-upstream-pr-branch.sh`) is retired by the user's decision of 2026-10-25 ("Stop syncing"): do not run it, because a sync would merge upstream's deletion of `bindings/python` (#10615) into this fork (`bindings/python/docs/development/sync-upstream.md`). The record is `.notes/bench/DECISIONS.md` #182 and #190. diff --git a/benchmarks/experiments/CAMPAIGN.md b/benchmarks/experiments/CAMPAIGN.md index 86dfe226a4c..459535f5197 100644 --- a/benchmarks/experiments/CAMPAIGN.md +++ b/benchmarks/experiments/CAMPAIGN.md @@ -8,7 +8,7 @@ Per campaign, once: 1. Settle the instrument before any cell runs: the query set, the measurement set, the durability rule, the fairness invariants, and the answer checking. A change mid-campaign splits the rows, so this is the only free moment (DECISIONS #82d, #88, #89, #90; PROTOCOL.md). The method pages in the package documentation (`bindings/python/docs/benchmarks/`) describe the same instrument without numbers, so they change in this step too, never later. 2. Smoke every adapter, one cell per engine per lane, and keep the evidence (the smoke's rows and logs). Smoke wherever a machine is free: nothing in the fairness contract touches a smoke, and the bench host is reserved only while a published cell runs, so a smoke on the real corpora there is both faster and a better test than a capped slice on the laptop. One runner per host, always: `runner.py` sweeps every `dbbench=1` container on the machine at startup, so it takes a host-wide lock first (`$TMPDIR/dbbench-runner.lock`, whichever checkout it runs from) and a second runner is refused at startup (BUGS F58). -3. Pin ArcadeDB: sync upstream to the release the campaign runs and build the wheel by the routine in `bindings/python/docs/development/upstream-bindings-pr.md`, run our Java reproductions on the built engine to confirm every fix it should carry is intact, and record the engine commit inside that wheel as the campaign's pin (the pin is the commit, not the version string, DECISIONS #49). For a stable release the pin is the release's own jar commit (26.10.1: `d36b4ca3a`), the wheel is the official jars plus this tree's bridge and package, and the pair is placed by the recipe in `.notes/bench/repin-round5/REPIN-START.md`. The served arm's matched pair is built from the same image by `build_matched_pair.sh` (HANDOFF.md). One wheel, one pin, held start to finish. +3. Pin ArcadeDB: build the wheel for the release the campaign runs (the upstream sync is retired, so the engine tree does not move and the jars come from the release image; `bindings/python/docs/development/build-architecture.md#local-build`), run our Java reproductions on the built engine to confirm every fix it should carry is intact, and record the engine commit inside that wheel as the campaign's pin (the pin is the commit, not the version string, DECISIONS #49). For a stable release the pin is the release's own jar commit (26.10.1: `d36b4ca3a`), the wheel is the official jars plus this tree's bridge and package, and the pair is placed by the recipe in `.notes/bench/repin-round5/REPIN-START.md`. The served arm's matched pair is built from the same image by `build_matched_pair.sh` (HANDOFF.md). One wheel, one pin, held start to finish. 4. Re-pin every comparator to its latest stable release, smoking the risky jumps on their own (COMPARATORS.md, DECISIONS #87). 5. Run the campaign, landing each stage into the freeze (section 6 below). 6. Regenerate the query budgets from the campaign's own rows (`python3 derive_budgets.py > budgets.py` reads the freeze, `results/runs_paper.csv`) so the next campaign's caps are measurements at the corpora it will run rather than projections, and commit the table with the freeze (DECISIONS #106). diff --git a/bindings/python/AGENTS.md b/bindings/python/AGENTS.md index 99fdc5ef910..35f595d7804 100644 --- a/bindings/python/AGENTS.md +++ b/bindings/python/AGENTS.md @@ -13,9 +13,9 @@ Paths in this file are relative to the repository root (the directory that conta | changing docs | `bindings/python/docs/development/documentation.md` | | touching CI or the wheel build | `bindings/python/docs/development/ci-setup.md`, `bindings/python/docs/development/build-architecture.md` | | touching the Java bridge or the architecture | `bindings/python/docs/development/bridge.md`, `bindings/python/docs/development/architecture.md` | -| syncing upstream, or testing an engine fix | `bindings/python/docs/development/sync-upstream.md` | +| testing an engine fix, or why the upstream sync is retired (do not run `sync-upstream.sh`) | `bindings/python/docs/development/sync-upstream.md` | | reporting an engine bug upstream, sending a low-hanging engine fix, or checking an upstream fix to one of our issues | `bindings/python/docs/development/upstream-issues.md` | -| sending the periodic `bindings/python` update to ArcadeData/arcadedb | `bindings/python/docs/development/upstream-bindings-pr.md` | +| the periodic `bindings/python` update to ArcadeData/arcadedb (retired 2026-10-25; the bindings continue in ArcadeData/arcadedb-python) | `bindings/python/docs/development/upstream-bindings-pr.md` | | releasing | `bindings/python/docs/development/release.md` | | debugging a build, JVM or test failure | `bindings/python/docs/development/troubleshooting.md` | | making a performance claim, or working around an engine bug | `bindings/python/docs/guide/performance.md`, `bindings/python/docs/guide/known-issues.md`, `bindings/python/docs/benchmarks/protocol.md` | diff --git a/bindings/python/docs/development/contributing.md b/bindings/python/docs/development/contributing.md index a720c94a39d..f3a98096ef8 100644 --- a/bindings/python/docs/development/contributing.md +++ b/bindings/python/docs/development/contributing.md @@ -394,8 +394,8 @@ new page to the `nav` in `mkdocs.yml`. git clone https://github.com/YOUR_USERNAME/arcadedb-embedded-python.git cd arcadedb-embedded-python/bindings/python -# Track this repository. Do not call the remote "upstream": this repository's own -# scripts (sync-upstream.sh) use that name for ArcadeData/arcadedb. +# Track this repository. Do not call the remote "upstream": in this repository's +# checkout that name is ArcadeData/arcadedb (sync-upstream.sh, now retired, used it). git remote add humemai https://github.com/humemai/arcadedb-embedded-python.git ``` diff --git a/bindings/python/docs/development/release.md b/bindings/python/docs/development/release.md index 7a742b11969..7eb5b862e70 100644 --- a/bindings/python/docs/development/release.md +++ b/bindings/python/docs/development/release.md @@ -11,9 +11,11 @@ Complete workflow for releasing ArcadeDB Python bindings to PyPI with versioned ## Where the Version Comes From This fork does not set its own version. The wheel version is derived from the root -`pom.xml`, and `pom.xml` belongs to upstream ArcadeDB: it arrives with each -`./sync-upstream.sh` run. Never edit it here; a local change would conflict with the -next sync. +`pom.xml`, and `pom.xml` belongs to upstream ArcadeDB: it used to arrive with each +`./sync-upstream.sh` run. + +!!! warning "The upstream sync is retired (2026-10-25)" + ArcadeData/arcadedb deleted `bindings/python` in #10615 (2026-10-23), and by the user's decision of 2026-10-25 ("Stop syncing") `sync-upstream.sh` no longer runs ([Syncing Upstream](sync-upstream.md)). `pom.xml` therefore no longer advances on its own, and steps 1, 7 and 8 below, which were written around the sync, are retired. Do not cut a release by this checklist until the version source has been decided: ask first (see the root `AGENTS.md`). - Between releases, upstream's `main` reads `X.Y.Z-SNAPSHOT`, and the wheel builds as `X.Y.Z.dev0`. - Upstream cuts a release with a commit titled "Set release version to X.Y.Z", which @@ -23,14 +25,16 @@ next sync. !!! warning "Upstream uses the same version numbers" ArcadeDB's release tags carry the same numbers as ours (`26.9.1` is both their engine release and our wheel release). Always fetch `upstream` with `--no-tags` - (`sync-upstream.sh` does). If their tag is ever fetched, `git tag -a X.Y.Z` fails + (the retired `sync-upstream.sh` did). If their tag is ever fetched, `git tag -a X.Y.Z` fails with "already exists", but `git push origin X.Y.Z` still succeeds and publishes upstream's commit. Check what a tag points to before pushing it (step 4 below). `git config remote.upstream.tagopt --no-tags` makes `git fetch upstream` skip tags for good. ## Release Checklist (stable `X.Y.Z`) -### 1. Sync to Upstream's Release Commit +### 1. Sync to Upstream's Release Commit (retired) + +This step no longer runs: `sync-upstream.sh` is retired. What follows is the record of how it worked. ```bash # Find upstream's release commit without fetching its tags @@ -41,10 +45,8 @@ git log upstream/main --oneline --grep "Set release version to X.Y.Z" ./sync-upstream.sh --until ``` -If the merge stops on conflicts, resolve them as described in -[Syncing Upstream](sync-upstream.md#when-the-merge-stops-on-conflicts), including the -fork-owned restore and the `.github` prune the script skips in that case. Afterwards -`pom.xml` reads `X.Y.Z`. +If the merge stopped on conflicts, the fork-owned restore and the `.github` prune had to be done by hand (the earlier version of [Syncing Upstream](sync-upstream.md) in git history has the steps). Afterwards +`pom.xml` read `X.Y.Z`. ### 2. Build and Test @@ -185,7 +187,9 @@ Read the previous release at that page and keep its shape: - a Security section for upstream advisories that matter to embedded use; - what the bindings added, and a pointer to [Known Engine Issues](../guide/known-issues.md). -### 7. After the Release +### 7. After the Release (retired) + +This step no longer runs: `sync-upstream.sh` is retired. What follows is the record. Return `main` to upstream's development line with a normal sync. It brings the next `-SNAPSHOT` version; there is no version bump to make here. @@ -194,7 +198,9 @@ Return `main` to upstream's development line with a normal sync. It brings the n ./sync-upstream.sh ``` -### 8. Open the Upstream Pull Request, Last +### 8. Open the Upstream Pull Request, Last (retired) + +This step no longer runs: upstream deleted `bindings/python`, and the bindings continue in ArcadeData/arcadedb-python. What follows is the record. Only now regenerate the `python-bindings` branch and open the pull request to ArcadeData/arcadedb: it is built on the `upstream-main` mirror that this sync just @@ -444,6 +450,6 @@ uv run --project .. --group docs mike alias --update-aliases \ - [Documentation Development](documentation.md) - Working with MkDocs - [Testing Guide](testing.md) - Running test suite - [Contributing Guide](contributing.md) - Development workflow -- [Syncing Upstream](sync-upstream.md) - How upstream changes arrive +- [Syncing Upstream](sync-upstream.md) - Retired; why, and how to test an engine fix - [GitHub Actions Docs](https://docs.github.com/en/actions) - [PyPI Publishing Guide](https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/) diff --git a/bindings/python/docs/development/sync-upstream.md b/bindings/python/docs/development/sync-upstream.md index 19091866b69..5a96bf6fe6a 100644 --- a/bindings/python/docs/development/sync-upstream.md +++ b/bindings/python/docs/development/sync-upstream.md @@ -1,90 +1,44 @@ -# Syncing Upstream +# Syncing Upstream (retired) -This fork tracks `ArcadeData/arcadedb` through the local `upstream-main` branch. +!!! danger "Retired on 2026-10-25. Do not run `sync-upstream.sh`." + ArcadeData/arcadedb deleted `bindings/python` in [#10615](https://github.com/ArcadeData/arcadedb/pull/10615) (commit `1e0507a522`, 2026-10-23, 143 files). A sync would merge that deletion into this fork. The decision was "Stop syncing" (2026-10-25). The script now exits with an error and a message that names the decision; only `./sync-upstream.sh --help` still prints (the old help, for the record). -## Commands +Nothing needs the engine tree synced any more: -```bash -./sync-upstream.sh --status # how far main and upstream-main are behind -./sync-upstream.sh --dry-run # list what would be merged, change nothing -./sync-upstream.sh # sync to upstream/main -./sync-upstream.sh --until # sync only up to a given upstream commit -``` - -`--until` also works with `--status` and `--dry-run`. It is how a release is cut: sync -up to upstream's "Set release version to X.Y.Z" commit (see -[Release Workflow](release.md)). - -## What it does +- The wheels take their engine JARs from the `arcadedata/arcadedb:` release image, not from the Java sources in this tree. +- Pull requests to ArcadeData/arcadedb branch from upstream's `main` in the `tae898/arcadedb` fork, never from this repository ([Upstream Issues and Fixes](upstream-issues.md)). +- The Python bindings continue in [ArcadeData/arcadedb-python](https://github.com/ArcadeData/arcadedb-python). -In order: +The engine tree in this repository is therefore frozen at the last sync. Do not rely on it being current, and do not edit it ([AGENTS.md](https://github.com/humemai/arcadedb-embedded-python/blob/main/AGENTS.md)). -1. Fetches `upstream` with `--no-tags` (upstream's release tags use the same numbers as ours) and shows the commits it would merge. `--status` and `--dry-run` stop here. -2. Refuses to continue if tracked files have uncommitted changes. -3. Asks `Continue with merge? [y/N]` and waits. There is no flag to skip it; from a non-interactive shell, pipe the answer (`printf y | ./sync-upstream.sh`), or the script exits silently at the prompt without merging. -4. Resets `upstream-main` to the sync target (`upstream/main`, or the `--until` commit). -5. Merges `upstream-main` into `main`. -6. Keeps this fork's own copies of the paths listed in `FORK_OWNED_PATHS` in the script (the root `README.md`, `pyproject.toml`, `uv.lock`, and a few others), and keeps the paths in `FORK_EXCLUDED_PATHS` (such as `CLAUDE.md`) out of the fork. -7. Treats `.github/` as an **allowlist**: after the merge it deletes every tracked file under `.github/` that is not named in `FORK_GITHUB_ALLOWLIST`, so no upstream workflow arrives. A list of files to delete could only name the ones that existed when it was written; the allowlist also stops the ones upstream adds later. Adding a workflow of our own therefore means adding it to that list in the same commit, or the next sync deletes it. A file that upstream also has at the same path and that we replace with our own (`.github/dependabot.yml`: ours updates the pinned GitHub Actions, upstream's is a Maven config) goes in both `FORK_GITHUB_ALLOWLIST` and `FORK_OWNED_PATHS`, so the prune keeps it and the sync never merges upstream's version into it. +## What replaces each use -The lists live in `sync-upstream.sh` and nowhere else on purpose. +| You wanted to | Do this instead | +|---|---| +| Get upstream's newer engine into the wheel | Nothing is synced: build the wheel against the release image's JARs (`build.sh` does this by default), or against JARs you built from any upstream commit (below). | +| Test an upstream engine fix | Build the engine JARs of the upstream commit and pass their directory to `build.sh` (below). | +| Send the bindings to ArcadeData/arcadedb | Retired too: [Sending the Bindings Upstream](upstream-bindings-pr.md). | +| Cut a release | [Release Workflow](release.md) (read its note on the version source first). | -## When the merge stops on conflicts +## Testing an engine fix without a sync -The script resolves conflicts on its own only for the paths in `FORK_OWNED_PATHS` and -`FORK_EXCLUDED_PATHS`. Any other conflict makes it print the conflict steps and exit -**before** steps 6 and 7, so the fork-owned restore and the `.github` prune do not run. -The usual case is a modify/delete conflict on a `.github/` file that an earlier sync -pruned and upstream has since changed. Finish by hand: - -```bash -# 1. See what conflicts -git diff --name-only --diff-filter=U - -# 2. A pruned .github file: resolve it as a deletion -git rm --cached && rm -f - -# 3. Resolve anything else normally, then finish the merge -GIT_EDITOR=true git merge --continue -``` - -Then do what the script skipped: - -- Delete every tracked file under `.github/` that is not in `FORK_GITHUB_ALLOWLIST`. -- Check that the `FORK_OWNED_PATHS` files are unchanged against the pre-merge commit - (`git diff -- README.md pyproject.toml uv.lock ...`), and restore - any that changed. -- Check that the `FORK_EXCLUDED_PATHS` files (such as `CLAUDE.md`) are absent. -- Commit, and run `./sync-upstream.sh --status`. - -## After sync +Get the JARs of an upstream commit without building on your machine: run the `Build ArcadeDB engine jars` workflow on the commit's full 40-character SHA and download its `arcadedb-lib` artifact ([Upstream Issues and Fixes](upstream-issues.md#reporting-an-engine-bug) has the commands). Then: ```bash cd bindings/python -./scripts/build.sh linux/amd64 +./scripts/build.sh linux/amd64 3.12 uv run pytest -git push origin main ``` -The build takes its engine JARs from the `arcadedata/arcadedb:` image, not -from the Java sources you just synced. To test the synced engine code itself, build the -engine JARs and pass their directory as `build.sh`'s third argument (`JAR_LIB_DIR`); see -[Build Architecture](build-architecture.md#local-build). - -This is step one of the contribution routine. The whole order, through to regenerating the pull-request branch, is in [Sending the Bindings Upstream](upstream-bindings-pr.md); checking an upstream fix to one of our issues is in [Upstream Issues and Fixes](upstream-issues.md). +The third argument is the JAR directory; an environment variable named `JAR_LIB_DIR` is ignored. See [Build Architecture](build-architecture.md#local-build). ## Verifying an upstream fix -When an issue we filed is fixed upstream, the way to check it is to sync, build the -wheel, and measure the thing the issue was about. Do this on a development machine, -never on a machine that is in the middle of a benchmark run: a long measurement run -should use one engine build from start to finish, and swapping in a freshly synced -wheel partway through would split its results across two engines in a way that cannot -be undone afterwards. +When an issue we filed is fixed upstream, the way to check it is to build the wheel on the fix's engine and measure the thing the issue was about. Do this on a development machine, never on a machine that is in the middle of a benchmark run: a long measurement run should use one engine build from start to finish, and swapping in a freshly built wheel partway through would split its results across two engines in a way that cannot be undone afterwards. The verification routine: -1. `./sync-upstream.sh`, then `./scripts/build.sh linux/amd64` in `bindings/python`. If the fix has not reached a published upstream image yet, pass the engine's JAR directory as `build.sh`'s third argument (`./scripts/build.sh linux/amd64 3.12 `); otherwise the wheel does not contain it. An environment variable named `JAR_LIB_DIR` is ignored. +1. Build the engine JARs of the fix's merge commit (above), then `./scripts/build.sh linux/amd64 3.12 ` in `bindings/python`. If the fix is already in a published upstream image, `build.sh` without the third argument is enough; otherwise the wheel does not contain the fix. 2. Re-run the issue's own reproduction against the new build. A Java reproduction compiles against an image's `lib/*` and needs no wheel at all, which makes it the cheaper check where one exists. 3. Compare against the same reproduction on the build the issue was filed against. A fix is "verified" when the repro that reproduced stops reproducing, on the same host, not when the issue is closed. **Mind the JVM.** Upstream's published images run **Java 21**; the wheel bundles a **Java 25** JRE, and the embedded JVM adds `-XX:+UseCompactObjectHeaders` unless the JVM arguments already set it, a flag Java 21 will not even start with. Verify on the image's own Java when replying upstream (that is what the issue was filed on), and, for any fix that affects the bindings, run the same repro again with that build's jars on Java 25 and the same flag. A fix confirmed on a JVM we do not ship is not yet confirmed for us. @@ -92,3 +46,7 @@ The verification routine: 5. Apply what the answer changes in practice across the package, the same day: `src/`, `tests/`, `examples/`, and `docs/`, not only the code that found the issue. A maintainer's answer often carries a recommendation (a setting, a statement, a pattern to avoid) beyond the fix itself. Install the rebuilt wheel first, so the new behavior can be tested locally; a test for a new feature should fail on the last release (`uv run --no-project --with arcadedb-embedded== python -m pytest ...`) and must not pass vacuously on the new one. The routine is not finished at the reply. A benchmark host moves to a new engine only between runs, as a deliberate step: re-measure what the engine affects, and re-run an untouched comparison as a control. + +## What the sync did (record) + +For the record, the script kept a local `upstream-main` mirror of `upstream/main`, merged it into `main`, restored the fork-owned paths (`README.md`, `pyproject.toml`, `uv.lock`, and a few others) and kept `CLAUDE.md` out, then deleted every tracked file under `.github/` that was not on an allowlist, so no upstream workflow arrived. The lists, and the reasons behind them, are in the script's comments and in its git history. The local `upstream-main` branch is no longer maintained. diff --git a/bindings/python/docs/development/upstream-bindings-pr.md b/bindings/python/docs/development/upstream-bindings-pr.md index 66a9722bc01..8ab5781d25a 100644 --- a/bindings/python/docs/development/upstream-bindings-pr.md +++ b/bindings/python/docs/development/upstream-bindings-pr.md @@ -1,72 +1,12 @@ -# Sending the Bindings Upstream +# Sending the Bindings Upstream (retired) -Changes made here reach ArcadeData/arcadedb through one branch, `python-bindings`, regenerated from `main` rather than edited. The order below matters: each step exists because skipping it has cost something. +!!! danger "Retired on 2026-10-25. Do not run `make-upstream-pr-branch.sh`." + ArcadeData/arcadedb deleted `bindings/python` in [#10615](https://github.com/ArcadeData/arcadedb/pull/10615) (commit `1e0507a522`, 2026-10-23, 143 files). There is no upstream tree left to update, so the periodic pull request has nothing to carry, and the sync it was built on is retired too ([Syncing Upstream](sync-upstream.md)). The script now exits with an error and a message that names the decision ("Stop syncing", 2026-10-25). -## 1. Sync upstream into main +The Python bindings continue in [ArcadeData/arcadedb-python](https://github.com/ArcadeData/arcadedb-python). Send bindings changes there. -```bash -./sync-upstream.sh -``` +Engine bugs and low-hanging engine fixes still go to ArcadeData/arcadedb, by a different route that does not touch this repository's tree: [Upstream Issues and Fixes](upstream-issues.md). -Never hand-merge. The script owns the allowlist of what crosses and maintains the `upstream-main` mirror the pull-request branch is built on. See [Syncing Upstream](sync-upstream.md). +## What the routine was (record) -## 2. Reconcile the bindings with what arrived - -Upstream changes can move the Java surface the bindings wrap. Read what the sync brought and update the source, tests, examples, and documentation where it lands. A sync that compiles is not a sync that is correct: run the tests and the examples, not just the build. - -```bash -cd bindings/python -./scripts/build.sh linux/amd64 -uv run pytest -``` - -`build.sh` without a third argument packages the JARs of the published -`arcadedata/arcadedb:` image, which can lag the sources you just synced. -To test the synced engine code, build its JARs and pass their directory as the third -argument, `JAR_LIB_DIR` (see [Build Architecture](build-architecture.md#local-build)). - -## 3. Push and let CI prove it - -```bash -git push origin main -``` - -Wait for the workflows to finish and read them. A run that reports success while a job skipped its real work is not a pass: the examples workflow skips a dataset-dependent example when its third-party host is unreachable. Check that the examples you care about actually ran. - -## 4. Regenerate the pull-request branch - -```bash -./make-upstream-pr-branch.sh # regenerate and force push -./make-upstream-pr-branch.sh --no-push # regenerate only -``` - -The branch is never edited directly. It is rebuilt from `main` on top of `upstream-main`, so `main` stays the single source of truth and the branch can always be thrown away. - -What crosses is the bare minimum: upstream's existing `bindings/python` tree with our file contents, plus whatever the wheel build, the tests, and the examples strictly need. Fork-only tooling, the benchmark harness, the notes, and result archives never cross. - -What the script does beyond that: - -- Besides `bindings/python`, it ships `.github/workflows/test-python-bindings.yml` and `.github/workflows/test-python-examples.yml`, so an edit to either workflow must also work in upstream's repository. -- Our copies of those two workflows are never synced from upstream (they are on the sync allowlist as ours), so they can lag upstream's action pins: after regenerating, run `git diff upstream-main python-bindings -- .github/workflows`. A workflow change always crosses when it changes which engine jars the tests run against (the `engine-from-source` input, `JAR_LIB_DIR`) or what the new tests need to run (a package, an environment variable, a step), and the pull request says why. If the only differences are older pins or changes the new tests do not need, restore upstream's files on the branch (`git checkout upstream-main -- .github/workflows/test-python-bindings.yml .github/workflows/test-python-examples.yml`, then amend), so the pull request does not revert upstream's bumps. The 26.10.2 update did this (ArcadeData/arcadedb#10120). -- It leaves out `examples/benchmark_results`, `examples/scripts`, `scripts/profile-python`, `scripts/fix_markdown.py`, `scripts/list_image_jars_by_size.sh`, and `scripts/build_and_install_locally.sh`. -- It refuses to run while tracked files have uncommitted changes, and it warns when the branch has a top-level entry under `bindings/python` that upstream's tree lacks. -- It ships only the tree: set the pull request's title and body by hand. The earlier ones - (ArcadeData/arcadedb#5807, #6889) open with "Periodic update of `bindings/python` from the - downstream packaging fork (humemai/arcadedb-embedded-python), rebased on current `main`. - No engine code.", say that the change has been shipping on PyPI as `arcadedb-embedded`, - most recently as which version, then explain each file that crosses outside - `bindings/python` and why, show the result of upstream's checks on the first push, and - state the scope. In a release cycle the pull request comes last, after the release and - the sync that follows it ([Release Workflow](release.md#8-open-the-upstream-pull-request-last)). - -## What upstream expects - -Their checks run on the pull request and some of them are stricter than ours, so run their pre-commit configuration locally before pushing the branch, over everything that crosses: - -```bash -uvx pre-commit run --files $(git ls-files 'bindings/python/**' .github/workflows/test-python-bindings.yml .github/workflows/test-python-examples.yml) -``` - -Our own `pre-commit` job covers only `bindings/python`, and the two workflows cross too. - -Reporting an engine bug, and checking a fix to one of our issues, are in [Upstream Issues and Fixes](upstream-issues.md). +Changes made here reached ArcadeData/arcadedb through one branch, `python-bindings`, regenerated from `main` on top of the `upstream-main` mirror by `make-upstream-pr-branch.sh`, after `sync-upstream.sh` and a reconcile of the bindings with whatever the sync brought. It carried upstream's existing `bindings/python` tree with our file contents, plus the two test workflows, and nothing else. The earlier pull requests were ArcadeData/arcadedb#5807, #6889 and #10120 (the 26.10.2 update). The script keeps the full rules in its comments and its git history. diff --git a/bindings/python/docs/development/upstream-issues.md b/bindings/python/docs/development/upstream-issues.md index 077ec1562cd..743bd641107 100644 --- a/bindings/python/docs/development/upstream-issues.md +++ b/bindings/python/docs/development/upstream-issues.md @@ -1,6 +1,6 @@ # Upstream Issues and Fixes -How an engine bug found here reaches ArcadeData/arcadedb, and how we check the fix. The periodic update of `bindings/python` is a separate routine: [Sending the Bindings Upstream](upstream-bindings-pr.md). +How an engine bug found here reaches ArcadeData/arcadedb, and how we check the fix. The periodic update of `bindings/python` to ArcadeData/arcadedb is retired (upstream deleted `bindings/python` in #10615; the bindings continue in ArcadeData/arcadedb-python): [Sending the Bindings Upstream](upstream-bindings-pr.md). ## Reporting an engine bug @@ -27,15 +27,15 @@ Most issues stay issues. Send a pull request only when the fix is small and you - Branch from upstream's `main` in the `tae898/arcadedb` fork (the `fork` remote; its Actions are disabled, so its pushes cost nothing) and open the pull request with `gh pr create -R ArcadeData/arcadedb --head tae898:`. - The body references the issue (`Fixes #N`). Add a test that fails before the change and passes after it, put the SPDX header on new files, and run upstream's pre-commit configuration, as upstream's `CONTRIBUTING.md` asks. -- Keep it to the engine change. The periodic bindings update is a different pull request. +- Keep it to the engine change. ## When upstream fixes one of our issues -The same four steps of [Sending the Bindings Upstream](upstream-bindings-pr.md) run in a different order, with two additions, and the whole loop belongs to one sitting: a fix verified today against an engine that moves every day is a claim with an expiry date. +The whole loop belongs to one sitting: a fix verified today against an engine that moves every day is a claim with an expiry date. The upstream sync is retired ([Syncing Upstream](sync-upstream.md)), so the fixed engine arrives as built JARs, not as a merge. -1. **Sync** (step 1) through the merge commit of the fix, and reconcile (step 2) whatever the sync touched. -2. **Verify with the reproduction that was filed**, never with the description of the fix. The reproduction is Java against the engine API, kept with the issue. Run it on the release the issue was filed against and on the synced engine, same JDK, and keep both outputs. Every number the reproduction prints must be exact afterwards; one that is merely better is a different bug. Re-run the reproductions of the fixes verified earlier on the same engine, since a later merge can undo an earlier one. -3. **Build, test, push, and read the workflows** (step 3), with the wheel's engine jar named by its file inside the wheel rather than by the wheel's version string. If the fix is not yet in a published upstream image, pass the engine's JAR directory as `build.sh`'s third argument (`./scripts/build.sh linux/amd64 3.12 `) so the wheel contains it; an environment variable named `JAR_LIB_DIR` is ignored. A sync that touches only Java does not trigger the two test workflows, which are path-filtered to the bindings; dispatch them by hand on the pushed commit (`gh workflow run "Test Python Bindings" --ref main`, and the same for the examples) and read those runs. +1. **Build the engine JARs** of the fix's merge commit with the `Build ArcadeDB engine jars` workflow (above), and reconcile the bindings with whatever that commit changed in the Java surface they wrap. +2. **Verify with the reproduction that was filed**, never with the description of the fix. The reproduction is Java against the engine API, kept with the issue. Run it on the release the issue was filed against and on the fixed engine, same JDK, and keep both outputs. Every number the reproduction prints must be exact afterwards; one that is merely better is a different bug. Re-run the reproductions of the fixes verified earlier on the same engine, since a later merge can undo an earlier one. +3. **Build, test, push, and read the workflows**, with the wheel's engine jar named by its file inside the wheel rather than by the wheel's version string. If the fix is not yet in a published upstream image, pass the engine's JAR directory as `build.sh`'s third argument (`./scripts/build.sh linux/amd64 3.12 `) so the wheel contains it; an environment variable named `JAR_LIB_DIR` is ignored. A change outside the bindings does not trigger the two test workflows, which are path-filtered to the bindings; dispatch them by hand on the pushed commit (`gh workflow run "Test Python Bindings" --ref main`, and the same for the examples) and read those runs. 4. **Reply once on the issue** with the before-and-after table, the commit and JDK tested, and that the bindings suite is green on that engine. Read the thread first so the reply is not a duplicate. Nothing that was not run goes in it. 5. **Update the tracking the same day**: record the fix, its pull request, and the verification next to the issue, and mark the issue fixed with the numbers. Any workaround that exists because of the defect is marked for removal at the next deliberate engine upgrade of whatever depends on it, not removed now, because anything still running on the old engine still has the defect. diff --git a/bindings/python/mkdocs.yml b/bindings/python/mkdocs.yml index 7fea9587b5f..75fb487e95c 100644 --- a/bindings/python/mkdocs.yml +++ b/bindings/python/mkdocs.yml @@ -219,8 +219,8 @@ nav: - CI/CD Setup: development/ci-setup.md - Contributing: development/contributing.md - Documentation: development/documentation.md - - Syncing Upstream: development/sync-upstream.md - - Sending the Bindings Upstream: development/upstream-bindings-pr.md + - Syncing Upstream (retired): development/sync-upstream.md + - Sending the Bindings Upstream (retired): development/upstream-bindings-pr.md - Upstream Issues and Fixes: development/upstream-issues.md - Testing: development/testing.md - Release Workflow: development/release.md diff --git a/make-upstream-pr-branch.sh b/make-upstream-pr-branch.sh index d593caeec1d..2ce579a6bd0 100755 --- a/make-upstream-pr-branch.sh +++ b/make-upstream-pr-branch.sh @@ -1,4 +1,6 @@ #!/usr/bin/env bash +# RETIRED on 2026-10-25: this script refuses to run (see the guard below). +# # Regenerate the `python-bindings` branch for the periodic upstream PR # (ArcadeData/arcadedb). Encodes the rule proven on PR #5010: # @@ -23,6 +25,22 @@ set -euo pipefail RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m' +# RETIRED 2026-10-25 (user decision "Stop syncing"). Do not remove this guard. +# +# ArcadeData/arcadedb deleted bindings/python in #10615 (commit 1e0507a522, +# 2026-10-23, 143 files), so there is no upstream tree to send the bindings to: +# the regenerated branch would add all of bindings/python back as a new tree. +# The bindings continue in ArcadeData/arcadedb-python. The rest of this file is +# kept as the record of how the periodic upstream PR was made; it is unreachable. +echo -e "${RED}RETIRED: make-upstream-pr-branch.sh no longer runs.${NC}" >&2 +echo "" >&2 +echo "ArcadeData/arcadedb deleted bindings/python in #10615 (commit 1e0507a522," >&2 +echo "2026-10-23, 143 files), so the periodic bindings PR to it has no tree to update." >&2 +echo "Decision: \"Stop syncing\" (the user, 2026-10-25). The Python bindings continue in" >&2 +echo "https://github.com/ArcadeData/arcadedb-python" >&2 +echo "See bindings/python/docs/development/upstream-bindings-pr.md." >&2 +exit 1 + BRANCH="python-bindings" PUSH_REMOTE="${PUSH_REMOTE:-fork}" BASE="upstream-main" diff --git a/sync-upstream.sh b/sync-upstream.sh index 72bf6e9b8f1..08b3eab56ab 100755 --- a/sync-upstream.sh +++ b/sync-upstream.sh @@ -1,4 +1,6 @@ #!/bin/bash +# RETIRED on 2026-10-25: this script refuses to run (see the guard below). +# # Script to sync upstream changes while preserving fork-specific customizations # # This fork maintains two protected path categories during sync: @@ -20,13 +22,41 @@ BLUE='\033[0;34m' CYAN='\033[0;36m' NC='\033[0m' # No Color +# RETIRED 2026-10-25 (user decision "Stop syncing"). Do not remove this guard. +# +# ArcadeData/arcadedb deleted bindings/python in #10615 (commit 1e0507a522, +# 2026-10-23, 143 files). Running this script would merge that deletion into +# this fork. Nothing needs the engine tree synced any more: the wheels take +# their jars from the release image, and upstream engine PRs branch from +# upstream/main. The Python bindings continue in ArcadeData/arcadedb-python. +# The rest of this file is kept as the record of how the sync worked; it is +# unreachable except for --help. +retired() { + echo -e "${RED}RETIRED: sync-upstream.sh no longer runs.${NC}" >&2 + echo "" >&2 + echo "ArcadeData/arcadedb deleted bindings/python in #10615 (commit 1e0507a522," >&2 + echo "2026-10-23, 143 files). A sync would merge that deletion into this fork." >&2 + echo "Decision: \"Stop syncing\" (the user, 2026-10-25). Nothing needs the engine tree" >&2 + echo "synced any more: wheels take their jars from the release image, and upstream" >&2 + echo "engine PRs branch from upstream/main." >&2 + echo "" >&2 + echo "The Python bindings continue in https://github.com/ArcadeData/arcadedb-python" >&2 + echo "See bindings/python/docs/development/sync-upstream.md. './sync-upstream.sh --help'" >&2 + echo "still prints the old help." >&2 + exit 1 +} +case "${1:-}" in + --help|-h) ;; + *) retired ;; +esac + FORK_OWNED_PATHS=( "README.md" # Repo-root uv dev environment (fork-only; upstream has no root Python project) "pyproject.toml" "uv.lock" ".python-version" - # Fork tooling for the periodic upstream bindings PR + # Fork tooling for the periodic upstream bindings PR (retired 2026-10-25) "make-upstream-pr-branch.sh" ".github/dependabot.yml" # ours since #378 (github-actions only); upstream has a maven one at the same path: never merge theirs ) @@ -218,6 +248,11 @@ print_list() { show_help() { cat << EOF +RETIRED 2026-10-25: this script refuses to run. Only --help works. +ArcadeData/arcadedb deleted bindings/python in #10615 (1e0507a522, 2026-10-23); a sync +would merge that deletion into this fork. See bindings/python/docs/development/sync-upstream.md. +The text below is the old help, kept as the record. + ${BLUE}🔄 Upstream Sync Script${NC} Syncs this fork with upstream ArcadeData/arcadedb while keeping: From e28136881d12371231e491f63a77116163670cd4 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Sun, 25 Oct 2026 16:20:35 +0900 Subject: [PATCH 2/2] docs(release): pom.xml stays at 26.11.1-SNAPSHOT; the remaining releases here match it Co-Authored-By: Claude Opus 5.5 --- bindings/python/docs/development/release.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bindings/python/docs/development/release.md b/bindings/python/docs/development/release.md index 7eb5b862e70..699ea364989 100644 --- a/bindings/python/docs/development/release.md +++ b/bindings/python/docs/development/release.md @@ -15,7 +15,7 @@ This fork does not set its own version. The wheel version is derived from the ro `./sync-upstream.sh` run. !!! warning "The upstream sync is retired (2026-10-25)" - ArcadeData/arcadedb deleted `bindings/python` in #10615 (2026-10-23), and by the user's decision of 2026-10-25 ("Stop syncing") `sync-upstream.sh` no longer runs ([Syncing Upstream](sync-upstream.md)). `pom.xml` therefore no longer advances on its own, and steps 1, 7 and 8 below, which were written around the sync, are retired. Do not cut a release by this checklist until the version source has been decided: ask first (see the root `AGENTS.md`). + ArcadeData/arcadedb deleted `bindings/python` in #10615 (2026-10-23), and by the user's decision of 2026-10-25 ("Stop syncing") `sync-upstream.sh` no longer runs ([Syncing Upstream](sync-upstream.md)). `pom.xml` therefore no longer advances on its own, and steps 1, 7 and 8 below, which were written around the sync, are retired. It does not need to advance: `pom.xml` stays at `26.11.1-SNAPSHOT`, and the only releases this repository still cuts match that base, the `26.11.1.devN` native previews and the final `26.11.1` warning shims for the old package names. From 26.11.1 on, `arcadedb` and `arcadedb-native` are released from ArcadeData/arcadedb-python, and this repository is archived. Any other release from here: ask first (see the root `AGENTS.md`). - Between releases, upstream's `main` reads `X.Y.Z-SNAPSHOT`, and the wheel builds as `X.Y.Z.dev0`. - Upstream cuts a release with a commit titled "Set release version to X.Y.Z", which