diff --git a/.github/ISSUE_TEMPLATE/release.md b/.github/ISSUE_TEMPLATE/release.md index 1ac1c146..3e1129a7 100644 --- a/.github/ISSUE_TEMPLATE/release.md +++ b/.github/ISSUE_TEMPLATE/release.md @@ -61,7 +61,7 @@ Every step says who does it: script (run the command, read the result), CI (a wo uv run --no-project python scripts/bump_engine.py --engine --dev-version ``` -- [ ] 8. script: refresh the notices the new engine needs. First refresh the kept engine NOTICE (`git -C show :NOTICE > native/wheel/ARCADEDB_ENGINE_NOTICE`; the generated notices embed it). The first `Test native` run on the bump PR stops at the notices drift check on purpose and uploads the generated file. Download it, commit it, push, and wait for the run to go green. +- [ ] 8. script: refresh the notices the new engine needs. First refresh the kept engine NOTICE (`git -C show :NOTICE > native/wheel/ARCADEDB_ENGINE_NOTICE`; the generated notices embed it). The first `Test native (arcadedb)` run on the bump PR stops at the notices drift check on purpose and uploads the generated file. Download it, commit it, push, and wait for the run to go green. ```bash gh run list -R ArcadeData/arcadedb-python --workflow test-native.yml --branch --limit 1 diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index f5ae8cc2..c51e4294 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -8,7 +8,7 @@ A short description of the change. ## Performance changes -For a change that claims a speed or memory difference: the timed A/B result (`ab.py`, with the noise run), or say why none applies. Leave this out otherwise. +For a change that claims a speed or memory difference: the timed A/B result (interleaved runs of the new build against `main` and a control run of `main` against itself, see CONTRIBUTING.md), or say why none applies. Leave this out otherwise. ## Docs diff --git a/CLAUDE.md b/CLAUDE.md index a82c4c87..ad47f782 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -20,7 +20,7 @@ This repository builds two distributions that both import as `arcadedb`: `arcade ./scripts/build.sh # current platform, auto-detected ./scripts/build.sh linux/amd64 3.12 # Linux builds run in Docker (manylinux) ./scripts/build.sh linux/amd64 3.12 /path/to/lib # embed local ArcadeDB jars instead of the image's -ARCADEDB_TAG=26.11.1-SNAPSHOT ./scripts/build.sh # pin the arcadedata/arcadedb image (default: latest) +ARCADEDB_TAG=26.12.1-SNAPSHOT ./scripts/build.sh # pin the arcadedata/arcadedb image (default: latest) ``` - Linux builds use `scripts/Dockerfile.build`; macOS/Windows use `scripts/build-native.sh` with a native JDK 25+ (jlink). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d816b517..f19c318b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -51,7 +51,7 @@ On Linux the build runs in Docker. On macOS and Windows it needs a JDK 25 or lat ```shell ./scripts/build.sh # an arcadedb-jvm wheel for your platform in dist/ -ARCADEDB_TAG=26.11.1-SNAPSHOT ./scripts/build.sh # pin the arcadedata/arcadedb image (default: latest) +ARCADEDB_TAG=26.12.1-SNAPSHOT ./scripts/build.sh # pin the arcadedata/arcadedb image (default: latest) ``` Tests run against the built wheel, not the source tree. Use `python -m pytest`, not the `pytest` entry point: with `--with`, the entry point does not see the packages of the environment. @@ -74,18 +74,20 @@ bash native/scripts/build-image.sh uv run --no-project --python 3.12 python native/wheel/build_wheel.py --lib-dir --version --engine --out ``` +`build-image.sh` takes GraalVM Community Edition 25.4.4.1.1 from `GRAALVM_HOME_P5`; set it to your install. `build_wheel.py` takes `--graalvm-home`. + `native/wheel/parity_wheel.py` runs the test suite against the installed wheel; its arguments are in the "Run the parity suite" step of `test-native.yml`. See [Continuous integration](docs/modules/ROOT/pages/development/ci.adoc) for when each workflow runs. ### Tests - Every fix comes with a regression test. Write the test first and run every test the change could affect. -- No skips: CI fails on a skipped test that is not listed in `scripts/check_test_skips.py`. A test that cannot run on a platform goes in `collect_ignore` in `tests/conftest.py`, and a known engine bug uses a strict `xfail`. +- No skips: CI fails on a skipped test that is not listed in `scripts/check_test_skips.py`. A test file that cannot run on a platform is a row, with its reason, in `WINDOWS_NOT_APPLICABLE` or `NATIVE_NOT_APPLICABLE` in `tests/conftest.py` (both feed `collect_ignore`), and a known engine bug uses a strict `xfail`. - Never bind or address a hand-picked port in a test. Use `server.get_http_port()` or the `_free_ports()` pattern in `tests/test_server_wire_protocols.py`. - Never assert on raw wall-clock time. A JVM pause makes it flaky. ### Pre-commit -This project uses [pre-commit](https://pre-commit.com/) (black, isort, shfmt, pretty-format-yaml, actionlint). Run it on the files you changed before you push, and again if it reformatted something: +This project uses [pre-commit](https://pre-commit.com/) (black, isort, shfmt, pretty-format-yaml, actionlint, and a local check that the old package name does not come back). Run it on the files you changed before you push, and again if it reformatted something: ```shell uvx pre-commit run --files diff --git a/README.md b/README.md index 86b2ef3c..eec173d7 100644 --- a/README.md +++ b/README.md @@ -96,13 +96,13 @@ Releases from this repository start with 26.11.1. The 26.10.x wheels were built ## Documentation -- Here: [`docs/`](https://github.com/ArcadeData/arcadedb-python/tree/main/docs/modules/ROOT/pages) is an Antora component (AsciiDoc) with Installation, [Which package?](https://github.com/ArcadeData/arcadedb-python/blob/main/docs/modules/ROOT/pages/which-package.adoc), Quickstart, Databases and queries, Vectors, Faceted queries, and Concurrency, processes and crashes. It covers only what is specific to the Python bindings. +- Here: [`docs/`](https://github.com/ArcadeData/arcadedb-python/tree/main/docs/modules/ROOT/pages) is an Antora component (AsciiDoc) with Installation, [Which package?](https://github.com/ArcadeData/arcadedb-python/blob/main/docs/modules/ROOT/pages/which-package.adoc), Quickstart, the how-to pages (databases and queries, transactions, bulk loads, vectors, faceted queries, concurrency, durability, and more) and the development pages (Continuous integration, Release). It covers only what is specific to the Python bindings. - Published at [docs.arcadedb.com/arcadedb-python](https://docs.arcadedb.com/arcadedb-python/), the same site as the main ArcadeDB manual, rebuilt daily from `main` (ArcadeData/arcadedb-docs#505). The `Docs` workflow builds it as a site of its own on every change under `docs/` and attaches it as the `docs-site` artifact. To build it locally, run `npx -p @antora/cli@3.2.1 -p @antora/site-generator@3.2.1 antora docs/antora-playbook.yml` and open `docs/build/site/index.html`. - ArcadeDB itself (SQL, openCypher, schema, server): [docs.arcadedb.com](https://docs.arcadedb.com). It is not repeated here. ## Examples -[`examples/`](https://github.com/ArcadeData/arcadedb-python/tree/main/examples) holds 30 runnable scripts, from a first document store to vector search, time series, graph algorithms and server mode. CI runs them ([`test-python-examples.yml`](https://github.com/ArcadeData/arcadedb-python/blob/main/.github/workflows/test-python-examples.yml)). +[`examples/`](https://github.com/ArcadeData/arcadedb-python/tree/main/examples) holds 30 runnable scripts, from a first document store to vector search, time series, graph algorithms and server mode. CI runs them ([`test-python-examples.yml`](https://github.com/ArcadeData/arcadedb-python/blob/main/.github/workflows/test-python-examples.yml), and the `examples` job of [`test-native.yml`](https://github.com/ArcadeData/arcadedb-python/blob/main/.github/workflows/test-native.yml) for the native package). ## Build and test from source @@ -110,17 +110,17 @@ Releases from this repository start with 26.11.1. The 26.10.x wheels were built ./scripts/build.sh # an arcadedb-jvm wheel for your platform in dist/ ``` -The `arcadedb` (native) wheel is built from `native/` with GraalVM Community Edition; [`CLAUDE.md`](https://github.com/ArcadeData/arcadedb-python/blob/main/CLAUDE.md) has the commands. A native image build takes about 5 to 30 minutes per platform (macOS is the slowest). +The `arcadedb` (native) wheel is built from `native/` with GraalVM Community Edition; [`CONTRIBUTING.md`](https://github.com/ArcadeData/arcadedb-python/blob/main/CONTRIBUTING.md) has the commands. A native image build takes about 5 to 30 minutes per platform (macOS is the slowest). The JVM build runs in Docker on Linux. macOS and Windows need a JDK 25 or later. The jars come from the `arcadedata/arcadedb` Docker image: `latest` by default, or the tag in `ARCADEDB_TAG` (for example `ARCADEDB_TAG=26.10.2 ./scripts/build.sh`). The package version is read from the engine jar in that image, so an `A.B.C-SNAPSHOT` image builds `A.B.C.devN`. The native build does not use the image. It resolves `arcadedb-engine` and `arcadedb-server` from Maven Central at `arcadedb.version` in `native/pom.xml` (release versions only, because a snapshot is not on Maven Central) and compiles them with GraalVM Community Edition. Neither build compiles the engine from source: the engine code is upstream's, unmodified. -Tests run against the built wheel, not the source tree: install the wheel with its `test` extra into a virtual environment, then run `pytest`. [`CLAUDE.md`](https://github.com/ArcadeData/arcadedb-python/blob/main/CLAUDE.md) has the exact commands, the build options and the lint setup. +Tests run against the built wheel, not the source tree: `uv run --no-project --with "$W[test]" python -m pytest` (`W` is the wheel in `dist/`) makes a throwaway environment from the wheel and its `test` extra. Use `python -m pytest`, not the `pytest` entry point. [`CONTRIBUTING.md`](https://github.com/ArcadeData/arcadedb-python/blob/main/CONTRIBUTING.md) has the exact commands, and [`CLAUDE.md`](https://github.com/ArcadeData/arcadedb-python/blob/main/CLAUDE.md) the build options and the lint setup. ## Contributing -Open an [issue](https://github.com/ArcadeData/arcadedb-python/issues) or a pull request. Run `pre-commit run --all-files` first, and add a regression test with every fix. The rest of the bindings work from `humemai/arcadedb-embedded-python` is ported here one pull request at a time; [#19](https://github.com/ArcadeData/arcadedb-python/issues/19) lists what is done and what is next. +Open an [issue](https://github.com/ArcadeData/arcadedb-python/issues) or a pull request. Run `uvx pre-commit run --all-files` first, and add a regression test with every fix. The rest of the bindings work from `humemai/arcadedb-embedded-python` is ported here one pull request at a time; [#19](https://github.com/ArcadeData/arcadedb-python/issues/19) lists what is done and what is next. ## Where this code comes from diff --git a/docs/modules/ROOT/pages/development/ci.adoc b/docs/modules/ROOT/pages/development/ci.adoc index 39dbcacb..1772e277 100644 --- a/docs/modules/ROOT/pages/development/ci.adoc +++ b/docs/modules/ROOT/pages/development/ci.adoc @@ -16,7 +16,7 @@ Each package has one test workflow and one release workflow: | `release-native.yml` |=== -The release workflows call the test workflows (`workflow_call`) and then publish; see xref:development/release.adoc[Release]. `release-notes.yml` runs after both of them have succeeded for a tag and creates the GitHub Release (and closes the milestone of a final release). Two workflows test the `arcadedb-jvm` package: `test-jvm.yml` builds a wheel and runs the unit tests, and `test-python-examples.yml` runs the scripts in `examples/`. Both build the wheel once for each platform, on a native runner of that platform (`Build arcadedb-jvm wheel ()` and `Build wheel for examples ()`), and install that one file into each Python they test. `test-native.yml` builds and tests the `arcadedb` (native) package; see <<_native_build,Native build>>. +The release workflows call the test workflows (`workflow_call`) and then publish; see xref:development/release.adoc[Release]. `release-notes.yml` runs after both of them have succeeded for a tag and creates the GitHub Release (and closes the milestone of a final release). Two workflows test the `arcadedb-jvm` package: `test-jvm.yml` builds a wheel and runs the unit tests, and `test-python-examples.yml` runs the scripts in `examples/`. Both build the wheel once for each platform, on a native runner of that platform (`Build arcadedb-jvm wheel ()` and `Build wheel for examples ()`), and install that one file into each Python they test. `test-native.yml` builds and tests the `arcadedb` (native) package; see <>. The workflow `docs.yml` builds these docs (the Antora component in `docs/`) as a site of its own with `docs/antora-playbook.yml`, on every pull request and push to `main` that changes `docs/`, and attaches the site as the `docs-site` artifact. Download it from the run to read a page as it will render. The test workflows ignore `docs/**` and `*.adoc`, so for a docs-only pull request this job is the only check. A warning fails it: a broken `xref:`, a misspelled `+{attribute}+`, an AsciiDoc syntax problem. Pages link into the main ArcadeDB manual as `+{arcadedb-docs-url}/+`, not with an `arcadedb::` xref, which the component on its own cannot resolve. To build the site locally, run `npx -p @antora/cli@3.2.1 -p @antora/site-generator@3.2.1 antora docs/antora-playbook.yml` from the repository root and open `docs/build/site/index.html`. @@ -65,7 +65,7 @@ The two packages get their engine from different places, so between releases the * `test-jvm.yml` and `test-python-examples.yml` take the engine jars from the `arcadedata/arcadedb` image tag, `latest` by default. `latest` is upstream's newest snapshot and moves nightly, which is why both workflows also run every night. * `test-native.yml` links `DEV_ENGINE`, the newest release on Maven Central, unless the tag is a final or post version (then it links the release the tag is named after) or the `engine` input of a manual run says otherwise. -A manual run of `test-native.yml` with a `devN` version may set `engine` to `A.B.C-SNAPSHOT`: `native/pom.xml` then resolves the engine jars from the Sonatype snapshot repository (snapshots on, releases off, so a release version still comes from Maven Central alone). The notices comparison of `build_wheel.py` is skipped for such a run, because `THIRD_PARTY_NOTICES.md` names a release and its jars. The plan job refuses a SNAPSHOT engine for a final or post version and for a release run, so nothing that is published depends on one. The `legs` input of a manual run chooses all four platforms (`full`, the default) or the one platform and one Python of a pull request (`pull-request`). +A manual run of `test-native.yml` with a `devN` version may set `engine` to `A.B.C-SNAPSHOT`: `native/pom.xml` then resolves the engine jars from the Sonatype snapshot repository (snapshots on, releases off, so a release version still comes from Maven Central alone). The notices comparison of `build_wheel.py` is skipped for such a run, because `THIRD_PARTY_NOTICES.md` names a release and its jars. The plan job refuses a SNAPSHOT engine for a final or post version. A release run links a snapshot only for a `.devN` tag, and only the one pinned in `scripts/dev_engine_pin.json` (see xref:development/release.adoc[Release]); every other snapshot is refused, so a final or post release never depends on one. The `legs` input of a manual run chooses all four platforms (`full`, the default) or the one platform and one Python of a pull request (`pull-request`). The dev-engine bump after each release, described in xref:development/release.adoc[Release], moves `DEV_ENGINE` to the new release and closes the gap until upstream publishes the next one. @@ -74,6 +74,7 @@ The dev-engine bump after each release, described in xref:development/release.ad * A pull request that changes only documentation (`docs/**`, `*.adoc`, `CLAUDE.md`, `examples/README.md`, `LICENSE*`) runs nothing. * A draft pull request skips the platform jobs. Marking it ready for review starts them. * A new push to a pull request cancels the run it replaces. A run on `main` is never cancelled. +* A JVM crash in an example leaves a HotSpot `hs_err_pid.log`. `test-python-examples.yml` uploads these files as the artifacts `jvm-crash-logdir--py` and `jvm-crash-cwd--py` (issue 352); read them before you re-run a red example leg. == Repository automation @@ -89,6 +90,7 @@ These workflows follow ArcadeData/arcadedb and need no secret. Issues and pull r | `dependabot-auto-merge.yml` | Every hour, squash-merges Dependabot pull requests that are patch or minor updates, mergeable, and have a non-empty list of checks that all succeeded. Majors and grouped updates stay open. Logic in `.github/scripts/dependabot_merge.py`. | `dependency-review.yml` | Fails a pull request that adds a dependency with a known vulnerability (moderate or worse) or a GPL, AGPL or SSPL licence. Policy in `.github/dependency-review-config.yml`. | `gitignore-hygiene.yml` | Fails when a `.gitignore` pattern matches a tracked file. +| `assign-maintainer.yml` | Assigns every new issue and pull request to the maintainer when it is opened. Nothing from a pull request head is checked out or run. |=== To close a release, close its milestone and open the next one with its due date: `assign-milestone.yml` picks the open milestone that is due first. @@ -97,10 +99,11 @@ To close a release, close its milestone and open the next one with its due date: The workflow `test-native.yml` builds the native image of the `arcadedb` package and runs the test suite against the installed wheel. It does not follow the table above. It never publishes; `release-native.yml` calls it on a version tag. -* A pull request runs it only when native paths change: `native/**`, `src/arcadedb/**`, `tests/**`, `scripts/select_native_matrix.py` or `test-native.yml` itself. Changes under `tests/` and `src/arcadedb/` are included because both backends share those files, so a shared test or source can break native parity without touching a native file; docs outside those paths do not run it. It then runs on `linux-x86_64` with Python 3.12 only. -* A push to `main`, the nightly run, a manual run (dispatch) run all four platforms: `linux-x86_64`, `linux-aarch64`, `macos-arm64` and `windows-x64` (Python 3.12 and 3.14). A release (`release-native.yml` calls the workflow with `release: true`) runs the same four platforms on all four Python versions, 16 legs. +* A pull request runs it only when native paths change: `+native/**+`, `+src/arcadedb/**+`, `+tests/**+`, `+examples/**+` (not `examples/README.md`), `pyproject.toml`, `setup.py`, `scripts/select_native_matrix.py`, `scripts/smoke_wheel.py`, `scripts/run_examples.sh`, `scripts/native_examples_skip.txt` or `test-native.yml` itself. Changes under `tests/` and `src/arcadedb/` are included because both backends share those files, so a shared test or source can break native parity without touching a native file; docs outside those paths do not run it. It then runs on `linux-x86_64` with Python 3.12 only. +* A push to `main` (only when a native path changed, same filter as a pull request), the nightly run (skipped when nothing changed in the last 25 hours, except on Mondays) and a manual run (dispatch) run all four platforms: `linux-x86_64`, `linux-aarch64`, `macos-arm64` and `windows-x64` (Python 3.12 and 3.14). A release (`release-native.yml` calls the workflow with `release: true`) runs the same four platforms on all four Python versions, 16 legs. * The `pyver` job installs the already built `linux-x86_64` wheel (no rebuild) into the Pythons that the full parity runs of the event leave out (3.11, 3.13 and 3.14 on a pull request; 3.11 and 3.13 on a push to `main`, the nightly run and a dispatch; none on a release) into 3.15 and into the free-threaded 3.14t, and runs 13 test files there (14 where polars installs; 3.15 has no jpype1 wheel, so it leaves out the two files that need jpype1), about 20 seconds of tests per Python. It is informational: `release-native.yml` reads only the `native-passed-*` markers and the `native-wheel-*` wheels, so a red `pyver` leg never blocks a release. The 3.14t leg has no polars (no free-threaded wheel exists), so it skips `test_dataframe_ingest.py`. `scripts/select_native_matrix.py` decides its legs. * Every test file runs in its own process (`native/wheel/parity_wheel.py`). A process that dies without writing its result is a `CRASH` row in `parity-wheel.tsv` with its exit code, for example `CRASH(rc=0xC0000005,3221225477)`, and it fails the leg. On the Windows legs only (the crashes of issue 295 happen there), the harness runs such a file once more in a fresh process (`--retry-crash`). A retry that produces a result is a `FLAKY-CRASH` row: the gate lets the leg pass, the job summary lists the file with the first exit code, and on a release or a dispatch the `crash-report` job adds one comment to issue 295 (run URL, Python, wheel version, exit codes). A second crash, a failing test, a timeout and every crash on Linux or macOS stay failures. A process that wrote its result and then died in shutdown is a `DIRTY-EXIT` record (exit code, file, attempt) in the metadata, the job summary and the issue comment: it is not retried and does not fail the leg. Each Windows test process also writes a faulthandler trace, `parity-wheel-xml/.attemptN.fault.txt`, into the parity artifact of the leg. Only a retry hides a crash, so treat a `FLAKY-CRASH` as an open bug: read the trace in the artifact before a release is announced. +* The `examples` job runs every script of `examples/` against the installed wheel of each platform (Python 3.12 and 3.14; a pull request: `linux-x86_64`, Python 3.12) through `scripts/run_examples.sh`, the runner that `test-python-examples.yml` uses too. `scripts/native_examples_skip.txt` lists, with a reason, the examples that need the JVM package. `Examples summary` is the one check for all its legs. These legs do not decide what a release publishes. * A native image build takes about 5 to 30 minutes for each platform (macOS is the slowest). The organisation shares one runner limit with upstream ArcadeDB, so do not add runs lightly. Change the paths or the matrix of `test-native.yml` only for a reason that you can write down. == Change the selection diff --git a/docs/modules/ROOT/pages/development/release.adoc b/docs/modules/ROOT/pages/development/release.adoc index 36139ca2..08e3faa5 100644 --- a/docs/modules/ROOT/pages/development/release.adoc +++ b/docs/modules/ROOT/pages/development/release.adoc @@ -132,7 +132,7 @@ Each step is a script, CI, or the owner's decision. The owner's tag push is the | Step | What | Who | 1 -| Preflight. Checks that the upstream GitHub release exists (not draft), that Maven Central has `arcadedb-engine` and `arcadedb-server`, that Docker Hub has the image for `linux/amd64` and `linux/arm64`, that `scripts/check_version.py` accepts `` for both packages against the git tags and PyPI, that the latest conclusive runs of `Test native`, `Test JVM` and `Test Python Examples` on `main` are green on every job, that the release workflows derive `` and the engine from the tag (the real step code of the workflows runs with `TAG=`), and that the native readiness variable is set for a final release. It only reads, prints `PASS`, `FAIL` or `SKIP` with the evidence, fails closed on network errors, and exits with the number of FAILs. +| Preflight. Checks that the upstream GitHub release exists (not draft), that Maven Central has `arcadedb-engine` and `arcadedb-server`, that Docker Hub has the image for `linux/amd64` and `linux/arm64`, that `scripts/check_version.py` accepts `` for both packages against the git tags and PyPI, that the latest conclusive runs of `Test native (arcadedb)`, `Test JVM (arcadedb-jvm)` and `Test Python Examples` on `main` are green on every job, that the release workflows derive `` and the engine from the tag (the real step code of the workflows runs with `TAG=`), and that the native readiness variable is set for a final release. It only reads, prints `PASS`, `FAIL` or `SKIP` with the evidence, fails closed on network errors, and exits with the number of FAILs. | script: `scripts/release_preflight.py` | 2 @@ -191,7 +191,7 @@ A release does not end on PyPI: it ends with a GitHub Release, as in the upstrea * The notes come from `scripts/release_notes.py`. The milestone is named after the base `A.B.C` (`26.11.1` for `26.11.1`, `26.11.1.dev2` and `26.11.1.post1`). A pull request is listed when it is merged, has that milestone and its merge commit is in the tag, grouped by its first matching label: `bug` (Bug fixes), `performance` (Performance), `enhancement` (Features), `documentation` (Documentation), everything else under Other. Each line is the title and `(#N)`. The notes also have the install lines, the engine version and the PyPI pages of both packages. * A final release lists the whole milestone. A `devN` lists only what is not in the previous dev tag of the same base. A pull request without the milestone is not in the notes of a final or dev release, so check the milestone before the tag. * A `postN` does not use a milestone, because the final release closed it. Its notes list every pull request merged between the previous tag of the same base (the final or `post(N-1)`) and the tag, found from the commits of that range, whatever milestone the pull request has or none. -* After a final release (not `devN`, not `postN`) the milestone `A.B.C` is closed, after what is still open in it moves to the next milestone. The next milestone is the open one with the earliest due date after this one. When there is none, the workflow creates one named after the next upstream engine version (the base of the upstream `main` version, the same command as in <<_choosing_the_next_dev_version,Choosing the next dev version>>) and due 30 days after the closed one. When that version cannot be read, or is the same version, nothing is moved or closed and the run fails with a message: create the next milestone with a due date and re-run the workflow. Every move is logged. The milestone auto-assignment of merged pull requests, where it is set up, then picks the next milestone, because the closed one is no longer open. +* After a final release (not `devN`, not `postN`) the milestone `A.B.C` is closed, after what is still open in it moves to the next milestone. The next milestone is the open one with the earliest due date after this one. When there is none, the workflow creates one named after the next upstream engine version (the base of the upstream `main` version, the same command as in <>) and due 30 days after the closed one. When that version cannot be read, or is the same version, nothing is moved or closed and the run fails with a message: create the next milestone with a due date and re-run the workflow. Every move is logged. The milestone auto-assignment of merged pull requests, where it is set up, then picks the next milestone, because the closed one is no longer open. * A run for a release that already exists edits its notes and flag, so a re-run is safe. Preview the notes before the tag, and read them again after the release: @@ -219,7 +219,7 @@ curl -fsS https://raw.githubusercontent.com/ArcadeData/arcadedb/main/pom.xml | s `native/wheel/THIRD_PARTY_NOTICES.md` names the engine and is generated from the build log of the native image, so `build_wheel.py` fails on any difference. `bump_engine.py` reports it as `TODO` and does not edit it, and its `--check` stays at exit 1 until the file names the new engine. To regenerate: . First refresh the kept copy of the engine `NOTICE`: `git -C show :NOTICE > native/wheel/ARCADEDB_ENGINE_NOTICE`. The generated notices embed it, so a stale copy costs one more CI round (the 26.11.1 bump needed two because its `NOTICE` gained an entry). -. Push the bump PR. The first `Test native` run stops at the notices drift check on purpose, and its `native-build-record-linux-x86_64` artifact holds `THIRD_PARTY_NOTICES.generated.md`, the output of `gen_notices.py` for the new engine (the text is the same on every platform). +. Push the bump PR. The first `Test native (arcadedb)` run stops at the notices drift check on purpose, and its `native-build-record-linux-x86_64` artifact holds `THIRD_PARTY_NOTICES.generated.md`, the output of `gen_notices.py` for the new engine (the text is the same on every platform). . Download the artifact with `gh run download -R ArcadeData/arcadedb-python -n native-build-record-linux-x86_64 -D `, copy `THIRD_PARTY_NOTICES.generated.md` over `native/wheel/THIRD_PARTY_NOTICES.md`, and commit it to the bump branch. . When the next run still fails on `NOTICE`, run the `gen_notices.py ... --notice` command that the failure message prints. . Check with `uv run --no-project python scripts/bump_engine.py --engine --dev-version --check`, which must exit 0, and wait for the CI of the PR to turn green. diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index 0144526c..8352ac64 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -181,7 +181,7 @@ The measurement is a laptop A/B on the `arcadedb` (native) wheel, one fresh proc == Graph batches next to deletes (fixed in ArcadeDB 26.11.1) -NOTE: *Fixed in ArcadeDB 26.11.1.* Before 26.11.1, a graph batch did not reuse the space of deleted edges (https://github.com/ArcadeData/arcadedb/issues/10746[ArcadeData/arcadedb#10746]). The engine wrote the edge records of each `graph_batch()` flush onto new pages and never looked at the free space that deletes left on older pages, so the edge bucket gained one 64 KiB page per flush while the live data stayed flat. From 26.11.1 (https://github.com/ArcadeData/arcadedb/pull/10836[ArcadeData/arcadedb#10836]) each edge goes where an ordinary insert would put it while an existing page has room, and only the rest of a flush goes onto new pages, so a large import stays as fast as before. The 26.11.1 packages ship this fix. The `26.11.1.devN` pre-releases embed engine 26.10.2 and still show the growth described below. +NOTE: *Fixed in ArcadeDB 26.11.1.* Before 26.11.1, a graph batch did not reuse the space of deleted edges (https://github.com/ArcadeData/arcadedb/issues/10746[ArcadeData/arcadedb#10746]). The engine wrote the edge records of each `graph_batch()` flush onto new pages and never looked at the free space that deletes left on older pages, so the edge bucket gained one 64 KiB page per flush while the live data stayed flat. From 26.11.1 (https://github.com/ArcadeData/arcadedb/pull/10836[ArcadeData/arcadedb#10836]) each edge goes where an ordinary insert would put it while an existing page has room, and only the rest of a flush goes onto new pages, so a large import stays as fast as before. The 26.11.1 packages ship this fix. On engine 26.10.2, a one-off bulk load is not affected. The workloads that are affected call `graph_batch()` again and again on a type that also sees deletes, for example a sliding window that adds a small batch and deletes the oldest one. diff --git a/docs/modules/ROOT/pages/how-to/dataframes-and-columns.adoc b/docs/modules/ROOT/pages/how-to/dataframes-and-columns.adoc index eb0796a6..43ba7f45 100644 --- a/docs/modules/ROOT/pages/how-to/dataframes-and-columns.adoc +++ b/docs/modules/ROOT/pages/how-to/dataframes-and-columns.adoc @@ -271,7 +271,7 @@ refused: True [[reading]] == Reading columns -The four readers take the whole result, so they hold every row in memory. For a result that is larger than memory read the rows one by one (xref:how-to/databases-and-queries.adoc[Databases and queries]) or write it with `export_to_csv`. A result set is read once (xref:how-to/databases-and-queries.adoc#a-result-set-is-read-once[A result set is read once]), so call one reader for each query. +The four readers take the whole result, so they hold every row in memory. For a result that is larger than memory read the rows one by one (xref:how-to/databases-and-queries.adoc[Databases and queries]) or write it with `export_to_csv`. A result set is read once (xref:how-to/databases-and-queries.adoc#result-set-single-pass[A result set is read once]), so call one reader for each query. The next block stores one record with every value and one record with none, then reads them with each reader. The records are the same for all of them, and the readers differ in what they do with the gap: diff --git a/docs/modules/ROOT/pages/how-to/vectors.adoc b/docs/modules/ROOT/pages/how-to/vectors.adoc index 2eef4df3..741bf839 100644 --- a/docs/modules/ROOT/pages/how-to/vectors.adoc +++ b/docs/modules/ROOT/pages/how-to/vectors.adoc @@ -68,7 +68,7 @@ Name the setting in the statement or in the call when you compare results or nee == Insert vectors -A vector is a property value like any other. Bind it as a parameter of `INSERT`. A NumPy `float32` array crosses into the engine as one buffer. A list of Python floats also works, and a list that is the only parameter of a statement is read as the whole parameter list, so give a vector as a list only next to other parameters or by name. The details are in xref:how-to/databases-and-queries.adoc#_a_query_vector[A query vector]. +A vector is a property value like any other. Bind it as a parameter of `INSERT`. A NumPy `float32` array crosses into the engine as one buffer. A list of Python floats also works, and a list that is the only parameter of a statement is read as the whole parameter list, so give a vector as a list only next to other parameters or by name. The details are in xref:how-to/databases-and-queries.adoc#a-query-vector[A query vector]. [source,python] ---- @@ -214,7 +214,7 @@ for statement in (inside, after): ['python'] ---- -Three mistakes raise `ArcadeDBError`: a binding variable that is not a node variable of the `MATCH` pattern, `OPTIONAL MATCH` before `SEARCH`, and an index that does not exist. The last one surfaces while you read the rows, as any error of a lazy result does (see xref:how-to/databases-and-queries.adoc#_a_result_set_is_read_once[A result set is read once]). +Three mistakes raise `ArcadeDBError`: a binding variable that is not a node variable of the `MATCH` pattern, `OPTIONAL MATCH` before `SEARCH`, and an index that does not exist. The last one surfaces while you read the rows, as any error of a lazy result does (see xref:how-to/databases-and-queries.adoc#result-set-single-pass[A result set is read once]). ==== CREATE VECTOR INDEX @@ -455,13 +455,13 @@ INT8 True These rules hold for the quantization arguments of the method: -* The `pq_*` and `fused_pq` arguments need `quantization="PRODUCT"`. `rerank_oversample` needs `"PRODUCT"` or, on ArcadeDB 26.11.1 and later, `"BINARY"`: a `BINARY` search scores the one-bit codes and re-ranks `k * rerank_oversample` candidates on the stored vectors (default 10), or the whole `efSearch` beam when that is larger (https://github.com/ArcadeData/arcadedb/issues/10968[ArcadeData/arcadedb#10968]). The call raises `ValueError` otherwise. The same keys are `METADATA` keys in a statement (`pqSubspaces`, `pqClusters`, `pqCenterGlobally`, `pqTrainingLimit`, `rerankOversample`, `fusedPQ`). With `PRODUCT`, every graph rebuild retrains the codebook and re-encodes every vector (https://github.com/ArcadeData/arcadedb/issues/10847[ArcadeData/arcadedb#10847]), so measure recall and rebuild time on your own data before you choose it. +* The `pq_*` and `fused_pq` arguments need `quantization="PRODUCT"`. `rerank_oversample` needs `"PRODUCT"` or, on ArcadeDB 26.11.1 and later, `"BINARY"`: a `BINARY` search scores the one-bit codes and re-ranks `k * rerank_oversample` candidates on the stored vectors (default 10), or the whole `efSearch` beam when that is larger (https://github.com/ArcadeData/arcadedb/issues/10968[ArcadeData/arcadedb#10968]). The call raises `ValueError` otherwise. The same keys are `METADATA` keys in a statement (`pqSubspaces`, `pqClusters`, `pqCenterGlobally`, `pqTrainingLimit`, `rerankOversample`, `fusedPQ`). With `PRODUCT`, from ArcadeDB 26.11.1 a graph rebuild keeps the trained codebook and encodes only the vectors added since (https://github.com/ArcadeData/arcadedb/issues/10847[ArcadeData/arcadedb#10847]). The engine trains a new codebook and re-encodes every vector when the index has grown past `arcadedb.vectorIndex.pqRetrainGrowthFactor` times its size at training (default 2.0), or when the new vectors encode worse than the training ones (`arcadedb.vectorIndex.pqRetrainErrorRatio`, default 1.25). Setting the growth factor to 0 restores a retrain on every rebuild, which is what engines before 26.11.1 do. Measure recall and rebuild time on your own data before you choose `PRODUCT`. * `encoding="INT8"` is for a `BINARY` property that already holds INT8 bytes. Set `quantization` to `"NONE"` with it. The call raises `ValueError` when both are `INT8`, because the vector would be quantized twice. * A bucket that holds fewer vectors than `pq_clusters` is too small to train a codebook. The index still builds: the engine skips PQ for that bucket and searches it exactly, so `find_nearest_approximate` returns the exact neighbours with exact scores from it. A bucket with `pq_clusters` vectors or more is trained and answers approximately. == Rebuild the graph -Inserts, updates and deletes collect as pending changes. When enough of them have collected, the engine rebuilds the graph in the background, and searches keep running. How many is enough comes from the database setting `arcadedb.vectorIndex.rebuildGraphRatio` and from `mutations_before_rebuild`; see xref:how-to/databases-and-queries.adoc#_database_settings[Database settings] for how to change the ratio. +Inserts, updates and deletes collect as pending changes. When enough of them have collected, the engine rebuilds the graph in the background, and searches keep running. How many is enough comes from the database setting `arcadedb.vectorIndex.rebuildGraphRatio` and from `mutations_before_rebuild`; see xref:how-to/databases-and-queries.adoc#database-settings[Database settings] for how to change the ratio. The background rebuild starts on the first search after the threshold is reached, so an index that only takes writes rebuilds when it is next searched. diff --git a/docs/modules/ROOT/pages/installation.adoc b/docs/modules/ROOT/pages/installation.adoc index 3e64546a..f19ee806 100644 --- a/docs/modules/ROOT/pages/installation.adoc +++ b/docs/modules/ROOT/pages/installation.adoc @@ -62,7 +62,7 @@ Every wheel is platform-specific and built on a native runner of its platform, s | `win_amd64` |=== -Each native wheel is about 38 MB. These platforms are not built, and the installer finds no wheel for them: +Each native wheel is about 66 MB. These platforms are not built, and the installer finds no wheel for them: * *Intel Macs* (macOS x86_64). Apple has sold only Apple Silicon Macs since 2023, and macOS 26 is the last release for Intel Macs. On an Intel Mac, run the Linux wheel in Docker or in a Linux virtual machine. * *Alpine Linux and other musl-based distributions.* The Linux wheels are for glibc, so the installer reports that no wheel matches the platform. Use a glibc image, for example `python:3.12-slim`. diff --git a/examples/README.md b/examples/README.md index e3f0c06c..0133c5cd 100644 --- a/examples/README.md +++ b/examples/README.md @@ -11,10 +11,13 @@ This directory contains hands-on examples demonstrating ArcadeDB Python bindings cd examples # Then run the basic document store example -python 01_simple_document_store.py +uv run --with arcadedb python 01_simple_document_store.py +# or install the package first: uv add arcadedb, then uv run python 01_simple_document_store.py ``` -The dataset downloader is [download_data.py](./download_data.py): its docstring lists the datasets, and `python download_data.py --help` the options. +The dataset downloader is [download_data.py](./download_data.py): its docstring lists the datasets, and `uv run --no-project python download_data.py --help` the options. + +Examples 02, 03, 14, 15 and 25 use Java objects and run on `arcadedb-jvm` only (`scripts/native_examples_skip.txt`). ## Available Examples @@ -75,7 +78,7 @@ High-performance CSV import for document data: - Batch processing for optimal performance - Index creation strategies -**Note:** Download the MovieLens dataset first with `python download_data.py movielens-`. +**Note:** Download the MovieLens dataset first with `uv run --no-project python download_data.py movielens-`. **Learn:** ETL patterns, bulk import, schema management, performance tuning @@ -400,7 +403,7 @@ DataFrames and Arrow tables loaded straight into a type, and a result back out a column refused before a row is written - `to_polars()` on the result - synthetic data; flags: `--rows` (default 100,000) and `--db-path` -- requires `uv add 'arcadedb-jvm[pandas,polars]'` +- requires `uv add 'arcadedb[pandas,polars]'` (or `'arcadedb-jvm[pandas,polars]'`) **Learn:** How a DataFrame becomes documents or vertices without a row loop, and what each Arrow type is stored as diff --git a/tests/README.md b/tests/README.md index f1681e43..2d544e90 100644 --- a/tests/README.md +++ b/tests/README.md @@ -8,7 +8,7 @@ Test suite for the ArcadeDB Python bindings. It runs against the built wheel, so ```bash # The wheel and its test extra go into a throwaway environment on every run, so there is no -# environment to create. --reinstall-package, as a rebuild keeps the version (26.11.1.dev0) +# environment to create. --reinstall-package, as a rebuild keeps the version (for example 26.12.1.dev0) # Run all tests uv run --no-project --reinstall-package arcadedb-jvm --with "$(ls dist/*.whl | head -1)[test]" python -m pytest @@ -34,7 +34,7 @@ The server tests do not skip when the server stack is absent: the wheel always s ## The native build -The same suite runs against the native `arcadedb` wheel (`native/wheel/parity_wheel.py`). A test is reported as a skip on the native build in two explicit ways, never by what its failure says: +The same suite runs against the native `arcadedb` wheel (`native/wheel/parity_wheel.py`). A test file about the JVM itself, the JPype wheel or the JVM wheel's build tooling is not collected on the native build: it is a row of `NATIVE_NOT_APPLICABLE` in `tests/conftest.py` (on Windows, `WINDOWS_NOT_APPLICABLE`). A test is reported as a skip on the native build in two explicit ways, never by what its failure says: - `NotOnNativeYet`: the native code raised it because the feature needs a Java object API the library does not expose. The report says `NOTYET`, and a final release allows none. A call that is JVM-only by design (`get_java_database`, `get_java_document`, a Java object a `Schema` method returns) raises it with a message that names the `arcadedb-jvm` build; a shared test that reaches one is `NA-JVM`, or reads the value through a statement or the Python API instead. - A row of `NATIVE_NA_JVM_TESTS` in `tests/conftest.py`, keyed `::[::]`: the test reaches into the JVM on purpose (`jpype.JClass`, a Java array, `db._java_db`, counted bridge crossings). The report says `NA-JVM`. Add a row only for such a test, and give the reason. When the behaviour is shared, write a native test beside it (`test_native_*.py`, collected on the native build only).