Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/release-jvm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ jobs:
with:
arcadedb-tag: ${{ needs.validate-version.outputs.arcadedb-tag }}
build-version: ${{ needs.validate-version.outputs.version }}
release: true

test-examples:
name: Example tests
Expand Down
8 changes: 7 additions & 1 deletion .github/workflows/test-jvm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,11 @@ on:
description: "Override package version (PEP 440) for build.sh"
required: false
type: string
release:
description: "A release run (release-jvm.yml): every platform x Python leg (scripts/select_matrix.py, event `release`)"
required: false
type: boolean
default: false

# Allow manual trigger
workflow_dispatch:
Expand Down Expand Up @@ -244,7 +249,8 @@ jobs:
id: matrix
shell: bash
env:
EVENT_NAME: ${{ github.event_name }}
# In a workflow_call github.event_name is the caller's event (a tag push), so a release says so through its input
EVENT_NAME: ${{ inputs.release && 'release' || github.event_name }}
run: |
matrix="$(python3 scripts/select_matrix.py jvm --event "$EVENT_NAME")"
wheel_matrix="$(python3 scripts/select_matrix.py jvm --event "$EVENT_NAME" --wheels)"
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Each package has one test and one release workflow: `test-jvm.yml` and `release-

Repository automation (no secret, modelled on ArcadeData/arcadedb): `classify-issue.yml` (rule-based labels), `label-issue-in-progress.yml`, `assign-milestone.yml` (open milestone due first), `dependabot-auto-merge.yml` (patch and minor, all checks green; `.github/scripts/dependabot_merge.py`), `dependency-review.yml`, `gitignore-hygiene.yml`; docs page `development/ci.adoc`. A pull request states its issue with `Closes #N` (the in-progress label and the milestone follow from it).

`.github/workflows/test-jvm.yml` builds ONE wheel per platform (linux/amd64, linux/arm64, darwin/arm64, windows/amd64), tagged `py3-none-<platform>` by `setup.py` (`bdist_wheel.get_tag`; `scripts/verify_wheel_tag.py` checks it), and tests that wheel on Python 3.11–3.14; also runs nightly against `arcadedata/arcadedb:latest`. `requires-python` is `>=3.11,<3.15` because the tag no longer keeps pip off 3.15 and JPype has no 3.15 wheel; drop the cap with it. The org shares a runner cap with upstream, so the unit tests run 10 legs on every event (linux/amd64 on all four Pythons, the other three platforms on 3.11 and 3.14, the shape of test-native.yml), and the examples run all four platforms on 3.14 on a pull request (4 legs, 28 checks in all) and 16 legs on push to main, the nightly run, dispatch and the release (40 checks). `scripts/select_matrix.py` holds both lists (`PULL_REQUEST_LEGS`, keyed `jvm` and `examples`), the first job of each workflow runs it, and `tests/test_select_matrix.py` pins the full matrix; docs page `development/ci.adoc`. Keep the job names `Test arcadedb (<platform>, Python <version>)` unchanged. Both workflows accept an `arcadedb-tag` input to pin the image. `test-python-examples.yml` runs `examples/`. A dependency-floor audit locks the dependencies and extras on a throwaway uv project with `--resolution lowest-direct` for each Python classifier and runs `uvx pip-audit` on the export, so raise lower bounds when a floor has an advisory. No tool is installed with pip anywhere in CI or the scripts: one-off tools run through `uvx tool==X`, the wheel is built with `uvx --from "build[uv]==1.6.1" pyproject-build --installer uv`, and the uv install URL is pinned (`0.12.0`).
`.github/workflows/test-jvm.yml` builds ONE wheel per platform (linux/amd64, linux/arm64, darwin/arm64, windows/amd64), tagged `py3-none-<platform>` by `setup.py` (`bdist_wheel.get_tag`; `scripts/verify_wheel_tag.py` checks it), and tests that wheel on Python 3.11–3.14; also runs nightly against `arcadedata/arcadedb:latest`. `requires-python` is `>=3.11,<3.15` because the tag no longer keeps pip off 3.15 and JPype has no 3.15 wheel; drop the cap with it. The org shares a runner cap with upstream, so `scripts/select_matrix.py` picks a smaller matrix for pull requests, pushes and the nightly run and the full one for a release or a dispatch; `tests/test_select_matrix.py` pins it and the docs page `development/ci.adoc` lists the legs. Keep the job names `Test arcadedb (<platform>, Python <version>)` unchanged. Both workflows accept an `arcadedb-tag` input to pin the image. `test-python-examples.yml` runs `examples/`. A dependency-floor audit locks the dependencies and extras on a throwaway uv project with `--resolution lowest-direct` for each Python classifier and runs `uvx pip-audit` on the export, so raise lower bounds when a floor has an advisory. No tool is installed with pip anywhere in CI or the scripts: one-off tools run through `uvx tool==X`, the wheel is built with `uvx --from "build[uv]==1.6.1" pyproject-build --installer uv`, and the uv install URL is pinned (`0.12.0`).

`.github/workflows/release-jvm.yml` publishes `arcadedb-jvm` on a bare-version tag (`26.11.1.dev0`, `26.11.1`, `26.11.1.post1`; no `v`, never `aN`/`bN`/`rcN`). Its first job runs `scripts/check_version.py` (the devN/postN counting rule, read-only, fails closed); `scripts/release_preflight.py <VERSION>` checks everything before the tag (read-only); `scripts/after_tag_verify.sh` (`PACKAGE=native` for `arcadedb`) checks the finished release; `scripts/bump_engine.py` moves the dev-engine pins after a release. The monthly steps are the "Release day" section of `development/release.adoc` and the issue template `.github/ISSUE_TEMPLATE/release.md`. `.github/workflows/release-notes.yml` (`workflow_run` on both release workflows, `workflow_dispatch` for a published tag) creates the GitHub Release once both packages are published, with notes from `scripts/release_notes.py` (merged pull requests of the milestone named after the base version for a final and a `devN` release, the commit range since the previous tag for `postN`, grouped by label; `--dry-run` previews), a pre-release for `devN`, and after a final release moves the open items to the next milestone (created from the next upstream engine version when none exists) and closes the milestone. The rule is stated once, on the docs page `development/release.adoc`.

Expand Down
10 changes: 7 additions & 3 deletions docs/modules/ROOT/pages/development/ci.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,16 @@ The script `scripts/select_matrix.py` chooses the combinations. The first job of
| 10 legs: `linux/amd64` on all four Python versions. `linux/arm64`, `darwin/arm64` and `windows/amd64` on Python 3.11 and 3.14.
| All four platforms on Python 3.14.

| Push to `main`, the nightly run, a manual run, a release
| Push to `main`, the nightly run
| The same 10 legs.
| All 16 combinations.

| A manual run (dispatch), a release
| All 16 combinations.
| All 16 combinations.
|===

The unit tests run the same 10 legs on every event, like the native package (all Python versions on one Linux platform, the oldest and the newest elsewhere). They were 16 legs on `main`, the nightly run and a release before the wheel became `py3-none`; the extra legs tested the same file again. A pull request runs 28 checks: 14 test jobs (10 and 4), 8 wheel builds (4 for each workflow), the two `Download ArcadeDB JARs` jobs, the two summary jobs, `Bandit security scan` and `Dependency floor audit`. The other events run 40: the examples run all 16 combinations there, from 4 wheel builds.
The unit tests run 10 legs on a pull request, a push to `main` and the nightly run (all Python versions on one Linux platform, the oldest and the newest elsewhere), and all 16 combinations on a release and a manual run, like the native package: what is published is tested on every platform and Python it claims. `release-jvm.yml` calls the workflow with `release: true`, which the matrix step turns into the event `release`, because in a called workflow `github.event_name` is the caller's tag push. A pull request runs 28 checks: 14 test jobs (10 and 4), 8 wheel builds (4 for each workflow), the two `Download ArcadeDB JARs` jobs, the two summary jobs, `Bandit security scan` and `Dependency floor audit`. A push and the nightly run run 40: the examples run all 16 combinations there, from 4 wheel builds. A manual run and a release run 46, with the 6 extra unit-test legs.

The names of the test jobs do not change: `Test arcadedb (<platform>, Python <version>)`. A check keeps its history when a pull request runs fewer of them.

Expand Down Expand Up @@ -98,4 +102,4 @@ The workflow `test-native.yml` builds the native image of the `arcadedb` package

== Change the selection

Edit `JVM_LEGS` (unit tests, every event) or `PULL_REQUEST_LEGS` (examples on a pull request) in `scripts/select_matrix.py`. The test `tests/test_select_matrix.py` fixes the legs, so a change that shrinks them fails. When you add a Python version, add it to `PYTHONS` in the script and to the classifiers in `pyproject.toml`. A test checks that both lists are the same, that `requires-python` starts at the oldest of them and stops before the minor after the newest. When you drop one, remove it from the same places and update the counts in the test.
Edit `JVM_LEGS` (unit tests on the events in `JVM_REDUCED_EVENTS`: a pull request, a push and the nightly run; every other event runs the full matrix) or `PULL_REQUEST_LEGS` (examples on a pull request) in `scripts/select_matrix.py`. The test `tests/test_select_matrix.py` fixes the legs, so a change that shrinks them fails. When you add a Python version, add it to `PYTHONS` in the script and to the classifiers in `pyproject.toml`. A test checks that both lists are the same, that `requires-python` starts at the oldest of them and stops before the minor after the newest. When you drop one, remove it from the same places and update the counts in the test.
23 changes: 15 additions & 8 deletions scripts/select_matrix.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,19 @@

The wheels support four platforms and four Python versions.

- `jvm` (test-jvm.yml) runs the same 10 legs on every event: a pull request, a push to main, the nightly
schedule, a manual dispatch and the release workflow (which calls the workflow as `workflow_call`). The JVM
wheel is one file for each platform (py3-none-<platform>, built once by the `build` job), so a Python version
is a test leg, not a build. Like the native package (test-native.yml), the Pythons are all four on
- `jvm` (test-jvm.yml) runs 10 legs on a pull request, a push to main and the nightly schedule, and all 16 for
a release (release-jvm.yml calls the workflow with `release: true`, which test-jvm.yml passes as the event
`release`) and a manual dispatch. The JVM wheel is one file for each platform (py3-none-<platform>, built once
by the `build` job), so a Python version is a test leg, not a build. The 10 legs are all four Pythons on
linux/amd64 and the oldest and the newest on the other platforms: a bug that depends on the interpreter shows
on the ends of the range, and one that depends on the platform shows on any version of it. The organisation
shares one runner cap with ArcadeData/arcadedb, so every extra macOS or Windows leg delays upstream too.
shares one runner cap with ArcadeData/arcadedb, so every extra macOS or Windows leg delays upstream too; what
is published, and a run started by hand before a release, gets every combination, like test-native.yml.
- `examples` (test-python-examples.yml) runs the newest Python on every platform on a pull request, and every
combination (16 legs) on every other event.

The selection fails open to the full matrix: an event name this script does not know runs everything, never less
(for `jvm`, whose legs do not depend on the event, that is the same 10 legs).
(for `jvm` too: only a pull request, a push and the nightly schedule get its 10 legs).

usage:
select_matrix.py jvm [--event NAME] # test-jvm.yml: the test legs
Expand Down Expand Up @@ -53,7 +54,7 @@
OLDEST = PYTHONS[0]
NEWEST = PYTHONS[-1]

# The legs of `jvm`, the same for every event: all Pythons on linux/amd64 (the cheap, fast runner and the one most
# The legs of `jvm` on a pull request, a push and the nightly run: all Pythons on linux/amd64 (the cheap, fast runner and the one most
# contributors use), the oldest and the newest on the three other platforms (the shape of test-native.yml). The wheel
# is py3-none-<platform>, one file for each platform, so these are test legs against that file, not builds.
JVM_LEGS = {
Expand All @@ -63,6 +64,10 @@
"windows/amd64": (OLDEST, NEWEST),
}

# The events on which `jvm` runs only JVM_LEGS. Every other event (a release, a manual dispatch, an event this
# script does not know) runs every platform x Python combination, so the selection fails open.
JVM_REDUCED_EVENTS = ("pull_request", "push", "schedule")

# The legs of a pull request, for the workflows whose legs depend on the event.
#
# examples: the newest Python only, on every platform. The examples are slow (torch,
Expand Down Expand Up @@ -91,7 +96,9 @@ def select(workflow: str, event: str) -> dict[str, list[dict[str, str]]]:
f"unknown workflow {workflow!r}; use one of {sorted(WORKFLOWS)}"
)
if workflow == "jvm":
return {"include": _legs(JVM_LEGS)}
if event in JVM_REDUCED_EVENTS:
return {"include": _legs(JVM_LEGS)}
return {"include": _legs({platform: PYTHONS for platform in RUNNERS})}
if event == "pull_request":
return {"include": _legs(PULL_REQUEST_LEGS[workflow])}
return {"include": _legs({platform: PYTHONS for platform in RUNNERS})}
Expand Down
54 changes: 47 additions & 7 deletions tests/test_select_matrix.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,10 @@ def test_every_event_but_pull_request_runs_the_full_examples_matrix(event):
assert _pairs(matrix) == FULL


@pytest.mark.parametrize("event", ["pull_request", *NON_PULL_REQUEST_EVENTS])
def test_jvm_tests_the_one_wheel_of_a_platform_on_the_same_ten_legs_on_every_event(event):
@pytest.mark.parametrize("event", ["pull_request", "push", "schedule"])
def test_jvm_tests_the_one_wheel_of_a_platform_on_ten_legs_on_a_pull_request_a_push_and_the_nightly(
event,
):
# The wheel is py3-none-<platform>: every Python of linux/amd64, the oldest and the newest elsewhere (as test-native.yml)
pairs = _pairs(sm.select("jvm", event))
assert {v for p, v in pairs if p == "linux/amd64"} == set(PYTHONS)
Expand All @@ -78,6 +80,19 @@ def test_jvm_tests_the_one_wheel_of_a_platform_on_the_same_ten_legs_on_every_eve
assert len(pairs) == 10


@pytest.mark.parametrize(
"event", ["release", "workflow_dispatch", "workflow_call", "", "some_new_event"]
)
def test_jvm_tests_every_platform_and_python_for_a_release_a_dispatch_and_any_other_event(
event,
):
# What is published (release-jvm.yml passes release: true, test-jvm.yml turns it into the event `release`), a run
# started by hand before a release, and any event the script does not know (it fails open) get all 16 legs
matrix = sm.select("jvm", event)
assert len(matrix["include"]) == 16
assert _pairs(matrix) == FULL


@pytest.mark.parametrize("event", ["pull_request", *NON_PULL_REQUEST_EVENTS])
def test_jvm_builds_one_wheel_for_each_platform_whatever_the_python(event):
# One build leg for each platform of the test legs, in the order of RUNNERS; no Python in it
Expand All @@ -94,7 +109,9 @@ def test_pull_request_examples_matrix():
@pytest.mark.parametrize("workflow", ["jvm", "examples"])
def test_pull_request_matrix_is_part_of_the_full_matrix(workflow):
assert _pairs(sm.select(workflow, "pull_request")) < FULL
assert _pairs(sm.select(workflow, "pull_request")) <= _pairs(sm.select(workflow, "push"))
assert _pairs(sm.select(workflow, "pull_request")) <= _pairs(
sm.select(workflow, "push")
)


def test_unknown_workflow_is_refused():
Expand Down Expand Up @@ -124,7 +141,7 @@ def test_pythons_match_the_classifiers_of_pyproject():


def test_command_line_prints_one_line_of_matrix_json():
for event, count in (("pull_request", 10), ("push", 10)):
for event, count in (("pull_request", 10), ("push", 10), ("release", 16)):
out = subprocess.run( # nosec B603 - fixed argument list, no untrusted input
[sys.executable, "-I", str(SCRIPT_PATH), "jvm", "--event", event],
capture_output=True,
Expand All @@ -145,20 +162,43 @@ def test_workflow_takes_its_matrix_from_the_script(workflow):
assert "python-version:\n" not in strategy.split("steps:", 1)[0]


def test_a_jvm_release_run_asks_for_the_full_matrix():
# In a workflow_call github.event_name is the caller's (a tag push), so the release says so through an input
text = WORKFLOWS["jvm"].read_text(encoding="utf-8")
call = text.split(" workflow_call:\n", 1)[1].split("\n workflow_dispatch:\n", 1)[
0
]
assert (
" release:\n" in call
and "type: boolean" in call.split(" release:\n", 1)[1]
)
assert "EVENT_NAME: ${{ inputs.release && 'release' || github.event_name }}" in text
release = (ROOT / ".github" / "workflows" / "release-jvm.yml").read_text(
encoding="utf-8"
)
job = release.split(" uses: ./.github/workflows/test-jvm.yml\n", 1)[1].split(
"\n\n", 1
)[0]
assert " release: true" in job


def test_the_jvm_workflow_builds_each_platform_once_and_tests_that_wheel():
text = WORKFLOWS["jvm"].read_text(encoding="utf-8")
assert "scripts/select_matrix.py jvm --event \"$EVENT_NAME\" --wheels" in text
assert 'scripts/select_matrix.py jvm --event "$EVENT_NAME" --wheels' in text
assert "matrix: ${{ fromJSON(needs.download-jars.outputs.wheel-matrix) }}" in text
build, test = text.split("\n build:\n")[1].split("\n test:\n")
# no Python in the build matrix, and the test job builds nothing: it installs the wheel the build job uploaded
assert "python-version: ${{ matrix.python-version }}" not in build
assert "scripts/build.sh" in build and "scripts/build.sh" not in test.split("\n test-summary:")[0]
assert (
"scripts/build.sh" in build
and "scripts/build.sh" not in test.split("\n test-summary:")[0]
)
assert "actions/download-artifact" in test and "name: wheel-" in test


def test_the_examples_workflow_builds_each_platform_once_and_runs_the_examples_against_that_wheel():
text = WORKFLOWS["examples"].read_text(encoding="utf-8")
assert "scripts/select_matrix.py examples --event \"$EVENT_NAME\" --wheels" in text
assert 'scripts/select_matrix.py examples --event "$EVENT_NAME" --wheels' in text
assert "matrix: ${{ fromJSON(needs.download-jars.outputs.wheel-matrix) }}" in text
build, rest = text.split("\n build:\n")[1].split("\n test-examples:\n")
test = rest.split("\n test-examples-summary:")[0]
Expand Down
Loading