Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 0 additions & 14 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,17 +83,3 @@ jobs:
name: distributions
path: dist
- run: bash tools/check-collector.sh

reader:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: "1.4.2"
- run: bash tools/check-reader.sh
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: orrery-reader
path: reader-dist/*.tgz
if-no-files-found: error
23 changes: 2 additions & 21 deletions .github/workflows/push.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ permissions:
# run, and the app may bypass the main ruleset. On that release commit, the
# remaining jobs verify, tag, then publish, each only after the one before.
#
# PyPI and npm trust this file by name, in the pypi environment, which only
# main can use. Renaming it stops publication.
# PyPI trusts this file by name, in the pypi environment, which only main can
# use. Renaming it stops publication.
jobs:
versioning:
name: Update versioning
Expand Down Expand Up @@ -76,22 +76,3 @@ jobs:
- uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
with:
packages-dir: dist/

publish-reader:
needs: [verify, publish]
runs-on: ubuntu-latest
environment: pypi
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: "24"
registry-url: "https://registry.npmjs.org"
- uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
with:
name: orrery-reader
path: reader-dist
- run: bash tools/publish-reader.sh
3 changes: 0 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,3 @@ dist/
.coverage
.collector/
.test-artifacts/
node_modules/
reader-dist/
typescript/orrery/dist/
13 changes: 5 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ The host can also start telemetry and graph modules together with `start_service

Preserve the selected source files until publication succeeds. `microcosm-emitter-publish-graph` asks the service to retry a publication by ID and can re-enqueue a preserved directory on another machine. It never rewrites the original build receipt; the queue and separate latest-status file record retries.

The existing TypeScript package `@policyengine/microcosm-provider-orrery` reads immutable published graph files on a server or in a browser. It remains separate from the Python distribution and does not change the hosted telemetry API.

## Development

Use Python 3.13 or 3.14 on Linux or macOS.
Expand All @@ -46,23 +44,22 @@ uv sync --locked
bash tools/check-quality.sh
uv run --no-sync pytest -m 'not collector'
bash tools/check-artifacts.sh
bash tools/check-reader.sh
bash tools/check-changelog.sh
```

Tests isolate ambient credentials and use synthetic identities. The artifact check builds one source archive and wheel, installs that wheel into a clean environment away from the source checkout, verifies import boundaries, and exercises telemetry-only, graph-only and combined service startup with packaged migrations. Reader checks cover tests, formatting, types, build and an isolated tarball installation; they do not inspect visual styling.
Tests isolate ambient credentials and use synthetic identities. The artifact check builds one source archive and wheel, installs that wheel into a clean environment away from the source checkout, verifies import boundaries, and exercises telemetry-only, graph-only and combined service startup with packaged migrations.

The collector compatibility job tests the installed wheel against a pinned dashboard collector and disposable PostgreSQL. It covers the HTTP exchange, event ingestion, blocked outcomes and supplied failure codes in dashboard run documents, rejected identities, and duplicate delivery after a lost acknowledgement. Local execution requires that pinned collector checkout and a loopback test database; `tools/check-collector.sh` verifies these inputs. Never point this test at a deployed database. The pin verifies source compatibility, not which revision the hosted collector currently runs.

## Publication

Every merge to `main` releases the Python distribution `microcosm-emitter` to PyPI and the TypeScript reader `@policyengine/microcosm-provider-orrery` to npm at one shared version. Each pull request adds a changelog fragment named `changelog.d/<name>.<type>.md`, where the type is `breaking`, `added`, `changed`, `fixed` or `removed`; `tools/check-changelog.sh` refuses a pull request without one.
Every merge to `main` releases the Python distribution `microcosm-emitter` to PyPI. Each pull request adds a changelog fragment named `changelog.d/<name>.<type>.md`, where the type is `breaking`, `added`, `changed`, `fixed` or `removed`; `tools/check-changelog.sh` refuses a pull request without one.

`push.yaml` runs twice for each merge:

1. On the merge commit, it bumps the version in `pyproject.toml`, both `package.json` files and `uv.lock`: a major bump for a breaking fragment, a minor bump for an added or removed fragment, and a patch bump otherwise. It collects the fragments into `CHANGELOG.md` and commits the result as "Update microcosm-emitter" using the PolicyEngine GitHub App, which may bypass the `main` ruleset.
2. On that commit, it runs every CI check, tags the commit with the bare version (for example `0.2.0`), publishes the tested wheel and source archive to PyPI, and then publishes the tested reader tarball to npm. Each step starts only after the one before succeeds.
1. On the merge commit, it bumps the version in `pyproject.toml` and `uv.lock`: a major bump for a breaking fragment, a minor bump for an added or removed fragment, and a patch bump otherwise. It collects the fragments into `CHANGELOG.md` and commits the result as "Update microcosm-emitter" using the PolicyEngine GitHub App, which may bypass the `main` ruleset.
2. On that commit, it runs every CI check, tags the commit with the bare version (for example `0.2.0`), and publishes the tested wheel and source archive to PyPI. Each step starts only after the one before succeeds.

Both registries accept uploads only through trusted publishing from `push.yaml` in the `pypi` environment, which only `main` can use. CI holds no long-lived registry token. Renaming the workflow or the environment stops publication until both registries' trusted-publisher settings change to match.
PyPI accepts uploads only through trusted publishing from `push.yaml` in the `pypi` environment, which only `main` can use. CI holds no long-lived registry token. Renaming the workflow or the environment stops publication until PyPI's trusted-publisher settings change to match.

No hosted-service deployment or consumer activation follows from publishing this package.
1 change: 1 addition & 0 deletions changelog.d/remove-typescript-reader.removed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Remove the unused TypeScript reader `@policyengine/microcosm-provider-orrery`, so each release now publishes only the `microcosm-emitter` Python distribution to PyPI.
15 changes: 0 additions & 15 deletions package.json

This file was deleted.

75 changes: 39 additions & 36 deletions tests/test_workflows.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

ROOT = Path(__file__).resolve().parents[1]
RELEASE_COMMIT = "Update microcosm-emitter"
VERSIONED_FILES = ("pyproject.toml", "package.json", "typescript/orrery/package.json")
VERSIONED_FILE = "pyproject.toml"


def workflow_paths():
Expand All @@ -30,16 +30,24 @@ def bump_version():


def release_tree(root, *fragments):
for name in VERSIONED_FILES:
(root / name).parent.mkdir(parents=True, exist_ok=True)
shutil.copy(ROOT / name, root / name)
shutil.copy(ROOT / VERSIONED_FILE, root / VERSIONED_FILE)
(root / "changelog.d").mkdir()
(root / "changelog.d/.gitkeep").touch()
for fragment in fragments:
(root / "changelog.d" / fragment).write_text("A change.\n")
return root


def remove_version_line(pyproject_text):
return re.sub(
r'^version = "[^"]+"\n', "", pyproject_text, count=1, flags=re.MULTILINE
)


def add_tool_version_line(pyproject_text):
return pyproject_text + '\n[tool.example]\nversion = "9.9.9"\n'


def test_shell_logic_lives_in_scripts():
for path in workflow_paths():
config = workflow(path.name)
Expand All @@ -65,7 +73,7 @@ def test_verification_includes_real_collector_and_matrix():
config = workflow("ci.yml")
assert set(config["on"]) == {"workflow_call"}
jobs = config["jobs"]
assert set(jobs) == {"quality", "tests", "build-packages", "collector", "reader"}
assert set(jobs) == {"quality", "tests", "build-packages", "collector"}
assert jobs["tests"]["strategy"]["matrix"] == {
"os": ["ubuntu-latest", "macos-latest"],
"python": ["3.13", "3.14"],
Expand All @@ -78,6 +86,7 @@ def test_release_commit_is_verified_then_tagged_then_published():
config = workflow("push.yaml")
assert config["on"]["push"]["branches"] == ["main"]
jobs = config["jobs"]
assert set(jobs) == {"versioning", "verify", "tag", "publish"}
assert f"!= '{RELEASE_COMMIT}'" in jobs["versioning"]["if"]
assert jobs["versioning"]["steps"][-2]["run"] == "bash tools/update-version.sh"
assert jobs["versioning"]["steps"][-1]["with"]["message"] == RELEASE_COMMIT
Expand All @@ -86,7 +95,6 @@ def test_release_commit_is_verified_then_tagged_then_published():
assert jobs["tag"]["needs"] == "verify"
assert jobs["tag"]["steps"][-1]["run"] == "bash tools/publish-git-tag.sh"
assert jobs["publish"]["needs"] == "tag"
assert jobs["publish-reader"]["needs"] == ["verify", "publish"]


def test_only_publication_holds_oidc_and_only_versioning_holds_secrets():
Expand All @@ -96,29 +104,18 @@ def test_only_publication_holds_oidc_and_only_versioning_holds_secrets():
oidc_jobs = {
name for name, job in jobs.items() if "id-token" in job.get("permissions", {})
}
assert oidc_jobs == {"publish", "publish-reader"}
for name in oidc_jobs:
assert jobs[name]["environment"] == "pypi"
assert oidc_jobs == {"publish"}
assert jobs["publish"]["environment"] == "pypi"
assert jobs["publish"]["permissions"] == {"id-token": "write"}
text = (ROOT / ".github/workflows/push.yaml").read_text()
assert set(re.findall(r"secrets\.(\w+)", text)) == {"APP_ID", "APP_PRIVATE_KEY"}
assert text.count("secrets.") == 2


def test_registries_receive_only_verified_artifacts():
jobs = workflow("push.yaml")["jobs"]
publish = jobs["publish"]
def test_pypi_receives_only_verified_artifacts():
publish = workflow("push.yaml")["jobs"]["publish"]
assert publish["steps"][0]["with"]["name"] == "distributions"
assert not any("run" in step for step in publish["steps"])
reader = jobs["publish-reader"]
download = next(
step for step in reader["steps"] if "download-artifact@" in step.get("uses", "")
)
assert download["with"]["name"] == "orrery-reader"
assert reader["steps"][-1]["run"] == "bash tools/publish-reader.sh"
publish = (ROOT / "tools/publish-reader.sh").read_text()
# A bare "reader-dist/<file>" is a GitHub shorthand to npm, not a path.
assert 'npm publish --access public --provenance "./${archives[0]}"' in publish


@pytest.mark.parametrize(
Expand All @@ -135,20 +132,21 @@ def test_fragment_types_choose_the_bump(bump_version, types, expected):
assert bump_version["next_version"]("1.4.2", types) == expected


def test_release_writes_one_version_to_every_package(tmp_path, bump_version):
def test_release_rewrites_only_the_version_line(tmp_path, bump_version):
root = release_tree(tmp_path, "1.added.md", "repair.fixed.md")
current, upcoming = bump_version["plan_release"](root)
assert upcoming == bump_version["next_version"](current, {"added", "fixed"})
bump_version["write_version"](root, current, upcoming)
assert bump_version["current_version"](root) == upcoming
for name in VERSIONED_FILES:
original = (ROOT / name).read_text().splitlines()
updated = (root / name).read_text().splitlines()
assert len(updated) == len(original)
changed = [
pair for pair in zip(original, updated, strict=True) if pair[0] != pair[1]
]
assert changed == [(changed[0][0], changed[0][0].replace(current, upcoming))]
original = (ROOT / VERSIONED_FILE).read_text().splitlines()
updated = (root / VERSIONED_FILE).read_text().splitlines()
assert len(updated) == len(original)
changed = [
(original_line, updated_line)
for original_line, updated_line in zip(original, updated, strict=True)
if original_line != updated_line
]
assert changed == [(f'version = "{current}"', f'version = "{upcoming}"')]


@pytest.mark.parametrize(
Expand All @@ -162,13 +160,18 @@ def test_release_refuses_missing_or_untyped_fragments(
bump_version["plan_release"](root)


def test_release_refuses_diverged_package_versions(tmp_path, bump_version):
@pytest.mark.parametrize(
"edit_pyproject",
[remove_version_line, add_tool_version_line],
ids=["missing", "duplicated"],
)
def test_release_refuses_a_missing_or_duplicated_version_line(
tmp_path, bump_version, edit_pyproject
):
root = release_tree(tmp_path, "1.fixed.md")
reader = root / "typescript/orrery/package.json"
reader.write_text(
re.sub(r'"version": "[^"]+"', '"version": "9.9.9"', reader.read_text(), count=1)
)
with pytest.raises(ValueError, match="one shared version"):
pyproject = root / VERSIONED_FILE
pyproject.write_text(edit_pyproject(pyproject.read_text()))
with pytest.raises(ValueError, match="one version line"):
bump_version["plan_release"](root)


Expand Down
42 changes: 17 additions & 25 deletions tools/bump-version.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
"""Choose the next shared release version from pending changelog fragments.
"""Choose the next release version from pending changelog fragments.

A breaking fragment bumps the major version, an added or removed fragment the
minor version, and any other fragment the patch version. The Python
distribution and both reader package manifests always share one version.
minor version, and any other fragment the patch version. The version is read
from and written to the single version line in pyproject.toml.

Prints the next version. With --check, writes nothing.
"""
Expand All @@ -17,13 +17,8 @@
"Name each fragment changelog.d/<name>.<type>.md, where <type> is one of: "
+ ", ".join(FRAGMENT_TYPES)
)
VERSIONED_FILE = "pyproject.toml"
PYPROJECT_VERSION = re.compile(r'^version = "(\d+\.\d+\.\d+)"$', re.MULTILINE)
PACKAGE_VERSION = re.compile(r'^ "version": "(\d+\.\d+\.\d+)",$', re.MULTILINE)
VERSIONED_FILES = {
"pyproject.toml": PYPROJECT_VERSION,
"package.json": PACKAGE_VERSION,
"typescript/orrery/package.json": PACKAGE_VERSION,
}


def fragment_types(changelog_dir: Path) -> set[str]:
Expand Down Expand Up @@ -57,14 +52,12 @@ def next_version(version: str, types: set[str]) -> str:


def current_version(root: Path) -> str:
found = {
name: pattern.findall((root / name).read_text())
for name, pattern in VERSIONED_FILES.items()
}
versions = {version for matches in found.values() for version in matches}
if len(versions) != 1 or any(len(matches) != 1 for matches in found.values()):
raise ValueError(f"Expected one shared version, found {found}.")
return versions.pop()
versions = PYPROJECT_VERSION.findall((root / VERSIONED_FILE).read_text())
if len(versions) != 1:
raise ValueError(
f"Expected one version line in {VERSIONED_FILE}, found {versions}."
)
return versions[0]


def plan_release(root: Path) -> tuple[str, str]:
Expand All @@ -73,15 +66,14 @@ def plan_release(root: Path) -> tuple[str, str]:


def write_version(root: Path, current: str, upcoming: str) -> None:
for name, pattern in VERSIONED_FILES.items():
path = root / name
path.write_text(
pattern.sub(
lambda match: match.group(0).replace(current, upcoming),
path.read_text(),
count=1,
)
path = root / VERSIONED_FILE
path.write_text(
PYPROJECT_VERSION.sub(
lambda match: match.group(0).replace(current, upcoming),
path.read_text(),
count=1,
)
)


def main(arguments: list[str]) -> None:
Expand Down
2 changes: 1 addition & 1 deletion tools/check-changelog.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
set -euo pipefail

# Refuse a pull request whose merge could not be released: changelog.d must hold
# at least one typed fragment, and all package manifests must share one version.
# at least one typed fragment, and pyproject.toml exactly one version line.
uv run --no-project python tools/bump-version.py --check
Loading
Loading