Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions .github/workflows/docs-site.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: Documentation site

# Builds the documentation site (site/, Astro Starlight) from docs/ and the source on every
# pull request and push to main, so a broken page, link, anchor or docstring fails CI
# (ITL-1170). This workflow only builds: it publishes nothing. Publishing to GitHub Pages
# is a separate step the developer enables.
#
# The site's address comes from the repository variables DOCS_SITE_URL (origin, e.g.
# https://docs.example.org) and DOCS_SITE_BASE (base path, e.g. /); one that is not
# defined expands to an empty string, and the build then uses the default,
# https://metamorphic-labs.github.io/orcapod-python/ (site_settings in
# scripts/docs_site_content.py).

on:
push:
branches: [main]
pull_request:
workflow_dispatch:

permissions:
contents: read

concurrency:
group: docs-site-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
name: Build the site
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
fetch-depth: 0 # required: hatch-vcs needs full tag history

- name: Install uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1

- name: Set up Python 3.11
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.11"

- name: Install Python dependencies
run: uv sync --locked --dev --python "3.11"

- name: Set up Node
# The version is the engines.node field of site/package.json.
uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with:
node-version-file: site/package.json

- name: Install pnpm
# The exact version in the packageManager field of site/package.json, installed
# with npm: the repository's actions policy does not allow pnpm/action-setup.
working-directory: site
run: npm install --global "$(node -p "require('./package.json').packageManager")"

- name: Install site dependencies
working-directory: site
run: pnpm install --frozen-lockfile

- name: Check site dependency licenses
working-directory: site
run: pnpm licenses list --json --prod | uv run python ../scripts/check_site_licenses.py

- name: Build the site
working-directory: site
env:
DOCS_SITE_URL: ${{ vars.DOCS_SITE_URL }}
DOCS_SITE_BASE: ${{ vars.DOCS_SITE_BASE }}
run: pnpm build
5 changes: 5 additions & 0 deletions .github/workflows/run-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -173,3 +173,8 @@ jobs:
LGPL-2.0-only, LGPL-2.0-or-later,
LGPL-2.1-only, LGPL-2.1-or-later,
LGPL-3.0-only, LGPL-3.0-or-later
# Accepted exceptions, each listed in docs/dev/security-exceptions.md.
# GHSA-fc5q-39xc-vjpf (picomatch): build-time only (the documentation site's
# toolchain, site/pnpm-lock.yaml); no patched version as of 2026-11-17.
# Tracked by ITL-1207: remove when a fix is published.
allow-ghsas: GHSA-fc5q-39xc-vjpf
4 changes: 1 addition & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -170,9 +170,6 @@ venv.bak/
# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
Expand Down Expand Up @@ -218,6 +215,7 @@ docs/dev/*
!docs/dev/design_issues
!docs/dev/notes
!docs/dev/orcapod-design.md
!docs/dev/security-exceptions.md

# General JSON files at project root
/*.json
43 changes: 38 additions & 5 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ Documents written during development are kept in the repository, under docs/dev/
- Never put them in the published documentation (docs/ outside docs/dev/) or at the top
level of the repository. There is no top-level superpowers/; `scripts/design_issues.py
--check` refuses one.
- docs/dev/ is developer material: mkdocs.yml excludes it from the published site
(exclude_docs), and the documentation sample test does not run it.
- docs/dev/ is developer material: the site build never publishes it (EXCLUDED in
scripts/docs_site_content.py), and the documentation sample test does not run it.

## Backward compatibility

Expand Down Expand Up @@ -88,12 +88,11 @@ other ReST cross-reference roles. Use plain ``backtick`` notation to refer to na

docs/ outside docs/dev/ holds library documentation only. docs/dev/ (the design issues
log, the design specification, specs, plans and their scripts) is developer material, kept out
of the published site (exclude_docs in mkdocs.yml) and out of the sample test.
of the published site (EXCLUDED in scripts/docs_site_content.py) and out of the sample test.

Every fenced python block in README.md, docs/index.md, docs/getting-started.md,
docs/troubleshooting.md, docs/concepts/, docs/reference/ and docs/tutorial/ is executed by
tests/test_docs/test_doc_samples.py, and tests/test_docs/test_mkdocs_build.py runs
mkdocs build --strict. When writing docs:
tests/test_docs/test_doc_samples.py. When writing docs:

- Blocks on one page share a namespace and run top to bottom.
- Every line a block prints must appear as a "# ..." comment in that block; "..." inside the
Expand All @@ -109,6 +108,36 @@ mkdocs build --strict. When writing docs:
- Output that the library does not guarantee to be ordered must not be shown as if it were:
print sorted(job.nodes), and sort rows read back from a database before showing them.

## Documentation site

The documentation is published as an Astro Starlight site, built from docs/ (developer,
November 16, 2026, ITL-1170; site/README.md has the commands). mkdocs is gone.

- docs/ is the only source. scripts/docs_site_content.py writes the site's pages
(site/src/content/docs/, site/public/, both ignored by git) on every build: never edit or
commit them, and never copy page text into site/.
- Write pages as before: a "# <title>" first line, mkdocs-style admonitions
(!!! note "Title", ??? success "Solution") and relative .md links; the script converts
them. A link to a missing or unpublished page or file fails the build, and the site build
validates every link and anchor.
- What is published is decided in scripts/docs_site_content.py (PUBLISHED_FILES,
PUBLISHED_FOLDERS, EXCLUDED). A new top-level entry under docs/ fails the build until it is
added to one of them; docs/dev/ is never published.
- The API reference is generated from the source by scripts/docs_site_api.py (griffe).
docs/api/*.md are its input: an introduction and one "::: <object path>" line per object;
never write a signature or a docstring there. A docstring griffe warns about (an Args:
entry the signature does not have) fails the build.
- Build locally with "cd site && pnpm install && pnpm build"; CI builds the site on every pull
request (.github/workflows/docs-site.yml) and tests/test_docs/test_site_content.py runs the
content build without Node. That workflow only builds: publishing to GitHub Pages
(https://metamorphic-labs.github.io/orcapod-python/) is a separate step the developer
enables. An agent never adds a deploy job or changes a Pages setting.
- The site's address is never hard-coded: DOCS_SITE_URL (origin) and DOCS_SITE_BASE
(base path) come from the environment, in CI from the repository variables of the same
names, with the address above when unset or empty; site_settings in
scripts/docs_site_content.py is the one place that reads and checks them. An agent never
creates or changes those variables.

## Linear issue tracking

All work must be linked to a Linear issue. Before starting any feature, bug fix, or
Expand Down Expand Up @@ -312,6 +341,10 @@ What runs where (developer, November 16, 2026; ITL-1181):
needs a reason.
- The Graphviz `dot` binary is optional. The tests that render a graph to an image skip
when `dot` is not on `PATH` and run when it is.
- A security advisory that a CI check accepts (`allow-ghsas` of the dependency review) is
listed in `docs/dev/security-exceptions.md`, the one list of exceptions: one is added only
with the developer's approval and a Linear issue with a due date, and the list is
reviewed at every release.

## Logical type identifiers

Expand Down
43 changes: 38 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@ Documents written during development are kept in the repository, under `docs/dev
- Never put them in the published documentation (`docs/` outside `docs/dev/`) or at the top
level of the repository. There is no top-level `superpowers/`; `scripts/design_issues.py
--check` refuses one.
- `docs/dev/` is developer material: `mkdocs.yml` excludes it from the published site
(`exclude_docs`), and the documentation sample test does not run it.
- `docs/dev/` is developer material: the site build never publishes it (`EXCLUDED` in
`scripts/docs_site_content.py`), and the documentation sample test does not run it.

## Backward compatibility

Expand Down Expand Up @@ -92,12 +92,11 @@ to refer to names (e.g. ``PollingSource``, ``iter_data``, ``CursorInvalidatedErr

`docs/` outside `docs/dev/` holds library documentation only. `docs/dev/` (the design issues
log, the design specification, specs, plans and their scripts) is developer material, kept out
of the published site (`exclude_docs` in `mkdocs.yml`) and out of the sample test.
of the published site (`EXCLUDED` in `scripts/docs_site_content.py`) and out of the sample test.

Every fenced `python` block in `README.md`, `docs/index.md`, `docs/getting-started.md`,
`docs/troubleshooting.md`, `docs/concepts/`, `docs/reference/` and `docs/tutorial/` is executed
by `tests/test_docs/test_doc_samples.py`, and `tests/test_docs/test_mkdocs_build.py` runs
`mkdocs build --strict`. When writing docs:
by `tests/test_docs/test_doc_samples.py`. When writing docs:

- Blocks on one page share a namespace and run top to bottom.
- Every line a block prints must appear as a `# ...` comment in that block; `...` inside the
Expand All @@ -113,6 +112,36 @@ by `tests/test_docs/test_doc_samples.py`, and `tests/test_docs/test_mkdocs_build
- Output that the library does not guarantee to be ordered must not be shown as if it were:
print `sorted(job.nodes)`, and sort rows read back from a database before showing them.

## Documentation site

The documentation is published as an Astro Starlight site, built from `docs/` (developer,
November 16, 2026, ITL-1170; `site/README.md` has the commands). mkdocs is gone.

- `docs/` is the only source. `scripts/docs_site_content.py` writes the site's pages
(`site/src/content/docs/`, `site/public/`, both ignored by git) on every build: never
edit or commit them, and never copy page text into `site/`.
- Write pages as before: a `# <title>` first line, mkdocs-style admonitions (`!!! note "Title"`,
`??? success "Solution"`) and relative `.md` links; the script converts them. A link to a
missing or unpublished page or file fails the build, and the site build validates every
link and anchor.
- What is published is decided in `scripts/docs_site_content.py` (`PUBLISHED_FILES`,
`PUBLISHED_FOLDERS`, `EXCLUDED`). A new top-level entry under `docs/` fails the build until
it is added to one of them; `docs/dev/` is never published.
- The API reference is generated from the source by `scripts/docs_site_api.py` (griffe).
`docs/api/*.md` are its input: an introduction and one `::: <object path>` line per object;
never write a signature or a docstring there. A docstring griffe warns about (an `Args:`
entry the signature does not have) fails the build.
- Build locally with `cd site && pnpm install && pnpm build`; CI builds the site on every pull
request (`.github/workflows/docs-site.yml`) and `tests/test_docs/test_site_content.py` runs
the content build without Node. That workflow only builds: publishing to GitHub Pages
(`https://metamorphic-labs.github.io/orcapod-python/`) is a separate step the developer
enables. An agent never adds a deploy job or changes a Pages setting.
- The site's address is never hard-coded: `DOCS_SITE_URL` (origin) and `DOCS_SITE_BASE`
(base path) come from the environment, in CI from the repository variables of the same
names, with the address above when unset or empty; `site_settings` in
`scripts/docs_site_content.py` is the one place that reads and checks them. An agent never
creates or changes those variables.

## Linear issue tracking

All work must be linked to a Linear issue. Before starting any feature, bug fix, or
Expand Down Expand Up @@ -323,6 +352,10 @@ What runs where (developer, November 16, 2026; ITL-1181):
needs a reason.
- The Graphviz `dot` binary is optional. The tests that render a graph to an image skip
when `dot` is not on `PATH` and run when it is.
- A security advisory that a CI check accepts (`allow-ghsas` of the dependency review) is
listed in `docs/dev/security-exceptions.md`, the one list of exceptions: one is added only
with the developer's approval and a Linear issue with a due date, and the list is
reviewed at every release.

## Logical type identifiers

Expand Down
2 changes: 2 additions & 0 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ All of these hold before the developer runs the Release workflow:
6. **The Linear release is named** (see *Linear Release System*): the open release of the
Linear release pipeline has the version `vX.Y.Z`, and nothing merges into `main` until
`linear-complete` has finished. Not for a hotfix run, which leaves the open release as it is.
7. **The security exceptions are reviewed**: every exception in
`docs/dev/security-exceptions.md` whose advisory has a published fix is removed.

## Golden hash values

Expand Down
10 changes: 10 additions & 0 deletions changelog.d/ITL-1170.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
### Changed

- The documentation is built as an [Astro Starlight](https://starlight.astro.build) site
from `docs/`, with an API reference generated from the source code on every build. CI
builds the site on every pull request; it is not published yet. mkdocs, mkdocs-material
and mkdocstrings are no longer development dependencies; `griffelib` is. Building the site
needs Node and pnpm (`site/README.md`). The site's address is configurable: `DOCS_SITE_URL`
(origin) and `DOCS_SITE_BASE` (base path), from the environment or, in CI, the repository
variables of the same names; unset or empty, it is
`https://metamorphic-labs.github.io/orcapod-python/`. (ITL-1170)
6 changes: 6 additions & 0 deletions docs/api/databases.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# Databases
<!--
Input of the API reference generator (scripts/docs_site_api.py), not a page written by
hand: this file chooses the objects (one `::: <object path>` line each) and gives the
introduction. Every signature and docstring on the published page comes from the source
code. See site/README.md.
-->

Database backends for persistent storage of pipeline computation results.

Expand Down
6 changes: 6 additions & 0 deletions docs/api/function-pods.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# Function Pods
<!--
Input of the API reference generator (scripts/docs_site_api.py), not a page written by
hand: this file chooses the objects (one `::: <object path>` line each) and gives the
introduction. Every signature and docstring on the published page comes from the source
code. See site/README.md.
-->

Function pods wrap Python functions to transform individual data in a stream.

Expand Down
6 changes: 6 additions & 0 deletions docs/api/nodes.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# Nodes
<!--
Input of the API reference generator (scripts/docs_site_api.py), not a page written by
hand: this file chooses the objects (one `::: <object path>` line each) and gives the
introduction. Every signature and docstring on the published page comes from the source
code. See site/README.md.
-->

Nodes are the elements of a compiled pipeline graph. The plain node classes describe the
graph of a `Pipeline` blueprint; the job node classes below add database-backed execution.
Expand Down
6 changes: 6 additions & 0 deletions docs/api/operators.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# Operators
<!--
Input of the API reference generator (scripts/docs_site_api.py), not a page written by
hand: this file chooses the objects (one `::: <object path>` line each) and gives the
introduction. Every signature and docstring on the published page comes from the source
code. See site/README.md.
-->

Operators perform structural transformations on streams without inspecting or synthesizing data values. An operator may add a tag (`AddTags`), never data.

Expand Down
6 changes: 6 additions & 0 deletions docs/api/pipeline.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# Pipeline
<!--
Input of the API reference generator (scripts/docs_site_api.py), not a page written by
hand: this file chooses the objects (one `::: <object path>` line each) and gives the
introduction. Every signature and docstring on the published page comes from the source
code. See site/README.md.
-->

Top-level objects for composing and executing Orcapod data pipelines. `Pipeline` is a pure
blueprint; `PipelineJob` binds a pipeline to sources and a database and runs it.
Expand Down
6 changes: 6 additions & 0 deletions docs/api/sources.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# Sources
<!--
Input of the API reference generator (scripts/docs_site_api.py), not a page written by
hand: this file chooses the objects (one `::: <object path>` line each) and gives the
introduction. Every signature and docstring on the published page comes from the source
code. See site/README.md.
-->

Source classes provide the entry point for external data into Orcapod pipelines.
All sources convert their input to a PyArrow Table and use `SourceStreamBuilder` for
Expand Down
6 changes: 6 additions & 0 deletions docs/api/streams.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# Streams
<!--
Input of the API reference generator (scripts/docs_site_api.py), not a page written by
hand: this file chooses the objects (one `::: <object path>` line each) and gives the
introduction. Every signature and docstring on the published page comes from the source
code. See site/README.md.
-->

Streams are immutable sequences of (Tag, Data) pairs backed by PyArrow tables.

Expand Down
6 changes: 6 additions & 0 deletions docs/api/types.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# Types
<!--
Input of the API reference generator (scripts/docs_site_api.py), not a page written by
hand: this file chooses the objects (one `::: <object path>` line each) and gives the
introduction. Every signature and docstring on the published page comes from the source
code. See site/README.md.
-->

Core type definitions used throughout Orcapod.

Expand Down
18 changes: 18 additions & 0 deletions docs/dev/security-exceptions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Accepted security advisory exceptions

This is the one list of security advisories that a CI check is told to accept. Each row
is an exception to a check that would otherwise fail, such as the `Dependency review` job
in `.github/workflows/run-tests.yml` (`allow-ghsas`).

Rules (developer, November 17, 2026):

- An exception is added only with the developer's approval.
- Each exception has a Linear issue with a due date, named in its row and in the comment
next to the setting that allows it.
- The list is reviewed at every release (`RELEASING.md`, pre-release checklist). An
exception whose advisory has a published fix is removed: upgrade the package, remove
the setting, delete the row.

| Advisory | Package and versions | Where allowed | Why accepted | Accepted | Tracking issue | How to check | Removal |
|---|---|---|---|---|---|---|---|
| [GHSA-fc5q-39xc-vjpf](https://github.com/advisories/GHSA-fc5q-39xc-vjpf) (high: ReDoS through single-branch repeated extglobs) | npm `picomatch` <= 4.0.7: 2.3.2 and 4.0.7 in `site/pnpm-lock.yaml`, through `astro` (`unstorage` > `anymatch`, `@astrojs/internal-helpers`, `vite`, `fdir`) | `allow-ghsas` of the `Dependency review` step, `.github/workflows/run-tests.yml` | Build-time only: the documentation site's toolchain matches the repository's own file globs; the published site is static HTML and the Python package does not contain it. No patched version exists (4.0.7 is the latest release) | 2026-11-17 | ITL-1207 (due 2026-12-15) | `gh api /advisories/GHSA-fc5q-39xc-vjpf -q '.vulnerabilities[].first_patched_version'` is no longer `null`; or `cd site && pnpm audit` | Update `site/pnpm-lock.yaml` to a fixed `picomatch` (a newer `astro`, or a pnpm override), remove the GHSA from `allow-ghsas`, delete this row |
Loading
Loading