[anth-volk]

Fixes #10

Summary

Carries over PolicyEngine/calibration-diagnostics#219, which is still open there: the Postgres store applies each stored event batch onto the run's stored columns instead of re-reading every event of the run and re-materializing it for every batch (calibration-diagnostics#217, O(n²) over a run). Run state, database contents and HTTP responses are unchanged, and there is no schema change or migration.

Source: calibration-diagnostics#219, branch collector-incremental-batch-apply, head 3dfbe0558dfccb3edea5e59476a2e180a2fd7043.

How it was carried over

tools/extract-from-calibration-diagnostics.sh accepts any branch or commit as EXTRACTION_SOURCE_REF, so it carried over #219's head directly, without cherry-picks:

EXTRACTION_SOURCE_REF=3dfbe0558dfccb3edea5e59476a2e180a2fd7043 \
  tools/extract-from-calibration-diagnostics.sh ../collector-extraction tools/extraction-source.env
git fetch ../collector-extraction main
git merge --no-ff FETCH_HEAD

git-filter-repo rewrote #219's commits with their authors and dates:

calibration-diagnostics commit	Here	Change
6993387a2a0d	b768c55	Add an incremental event-batch reducer equivalent to full materialization
e8815473cdfe	548af5a	Apply each stored event batch incrementally in the Postgres store
15bb5b38e6fe	595c557	Test incremental batch application with non-UTC timestamps
3dfbe0558dfc	ae1dc45	Document that the producer read must follow the run row lock

#219 also merged calibration-diagnostics main (bbe4ea5b7881). That merge changes nothing under telemetry-service/, so the filter drops it. The merge here was conflict-free: none of #219's four files changed in this repo since the extraction.

After the merge, telemetry_collector/, migrations/, alembic.ini and Dockerfile here have the same git object IDs as their counterparts under telemetry-service/ at 3dfbe05, so the image built from this branch runs exactly #219's code.

The second commit updates tools/extraction-source.env (source_commit 3dfbe05…, extracted_commit ae1dc45…) and adds the changelog fragment (a fixed fragment, so a patch release).

Merging

• Merge with a merge commit (the only method this repo allows), so that main keeps ae1dc45, the commit the record names.
• When to merge: this PR is meant to be the first deployment from this repo. Follow the switch-over runbook in calibration-diagnostics#233. In short: calibration-diagnostics#219 merges there first and deploys from there; the switch PR stops calibration-diagnostics from deploying; COLLECTOR_DEPLOY_ENABLED is set to true; then this PR merges. If it merges while the variable is still unset, the release is tagged but not deployed, and the next merge deploys it.
• Merge before microcosm-collector#12, and before any other PR that adds an Alembic migration, a dependency or a runtime setting. The first deployment from this repo deploys everything on main, and Accept artifact submissions into private storage and run worker jobs #12 adds migration 0002_artifact_intake. microcosm-collector#9, already merged, adds only telemetry_collector/orrery/, which the service doesn't import.
• If #219 gains more commits before it merges, re-run the extraction from its new head on this branch and merge again. The recorded commit is already in this branch's history, so the tool only adds the new commits.
• #219 should be merged with "Create a merge commit", or closed unmerged if the team prefers to ship it only from here. A squash or rebase merge there would give its changes new commits, and a later extraction from calibration-diagnostics main would then refuse to run, because ae1dc45 would not be in the new history. After the switch, nothing should need carrying over, so this matters only if telemetry-service/ changes again before calibration-diagnostics#234 removes it.
• microcosm-collector#4 is independent: it touches no file this PR touches. The release version depends on the fragments pending on main when this merges (this PR adds a fixed fragment).
• Update the branch before merging. The main ruleset requires branches to be up to date. 564e12b and ea621b9 merged main after microcosm-collector#9 and its 0.3.0 release; any later merge to main needs another update.

Tests

Red and green per commit:

Commit	Red (before the change)	Green (after)
832885b Carry over calibration-diagnostics#219	#219's tests/test_incremental_batch_apply.py and tests/test_storage_postgres.py on this repo's previous code: both modules fail at collection (ImportError: cannot import name 'MATERIALIZED_RUN_COLUMNS'), 0 tests run	16 passed (8 + 8). The 8 PostgreSQL tests ran against a local PostgreSQL 16 container with TEST_DATABASE_URL set; without it they skip
7e557fe Record the carry-over	tools/check-changelog.sh exits 1 (no fragment)	tools/check-changelog.sh passes (0.2.0 → 0.2.1); tests/test_extraction.py and tests/test_release_versioning.py: 22 passed

After committing, the same files passed again (16 and 22). After each update from main (564e12b, ea621b9, which brought in microcosm-collector#9 and its release; no change of their own), the four files passed again together (38) and the changelog check passes (now 0.3.0 → 0.3.1). Throughout, tools/check-quality.sh (Ruff lint and format, bash -n on every script) passes. Locally I ran only these test files. CI runs the whole suite, the migration and alembic check, actionlint and the image build.

Deployment

This PR changes no workflow, Dockerfile, deploy script or configuration. Its merge deploys only if COLLECTOR_DEPLOY_ENABLED is true at that time. The deployment's variables, secrets and Google Cloud access were checked end to end for the switch; the table is in calibration-diagnostics#233.

🤖 Generated with Claude Code