Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
version: 2

updates:
# The package pins no runtime dependencies beyond numpy/scipy, but the test
# and CI toolchain moves. Weekly PRs keep the reproduction honest against
# current versions rather than the ones that happened to be installed on the
# day of release.
- package-ecosystem: pip
directory: /
schedule:
interval: weekly
day: monday
open-pull-requests-limit: 3
commit-message:
prefix: deps

- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
day: monday
open-pull-requests-limit: 3
commit-message:
prefix: ci
7 changes: 7 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ on:
push:
branches: [master]
pull_request:
# A package whose claim is reproducibility should keep proving it. The
# weekly run re-asserts the paper's numbers against whatever numpy, scipy
# and Python resolve to that week, so drift surfaces here rather than in a
# reader's environment.
schedule:
- cron: "0 6 * * 1"
workflow_dispatch:

permissions:
contents: read
Expand Down
34 changes: 34 additions & 0 deletions .github/workflows/draft-pdf.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Draft PDF

# Compiles paper.md with the Open Journals (inara) toolchain and uploads the
# rendered paper.pdf as a build artifact. This is the JOSS-documented preview
# workflow: it proves the paper compiles before submission. It does not contact
# JOSS and does not submit anything.

on:
push:
paths:
- 'paper.md'
- 'paper.bib'
- '.github/workflows/draft-pdf.yml'
workflow_dispatch:

jobs:
paper:
runs-on: ubuntu-latest
name: Paper Draft
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Build draft PDF
uses: openjournals/openjournals-draft-action@master
with:
journal: joss
paper-path: paper.md

- name: Upload rendered paper
uses: actions/upload-artifact@v4
with:
name: paper
path: paper.pdf
2 changes: 1 addition & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
permissions:
id-token: write # required for Trusted Publishing
steps:
- uses: actions/download-artifact@v4
- uses: actions/download-artifact@v8
with:
name: dist
path: dist/
Expand Down
33 changes: 33 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,38 @@
# Changelog

## Unreleased

Screen-result export. **No change to package behavior:**
`tests/test_reproduces_paper.py` is untouched and the demo reproduces the same
numbers as 0.1.1.

- `--export PATH` writes a `pit-screen-results` v1.0 record — per-signal summary
statistics and the screen settings that produced the verdicts. Fully offline
(local file, no network call) and free of clocks, so the same screen exports
byte-identical bytes.
- `docs/results-schema.md`: the schema as a standalone versioned interchange
spec, free for other tools to adopt.
- New module `pit_release_gate.results`: `summarize_signal`, `screen_config`,
`build_results`, `validate_results`, `write_results`. Totals are derived, so
they cannot disagree with the per-signal rows.
- Still no telemetry of any kind, now enforced structurally rather than by
policy: no module in the package imports a transport, and a test asserts that
over every module — no background thread, no `atexit` hook, nothing to opt
out of.

## 0.1.1 — 2026-08-16

Documentation and submission materials only. **No change to package behavior:**
the test suite, including `tests/test_reproduces_paper.py`, is untouched and the
demo reproduces the same numbers as 0.1.0.

- `paper.md` and `paper.bib`: software paper prepared for submission to the
Journal of Open Source Software.
- README: added *Statement of need*, *API overview*, and *Community guidelines*
sections.
- CI: `draft-pdf.yml` builds a preview PDF of `paper.md` with the Open Journals
toolchain and uploads it as a build artifact.

## 0.1.0 — 2026-08-15

First public release.
Expand Down
107 changes: 107 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
[![CI](https://github.com/MaxWellApexLab/pit-release-gate/actions/workflows/ci.yml/badge.svg)](https://github.com/MaxWellApexLab/pit-release-gate/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/MaxWellApexLab/pit-release-gate/branch/master/graph/badge.svg)](https://codecov.io/gh/MaxWellApexLab/pit-release-gate)
[![PyPI](https://img.shields.io/pypi/v/pit-release-gate)](https://pypi.org/project/pit-release-gate/)
[![Downloads](https://img.shields.io/pypi/dm/pit-release-gate)](https://pypi.org/project/pit-release-gate/)
[![Python](https://img.shields.io/pypi/pyversions/pit-release-gate)](https://pypi.org/project/pit-release-gate/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![PIT Hygiene](https://img.shields.io/badge/PIT%20Hygiene-pledged-2ea44f)](https://github.com/MaxWellApexLab/pit-hygiene)

Completeness-aware release control for staggered-arrival cross-sectional data.

Expand All @@ -20,6 +22,24 @@ correlation fitted honestly on prior *completed* periods — and grades the requ
completeness per signal, so benign signals release early and susceptible signals are
withheld until enough of the cross-section has arrived to suppress the bias.

## Statement of need

Point-in-time discipline in ML pipelines currently rests on tooling that answers one
question: *was this value readable at time t?* Feature-store as-of joins, bitemporal
and vintage-aware storage, and purged or embargoed cross-validation all enforce
read-time correctness, and they do it well.

None of them answers a second question: given that every value read was legitimately
readable, was the *set* of entities that had reported by t a selected sample? An
as-of join over an incomplete cross-section is a correct join over a biased sample.
The two failures need different remedies — the first is fixed by timestamp hygiene,
the second only by waiting or by an explicit correction. Researchers building
cross-sectional signals on staggered-arrival panels have had no routine, per-signal
screen for the second. `pit-release-gate` is that screen, plus the release controller
that acts on it: one `fit_trailing` call per signal, so reporting a susceptibility
estimate alongside a released signal costs about as much as reporting a standard
error.

## Install

```bash
Expand Down Expand Up @@ -56,6 +76,21 @@ values, and per-entity filing-arrival times, then call
`ReleaseController.decide(store, t)` at each evaluation time — it returns
`WITHHOLD`, `REWEIGHT_RELEASE`, or `RELEASE` plus the released values.

## API overview

Five public components, all importable from the top-level `pit_release_gate` package:

| component | what it does |
|---|---|
| [`AsOfDataStore`](src/pit_release_gate/store.py) | Holds one period's as-filed records for a cross-sectional group: design matrix, signal values, and a filing-arrival time per entity. |
| [`CompletenessMonitor`](src/pit_release_gate/monitor.py) | Reports the arrived fraction at an evaluation time, plus a composition-shift gauge for the arrived subset. |
| [`SusceptibilityGate`](src/pit_release_gate/gate.py) | Estimates ρ̂, the partial correlation between filing latency and the complete-cross-section residual given observables. `fit_trailing` enforces the honest-estimation contract: prior *completed* periods only. |
| [`PropensityReweighter`](src/pit_release_gate/reweight.py) | Inverse-filing-propensity weights. Included to make a negative result executable: reweighting on observables corrects composition, but cannot remove selection on the disturbance. |
| [`ReleaseController`](src/pit_release_gate/controller.py) | Maps \|ρ̂\| to a required completeness `φ_req = min(1, φ_min + κ·\|ρ̂\|)` and returns `WITHHOLD` / `REWEIGHT_RELEASE` / `RELEASE` at each evaluation time. |

`make_group`, `run_demo` and `demo` ([`simulate.py`](src/pit_release_gate/simulate.py))
generate and run the known-ground-truth worked example described below.

## The known-ground-truth demo

The package ships a self-contained worked example with a *planted* leakage strength,
Expand All @@ -80,6 +115,44 @@ A sensitivity sweep of the policy slope κ shows the timeliness–bias dial:
κ = 2.0 → 100% (bias exactly 0). The demo is deterministic (fixed seed), and
`tests/test_reproduces_paper.py` asserts these numbers.

## Export your screen result

```bash
pit-release-gate --export results.json
```

writes a `pit-screen-results` v1.0 record: per screened signal, how many periods
were screened, how many the measure flagged, mean and max ρ̂, the required
completeness that was assigned, and the verdict (`benign` / `susceptible`), plus
the five settings that produced those verdicts. It is the file a *screened with*
badge should point at. The format is a standalone versioned interchange spec —
[`docs/results-schema.md`](docs/results-schema.md) — that any other tool is free
to emit or consume.

**`--export` is fully offline**: it writes a local file and makes no network
call. Records carry summary statistics only — never input rows, file paths,
usernames, hostnames, or any environment detail beyond the tool version — and no
clock is read while building one, so the same screen always exports byte-identical
bytes.

Where the record goes afterwards is entirely your business — commit it next to
your badge, publish it, or keep it. This tool does not send it anywhere.

**No telemetry — structurally, not merely by default.** No module in this
package imports a transport at all, and a test walks every module in the package
and fails the suite if one ever does. There is no background thread, no `atexit`
hook, no anonymous usage counter, and nothing to opt out of.

Programmatic use, for screens on your own data:

```python
from pit_release_gate import build_results, screen_config, summarize_signal, validate_results

sig = summarize_signal("accruals", rhos=[...], phi_reqs=[...], rho_threshold=0.10)
record = build_results([sig], screen_config(0.10, 0.35, 1.0, trailing_k=8, min_entities=6))
assert validate_results(record) == []
```

## Papers

The method and its evaluation are developed in three public papers:
Expand All @@ -95,6 +168,31 @@ The method and its evaluation are developed in three public papers:
This package is the reference implementation of paper 3's release controller; its
demo reproduces the paper's controlled experiment.

## Badge

If you have run the susceptibility screen on your own data — whatever the result — you
are welcome to say so:

```markdown
[![screened with pit-release-gate](https://img.shields.io/badge/screened%20with-pit--release--gate-blue)](https://github.com/MaxWellApexLab/pit-release-gate)
```

The badge reads **screened with**, not *passed* — it states that the screen was run, the
same way a formatter badge states that the formatter was run. A benign result and a
susceptible result are equally worth badging; the second one arguably more, because it
means the screen found something and your pipeline now waits for it.

**Make it point at something.** A badge is worth reading only if there is evidence behind
it. Commit your screen output — `pit-release-gate --export results.json` produces exactly
that file: which signals came out benign, which came out susceptible, and the required
completeness each was assigned — and link the badge at it rather
than at this repo. A worked example is the OSAP screen in the
[PIT audit registry](https://github.com/MaxWellApexLab/pit-audit-registry/blob/main/audits/2026-08_osap/report.md).

**Related:** the [PIT Hygiene pledge](https://github.com/MaxWellApexLab/pit-hygiene) is a
broader, tool-neutral statement about how a staggered-arrival pipeline is built; this badge
is the narrower statement that this particular screen was run.

## Cite this

See [`CITATION.cff`](CITATION.cff). If you use this software, please cite paper 3:
Expand All @@ -110,6 +208,15 @@ See [`CITATION.cff`](CITATION.cff). If you use this software, please cite paper
}
```

## Community guidelines

- **Report a bug or request a feature:** open an issue at
[github.com/MaxWellApexLab/pit-release-gate/issues](https://github.com/MaxWellApexLab/pit-release-gate/issues).
- **Contribute:** see [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup,
test requirements, and pull-request process.
- **Get help:** open an issue with a minimal reproducing example, or email
maxwellapexlab@proton.me.

## License

MIT — see [LICENSE](LICENSE).
Expand Down
Loading
Loading