Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion .claude/commands/create-newsletter.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,22 @@ Create a new newsletter HTML file from PolicyEngine research posts and optional
- Keep the sign-off, and the footer with all four Mailchimp merge tags
- Run `bash scripts/lint-newsletters.sh`

4. **Show result**:
4. **Check comparables are fresh** (required before sending, Max 2026-10-21):
- Name the file for its send date, then run
`python -m newsletter_uploader.comparables --require-dates editions/<new file>`.
Every policyengine.org analysis the edition links needs a comparables search in
PolicyEngine/policyengine-scorecard on the send date or in the scorecard's
promotion window before it (7 days by default, counting both ends), against the
results the analysis reads now, with no gap still pending.
- For each analysis it lists, run the comparables review with search kind
`prepromotion` (`/comparables-review` and the `policyengine-comparables-review`
skill in the PolicyEngine plugin), merge the scorecard PR, and check again.
CI and `upload-newsletter` both refuse an edition that fails this.
- The check confirms the search is fresh and every gap over the threshold has a
written cause; it does not put the comparable in the edition. Whether an edition
shows a comparable beside our figure is not yet decided.

5. **Show result**:
- Path to generated file
- Preview of content
- Next steps: upload with `upload-newsletter` command
Expand Down
8 changes: 7 additions & 1 deletion .claude/commands/upload-draft.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,13 @@ This is a wrapper around the `upload-newsletter` command that:

1. Validates the HTML file exists
2. Prompts for required fields if missing
3. Runs the upload command
3. Runs the upload command, which first checks that every policyengine.org analysis
the edition links has a fresh comparables search in the scorecard, judged on the
later of the edition's date and today (`src/newsletter_uploader/comparables.py`), and
refuses to upload otherwise, or when the file name does not start with its date. Fix
it by running the comparables review (search kind `prepromotion`), never by renaming
or re-dating the file. To change a draft already uploaded, run the same command with
`--campaign-id`; it runs the same check.
4. Shows the Mailchimp edit URL

## Arguments
Expand Down
57 changes: 56 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,63 @@ jobs:
chmod +x scripts/lint-newsletters.sh
./scripts/lint-newsletters.sh

test:
comparables:
# Max, 2026-10-21: "even if we've already published something, we should
# do a fresh comp review before big pushes like newsletters". Every
# policyengine.org analysis an edition links needs a comparables search in
# PolicyEngine/policyengine-scorecard within the scorecard's promotion
# window before the edition's date (src/newsletter_uploader/comparables.py).
# It checks the editions this change adds or modifies (on a push, the
# pushed range), so an edition already sent is not re-judged against
# later data; each must be named for its date. It downloads the scorecard
# index and app-v2's registries only when such an edition is due, and
# fails if it cannot read them.
name: Fresh comparables for promoted analyses
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.11"

- name: Install
run: pip install -e .

- name: Every analysis a changed edition links has a fresh comparables search
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
# reads each source at the commit its main branch points to
GITHUB_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
# The commit to compare with: the pull request's base, or main before
# the push. A new branch's first push has no such commit, so its
# files are compared with the empty tree. A base the checkout
# cannot read fails the job rather than passing with nothing checked.
if [ -z "$BASE_SHA" ] || [ "$BASE_SHA" = "0000000000000000000000000000000000000000" ]; then
BASE_SHA=$(git hash-object -t tree /dev/null)
elif ! git cat-file -e "$BASE_SHA^{commit}"; then
echo "::error::cannot read the base commit $BASE_SHA"
exit 1
fi
git diff --name-only -z --diff-filter=AMR "$BASE_SHA" HEAD > "$RUNNER_TEMP/all-changed"
# Keep Git's NUL delimiters through selection and checker invocation,
# so Unicode and control characters cannot hide a changed edition.
# The selector also rejects social posts outside editions/.
python scripts/select_changed_editions.py "$RUNNER_TEMP/all-changed" "$RUNNER_TEMP/changed"
if [ ! -s "$RUNNER_TEMP/changed" ]; then
echo "No edition changed; nothing to check."
exit 0
fi
xargs -0 python -m newsletter_uploader.comparables --require-dates < "$RUNNER_TEMP/changed"

test:
# Ubuntu 26.04 has no setup-python builds for Python 3.8 or 3.9.
runs-on: ubuntu-24.04
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,4 @@ __pycache__/
.coverage
htmlcov/
.pytest_cache/
.hypothesis/
29 changes: 22 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## Repository Overview

This is a simple newsletter repository containing HTML email templates for PolicyEngine's Mailchimp campaigns. No build tools, testing frameworks, or development servers are needed - this is pure HTML/CSS.
This repository holds the HTML email templates for PolicyEngine's Mailchimp campaigns, and a Python package (`newsletter_uploader`) that uploads them and checks, before a send, that every analysis an edition links has a fresh comparables search. The editions themselves are plain HTML/CSS with no build step.

## Repository Structure

Expand Down Expand Up @@ -137,6 +137,20 @@ Every link goes to policyengine.org, never to the address of the platform that h
- External assets use full URLs pinned to a commit SHA (no relative paths; see "Images" above)
- The hidden preheader matches the campaign's preview text
- Every link goes to policyengine.org, never a hosting address (see "Links" above)
4. **Fresh comparables before sending (Max, 2026-10-21).** Every policyengine.org
analysis the edition links needs a comparables search recorded in
PolicyEngine/policyengine-scorecard on the send date or in the scorecard's promotion
window before it (7 days by default, counting both ends), against the results the
analysis reads now, with no gap still pending
(`python -m newsletter_uploader.comparables --require-dates editions/<file>`).
For any it lists, run the comparables review with search kind `prepromotion` (the
PolicyEngine plugin's `policyengine-comparables-review` skill) first. Name the file
for its send date: CI checks the editions a change adds or modifies on their dates,
and `upload-newsletter` refuses an undated file and judges the edition on the later
of its date and the day you upload. The same applies to the social posts kept beside
an edition, which live in `editions/` as `<date>-<audience>-social.md` (CI fails a social
post anywhere else). The check confirms the search is fresh; it does not put the comparable
in the edition.

## Email Compatibility

Expand Down Expand Up @@ -168,7 +182,7 @@ Use a full commit SHA, not a branch name or `main`; see "Images" under "Newslett

## File Naming Conventions

- Newsletter editions: `YYYY-MM-DD-{audience}.html` (e.g., `2026-10-20-uk.html`). The file name carries the planned date and isn't renamed if the send moves; the `<title>` and header carry the send date (`2026-10-20-uk.html` is dated 21 October, `2026-10-26-uk.html` 27 October).
- Newsletter editions: `YYYY-MM-DD-{audience}.html` or `YYYY-MM-DD.html`. The file name carries the planned date and is not renamed if the send moves; the `<title>` and header carry the send date. The comparables check uses the file date in CI and the later of the file date and today on upload.
- Images: Descriptive names in `assets/images/`

## Configuration
Expand All @@ -179,7 +193,7 @@ Use a full commit SHA, not a branch name or `main`; see "Images" under "Newslett

## Uploading to Mailchimp

This repo includes a Python package (`newsletter_uploader`) with full test coverage.
This repo includes a Python package (`newsletter_uploader`), tested with pytest.

### Installation

Expand Down Expand Up @@ -219,7 +233,7 @@ The command creates a **draft campaign** (not sent) that you can review, test, a

### Updating Existing Drafts

**IMPORTANT:** When making changes to a newsletter that has already been uploaded to Mailchimp, **update the existing campaign** instead of creating a new one. Use `upload-newsletter` with `--campaign-id`, which runs the same checks as a new upload (see "Links"):
**IMPORTANT:** When making changes to a newsletter that has already been uploaded to Mailchimp, **update the existing campaign** instead of creating a new one. Use `upload-newsletter` with `--campaign-id`, which runs the same hosting-link and fresh-comparables checks as a new upload:

```bash
upload-newsletter editions/your-newsletter.html \
Expand All @@ -228,7 +242,8 @@ upload-newsletter editions/your-newsletter.html \
--preview "Same preview text as the hidden preheader"
```

Don't PUT the HTML to Mailchimp's `/content` endpoint yourself; that skips the checks.
Do not PUT the HTML to Mailchimp's `/content` endpoint yourself; that skips the checks.


To delete an accidentally created duplicate campaign:
```python
Expand All @@ -245,7 +260,7 @@ requests.delete(url, auth=("anystring", API_KEY))
### Development

```bash
# Run tests (93% coverage)
# Run tests (with coverage: pytest --cov=newsletter_uploader)
pytest -v

# Format and lint
Expand All @@ -255,7 +270,7 @@ ruff check src/ tests/

The package is structured with:
- `src/newsletter_uploader/` - Core package modules
- `tests/` - Comprehensive test suite
- `tests/` - pytest suite
- CI runs on GitHub Actions for Python 3.8-3.11

## Claude Code Automations
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,6 @@ ruff check src/ tests/
- `audience.py` - Audience targeting logic
- `uploader.py` - Newsletter uploader
- `cli.py` - Command-line interface
- `tests/` - Test suite (93% coverage)
- `tests/` - pytest suite
- `editions/` - Newsletter HTML files
- `.env` - Contains `MAILCHIMP_API_KEY` (gitignored)
41 changes: 41 additions & 0 deletions scripts/select_changed_editions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
"""Select changed editions without decoding Git's NUL-delimited filenames."""

import os
import re
import sys
from pathlib import Path


def main():
"""Reject misplaced social posts and write selected names as NUL-delimited bytes."""
names = [name for name in Path(sys.argv[1]).read_bytes().split(b"\0") if name]
misplaced = [
name
for name in names
if re.search(rb"(^|/)[^/]*social[^/]*\.md$", name, re.IGNORECASE)
and not name.startswith(b"editions/")
]
if misplaced:
print(
"::error::social posts go in editions/ as <date>-<audience>-social.md, "
"so their comparables can be checked:"
)
for name in misplaced:
print(repr(os.fsdecode(name)))
return 1

selected = [
name
for name in names
if name.lower().startswith(b"editions/")
and name.lower().endswith((b".html", b".htm", b".md"))
]
Path(sys.argv[2]).write_bytes(b"".join(name + b"\0" for name in selected))
print("Changed editions:")
for name in selected:
print(repr(os.fsdecode(name)))
return 0


if __name__ == "__main__":
sys.exit(main())
19 changes: 18 additions & 1 deletion src/newsletter_uploader/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

import click

from . import comparables
from .audience import AudienceType
from .mailchimp_client import MailchimpClient
from .uploader import NewsletterUploader
Expand Down Expand Up @@ -79,6 +80,22 @@ def main(html_file, audience, subject, preview, title, api_key, list_id, campaig
)
sys.exit(1)

# Fresh comparables before a push (Max, 2026-10-21): every analysis an
# edition links needs a recent comparables search in the scorecard,
# judged on the later of the edition's date and today
# (newsletter_uploader/comparables.py). An undated file is refused.
due, problems = comparables.run_check(
[html_file], today=comparables.today_utc(), require_dates=True
)
if problems:
click.echo("❌ Comparables are not fresh for this edition:", err=True)
for problem in problems:
click.echo(f" {problem}", err=True)
click.echo(comparables.HOW_TO_FIX, err=True)
sys.exit(1)
if due:
click.echo("✓ Comparables are fresh for every analysis this edition links")

# Create client and uploader
if campaign_id:
click.echo("📧 Updating Mailchimp draft campaign")
Expand All @@ -97,7 +114,7 @@ def main(html_file, audience, subject, preview, title, api_key, list_id, campaig

# Upload newsletter
click.echo("Reading HTML file...")
html_content = html_file.read_text()
html_content = html_file.read_text(encoding="utf-8")
click.echo(f"✓ Read {len(html_content)} characters")

if campaign_id:
Expand Down
Loading
Loading