[MaxGhenis]

Max, 2026-10-29: "we need a rule to never share vercel links publicly - should always be on policyengine.org, esp in newsletters".

The 29 October US edition (#35) went out to 762 people with seven ballot-measures-2026.vercel.app links. The policyengine.org route (PolicyEngine/policyengine-app-v2#1415) wasn't live, and the drafting session swapped the links to Vercel to make them work. Earlier editions did the same thing:

• 2025-11-25-uk.html linked a git-dev preview deployment;
• 2026-03-18-us.html linked policyengine.github.io/tanf-calculator;
• 2026-04-01-us.html linked a Vercel app.

What changes

• src/newsletter_uploader/hosting_links.py finds hosting addresses in an edition:

  • *.vercel.app, vercel.com, *.netlify.app, *.pages.dev, *.modal.run, *.run.app, *.appspot.com, and the GitHub Pages sites of PolicyEngine and Axiom (policyengine.github.io, theaxiomfoundation.github.io);
  • in hrefs, image srcs, link text, alt text, and URLs inside another URL's query string;
  • after it decodes HTML entities and percent-escapes, up to three rounds deep.

  It reads each host the way a browser or a reader does:

  • Tabs and newlines drop out inside URL attributes (href, src, srcset, action and the like), as browsers drop them from a URL. In alt or title text a line break stays a break between words, so "We left Modal.⏎Run it" passes.
  • Inline tags such as <wbr>, <span> or <b> don't split the host a reader sees: x.vercel<wbr>.app counts.
  • NFKC folds full-width letters; the dots 。, ． and ｡ become .; the characters a hostname ignores (UTS #46 and stringprep B.1: soft hyphen, zero-width spaces, variation selectors and the rest) drop out.
  • Underscores around a host read as Markdown emphasis: _policyengine.github.io/tanf_ counts.

  Labels may be Unicode, so a longer host that only starts like a hosting one, such as vercel.app.例子.org or vercel.app。example.org, passes. Other people's github.io sites (for example fhoces.github.io) pass. A reviewed exception goes in ALLOWED_HOSTS.

• upload-newsletter refuses such HTML, subject lines, preview text and campaign titles before it calls Mailchimp. The check runs for new campaigns and for --campaign-id updates, so an empty campaign never gets created.

• lint-newsletters.sh fails any edition dated 2026-10-30 or later that links one, and adds a GitHub Actions error annotation on the line. It checks HTML editions, Markdown editions and the root social-posts-*.md files. Older editions went out as they are, so the lint skips them. If the checker crashes, the lint fails rather than passing the file.

• CLAUDE.md gets a "Links" section with the rule. Its "Updating existing drafts" section now uses upload-newsletter --campaign-id, which runs the check, in place of a raw PUT to Mailchimp's /content endpoint, which skipped it.

The open PRs' editions are all dated before 30 October, so none of them starts failing.

Invariants (tested)

• Completeness: the check reports any host whose last labels are a hosting domain, wherever it sits in running text or markup. It still reports it when the URL sits percent-encoded inside another URL or appears entirely as HTML entities (Hypothesis: test_property_any_hosting_host_in_any_text_is_caught, test_property_a_host_in_running_text_is_caught, test_property_encoding_never_hides_a_host).
• Folding never hides a host: tabs, newlines, 
, 	 or %0A anywhere in a URL attribute, and any of the dots ., 。, ．, ｡, . or %2E, still give the hosting host (test_property_whitespace_and_unicode_dots_never_hide_a_host). A host after a line break in any attribute is caught (test_property_a_host_after_a_line_break_in_any_attribute_is_caught).
• Soundness: the check never reports a host that is_hosting_host rejects (test_property_other_hosts_are_never_flagged, test_property_a_longer_host_that_starts_like_a_hosting_one_passes). Lookalikes such as notvercel.app, myvercel.com, vercel.app.example.com, vercel.app.例子.org and foo_vercel.app pass, and only URL attributes drop line breaks (test_only_url_attributes_drop_line_breaks).
• Location: the reported line is the line the link sits on (test_property_reported_line_is_the_line_the_link_sits_on). A host inside an attribute value that spans lines reports the line where the value starts; every line after it keeps its number.
• Linear time: 20,000 hits on one line take well under the test's 5 s limit, and so do 900 KB of unclosed inline tags (test_many_hits_on_one_line_stay_fast, test_unclosed_inline_tags_stay_fast).
• No partial upload: when the check fails on the HTML, subject line, preview text or title, the Mailchimp client receives no call at all (mock_calls == []), for both upload and update.
• Real editions: the check finds every ballot-measures-2026.vercel.app occurrence in the sent 29 October edition, plus the hosting links in the three older editions above. On all 35 edition and social-post files in the repo, it gives the same 12 hits as the first version (4891a55).

Checks run here

• pytest: 178 passed on Python 3.11 and on Python 3.8, the low end of the CI matrix.
• black --check and ruff check are clean.
• lint-newsletters.sh passes on every file in the repo.
• Mutation checks, across rounds 1 to 3 (eleven of them run by the round-3 reviewer): each of these makes at least one test fail:
  • removing the uploader call;
  • dropping the decoding, or decoding only once;
  • matching a domain suffix without its leading dot;
  • moving the lint cutoff a day earlier;
  • skipping the URL-attribute join, joining every attribute, or dropping the guard on runaway values;
  • skipping the folding, the 。 mapping or the ignored-character removal, or leaving out its supplementary-plane ranges;
  • dropping either host pattern, the nested-match sweep, the inline-tag view or its tail replacement;
  • keeping underscores, or matching attribute names case-sensitively;
  • skipping the subject, preview or title check.
• The check takes 0.09 s over all 35 files.

Known limits

This package and TheAxiomFoundation/newsletters#15 share the same matching code, so neither backs up the other on these:

• The check doesn't join a host split by <nobr>, <o:p>, an HTML comment or a tag that spans lines.
• It misses a host glued to a double-hyphen dash, as in x.vercel.app--now.
• An attribute value with an unbalanced quote can swallow the next href, and the check then leaves that href unjoined. It also leaves any value containing < unjoined.
• A < inside an inline tag's quoted attribute ends the tag early, so the check misses x.vercel<span title="a<b">.app. On broken markup such as <a Run</b>.app it can flag run.app.
• Inline elements styled display:block or inline-block with no whitespace between them can glue words a reader sees apart, such as "We left Modal." and "Run it". Reading through inline tags adds no hit on any file in the repo.

Review

• Round 1 (GPT-6.1 Sol, on the Axiom port at e98e989) asked for three changes, which 88c9d3c makes: check the subject and preview text; read tabs and newlines inside a URL as a browser does; fold Unicode dots and hostnames before comparing domains.
• Round 2 (Opus 5.5, at 88c9d3c) requested changes. Its one P2: stripping whitespace from every attribute value hid hosts in alt text and invented one ("Modal.⏎Run"). 91f450b fixes it and all ten P3s.
• Round 3 (Opus 5.5, at 91f450b) approved with only P3s left. 8d6470b fixes R3-1: the inline-tag pattern ran in quadratic time on unclosed tags (300 KB took 7.5 s; now 0.05 s). The rest are under "Known limits".
• Round 3b (Opus 5.5, at 8d6470b): approved. The change alters no hit on any file in the repo, and the 300 KB flood drops from about 7.5 s to 0.05 s.

axiom: n/a: newsletter tooling

🤖 Generated with Claude Code