diff --git a/CLAUDE.md b/CLAUDE.md index 8e2d336..6458ee4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,6 +13,8 @@ Open source newsletter sender: Cloudflare Worker + D1 + R2, Amazon SES for deliv - Installation-specific values (organization, lists, senders, domain, D1 id) belong in `config/*.json`, never in `src/` or `cloudflare.config.ts`. - Schema changes go in a new file under `migrations/`, never by editing an applied one. - Pages, logs, stats, MCP tool output and the audit log never show an address or a name. +- An assistant adds people only to lists it made (`src/lists.ts`). A newsletter from the config takes people only from forms and imports. +- Opens and clicks keep only a machine-or-not flag (`src/tracking.ts`), never the IP address or user agent it was judged by. ## Code diff --git a/README.md b/README.md index 3c4f1e7..d8598cb 100644 --- a/README.md +++ b/README.md @@ -8,12 +8,13 @@ You keep the list, the templates and the numbers. Amazon SES only delivers the m - **Sends** an HTML issue to a list, one mail per recipient, with the merge tags `*|FNAME|*`, `*|UNSUB|*` and `*|ARCHIVE|*`. A send that stops halfway resumes without mailing anyone twice. - **One-click unsubscribe** ([RFC 8058](https://www.rfc-editor.org/rfc/rfc8058)): the `List-Unsubscribe` headers that Gmail, Yahoo and Apple Mail act on, plus the footer link. -- **Counts** clicks per link, and opens through an optional pixel per list, which feeds a resend to those who did not open. +- **Counts** clicks per link, and opens through an optional pixel per list, which feeds a resend to those who did not open. Clicks by link scanners at mail servers and opens by Apple Mail's privacy proxy are counted apart from people's, so a B2B list does not look busier than it is. +- **Numbers that matter:** open, click, click-to-open, unsubscribe, bounce and complaint rates per send; sending health over 30 days against the limits Amazon SES and Gmail hold senders to; and each list's growth by month. - **Bounces and spam complaints** arrive from SES through SNS; each message's signature is checked, and the address is suppressed. - **Imports** a Mailchimp audience export: subscribers with their opt-in time, unsubscribed and cleaned addresses as suppressions. - **Sign-ups** from your own sites: a browser form, a server route with a key, or a Cloudflare Worker over a service binding ([docs/signup.md](docs/signup.md)). -- **A dashboard** for admins with lists, sends, rates, clicks per link and the audit log, and no personal data ([docs/dashboard.md](docs/dashboard.md)). -- **An MCP server** for assistants (Claude, ChatGPT, Grok, Cursor, Codex …) with OAuth 2.1. An assistant can read stats, write drafts and send tests to the team; it can only *ask* to send to a list, and a person approves each send ([docs/mcp.md](docs/mcp.md)). +- **A dashboard** for the team with sending health, lists and their growth, sends, rates, clicks per link and the audit log, and no personal data ([docs/dashboard.md](docs/dashboard.md)). +- **An MCP server** for everyone at the company, through their own assistant (Claude, ChatGPT, Grok, Cursor, Codex …), with OAuth 2.1. An assistant reads the numbers, writes drafts, sends tests to the person using it, and makes lists for one mailing, such as an event's guests. It can only *ask* to send to a list: an admin gets the request by mail and approves each send ([docs/mcp.md](docs/mcp.md)). - **A web archive** page per issue, without names or tracking. ## Architecture @@ -36,7 +37,7 @@ issue HTML ──► CLI ──/admin──► Worker ──SendEmail──► A 1. `send … --dry-run` counts the audience (subscribers of the list, minus suppressed addresses), lists the links it will track, and writes a sample mail to `.dev/`. 2. A live send needs `--confirm `, the number the dry run printed. Without it nothing goes out. 3. Creating a send writes one delivery row per recipient, each with a random token. Creating it again with the same content does nothing; other content under the same key is refused. -4. The CLI asks the Worker for batches of up to 14 mails (SES's default rate), one batch a second. Each row is claimed before SES is called and settled after, so a stopped send resumes where it stopped (`resume `). When SES's answer is lost or SES answers with a server error, the row becomes `unknown` and is not retried on its own ("rather none than two"); `retry-unknown` sends those again. An address SES refuses is `failed` (`retry-failed` puts those back in line). An account or settings error (access denied, sending paused, an unverified domain) stops the send with the rest pending: fix the cause, then `resume`. After the last batch the CLI says if any rows are still sending, unknown or failed, and what to run. +4. The CLI asks the Worker for batches of up to 14 mails (SES's default rate), one batch a second. Each row is claimed before SES is called and settled as soon as SES answers, so a stopped send resumes where it stopped (`resume `). When SES's answer is lost or SES answers with a server error, the row becomes `unknown` and is not retried on its own ("rather none than two"); `retry-unknown` sends those again. An address SES refuses is `failed` (`retry-failed` puts those back in line). An account or settings error (access denied, sending paused, an unverified domain or the SES sandbox, the day's quota spent, or SES refusing every mail of a batch the same way) stops the send with the rest pending: fix the cause, then `resume`. A row whose answer was lost turns `sent` by itself when SES's delivery event for it arrives. After the last batch the CLI says if any rows are still sending, unknown or failed, and what to run. 5. Unsubscribes, bounces and complaints are checked again at send time. ## Getting started @@ -74,7 +75,7 @@ Images next to the issue HTML (`src="hero.jpg"`) are uploaded to R2 and served f 1. Copy `config/example.json` to `config/.json` and fill it in: the `deploy` section (public URL, AWS region, D1, R2), your organization, lists, forms and admins. 2. Follow [docs/setup.md](docs/setup.md) for SES, DNS, SNS, IAM, D1, R2, Cloudflare Access and the secrets. -3. Deploy with `UTSKICK_CONFIG=config/.json npx cf deploy`. Without `UTSKICK_CONFIG` a deploy stops, so nobody ships mail whose links point at someone else's Worker. +3. Apply the migrations first (`cf d1 migrations apply `), then deploy with `UTSKICK_CONFIG=config/.json npx cf deploy`. Code that ships before its migration answers readers' clicks with errors. Without `UTSKICK_CONFIG` a deploy stops, so nobody ships mail whose links point at someone else's Worker. More: [docs/signup.md](docs/signup.md) (forms), [docs/dashboard.md](docs/dashboard.md), [docs/mcp.md](docs/mcp.md) (assistants), [docs/examples/](docs/examples/) (connecting a Cloudflare Pages site). @@ -85,10 +86,10 @@ More: [docs/signup.md](docs/signup.md) (forms), [docs/dashboard.md](docs/dashboa - **SNS webhook.** Nothing in a message is used before its signature checks out: SignatureVersion 1 or 2, the signing certificate fetched only from `sns..amazonaws.com` with the `SimpleNotificationService-…pem` path, and only the configured topic accepted. That includes a subscription confirmation's `SubscribeURL`. - **Reader tokens.** Each delivery has its own 128-bit random token, used in its unsubscribe, click and open URLs. A token identifies one mail, not a person, and the API never returns one. Like any newsletter footer link, a forwarded mail lets its new reader unsubscribe the original one. - **Unsubscribe.** A GET only shows a button, because mail scanners open links. A POST, the one RFC 8058 clients send, unsubscribes. In a test mail the POST is logged and takes nobody off a list. -- **Bounces.** A hard bounce or a spam complaint suppresses the address on every list. A soft bounce does so when a second mail bounces. -- **Sign-ups.** Browser forms are allowed by exact origin (CORS), server routes by a per-form key, sites on Cloudflare over a service binding. Each list's consent id must match the config, a honeypot field catches form bots, and requests are rate-limited per client (per /64 for IPv6). The answer is the same for a new, known or suppressed address. Without double opt-in, a sign-up never lifts a suppression and never changes an existing contact. -- **Assistants (MCP).** OAuth 2.1 with PKCE S256 only, tokens bound to the `/mcp` resource, rotating refresh tokens, revocation per client, and least-privilege scopes (`stats.read`, `subscribers.read`, `drafts.write`, `test.send`). Sending to a list is never a scope: a person approves each send on a page that shows the list, the subject, the count, the text part, every link and anything hidden; the approval works once, for ten minutes. Every tool call is audited. Tokens, codes and secrets are stored only as hashes. -- **Admins.** Sign-in for the dashboard, the consent page and approvals goes through Cloudflare Access; Utskick checks the Access JWT itself and allows only the addresses in `admins`. The development stand-in works only when both the configured and the requested host are local. +- **Bounces.** A hard bounce or a spam complaint suppresses the address on every list. A full mailbox does so when a second mail bounces for it within 30 days; other soft bounces (a busy server, one slow to trust a new sender) do not. Only the mail's own recipient is suppressed, never other addresses a bounce report names. +- **Sign-ups.** Browser forms are allowed by exact origin (CORS), server routes by a per-form key, sites on Cloudflare over a service binding. Each list's consent id must match the config, a honeypot field catches form bots, and requests are rate-limited per client (per /64 for IPv6). The answer is the same for a new, known or suppressed address. Without double opt-in, a sign-up never lifts a suppression and never changes an existing contact. A site over a service binding has the same rate limit. +- **Assistants (MCP).** OAuth 2.1 with PKCE S256 only, tokens bound to the `/mcp` resource, rotating refresh tokens, revocation per client and per person, and least-privilege scopes (`stats.read`, `subscribers.read`, `drafts.write`, `test.send`, `lists.write`). Sending to a list is never a scope: an admin approves each send on a page that shows the list, the subject, the count, the text part, every link and anything hidden; the approval works once, within 24 hours by default. With `approval.notifyFrom` set, the admins get the approval link by mail and the assistant never holds it. An assistant can add addresses to a list it made, and no tool returns one. Every tool call is audited. Tokens, codes and secrets are stored only as hashes. +- **Members and admins.** Sign-in for the dashboard, the consent page and approvals goes through Cloudflare Access; Utskick checks the Access JWT itself. `members` (addresses, or `@domain` for a whole company) may connect an assistant and read the dashboard; only `admins` approve sends. Every call an assistant makes checks that its person is still a member. The development stand-in works only when both the configured and the requested host are local. - **Pages.** Every reader and admin page is server-rendered without scripts. The dashboard and approval pages carry a strict Content-Security-Policy and cannot be framed; the web archive is served in a CSP sandbox. - **Limits on abuse.** At most one open per mail, and one click per mail and link, are logged an hour; request bodies are read only up to a small limit. - **Local stand-ins stay local.** `SES_ENDPOINT` (the SES mock) is ignored unless the public URL is a local host, and the CLI refuses to send the admin token over plain HTTP to anything but localhost. @@ -98,7 +99,8 @@ Report a vulnerability as described in [SECURITY.md](SECURITY.md). ## Privacy - Leaving a list deletes the address from it, and the contact once it is on no list. A SHA-256 hash of the address stays on the suppression list, so an old import cannot sign the person up again; it is a pseudonym, not anonymous. -- Opens and clicks are stored as "delivery N, link M, at time T", without IP address or user agent. Lists without `opens` get no pixel. In the EU, an open pixel may need consent (EDPB Guidelines 2/2023); say so where people sign up, or turn it off. +- Opens and clicks are stored as "delivery N, link M, at time T, machine or not", without IP address or user agent. Whether a machine made it is decided from the network and the user agent when the request comes in, and only the yes or no is kept. Lists without `opens` get no pixel. +- A list a colleague makes through an assistant holds addresses that colleague typed in, for one purpose the admin sees before any send. Deleting it deletes every address on no other list. In the EU, an open pixel may need consent (EDPB Guidelines 2/2023); say so where people sign up, or turn it off. - Pages, logs, stats, the dashboard and the audit log show no address or name. - D1 and R2 can be created in Cloudflare's EU jurisdiction, and SES runs in the AWS region you choose. diff --git a/cli/utskick.ts b/cli/utskick.ts index 678f4dd..7dfee9f 100644 --- a/cli/utskick.ts +++ b/cli/utskick.ts @@ -115,8 +115,17 @@ set and the sender's domain in SES, then run \`resume ${key}\`.`); } } +// What the dry run found wrong with the mail. Older Workers send no warnings. One line each, whatever the text holds. +function printWarnings(r: any) { + if (!Array.isArray(r.warnings)) return; + for (const w of r.warnings) console.log(`warning: ${String(w).replace(/[\s\x00-\x1f\x7f]+/g, ' ').trim()}`); +} + function printDryRun(r: any) { console.log(`${r.recipients} recipients${r.excluded ? `, ${r.excluded} subscribers left out (unsubscribed, bounced or complained)` : ''}`); + // Older Workers do not send the size either. + if (Number.isFinite(r.bytes)) console.log(`size: ${(r.bytes / 1024).toFixed(1)} KB (the sample's HTML)`); + printWarnings(r); console.log(`subject: ${r.sample.subject}\nfrom: ${r.sample.from}\nreply-to: ${r.sample.replyTo}`); console.log(`tracked links (${r.links.length}):`); r.links.forEach((l: string, n: number) => console.log(` ${n} ${l}`)); @@ -156,6 +165,8 @@ async function send(kind: 'live' | 'resend') { printDryRun(dry); fail(`\nNothing sent. This goes to ${dry.recipients} people on ${list()}. To send, run again with --confirm ${dry.recipients}`); } + // The count matches, so this goes out. Show once more what the dry run warned about. + printWarnings(dry); const r = await api('POST', '/admin/sends', body); console.log(`created ${key}: ${r.recipients} recipients`); await deliver(key); diff --git a/config/example.json b/config/example.json index b5c9dd4..a0fb018 100644 --- a/config/example.json +++ b/config/example.json @@ -34,9 +34,21 @@ "admins": [ "you@example.com" ], + "members": [ + "@example.com" + ], "testAllowlist": [ "you@example.com" ], + "adhoc": { + "from": "Example Org ", + "opens": false, + "fname": "there" + }, + "approval": { + "hours": 24, + "notifyFrom": "Utskick " + }, "access": { "teamDomain": "your-team.cloudflareaccess.com", "audience": "the-AUD-tag-of-your-Access-application" diff --git a/config/sourceful.json b/config/sourceful.json index 3e272ef..7e92533 100644 --- a/config/sourceful.json +++ b/config/sourceful.json @@ -41,9 +41,22 @@ "admins": [ "fredrik@sourceful-labs.com" ], + "members": [ + "@sourceful-labs.com", + "@sourceful.energy" + ], "testAllowlist": [ "fredrik@sourceful-labs.com" ], + "adhoc": { + "from": "Sourceful Energy ", + "opens": false, + "fname": "there" + }, + "approval": { + "hours": 24, + "notifyFrom": "Utskick " + }, "access": { "teamDomain": "sourceful.cloudflareaccess.com", "audience": "83aeb3bf704fc1e26213a621c0c2ed12c788e21a502f2195fb8d3d5aa906d56f" diff --git a/docs/dashboard.md b/docs/dashboard.md index 07c0257..25dc5b1 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -1,6 +1,6 @@ # Dashboard -`${PUBLIC_URL}/dashboard` shows an admin how the newsletter does: the lists, the sends, the sends that wait for approval, and the latest audit entries. It only reads. It has no form and no button, and no page changes anything. +`${PUBLIC_URL}/dashboard` shows every member how the mail does: sending health, the lists and how they grow, the sends, the sends that wait for approval, and the latest audit entries. It only reads. It has no form and no button, and no page changes anything. The code is in `src/dashboard.ts`, the tests in `test/dashboard.test.ts`. @@ -8,40 +8,54 @@ The code is in `src/dashboard.ts`, the tests in `test/dashboard.test.ts`. ### `/dashboard` -- **Lists**, one row for each list in the config: subscribers, addresses suppressed for that list (those who left it), new subscriptions in the last 30 days, and where they came from (`subscriptions.source`, such as `mailchimp-import` or `ftw-web`). "New" goes by `consented_at`, the time the person signed up, so an import of old subscribers adds none. Under the table: how many addresses are suppressed on every list (hard bounces, spam complaints, addresses Mailchimp cleaned). -- **Sends**, the latest 25, newest first: key, kind, created, recipients, sent, delivered, unique opens and unique clicks with their share of mails sent, hard bounces, complaints and unsubscribes. Test sends are hidden; `/dashboard?tests=1` shows them. A list without the open pixel (`opens: false` in the config) shows "no pixel" instead of an open rate. -- **Waiting for approval**: the `request_send` approvals that are pending and not yet expired: the request id, the key, the list, the subject, the recipients when the assistant asked, which assistant asked, which assistants made and last changed the draft it copied, when, and when the request ends. Drafts are shared by every assistant, so the column says when the last change came from another assistant than the one asking. The page cannot link to the approval page. Utskick keeps only the SHA-256 of an approval link ([mcp.md](mcp.md)), and the link itself went to the assistant that asked, so an admin gets it from there. +- **Sending health, last 30 days**: mails sent to lists and delivered, the hard bounce rate and the complaint rate, each with a level (ok, watch, act now), unsubscribes, and rows that failed or are unknown. Amazon SES reviews an account at 5% hard bounces or 0.1% complaints and may pause it at 10% or 0.5%; the levels warn at 2% and 0.05%. Gmail does not report spam complaints to SES; its spam rate is only in Google Postmaster Tools. If mail went out and no delivery event came back, the page says so: SES's events do not reach Utskick, and bounces and complaints then go unseen. `deliverability` gives the same to an assistant. +- **Lists**, one row for each list in the config: subscribers, addresses suppressed for that list (those who left it), new subscriptions in the last 30 days, and where they came from (`subscriptions.source`, such as `mailchimp-import` or `ftw-web`). "New" goes by `consented_at`, the time the person signed up, so an import of old subscribers adds none. Under the table: how many addresses are suppressed on every list (hard bounces, spam complaints, addresses Mailchimp cleaned). A list a colleague made through an assistant says so; a deleted one is not shown. +- **Growth, by month**: for each newsletter, how many joined and how many left in each of the last 12 months, from the `list_changes` table (counts per day and list, nothing about anyone). A sign-up counts in the month the person signed up, so an import fills in the months before it, but only with those still subscribed. `list_growth` gives the same to an assistant. +- **Sends**, the latest 25, newest first: key, kind, created, recipients, sent, delivered, unique opens and unique clicks with their share of mails sent, hard bounces, complaints and unsubscribes. Opens count any open, as most tools do; clicks count people only. Test sends are hidden; `/dashboard?tests=1` shows them. A list without the open pixel (`opens: false` in the config) shows "no pixel" instead of an open rate. +- **Waiting for approval**: the `request_send` approvals that are pending and not yet expired: the request id, the key, the list, the subject, the recipients when the assistant asked, which assistant asked, which assistants made and last changed the draft it copied, when, and when the request ends. Drafts are shared by every assistant, so the column says when the last change came from another assistant than the one asking. The page cannot link to the approval page. Utskick keeps only the SHA-256 of an approval link ([mcp.md](mcp.md)); the link itself went to the admins by mail, or to the assistant that asked. - **Audit log**: the latest 20 rows of `audit_log`: time, assistant, tool, scope, outcome and the summary of the arguments. What the CLI changed through `/admin` is there too, as `admin-token`, with the route as the tool and counts as the summary. ### `/dashboard/sends/` -- The numbers from `stats()` in `src/send.ts`, the same that `send_stats` and `node cli/utskick.ts stats ` give: recipients, sent, delivered, unique and total opens and clicks, hard and soft bounces, complaints, unsubscribes. -- Clicks per link as bars, in the order the links come in the mail: how many mails had a click on the link, and their share of mails sent. +- The numbers from `stats()` in `src/send.ts`, the same that `send_stats` and `node cli/utskick.ts stats ` give: recipients, sent, delivered, unique opens (and how many of them only machines made), unique clicks by people (and how many more mails only scanners clicked), click-to-open, hard and soft bounces, complaints, unsubscribes. +- Clicks per link as bars, in the order the links come in the mail: how many mails had a click on the link by a person, and their share of mails sent. - Delivery status: how many rows are sent, pending, sending, unknown, skipped or failed, and what each means. `unknown`: SES may or may not have the mail (no answer, an SES server error, or a run that stopped between SES and the database); check, then `node cli/utskick.ts retry-unknown ` sends them again. `failed`: SES refused that address and the mail did not go out; once the cause is fixed (an unverified sender, say), `node cli/utskick.ts retry-failed ` puts them back in line. -- First opens and first clicks over time, as inline SVG: an hour a bar for the 72 hours after the first mail went out, then a day a bar up to day 33. Each mail counts once, at its first open or its first click, so the bars add up to the unique counts. A line under the chart counts any that came after day 33. +- First opens and first clicks by people over time, as inline SVG: an hour a bar for the 72 hours after the first mail went out, then a day a bar up to day 33. Each mail counts once, at its first open or its first click, so the bars add up to the unique counts. A line under the chart counts any that came after day 33. - For a live send, its resends; for a resend, the live send it repeats. Each is a link. +### People and machines + +Mail servers and security gateways (Microsoft Defender, Proofpoint, Mimecast, Gmail) follow every link in a mail to check it, often before anyone has read it. Apple Mail's privacy proxy loads every image, the open pixel too, as soon as a mail arrives. On a list of companies that can be most of the clicks. `src/tracking.ts` marks an open or a click as a machine's when it comes: + +- from a network of mail scanners, clouds or hosting (Microsoft, Google, Amazon, Proofpoint, Mimecast and the like), or for an open, Apple's; +- with a user agent that says it is a program, or none; +- for a click, within 30 seconds of the mail going out, or as the third of a mail's links clicked within 2 seconds. + +Web proxies that carry a company's staff (Prisma Access, Zscaler, Forcepoint and the like) count as people. Someone on a cloud desktop (Windows 365, a VPN on AWS) counts as a machine: the rule cannot tell them from a scanner. + +Only the yes or no is stored. A machine's click never hides a person's later click on the same link, and one mail logs at most 30 clicks an hour. A resend still skips anyone whose mail was opened or clicked by anything, machine or not, so nobody gets an issue twice. + Every page ends with a note: opens overstate reading. Apple Mail Privacy Protection loads every image in a mail, the open pixel too, as soon as the mail arrives, and Gmail fetches images through its own proxy. Clicks, and replies to the list's reply-to address, are the honest signal. Replies go to that inbox; Utskick never sees them. A send key must match `[a-z0-9][a-z0-9-]{1,79}`, as when a send is made. Any other path under `/dashboard` is a 404. ## Signing in -The dashboard signs in an admin the way `/oauth/authorize` and `/approve/` do: `signedInAdmin()` in `src/identity.ts`. In production, Cloudflare Access signs the person in, and Utskick checks the JWT that Access adds (`Cf-Access-Jwt-Assertion`) itself: the signature against the team's keys, the issuer, the audience (`access.audience`) and the expiry. It then looks the address up in `admins` in the config. Under `cf dev`, `DEV_IDENTITY` stands in for Access, as [mcp.md](mcp.md) describes. +The dashboard signs in a member the way `/oauth/authorize` does: `signedInMember()` in `src/identity.ts`. Admins are members too. In production, Cloudflare Access signs the person in, and Utskick checks the JWT that Access adds (`Cf-Access-Jwt-Assertion`) itself: the signature against the team's keys, the issuer, the audience (`access.audience`) and the expiry. It then looks the address up in `members` and `admins` in the config. Under `cf dev`, `DEV_IDENTITY` stands in for Access, as [mcp.md](mcp.md) describes. - Not signed in: 401, the same "Sign-in needed" page as `/oauth/authorize`. No sign-in set up for the installation: 503. -- Signed in, but not an admin: 403. Unlike `/oauth/authorize`, this page does not repeat the address. +- Signed in, but not a member: 403. Unlike `/oauth/authorize`, this page does not repeat the address. - Who comes before what: without sign-in, every path under `/dashboard` answers 401, so nobody can probe which send keys exist. - Only GET and HEAD. Anything else is a 405. ### Paths Cloudflare Access must cover -All of them belong to **one** self-hosted Access application for your Utskick host, with a policy that allows the admins' addresses: +All of them belong to **one** self-hosted Access application for your Utskick host, with a policy that allows everyone in `members`: | Path | For | |---|---| | `dashboard` and `dashboard/*` | this dashboard | -| `oauth/authorize` | an admin connects an assistant | +| `oauth/authorize` | a member connects an assistant | | `approve/*` | an admin approves or declines a send | Add the two dashboard paths to the application that already holds `oauth/authorize` and `approve`. Utskick accepts one audience (`access.audience`), and a second Access application would have an AUD tag of its own, so its sign-ins would get a 401. As with every change in Cloudflare, this needs the owner's go-ahead ([setup.md](setup.md)). diff --git a/docs/mcp.md b/docs/mcp.md index a6d80be..994bfd5 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -1,12 +1,14 @@ # Assistants: MCP and OAuth -Utskick has one MCP endpoint, `${PUBLIC_URL}/mcp`. Claude, ChatGPT, Grok, other assistants, Cursor, Claude Code and Codex can connect to it. An assistant can read stats and list counts, write and preview drafts, send tests to the team's own addresses, and ask for a send to a list. It cannot send to a list. An admin approves each such send on a page of its own. +Utskick has one MCP endpoint, `${PUBLIC_URL}/mcp`. Claude, ChatGPT, Grok, other assistants, Cursor, Claude Code and Codex can connect to it, for everyone in `members`, a whole company if you like. An assistant can read the numbers, write and preview drafts, send tests to the person using it, make a list of people that person names for one mailing, and ask for a send to a list. It cannot send to a list. An admin approves each such send on a page of its own. + +A colleague's usual way, which the server's instructions give every assistant: make a list and add people (or pick a newsletter), write a draft with an unsubscribe link, preview it and fix its warnings, send a test to themselves, ask for the send, and read the numbers once it has gone out. The server follows the MCP authorization spec, revision 2026-07-28, as [srcfl/ftw-cloud](https://github.com/srcfl/ftw-cloud) does. The code is in `src/oauth.ts`, `src/clients.ts`, `src/identity.ts`, `src/mcp.ts`, `src/tools.ts`, `src/drafts.ts` and `src/approvals.ts`; the tables are in `migrations/0002_oauth.sql` and `migrations/0004_auth.sql`. ## Connecting a client -Every client needs only the URL. It finds the rest itself: a request without a token gets a 401 whose `WWW-Authenticate` header points at `/.well-known/oauth-protected-resource/mcp`, which names the authorization server, whose metadata is at `/.well-known/oauth-authorization-server`. The client then sends an admin's browser to `/oauth/authorize`. The admin signs in, ticks what the assistant may do, and the browser returns to the client with a code. +Every client needs only the URL. It finds the rest itself: a request without a token gets a 401 whose `WWW-Authenticate` header points at `/.well-known/oauth-protected-resource/mcp`, which names the authorization server, whose metadata is at `/.well-known/oauth-authorization-server`. The client then sends the person's browser to `/oauth/authorize`. They sign in as a member, tick what the assistant may do, and the browser returns to the client with a code. | Client | How | |---|---| @@ -22,14 +24,15 @@ A client that cannot store tokens across restarts asks the admin to sign in agai ## Scopes -Each tool needs one scope. The consent page lists the scopes the client asked for, all ticked; the admin unticks what the assistant should not have. A client that asks for none is offered all four. +Each tool needs one scope. The consent page lists the scopes the client asked for, all ticked; the member unticks what the assistant should not have. A client that asks for none is offered all five. | Scope | Tools | What it allows | |---|---|---| -| `stats.read` | `sends_list`, `send_stats` | Each send's numbers and clicks per link. Never who got a mail, opened it or clicked. | -| `subscribers.read` | `subscriber_counts` | How many are on each list, how many can be mailed, how many left. Counts only. | +| `stats.read` | `sends_list`, `send_stats`, `deliverability` | The latest sends side by side with their rates, one send's numbers and clicks per link, people and machines apart, and sending health over 30 days. Never who got a mail, opened it or clicked. | +| `subscribers.read` | `subscriber_counts`, `list_growth` | How many are on each list, how many can be mailed, how many left, and each newsletter's growth by month. Counts only. | | `drafts.write` | `drafts_list`, `draft_save`, `draft_preview`, `request_send`, `approval_status` | Write drafts in HTML or Markdown, preview them with the sample name Alex and no real recipient, and ask an admin to approve a send. | -| `test.send` | `test_send` | Send a draft or an earlier issue as a test (`[Test]` in the subject) to the addresses in `testAllowlist` in the config, and nowhere else. At most 20 test sends an hour, for all assistants together, of up to 20 addresses each. | +| `test.send` | `test_send` | Send a draft or an earlier issue as a test (`[Test]` in the subject) to the person who connected the assistant (the default), or to the addresses in `testAllowlist` in the config, and nowhere else. At most 20 test sends an hour, for all assistants together, of up to 20 addresses each. | +| `lists.write` | `list_create`, `list_add`, `list_remove`, `list_delete` | Make a list for one mailing, add up to 500 people a call, take them off, and delete the list when the mailing is done. Addresses go in; only counts come out. Never a newsletter from the config: people join those themselves. | Drafts are shared: every assistant with `drafts.write` can read and change every draft. Each draft records the assistant that made it and the one that changed it last; `drafts_list`, the approval page and the dashboard show both. @@ -44,16 +47,32 @@ Not built, on purpose: 1. The assistant calls `request_send` with a draft id and, if it likes, a send key such as `the-source-23` (the key is part of the web version's address, `/a/`). The default key is the list and today's date. 2. Utskick checks that the draft has an `*|UNSUB|*` link and that the key is free, counts the recipients with a dry run, and stores a copy of the draft's subject and HTML. What is approved is what goes out, whatever happens to the draft afterwards. -3. The assistant gets an approval link, `/approve/`, and gives it to the user. Nothing has been sent. -4. An admin opens the link and signs in. The page shows the list, the subject, the sender, the recipient count now and when the assistant asked, the key, which assistant asked for whose grant, which assistants made and last changed the draft, and the mail itself. Beside the preview it shows what a browser preview hides: the text/plain part as it goes out, every link target as text, what the mail loads, and a warning for parts only Outlook shows (`', '

Leave: *|UNSUB|*

']) { + const r = await create(env, { key: 'k-1', html: bad }); + assert.equal(r.status, 400, bad); + assert.match(r.body.error, /^no_unsubscribe_link/); + } + const t = await create(env, { key: 'k-test', kind: 'test', html: '

Hi

', testTo: [{ email: 'me@example.com' }] }); + assert.equal(t.status, 200); +}); + +test('the dry run warns of merge tags left in the mail and of a mail Gmail would cut', async () => { + const env = await listOf(1); + const r = await create(env, { key: 'k-1', html: html.replace('', `

*|MC:SUBJECT|* *|LNAME|*

${'

padding

'.repeat(9000)}`), dryRun: true }); + assert.equal(r.status, 200); + assert.ok(r.body.bytes > 100 * 1024); + assert.equal(r.body.warnings.length, 2); + assert.match(r.body.warnings[0], /\*\|MC:SUBJECT\|\*, \*\|LNAME\|\*/); + assert.match(r.body.warnings[1], /Gmail cuts/); + assert.deepEqual((await create(env, { key: 'k-2', html, dryRun: true })).body.warnings, []); +}); + +test('when SES refuses every mail of a batch the same way, the send stops with the rows back in line', async () => { + const env = await listOf(5); + await create(env, { key: 'k-1', html }); + ses.fail.push(...Array(5).fill({ status: 400, type: 'BadRequest' })); + const res = await handle(admin('/admin/sends/k-1/batch', { max: 10 }), env); + assert.equal(res.status, 502); + assert.match((await res.json() as { error: string }).error, /^ses_account_error: BadRequest for every mail in a batch/); + assert.deepEqual(await statuses(env), { pending: 5 }); + // Fixed: the send goes on. + assert.deepEqual(await (await handle(admin('/admin/sends/k-1/batch', { max: 10 }), env)).json(), { sent: 5, skipped: 0, failed: 0, retry: 0, pending: 0, unknown: 0, sending: 0 }); +}); + +test('one address SES refuses fails alone, and the rest go out', async () => { + const env = await listOf(4); + await create(env, { key: 'k-1', html }); + ses.fail.push({ status: 400, type: 'MessageRejected' }); + const r = await (await handle(admin('/admin/sends/k-1/batch', { max: 10 }), env)).json() as Record; + assert.deepEqual([r.sent, r.failed], [3, 1]); +}); + +test('the sandbox, an unknown sender and a spent daily quota stop the send at once', async () => { + for (const [type, message] of [['MessageRejected', 'Email address is not verified. The following identities failed the check: p0@example.com'], ['Throttling', 'Daily message quota exceeded.']]) { + const env = await listOf(2); + await create(env, { key: 'k-1', html }); + const original = globalThis.fetch; + globalThis.fetch = (async () => Response.json({ message }, { status: 400, headers: { 'x-amzn-ErrorType': type } })) as typeof fetch; + try { + const res = await handle(admin('/admin/sends/k-1/batch', { max: 10 }), env); + assert.equal(res.status, 502, type); + assert.doesNotMatch(JSON.stringify(await res.json()), /@/, 'SES\'s text, which names the reader, is not passed on'); + } finally { + globalThis.fetch = original; + } + assert.deepEqual(await statuses(env), { pending: 2 }, type); + assert.doesNotMatch(JSON.stringify(await rows(env, 'SELECT error FROM deliveries')), /@/); + } +}); + +test('the Worker\'s own limit on outgoing requests stops the send, with the rows not yet sent back in line', async () => { + const env = await listOf(3); + await create(env, { key: 'k-1', html }); + const original = globalThis.fetch; + globalThis.fetch = (async () => { throw new Error('Too many subrequests.'); }) as typeof fetch; + try { + assert.equal((await handle(admin('/admin/sends/k-1/batch', { max: 10 }), env)).status, 502); + } finally { + globalThis.fetch = original; + } + assert.deepEqual(await statuses(env), { pending: 3 }); +}); + +test('each row is settled as SES answers: a settle that fails leaves that one row, not the batch', async t => { + const env = await listOf(4); + await create(env, { key: 'k-1', html }); + const [target] = (await rows(env, 'SELECT id FROM deliveries ORDER BY id DESC LIMIT 1')).map(r => r.id); + const prepare = env.DB.prepare; + env.DB.prepare = ((sql: string) => { + const statement = prepare(sql); + if (!sql.includes("status = 'sent', message_id")) return statement; + return { ...statement, bind: (...p: (string | number | null)[]) => { + const bound = statement.bind(...p); + return { ...bound, run: async () => { if (p[2] === target) throw new Error('D1_ERROR: Network connection lost.'); return bound.run(); } }; + } }; + }) as typeof env.DB.prepare; + const logged = t.mock.method(console, 'error', () => {}); + const r = await runBatch(env, 'k-1', 10); + assert.equal(r.sent, 4); + assert.deepEqual(await statuses(env), { sending: 1, sent: 3 }); + assert.equal(logged.mock.callCount(), 1); + assert.doesNotMatch(String(logged.mock.calls[0].arguments[0]), /@/); +}); + +test('a second resend of an issue reaches nobody the first resend reached', async () => { + const env = await listOf(3); + await create(env, { key: 'k-1', html }); + await runBatch(env, 'k-1', 10, new Date('2026-11-01T10:00:00Z')); + const resend = (key: string) => create(env, { key, kind: 'resend', parent: 'k-1', html, subject: 'Again', now: '2026-11-08T10:00:00Z' }); + assert.equal((await resend('k-1-resend')).body.recipients, 3); + assert.equal((await resend('k-1-resend-2')).body.recipients, 0); +}); + +test('a name with $ patterns in it lands in the greeting as it is', () => { + for (const name of ['Al$`', "Bob$'", 'Cy$&']) { + const out = render('

Hi *|FNAME|*!

u', { firstName: name, fallback: 'there', unsubscribeUrl: 'U', archiveUrl: 'A' }); + assert.equal(out, `

Hi ${escapeHtml(name)}!

u`); + } +}); + +test('an address with control or invisible characters, or a trailing dot, is not an address', () => { + for (const bad of ['vic\u0001tim@example.com', 'vic​tim@example.com', 'victim@example.com.', 'a\u007f@example.com']) assert.equal(normalizeEmail(bad), null, JSON.stringify(bad)); + assert.equal(normalizeEmail(' Anna@Example.COM '), 'anna@example.com'); +}); + +test('a short admin token works only on a local Worker', async () => { + const env = testEnv(); + assert.equal((await handle(admin('/admin/sends/x'), env)).status, 404, 'news.test is local, so the test token is fine'); + const live = { ...env, PUBLIC_URL: 'https://news.example.com' }; + assert.equal((await handle(admin('/admin/sends/x'), live)).status, 503); + const long = { ...live, ADMIN_TOKEN: 'x'.repeat(40) }; + const res = await handle(new Request('https://news.example.com/admin/sends/x', { headers: { Authorization: `Bearer ${'x'.repeat(40)}` } }), long); + assert.equal(res.status, 404); +}); + +// review: content SES will never take must not stop the send, and the sends after it, for ever. +test('a batch SES refuses the same way twice stays failed, and the send goes on', async () => { + const env = await listOf(4); + await create(env, { key: 'k-1', html }); + ses.fail.push(...Array(4).fill({ status: 400, type: 'MessageRejected' })); + assert.equal((await handle(admin('/admin/sends/k-1/batch', { max: 10 }), env)).status, 502); + assert.deepEqual(await statuses(env), { pending: 4 }); + ses.fail.push(...Array(4).fill({ status: 400, type: 'MessageRejected' })); + const second = await handle(admin('/admin/sends/k-1/batch', { max: 10 }), env); + assert.equal(second.status, 200); + assert.deepEqual(await statuses(env), { failed: 4 }); +}); + +test('one token cannot fill the events table, whatever the clicks turn into', async () => { + const env = await listOf(1); + await create(env, { key: 'k-1', html: html.replace('', 'yz') }); + await runBatch(env, 'k-1', 10); + const [d] = await rows(env, 'SELECT token FROM deliveries'); + const ua = { 'User-Agent': 'Mozilla/5.0 (Macintosh) Safari/605.1.15' }; + for (let i = 0; i < 60; i++) await handle(new Request(`https://news.test/c/${d.token}/${i % 3}`, { headers: ua }), env); + assert.ok((await rows(env, 'SELECT COUNT(*) AS n FROM events'))[0].n as number <= 30); +}); diff --git a/test/send.test.ts b/test/send.test.ts index 9bb4b33..4931b79 100644 --- a/test/send.test.ts +++ b/test/send.test.ts @@ -124,7 +124,7 @@ test('FTW mail counts clicks but has no open pixel', async () => { ses = fakeSes(); const env = testEnv(); await handle(admin('/admin/import', { list: 'ftw-news', source: 'ftw-web', consent: 'news-2026-10-30', contacts: [{ email: 'a@example.com' }], suppressions: [] }), env); - await create(env, { key: 'ftw-news-1', list: 'ftw-news', kind: 'live', subject: 'FTW', html: 'FTW' }); + await create(env, { key: 'ftw-news-1', list: 'ftw-news', kind: 'live', subject: 'FTW', html: 'FTWu' }); await batch(env, 'ftw-news-1'); const html = ses.outbox[0].Content.Simple.Body.Html.Data; assert.match(html, /href="https:\/\/news\.test\/c\/[0-9a-f]{32}\/0"/); @@ -268,7 +268,7 @@ test('List-Id leaves out a list name that is not plain ASCII', async () => { const listId = (key: string) => ses.outbox.find(m => m.EmailTags?.[0].Value === key)!.Content.Simple.Headers.find(h => h.Name === 'List-Id')!.Value; for (const list of ['ftw-news', 'ftw-box']) { await handle(admin('/admin/import', { list, source: 's', consent: 'c', contacts: [{ email: 'a@example.com' }], suppressions: [] }), env); - await create(env, { key: list, list, kind: 'live', subject: 'S', html: '

x

' }); + await create(env, { key: list, list, kind: 'live', subject: 'S', html: '

x

u' }); await batch(env, list); } assert.equal(listId('ftw-news'), 'FTW news '); @@ -280,10 +280,10 @@ test('a resend is refused for a list without the open pixel', async () => { ses = fakeSes(); const env = testEnv(); await handle(admin('/admin/import', { list: 'ftw-news', source: 's', consent: 'c', contacts: [{ email: 'a@example.com' }], suppressions: [] }), env); - await create(env, { key: 'ftw-news-1', list: 'ftw-news', kind: 'live', subject: 'FTW', html: '

x

' }); + await create(env, { key: 'ftw-news-1', list: 'ftw-news', kind: 'live', subject: 'FTW', html: '

x

u' }); await runBatch(env, 'ftw-news-1', 10, new Date('2026-11-01T10:00:00Z')); for (const dryRun of [true, false]) { - const { status, body } = await create(env, { key: 'ftw-news-1-resend', list: 'ftw-news', kind: 'resend', parent: 'ftw-news-1', subject: 'Again', html: '

x

', dryRun, now: '2026-11-08T10:00:00Z' }); + const { status, body } = await create(env, { key: 'ftw-news-1-resend', list: 'ftw-news', kind: 'resend', parent: 'ftw-news-1', subject: 'Again', html: '

x

u', dryRun, now: '2026-11-08T10:00:00Z' }); assert.equal(status, 400); assert.match(body.error, /^resend_needs_opens/); } diff --git a/test/ses.test.ts b/test/ses.test.ts index b7884e4..7e20d5c 100644 --- a/test/ses.test.ts +++ b/test/ses.test.ts @@ -11,7 +11,10 @@ const mail = { from: 'a@news.test', to: 'b@example.com', subject: 'S', html: '

{ const error = (status: number, headers: Record, body: unknown) => SesError.from(new Response(JSON.stringify(body), { status, headers })); const cases: [SesError, string, string][] = [ - [await error(400, { 'x-amzn-ErrorType': 'MessageRejected:http://internal.amazon.com/coral/com.amazonaws.sesv2/' }, { message: 'Email address is not verified: b@example.com' }), 'MessageRejected 400', 'recipient'], + [await error(400, { 'x-amzn-ErrorType': 'MessageRejected:http://internal.amazon.com/coral/com.amazonaws.sesv2/' }, { message: 'Content rejected for b@example.com' }), 'MessageRejected 400', 'recipient'], + // The sandbox, or a sender SES does not know: every other mail would be refused too. + [await error(400, { 'x-amzn-ErrorType': 'MessageRejected:http://internal.amazon.com/coral/com.amazonaws.sesv2/' }, { message: 'Email address is not verified. The following identities failed the check: b@example.com' }), 'MessageRejected 400 unverified', 'account'], + [await error(400, {}, { code: 'ThrottlingException', message: 'Daily message quota exceeded.' }), 'Throttling 400 quota', 'account'], [await error(400, {}, { __type: 'com.amazonaws.sesv2#BadRequestException', message: 'Illegal address' }), 'BadRequest 400', 'recipient'], [await error(429, {}, { message: 'Maximum sending rate exceeded.' }), 'HTTP 429', 'throttled'], [await error(400, {}, { code: 'ThrottlingException' }), 'Throttling 400', 'throttled'], diff --git a/test/stats.test.ts b/test/stats.test.ts new file mode 100644 index 0000000..f745489 --- /dev/null +++ b/test/stats.test.ts @@ -0,0 +1,112 @@ +import { test, afterEach } from 'node:test'; +import assert from 'node:assert/strict'; +import { handle } from '../src/app.ts'; +import { subscribe } from '../src/contacts.ts'; +import { stats } from '../src/send.ts'; +import { deliverability, listGrowth, sendSummaries } from '../src/stats.ts'; +import { iso } from '../src/util.ts'; +import { admin, aged, browser, fakeSes, reader, rows, testEnv } from './helpers.ts'; + +let ses: ReturnType; +afterEach(() => ses?.restore()); + +const html = `

Hi *|FNAME|*

+A B C +Unsubscribe`; + +async function sent(n = 4, key = 'the-source-50') { + ses = fakeSes(); + const env = testEnv(); + const contacts = Array.from({ length: n }, (_, i) => ({ email: `r${i}@example.com`, consentedAt: '2026-09-15T10:00:00Z' })); + await handle(admin('/admin/import', { list: 'the-source', source: 'mailchimp-import', consent: 'c', contacts, suppressions: [] }), env); + await handle(admin('/admin/sends', { key, list: 'the-source', kind: 'live', subject: 'S', html }), env); + await handle(admin('/admin/sends/' + key + '/batch', { max: 50 }), env); + const tokens = (await rows(env, 'SELECT token FROM deliveries ORDER BY id')).map(r => r.token as string); + return { env, tokens }; +} + +// A request as it reaches the Worker, with Cloudflare's view of the network it came from. +function from(url: string, cf: Record, agent = browser) { + const r = new Request(url, { headers: { 'User-Agent': agent } }); + Object.defineProperty(r, 'cf', { value: cf }); + return r; +} + +test('scanners and machines are told apart from people, and a person\'s click still counts after a scanner\'s', async () => { + const { env, tokens } = await sent(); + const c = (t: string, n: number) => `https://news.test/c/${t}/${n}`; + // Right after the send: only a scanner clicks this soon. + await handle(reader(c(tokens[0], 0)), env); + await aged(env); + // A mail server's checks, by network and by user agent; a click with no user agent. + await handle(from(c(tokens[1], 0), { asn: 8075, asOrganization: 'MICROSOFT-CORP-MSN-AS-BLOCK' }), env); + await handle(from(c(tokens[1], 1), { asn: 64500, asOrganization: 'Proofpoint, Inc.' }), env); + await handle(new Request(c(tokens[1], 2)), env); + // A scanner that follows every link in turn, from a network nobody knows. + for (const n of [0, 1, 2]) await handle(reader(c(tokens[2], n)), env); + // A person, later, on the link a scanner clicked first. + await handle(reader(c(tokens[0], 0)), env); + // Opens: Apple's proxy, and a person. + await handle(from(`https://news.test/o/${tokens[3]}.gif`, { asn: 714, asOrganization: 'APPLE-ENGINEERING' }), env); + await handle(reader(`https://news.test/o/${tokens[0]}.gif`), env); + + const s = await stats(env, 'the-source-50'); + assert.deepEqual(s.clicks, { unique: 1, total: 1, machine: 2, any: 3 }); + assert.deepEqual(s.opens, { unique: 1, total: 1, machine: 1, any: 2 }); + assert.deepEqual(s.links.map(l => [l.unique, l.machine]), [[1, 2], [0, 2], [0, 2]]); + assert.deepEqual(s.rates, { delivered: 0, open: 0.5, openPeople: 0.25, click: 0.25, clickToOpen: 0.5, unsubscribe: 0, bounce: 0, complaint: 0 }); + // Nothing but the verdict is kept. + assert.deepEqual(Object.keys((await rows(env, 'SELECT * FROM events LIMIT 1'))[0]).sort(), ['at', 'delivery_id', 'detail', 'id', 'link', 'machine', 'type']); + + // The dashboard counts the person's click only, and says how many only scanners touched. + env.DEV_IDENTITY = 'admin@example.com'; + const page = await (await handle(new Request('https://news.test/dashboard/sends/the-source-50'), env)).text(); + assert.match(page, /
Unique clicks<\/div>
1<\/div>
25\.0% of sent · 1 in all · 2 more only by scanners/); +}); + +test('sends side by side, sending health, and when SES events do not arrive', async () => { + const { env } = await sent(40); + const summary = await sendSummaries(env, { limit: 10 }); + assert.equal(summary.length, 1); + assert.deepEqual([summary[0].sent, summary[0].rates], [40, { open: 0, click: 0, unsubscribe: 0 }]); + let health = await deliverability(env); + assert.equal(health.sent, 40); + assert.equal(health.eventsMissing, true, 'mail went out and no delivery event came back'); + assert.equal(health.health.overall, 'ok'); + + // Two hard bounces and one complaint in 40: 5% bounces, 2.5% complaints. + const ids = (await rows(env, 'SELECT id FROM deliveries ORDER BY id')).map(r => r.id as number); + for (const id of ids) await env.DB.prepare(`INSERT INTO events (delivery_id, type) VALUES (?, 'delivery')`).bind(id).run(); + for (const id of ids.slice(0, 2)) await env.DB.prepare(`INSERT INTO events (delivery_id, type) VALUES (?, 'bounce')`).bind(id).run(); + await env.DB.prepare(`INSERT INTO events (delivery_id, type) VALUES (?, 'complaint')`).bind(ids[5]).run(); + health = await deliverability(env); + assert.deepEqual([health.bounced, health.complaints, health.rates.bounce, health.rates.complaint], [2, 1, 0.05, 0.025]); + assert.deepEqual(health.health, { bounce: 'act', complaint: 'act', overall: 'act' }); + assert.equal(health.eventsMissing, false); + + env.DEV_IDENTITY = 'admin@example.com'; + const page = await (await handle(new Request('https://news.test/dashboard'), env)).text(); + assert.match(page, /id="health"[\s\S]*Hard bounces[\s\S]*5\.0%[\s\S]*act now/); + assert.doesNotMatch(page, /id="events-missing"/); +}); + +test('list growth counts who joined and left, by month, and counts a person once', async () => { + const { env, tokens } = await sent(3); + const now = new Date(); + const month = iso(now).slice(0, 7); + await subscribe(env, { email: 'new@example.com', list: 'the-source', consent: 'c', source: 'web' }); + await subscribe(env, { email: 'new@example.com', list: 'the-source', consent: 'c', source: 'web' }); + await handle(new Request(`https://news.test/u/${tokens[0]}`, { method: 'POST' }), env); + await handle(admin('/admin/import', { list: 'the-source', source: 'mailchimp-import', consent: 'c', contacts: [{ email: 'r1@example.com', consentedAt: '2026-09-15T10:00:00Z' }], suppressions: [] }), env); + + const g = await listGrowth(env, { months: 24 }, now); + const source = g.lists.find(l => l.list === 'the-source')!; + assert.equal(source.subscribers, 3); + assert.deepEqual(source.months.find(m => m.month === '2026-09'), { month: '2026-09', joined: 3, departed: 0, net: 3 }, 'an import counts people when they signed up'); + assert.deepEqual(source.months.find(m => m.month === month), { month, joined: 1, departed: 1, net: 0 }); + assert.equal(g.lists.some(l => l.list === 'ftw-news'), true); + + env.DEV_IDENTITY = 'admin@example.com'; + const page = await (await handle(new Request('https://news.test/dashboard'), env)).text(); + assert.match(page, new RegExp(`${month}\\+1 −1`)); +}); diff --git a/test/tracking.test.ts b/test/tracking.test.ts index bc48279..1e1379e 100644 --- a/test/tracking.test.ts +++ b/test/tracking.test.ts @@ -1,7 +1,7 @@ import { test, afterEach } from 'node:test'; import assert from 'node:assert/strict'; import { handle } from '../src/app.ts'; -import { admin, fakeSes, rows, testEnv } from './helpers.ts'; +import { admin, aged, fakeSes, reader, rows, testEnv } from './helpers.ts'; let ses: ReturnType; afterEach(() => ses?.restore()); @@ -19,11 +19,12 @@ async function sent() { await handle(admin('/admin/import', { list: 'the-source', source: 'mailchimp-import', consent: 'mailchimp-the-source', contacts, suppressions: [] }), env); await handle(admin('/admin/sends', { key: 'the-source-22', list: 'the-source', kind: 'live', subject: 'The Source #22', html }), env); await handle(admin('/admin/sends/the-source-22/batch', { max: 10 }), env); + await aged(env); const tokens = (await rows(env, 'SELECT token FROM deliveries ORDER BY id')).map(r => r.token as string); return { env, tokens }; } -const get = (path: string, method = 'GET') => new Request(`https://news.test${path}`, { method }); +const get = (path: string, method = 'GET') => reader(`https://news.test${path}`, method); test('a click redirects to the link and is logged against it', async () => { const { env, tokens } = await sent(); @@ -66,11 +67,11 @@ test('stats count unique and total clicks per link, opens, and statuses', async assert.equal(stats.recipients, 3); assert.deepEqual(stats.status, { sent: 3 }); // Repeats within the hour count once (see the next test); the unique figures are what matter. - assert.deepEqual(stats.opens, { unique: 1, total: 1 }); - assert.deepEqual(stats.clicks, { unique: 2, total: 3 }); + assert.deepEqual(stats.opens, { unique: 1, total: 1, machine: 0, any: 1 }); + assert.deepEqual(stats.clicks, { unique: 2, total: 3, machine: 0, any: 2 }); assert.deepEqual(stats.links, [ - { n: 0, url: 'https://ftw.sourceful.energy', unique: 2, total: 2 }, - { n: 1, url: 'https://sourceful.energy/master-plan?a=1&b=2', unique: 1, total: 1 }, + { n: 0, url: 'https://ftw.sourceful.energy', unique: 2, total: 2, machine: 0, any: 2 }, + { n: 1, url: 'https://sourceful.energy/master-plan?a=1&b=2', unique: 1, total: 1, machine: 0, any: 1 }, ]); }); @@ -121,7 +122,7 @@ test('a click on a link with odd characters redirects to its encoded form', asyn ses = fakeSes(); const env = testEnv(); await handle(admin('/admin/import', { list: 'the-source', source: 'mailchimp-import', consent: 'mailchimp-the-source', contacts: [{ email: 'a@example.com' }], suppressions: [] }), env); - await handle(admin('/admin/sends', { key: 'odd', list: 'the-source', kind: 'live', subject: 'S', html: 'x' }), env); + await handle(admin('/admin/sends', { key: 'odd', list: 'the-source', kind: 'live', subject: 'S', html: 'xu' }), env); await handle(admin('/admin/sends/odd/batch', { max: 10 }), env); const [{ token }] = await rows<{ token: string }>(env, 'SELECT token FROM deliveries'); const res = await handle(get(`/c/${token}/0`), env); diff --git a/test/unsubscribe.test.ts b/test/unsubscribe.test.ts index abd5853..e203247 100644 --- a/test/unsubscribe.test.ts +++ b/test/unsubscribe.test.ts @@ -73,7 +73,7 @@ test('an unsubscribed address gets no later issue, and an import cannot bring it await handle(oneClick(tokens['bo@example.com']), env); const reimport = await handle(admin('/admin/import', { list: 'the-source', source: 'mailchimp-import', consent: 'mailchimp-the-source', contacts: [{ email: 'bo@example.com', consentedAt: '2024-01-01T00:00:00Z' }], suppressions: [] }), env); assert.equal((await reimport.json() as Record).skipped, 1); - const next = await handle(admin('/admin/sends', { key: 'the-source-23', list: 'the-source', kind: 'live', subject: 'S', html: '

x

' }), env); + const next = await handle(admin('/admin/sends', { key: 'the-source-23', list: 'the-source', kind: 'live', subject: 'S', html: '

x

u' }), env); assert.equal((await next.json() as Record).recipients, 1); }); diff --git a/test/webhook.test.ts b/test/webhook.test.ts index 9bc3879..0a7a7ad 100644 --- a/test/webhook.test.ts +++ b/test/webhook.test.ts @@ -15,7 +15,7 @@ async function sentTo(emails: string[]) { const env = testEnv(); const contacts = emails.map(email => ({ email, consentedAt: '2024-01-01T00:00:00Z' })); await handle(admin('/admin/import', { list: 'the-source', source: 'mailchimp-import', consent: 'mailchimp-the-source', contacts, suppressions: [] }), env); - await handle(admin('/admin/sends', { key: 'the-source-22', list: 'the-source', kind: 'live', subject: 'S', html: '

x

' }), env); + await handle(admin('/admin/sends', { key: 'the-source-22', list: 'the-source', kind: 'live', subject: 'S', html: '

x

u' }), env); await handle(admin('/admin/sends/the-source-22/batch', { max: 10 }), env); return env; } @@ -40,26 +40,64 @@ test('a permanent bounce suppresses the address on every list', async () => { assert.deepEqual(await rows(env, 'SELECT scope, reason FROM suppressions'), [{ scope: '*', reason: 'bounce' }]); assert.deepEqual(await rows(env, `SELECT email FROM contacts ORDER BY email`), [{ email: 'ok@example.com' }]); assert.deepEqual(await rows(env, `SELECT e.type, d.message_id FROM events e JOIN deliveries d ON d.id = e.delivery_id`), [{ type: 'bounce', message_id: 'msg-2' }]); - const next = await handle(admin('/admin/sends', { key: 'the-source-23', list: 'the-source', kind: 'live', subject: 'S', html: '

x

' }), env); + const next = await handle(admin('/admin/sends', { key: 'the-source-23', list: 'the-source', kind: 'live', subject: 'S', html: '

x

u' }), env); assert.equal((await next.json() as Record).recipients, 1); }); -test('a soft bounce on a second mail suppresses the address', async () => { - const env = await sentTo(['full@example.com']); - await handle(admin('/admin/sends', { key: 'the-source-23', list: 'the-source', kind: 'live', subject: 'S2', html: '

y

' }), env); - await handle(admin('/admin/sends/the-source-23/batch', { max: 10 }), env); - const soft = (id: string) => post(sesEvent({ +test('a full mailbox on a second mail suppresses the address; a busy server does not', async () => { + const twoMails = async () => { + const env = await sentTo(['full@example.com']); + await handle(admin('/admin/sends', { key: 'the-source-23', list: 'the-source', kind: 'live', subject: 'S2', html: '

y

u' }), env); + await handle(admin('/admin/sends/the-source-23/batch', { max: 10 }), env); + return env; + }; + const soft = (id: string, bounceSubType: string) => post(sesEvent({ eventType: 'Bounce', mail: { messageId: id }, - bounce: { bounceType: 'Transient', bouncedRecipients: [{ emailAddress: 'full@example.com' }] }, + bounce: { bounceType: 'Transient', bounceSubType, bouncedRecipients: [{ emailAddress: 'full@example.com' }] }, })); - await handle(soft('msg-1'), env); - await handle(soft('msg-1'), env); + let env = await twoMails(); + await handle(soft('msg-1', 'General'), env); + await handle(soft('msg-2', 'General'), env); + assert.deepEqual(await rows(env, 'SELECT * FROM suppressions'), [], 'a server that is slow to trust a new sender says nothing about the address'); + ses.restore(); + env = await twoMails(); + await handle(soft('msg-1', 'MailboxFull'), env); + await handle(soft('msg-1', 'MailboxFull'), env); assert.deepEqual(await rows(env, 'SELECT * FROM suppressions'), [], 'two soft bounces of one mail are still one'); - await handle(soft('msg-2'), env); + await handle(soft('msg-2', 'MailboxFull'), env); assert.deepEqual(await rows(env, 'SELECT scope, reason FROM suppressions'), [{ scope: '*', reason: 'bounce' }]); }); +test('a bounce report suppresses the mail\'s own recipient, not the addresses the report names', async () => { + const env = await sentTo(['ok@example.com', 'sly@example.com']); + const [sly] = await rows(env, `SELECT d.message_id FROM deliveries d JOIN contacts c ON c.id = d.contact_id WHERE c.email = 'sly@example.com'`); + await handle(post(sesEvent({ + eventType: 'Bounce', + mail: { messageId: sly.message_id, destination: ['sly@example.com'] }, + bounce: { bounceType: 'Permanent', bouncedRecipients: [{ emailAddress: 'ok@example.com' }, { emailAddress: 'victim@example.com' }] }, + })), env); + assert.deepEqual(await rows(env, 'SELECT email FROM contacts'), [{ email: 'ok@example.com' }]); + assert.equal((await rows(env, 'SELECT COUNT(*) AS n FROM suppressions')).at(0)!.n, 1); + // A mail Utskick cannot place: only an address the mail itself went to counts. + await handle(post(sesEvent({ + eventType: 'Complaint', + mail: { messageId: 'msg-unknown', destination: ['nobody@example.com'] }, + complaint: { complainedRecipients: [{ emailAddress: 'ok@example.com' }] }, + })), env); + assert.deepEqual(await rows(env, 'SELECT email FROM contacts'), [{ email: 'ok@example.com' }]); +}); + +test('an event for a row whose settle was lost marks it sent, and a message that is not JSON is ignored', async () => { + const env = await sentTo(['ok@example.com']); + await env.DB.prepare(`UPDATE deliveries SET status = 'unknown', message_id = NULL`).run(); + await handle(post(sesEvent({ eventType: 'Delivery', mail: { messageId: 'msg-9', destination: ['ok@example.com'], tags: { send: ['the-source-22'] } } })), env); + assert.deepEqual(await rows(env, 'SELECT status, message_id FROM deliveries'), [{ status: 'sent', message_id: 'msg-9' }]); + assert.deepEqual(await rows(env, 'SELECT type FROM events'), [{ type: 'delivery' }]); + const res = await handle(post(snsMessage({ Type: 'Notification', Message: 'not json' })), env); + assert.equal(res.status, 200); +}); + test('a transient bounce is logged but suppresses nothing', async () => { const env = await sentTo(['full@example.com']); await handle(post(sesEvent({ @@ -159,3 +197,17 @@ test('the webhook refuses a huge body without reading it all, and a null body wi assert.equal((await handle(big, env)).status, 413); assert.equal((await handle(post(null), env)).status, 400); }); + +// review: a reader who reports spam and unsubscribes at once has no contact row left when the complaint comes. +test('a complaint after the reader already unsubscribed still suppresses the address everywhere', async () => { + const env = await sentTo(['angry@example.com']); + const [d] = await rows(env, 'SELECT token, message_id FROM deliveries'); + await handle(new Request(`https://news.test/u/${d.token}`, { method: 'POST' }), env); + assert.deepEqual(await rows(env, 'SELECT * FROM contacts'), []); + await handle(post(sesEvent({ + eventType: 'Complaint', + mail: { messageId: d.message_id, destination: ['angry@example.com'] }, + complaint: { complainedRecipients: [{ emailAddress: 'angry@example.com' }] }, + })), env); + assert.deepEqual(await rows(env, 'SELECT scope, reason FROM suppressions ORDER BY scope'), [{ scope: '*', reason: 'complaint' }, { scope: 'the-source', reason: 'unsubscribe' }]); +});