Postmaster v9.4.1 qualitative provider click classification

CHANGELOG.md

@@ -2,6 +2,25 @@
 
 Postmaster MCP follows Semantic Versioning for stable releases. Every stable release should update `VERSION`, this changelog, and publish an immutable Git tag/release named `vX.Y.Z`.
 
+## 9.4.1 - 2026-08-20
+
+### Added
+- Query-time qualitative classification for link-click telemetry. Every returned `link` event can now include `provider_likelihood` (0-100), `provider_classification` (`likely_human`, `uncertain`, `likely_email_provider`, `known_email_proxy`), `provider_guess` (`google`, `microsoft`, `yahoo`, `other` or null) and human-readable `classification_reasons`.
+- Combined-evidence heuristic model: explicit `GoogleImageProxy`/Gmail proxy signatures are treated as known proxy evidence; near-simultaneous second requests on the same `delivery_id + link_id`, changed fingerprints, country/browser/User-Agent changes and proxy source metadata increase provider likelihood, while a fingerprint seen consistently across multiple links in the same delivery lowers it.
+- Qualitative unique-click summary fields including `likely_provider_unique_clicks`, `likely_human_or_unclassified_unique_clicks`, `uncertain_unique_clicks`, provider suspects and potential-provider share while preserving the original fingerprint unique count beside them.
+- Tracking dashboard qualitative summary plus per-link-event provider score, classification, provider guess and reasons.
+- `build_status.provider_qualitative_classification=true` and `tracking_status.link_tracking.provider_classification_query_time=true` capability reporting.
+- Ground-truth regression fixtures for the observed Gmail duplicate-fetch pattern, stable human fingerprints across multiple links, same-fingerprint Libero-style clicks and explicit `GoogleImageProxy` traffic.
+
+### Changed
+- Provider classification is recalculated from stored click evidence on every query instead of being persisted as authoritative database state, so historical events automatically benefit from future heuristic refinements.
+- `get_tracking_summary`, `get_tracking_campaign` and `list_tracking_events` expose the qualitative interpretation layer without removing or renaming the v9.4 analytics fields.
+
+### Compatibility / deployment
+- The v9.4 unique-click definition remains exactly `delivery_id + link_id + client_fingerprint`; suspicious events are never deleted, hidden or rewritten in `tracking_clicks`.
+- No tracking schema migration is required and the v9.3/v9.4 single-YAML Portainer bootstrap remains unchanged.
+- No new public HTTP endpoint is introduced. Cloudflare Access public bypass requirements remain `/track/open/*`, `/api/amp/*` and `/t/c/*` according to the features in use. `/files/*` remains the separate pre-existing signed file-handoff concern.
+
 ## 9.4.0 - 2026-08-19
 
 ### Added
@@ -95,4 +114,4 @@ Postmaster MCP follows Semantic Versioning for stable releases. Every stable rel
 - CI coverage for runtime import, bootstrap, model provisioning, MIME regressions and knowledge operations.
 
 ### Changed
-- Public project naming and configuration became provider-agnostic.
+- Public project naming and configuration became provider-agnostic.

VERSION

@@ -1 +1 @@
-9.4.0
+9.4.1

docs/LINK_TRACKING.md

@@ -1,6 +1,6 @@
-# Link tracking and clean Sent copies (v9.4)
+# Link tracking, qualitative provider classification and clean Sent copies (v9.4 / v9.4.1)
 
-Postmaster v9.4 adds per-link click telemetry to the existing per-recipient analytics pipeline. The existing open-tracking pixel remains unchanged: its route, event format, enrichment, client fingerprint, country/source, browser/OS parsing, campaign/delivery correlation and dashboard continue to work as before.
+Postmaster v9.4 adds per-link click telemetry to the existing per-recipient analytics pipeline. v9.4.1 adds a second, explicitly qualitative interpretation layer that estimates whether a unique click context may have been generated by mailbox-provider infrastructure. The existing open-tracking pixel, raw click records and stable fingerprint unique-click definition remain unchanged.
 
 ## Recipient versus Sent architecture
 
@@ -24,7 +24,7 @@ Link tracking follows the existing `track_opens` tracking opt-in (including the
 
 ## Public click endpoint
 
-The only new public callback path required by v9.4 is:
+The public click callback is:
 
 ```text
 GET /t/c/<token>
@@ -34,6 +34,8 @@ The public base is resolved exactly like existing mail callbacks: `PUBLIC_EMAIL_
 
 The token is random and opaque; it does not encode recipient data or the destination. `/t/c/<token>` resolves a server-side `tracking_links` record, records a `link` event and issues a `302` to the stored `original_url`. Request/query parameters are never used as the destination, so `?url=https://...` cannot create an open redirect. Invalid tokens return `404` without a destination hint.
 
+v9.4.1 does **not** add another public endpoint. Provider classification runs only when protected analytics/MCP/dashboard queries read already stored click evidence.
+
 ## HTML rewriting
 
 Recipient HTML rewrites eligible `http://` and `https://` anchors only. Postmaster does not rewrite `mailto:`, `tel:`, `cid:`, `data:`, `javascript:`, `#fragment` or other non-web URLs. A URL already pointing to this deployment's `/t/c/` path is not wrapped twice. A legitimate third-party URL whose path happens to contain `/t/c/` remains eligible.
@@ -50,23 +52,117 @@ Existing pixel tables are not replaced. v9.4 adds:
 
 The click fingerprint reuses the existing analytics HMAC derivation; raw IP is not stored. Country/source/browser/OS parsing reuses the pixel enrichment helpers. The browser field continues to contain the parsed browser/version label used by the existing pipeline.
 
-## Unique click
+v9.4.1 deliberately adds **no provider-classification columns** to `tracking_clicks`. The score is derived at read time, so old events automatically receive the current classifier and future heuristic improvements do not require rewriting historical data.
+
+## Stable unique click
 
-A v9.4 unique click is:
+The authoritative unique-click definition remains:
 
 ```text
 delivery_id + link_id + client_fingerprint
 ```
 
-If both IP and User-Agent are unavailable, the existing keyed fingerprint pipeline produces the stable HMAC of the empty pair; repeated unknown fetches for that delivery/link therefore collapse consistently. v9.4 does not classify human/bot/scanner clicks. User-Agent, source and fingerprint are retained for a future evidence-based classifier.
+If both IP and User-Agent are unavailable, the existing keyed fingerprint pipeline produces the stable HMAC of the empty pair; repeated unknown fetches for that delivery/link therefore collapse consistently.
+
+The qualitative layer never removes a suspicious event from this count. A report can therefore show both:
+
+```text
+unique_clicks: 24
+likely_provider_unique_clicks: 3
+likely_human_or_unclassified_unique_clicks: 21
+```
+
+## v9.4.1 qualitative provider classification
+
+Every returned link event can include:
+
+```text
+provider_likelihood: 0..100
+provider_classification:
+  likely_human
+  uncertain
+  likely_email_provider
+  known_email_proxy
+provider_guess:
+  null
+  google
+  microsoft
+  yahoo
+  other
+classification_reasons:
+  - "second request on same delivery/link after 3.64s"
+  - "fingerprint changed"
+  - "country changed IT → US"
+  - "browser changed Chrome 151.0.0.0 → Chrome 149.0.0.0"
+```
+
+This is a heuristic interpretation, not ground truth. Timing, geography, browser version or fingerprint change alone are not considered proof. Signals are combined.
 
-## Analytics, MCP and dashboard
+Current `heuristic-v1-query-time` behavior includes:
+
+- explicit `GoogleImageProxy`, Gmail image-proxy source or parsed Google Image Proxy evidence -> known proxy, score 100, provider `google`;
+- a second request for the same `delivery_id + link_id` within 5 seconds with a different fingerprint -> strong provider signal;
+- the same pattern within 15 seconds -> medium signal;
+- fast country, browser and User-Agent changes -> additional independent signals;
+- source metadata that explicitly indicates a proxy -> additional signal;
+- a fingerprint that appears consistently across multiple different links in the same delivery -> negative provider evidence because it is consistent with one human browser context;
+- a single anomaly-free click -> low provider score.
+
+Thresholds are intentionally simple and inspectable:
+
+```text
+0..34   likely_human
+35..69  uncertain
+70..99  likely_email_provider
+100     known_email_proxy when an explicit known signature matched
+```
+
+For provider-like rows without a vendor-specific signature, `provider_guess` may be `other`. Microsoft/Yahoo guesses are only hints when their names/signatures are present in the stored client metadata; they are not treated as known proxies by name alone.
+
+### Ground-truth fixture examples
+
+The v9.4.1 regression tests encode the first observed real-world patterns discussed during development:
+
+```text
+Gmail A — IT / Chrome 151                     low (~0-10)
+Gmail A — US / Chrome 149, +3.637 s           high (~85-100)
+Same human fingerprint on another link        low
+Libero-style same fingerprint across links    low
+GoogleImageProxy explicit                     100
+```
 
-Available analytics include total clicks, unique clicks, unique recipients, first/last click, campaign/delivery/link filtering, event detail, top links and destination host.
+These fixtures are deliberately evidence-based and can be extended as more Gmail, Outlook, Yahoo, Libero and other mailbox-provider cases are observed.
 
-Existing `tracking_status` and `get_tracking_campaign` are extended. v9.4 adds read-only `get_tracking_summary`, `list_tracking_links` and `list_tracking_events`. Listing/event tools never return opaque click tokens.
+## Qualitative campaign summary
 
-The existing pixel dashboard is preserved. v9.4 adds **Top links** and a unified **Tracking events** table distinguishing `pixel`, `amp_xhr` and `link`; link rows include label, `link_id`, destination host and position.
+`get_tracking_summary` and the link-tracking block inside `get_tracking_campaign` retain the v9.4 counters and additionally expose a qualitative estimate such as:
+
+```text
+Observed click events: 37
+Unique fingerprint clicks: 24
+Likely human: 18
+Uncertain: 3
+Likely provider/proxy: 3
+Potential provider share: 3 / 24 (12.5%)
+Provider suspects: google 1, other 2
+Confidence: high
+```
+
+`confidence` describes the strength of the provider-like rows that were found; it does not mean Postmaster can prove which requests were human.
+
+## MCP and dashboard
+
+Existing `tracking_status` and `get_tracking_campaign` remain available. v9.4 adds read-only `get_tracking_summary`, `list_tracking_links` and `list_tracking_events`.
+
+In v9.4.1:
+
+- `get_tracking_summary` adds `qualitative_estimate`, `likely_provider_unique_clicks`, `likely_human_or_unclassified_unique_clicks`, `uncertain_unique_clicks`, `potential_provider_share`, `provider_suspects` and `provider_classification_model`;
+- `get_tracking_campaign` receives those fields inside its existing link-tracking summary;
+- `list_tracking_events` adds the provider fields to `link` rows while existing `pixel` and `amp_xhr` rows remain structurally compatible;
+- `build_status.provider_qualitative_classification=true` reports the capability;
+- the dashboard adds a qualitative summary and event-level provider score/classification/reasons beside the unchanged raw fingerprint data.
+
+Opaque click tokens are never returned by listing/event tools.
 
 ## Clean Sent behavior and historical finding
 
@@ -76,41 +172,74 @@ v9.4 builds outbound and Sent MIME independently from the same canonical body/at
 
 This is the primary self-open/self-click defense; Postmaster does not guess sender identity from IP, country, User-Agent, browser or fingerprint.
 
-## Cloudflare Access — manual deployment requirement
+## Cloudflare Access — exactly which endpoints to open
+
+Keep Postmaster protected by default. The public callback surface for tracking/AMP is intentionally narrow.
+
+Required Access bypasses according to enabled features:
 
-Keep Postmaster protected by default. Existing public callback paths remain:
+```text
+GET /track/open/*    # open-tracking pixel
+GET /api/amp/*       # AMP callback/status path family; current route is /api/amp/status
+GET /t/c/*           # per-link click redirect
+```
+
+For a deployment using all three features, configure public Bypass rules for exactly:
 
 ```text
 /track/open/*
 /api/amp/*
+/t/c/*
 ```
 
-v9.4 adds exactly one new required public path:
+v9.4.1 introduces no additional anonymous route.
+
+Keep these protected by Cloudflare Access:
 
 ```text
-/t/c/*
+/
+/mcp
+/dashboard/*
+mail/account/task/memory/skill APIs
+file-management administration
+tracking analytics and MCP read tools
+all other private/general application routes
 ```
 
-**Cloudflare Access must bypass `/t/c/*`.** Without this bypass a recipient reaches the Access login/challenge instead of Postmaster and the redirect fails.
+Do not disable Access globally and do not expose the raw Docker port directly to the Internet.
 
-Keep `/mcp`, dashboard/admin/private APIs, mail/task/memory/skill/file-management routes and tracking analytics protected. Do not disable Access globally.
+### Separate signed file-handoff endpoint
+
+`GET`/`HEAD /files/{file_id}` is a pre-existing v9.3 signed file-handoff route. It is **not** a v9.4/v9.4.1 tracking bypass requirement. If the deployment intentionally relies on signed public HTTP file handoff, the operator must separately decide to make `/files/*` reachable through the external proxy policy; authorization is then provided by the route's expiry/HMAC signature. If the deployment uses MCP resource handoff instead, `/files/*` does not need to be opened merely for tracking.
+
+## Portainer deployment
+
+`postmaster-mcp.yml` remains unchanged. With:
+
+```yaml
+POSTMASTER_VERSION: latest
+POSTMASTER_CHECK_UPDATES_ON_START: "true"
+```
 
-### Separate v9.3 file-handoff note
+a stack/container restart after the stable v9.4.1 release is published is sufficient for the bootstrap to resolve and cache the new release. No compose variable, database migration or new volume is required for the qualitative classifier.
 
-`/files/{file_id}` is a pre-existing v9.3 signed HTTP file-handoff route. v9.4 does not add `/files/*` to its Cloudflare bypass or change that policy. If a deployment intentionally uses signed HTTP file handoff, the operator must separately ensure that pre-existing route is reachable according to the deployment's proxy policy.
+Cloudflare Access is external/manual configuration and is never changed automatically by the container.
 
-## Live preflight
+## Live preflight for v9.4.1
 
-After release/deploy and after the operator adds the `/t/c/*` bypass:
+After release/deploy:
 
-1. `build_status` reports `9.4.0`, `link_tracking=true`, `sent_copy_tracking_sanitized=true`.
-2. Send a tracked email with at least two distinct HTTP/HTTPS URLs.
-3. Recipient MIME contains the unchanged `/track/open/...gif` pixel and distinct `/t/c/<token>` URLs.
-4. Sent MIME contains neither recipient pixel nor `/t/c/`, and contains the original URLs.
-5. Anonymous `GET https://<PUBLIC_HOST>/t/c/<valid-token>` reaches Postmaster without Cloudflare login, records a `link` event and redirects to the exact stored destination.
-6. Invalid token plus `?url=https://evil.example/` does not redirect.
-7. Link #1/#2 are independently visible in total/unique/top-link analytics with enrichment fields.
-8. Clicking a URL extracted from Sent goes directly to the original destination and creates no Postmaster click event.
-9. `/mcp`, dashboard and all other private paths remain protected by Cloudflare Access.
+1. `build_status` reports `9.4.1`, `link_tracking=true`, `sent_copy_tracking_sanitized=true`, `provider_qualitative_classification=true`.
+2. `tracking_status.link_tracking.provider_classification_query_time=true`.
+3. Send a tracked email with at least two distinct HTTP/HTTPS URLs.
+4. Recipient MIME contains the unchanged `/track/open/...gif` pixel and distinct `/t/c/<token>` URLs.
+5. Sent MIME contains neither recipient pixel nor `/t/c/`, and contains the original URLs.
+6. Anonymous `GET https://<PUBLIC_HOST>/t/c/<valid-token>` reaches Postmaster without Cloudflare login, records a `link` event and redirects to the exact stored destination.
+7. Invalid token plus `?url=https://evil.example/` does not redirect.
+8. `list_tracking_events(event_type="link")` exposes provider fields while the underlying database schema still contains no provider score/classification columns.
+9. `get_tracking_summary` returns both the unchanged `unique_clicks` and the new qualitative counters.
+10. A known `GoogleImageProxy` fixture/query classifies as `known_email_proxy` with score 100; the Gmail duplicate-fetch ground-truth fixture remains high and the stable human fingerprint fixtures remain low.
+11. Clicking a URL extracted from Sent goes directly to the original destination and creates no Postmaster click event.
+12. `/mcp`, dashboard and all other private paths remain protected by Cloudflare Access.
 
-The container cannot prove Cloudflare policy by itself. v9.4 is fully operational only after the external `/t/c/*` bypass is configured and the anonymous live redirect test succeeds.
+The container cannot prove Cloudflare policy by itself. Tracking is fully operational only after the external callback bypasses required by the features in use are configured and anonymous live callback tests succeed.

src/postmaster/link_tracking_queries.py

@@ -2,8 +2,42 @@
 
 from typing import Any
 
+from .provider_classification import classify_click_events, summarize_click_classification
+
 
 class LinkTrackingQueriesMixin:
+    def _classification_events(
+        self,
+        *,
+        campaign_id: str | None = None,
+        delivery_id: str | None = None,
+        link_id: str | None = None,
+        account_id: str | None = None,
+    ) -> list[dict[str, Any]]:
+        clauses: list[str] = []
+        params: list[Any] = []
+        for column, value in (
+            ("c.campaign_id", campaign_id),
+            ("c.delivery_id", delivery_id),
+            ("c.link_id", link_id),
+            ("c.account_id", account_id),
+        ):
+            if value:
+                clauses.append(f"{column}=?")
+                params.append(value)
+        where = (" WHERE " + " AND ".join(clauses)) if clauses else ""
+        with self._connect() as conn:
+            rows = conn.execute(
+                f"""
+                SELECT c.*
+                FROM tracking_clicks c
+                {where}
+                ORDER BY c.observed_at ASC,c.id ASC
+                """,
+                params,
+            ).fetchall()
+        return [dict(row) for row in rows]
+
     def list_links(
         self,
         *,
@@ -90,7 +124,7 @@ def list_click_events(
                 """,
                 params,
             ).fetchall()
-        return [dict(row) for row in rows]
+        return classify_click_events([dict(row) for row in rows])
 
     def summary(
         self,
@@ -134,6 +168,21 @@ def summary(
             "If IP and User-Agent are unavailable, the existing keyed fingerprint pipeline "
             "uses the stable HMAC of the empty pair, collapsing unknown repeat fetches for that delivery/link."
         )
+        qualitative = summarize_click_classification(
+            self._classification_events(
+                campaign_id=campaign_id,
+                delivery_id=delivery_id,
+                link_id=link_id,
+                account_id=account_id,
+            )
+        )
+        out["qualitative_estimate"] = qualitative
+        out["likely_provider_unique_clicks"] = qualitative["likely_provider_unique_clicks"]
+        out["likely_human_or_unclassified_unique_clicks"] = qualitative["likely_human_or_unclassified_unique_clicks"]
+        out["uncertain_unique_clicks"] = qualitative["uncertain_unique_clicks"]
+        out["potential_provider_share"] = qualitative["potential_provider_share"]
+        out["provider_suspects"] = qualitative["provider_suspects"]
+        out["provider_classification_model"] = qualitative["classification_model"]
         return out
 
     def top_links(