feat: connector verification loop — evidence claims, run-and-watch, scenario record/replay - #140
feat: connector verification loop — evidence claims, run-and-watch, scenario record/replay#140tnunamak wants to merge 15 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
27d45cf to
d3d71ff
Compare
|
Round-2 repairs for the final re-review are on the branch (
Still honestly open (tracked for follow-up PRs): per-connector provider-authority policy; browser replay driver (Playwright HAR routing per the layered plan); IMAP seedable-fake driver for gmail; subprocess-output driver for slackdump/gmcli-class connectors; overlap/dedup/idempotence scenario fixtures; built-package digest binding (publication pipeline); response-body field-level secret handling. Current head: 113-test harness batch (111 pass, 2 honest environment skips), both connector spikes green, tsc/biome clean. Assisted-by: AI |
|
Round-3 repairs on the branch (
Verification at head: 141-test batch (139 pass, 2 environment skips), both connector spikes green, tsc/biome clean. Remaining follow-ups per the review's own disposition table (all conditioned as it required): provider-authority policy (labels already narrowed), browser HAR driver, gmail IMAP seedable fake, subprocess-output driver, overlap/dedup fixtures, package-digest binding. Assisted-by: AI |
|
Round-4 repairs (
Verification at head: 165-test batch (163 pass, 2 environment skips), both spikes green, tsc/biome clean. The review's structural point is taken as the standing design rule: exhaustiveness relationships (claim ↔ producer ↔ observations ↔ message union ↔ field disposition ↔ downgrade) live in the type system, not prose — that is what the Assisted-by: AI |
abd5db8 to
12a85c6
Compare
|
Closure packet (per the review sequence's requirements) — exact head Round-7 disposition: P1-1 (RECORD op directive ignored) → closed: op validated at the wire boundary, captured index-aligned with ids, compared on replay ( Negative-control inventory (cumulative): eligibility — one test per condition incl. all four digest-presence cases, driver-evidence vacuity (zero interactions + expected records / + trace), unsupported-surface downgrade; wire boundary — unknown type (record+verify), malformed RECORD fields, invalid op, empty/partial recovery hints, missing/blank locators, numeric ids, empty DONE.error, mixed-invalid key arrays, fractional/negative counts, blank continuation boundaries; trace — drop/alter for every truth-bearing kind incl. continuation flips, gap lifecycle drops, digest drift; runtime parity — 16 curated malformed continuation cases driven through both validators; interaction — per-field prompt mutations; isolation — child-process canary (environment-gated); recorder — fire-and-forget-at-exit, truncation, protocol-corrupt stdout; provenance — order-independence under simulated residue. Honest gaps, stated: where the runtime type declares fields optional they remain validated-when-present only (no invented strictness); the runtime itself has no emission-side validation for key-array elements/recovery-hint shape/nested diagnostics — the oracle's strictness there is oracle policy, documented as such; namespace isolation is capability-gated with honest Ready for the short final closure review. Assisted-by: AI |
|
Closure packet — final delta (head Round-8 disposition: P1 (legacy ops-less scenarios could earn the canonical claim) → closed via the review's blessed simpler option: Pre-submission internal audit (the reviewer's own closure matrix applied to the delta before this push): one P2 found and fixed (a stale comment still describing ops as backward-compatible-optional), one P3 hardened (verify's Verification: 233-test focused batch (231 pass, 2 environment-gated skips), both real-connector spikes green under mandatory ops. Exact-head workflow runs (manually dispatched after a push-trigger drop; results are head-exact): polyfill-connectors https://github.com/PDP-Connect/pdpp/actions/runs/31829065426 ✓, reference-implementation https://github.com/PDP-Connect/pdpp/actions/runs/31829081713 ✓, docker-images in progress at packet time (passed on the content-identical prior head Deferred list unchanged. Ready for final closure review. Assisted-by: AI |
d89d8cb to
f537163
Compare
…g/verification workflow Adds the connector-evidence-claims vocabulary (canonical_recorded_replay, diagnostic_replay, and their eligibility conditions) plus the authoring guide and verification workflow docs that define how a connector earns each claim level. Content-selected from PR #140 (feat/connector-verification, 61d18d6..9916ed6) Cherry-picked-from-content: 9916ed6 Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> Assisted-by: AI
Adds the scenario tooling (claims, format, isolation, record, replay, validate, verify, wire-registry) plus scenario-record/scenario-verify CLIs and scenario-verify-strict, and the oura/spotify scenario spikes proving the loop end-to-end against synthetic in-test providers. Imports of connector-runtime-protocol/local-device-envelope are rewritten from the branch's original pre-extraction relative paths (packages/polyfill-connectors/src/connector-runtime-protocol.ts, etc.) to the vendored @pdpp/connector-protocol and @pdpp/collector-runtime package imports main uses today — those modules physically left this repo for data-connect on 2026-08-17. Content-selected from PR #140 (feat/connector-verification, 61d18d6..9916ed6) Cherry-picked-from-content: 9916ed6 Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> Assisted-by: AI
…tor-init scaffolding Adds bin/connector-dev.ts (the run-and-watch loop for iterating on a connector locally, with interaction/credentials/scope-state fixtures and run-summary reporting) and bin/connector-init.ts (new-connector scaffolding), plus fixture provenance labels for the pilot-real-shape fixtures this cluster depends on (claude_code, codex, github, gmail, jellyfin, slack, venmo, ynab). connector-init.ts's generated manifest registry URL is updated to registry.pdpp.dev (main's current canonical domain per the domain-sweep commit 7f9b07c) rather than the branch's original stale registry.pdpp.org. Imports of connector-runtime-protocol/ safe-emit/is-main-module are rewritten to the vendored @pdpp/connector-protocol package main uses today. Content-selected from PR #140 (feat/connector-verification, 61d18d6..9916ed6) Cherry-picked-from-content: 9916ed6 Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> Assisted-by: AI
…ke, jellyfin schema gen Adds bin/observe-schema.ts (schema observation reports from a connector's own manifest) and the generate-validators spike, plus jellyfin's and oura's generated schema validators. jellyfin.json gains the cursor_field declaration (null, matching every other connector's manifest convention on main) that was missing before; its registry URLs are left as main's current registry.pdpp.dev, not the branch's stale registry.pdpp.org. Content-selected from PR #140 (feat/connector-verification, 61d18d6..9916ed6) Cherry-picked-from-content: 9916ed6 Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> Assisted-by: AI
…misc conflict-free additions interaction-handler.ts gains promptStdinMasked for OTP/password/token interaction prompts so secrets aren't echoed to the terminal, plus browser-launch.ts/.test.ts (a fail-fast check for a 0x0 viewport after launch — window access typed via a narrow globalThis cast since this package's browser-context evaluate callbacks run under both a DOM-lib and a non-DOM-lib tsconfig depending on which package typechecks them) and a .gitignore entry the scenario/connector-dev tooling depends on. Also carries small independent additions to client-caption.ts, the run-history-status query (explanatory comment + formatting only, no logic change), owner-connection-collection-scope.ts, and the google-provider-auth composite-dispatch test. Content-selected from PR #140 (feat/connector-verification, 61d18d6..9916ed6) Cherry-picked-from-content: 9916ed6 Signed-off-by: Tim Nunamaker <tnunamak@gmail.com> Assisted-by: AI
9916ed6 to
ae7d298
Compare
…ifest The rebased branch carried "cursor_field": null on both jellyfin streams. SourceDeclaration validation requires a string when the key is present, so registration returned 500 (source.declaration_invalid) and failed seven tests across connector-public-catalog-completeness, polyfill-range-filters, source-declaration-boundary, and static-secret-owner-capture-route. main omits the key entirely; jellyfin.json now matches main byte for byte. Fallout from the manifest re-disposition noted in local/pr140-handoff.md. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
Import ordering in connector-dev, scenario-record, scenario-verify, and scenario/verify; object destructuring in interaction-handler. buildRunSummary was at cognitive complexity 31 against a ceiling of 20. The per-stream emitted_at min/max tracking is extracted to widenRecordWindow -- a self-contained concern that carried most of the nesting weight. Behavior preserved: src/run-summary.test.ts passes 12/12, including the per-stream elapsed-time case that covers this path. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…owlist The 8-20 rebase left the allowlist holding pre-rebase coordinates while the source moved, so check:noAwaitInLoops-conformance reported 5 stale entries and 12 unlisted findings. This gate was already failing at ae7d298, before the two fixes on top of it. Five surviving intentional sequential awaits re-pointed to their current coordinates (verified by enclosing function, not line arithmetic): browser-launch runAttempt 516->594, readFile 938->1032, closeRemoteDevToolsTarget 1008->1102, fetchRemoteDevToolsTargets 1016->1110, and interaction-handler 119->174. None referenced deleted code. The interaction-handler entry was mis-categorized before the rebase: line 174 is respondViaTerminal's per-credential-field prompt loop, not a retry/poll loop. Recategorized as shared_mutable_accumulator with an accurate note. Seven test-fixture stub connectors added by this PR were never listed. They emit protocol messages in deliberate order, so they take ordered_protocol_emission, except the stub connector's page fetch, which is genuine cursor-dependent pagination. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…cenario fixtures Two failures from the 8-20 reorg moving modules into @pdpp/connector-protocol. Both reproduce off-CI, so they are branch defects rather than CI flake. connector-init scaffolded connectors importing ../../src/pdpp-safe-text.ts, which no longer exists in this package. Now emits the bare specifier real connectors use (connectors/anthropic/schemas.ts:26). The scaffolder's other relative imports still resolve and are untouched. scenario-cli.test.ts wrote generated connector fixtures to os.tmpdir() and imported src/connector-runtime.ts by absolute path. That module imports @pdpp/connector-protocol, and Node resolves package exports by walking up from the importing file, so from /tmp it never reaches this package's node_modules: the subprocess died with ERR_PACKAGE_PATH_NOT_EXPORTED before the assertion ran. The three raw-egress tests were passing their spawn step and failing for a reason unrelated to egress. Only the two fixture writers that import connector-runtime now write into the package's gitignored tmp/; stub providers importing node: builtins keep using os.tmpdir(). No assertion changed -- the egress tests pass because the guard in scenario/subprocess-fetch-preloads.ts actually fires and the canary server records no hit. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…d of guessing Two live runs failed with evidence in hand and the wrong claim printed. Reddit served its own network-security interstitial and heb served an Imperva JSON block body; both printed a generic "page may have changed" message because detectCloudflareChallenge correctly found no Cloudflare signals and nothing else covered the shape. Evidence collected, claim mislabeled -- the defect class this tooling exists to eliminate, inside the tooling itself. Adds detectProviderBlockInterstitial alongside detectCloudflareChallenge, at the layer where page content is actually available. Deterministic predicates only: Imperva requires both an errorCode field and an incident/proxy id, so a page merely mentioning errorCode cannot false-positive. Messages name the signals that fired and cite the captured artifact; they never speculate about why (IP reputation vs fingerprint vs rate limit are indistinguishable from this evidence). No match falls back to the unchanged generic message. heb's isIncapsulaBlocked also gained the JSON-body predicate: its original empty-shell heuristic requires an iframe, which a raw JSON error body wrapped by page.content() does not have. Reproduced directly before fixing. Not added to classifyFailureEnvironment: that taxonomy sees only checkpoint URLs and DOM byte sizes, and answers whether the browser environment broke -- a different question from what the page said. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…n fallback too Same defect 90e5da5 fixed for reddit and heb, in the third consumer of detectCloudflareChallenge: fallbackForUnexpectedLoginUi consulted only the Cloudflare detector, so a non-Cloudflare provider block printed a generic "login inputs were not found" message while the page content proved a block. Wired to the same template: Cloudflare still wins when it fires, a provider block names its signals and cites the captured artifact, and the generic fallback is byte-identical when neither detector fires. Assistance and completeAssistance handoff semantics are unchanged -- only the diagnostic message text differs. Tests assert the message never speculates about why. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
The recorder patches globalThis.fetch inside the connector process, which structurally cannot observe traffic leaving from a browser page. That is 29 of 45 connectors. This adds the replay half of a network-layer HAR driver, using the extension points the format already reserved: a new driver in environment.network, its own DRIVER_EVIDENCE_POLICIES entry, and no change to the recorded-http path. Network layer, not DOM layer: DOM-layer capture (rrweb) has disqualifying evidence fidelity -- canvas blank without a plugin, multi-tab streams unmerged, privacy-masked subtrees irreversibly excluded -- and network layer matches the existing recorded request/response shape, so one scenario format spans both drivers. Replay pins the fixture envelope, not just the HAR: captured storage state (a warm profile means the app's own JS takes the login-wall path without it), both the in-page and connector-process clocks (connectors compute date-window request params, so an unpinned clock misses the HAR on any later day), and serviceWorkers: "block" (service workers can bypass routeFromHAR and reach the real network). routeFromHAR uses notFound: "abort" -- an unmatched request fails loudly rather than silently reaching a provider. Honesty properties, both enforced rather than documented. A browser scenario can never reach the canonical recorded_replay claim: browser replay proves data mapping (recorded responses -> emitted records), not page choreography, which anti-bot JS and timer nondeterminism make non-deterministic. And every passing browser replay carries a staleness limitation naming its capture timestamp, typed as a template literal and living inside the claim object, so no caller can quote a PASS without the disclaimer riding along. There is no auto-re-record path; re-capture is an explicit human act. Also corrects a limitation that was itself a claim exceeding its evidence: a recorded-browser scenario was reported as "environment driver not declared for every run" when it plainly declares one. Declared-but-not-canonical is now its own limitation; the original string and its legacy tests are unchanged. WebSocket/SSE frames and download bodies are not captured by HAR. Verified unexercised across all 45 connectors, and documented as named boundaries. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
The record half of the recorded-browser driver. Opt-in via --record-har (and explicit typed launch options); when omitted, no recordHar key is added to the launch options at all, so normal connector runs are unchanged. Captures the fixture envelope the replay side needs: the HAR itself with content: "embed" (routeFromHAR needs real response bodies), plus a storage-state snapshot written to its own file, since replay from a cold context would take the login-wall path and miss every subsequent request. Secret hygiene, with the residual exposure stated rather than implied. Redacted from the HAR: Cookie/Set-Cookie/Authorization/x-csrf/x-xsrf headers and the structured cookie arrays on both sides, and form-encoded POST bodies whose field names match password/secret/token/otp/pin. NOT redacted: response and request body content, which is an opaque per-provider blob this layer cannot safely field-redact. The storage state is deliberately unredacted -- the session cookie value is precisely what replay needs, and blanking it would break replay while removing no real risk given the HAR's own cookies are handled. Both artifacts are privacy_class local-only: never share or commit without a manual scrub. environment.network is stamped recorded-browser only when the HAR actually flushed with nonzero entries and the storage state flushed -- never merely because the flag was passed, matching this file's existing "never a constant" posture for evidence claims. Playwright buffers the HAR in memory and writes on context close, so a crash leaves no HAR and no scenario rather than a truncated one. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…captures storage state Found on a real reddit capture: the HAR flushed with 50 entries but no storage state was written, so resolveRunEnvironment correctly refused to stamp recorded-browser and the scenario came out as recorded-http with zero interactions. Patchright ties a persistent context's CDP transport to its pages. Closing the connector's last page tears the context down enough that a later context.storageState() throws "Target page, context or browser has been closed" -- even though context.close() has not run yet. runInBrowser's teardown closed the page before release(), whose storageState() read happens just before context.close(), so a recording run always lost storage state silently: writeStorageStateBestEffort swallows the error to the subprocess's own stderr, which scenario-record discards on a successful run. HAR survived because network-event buffering is independent of page lifecycle, which is why the two artifacts diverged instead of failing together. shouldCloseBrowserPageAfterRun now returns false while recording. This is a no-op in shape: release()'s own context.close() closes the same page a moment later. Non-recording runs are unchanged. The env-var names are mirrored as string literals rather than imported, to keep patchright out of a fetch-only connector's load path. A rename on either side would silently make the check under-detect and reintroduce this exact bug with every behavioral test still passing, so a test pins the mirrors to browser-launch's exported source of truth. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
…atchright scenario-verify against a recorded-browser scenario died with ERR_MODULE_NOT_FOUND: patchright before any of the preload's patching ran. writeBrowserHarReplayPreload wrote its generated module into the evidence workspace, which is mkdtemp'd under os.tmpdir(). Node resolves bare specifiers by walking up from the importing file, so from /tmp that walk reaches / without finding this package's node_modules. The fetch replay preload never hit this because it imports only node: builtins. Same defect class as 2b674fd, and the same house fix: write into the package's gitignored tmp/, still 0600, with explicit cleanup since nothing else owns this file. Considered baking patchright's absolute path into the generated source instead and rejected it: that freezes one install layout into generated code, and a different one (flat node_modules, other pnpm hoist settings, Yarn PnP) could leave the path stale or inconsistent with patchright's own asset lookups. Relocating lets ordinary resolution run fresh in every environment. Deliberately narrower than 2b674fd: only the preload moved, since the isolation workspace still owns HAR/storage-state resolution and the UDS bridge socket. This closes the record->replay round trip, which had never run end to end. A hermetic loopback fixture now records (recorded-browser, 2 HAR entries) and replays to PASS with the stub server down. Egress denial is proven by the case that actually distinguishes it: deleting a HAR entry while a reachable canary is bound to the same origin yields a replay FAIL with zero canary hits, since a generic fetch error alone cannot tell aborted from unreachable. Assisted-by: AI Signed-off-by: Tim Nunamaker <tnunamak@gmail.com>
1 participant
Rebased onto main
cbf598a24on 2026-08-20; true delta is 5 commits / 73 files (content-selected replay — see disposition report for what was carried vs. deferred vs. skipped).Settled state — exact head
f6a1eea43(33 commits), substantive suites green head-exact (polyfill-connectors and reference-implementation via dispatched runs after a push-trigger drop; prior content-identical head92f6ea8f4was 11/11 green including the full suite). Based oncodex/uat-health-candidate-0812(#125 train); retargets to main when #125 merges.What this delivers
The connector verification loop: evidence-claims vocabulary (
docs/reference/connector-evidence-claims.md), run-and-watch (bin/connector-dev.ts,pdpp.run-summary/1), scenario record/strict-replay (pdpp.connector-scenario/1,bin/scenario-record.ts/bin/scenario-verify.ts), fixture provenance labels + enforcement,connector-initscaffolding,observe-schemaobservation reports,generate-validatorsspike, and workflow docs. Proven end-to-end on the unmodified oura and spotify connectors against synthetic in-test providers (synthetic-spikeevidence class by the tools' own grounding rules — the firstnon_loopback_contact_observedcapture requires a machine with real accounts).Verification at head
Claim eligibility (all conditions independent, each with its own limitation string and negative test)
Canonical
recorded_replayrequires: registered connector identity · capture-time declaration digest · capture-time source digest · current declaration digest computed · current source digest computed · supported environment driver on every run · protocol trace on every run · OS-namespace descendant isolation · no unsupported evidence surface (ASSISTANCE) · driver evidence (recorded-http: ≥1 recorded interaction; consumption enforced by the matcher). Anything less:diagnostic_replay: PASS/recorded_replay: WITHHELD/ named limitations, plus machine-readableclaim:andscenario status: candidate oraclelines.Wire and trace contract
Untrusted subprocess output is validated before normalization: unknown message types reject on record and verify sides; RECORD requires stream/key/object-data/emitted_at with the
opdirective validated, captured, and compared (delete↔upsert mutations fail); INTERACTION prompts compare field-exactly (kind/message/canonical schema/timeout;request_idvolatile); RECORD ops are mandatory in the v1 format and STATE has symmetric wire validation.TRACE_POLICYdispositions all 13 runtime message kinds under asatisfiesclause (compile-time exhaustive); tracked kinds get field-level dispositions (compared / digested / excluded-volatile) with continuation evidence via the runtime's own validator (parity-tested on 16 curated malformed cases). Evidence labels state observed facts only:non_loopback_contact_observeduntil provider-authority policy exists;author_livewithheld by all tooling; secrets redact by default (--persist-otpopt-in).Review history
Seven independent external review rounds plus two internal adversarial passes; every P0–P2 across all rounds repaired in-branch with per-finding disposition comments. Every strictness increase caught at least one real pre-existing bug (test-order dependence, silent trace field drops, JSON-path rooting mismatch, untracked fixture labels).
Honestly deferred (per the reviews' own disposition tables)
Provider-authority policy (labels already narrowed so nothing overclaims meanwhile) · browser HAR replay driver · gmail IMAP seedable fake · subprocess-output driver (slackdump/gmcli class) · overlap/dedup/idempotence scenario fixtures · built-package digest binding (publication pipeline) · response-body field-level secret handling (scenarios remain local-only).
Assisted-by: AI