Migrates Dev Proxy to Kestrel - #1735
Merged
garrytrinder merged 47 commits intoJun 29, 2026
Merged
garrytrinder merged 47 commits into
garrytrinder merged 47 commits into
Conversation
Validates selective decrypt, non-destructive ClientHello ALPN peek, h2-only/gRPC blind-tunnel, keep-alive isolation, and unbuffered streaming. Not part of DevProxy.sln. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce engine-agnostic Dev Proxy model that will replace the leaked Unobtanium/Titanium types in the plugin SDK: - IHttpHeader/IHeaderCollection/IHttpMessage/IHttpRequest/IHttpResponse and IProxySession (logical SessionId, Respond() mocking primitive), with concrete HttpHeader + HeaderCollection. - Bodies exposed as ReadOnlyMemory<byte> (decompressed-body contract). - Body strategy: BodyMode, BodyCapabilities, BodyModeResolver (pure, reconciles streaming vs full-body plugin access). - ForwardingInvariants documenting the forwarding contract + shared HopByHopHeaders set. - New DevProxy.Abstractions.Tests (xUnit), 33 tests. Purely additive: existing Titanium-based code is untouched and the full solution builds clean. Adapter wiring follows in Phase 2/3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduces DevProxy.Proxy.Titanium, the sole project referencing Unobtanium.Web.Proxy, which projects Titanium request/response/header/session types onto the engine-agnostic canonical model in DevProxy.Abstractions. - TitaniumHeaderCollection, TitaniumHttpMessageAdapter (shared DRY base over RequestResponseBase), TitaniumRequestAdapter, TitaniumResponseAdapter, TitaniumProxySession. - Body access guarded by HasBody (Titanium throws BodyNotFoundException otherwise); mutation routed through the session's SetRequestBody/ SetResponseBody since Titanium exposes no public body setter. - DevProxy.Proxy.Titanium.Tests: 38 adapter-fidelity unit tests. Purely additive and ship-safe: the existing engine and plugins are untouched. Full solution builds 0/0; 71 tests green. Cross-engine parity tests are deferred to Phase 4 (require the Kestrel engine to exist). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n (Phase 3 slice 1) Introduces the dual-field strangler that lets plugins migrate off Titanium types one at a time while staying green: - ProxyHttpEventArgsBase exposes IProxySession ProxySession alongside the existing Titanium SessionEventArgs Session. HasRequestUrlMatch now reads the canonical request. - ProxyEngine builds a TitaniumProxySession (keyed by the existing per-session GetHashCode key) and threads it through the request/response arg construction sites; DevProxy host references DevProxy.Proxy.Titanium. - CachingGuidancePlugin fully migrated to e.ProxySession.Request and is now Titanium-free. Verified live: the cache-window warning fires through the real proxy pipeline reading IHttpRequest.RequestUri via the adapter. Full solution builds 0/0; 71 tests green. The other 21 Titanium-touching plugins are untouched and keep compiling against Session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add canonical IHttpRequest overloads alongside the existing Titanium Request overloads for IsGraphRequest/IsSdkRequest/IsGraphBetaRequest/BuildGraphResponseHeaders. Extract a shared BuildGraphResponseHeadersCore to keep both overloads DRY. Additive only; Titanium overloads remain until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrate ODSPSearchGuidancePlugin, GraphSelectGuidancePlugin, GraphSdkGuidancePlugin, and GraphClientRequestIdGuidancePlugin to the canonical model: read request/response via e.ProxySession, helpers take IHttpRequest, header checks via Contains. Each plugin is now Titanium-free in source (LoggingContext still takes e.Session until the final wave). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrate AuthPlugin, CrudApiPlugin, GraphMockResponsePlugin, MockResponsePlugin, and OpenAIMockResponsePlugin to the canonical model: request/response via e.ProxySession, GenericResponse -> IProxySession.Respond, canonical HttpHeader, header methods via the canonical IHeaderCollection. Helper signatures and the CrudApi action-handler delegate now use IProxySession/IHttpRequest/ProxyRequestArgs. LoggingContext still takes e.Session until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…treaming off Titanium Change HttpUtils.IsStreamingResponse/GetBodyFromStreamingResponse to take the canonical IHttpResponse. Migrate OpenAIUsageDebuggingPlugin fully to e.ProxySession (status comparisons cast HttpStatusCode to int). Point OpenAITelemetryPlugin's streaming call site at e.ProxySession.Response. DevToolsPlugin and the OpenAITelemetry RequestLog.Context.Session readers remain for the final LoggingContext wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…s off Titanium Flip ThrottlerInfo.ShouldThrottle delegate from Func<Request,...> to Func<IHttpRequest,...> (PluginEvents) and migrate the coordinated consumers: GraphUtils.BuildThrottleKey, RateLimitingPlugin, GraphRandomErrorPlugin, GenericRandomErrorPlugin, LanguageModelRateLimitingPlugin, RetryAfterPlugin. RetryAfter's GenericResponse becomes IProxySession.Respond with canonical HttpHeader. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… wave) Complete the strangler migration: flip LoggingContext and the proxy event args to the engine-agnostic IProxySession model, migrate all remaining plugins off Titanium.Web.Proxy types, and drop the Unobtanium.Web.Proxy package reference from DevProxy.Abstractions and DevProxy.Plugins. - LoggingContext/RequestLog now carry IProxySession (was SessionEventArgs); ProxyHttpEventArgsBase no longer exposes the Titanium Session. - All Behavior/Generation/Guidance/Inspection/Manipulation/Mocking/Reporting plugins read e.ProxySession.Request/Response and the canonical header API. - DevToolsPlugin re-keys _responseBody/CDP requestId on the phase-stable ProxySession.SessionId instead of the Titanium Request hash code. - Add Url setter and response HttpVersion to the canonical model (+ Titanium adapters) to preserve Rewrite and HAR behavior. - Remove dead Titanium FuncExtensions helper. DevProxy.Abstractions and DevProxy.Plugins are now Titanium-free; only the host engine and the Titanium adapter retain the dependency. Build 0/0, 33+38 tests green, live smoke test verified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce DevProxy.Proxy.Kestrel: a forward-proxy engine built on ASP.NET Core Kestrel that runs the Dev Proxy plugin pipeline against the canonical HTTP model, as the eventual replacement for the Titanium engine. Slice-1 scope (dev-toggle via DEV_PROXY_ENGINE=kestrel, for golden-output comparison against Titanium; not a shipped fallback): - KestrelProxyEngine hosts a raw TCP endpoint via UseConnectionHandler. - ProxyConnectionHandler: HTTP/1.1 parse, CONNECT->SslStream MITM, forward, write-back; catches client disconnect/cancel as normal teardown. - PluginPipeline mirrors ProxyEngine semantics (watched/mocked/notwatched, Before/After request+response, per-exchange SessionData) and emits request logs inside the requestId scope so the console formatter groups plugin and engine lines identically to Titanium. - UpstreamForwarder/ResponseWriter apply ForwardingInvariants (hop-by-hop stripping, decompressed bodies, Content-Length/Encoding fix-up). - In-memory CertificateAuthority + DuplexPipeStream ported from the POC. Engine selection wired via DEV_PROXY_ENGINE in DI; ProxyConsoleFormatter suppresses the engine-name prefix for KestrelProxyEngine (as for ProxyEngine). Deferred hardening (tracked): keep-alive, blind-tunnel for non-watched and h2-only/gRPC (ClientHello peek), WebSocket relay, chunked/trailers/100-continue, body-mode streaming, process filter, persistent CA, host-watch DRY consolidation. Full solution builds 0/0; 71 existing tests green; live-verified plain HTTP passthrough, HTTPS MITM, GenericRandomError/RetryAfter mocking with correct console output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add DevProxy.Proxy.Kestrel.Tests with 28 unit tests covering the HTTP/1.1 parser, host watch list (including order-dependent exclusion semantics), response writer, and the mutable HTTP model / CanonicalProxySession. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
At CONNECT time the engine now non-destructively peeks the TLS ClientHello (SNI + ALPN) and decides whether to intercept: - non-watched hosts are blind-tunnelled (encrypted bytes relayed, never MITM'd) - watched hosts whose client is h2-only (gRPC, no http/1.1 fallback) are blind-tunnelled so they stop failing - otherwise the engine terminates TLS advertising http/1.1 only, so h2-capable clients downgrade and are intercepted as HTTP/1.1 Adds a minimal tolerant TlsClientHello parser, wires HostWatchList into the connection handler, and relays blind tunnels bidirectionally with a linked cancellation token so both copy tasks finish before the streams are disposed. Covered by 9 new TlsClientHello unit tests (37 Kestrel tests total). Verified live: watched host downgrades h2->http/1.1 and is intercepted; example.com (non-watched) validates its real cert without -k, proving it is never MITM'd. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… 4 slice 3) Replace the static one-shot HTTP/1.1 reader with a stateful per-connection Http1ConnectionReader that owns the inter-request buffer, so bytes read past one request's body (a pipelined next request) are retained instead of dropped. This makes client-side keep-alive correct. - Http1ConnectionReader: stateful ReadHeadAsync/ReadBodyAsync owning _pending. - Http1RequestReader: reduced to stateless parse helpers (ParseHead, GetContentLength, IndexOfDoubleCrlf). - ProxyConnectionHandler.ServeConnectionAsync: keep-alive loop over both plain HTTP and decrypted CONNECT streams; each exchange builds a fresh CanonicalProxySession (new GUID + Interlocked requestId) so plugin state never leaks across requests on a reused connection. - ShouldKeepAlive (RFC 9112 9.3): persistent by default on HTTP/1.1, opt-in on HTTP/1.0; forces Connection: close for Connection: close, Transfer-Encoding (chunked, not yet reframable) and Expect: 100-continue (unsupported), so an unframable body can never corrupt the next request. - ResponseWriter.WriteAsync: keepAlive flag drives the Connection header. Tests: +15 Kestrel (stateful reader incl. pipelining/surplus-retention, ShouldKeepAlive matrix, keep-alive/close response-writer). 123 green total. Verified live: two URLs in one curl invocation reuse one TCP connection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Watched-host WebSocket upgrades (ws:// plain + wss:// after MITM) now relay through the Kestrel engine for parity with Titanium, which relays WS opaquely (no plugin sees frames). HttpClient can't carry a 101 upgrade, so WebSocketRelay opens its own raw socket to the origin (TLS w/ http/1.1 ALPN for wss), replays the handshake in origin-form (preserving Upgrade/Connection/Sec-WebSocket-*, stripping Proxy-*), writes the origin's 101 back to the client verbatim, then splices frames raw via a shared StreamRelay (also now used by BlindTunnelAsync, DRY). Critical ordering: the response pipeline (which flushes the buffered req-log) runs in the onHandshakeResponse callback, right after the 101 and before the splice blocks, else a watched WS never logs until the socket closes. A handshakeObserved fallback flush covers origin-connect failure. Tests: ParseResponseHead unit tests, StreamRelay bidirectional copy, and an end-to-end RelayAsync test against a loopback fake origin (asserts origin-form handshake replay, header preservation, 101 verbatim write-back, frame splicing, and onHandshakeResponse invocation). 128 tests green (+5 Kestrel). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…slice 5) Closes the biggest remaining HTTP/1.1 correctness gap on the request read path. Previously a chunked request body was silently dropped (Content-Length defaulted to 0, so an empty body was forwarded) and Expect: 100-continue hung (the proxy never sent the interim 100). Response-side chunked already worked via HttpClient. - Http1ConnectionReader.ReadChunkedBodyAsync decodes chunked framing (size lines, chunk extensions, CRLF delimiters) into one buffer, consumes and drops trailers, and retains surplus for the next pipelined request. Shared _pending fill primitives (PullAsync/ReadExactlyAsync/ReadLineAsync/TakeFromPending), reused by ReadBodyAsync. - Http1RequestReader.DetectBodyFraming (None/ContentLength/Chunked/Conflicting) and HasExpectContinue. A request declaring both Content-Length and chunked Transfer-Encoding is Conflicting and refused with 400 (RFC 9112 6.3.3 smuggling guard). - ResponseWriter.WriteContinueAsync sends 100 Continue; ExchangeAsync answers Expect: 100-continue itself before reading the body (the buffering model can't relay the origin's interim 100). Expect is stripped on forward. - ShouldKeepAlive no longer forces close for chunked/Expect; malformed chunked framing replies 400 and closes. 146 tests green (+18). Verified live through the MITM proxy: a chunked upload reached httpbin with the body intact and reframed to Content-Length (no Transfer-Encoding), the client got 100 Continue, two chunked POSTs reused one TCP connection, and a raw Content-Length+Transfer-Encoding request got 400. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Forward text/event-stream responses to the client piece-by-piece (re-framed as HTTP/1.1 chunked) instead of buffering the whole body. Buffering withheld every event until the stream ended and hung the engine on unbounded streams. - UpstreamForwarder.ForwardAsync returns an OriginResponse: buffered for normal responses, or a live body stream for text/event-stream (message stays open). - StreamingResponseWriter pumps origin->client one chunk per read (flushed so events arrive live), accumulating a capped copy (4 MiB, reusing BodyModeResolver.DefaultInMemoryLimitBytes); beyond the cap accumulation is dropped but relaying continues, so the engine never hangs or OOMs. - PluginPipeline.RunStreamingResponseAsync runs the body pump between BeforeResponse and AfterResponse, exposing the accumulated body to read-only AfterResponse inspectors (OpenAITelemetry/OpenAIUsageDebugging) for parity. - Keep-alive preserved (chunked is self-delimiting). Tests: +12 (StreamingResponseWriter chunk framing/accumulation/cap, forwarder streaming detection). 158 green. Live-verified: local finite SSE streamed ~0.5s per event (not buffered), Transfer-Encoding: chunked, Connection: keep-alive, connection reused across two streamed responses, non-watched passthrough intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rocess-names) Implements the last functional parity gap with the Titanium engine: restricting MITM to specific processes via --watch-pids / --watch-process-names. Matches Titanium semantics exactly — filtering applies ONLY at the CONNECT decision (IsProxiedProcess in ProxyEngine.cs), so plain HTTP stays unfiltered (documented parity quirk). - ProcessFilter resolves the client's PID from its source port and matches against WatchPids then WatchProcessNames (ordinal/case-sensitive HashSet to mirror Titanium's default Contains comparer). No filter configured -> watch all (IsEmpty short-circuit, zero overhead). - ConnectionProcessResolver shells out per-platform: lsof -i on Unix, netstat -ano on Windows. LsofParser/NetstatParser are pure, fully unit-tested partial classes (GeneratedRegex). Chosen over Windows P/Invoke (GetExtendedTcpTable) for testability and boring-tech symmetry. NOTE: the Windows netstat path is parser-tested but not live-verifiable on macOS. - Wired into ProxyConnectionHandler.HandleConnectAsync (after the h2-only check): unresolved/non-matching process -> blind-tunnel (never MITM'd). KestrelProxyEngine constructs the filter from configuration.WatchPids/WatchProcessNames. - Updated the CONNECT-flow doc diagram. Tests: +21 Kestrel (12 ProcessFilter + 4 Lsof + 5 Netstat) -> 179 green (33 Abstractions + 38 Titanium + 108 Kestrel). Live-verified on macOS: matching process name -> MITM intercept + req-log; non-matching -> blind tunnel (real cert, no -k, zero req-log lines). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cache) The Kestrel engine previously minted a brand-new in-memory root CA on every start, so the OS would distrust it after each restart. This makes the root (and per-host leaves) persistent on disk, mirroring the Titanium engine's CertificateDiskCache layout exactly: root at <configDir>/rootCert.pfx (overridable via DEV_PROXY_CERT_PATH), leaves at <configDir>/crts/<host>.pfx, both PKCS#12 with an empty password. configDir matches Titanium's resolution (ApplicationData/dev-proxy on mac/Linux, the executable dir on Windows). EUREKA: because the format + path match, the persistent CA loads a root that the Titanium engine ALREADY created and the OS ALREADY trusts — existing users get interception with zero new trust code (verified live: curl WITHOUT -k against a watched HTTPS host returned 200 + full MITM req-log). - LoadOrCreateRoot: load + validate (isCA, has private key, not expired) the on-disk root; regenerate + re-save (and purge the now-stale leaf cache) if absent/invalid/ expired. Per decision #5 there is no cross-version compatibility contract. - Leaves are minted on demand, cached in memory + on disk, and reused across restarts. Leaf validity is clamped to the issuer's window so reusing a near-expiry root can't produce a leaf that outlives it (CertificateRequest.Create rejects that). Filename- unsafe hosts (IPv6 ':', wildcard '*', '/') are sanitized for the disk cache key. - Disk writes are best-effort: an unwritable location degrades to in-memory-only for the run instead of failing interception. - KestrelProxyEngine now builds the CA via CertificateAuthority.CreateDefault(logger). OS-trust install (mac keychain / Windows root store) + the first-run prompt remain Slice 5b (host-side, reusing the existing trust helpers). This slice only makes the root persistent. Tests: +12 Kestrel (CertificateAuthorityTests: first-run create+persist, reuse across instances, invalid/expired/non-CA regeneration, stale-leaf purge, leaf mint+sign+ persist+reuse, leaf-not-outliving-root clamp, IP-literal SAN, filename sanitization) -> 191 green (33 + 38 + 120). Live-verified on macOS: existing root reused (fingerprint unchanged, "Loaded persisted root certificate"), leaf cached + reused across restarts, MITM works without -k. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… 5b) Wire root-certificate OS trust for the Kestrel engine via a new IRootCertificateTrust abstraction (DevProxy.Abstractions/Proxy), implemented host-side by RootCertificateTrust (DevProxy/Proxy). The engine mints/persists its root (Slice 5a) then calls EnsureTrusted; the host performs the platform install: - macOS: trust in the login keychain via MacCertificateHelper, gated by the first-run flow (InstallCert + NoFirstRun + HasRunFlag + Y/n prompt, auto-"y" under CI/redirected stdin) — mirrors Titanium's FirstRunSetup. - Windows: install the public-only root into CurrentUser/Root, idempotent by identity (CA5380 suppressed — installing the Dev Proxy root is the proxy's explicit, user-consented purpose). NOT live-verifiable on macOS. - Linux: warn to trust manually. The trust decision is factored into a pure RootTrustPolicy.Decide table in Abstractions so it's exhaustively unit-testable without platform/console I/O; the host impl is just the thin I/O around it (option X — new interface, leave Titanium's FirstRunSetup untouched for low blast radius). CertCommand (ensure/remove) already works for Kestrel mode unchanged thanks to Slice 5a's on-disk path parity. The Kestrel engine ctor takes an optional IRootCertificateTrust (null = no trust, e.g. tests); the DEV_PROXY_ENGINE=kestrel DI toggle registers and injects RootCertificateTrust. 207 tests green (49 Abstractions incl. +16 RootTrustPolicy, 38 Titanium, 120 Kestrel). Live-verified on macOS: with an already-trusted persisted root and --no-first-run, a watched HTTPS request returned 200 WITHOUT -k plus a full MITM req-log — trust wiring doesn't regress the existing-root happy path. Fresh-machine keychain trust is covered by the unit-tested decision table (not live-exercised to avoid keychain GUI pollution); Windows store install is parser/logic-only on macOS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract a single ConnectionTeardown.IsExpected(exception) classifier and use it at every client/origin read/copy/write boundary, replacing the per-site hardcoded `ex is IOException or OperationCanceledException or ...` lists. An exception is "expected" (treat as EOF/close, don't log as error) when it's a ConnectionResetException, ConnectionAbortedException, OperationCanceledException, IOException, or SocketException — the normal result of a peer disconnecting, resetting, or the connection being cancelled mid-exchange. Boundaries updated to the shared classifier: - ProxyConnectionHandler.OnConnectedAsync outer catch (keep-alive read, body read, response write, streaming pump, blind tunnel) — now also covers SocketException/ConnectionAbortedException. - StreamRelay bidirectional-copy teardown. - WriteErrorAsync best-effort write to a possibly-gone client. Fixes the one genuine noise gap: the WebSocket relay catch logged a client disconnect mid-handshake / verbatim-101-write as an ERROR. It now logs at Debug when IsExpected and reserves Error for genuine faults. 216 tests green (49 Abstractions, 38 Titanium, 129 Kestrel; +9 ConnectionTeardown covering OCE/TaskCanceled/IOException/SocketException/ConnectionReset/ ConnectionAborted true and InvalidOperation/NullReference/unrelated false). Live-verified on macOS: aborting curl mid-stream (httpbin /drip, --max-time 1) produced ZERO error/exception log lines and the engine kept serving (a following request returned 200). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ce 5d) Replace the naive "split on the last colon" SplitHostPort with a pure, explicit ConnectAuthorityParser that correctly handles: - IPv6 literals: [::1] and [2001:db8::1]:8443 — the bare host (no brackets) is used for socket connect / cert minting, and a UrlHost property re-adds brackets for composing the absolute URL (https://[::1]:8443/path). - Explicit ports, including validation (reject port 0 / >65535 / non-numeric / empty). - Malformed targets: empty host, unterminated bracket, junk after ], bare (unbracketed) IPv6, invalid host names — all rejected. The authority is now parsed + validated BEFORE the proxy writes "200 Connection Established", so a malformed CONNECT is refused with a 400 instead of establishing an unusable tunnel. The decrypted-tunnel URL is composed from UrlHost so IPv6 targets produce valid absolute URLs. 237 tests green (49 Abstractions, 38 Titanium, 150 Kestrel; +21 parser tests covering reg-name/IPv4/punycode/IPv6 with and without ports plus 15 malformed cases). Live-verified on macOS: normal + explicit-:443 watched MITM return 200 (no -k); malformed CONNECT targets (port 0, bare IPv6 ::1:443, unterminated [::1) each return 400 Bad Request while a valid target returns 200 Connection Established; zero error-log noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Log the OS-assigned port (not the configured 0) for the Kestrel engine by reading the bound ListenOptions endpoint after StartAsync, so users running with --port 0 can connect. Extract the duplicated urlsToWatch host-derivation logic into a shared WatchedHostExtractor.ToHostRegex in DevProxy.Abstractions, used by both the Titanium ProxyEngine.LoadHostNamesFromUrls and the Kestrel HostWatchList. One implementation, one test suite (11 new tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Re-implement turning the OS system-proxy on and off without Titanium, behind a new ISystemProxyManager abstraction (Abstractions) so the Kestrel engine can use it without referencing the host (mirrors IRootCertificateTrust): - SystemProxyManager (host): Windows via WinINET (registry ProxyServer/ProxyEnable + InternetSetOption settings-changed/refresh broadcast), macOS via the existing toggle-proxy.sh, Linux warns. DllImport (not LibraryImport) to avoid enabling AllowUnsafeBlocks project-wide. - SystemProxyAddress (Abstractions, pure): normalizes wildcard/empty bind addresses to loopback for the client-facing proxy address; unit-tested. - KestrelProxyEngine: enable on start (using the actually-bound port, matters for --port 0), disable on stop. Previously the Kestrel engine set no system proxy at all — a parity gap. - StopCommand --force crash-cleanup now uses the shared manager, so a force-killed daemon restores the system proxy on Windows too (was macOS-only before). 259 tests green (+11 SystemProxyAddress). Windows WinINET path is parser/logic-tested but needs verification on a Windows host (cannot run on macOS). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Swap the Titanium RunTime OS-check helper for the .NET built-in equivalents (RunTime.IsWindows → OperatingSystem.IsWindows(), RunTime.IsMac → OperatingSystem.IsMacOS()) in ProxyEngine and CertCommand, and drop the now-unused `using Titanium.Web.Proxy.Helpers;` import from both. Strictly equivalent behavior; reduces Titanium API surface ahead of the cut-over. Left untouched (intentionally): - CertificateDiskCache.RunTime.IsUwpOnWindows — no built-in equivalent, and the file (a Titanium ICertificateCache impl) is deleted at cut-over anyway. - The Titanium.Web.Proxy.* log filters — still suppress real Titanium engine logs while it remains the default engine; their removal belongs in the cut-over commit. 259 tests green; Titanium engine boot-smoked clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build the DevProxy.Parity.Tests project that boots the real KestrelProxyEngine on a random localhost port against a deterministic FakeOrigin and asserts the observable proxy contract for every controllable parity-matrix row: - plain HTTP: GET/POST (small/large), large download, status relay, 204 empty-body framing, request-header passthrough - chunked request body reframing + SSE chunked response streaming - keep-alive reuse + Connection: close - plugin Respond() mocking short-circuit (origin never contacted) - Content-Length + Transfer-Encoding smuggling -> 400 guard 15 hermetic rows green. External/OS rows (HTTPS real-cert MITM, h2/gRPC, WebSocket, system-proxy, process filter) remain the scripted live pass and are already slice-verified; see files/parity-results.md for the full matrix. The harness boots only the Kestrel engine (the Titanium engine's process-global static ProxyServer + IServer make in-process co-hosting flaky, and it is deleted at cut-over); it asserts the reference behavior Titanium already exhibits. Total: 274 tests green (71 Abstractions + 38 Titanium + 150 Kestrel + 15 Parity). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make the ASP.NET Core Kestrel engine the sole proxy engine and remove the Unobtanium/Titanium.Web.Proxy adapter, dependency, and dev toggle in one change. - Remove the DEV_PROXY_ENGINE toggle; AddProxyEngine always wires Kestrel, the shared CertificateAuthority, IRootCertificateTrust, and ISystemProxyManager. - Introduce a single shared CertificateAuthority DI singleton (Kestrel AddKestrelCertificateAuthority) with the root X509Certificate2 derived from it; the engine, CertCommand, ProxyController, and EntraMockResponsePlugin all resolve the same root cert. The engine no longer self-creates/disposes its CA. - Add Trust/Untrust to IRootCertificateTrust and implement them host-side (mac keychain / Windows root store / Linux warn); CertCommand uses them. - Delete ProxyEngine, CertificateDiskCache, the DevProxy.Proxy.Titanium and DevProxy.Proxy.Titanium.Tests projects, the Unobtanium PackageReference, the Titanium.Web.Proxy.* log filters, and the THIRD PARTY NOTICES Unobtanium entry. - Update docs (CONTRIBUTING, copilot-instructions) and stale comments to Kestrel. Build 0/0; 236 tests green (71 Abstractions + 150 Kestrel + 15 Parity). Live-verified on macOS: watched HTTPS MITM without -k + req-log, cert-download API, and `cert ensure` trust path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two cut-over regressions surfaced by the macOS system-proxy live test:
1. Detached mode (`--detach`) was broken. The deleted Titanium ProxyEngine wrote
the initial ProxyInstanceState (gated on IsInternalDaemon); the Kestrel engine
never did, so the daemon orphaned, the parent's readiness poll timed out, and
the system proxy was left on. The host's UpdateStateWithApiUrlAsync only updated
an existing state file, so it silently no-opped.
- KestrelProxyEngine now publishes its actually-bound port back to the shared
IProxyConfiguration after binding (resolves --port 0 to the OS-assigned port).
- DevProxyCommand.WriteDaemonStateAsync writes the full ProxyInstanceState
(Pid, ApiUrl, LogFile, ConfigFile, resolved Port, AsSystemProxy) once the
engine has published its port, satisfying the parent's `Port > 0` readiness
contract used by `devproxy stop` and `devproxy status`.
2. Graceful shutdown stalled after real traffic. The engine awaited
app.StopAsync(CancellationToken.None), which waited unbounded for lingering
client keep-alive connections to drain — fast when idle, but >10s after
traffic, making `devproxy stop` falsely report "did not stop in time". The
drain is now bounded (5s) so remaining connections are force-closed.
Adds EnginePortPublishingTests regression coverage (verified failing without the
port write-back). 237 tests green. Live-verified on macOS: `--detach` registers
the daemon, `devproxy status`/`stop` find it, and the system proxy is restored on
both graceful API stop and SIGTERM.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With Titanium removed there is no second engine to compare against, so the "parity" framing no longer applies. The suite is now a black-box integration test that boots the Kestrel engine over a real socket against a FakeOrigin. Renamed the project, csproj, namespaces, *ParityTests classes/files, and the solution entry (project GUID unchanged); reworded "parity" comments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on/mocking plugins Adds a CapturingLoggerFactory (collects engine-emitted RequestLogs), an inline PluginConfig section builder, and extends KestrelProxyHarness to accept a logger factory and expose its urls-to-watch set. Covers 9 hermetic, HTTP-observable plugins end-to-end through the Kestrel engine: GenericRandomError, Latency, RateLimiting, RetryAfter, Rewrite, MockResponse, Auth (reject + allow), and GraphRandomError. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds PluginTestHost (a minimal DI ServiceProvider supplying HttpClient/ILogger<>/ IProxyConfiguration so plugins whose InitializeAsync builds file loaders can run in-process), TestDefaults (shared HttpClient), a FakeOrigin request recorder + /json endpoint, and a settable ConfigFile on TestProxyConfiguration. Mirrors the host's runtime-only package assets in the test csproj so plugin runtime dependencies (OpenIdConnect, JWT, OpenApi, OpenTelemetry, EF Sqlite, Newtonsoft, Azure.Identity) resolve. Covers MockRequestPlugin (outbound fire) and CrudApiPlugin (in-memory REST served through the engine, origin never contacted). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds TestExchange, which constructs the engine's real canonical session/request/ response types so guidance plugins gated on a fixed upstream host (graph.microsoft.com) can be driven at their hooks with production-fidelity inputs the loopback origin can't impersonate. Covers GraphBetaSupport, GraphClientRequestId, GraphSdk, ODataPaging, ODSPSearch, GraphConnector, and CachingGuidance, asserting the advisory log each emits. GraphSelectGuidancePlugin is documented as non-hermetic (needs the Graph metadata DB) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…y, MinimalPermissions) Drives reporter plugins through AfterRecordingStopAsync exactly as the host's recording controller does: feeds engine-shaped RequestLogs (TestExchange.AsRequestLog) into RecordingArgs with a seeded Reports GlobalData slot, then asserts the structured report stored under the plugin name. MinimalPermissions runs against a temp OpenAPI spec fixture to verify offline permission analysis. GraphMinimalPermissions(+Guidance), MinimalCsom, and ApiCenter reporters are documented as live-backend (Bucket 3) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ypeSpec) Drives each generator through AfterRecordingStopAsync with an InterceptedResponse log, asserting the stored report and the generated artifact. Generators write to the process CWD, so tests redirect it to a temp dir; assembly test parallelization is disabled to keep that global swap safe. A DisabledLanguageModelClient keeps the OpenAPI/TypeSpec generators hermetic (LM is cosmetic-only; they fall back to deterministic generation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Spawns the real DevProxy host with a devproxyrc.json + mocks file, waits for the Kestrel engine to report its bound port (--port 0), routes an HTTP request through it as an explicit proxy, and asserts MockResponsePlugin short-circuits with the configured body. Proves end-to-end that config-driven plugin discovery + the engine wire survive the Titanium -> Kestrel cut-over. Uses a plain-HTTP watched URL so no CA trust is needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The BodyModeResolver + BodyMode/BodyCapabilities/BodyContext/BodyDirection types (319 LOC, 12 tests) were a spike-era design that the Kestrel engine never wired: Resolve() is never called in production, no plugin declares BodyCapabilities, and IHttpMessage exposes no capability property. The engine uses a fixed 4 MiB in-memory inspection cap instead. Worse, the IHttpMessage doc claimed body buffering was 'driven by plugin BodyCapabilities', misleading plugin authors toward an API that does not exist. - Delete BodyModeResolver.cs, BodyHandling.cs, and BodyModeResolverTests.cs. - Inline the one used constant as ProxyConnectionHandler.InMemoryInspectionCapBytes. - Correct the IHttpMessage body-visibility doc to describe actual behavior. - Drop stale 'every engine adapter' language (abandoned multi-engine design) from ForwardingInvariants and HeaderCollection docs. The design is preserved in git and can be restored from this parent commit if/when capability-based body handling is actually needed. A code comment + plan.md follow-up track the genuine, separate unbounded-buffered-body memory risk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The file's 'I' prefix implied an interface, but it contains no interface — only the MessageType enum and the LoggingContext / StdioLoggingContext classes. Rename to match its primary type and theme. (StdioEvents.cs was left as-is: it correctly groups the stdio event-arg family, mirroring the existing ProxyEvents.cs convention.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Net-new capability unlocked by the Kestrel engine: mock WebSocket frames end-to-end. The engine owns transport (101 handshake + framing via the framework's WebSocket.CreateFromStream); the plugin owns behavior (on-connect scripted sends + match incoming client messages by equals/contains/regex/json and reply with mapped messages, plus optional close). A watched WS upgrade is dispatched to the plugin responder (instead of the origin relay) when the plugin calls IProxySession.HandleWebSocket(handler), leaving the session in the Watched phase. The plugin must NOT set ResponseState.HasBeenSet, which would short-circuit to the mocked-HTTP ResponseWriter path before the WS dispatch branch runs. DRY scheme-normalization footgun fix: the engine reports an intercepted WS upgrade with an http(s) scheme, but authors naturally write wss://host/* in urlsToWatch and mock URLs. A shared ProxyUtils.NormalizeWebSocketScheme normalizes wss->https / ws->http and is applied at both the watch-regex build site (PluginServiceExtensions.ConvertToRegex) and the plugin's mock-URL match, so either form works and a wss:// watch entry is no longer silently relayed to origin. Tests: responder loopback unit tests, matcher unit tests, 2 e2e ClientWebSocket-over-wss-MITM integration tests, and NormalizeWebSocketScheme unit tests. Full suite green (284). Live-verified through the host binary with urlsToWatch ["wss://ws.example.test/*"]: OPEN -> welcome (onConnect) -> pong (reactive ping->pong); proxy logged "Mocking WebSocket" + "WebSocket mock established". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The WebSocketMockResponsePlugin invented a `matchType: equals|contains|regex| json` enum, diverging from every other Dev Proxy mock plugin, which matches with the `bodyFragment` (case-insensitive contains) + `bodyRegex` field pair. Replace the enum with the conventional vocabulary so authors familiar with the HTTP/stdio mocks reach for the same fields: - `bodyFragment` — OrdinalIgnoreCase substring - `bodyRegex` — regular expression (1s timeout) - `bodyJson` — structural JSON equality (JsonNode.DeepEquals); accepts a JSON value directly in the mocks file, matching how the HTTP response body accepts object/array/string Precedence when more than one is set: bodyJson > bodyRegex > bodyFragment (mirrors stdio's "bodyRegex wins over bodyFragment"). Null match or no criteria remains a catch-all. The genuinely WS-specific structure (url-keyed mock, onConnect pushes, reactive rules, multi-message responses) is unchanged — only the match expression is aligned. Updates the mocks-file schema and matcher unit tests accordingly. Full suite green (283). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ut-over The hard cut-over to the Kestrel engine deleted ProxyEngine.cs, which owned the interactive key loop (r/s/c/w) and the --record auto-start. KestrelProxyEngine never re-implemented them, so the documented keyboard shortcuts (technical-reference/keyboard-shortcuts.md) stopped working and `devproxy --record` no longer started recording at launch. Restore both as a host-side InteractiveConsoleService (BackgroundService), keeping the Kestrel engine headless: the engine depends only on the proxy abstractions and must not reach into host concerns like IProxyStateController. The service waits for ApplicationStarted so the banner prints after the engine's startup logs, honors --record, prints hotkeys (text mode) or API instructions (JSON mode), and dispatches keys via a testable ConsoleHotkeyHandler over an ISystemConsole seam. Adds DevProxy.Tests (the host had no unit test project) with 10 tests covering key dispatch and both banners. Full suite green (293 tests). Live-verified: --record starts recording at launch, JSON banner prints, proxy serves 200 through the new service. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Windows-specific runtime paths (WinINET system proxy, current-user root-store cert trust, netstat-based process filter) are unit-tested but were never live-verified on a real Windows host after the Kestrel cut-over. Add a self-contained, clone-and-run checklist covering build/tests, cert trust, system proxy on/off, detached daemon lifecycle, process filter, the restored interactive console, and a core proxy + plugin smoke pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A HEAD response carries the Content-Length a GET would return but no body (RFC 9110 §9.3.2). The engine was recomputing Content-Length from the (empty) HEAD body, so clients saw Content-Length: 0 instead of the real resource size. ResponseWriter and UpstreamForwarder are now method-aware: HEAD preserves the origin's Content-Length, never takes the streaming path, and never writes a body.
Resolve the OS-assigned API port from the bound server address before printing the JSON-mode banner so the curl commands reference the actual port instead of the literal 0.
…ction The forward catch excluded all OperationCanceledException so client cancellation would bubble up as a silent teardown. But an HttpClient request timeout also throws TaskCanceledException (an OperationCanceledException), so a stalled origin was misclassified as a client disconnect: the client got a dropped/reset connection after the default 100s timeout instead of a gateway error. Add a pure UpstreamFailure.Classify helper (mirroring ConnectionTeardown) that tells the two apart by the connection token: OperationCanceledException with the token NOT cancelled is an upstream timeout -> 504 Gateway Timeout; with the token cancelled it is a genuine client teardown -> silent; anything else (TLS handshake failure on an invalid upstream cert, DNS, connection refused) -> 502 Bad Gateway. Wire it into the single forward catch in ProxyConnectionHandler and add the 504 reason phrase. Closes the "upstream timeout" and "invalid upstream cert" parity rows in code with 7 unit tests on the classifier. Full suite green (306 tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The test deliberately constructs a NullReferenceException to prove the teardown classifier does not mask a latent bug as an expected connection close. CA2201 flags constructing reserved exception types, so scope a justified suppression to that single line instead of weakening the test. Solution now builds with 0 warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This is a major version bump because the migration from Titanium/Unobtanium to the Kestrel-based proxy engine introduces breaking changes: - Plugin API surface moved from Titanium types (SessionEventArgs/Request/ Response/HttpHeader) to the canonical model (IProxySession/IHttpRequest/ IHttpResponse/HeaderCollection). Third-party plugins must recompile. - Behavioral changes around HTTP/2 handling (h2-only/gRPC blind-tunnel). Copies schemas/v3.1.0 -> schemas/v4.0.0 verbatim (41 files) and updates <Version> in all four shipping projects (incl. DevProxy.Proxy.Kestrel), schema URLs in config JSON + MockResponsePlugin, installer/script/Dockerfile version strings, and refreshes packages.lock.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
garrytrinder
approved these changes
Jun 29, 2026
waldekmastykarz
added a commit
that referenced
this pull request
Jul 4, 2026
* Add throwaway Kestrel proxy spike (Phase 0B de-risking) Validates selective decrypt, non-destructive ClientHello ALPN peek, h2-only/gRPC blind-tunnel, keep-alive isolation, and unbuffered streaming. Not part of DevProxy.sln. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add canonical HTTP proxy model to DevProxy.Abstractions (Phase 1) Introduce engine-agnostic Dev Proxy model that will replace the leaked Unobtanium/Titanium types in the plugin SDK: - IHttpHeader/IHeaderCollection/IHttpMessage/IHttpRequest/IHttpResponse and IProxySession (logical SessionId, Respond() mocking primitive), with concrete HttpHeader + HeaderCollection. - Bodies exposed as ReadOnlyMemory<byte> (decompressed-body contract). - Body strategy: BodyMode, BodyCapabilities, BodyModeResolver (pure, reconciles streaming vs full-body plugin access). - ForwardingInvariants documenting the forwarding contract + shared HopByHopHeaders set. - New DevProxy.Abstractions.Tests (xUnit), 33 tests. Purely additive: existing Titanium-based code is untouched and the full solution builds clean. Adapter wiring follows in Phase 2/3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Titanium adapter mapping engine types onto canonical model (Phase 2) Introduces DevProxy.Proxy.Titanium, the sole project referencing Unobtanium.Web.Proxy, which projects Titanium request/response/header/session types onto the engine-agnostic canonical model in DevProxy.Abstractions. - TitaniumHeaderCollection, TitaniumHttpMessageAdapter (shared DRY base over RequestResponseBase), TitaniumRequestAdapter, TitaniumResponseAdapter, TitaniumProxySession. - Body access guarded by HasBody (Titanium throws BodyNotFoundException otherwise); mutation routed through the session's SetRequestBody/ SetResponseBody since Titanium exposes no public body setter. - DevProxy.Proxy.Titanium.Tests: 38 adapter-fidelity unit tests. Purely additive and ship-safe: the existing engine and plugins are untouched. Full solution builds 0/0; 71 tests green. Cross-engine parity tests are deferred to Phase 4 (require the Kestrel engine to exist). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wire canonical IProxySession through ProxyEvents; migrate first plugin (Phase 3 slice 1) Introduces the dual-field strangler that lets plugins migrate off Titanium types one at a time while staying green: - ProxyHttpEventArgsBase exposes IProxySession ProxySession alongside the existing Titanium SessionEventArgs Session. HasRequestUrlMatch now reads the canonical request. - ProxyEngine builds a TitaniumProxySession (keyed by the existing per-session GetHashCode key) and threads it through the request/response arg construction sites; DevProxy host references DevProxy.Proxy.Titanium. - CachingGuidancePlugin fully migrated to e.ProxySession.Request and is now Titanium-free. Verified live: the cache-window warning fires through the real proxy pipeline reading IHttpRequest.RequestUri via the adapter. Full solution builds 0/0; 71 tests green. The other 21 Titanium-touching plugins are untouched and keep compiling against Session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 Wave 1: add IHttpRequest overloads to ProxyUtils Add canonical IHttpRequest overloads alongside the existing Titanium Request overloads for IsGraphRequest/IsSdkRequest/IsGraphBetaRequest/BuildGraphResponseHeaders. Extract a shared BuildGraphResponseHeadersCore to keep both overloads DRY. Additive only; Titanium overloads remain until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 guidance wave: migrate 4 guidance plugins off Titanium Migrate ODSPSearchGuidancePlugin, GraphSelectGuidancePlugin, GraphSdkGuidancePlugin, and GraphClientRequestIdGuidancePlugin to the canonical model: read request/response via e.ProxySession, helpers take IHttpRequest, header checks via Contains. Each plugin is now Titanium-free in source (LoggingContext still takes e.Session until the final wave). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 mocking wave: migrate 5 mocking plugins off Titanium Migrate AuthPlugin, CrudApiPlugin, GraphMockResponsePlugin, MockResponsePlugin, and OpenAIMockResponsePlugin to the canonical model: request/response via e.ProxySession, GenericResponse -> IProxySession.Respond, canonical HttpHeader, header methods via the canonical IHeaderCollection. Helper signatures and the CrudApi action-handler delegate now use IProxySession/IHttpRequest/ProxyRequestArgs. LoggingContext still takes e.Session until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 inspection-lite: migrate HttpUtils + OpenAI usage/telemetry streaming off Titanium Change HttpUtils.IsStreamingResponse/GetBodyFromStreamingResponse to take the canonical IHttpResponse. Migrate OpenAIUsageDebuggingPlugin fully to e.ProxySession (status comparisons cast HttpStatusCode to int). Point OpenAITelemetryPlugin's streaming call site at e.ProxySession.Response. DevToolsPlugin and the OpenAITelemetry RequestLog.Context.Session readers remain for the final LoggingContext wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 throttling group: migrate ThrottlerInfo + 5 throttling plugins off Titanium Flip ThrottlerInfo.ShouldThrottle delegate from Func<Request,...> to Func<IHttpRequest,...> (PluginEvents) and migrate the coordinated consumers: GraphUtils.BuildThrottleKey, RateLimitingPlugin, GraphRandomErrorPlugin, GenericRandomErrorPlugin, LanguageModelRateLimitingPlugin, RetryAfterPlugin. RetryAfter's GenericResponse becomes IProxySession.Respond with canonical HttpHeader. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Migrate plugins off Titanium onto canonical HTTP model (Phase 3 final wave) Complete the strangler migration: flip LoggingContext and the proxy event args to the engine-agnostic IProxySession model, migrate all remaining plugins off Titanium.Web.Proxy types, and drop the Unobtanium.Web.Proxy package reference from DevProxy.Abstractions and DevProxy.Plugins. - LoggingContext/RequestLog now carry IProxySession (was SessionEventArgs); ProxyHttpEventArgsBase no longer exposes the Titanium Session. - All Behavior/Generation/Guidance/Inspection/Manipulation/Mocking/Reporting plugins read e.ProxySession.Request/Response and the canonical header API. - DevToolsPlugin re-keys _responseBody/CDP requestId on the phase-stable ProxySession.SessionId instead of the Titanium Request hash code. - Add Url setter and response HttpVersion to the canonical model (+ Titanium adapters) to preserve Rewrite and HAR behavior. - Remove dead Titanium FuncExtensions helper. DevProxy.Abstractions and DevProxy.Plugins are now Titanium-free; only the host engine and the Titanium adapter retain the dependency. Build 0/0, 33+38 tests green, live smoke test verified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Kestrel proxy engine vertical slice (Phase 4) Introduce DevProxy.Proxy.Kestrel: a forward-proxy engine built on ASP.NET Core Kestrel that runs the Dev Proxy plugin pipeline against the canonical HTTP model, as the eventual replacement for the Titanium engine. Slice-1 scope (dev-toggle via DEV_PROXY_ENGINE=kestrel, for golden-output comparison against Titanium; not a shipped fallback): - KestrelProxyEngine hosts a raw TCP endpoint via UseConnectionHandler. - ProxyConnectionHandler: HTTP/1.1 parse, CONNECT->SslStream MITM, forward, write-back; catches client disconnect/cancel as normal teardown. - PluginPipeline mirrors ProxyEngine semantics (watched/mocked/notwatched, Before/After request+response, per-exchange SessionData) and emits request logs inside the requestId scope so the console formatter groups plugin and engine lines identically to Titanium. - UpstreamForwarder/ResponseWriter apply ForwardingInvariants (hop-by-hop stripping, decompressed bodies, Content-Length/Encoding fix-up). - In-memory CertificateAuthority + DuplexPipeStream ported from the POC. Engine selection wired via DEV_PROXY_ENGINE in DI; ProxyConsoleFormatter suppresses the engine-name prefix for KestrelProxyEngine (as for ProxyEngine). Deferred hardening (tracked): keep-alive, blind-tunnel for non-watched and h2-only/gRPC (ClientHello peek), WebSocket relay, chunked/trailers/100-continue, body-mode streaming, process filter, persistent CA, host-watch DRY consolidation. Full solution builds 0/0; 71 existing tests green; live-verified plain HTTP passthrough, HTTPS MITM, GenericRandomError/RetryAfter mocking with correct console output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Kestrel proxy engine test project (Phase 4) Add DevProxy.Proxy.Kestrel.Tests with 28 unit tests covering the HTTP/1.1 parser, host watch list (including order-dependent exclusion semantics), response writer, and the mutable HTTP model / CanonicalProxySession. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add selective decrypt + ALPN blind-tunnel to Kestrel engine (Phase 4) At CONNECT time the engine now non-destructively peeks the TLS ClientHello (SNI + ALPN) and decides whether to intercept: - non-watched hosts are blind-tunnelled (encrypted bytes relayed, never MITM'd) - watched hosts whose client is h2-only (gRPC, no http/1.1 fallback) are blind-tunnelled so they stop failing - otherwise the engine terminates TLS advertising http/1.1 only, so h2-capable clients downgrade and are intercepted as HTTP/1.1 Adds a minimal tolerant TlsClientHello parser, wires HostWatchList into the connection handler, and relays blind tunnels bidirectionally with a linked cancellation token so both copy tasks finish before the streams are disposed. Covered by 9 new TlsClientHello unit tests (37 Kestrel tests total). Verified live: watched host downgrades h2->http/1.1 and is intercepted; example.com (non-watched) validates its real cert without -k, proving it is never MITM'd. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add keep-alive + per-request state isolation to Kestrel engine (Phase 4 slice 3) Replace the static one-shot HTTP/1.1 reader with a stateful per-connection Http1ConnectionReader that owns the inter-request buffer, so bytes read past one request's body (a pipelined next request) are retained instead of dropped. This makes client-side keep-alive correct. - Http1ConnectionReader: stateful ReadHeadAsync/ReadBodyAsync owning _pending. - Http1RequestReader: reduced to stateless parse helpers (ParseHead, GetContentLength, IndexOfDoubleCrlf). - ProxyConnectionHandler.ServeConnectionAsync: keep-alive loop over both plain HTTP and decrypted CONNECT streams; each exchange builds a fresh CanonicalProxySession (new GUID + Interlocked requestId) so plugin state never leaks across requests on a reused connection. - ShouldKeepAlive (RFC 9112 9.3): persistent by default on HTTP/1.1, opt-in on HTTP/1.0; forces Connection: close for Connection: close, Transfer-Encoding (chunked, not yet reframable) and Expect: 100-continue (unsupported), so an unframable body can never corrupt the next request. - ResponseWriter.WriteAsync: keepAlive flag drives the Connection header. Tests: +15 Kestrel (stateful reader incl. pipelining/surplus-retention, ShouldKeepAlive matrix, keep-alive/close response-writer). 123 green total. Verified live: two URLs in one curl invocation reuse one TCP connection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add WebSocket transparent relay to Kestrel engine (Phase 4 slice 4) Watched-host WebSocket upgrades (ws:// plain + wss:// after MITM) now relay through the Kestrel engine for parity with Titanium, which relays WS opaquely (no plugin sees frames). HttpClient can't carry a 101 upgrade, so WebSocketRelay opens its own raw socket to the origin (TLS w/ http/1.1 ALPN for wss), replays the handshake in origin-form (preserving Upgrade/Connection/Sec-WebSocket-*, stripping Proxy-*), writes the origin's 101 back to the client verbatim, then splices frames raw via a shared StreamRelay (also now used by BlindTunnelAsync, DRY). Critical ordering: the response pipeline (which flushes the buffered req-log) runs in the onHandshakeResponse callback, right after the 101 and before the splice blocks, else a watched WS never logs until the socket closes. A handshakeObserved fallback flush covers origin-connect failure. Tests: ParseResponseHead unit tests, StreamRelay bidirectional copy, and an end-to-end RelayAsync test against a loopback fake origin (asserts origin-form handshake replay, header preservation, 101 verbatim write-back, frame splicing, and onHandshakeResponse invocation). 128 tests green (+5 Kestrel). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add chunked request bodies + 100-continue to Kestrel engine (Phase 4 slice 5) Closes the biggest remaining HTTP/1.1 correctness gap on the request read path. Previously a chunked request body was silently dropped (Content-Length defaulted to 0, so an empty body was forwarded) and Expect: 100-continue hung (the proxy never sent the interim 100). Response-side chunked already worked via HttpClient. - Http1ConnectionReader.ReadChunkedBodyAsync decodes chunked framing (size lines, chunk extensions, CRLF delimiters) into one buffer, consumes and drops trailers, and retains surplus for the next pipelined request. Shared _pending fill primitives (PullAsync/ReadExactlyAsync/ReadLineAsync/TakeFromPending), reused by ReadBodyAsync. - Http1RequestReader.DetectBodyFraming (None/ContentLength/Chunked/Conflicting) and HasExpectContinue. A request declaring both Content-Length and chunked Transfer-Encoding is Conflicting and refused with 400 (RFC 9112 6.3.3 smuggling guard). - ResponseWriter.WriteContinueAsync sends 100 Continue; ExchangeAsync answers Expect: 100-continue itself before reading the body (the buffering model can't relay the origin's interim 100). Expect is stripped on forward. - ShouldKeepAlive no longer forces close for chunked/Expect; malformed chunked framing replies 400 and closes. 146 tests green (+18). Verified live through the MITM proxy: a chunked upload reached httpbin with the body intact and reframed to Content-Length (no Transfer-Encoding), the client got 100 Continue, two chunked POSTs reused one TCP connection, and a raw Content-Length+Transfer-Encoding request got 400. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Kestrel engine: incremental SSE streaming with capped tee (Slice 6) Forward text/event-stream responses to the client piece-by-piece (re-framed as HTTP/1.1 chunked) instead of buffering the whole body. Buffering withheld every event until the stream ended and hung the engine on unbounded streams. - UpstreamForwarder.ForwardAsync returns an OriginResponse: buffered for normal responses, or a live body stream for text/event-stream (message stays open). - StreamingResponseWriter pumps origin->client one chunk per read (flushed so events arrive live), accumulating a capped copy (4 MiB, reusing BodyModeResolver.DefaultInMemoryLimitBytes); beyond the cap accumulation is dropped but relaying continues, so the engine never hangs or OOMs. - PluginPipeline.RunStreamingResponseAsync runs the body pump between BeforeResponse and AfterResponse, exposing the accumulated body to read-only AfterResponse inspectors (OpenAITelemetry/OpenAIUsageDebugging) for parity. - Keep-alive preserved (chunked is self-delimiting). Tests: +12 (StreamingResponseWriter chunk framing/accumulation/cap, forwarder streaming detection). 158 green. Live-verified: local finite SSE streamed ~0.5s per event (not buffered), Transfer-Encoding: chunked, Connection: keep-alive, connection reused across two streamed responses, non-watched passthrough intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 4 Slice 7: process filtering at CONNECT (--watch-pids/--watch-process-names) Implements the last functional parity gap with the Titanium engine: restricting MITM to specific processes via --watch-pids / --watch-process-names. Matches Titanium semantics exactly — filtering applies ONLY at the CONNECT decision (IsProxiedProcess in ProxyEngine.cs), so plain HTTP stays unfiltered (documented parity quirk). - ProcessFilter resolves the client's PID from its source port and matches against WatchPids then WatchProcessNames (ordinal/case-sensitive HashSet to mirror Titanium's default Contains comparer). No filter configured -> watch all (IsEmpty short-circuit, zero overhead). - ConnectionProcessResolver shells out per-platform: lsof -i on Unix, netstat -ano on Windows. LsofParser/NetstatParser are pure, fully unit-tested partial classes (GeneratedRegex). Chosen over Windows P/Invoke (GetExtendedTcpTable) for testability and boring-tech symmetry. NOTE: the Windows netstat path is parser-tested but not live-verifiable on macOS. - Wired into ProxyConnectionHandler.HandleConnectAsync (after the h2-only check): unresolved/non-matching process -> blind-tunnel (never MITM'd). KestrelProxyEngine constructs the filter from configuration.WatchPids/WatchProcessNames. - Updated the CONNECT-flow doc diagram. Tests: +21 Kestrel (12 ProcessFilter + 4 Lsof + 5 Netstat) -> 179 green (33 Abstractions + 38 Titanium + 108 Kestrel). Live-verified on macOS: matching process name -> MITM intercept + req-log; non-matching -> blind tunnel (real cert, no -k, zero req-log lines). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 5 Slice 5a: persistent certificate authority (disk root + leaf cache) The Kestrel engine previously minted a brand-new in-memory root CA on every start, so the OS would distrust it after each restart. This makes the root (and per-host leaves) persistent on disk, mirroring the Titanium engine's CertificateDiskCache layout exactly: root at <configDir>/rootCert.pfx (overridable via DEV_PROXY_CERT_PATH), leaves at <configDir>/crts/<host>.pfx, both PKCS#12 with an empty password. configDir matches Titanium's resolution (ApplicationData/dev-proxy on mac/Linux, the executable dir on Windows). EUREKA: because the format + path match, the persistent CA loads a root that the Titanium engine ALREADY created and the OS ALREADY trusts — existing users get interception with zero new trust code (verified live: curl WITHOUT -k against a watched HTTPS host returned 200 + full MITM req-log). - LoadOrCreateRoot: load + validate (isCA, has private key, not expired) the on-disk root; regenerate + re-save (and purge the now-stale leaf cache) if absent/invalid/ expired. Per decision #5 there is no cross-version compatibility contract. - Leaves are minted on demand, cached in memory + on disk, and reused across restarts. Leaf validity is clamped to the issuer's window so reusing a near-expiry root can't produce a leaf that outlives it (CertificateRequest.Create rejects that). Filename- unsafe hosts (IPv6 ':', wildcard '*', '/') are sanitized for the disk cache key. - Disk writes are best-effort: an unwritable location degrades to in-memory-only for the run instead of failing interception. - KestrelProxyEngine now builds the CA via CertificateAuthority.CreateDefault(logger). OS-trust install (mac keychain / Windows root store) + the first-run prompt remain Slice 5b (host-side, reusing the existing trust helpers). This slice only makes the root persistent. Tests: +12 Kestrel (CertificateAuthorityTests: first-run create+persist, reuse across instances, invalid/expired/non-CA regeneration, stale-leaf purge, leaf mint+sign+ persist+reuse, leaf-not-outliving-root clamp, IP-literal SAN, filename sanitization) -> 191 green (33 + 38 + 120). Live-verified on macOS: existing root reused (fingerprint unchanged, "Loaded persisted root certificate"), leaf cached + reused across restarts, MITM works without -k. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OS-trust install + first-run wiring for the Kestrel engine (Slice 5b) Wire root-certificate OS trust for the Kestrel engine via a new IRootCertificateTrust abstraction (DevProxy.Abstractions/Proxy), implemented host-side by RootCertificateTrust (DevProxy/Proxy). The engine mints/persists its root (Slice 5a) then calls EnsureTrusted; the host performs the platform install: - macOS: trust in the login keychain via MacCertificateHelper, gated by the first-run flow (InstallCert + NoFirstRun + HasRunFlag + Y/n prompt, auto-"y" under CI/redirected stdin) — mirrors Titanium's FirstRunSetup. - Windows: install the public-only root into CurrentUser/Root, idempotent by identity (CA5380 suppressed — installing the Dev Proxy root is the proxy's explicit, user-consented purpose). NOT live-verifiable on macOS. - Linux: warn to trust manually. The trust decision is factored into a pure RootTrustPolicy.Decide table in Abstractions so it's exhaustively unit-testable without platform/console I/O; the host impl is just the thin I/O around it (option X — new interface, leave Titanium's FirstRunSetup untouched for low blast radius). CertCommand (ensure/remove) already works for Kestrel mode unchanged thanks to Slice 5a's on-disk path parity. The Kestrel engine ctor takes an optional IRootCertificateTrust (null = no trust, e.g. tests); the DEV_PROXY_ENGINE=kestrel DI toggle registers and injects RootCertificateTrust. 207 tests green (49 Abstractions incl. +16 RootTrustPolicy, 38 Titanium, 120 Kestrel). Live-verified on macOS: with an already-trusted persisted root and --no-first-run, a watched HTTPS request returned 200 WITHOUT -k plus a full MITM req-log — trust wiring doesn't regress the existing-root happy path. Fresh-machine keychain trust is covered by the unit-tested decision table (not live-exercised to avoid keychain GUI pollution); Windows store install is parser/logic-only on macOS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Graceful-teardown audit: DRY connection-close classifier (Slice 5c) Extract a single ConnectionTeardown.IsExpected(exception) classifier and use it at every client/origin read/copy/write boundary, replacing the per-site hardcoded `ex is IOException or OperationCanceledException or ...` lists. An exception is "expected" (treat as EOF/close, don't log as error) when it's a ConnectionResetException, ConnectionAbortedException, OperationCanceledException, IOException, or SocketException — the normal result of a peer disconnecting, resetting, or the connection being cancelled mid-exchange. Boundaries updated to the shared classifier: - ProxyConnectionHandler.OnConnectedAsync outer catch (keep-alive read, body read, response write, streaming pump, blind tunnel) — now also covers SocketException/ConnectionAbortedException. - StreamRelay bidirectional-copy teardown. - WriteErrorAsync best-effort write to a possibly-gone client. Fixes the one genuine noise gap: the WebSocket relay catch logged a client disconnect mid-handshake / verbatim-101-write as an ERROR. It now logs at Debug when IsExpected and reserves Error for genuine faults. 216 tests green (49 Abstractions, 38 Titanium, 129 Kestrel; +9 ConnectionTeardown covering OCE/TaskCanceled/IOException/SocketException/ConnectionReset/ ConnectionAborted true and InvalidOperation/NullReference/unrelated false). Live-verified on macOS: aborting curl mid-stream (httpbin /drip, --max-time 1) produced ZERO error/exception log lines and the engine kept serving (a following request returned 200). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden CONNECT authority parsing: IPv6, ports, malformed targets (Slice 5d) Replace the naive "split on the last colon" SplitHostPort with a pure, explicit ConnectAuthorityParser that correctly handles: - IPv6 literals: [::1] and [2001:db8::1]:8443 — the bare host (no brackets) is used for socket connect / cert minting, and a UrlHost property re-adds brackets for composing the absolute URL (https://[::1]:8443/path). - Explicit ports, including validation (reject port 0 / >65535 / non-numeric / empty). - Malformed targets: empty host, unterminated bracket, junk after ], bare (unbracketed) IPv6, invalid host names — all rejected. The authority is now parsed + validated BEFORE the proxy writes "200 Connection Established", so a malformed CONNECT is refused with a 400 instead of establishing an unusable tunnel. The decrypted-tunnel URL is composed from UrlHost so IPv6 targets produce valid absolute URLs. 237 tests green (49 Abstractions, 38 Titanium, 150 Kestrel; +21 parser tests covering reg-name/IPv4/punycode/IPv6 with and without ports plus 15 malformed cases). Live-verified on macOS: normal + explicit-:443 watched MITM return 200 (no -k); malformed CONNECT targets (port 0, bare IPv6 ::1:443, unterminated [::1) each return 400 Bad Request while a valid target returns 200 Connection Established; zero error-log noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Slice 5e: --port 0 actual-port logging + DRY host extraction Log the OS-assigned port (not the configured 0) for the Kestrel engine by reading the bound ListenOptions endpoint after StartAsync, so users running with --port 0 can connect. Extract the duplicated urlsToWatch host-derivation logic into a shared WatchedHostExtractor.ToHostRegex in DevProxy.Abstractions, used by both the Titanium ProxyEngine.LoadHostNamesFromUrls and the Kestrel HostWatchList. One implementation, one test suite (11 new tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 6 slice: engine-agnostic system-proxy on/off (no Titanium) Re-implement turning the OS system-proxy on and off without Titanium, behind a new ISystemProxyManager abstraction (Abstractions) so the Kestrel engine can use it without referencing the host (mirrors IRootCertificateTrust): - SystemProxyManager (host): Windows via WinINET (registry ProxyServer/ProxyEnable + InternetSetOption settings-changed/refresh broadcast), macOS via the existing toggle-proxy.sh, Linux warns. DllImport (not LibraryImport) to avoid enabling AllowUnsafeBlocks project-wide. - SystemProxyAddress (Abstractions, pure): normalizes wildcard/empty bind addresses to loopback for the client-facing proxy address; unit-tested. - KestrelProxyEngine: enable on start (using the actually-bound port, matters for --port 0), disable on stop. Previously the Kestrel engine set no system proxy at all — a parity gap. - StopCommand --force crash-cleanup now uses the shared manager, so a force-killed daemon restores the system proxy on Windows too (was macOS-only before). 259 tests green (+11 SystemProxyAddress). Windows WinINET path is parser/logic-tested but needs verification on a Windows host (cannot run on macOS). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 6 slice: replace Titanium RunTime with System.OperatingSystem Swap the Titanium RunTime OS-check helper for the .NET built-in equivalents (RunTime.IsWindows → OperatingSystem.IsWindows(), RunTime.IsMac → OperatingSystem.IsMacOS()) in ProxyEngine and CertCommand, and drop the now-unused `using Titanium.Web.Proxy.Helpers;` import from both. Strictly equivalent behavior; reduces Titanium API surface ahead of the cut-over. Left untouched (intentionally): - CertificateDiskCache.RunTime.IsUwpOnWindows — no built-in equivalent, and the file (a Titanium ICertificateCache impl) is deleted at cut-over anyway. - The Titanium.Web.Proxy.* log filters — still suppress real Titanium engine logs while it remains the default engine; their removal belongs in the cut-over commit. 259 tests green; Titanium engine boot-smoked clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add hermetic Kestrel parity-test gate (controllable rows) Build the DevProxy.Parity.Tests project that boots the real KestrelProxyEngine on a random localhost port against a deterministic FakeOrigin and asserts the observable proxy contract for every controllable parity-matrix row: - plain HTTP: GET/POST (small/large), large download, status relay, 204 empty-body framing, request-header passthrough - chunked request body reframing + SSE chunked response streaming - keep-alive reuse + Connection: close - plugin Respond() mocking short-circuit (origin never contacted) - Content-Length + Transfer-Encoding smuggling -> 400 guard 15 hermetic rows green. External/OS rows (HTTPS real-cert MITM, h2/gRPC, WebSocket, system-proxy, process filter) remain the scripted live pass and are already slice-verified; see files/parity-results.md for the full matrix. The harness boots only the Kestrel engine (the Titanium engine's process-global static ProxyServer + IServer make in-process co-hosting flaky, and it is deleted at cut-over); it asserts the reference behavior Titanium already exhibits. Total: 274 tests green (71 Abstractions + 38 Titanium + 150 Kestrel + 15 Parity). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Hard cut-over to the Kestrel proxy engine; remove Titanium Make the ASP.NET Core Kestrel engine the sole proxy engine and remove the Unobtanium/Titanium.Web.Proxy adapter, dependency, and dev toggle in one change. - Remove the DEV_PROXY_ENGINE toggle; AddProxyEngine always wires Kestrel, the shared CertificateAuthority, IRootCertificateTrust, and ISystemProxyManager. - Introduce a single shared CertificateAuthority DI singleton (Kestrel AddKestrelCertificateAuthority) with the root X509Certificate2 derived from it; the engine, CertCommand, ProxyController, and EntraMockResponsePlugin all resolve the same root cert. The engine no longer self-creates/disposes its CA. - Add Trust/Untrust to IRootCertificateTrust and implement them host-side (mac keychain / Windows root store / Linux warn); CertCommand uses them. - Delete ProxyEngine, CertificateDiskCache, the DevProxy.Proxy.Titanium and DevProxy.Proxy.Titanium.Tests projects, the Unobtanium PackageReference, the Titanium.Web.Proxy.* log filters, and the THIRD PARTY NOTICES Unobtanium entry. - Update docs (CONTRIBUTING, copilot-instructions) and stale comments to Kestrel. Build 0/0; 236 tests green (71 Abstractions + 150 Kestrel + 15 Parity). Live-verified on macOS: watched HTTPS MITM without -k + req-log, cert-download API, and `cert ensure` trust path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix detached-daemon state registration and bound graceful-shutdown drain Two cut-over regressions surfaced by the macOS system-proxy live test: 1. Detached mode (`--detach`) was broken. The deleted Titanium ProxyEngine wrote the initial ProxyInstanceState (gated on IsInternalDaemon); the Kestrel engine never did, so the daemon orphaned, the parent's readiness poll timed out, and the system proxy was left on. The host's UpdateStateWithApiUrlAsync only updated an existing state file, so it silently no-opped. - KestrelProxyEngine now publishes its actually-bound port back to the shared IProxyConfiguration after binding (resolves --port 0 to the OS-assigned port). - DevProxyCommand.WriteDaemonStateAsync writes the full ProxyInstanceState (Pid, ApiUrl, LogFile, ConfigFile, resolved Port, AsSystemProxy) once the engine has published its port, satisfying the parent's `Port > 0` readiness contract used by `devproxy stop` and `devproxy status`. 2. Graceful shutdown stalled after real traffic. The engine awaited app.StopAsync(CancellationToken.None), which waited unbounded for lingering client keep-alive connections to drain — fast when idle, but >10s after traffic, making `devproxy stop` falsely report "did not stop in time". The drain is now bounded (5s) so remaining connections are force-closed. Adds EnginePortPublishingTests regression coverage (verified failing without the port write-back). 237 tests green. Live-verified on macOS: `--detach` registers the daemon, `devproxy status`/`stop` find it, and the system proxy is restored on both graceful API stop and SIGTERM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename Parity.Tests to Integration.Tests post-cut-over With Titanium removed there is no second engine to compare against, so the "parity" framing no longer applies. The suite is now a black-box integration test that boots the Kestrel engine over a real socket against a FakeOrigin. Renamed the project, csproj, namespaces, *ParityTests classes/files, and the solution entry (project GUID unchanged); reworded "parity" comments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add per-plugin integration tests: harness infra + behavior/manipulation/mocking plugins Adds a CapturingLoggerFactory (collects engine-emitted RequestLogs), an inline PluginConfig section builder, and extends KestrelProxyHarness to accept a logger factory and expose its urls-to-watch set. Covers 9 hermetic, HTTP-observable plugins end-to-end through the Kestrel engine: GenericRandomError, Latency, RateLimiting, RetryAfter, Rewrite, MockResponse, Auth (reject + allow), and GraphRandomError. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CrudApi + MockRequest integration tests with InitializeAsync infra Adds PluginTestHost (a minimal DI ServiceProvider supplying HttpClient/ILogger<>/ IProxyConfiguration so plugins whose InitializeAsync builds file loaders can run in-process), TestDefaults (shared HttpClient), a FakeOrigin request recorder + /json endpoint, and a settable ConfigFile on TestProxyConfiguration. Mirrors the host's runtime-only package assets in the test csproj so plugin runtime dependencies (OpenIdConnect, JWT, OpenApi, OpenTelemetry, EF Sqlite, Newtonsoft, Azure.Identity) resolve. Covers MockRequestPlugin (outbound fire) and CrudApiPlugin (in-memory REST served through the engine, origin never contacted). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add guidance plugin integration tests (7 hermetic plugins) Adds TestExchange, which constructs the engine's real canonical session/request/ response types so guidance plugins gated on a fixed upstream host (graph.microsoft.com) can be driven at their hooks with production-fidelity inputs the loopback origin can't impersonate. Covers GraphBetaSupport, GraphClientRequestId, GraphSdk, ODataPaging, ODSPSearch, GraphConnector, and CachingGuidance, asserting the advisory log each emits. GraphSelectGuidancePlugin is documented as non-hermetic (needs the Graph metadata DB) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add reporting plugin integration tests (UrlDiscovery, ExecutionSummary, MinimalPermissions) Drives reporter plugins through AfterRecordingStopAsync exactly as the host's recording controller does: feeds engine-shaped RequestLogs (TestExchange.AsRequestLog) into RecordingArgs with a seeded Reports GlobalData slot, then asserts the structured report stored under the plugin name. MinimalPermissions runs against a temp OpenAPI spec fixture to verify offline permission analysis. GraphMinimalPermissions(+Guidance), MinimalCsom, and ApiCenter reporters are documented as live-backend (Bucket 3) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add generation plugin integration tests (HAR, mock, .http, OpenAPI, TypeSpec) Drives each generator through AfterRecordingStopAsync with an InterceptedResponse log, asserting the stored report and the generated artifact. Generators write to the process CWD, so tests redirect it to a temp dir; assembly test parallelization is disabled to keep that global swap safe. A DisabledLanguageModelClient keeps the OpenAPI/TypeSpec generators hermetic (LM is cosmetic-only; they fall back to deterministic generation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add process-level smoke test for config-driven plugin loading Spawns the real DevProxy host with a devproxyrc.json + mocks file, waits for the Kestrel engine to report its bound port (--port 0), routes an HTTP request through it as an explicit proxy, and asserts MockResponsePlugin short-circuits with the configured body. Proves end-to-end that config-driven plugin discovery + the engine wire survive the Titanium -> Kestrel cut-over. Uses a plain-HTTP watched URL so no CA trust is needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unwired body-mode subsystem; fix misleading body docs The BodyModeResolver + BodyMode/BodyCapabilities/BodyContext/BodyDirection types (319 LOC, 12 tests) were a spike-era design that the Kestrel engine never wired: Resolve() is never called in production, no plugin declares BodyCapabilities, and IHttpMessage exposes no capability property. The engine uses a fixed 4 MiB in-memory inspection cap instead. Worse, the IHttpMessage doc claimed body buffering was 'driven by plugin BodyCapabilities', misleading plugin authors toward an API that does not exist. - Delete BodyModeResolver.cs, BodyHandling.cs, and BodyModeResolverTests.cs. - Inline the one used constant as ProxyConnectionHandler.InMemoryInspectionCapBytes. - Correct the IHttpMessage body-visibility doc to describe actual behavior. - Drop stale 'every engine adapter' language (abandoned multi-engine design) from ForwardingInvariants and HeaderCollection docs. The design is preserved in git and can be restored from this parent commit if/when capability-based body handling is actually needed. A code comment + plan.md follow-up track the genuine, separate unbounded-buffered-body memory risk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename IProxyLogger.cs to LoggingContext.cs to match contents The file's 'I' prefix implied an interface, but it contains no interface — only the MessageType enum and the LoggingContext / StdioLoggingContext classes. Rename to match its primary type and theme. (StdioEvents.cs was left as-is: it correctly groups the stdio event-arg family, mirroring the existing ProxyEvents.cs convention.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add WebSocketMockResponsePlugin (reactive WS frame mocking) Net-new capability unlocked by the Kestrel engine: mock WebSocket frames end-to-end. The engine owns transport (101 handshake + framing via the framework's WebSocket.CreateFromStream); the plugin owns behavior (on-connect scripted sends + match incoming client messages by equals/contains/regex/json and reply with mapped messages, plus optional close). A watched WS upgrade is dispatched to the plugin responder (instead of the origin relay) when the plugin calls IProxySession.HandleWebSocket(handler), leaving the session in the Watched phase. The plugin must NOT set ResponseState.HasBeenSet, which would short-circuit to the mocked-HTTP ResponseWriter path before the WS dispatch branch runs. DRY scheme-normalization footgun fix: the engine reports an intercepted WS upgrade with an http(s) scheme, but authors naturally write wss://host/* in urlsToWatch and mock URLs. A shared ProxyUtils.NormalizeWebSocketScheme normalizes wss->https / ws->http and is applied at both the watch-regex build site (PluginServiceExtensions.ConvertToRegex) and the plugin's mock-URL match, so either form works and a wss:// watch entry is no longer silently relayed to origin. Tests: responder loopback unit tests, matcher unit tests, 2 e2e ClientWebSocket-over-wss-MITM integration tests, and NormalizeWebSocketScheme unit tests. Full suite green (284). Live-verified through the host binary with urlsToWatch ["wss://ws.example.test/*"]: OPEN -> welcome (onConnect) -> pong (reactive ping->pong); proxy logged "Mocking WebSocket" + "WebSocket mock established". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align WebSocket mock matching with HTTP/stdio mock vocabulary The WebSocketMockResponsePlugin invented a `matchType: equals|contains|regex| json` enum, diverging from every other Dev Proxy mock plugin, which matches with the `bodyFragment` (case-insensitive contains) + `bodyRegex` field pair. Replace the enum with the conventional vocabulary so authors familiar with the HTTP/stdio mocks reach for the same fields: - `bodyFragment` — OrdinalIgnoreCase substring - `bodyRegex` — regular expression (1s timeout) - `bodyJson` — structural JSON equality (JsonNode.DeepEquals); accepts a JSON value directly in the mocks file, matching how the HTTP response body accepts object/array/string Precedence when more than one is set: bodyJson > bodyRegex > bodyFragment (mirrors stdio's "bodyRegex wins over bodyFragment"). Null match or no criteria remains a catch-all. The genuinely WS-specific structure (url-keyed mock, onConnect pushes, reactive rules, multi-message responses) is unchanged — only the match expression is aligned. Updates the mocks-file schema and matcher unit tests accordingly. Full suite green (283). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore interactive console (hotkeys + --record) dropped in Kestrel cut-over The hard cut-over to the Kestrel engine deleted ProxyEngine.cs, which owned the interactive key loop (r/s/c/w) and the --record auto-start. KestrelProxyEngine never re-implemented them, so the documented keyboard shortcuts (technical-reference/keyboard-shortcuts.md) stopped working and `devproxy --record` no longer started recording at launch. Restore both as a host-side InteractiveConsoleService (BackgroundService), keeping the Kestrel engine headless: the engine depends only on the proxy abstractions and must not reach into host concerns like IProxyStateController. The service waits for ApplicationStarted so the banner prints after the engine's startup logs, honors --record, prints hotkeys (text mode) or API instructions (JSON mode), and dispatches keys via a testable ConsoleHotkeyHandler over an ISystemConsole seam. Adds DevProxy.Tests (the host had no unit test project) with 10 tests covering key dispatch and both banners. Full suite green (293 tests). Live-verified: --record starts recording at launch, JSON banner prints, proxy serves 200 through the new service. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Windows verification checklist for the Kestrel engine The Windows-specific runtime paths (WinINET system proxy, current-user root-store cert trust, netstat-based process filter) are unit-tested but were never live-verified on a real Windows host after the Kestrel cut-over. Add a self-contained, clone-and-run checklist covering build/tests, cert trust, system proxy on/off, detached daemon lifecycle, process filter, the restored interactive console, and a core proxy + plugin smoke pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Preserve Content-Length on HEAD responses A HEAD response carries the Content-Length a GET would return but no body (RFC 9110 §9.3.2). The engine was recomputing Content-Length from the (empty) HEAD body, so clients saw Content-Length: 0 instead of the real resource size. ResponseWriter and UpstreamForwarder are now method-aware: HEAD preserves the origin's Content-Length, never takes the streaming path, and never writes a body. * Fix API URL banner showing port 0 with --api-port 0 Resolve the OS-assigned API port from the bound server address before printing the JSON-mode banner so the curl commands reference the actual port instead of the literal 0. * Return 504 on upstream timeout instead of silently dropping the connection The forward catch excluded all OperationCanceledException so client cancellation would bubble up as a silent teardown. But an HttpClient request timeout also throws TaskCanceledException (an OperationCanceledException), so a stalled origin was misclassified as a client disconnect: the client got a dropped/reset connection after the default 100s timeout instead of a gateway error. Add a pure UpstreamFailure.Classify helper (mirroring ConnectionTeardown) that tells the two apart by the connection token: OperationCanceledException with the token NOT cancelled is an upstream timeout -> 504 Gateway Timeout; with the token cancelled it is a genuine client teardown -> silent; anything else (TLS handshake failure on an invalid upstream cert, DNS, connection refused) -> 502 Bad Gateway. Wire it into the single forward catch in ProxyConnectionHandler and add the 504 reason phrase. Closes the "upstream timeout" and "invalid upstream cert" parity rows in code with 7 unit tests on the classifier. Full suite green (306 tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clear CA2201 warning in ConnectionTeardownTests The test deliberately constructs a NullReferenceException to prove the teardown classifier does not mask a latent bug as an expected connection close. CA2201 flags constructing reserved exception types, so scope a justified suppression to that single line instead of weakening the test. Solution now builds with 0 warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump version to 4.0.0 (breaking: Kestrel engine + canonical plugin API) This is a major version bump because the migration from Titanium/Unobtanium to the Kestrel-based proxy engine introduces breaking changes: - Plugin API surface moved from Titanium types (SessionEventArgs/Request/ Response/HttpHeader) to the canonical model (IProxySession/IHttpRequest/ IHttpResponse/HeaderCollection). Third-party plugins must recompile. - Behavioral changes around HTTP/2 handling (h2-only/gRPC blind-tunnel). Copies schemas/v3.1.0 -> schemas/v4.0.0 verbatim (41 files) and updates <Version> in all four shipping projects (incl. DevProxy.Proxy.Kestrel), schema URLs in config JSON + MockResponsePlugin, installer/script/Dockerfile version strings, and refreshes packages.lock.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
waldekmastykarz
added a commit
that referenced
this pull request
Jul 24, 2026
* Add throwaway Kestrel proxy spike (Phase 0B de-risking) Validates selective decrypt, non-destructive ClientHello ALPN peek, h2-only/gRPC blind-tunnel, keep-alive isolation, and unbuffered streaming. Not part of DevProxy.sln. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add canonical HTTP proxy model to DevProxy.Abstractions (Phase 1) Introduce engine-agnostic Dev Proxy model that will replace the leaked Unobtanium/Titanium types in the plugin SDK: - IHttpHeader/IHeaderCollection/IHttpMessage/IHttpRequest/IHttpResponse and IProxySession (logical SessionId, Respond() mocking primitive), with concrete HttpHeader + HeaderCollection. - Bodies exposed as ReadOnlyMemory<byte> (decompressed-body contract). - Body strategy: BodyMode, BodyCapabilities, BodyModeResolver (pure, reconciles streaming vs full-body plugin access). - ForwardingInvariants documenting the forwarding contract + shared HopByHopHeaders set. - New DevProxy.Abstractions.Tests (xUnit), 33 tests. Purely additive: existing Titanium-based code is untouched and the full solution builds clean. Adapter wiring follows in Phase 2/3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Titanium adapter mapping engine types onto canonical model (Phase 2) Introduces DevProxy.Proxy.Titanium, the sole project referencing Unobtanium.Web.Proxy, which projects Titanium request/response/header/session types onto the engine-agnostic canonical model in DevProxy.Abstractions. - TitaniumHeaderCollection, TitaniumHttpMessageAdapter (shared DRY base over RequestResponseBase), TitaniumRequestAdapter, TitaniumResponseAdapter, TitaniumProxySession. - Body access guarded by HasBody (Titanium throws BodyNotFoundException otherwise); mutation routed through the session's SetRequestBody/ SetResponseBody since Titanium exposes no public body setter. - DevProxy.Proxy.Titanium.Tests: 38 adapter-fidelity unit tests. Purely additive and ship-safe: the existing engine and plugins are untouched. Full solution builds 0/0; 71 tests green. Cross-engine parity tests are deferred to Phase 4 (require the Kestrel engine to exist). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wire canonical IProxySession through ProxyEvents; migrate first plugin (Phase 3 slice 1) Introduces the dual-field strangler that lets plugins migrate off Titanium types one at a time while staying green: - ProxyHttpEventArgsBase exposes IProxySession ProxySession alongside the existing Titanium SessionEventArgs Session. HasRequestUrlMatch now reads the canonical request. - ProxyEngine builds a TitaniumProxySession (keyed by the existing per-session GetHashCode key) and threads it through the request/response arg construction sites; DevProxy host references DevProxy.Proxy.Titanium. - CachingGuidancePlugin fully migrated to e.ProxySession.Request and is now Titanium-free. Verified live: the cache-window warning fires through the real proxy pipeline reading IHttpRequest.RequestUri via the adapter. Full solution builds 0/0; 71 tests green. The other 21 Titanium-touching plugins are untouched and keep compiling against Session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 Wave 1: add IHttpRequest overloads to ProxyUtils Add canonical IHttpRequest overloads alongside the existing Titanium Request overloads for IsGraphRequest/IsSdkRequest/IsGraphBetaRequest/BuildGraphResponseHeaders. Extract a shared BuildGraphResponseHeadersCore to keep both overloads DRY. Additive only; Titanium overloads remain until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 guidance wave: migrate 4 guidance plugins off Titanium Migrate ODSPSearchGuidancePlugin, GraphSelectGuidancePlugin, GraphSdkGuidancePlugin, and GraphClientRequestIdGuidancePlugin to the canonical model: read request/response via e.ProxySession, helpers take IHttpRequest, header checks via Contains. Each plugin is now Titanium-free in source (LoggingContext still takes e.Session until the final wave). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 mocking wave: migrate 5 mocking plugins off Titanium Migrate AuthPlugin, CrudApiPlugin, GraphMockResponsePlugin, MockResponsePlugin, and OpenAIMockResponsePlugin to the canonical model: request/response via e.ProxySession, GenericResponse -> IProxySession.Respond, canonical HttpHeader, header methods via the canonical IHeaderCollection. Helper signatures and the CrudApi action-handler delegate now use IProxySession/IHttpRequest/ProxyRequestArgs. LoggingContext still takes e.Session until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 inspection-lite: migrate HttpUtils + OpenAI usage/telemetry streaming off Titanium Change HttpUtils.IsStreamingResponse/GetBodyFromStreamingResponse to take the canonical IHttpResponse. Migrate OpenAIUsageDebuggingPlugin fully to e.ProxySession (status comparisons cast HttpStatusCode to int). Point OpenAITelemetryPlugin's streaming call site at e.ProxySession.Response. DevToolsPlugin and the OpenAITelemetry RequestLog.Context.Session readers remain for the final LoggingContext wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 throttling group: migrate ThrottlerInfo + 5 throttling plugins off Titanium Flip ThrottlerInfo.ShouldThrottle delegate from Func<Request,...> to Func<IHttpRequest,...> (PluginEvents) and migrate the coordinated consumers: GraphUtils.BuildThrottleKey, RateLimitingPlugin, GraphRandomErrorPlugin, GenericRandomErrorPlugin, LanguageModelRateLimitingPlugin, RetryAfterPlugin. RetryAfter's GenericResponse becomes IProxySession.Respond with canonical HttpHeader. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Migrate plugins off Titanium onto canonical HTTP model (Phase 3 final wave) Complete the strangler migration: flip LoggingContext and the proxy event args to the engine-agnostic IProxySession model, migrate all remaining plugins off Titanium.Web.Proxy types, and drop the Unobtanium.Web.Proxy package reference from DevProxy.Abstractions and DevProxy.Plugins. - LoggingContext/RequestLog now carry IProxySession (was SessionEventArgs); ProxyHttpEventArgsBase no longer exposes the Titanium Session. - All Behavior/Generation/Guidance/Inspection/Manipulation/Mocking/Reporting plugins read e.ProxySession.Request/Response and the canonical header API. - DevToolsPlugin re-keys _responseBody/CDP requestId on the phase-stable ProxySession.SessionId instead of the Titanium Request hash code. - Add Url setter and response HttpVersion to the canonical model (+ Titanium adapters) to preserve Rewrite and HAR behavior. - Remove dead Titanium FuncExtensions helper. DevProxy.Abstractions and DevProxy.Plugins are now Titanium-free; only the host engine and the Titanium adapter retain the dependency. Build 0/0, 33+38 tests green, live smoke test verified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Kestrel proxy engine vertical slice (Phase 4) Introduce DevProxy.Proxy.Kestrel: a forward-proxy engine built on ASP.NET Core Kestrel that runs the Dev Proxy plugin pipeline against the canonical HTTP model, as the eventual replacement for the Titanium engine. Slice-1 scope (dev-toggle via DEV_PROXY_ENGINE=kestrel, for golden-output comparison against Titanium; not a shipped fallback): - KestrelProxyEngine hosts a raw TCP endpoint via UseConnectionHandler. - ProxyConnectionHandler: HTTP/1.1 parse, CONNECT->SslStream MITM, forward, write-back; catches client disconnect/cancel as normal teardown. - PluginPipeline mirrors ProxyEngine semantics (watched/mocked/notwatched, Before/After request+response, per-exchange SessionData) and emits request logs inside the requestId scope so the console formatter groups plugin and engine lines identically to Titanium. - UpstreamForwarder/ResponseWriter apply ForwardingInvariants (hop-by-hop stripping, decompressed bodies, Content-Length/Encoding fix-up). - In-memory CertificateAuthority + DuplexPipeStream ported from the POC. Engine selection wired via DEV_PROXY_ENGINE in DI; ProxyConsoleFormatter suppresses the engine-name prefix for KestrelProxyEngine (as for ProxyEngine). Deferred hardening (tracked): keep-alive, blind-tunnel for non-watched and h2-only/gRPC (ClientHello peek), WebSocket relay, chunked/trailers/100-continue, body-mode streaming, process filter, persistent CA, host-watch DRY consolidation. Full solution builds 0/0; 71 existing tests green; live-verified plain HTTP passthrough, HTTPS MITM, GenericRandomError/RetryAfter mocking with correct console output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Kestrel proxy engine test project (Phase 4) Add DevProxy.Proxy.Kestrel.Tests with 28 unit tests covering the HTTP/1.1 parser, host watch list (including order-dependent exclusion semantics), response writer, and the mutable HTTP model / CanonicalProxySession. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add selective decrypt + ALPN blind-tunnel to Kestrel engine (Phase 4) At CONNECT time the engine now non-destructively peeks the TLS ClientHello (SNI + ALPN) and decides whether to intercept: - non-watched hosts are blind-tunnelled (encrypted bytes relayed, never MITM'd) - watched hosts whose client is h2-only (gRPC, no http/1.1 fallback) are blind-tunnelled so they stop failing - otherwise the engine terminates TLS advertising http/1.1 only, so h2-capable clients downgrade and are intercepted as HTTP/1.1 Adds a minimal tolerant TlsClientHello parser, wires HostWatchList into the connection handler, and relays blind tunnels bidirectionally with a linked cancellation token so both copy tasks finish before the streams are disposed. Covered by 9 new TlsClientHello unit tests (37 Kestrel tests total). Verified live: watched host downgrades h2->http/1.1 and is intercepted; example.com (non-watched) validates its real cert without -k, proving it is never MITM'd. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add keep-alive + per-request state isolation to Kestrel engine (Phase 4 slice 3) Replace the static one-shot HTTP/1.1 reader with a stateful per-connection Http1ConnectionReader that owns the inter-request buffer, so bytes read past one request's body (a pipelined next request) are retained instead of dropped. This makes client-side keep-alive correct. - Http1ConnectionReader: stateful ReadHeadAsync/ReadBodyAsync owning _pending. - Http1RequestReader: reduced to stateless parse helpers (ParseHead, GetContentLength, IndexOfDoubleCrlf). - ProxyConnectionHandler.ServeConnectionAsync: keep-alive loop over both plain HTTP and decrypted CONNECT streams; each exchange builds a fresh CanonicalProxySession (new GUID + Interlocked requestId) so plugin state never leaks across requests on a reused connection. - ShouldKeepAlive (RFC 9112 9.3): persistent by default on HTTP/1.1, opt-in on HTTP/1.0; forces Connection: close for Connection: close, Transfer-Encoding (chunked, not yet reframable) and Expect: 100-continue (unsupported), so an unframable body can never corrupt the next request. - ResponseWriter.WriteAsync: keepAlive flag drives the Connection header. Tests: +15 Kestrel (stateful reader incl. pipelining/surplus-retention, ShouldKeepAlive matrix, keep-alive/close response-writer). 123 green total. Verified live: two URLs in one curl invocation reuse one TCP connection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add WebSocket transparent relay to Kestrel engine (Phase 4 slice 4) Watched-host WebSocket upgrades (ws:// plain + wss:// after MITM) now relay through the Kestrel engine for parity with Titanium, which relays WS opaquely (no plugin sees frames). HttpClient can't carry a 101 upgrade, so WebSocketRelay opens its own raw socket to the origin (TLS w/ http/1.1 ALPN for wss), replays the handshake in origin-form (preserving Upgrade/Connection/Sec-WebSocket-*, stripping Proxy-*), writes the origin's 101 back to the client verbatim, then splices frames raw via a shared StreamRelay (also now used by BlindTunnelAsync, DRY). Critical ordering: the response pipeline (which flushes the buffered req-log) runs in the onHandshakeResponse callback, right after the 101 and before the splice blocks, else a watched WS never logs until the socket closes. A handshakeObserved fallback flush covers origin-connect failure. Tests: ParseResponseHead unit tests, StreamRelay bidirectional copy, and an end-to-end RelayAsync test against a loopback fake origin (asserts origin-form handshake replay, header preservation, 101 verbatim write-back, frame splicing, and onHandshakeResponse invocation). 128 tests green (+5 Kestrel). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add chunked request bodies + 100-continue to Kestrel engine (Phase 4 slice 5) Closes the biggest remaining HTTP/1.1 correctness gap on the request read path. Previously a chunked request body was silently dropped (Content-Length defaulted to 0, so an empty body was forwarded) and Expect: 100-continue hung (the proxy never sent the interim 100). Response-side chunked already worked via HttpClient. - Http1ConnectionReader.ReadChunkedBodyAsync decodes chunked framing (size lines, chunk extensions, CRLF delimiters) into one buffer, consumes and drops trailers, and retains surplus for the next pipelined request. Shared _pending fill primitives (PullAsync/ReadExactlyAsync/ReadLineAsync/TakeFromPending), reused by ReadBodyAsync. - Http1RequestReader.DetectBodyFraming (None/ContentLength/Chunked/Conflicting) and HasExpectContinue. A request declaring both Content-Length and chunked Transfer-Encoding is Conflicting and refused with 400 (RFC 9112 6.3.3 smuggling guard). - ResponseWriter.WriteContinueAsync sends 100 Continue; ExchangeAsync answers Expect: 100-continue itself before reading the body (the buffering model can't relay the origin's interim 100). Expect is stripped on forward. - ShouldKeepAlive no longer forces close for chunked/Expect; malformed chunked framing replies 400 and closes. 146 tests green (+18). Verified live through the MITM proxy: a chunked upload reached httpbin with the body intact and reframed to Content-Length (no Transfer-Encoding), the client got 100 Continue, two chunked POSTs reused one TCP connection, and a raw Content-Length+Transfer-Encoding request got 400. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Kestrel engine: incremental SSE streaming with capped tee (Slice 6) Forward text/event-stream responses to the client piece-by-piece (re-framed as HTTP/1.1 chunked) instead of buffering the whole body. Buffering withheld every event until the stream ended and hung the engine on unbounded streams. - UpstreamForwarder.ForwardAsync returns an OriginResponse: buffered for normal responses, or a live body stream for text/event-stream (message stays open). - StreamingResponseWriter pumps origin->client one chunk per read (flushed so events arrive live), accumulating a capped copy (4 MiB, reusing BodyModeResolver.DefaultInMemoryLimitBytes); beyond the cap accumulation is dropped but relaying continues, so the engine never hangs or OOMs. - PluginPipeline.RunStreamingResponseAsync runs the body pump between BeforeResponse and AfterResponse, exposing the accumulated body to read-only AfterResponse inspectors (OpenAITelemetry/OpenAIUsageDebugging) for parity. - Keep-alive preserved (chunked is self-delimiting). Tests: +12 (StreamingResponseWriter chunk framing/accumulation/cap, forwarder streaming detection). 158 green. Live-verified: local finite SSE streamed ~0.5s per event (not buffered), Transfer-Encoding: chunked, Connection: keep-alive, connection reused across two streamed responses, non-watched passthrough intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 4 Slice 7: process filtering at CONNECT (--watch-pids/--watch-process-names) Implements the last functional parity gap with the Titanium engine: restricting MITM to specific processes via --watch-pids / --watch-process-names. Matches Titanium semantics exactly — filtering applies ONLY at the CONNECT decision (IsProxiedProcess in ProxyEngine.cs), so plain HTTP stays unfiltered (documented parity quirk). - ProcessFilter resolves the client's PID from its source port and matches against WatchPids then WatchProcessNames (ordinal/case-sensitive HashSet to mirror Titanium's default Contains comparer). No filter configured -> watch all (IsEmpty short-circuit, zero overhead). - ConnectionProcessResolver shells out per-platform: lsof -i on Unix, netstat -ano on Windows. LsofParser/NetstatParser are pure, fully unit-tested partial classes (GeneratedRegex). Chosen over Windows P/Invoke (GetExtendedTcpTable) for testability and boring-tech symmetry. NOTE: the Windows netstat path is parser-tested but not live-verifiable on macOS. - Wired into ProxyConnectionHandler.HandleConnectAsync (after the h2-only check): unresolved/non-matching process -> blind-tunnel (never MITM'd). KestrelProxyEngine constructs the filter from configuration.WatchPids/WatchProcessNames. - Updated the CONNECT-flow doc diagram. Tests: +21 Kestrel (12 ProcessFilter + 4 Lsof + 5 Netstat) -> 179 green (33 Abstractions + 38 Titanium + 108 Kestrel). Live-verified on macOS: matching process name -> MITM intercept + req-log; non-matching -> blind tunnel (real cert, no -k, zero req-log lines). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 5 Slice 5a: persistent certificate authority (disk root + leaf cache) The Kestrel engine previously minted a brand-new in-memory root CA on every start, so the OS would distrust it after each restart. This makes the root (and per-host leaves) persistent on disk, mirroring the Titanium engine's CertificateDiskCache layout exactly: root at <configDir>/rootCert.pfx (overridable via DEV_PROXY_CERT_PATH), leaves at <configDir>/crts/<host>.pfx, both PKCS#12 with an empty password. configDir matches Titanium's resolution (ApplicationData/dev-proxy on mac/Linux, the executable dir on Windows). EUREKA: because the format + path match, the persistent CA loads a root that the Titanium engine ALREADY created and the OS ALREADY trusts — existing users get interception with zero new trust code (verified live: curl WITHOUT -k against a watched HTTPS host returned 200 + full MITM req-log). - LoadOrCreateRoot: load + validate (isCA, has private key, not expired) the on-disk root; regenerate + re-save (and purge the now-stale leaf cache) if absent/invalid/ expired. Per decision #5 there is no cross-version compatibility contract. - Leaves are minted on demand, cached in memory + on disk, and reused across restarts. Leaf validity is clamped to the issuer's window so reusing a near-expiry root can't produce a leaf that outlives it (CertificateRequest.Create rejects that). Filename- unsafe hosts (IPv6 ':', wildcard '*', '/') are sanitized for the disk cache key. - Disk writes are best-effort: an unwritable location degrades to in-memory-only for the run instead of failing interception. - KestrelProxyEngine now builds the CA via CertificateAuthority.CreateDefault(logger). OS-trust install (mac keychain / Windows root store) + the first-run prompt remain Slice 5b (host-side, reusing the existing trust helpers). This slice only makes the root persistent. Tests: +12 Kestrel (CertificateAuthorityTests: first-run create+persist, reuse across instances, invalid/expired/non-CA regeneration, stale-leaf purge, leaf mint+sign+ persist+reuse, leaf-not-outliving-root clamp, IP-literal SAN, filename sanitization) -> 191 green (33 + 38 + 120). Live-verified on macOS: existing root reused (fingerprint unchanged, "Loaded persisted root certificate"), leaf cached + reused across restarts, MITM works without -k. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OS-trust install + first-run wiring for the Kestrel engine (Slice 5b) Wire root-certificate OS trust for the Kestrel engine via a new IRootCertificateTrust abstraction (DevProxy.Abstractions/Proxy), implemented host-side by RootCertificateTrust (DevProxy/Proxy). The engine mints/persists its root (Slice 5a) then calls EnsureTrusted; the host performs the platform install: - macOS: trust in the login keychain via MacCertificateHelper, gated by the first-run flow (InstallCert + NoFirstRun + HasRunFlag + Y/n prompt, auto-"y" under CI/redirected stdin) — mirrors Titanium's FirstRunSetup. - Windows: install the public-only root into CurrentUser/Root, idempotent by identity (CA5380 suppressed — installing the Dev Proxy root is the proxy's explicit, user-consented purpose). NOT live-verifiable on macOS. - Linux: warn to trust manually. The trust decision is factored into a pure RootTrustPolicy.Decide table in Abstractions so it's exhaustively unit-testable without platform/console I/O; the host impl is just the thin I/O around it (option X — new interface, leave Titanium's FirstRunSetup untouched for low blast radius). CertCommand (ensure/remove) already works for Kestrel mode unchanged thanks to Slice 5a's on-disk path parity. The Kestrel engine ctor takes an optional IRootCertificateTrust (null = no trust, e.g. tests); the DEV_PROXY_ENGINE=kestrel DI toggle registers and injects RootCertificateTrust. 207 tests green (49 Abstractions incl. +16 RootTrustPolicy, 38 Titanium, 120 Kestrel). Live-verified on macOS: with an already-trusted persisted root and --no-first-run, a watched HTTPS request returned 200 WITHOUT -k plus a full MITM req-log — trust wiring doesn't regress the existing-root happy path. Fresh-machine keychain trust is covered by the unit-tested decision table (not live-exercised to avoid keychain GUI pollution); Windows store install is parser/logic-only on macOS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Graceful-teardown audit: DRY connection-close classifier (Slice 5c) Extract a single ConnectionTeardown.IsExpected(exception) classifier and use it at every client/origin read/copy/write boundary, replacing the per-site hardcoded `ex is IOException or OperationCanceledException or ...` lists. An exception is "expected" (treat as EOF/close, don't log as error) when it's a ConnectionResetException, ConnectionAbortedException, OperationCanceledException, IOException, or SocketException — the normal result of a peer disconnecting, resetting, or the connection being cancelled mid-exchange. Boundaries updated to the shared classifier: - ProxyConnectionHandler.OnConnectedAsync outer catch (keep-alive read, body read, response write, streaming pump, blind tunnel) — now also covers SocketException/ConnectionAbortedException. - StreamRelay bidirectional-copy teardown. - WriteErrorAsync best-effort write to a possibly-gone client. Fixes the one genuine noise gap: the WebSocket relay catch logged a client disconnect mid-handshake / verbatim-101-write as an ERROR. It now logs at Debug when IsExpected and reserves Error for genuine faults. 216 tests green (49 Abstractions, 38 Titanium, 129 Kestrel; +9 ConnectionTeardown covering OCE/TaskCanceled/IOException/SocketException/ConnectionReset/ ConnectionAborted true and InvalidOperation/NullReference/unrelated false). Live-verified on macOS: aborting curl mid-stream (httpbin /drip, --max-time 1) produced ZERO error/exception log lines and the engine kept serving (a following request returned 200). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden CONNECT authority parsing: IPv6, ports, malformed targets (Slice 5d) Replace the naive "split on the last colon" SplitHostPort with a pure, explicit ConnectAuthorityParser that correctly handles: - IPv6 literals: [::1] and [2001:db8::1]:8443 — the bare host (no brackets) is used for socket connect / cert minting, and a UrlHost property re-adds brackets for composing the absolute URL (https://[::1]:8443/path). - Explicit ports, including validation (reject port 0 / >65535 / non-numeric / empty). - Malformed targets: empty host, unterminated bracket, junk after ], bare (unbracketed) IPv6, invalid host names — all rejected. The authority is now parsed + validated BEFORE the proxy writes "200 Connection Established", so a malformed CONNECT is refused with a 400 instead of establishing an unusable tunnel. The decrypted-tunnel URL is composed from UrlHost so IPv6 targets produce valid absolute URLs. 237 tests green (49 Abstractions, 38 Titanium, 150 Kestrel; +21 parser tests covering reg-name/IPv4/punycode/IPv6 with and without ports plus 15 malformed cases). Live-verified on macOS: normal + explicit-:443 watched MITM return 200 (no -k); malformed CONNECT targets (port 0, bare IPv6 ::1:443, unterminated [::1) each return 400 Bad Request while a valid target returns 200 Connection Established; zero error-log noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Slice 5e: --port 0 actual-port logging + DRY host extraction Log the OS-assigned port (not the configured 0) for the Kestrel engine by reading the bound ListenOptions endpoint after StartAsync, so users running with --port 0 can connect. Extract the duplicated urlsToWatch host-derivation logic into a shared WatchedHostExtractor.ToHostRegex in DevProxy.Abstractions, used by both the Titanium ProxyEngine.LoadHostNamesFromUrls and the Kestrel HostWatchList. One implementation, one test suite (11 new tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 6 slice: engine-agnostic system-proxy on/off (no Titanium) Re-implement turning the OS system-proxy on and off without Titanium, behind a new ISystemProxyManager abstraction (Abstractions) so the Kestrel engine can use it without referencing the host (mirrors IRootCertificateTrust): - SystemProxyManager (host): Windows via WinINET (registry ProxyServer/ProxyEnable + InternetSetOption settings-changed/refresh broadcast), macOS via the existing toggle-proxy.sh, Linux warns. DllImport (not LibraryImport) to avoid enabling AllowUnsafeBlocks project-wide. - SystemProxyAddress (Abstractions, pure): normalizes wildcard/empty bind addresses to loopback for the client-facing proxy address; unit-tested. - KestrelProxyEngine: enable on start (using the actually-bound port, matters for --port 0), disable on stop. Previously the Kestrel engine set no system proxy at all — a parity gap. - StopCommand --force crash-cleanup now uses the shared manager, so a force-killed daemon restores the system proxy on Windows too (was macOS-only before). 259 tests green (+11 SystemProxyAddress). Windows WinINET path is parser/logic-tested but needs verification on a Windows host (cannot run on macOS). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 6 slice: replace Titanium RunTime with System.OperatingSystem Swap the Titanium RunTime OS-check helper for the .NET built-in equivalents (RunTime.IsWindows → OperatingSystem.IsWindows(), RunTime.IsMac → OperatingSystem.IsMacOS()) in ProxyEngine and CertCommand, and drop the now-unused `using Titanium.Web.Proxy.Helpers;` import from both. Strictly equivalent behavior; reduces Titanium API surface ahead of the cut-over. Left untouched (intentionally): - CertificateDiskCache.RunTime.IsUwpOnWindows — no built-in equivalent, and the file (a Titanium ICertificateCache impl) is deleted at cut-over anyway. - The Titanium.Web.Proxy.* log filters — still suppress real Titanium engine logs while it remains the default engine; their removal belongs in the cut-over commit. 259 tests green; Titanium engine boot-smoked clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add hermetic Kestrel parity-test gate (controllable rows) Build the DevProxy.Parity.Tests project that boots the real KestrelProxyEngine on a random localhost port against a deterministic FakeOrigin and asserts the observable proxy contract for every controllable parity-matrix row: - plain HTTP: GET/POST (small/large), large download, status relay, 204 empty-body framing, request-header passthrough - chunked request body reframing + SSE chunked response streaming - keep-alive reuse + Connection: close - plugin Respond() mocking short-circuit (origin never contacted) - Content-Length + Transfer-Encoding smuggling -> 400 guard 15 hermetic rows green. External/OS rows (HTTPS real-cert MITM, h2/gRPC, WebSocket, system-proxy, process filter) remain the scripted live pass and are already slice-verified; see files/parity-results.md for the full matrix. The harness boots only the Kestrel engine (the Titanium engine's process-global static ProxyServer + IServer make in-process co-hosting flaky, and it is deleted at cut-over); it asserts the reference behavior Titanium already exhibits. Total: 274 tests green (71 Abstractions + 38 Titanium + 150 Kestrel + 15 Parity). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Hard cut-over to the Kestrel proxy engine; remove Titanium Make the ASP.NET Core Kestrel engine the sole proxy engine and remove the Unobtanium/Titanium.Web.Proxy adapter, dependency, and dev toggle in one change. - Remove the DEV_PROXY_ENGINE toggle; AddProxyEngine always wires Kestrel, the shared CertificateAuthority, IRootCertificateTrust, and ISystemProxyManager. - Introduce a single shared CertificateAuthority DI singleton (Kestrel AddKestrelCertificateAuthority) with the root X509Certificate2 derived from it; the engine, CertCommand, ProxyController, and EntraMockResponsePlugin all resolve the same root cert. The engine no longer self-creates/disposes its CA. - Add Trust/Untrust to IRootCertificateTrust and implement them host-side (mac keychain / Windows root store / Linux warn); CertCommand uses them. - Delete ProxyEngine, CertificateDiskCache, the DevProxy.Proxy.Titanium and DevProxy.Proxy.Titanium.Tests projects, the Unobtanium PackageReference, the Titanium.Web.Proxy.* log filters, and the THIRD PARTY NOTICES Unobtanium entry. - Update docs (CONTRIBUTING, copilot-instructions) and stale comments to Kestrel. Build 0/0; 236 tests green (71 Abstractions + 150 Kestrel + 15 Parity). Live-verified on macOS: watched HTTPS MITM without -k + req-log, cert-download API, and `cert ensure` trust path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix detached-daemon state registration and bound graceful-shutdown drain Two cut-over regressions surfaced by the macOS system-proxy live test: 1. Detached mode (`--detach`) was broken. The deleted Titanium ProxyEngine wrote the initial ProxyInstanceState (gated on IsInternalDaemon); the Kestrel engine never did, so the daemon orphaned, the parent's readiness poll timed out, and the system proxy was left on. The host's UpdateStateWithApiUrlAsync only updated an existing state file, so it silently no-opped. - KestrelProxyEngine now publishes its actually-bound port back to the shared IProxyConfiguration after binding (resolves --port 0 to the OS-assigned port). - DevProxyCommand.WriteDaemonStateAsync writes the full ProxyInstanceState (Pid, ApiUrl, LogFile, ConfigFile, resolved Port, AsSystemProxy) once the engine has published its port, satisfying the parent's `Port > 0` readiness contract used by `devproxy stop` and `devproxy status`. 2. Graceful shutdown stalled after real traffic. The engine awaited app.StopAsync(CancellationToken.None), which waited unbounded for lingering client keep-alive connections to drain — fast when idle, but >10s after traffic, making `devproxy stop` falsely report "did not stop in time". The drain is now bounded (5s) so remaining connections are force-closed. Adds EnginePortPublishingTests regression coverage (verified failing without the port write-back). 237 tests green. Live-verified on macOS: `--detach` registers the daemon, `devproxy status`/`stop` find it, and the system proxy is restored on both graceful API stop and SIGTERM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename Parity.Tests to Integration.Tests post-cut-over With Titanium removed there is no second engine to compare against, so the "parity" framing no longer applies. The suite is now a black-box integration test that boots the Kestrel engine over a real socket against a FakeOrigin. Renamed the project, csproj, namespaces, *ParityTests classes/files, and the solution entry (project GUID unchanged); reworded "parity" comments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add per-plugin integration tests: harness infra + behavior/manipulation/mocking plugins Adds a CapturingLoggerFactory (collects engine-emitted RequestLogs), an inline PluginConfig section builder, and extends KestrelProxyHarness to accept a logger factory and expose its urls-to-watch set. Covers 9 hermetic, HTTP-observable plugins end-to-end through the Kestrel engine: GenericRandomError, Latency, RateLimiting, RetryAfter, Rewrite, MockResponse, Auth (reject + allow), and GraphRandomError. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CrudApi + MockRequest integration tests with InitializeAsync infra Adds PluginTestHost (a minimal DI ServiceProvider supplying HttpClient/ILogger<>/ IProxyConfiguration so plugins whose InitializeAsync builds file loaders can run in-process), TestDefaults (shared HttpClient), a FakeOrigin request recorder + /json endpoint, and a settable ConfigFile on TestProxyConfiguration. Mirrors the host's runtime-only package assets in the test csproj so plugin runtime dependencies (OpenIdConnect, JWT, OpenApi, OpenTelemetry, EF Sqlite, Newtonsoft, Azure.Identity) resolve. Covers MockRequestPlugin (outbound fire) and CrudApiPlugin (in-memory REST served through the engine, origin never contacted). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add guidance plugin integration tests (7 hermetic plugins) Adds TestExchange, which constructs the engine's real canonical session/request/ response types so guidance plugins gated on a fixed upstream host (graph.microsoft.com) can be driven at their hooks with production-fidelity inputs the loopback origin can't impersonate. Covers GraphBetaSupport, GraphClientRequestId, GraphSdk, ODataPaging, ODSPSearch, GraphConnector, and CachingGuidance, asserting the advisory log each emits. GraphSelectGuidancePlugin is documented as non-hermetic (needs the Graph metadata DB) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add reporting plugin integration tests (UrlDiscovery, ExecutionSummary, MinimalPermissions) Drives reporter plugins through AfterRecordingStopAsync exactly as the host's recording controller does: feeds engine-shaped RequestLogs (TestExchange.AsRequestLog) into RecordingArgs with a seeded Reports GlobalData slot, then asserts the structured report stored under the plugin name. MinimalPermissions runs against a temp OpenAPI spec fixture to verify offline permission analysis. GraphMinimalPermissions(+Guidance), MinimalCsom, and ApiCenter reporters are documented as live-backend (Bucket 3) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add generation plugin integration tests (HAR, mock, .http, OpenAPI, TypeSpec) Drives each generator through AfterRecordingStopAsync with an InterceptedResponse log, asserting the stored report and the generated artifact. Generators write to the process CWD, so tests redirect it to a temp dir; assembly test parallelization is disabled to keep that global swap safe. A DisabledLanguageModelClient keeps the OpenAPI/TypeSpec generators hermetic (LM is cosmetic-only; they fall back to deterministic generation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add process-level smoke test for config-driven plugin loading Spawns the real DevProxy host with a devproxyrc.json + mocks file, waits for the Kestrel engine to report its bound port (--port 0), routes an HTTP request through it as an explicit proxy, and asserts MockResponsePlugin short-circuits with the configured body. Proves end-to-end that config-driven plugin discovery + the engine wire survive the Titanium -> Kestrel cut-over. Uses a plain-HTTP watched URL so no CA trust is needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unwired body-mode subsystem; fix misleading body docs The BodyModeResolver + BodyMode/BodyCapabilities/BodyContext/BodyDirection types (319 LOC, 12 tests) were a spike-era design that the Kestrel engine never wired: Resolve() is never called in production, no plugin declares BodyCapabilities, and IHttpMessage exposes no capability property. The engine uses a fixed 4 MiB in-memory inspection cap instead. Worse, the IHttpMessage doc claimed body buffering was 'driven by plugin BodyCapabilities', misleading plugin authors toward an API that does not exist. - Delete BodyModeResolver.cs, BodyHandling.cs, and BodyModeResolverTests.cs. - Inline the one used constant as ProxyConnectionHandler.InMemoryInspectionCapBytes. - Correct the IHttpMessage body-visibility doc to describe actual behavior. - Drop stale 'every engine adapter' language (abandoned multi-engine design) from ForwardingInvariants and HeaderCollection docs. The design is preserved in git and can be restored from this parent commit if/when capability-based body handling is actually needed. A code comment + plan.md follow-up track the genuine, separate unbounded-buffered-body memory risk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename IProxyLogger.cs to LoggingContext.cs to match contents The file's 'I' prefix implied an interface, but it contains no interface — only the MessageType enum and the LoggingContext / StdioLoggingContext classes. Rename to match its primary type and theme. (StdioEvents.cs was left as-is: it correctly groups the stdio event-arg family, mirroring the existing ProxyEvents.cs convention.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add WebSocketMockResponsePlugin (reactive WS frame mocking) Net-new capability unlocked by the Kestrel engine: mock WebSocket frames end-to-end. The engine owns transport (101 handshake + framing via the framework's WebSocket.CreateFromStream); the plugin owns behavior (on-connect scripted sends + match incoming client messages by equals/contains/regex/json and reply with mapped messages, plus optional close). A watched WS upgrade is dispatched to the plugin responder (instead of the origin relay) when the plugin calls IProxySession.HandleWebSocket(handler), leaving the session in the Watched phase. The plugin must NOT set ResponseState.HasBeenSet, which would short-circuit to the mocked-HTTP ResponseWriter path before the WS dispatch branch runs. DRY scheme-normalization footgun fix: the engine reports an intercepted WS upgrade with an http(s) scheme, but authors naturally write wss://host/* in urlsToWatch and mock URLs. A shared ProxyUtils.NormalizeWebSocketScheme normalizes wss->https / ws->http and is applied at both the watch-regex build site (PluginServiceExtensions.ConvertToRegex) and the plugin's mock-URL match, so either form works and a wss:// watch entry is no longer silently relayed to origin. Tests: responder loopback unit tests, matcher unit tests, 2 e2e ClientWebSocket-over-wss-MITM integration tests, and NormalizeWebSocketScheme unit tests. Full suite green (284). Live-verified through the host binary with urlsToWatch ["wss://ws.example.test/*"]: OPEN -> welcome (onConnect) -> pong (reactive ping->pong); proxy logged "Mocking WebSocket" + "WebSocket mock established". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align WebSocket mock matching with HTTP/stdio mock vocabulary The WebSocketMockResponsePlugin invented a `matchType: equals|contains|regex| json` enum, diverging from every other Dev Proxy mock plugin, which matches with the `bodyFragment` (case-insensitive contains) + `bodyRegex` field pair. Replace the enum with the conventional vocabulary so authors familiar with the HTTP/stdio mocks reach for the same fields: - `bodyFragment` — OrdinalIgnoreCase substring - `bodyRegex` — regular expression (1s timeout) - `bodyJson` — structural JSON equality (JsonNode.DeepEquals); accepts a JSON value directly in the mocks file, matching how the HTTP response body accepts object/array/string Precedence when more than one is set: bodyJson > bodyRegex > bodyFragment (mirrors stdio's "bodyRegex wins over bodyFragment"). Null match or no criteria remains a catch-all. The genuinely WS-specific structure (url-keyed mock, onConnect pushes, reactive rules, multi-message responses) is unchanged — only the match expression is aligned. Updates the mocks-file schema and matcher unit tests accordingly. Full suite green (283). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore interactive console (hotkeys + --record) dropped in Kestrel cut-over The hard cut-over to the Kestrel engine deleted ProxyEngine.cs, which owned the interactive key loop (r/s/c/w) and the --record auto-start. KestrelProxyEngine never re-implemented them, so the documented keyboard shortcuts (technical-reference/keyboard-shortcuts.md) stopped working and `devproxy --record` no longer started recording at launch. Restore both as a host-side InteractiveConsoleService (BackgroundService), keeping the Kestrel engine headless: the engine depends only on the proxy abstractions and must not reach into host concerns like IProxyStateController. The service waits for ApplicationStarted so the banner prints after the engine's startup logs, honors --record, prints hotkeys (text mode) or API instructions (JSON mode), and dispatches keys via a testable ConsoleHotkeyHandler over an ISystemConsole seam. Adds DevProxy.Tests (the host had no unit test project) with 10 tests covering key dispatch and both banners. Full suite green (293 tests). Live-verified: --record starts recording at launch, JSON banner prints, proxy serves 200 through the new service. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Windows verification checklist for the Kestrel engine The Windows-specific runtime paths (WinINET system proxy, current-user root-store cert trust, netstat-based process filter) are unit-tested but were never live-verified on a real Windows host after the Kestrel cut-over. Add a self-contained, clone-and-run checklist covering build/tests, cert trust, system proxy on/off, detached daemon lifecycle, process filter, the restored interactive console, and a core proxy + plugin smoke pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Preserve Content-Length on HEAD responses A HEAD response carries the Content-Length a GET would return but no body (RFC 9110 §9.3.2). The engine was recomputing Content-Length from the (empty) HEAD body, so clients saw Content-Length: 0 instead of the real resource size. ResponseWriter and UpstreamForwarder are now method-aware: HEAD preserves the origin's Content-Length, never takes the streaming path, and never writes a body. * Fix API URL banner showing port 0 with --api-port 0 Resolve the OS-assigned API port from the bound server address before printing the JSON-mode banner so the curl commands reference the actual port instead of the literal 0. * Return 504 on upstream timeout instead of silently dropping the connection The forward catch excluded all OperationCanceledException so client cancellation would bubble up as a silent teardown. But an HttpClient request timeout also throws TaskCanceledException (an OperationCanceledException), so a stalled origin was misclassified as a client disconnect: the client got a dropped/reset connection after the default 100s timeout instead of a gateway error. Add a pure UpstreamFailure.Classify helper (mirroring ConnectionTeardown) that tells the two apart by the connection token: OperationCanceledException with the token NOT cancelled is an upstream timeout -> 504 Gateway Timeout; with the token cancelled it is a genuine client teardown -> silent; anything else (TLS handshake failure on an invalid upstream cert, DNS, connection refused) -> 502 Bad Gateway. Wire it into the single forward catch in ProxyConnectionHandler and add the 504 reason phrase. Closes the "upstream timeout" and "invalid upstream cert" parity rows in code with 7 unit tests on the classifier. Full suite green (306 tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clear CA2201 warning in ConnectionTeardownTests The test deliberately constructs a NullReferenceException to prove the teardown classifier does not mask a latent bug as an expected connection close. CA2201 flags constructing reserved exception types, so scope a justified suppression to that single line instead of weakening the test. Solution now builds with 0 warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump version to 4.0.0 (breaking: Kestrel engine + canonical plugin API) This is a major version bump because the migration from Titanium/Unobtanium to the Kestrel-based proxy engine introduces breaking changes: - Plugin API surface moved from Titanium types (SessionEventArgs/Request/ Response/HttpHeader) to the canonical model (IProxySession/IHttpRequest/ IHttpResponse/HeaderCollection). Third-party plugins must recompile. - Behavioral changes around HTTP/2 handling (h2-only/gRPC blind-tunnel). Copies schemas/v3.1.0 -> schemas/v4.0.0 verbatim (41 files) and updates <Version> in all four shipping projects (incl. DevProxy.Proxy.Kestrel), schema URLs in config JSON + MockResponsePlugin, installer/script/Dockerfile version strings, and refreshes packages.lock.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
waldekmastykarz
added a commit
that referenced
this pull request
Jul 24, 2026
* Add throwaway Kestrel proxy spike (Phase 0B de-risking) Validates selective decrypt, non-destructive ClientHello ALPN peek, h2-only/gRPC blind-tunnel, keep-alive isolation, and unbuffered streaming. Not part of DevProxy.sln. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add canonical HTTP proxy model to DevProxy.Abstractions (Phase 1) Introduce engine-agnostic Dev Proxy model that will replace the leaked Unobtanium/Titanium types in the plugin SDK: - IHttpHeader/IHeaderCollection/IHttpMessage/IHttpRequest/IHttpResponse and IProxySession (logical SessionId, Respond() mocking primitive), with concrete HttpHeader + HeaderCollection. - Bodies exposed as ReadOnlyMemory<byte> (decompressed-body contract). - Body strategy: BodyMode, BodyCapabilities, BodyModeResolver (pure, reconciles streaming vs full-body plugin access). - ForwardingInvariants documenting the forwarding contract + shared HopByHopHeaders set. - New DevProxy.Abstractions.Tests (xUnit), 33 tests. Purely additive: existing Titanium-based code is untouched and the full solution builds clean. Adapter wiring follows in Phase 2/3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Titanium adapter mapping engine types onto canonical model (Phase 2) Introduces DevProxy.Proxy.Titanium, the sole project referencing Unobtanium.Web.Proxy, which projects Titanium request/response/header/session types onto the engine-agnostic canonical model in DevProxy.Abstractions. - TitaniumHeaderCollection, TitaniumHttpMessageAdapter (shared DRY base over RequestResponseBase), TitaniumRequestAdapter, TitaniumResponseAdapter, TitaniumProxySession. - Body access guarded by HasBody (Titanium throws BodyNotFoundException otherwise); mutation routed through the session's SetRequestBody/ SetResponseBody since Titanium exposes no public body setter. - DevProxy.Proxy.Titanium.Tests: 38 adapter-fidelity unit tests. Purely additive and ship-safe: the existing engine and plugins are untouched. Full solution builds 0/0; 71 tests green. Cross-engine parity tests are deferred to Phase 4 (require the Kestrel engine to exist). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wire canonical IProxySession through ProxyEvents; migrate first plugin (Phase 3 slice 1) Introduces the dual-field strangler that lets plugins migrate off Titanium types one at a time while staying green: - ProxyHttpEventArgsBase exposes IProxySession ProxySession alongside the existing Titanium SessionEventArgs Session. HasRequestUrlMatch now reads the canonical request. - ProxyEngine builds a TitaniumProxySession (keyed by the existing per-session GetHashCode key) and threads it through the request/response arg construction sites; DevProxy host references DevProxy.Proxy.Titanium. - CachingGuidancePlugin fully migrated to e.ProxySession.Request and is now Titanium-free. Verified live: the cache-window warning fires through the real proxy pipeline reading IHttpRequest.RequestUri via the adapter. Full solution builds 0/0; 71 tests green. The other 21 Titanium-touching plugins are untouched and keep compiling against Session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 Wave 1: add IHttpRequest overloads to ProxyUtils Add canonical IHttpRequest overloads alongside the existing Titanium Request overloads for IsGraphRequest/IsSdkRequest/IsGraphBetaRequest/BuildGraphResponseHeaders. Extract a shared BuildGraphResponseHeadersCore to keep both overloads DRY. Additive only; Titanium overloads remain until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 guidance wave: migrate 4 guidance plugins off Titanium Migrate ODSPSearchGuidancePlugin, GraphSelectGuidancePlugin, GraphSdkGuidancePlugin, and GraphClientRequestIdGuidancePlugin to the canonical model: read request/response via e.ProxySession, helpers take IHttpRequest, header checks via Contains. Each plugin is now Titanium-free in source (LoggingContext still takes e.Session until the final wave). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 mocking wave: migrate 5 mocking plugins off Titanium Migrate AuthPlugin, CrudApiPlugin, GraphMockResponsePlugin, MockResponsePlugin, and OpenAIMockResponsePlugin to the canonical model: request/response via e.ProxySession, GenericResponse -> IProxySession.Respond, canonical HttpHeader, header methods via the canonical IHeaderCollection. Helper signatures and the CrudApi action-handler delegate now use IProxySession/IHttpRequest/ProxyRequestArgs. LoggingContext still takes e.Session until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 inspection-lite: migrate HttpUtils + OpenAI usage/telemetry streaming off Titanium Change HttpUtils.IsStreamingResponse/GetBodyFromStreamingResponse to take the canonical IHttpResponse. Migrate OpenAIUsageDebuggingPlugin fully to e.ProxySession (status comparisons cast HttpStatusCode to int). Point OpenAITelemetryPlugin's streaming call site at e.ProxySession.Response. DevToolsPlugin and the OpenAITelemetry RequestLog.Context.Session readers remain for the final LoggingContext wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 throttling group: migrate ThrottlerInfo + 5 throttling plugins off Titanium Flip ThrottlerInfo.ShouldThrottle delegate from Func<Request,...> to Func<IHttpRequest,...> (PluginEvents) and migrate the coordinated consumers: GraphUtils.BuildThrottleKey, RateLimitingPlugin, GraphRandomErrorPlugin, GenericRandomErrorPlugin, LanguageModelRateLimitingPlugin, RetryAfterPlugin. RetryAfter's GenericResponse becomes IProxySession.Respond with canonical HttpHeader. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Migrate plugins off Titanium onto canonical HTTP model (Phase 3 final wave) Complete the strangler migration: flip LoggingContext and the proxy event args to the engine-agnostic IProxySession model, migrate all remaining plugins off Titanium.Web.Proxy types, and drop the Unobtanium.Web.Proxy package reference from DevProxy.Abstractions and DevProxy.Plugins. - LoggingContext/RequestLog now carry IProxySession (was SessionEventArgs); ProxyHttpEventArgsBase no longer exposes the Titanium Session. - All Behavior/Generation/Guidance/Inspection/Manipulation/Mocking/Reporting plugins read e.ProxySession.Request/Response and the canonical header API. - DevToolsPlugin re-keys _responseBody/CDP requestId on the phase-stable ProxySession.SessionId instead of the Titanium Request hash code. - Add Url setter and response HttpVersion to the canonical model (+ Titanium adapters) to preserve Rewrite and HAR behavior. - Remove dead Titanium FuncExtensions helper. DevProxy.Abstractions and DevProxy.Plugins are now Titanium-free; only the host engine and the Titanium adapter retain the dependency. Build 0/0, 33+38 tests green, live smoke test verified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Kestrel proxy engine vertical slice (Phase 4) Introduce DevProxy.Proxy.Kestrel: a forward-proxy engine built on ASP.NET Core Kestrel that runs the Dev Proxy plugin pipeline against the canonical HTTP model, as the eventual replacement for the Titanium engine. Slice-1 scope (dev-toggle via DEV_PROXY_ENGINE=kestrel, for golden-output comparison against Titanium; not a shipped fallback): - KestrelProxyEngine hosts a raw TCP endpoint via UseConnectionHandler. - ProxyConnectionHandler: HTTP/1.1 parse, CONNECT->SslStream MITM, forward, write-back; catches client disconnect/cancel as normal teardown. - PluginPipeline mirrors ProxyEngine semantics (watched/mocked/notwatched, Before/After request+response, per-exchange SessionData) and emits request logs inside the requestId scope so the console formatter groups plugin and engine lines identically to Titanium. - UpstreamForwarder/ResponseWriter apply ForwardingInvariants (hop-by-hop stripping, decompressed bodies, Content-Length/Encoding fix-up). - In-memory CertificateAuthority + DuplexPipeStream ported from the POC. Engine selection wired via DEV_PROXY_ENGINE in DI; ProxyConsoleFormatter suppresses the engine-name prefix for KestrelProxyEngine (as for ProxyEngine). Deferred hardening (tracked): keep-alive, blind-tunnel for non-watched and h2-only/gRPC (ClientHello peek), WebSocket relay, chunked/trailers/100-continue, body-mode streaming, process filter, persistent CA, host-watch DRY consolidation. Full solution builds 0/0; 71 existing tests green; live-verified plain HTTP passthrough, HTTPS MITM, GenericRandomError/RetryAfter mocking with correct console output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Kestrel proxy engine test project (Phase 4) Add DevProxy.Proxy.Kestrel.Tests with 28 unit tests covering the HTTP/1.1 parser, host watch list (including order-dependent exclusion semantics), response writer, and the mutable HTTP model / CanonicalProxySession. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add selective decrypt + ALPN blind-tunnel to Kestrel engine (Phase 4) At CONNECT time the engine now non-destructively peeks the TLS ClientHello (SNI + ALPN) and decides whether to intercept: - non-watched hosts are blind-tunnelled (encrypted bytes relayed, never MITM'd) - watched hosts whose client is h2-only (gRPC, no http/1.1 fallback) are blind-tunnelled so they stop failing - otherwise the engine terminates TLS advertising http/1.1 only, so h2-capable clients downgrade and are intercepted as HTTP/1.1 Adds a minimal tolerant TlsClientHello parser, wires HostWatchList into the connection handler, and relays blind tunnels bidirectionally with a linked cancellation token so both copy tasks finish before the streams are disposed. Covered by 9 new TlsClientHello unit tests (37 Kestrel tests total). Verified live: watched host downgrades h2->http/1.1 and is intercepted; example.com (non-watched) validates its real cert without -k, proving it is never MITM'd. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add keep-alive + per-request state isolation to Kestrel engine (Phase 4 slice 3) Replace the static one-shot HTTP/1.1 reader with a stateful per-connection Http1ConnectionReader that owns the inter-request buffer, so bytes read past one request's body (a pipelined next request) are retained instead of dropped. This makes client-side keep-alive correct. - Http1ConnectionReader: stateful ReadHeadAsync/ReadBodyAsync owning _pending. - Http1RequestReader: reduced to stateless parse helpers (ParseHead, GetContentLength, IndexOfDoubleCrlf). - ProxyConnectionHandler.ServeConnectionAsync: keep-alive loop over both plain HTTP and decrypted CONNECT streams; each exchange builds a fresh CanonicalProxySession (new GUID + Interlocked requestId) so plugin state never leaks across requests on a reused connection. - ShouldKeepAlive (RFC 9112 9.3): persistent by default on HTTP/1.1, opt-in on HTTP/1.0; forces Connection: close for Connection: close, Transfer-Encoding (chunked, not yet reframable) and Expect: 100-continue (unsupported), so an unframable body can never corrupt the next request. - ResponseWriter.WriteAsync: keepAlive flag drives the Connection header. Tests: +15 Kestrel (stateful reader incl. pipelining/surplus-retention, ShouldKeepAlive matrix, keep-alive/close response-writer). 123 green total. Verified live: two URLs in one curl invocation reuse one TCP connection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add WebSocket transparent relay to Kestrel engine (Phase 4 slice 4) Watched-host WebSocket upgrades (ws:// plain + wss:// after MITM) now relay through the Kestrel engine for parity with Titanium, which relays WS opaquely (no plugin sees frames). HttpClient can't carry a 101 upgrade, so WebSocketRelay opens its own raw socket to the origin (TLS w/ http/1.1 ALPN for wss), replays the handshake in origin-form (preserving Upgrade/Connection/Sec-WebSocket-*, stripping Proxy-*), writes the origin's 101 back to the client verbatim, then splices frames raw via a shared StreamRelay (also now used by BlindTunnelAsync, DRY). Critical ordering: the response pipeline (which flushes the buffered req-log) runs in the onHandshakeResponse callback, right after the 101 and before the splice blocks, else a watched WS never logs until the socket closes. A handshakeObserved fallback flush covers origin-connect failure. Tests: ParseResponseHead unit tests, StreamRelay bidirectional copy, and an end-to-end RelayAsync test against a loopback fake origin (asserts origin-form handshake replay, header preservation, 101 verbatim write-back, frame splicing, and onHandshakeResponse invocation). 128 tests green (+5 Kestrel). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add chunked request bodies + 100-continue to Kestrel engine (Phase 4 slice 5) Closes the biggest remaining HTTP/1.1 correctness gap on the request read path. Previously a chunked request body was silently dropped (Content-Length defaulted to 0, so an empty body was forwarded) and Expect: 100-continue hung (the proxy never sent the interim 100). Response-side chunked already worked via HttpClient. - Http1ConnectionReader.ReadChunkedBodyAsync decodes chunked framing (size lines, chunk extensions, CRLF delimiters) into one buffer, consumes and drops trailers, and retains surplus for the next pipelined request. Shared _pending fill primitives (PullAsync/ReadExactlyAsync/ReadLineAsync/TakeFromPending), reused by ReadBodyAsync. - Http1RequestReader.DetectBodyFraming (None/ContentLength/Chunked/Conflicting) and HasExpectContinue. A request declaring both Content-Length and chunked Transfer-Encoding is Conflicting and refused with 400 (RFC 9112 6.3.3 smuggling guard). - ResponseWriter.WriteContinueAsync sends 100 Continue; ExchangeAsync answers Expect: 100-continue itself before reading the body (the buffering model can't relay the origin's interim 100). Expect is stripped on forward. - ShouldKeepAlive no longer forces close for chunked/Expect; malformed chunked framing replies 400 and closes. 146 tests green (+18). Verified live through the MITM proxy: a chunked upload reached httpbin with the body intact and reframed to Content-Length (no Transfer-Encoding), the client got 100 Continue, two chunked POSTs reused one TCP connection, and a raw Content-Length+Transfer-Encoding request got 400. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Kestrel engine: incremental SSE streaming with capped tee (Slice 6) Forward text/event-stream responses to the client piece-by-piece (re-framed as HTTP/1.1 chunked) instead of buffering the whole body. Buffering withheld every event until the stream ended and hung the engine on unbounded streams. - UpstreamForwarder.ForwardAsync returns an OriginResponse: buffered for normal responses, or a live body stream for text/event-stream (message stays open). - StreamingResponseWriter pumps origin->client one chunk per read (flushed so events arrive live), accumulating a capped copy (4 MiB, reusing BodyModeResolver.DefaultInMemoryLimitBytes); beyond the cap accumulation is dropped but relaying continues, so the engine never hangs or OOMs. - PluginPipeline.RunStreamingResponseAsync runs the body pump between BeforeResponse and AfterResponse, exposing the accumulated body to read-only AfterResponse inspectors (OpenAITelemetry/OpenAIUsageDebugging) for parity. - Keep-alive preserved (chunked is self-delimiting). Tests: +12 (StreamingResponseWriter chunk framing/accumulation/cap, forwarder streaming detection). 158 green. Live-verified: local finite SSE streamed ~0.5s per event (not buffered), Transfer-Encoding: chunked, Connection: keep-alive, connection reused across two streamed responses, non-watched passthrough intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 4 Slice 7: process filtering at CONNECT (--watch-pids/--watch-process-names) Implements the last functional parity gap with the Titanium engine: restricting MITM to specific processes via --watch-pids / --watch-process-names. Matches Titanium semantics exactly — filtering applies ONLY at the CONNECT decision (IsProxiedProcess in ProxyEngine.cs), so plain HTTP stays unfiltered (documented parity quirk). - ProcessFilter resolves the client's PID from its source port and matches against WatchPids then WatchProcessNames (ordinal/case-sensitive HashSet to mirror Titanium's default Contains comparer). No filter configured -> watch all (IsEmpty short-circuit, zero overhead). - ConnectionProcessResolver shells out per-platform: lsof -i on Unix, netstat -ano on Windows. LsofParser/NetstatParser are pure, fully unit-tested partial classes (GeneratedRegex). Chosen over Windows P/Invoke (GetExtendedTcpTable) for testability and boring-tech symmetry. NOTE: the Windows netstat path is parser-tested but not live-verifiable on macOS. - Wired into ProxyConnectionHandler.HandleConnectAsync (after the h2-only check): unresolved/non-matching process -> blind-tunnel (never MITM'd). KestrelProxyEngine constructs the filter from configuration.WatchPids/WatchProcessNames. - Updated the CONNECT-flow doc diagram. Tests: +21 Kestrel (12 ProcessFilter + 4 Lsof + 5 Netstat) -> 179 green (33 Abstractions + 38 Titanium + 108 Kestrel). Live-verified on macOS: matching process name -> MITM intercept + req-log; non-matching -> blind tunnel (real cert, no -k, zero req-log lines). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 5 Slice 5a: persistent certificate authority (disk root + leaf cache) The Kestrel engine previously minted a brand-new in-memory root CA on every start, so the OS would distrust it after each restart. This makes the root (and per-host leaves) persistent on disk, mirroring the Titanium engine's CertificateDiskCache layout exactly: root at <configDir>/rootCert.pfx (overridable via DEV_PROXY_CERT_PATH), leaves at <configDir>/crts/<host>.pfx, both PKCS#12 with an empty password. configDir matches Titanium's resolution (ApplicationData/dev-proxy on mac/Linux, the executable dir on Windows). EUREKA: because the format + path match, the persistent CA loads a root that the Titanium engine ALREADY created and the OS ALREADY trusts — existing users get interception with zero new trust code (verified live: curl WITHOUT -k against a watched HTTPS host returned 200 + full MITM req-log). - LoadOrCreateRoot: load + validate (isCA, has private key, not expired) the on-disk root; regenerate + re-save (and purge the now-stale leaf cache) if absent/invalid/ expired. Per decision #5 there is no cross-version compatibility contract. - Leaves are minted on demand, cached in memory + on disk, and reused across restarts. Leaf validity is clamped to the issuer's window so reusing a near-expiry root can't produce a leaf that outlives it (CertificateRequest.Create rejects that). Filename- unsafe hosts (IPv6 ':', wildcard '*', '/') are sanitized for the disk cache key. - Disk writes are best-effort: an unwritable location degrades to in-memory-only for the run instead of failing interception. - KestrelProxyEngine now builds the CA via CertificateAuthority.CreateDefault(logger). OS-trust install (mac keychain / Windows root store) + the first-run prompt remain Slice 5b (host-side, reusing the existing trust helpers). This slice only makes the root persistent. Tests: +12 Kestrel (CertificateAuthorityTests: first-run create+persist, reuse across instances, invalid/expired/non-CA regeneration, stale-leaf purge, leaf mint+sign+ persist+reuse, leaf-not-outliving-root clamp, IP-literal SAN, filename sanitization) -> 191 green (33 + 38 + 120). Live-verified on macOS: existing root reused (fingerprint unchanged, "Loaded persisted root certificate"), leaf cached + reused across restarts, MITM works without -k. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OS-trust install + first-run wiring for the Kestrel engine (Slice 5b) Wire root-certificate OS trust for the Kestrel engine via a new IRootCertificateTrust abstraction (DevProxy.Abstractions/Proxy), implemented host-side by RootCertificateTrust (DevProxy/Proxy). The engine mints/persists its root (Slice 5a) then calls EnsureTrusted; the host performs the platform install: - macOS: trust in the login keychain via MacCertificateHelper, gated by the first-run flow (InstallCert + NoFirstRun + HasRunFlag + Y/n prompt, auto-"y" under CI/redirected stdin) — mirrors Titanium's FirstRunSetup. - Windows: install the public-only root into CurrentUser/Root, idempotent by identity (CA5380 suppressed — installing the Dev Proxy root is the proxy's explicit, user-consented purpose). NOT live-verifiable on macOS. - Linux: warn to trust manually. The trust decision is factored into a pure RootTrustPolicy.Decide table in Abstractions so it's exhaustively unit-testable without platform/console I/O; the host impl is just the thin I/O around it (option X — new interface, leave Titanium's FirstRunSetup untouched for low blast radius). CertCommand (ensure/remove) already works for Kestrel mode unchanged thanks to Slice 5a's on-disk path parity. The Kestrel engine ctor takes an optional IRootCertificateTrust (null = no trust, e.g. tests); the DEV_PROXY_ENGINE=kestrel DI toggle registers and injects RootCertificateTrust. 207 tests green (49 Abstractions incl. +16 RootTrustPolicy, 38 Titanium, 120 Kestrel). Live-verified on macOS: with an already-trusted persisted root and --no-first-run, a watched HTTPS request returned 200 WITHOUT -k plus a full MITM req-log — trust wiring doesn't regress the existing-root happy path. Fresh-machine keychain trust is covered by the unit-tested decision table (not live-exercised to avoid keychain GUI pollution); Windows store install is parser/logic-only on macOS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Graceful-teardown audit: DRY connection-close classifier (Slice 5c) Extract a single ConnectionTeardown.IsExpected(exception) classifier and use it at every client/origin read/copy/write boundary, replacing the per-site hardcoded `ex is IOException or OperationCanceledException or ...` lists. An exception is "expected" (treat as EOF/close, don't log as error) when it's a ConnectionResetException, ConnectionAbortedException, OperationCanceledException, IOException, or SocketException — the normal result of a peer disconnecting, resetting, or the connection being cancelled mid-exchange. Boundaries updated to the shared classifier: - ProxyConnectionHandler.OnConnectedAsync outer catch (keep-alive read, body read, response write, streaming pump, blind tunnel) — now also covers SocketException/ConnectionAbortedException. - StreamRelay bidirectional-copy teardown. - WriteErrorAsync best-effort write to a possibly-gone client. Fixes the one genuine noise gap: the WebSocket relay catch logged a client disconnect mid-handshake / verbatim-101-write as an ERROR. It now logs at Debug when IsExpected and reserves Error for genuine faults. 216 tests green (49 Abstractions, 38 Titanium, 129 Kestrel; +9 ConnectionTeardown covering OCE/TaskCanceled/IOException/SocketException/ConnectionReset/ ConnectionAborted true and InvalidOperation/NullReference/unrelated false). Live-verified on macOS: aborting curl mid-stream (httpbin /drip, --max-time 1) produced ZERO error/exception log lines and the engine kept serving (a following request returned 200). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden CONNECT authority parsing: IPv6, ports, malformed targets (Slice 5d) Replace the naive "split on the last colon" SplitHostPort with a pure, explicit ConnectAuthorityParser that correctly handles: - IPv6 literals: [::1] and [2001:db8::1]:8443 — the bare host (no brackets) is used for socket connect / cert minting, and a UrlHost property re-adds brackets for composing the absolute URL (https://[::1]:8443/path). - Explicit ports, including validation (reject port 0 / >65535 / non-numeric / empty). - Malformed targets: empty host, unterminated bracket, junk after ], bare (unbracketed) IPv6, invalid host names — all rejected. The authority is now parsed + validated BEFORE the proxy writes "200 Connection Established", so a malformed CONNECT is refused with a 400 instead of establishing an unusable tunnel. The decrypted-tunnel URL is composed from UrlHost so IPv6 targets produce valid absolute URLs. 237 tests green (49 Abstractions, 38 Titanium, 150 Kestrel; +21 parser tests covering reg-name/IPv4/punycode/IPv6 with and without ports plus 15 malformed cases). Live-verified on macOS: normal + explicit-:443 watched MITM return 200 (no -k); malformed CONNECT targets (port 0, bare IPv6 ::1:443, unterminated [::1) each return 400 Bad Request while a valid target returns 200 Connection Established; zero error-log noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Slice 5e: --port 0 actual-port logging + DRY host extraction Log the OS-assigned port (not the configured 0) for the Kestrel engine by reading the bound ListenOptions endpoint after StartAsync, so users running with --port 0 can connect. Extract the duplicated urlsToWatch host-derivation logic into a shared WatchedHostExtractor.ToHostRegex in DevProxy.Abstractions, used by both the Titanium ProxyEngine.LoadHostNamesFromUrls and the Kestrel HostWatchList. One implementation, one test suite (11 new tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 6 slice: engine-agnostic system-proxy on/off (no Titanium) Re-implement turning the OS system-proxy on and off without Titanium, behind a new ISystemProxyManager abstraction (Abstractions) so the Kestrel engine can use it without referencing the host (mirrors IRootCertificateTrust): - SystemProxyManager (host): Windows via WinINET (registry ProxyServer/ProxyEnable + InternetSetOption settings-changed/refresh broadcast), macOS via the existing toggle-proxy.sh, Linux warns. DllImport (not LibraryImport) to avoid enabling AllowUnsafeBlocks project-wide. - SystemProxyAddress (Abstractions, pure): normalizes wildcard/empty bind addresses to loopback for the client-facing proxy address; unit-tested. - KestrelProxyEngine: enable on start (using the actually-bound port, matters for --port 0), disable on stop. Previously the Kestrel engine set no system proxy at all — a parity gap. - StopCommand --force crash-cleanup now uses the shared manager, so a force-killed daemon restores the system proxy on Windows too (was macOS-only before). 259 tests green (+11 SystemProxyAddress). Windows WinINET path is parser/logic-tested but needs verification on a Windows host (cannot run on macOS). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 6 slice: replace Titanium RunTime with System.OperatingSystem Swap the Titanium RunTime OS-check helper for the .NET built-in equivalents (RunTime.IsWindows → OperatingSystem.IsWindows(), RunTime.IsMac → OperatingSystem.IsMacOS()) in ProxyEngine and CertCommand, and drop the now-unused `using Titanium.Web.Proxy.Helpers;` import from both. Strictly equivalent behavior; reduces Titanium API surface ahead of the cut-over. Left untouched (intentionally): - CertificateDiskCache.RunTime.IsUwpOnWindows — no built-in equivalent, and the file (a Titanium ICertificateCache impl) is deleted at cut-over anyway. - The Titanium.Web.Proxy.* log filters — still suppress real Titanium engine logs while it remains the default engine; their removal belongs in the cut-over commit. 259 tests green; Titanium engine boot-smoked clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add hermetic Kestrel parity-test gate (controllable rows) Build the DevProxy.Parity.Tests project that boots the real KestrelProxyEngine on a random localhost port against a deterministic FakeOrigin and asserts the observable proxy contract for every controllable parity-matrix row: - plain HTTP: GET/POST (small/large), large download, status relay, 204 empty-body framing, request-header passthrough - chunked request body reframing + SSE chunked response streaming - keep-alive reuse + Connection: close - plugin Respond() mocking short-circuit (origin never contacted) - Content-Length + Transfer-Encoding smuggling -> 400 guard 15 hermetic rows green. External/OS rows (HTTPS real-cert MITM, h2/gRPC, WebSocket, system-proxy, process filter) remain the scripted live pass and are already slice-verified; see files/parity-results.md for the full matrix. The harness boots only the Kestrel engine (the Titanium engine's process-global static ProxyServer + IServer make in-process co-hosting flaky, and it is deleted at cut-over); it asserts the reference behavior Titanium already exhibits. Total: 274 tests green (71 Abstractions + 38 Titanium + 150 Kestrel + 15 Parity). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Hard cut-over to the Kestrel proxy engine; remove Titanium Make the ASP.NET Core Kestrel engine the sole proxy engine and remove the Unobtanium/Titanium.Web.Proxy adapter, dependency, and dev toggle in one change. - Remove the DEV_PROXY_ENGINE toggle; AddProxyEngine always wires Kestrel, the shared CertificateAuthority, IRootCertificateTrust, and ISystemProxyManager. - Introduce a single shared CertificateAuthority DI singleton (Kestrel AddKestrelCertificateAuthority) with the root X509Certificate2 derived from it; the engine, CertCommand, ProxyController, and EntraMockResponsePlugin all resolve the same root cert. The engine no longer self-creates/disposes its CA. - Add Trust/Untrust to IRootCertificateTrust and implement them host-side (mac keychain / Windows root store / Linux warn); CertCommand uses them. - Delete ProxyEngine, CertificateDiskCache, the DevProxy.Proxy.Titanium and DevProxy.Proxy.Titanium.Tests projects, the Unobtanium PackageReference, the Titanium.Web.Proxy.* log filters, and the THIRD PARTY NOTICES Unobtanium entry. - Update docs (CONTRIBUTING, copilot-instructions) and stale comments to Kestrel. Build 0/0; 236 tests green (71 Abstractions + 150 Kestrel + 15 Parity). Live-verified on macOS: watched HTTPS MITM without -k + req-log, cert-download API, and `cert ensure` trust path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix detached-daemon state registration and bound graceful-shutdown drain Two cut-over regressions surfaced by the macOS system-proxy live test: 1. Detached mode (`--detach`) was broken. The deleted Titanium ProxyEngine wrote the initial ProxyInstanceState (gated on IsInternalDaemon); the Kestrel engine never did, so the daemon orphaned, the parent's readiness poll timed out, and the system proxy was left on. The host's UpdateStateWithApiUrlAsync only updated an existing state file, so it silently no-opped. - KestrelProxyEngine now publishes its actually-bound port back to the shared IProxyConfiguration after binding (resolves --port 0 to the OS-assigned port). - DevProxyCommand.WriteDaemonStateAsync writes the full ProxyInstanceState (Pid, ApiUrl, LogFile, ConfigFile, resolved Port, AsSystemProxy) once the engine has published its port, satisfying the parent's `Port > 0` readiness contract used by `devproxy stop` and `devproxy status`. 2. Graceful shutdown stalled after real traffic. The engine awaited app.StopAsync(CancellationToken.None), which waited unbounded for lingering client keep-alive connections to drain — fast when idle, but >10s after traffic, making `devproxy stop` falsely report "did not stop in time". The drain is now bounded (5s) so remaining connections are force-closed. Adds EnginePortPublishingTests regression coverage (verified failing without the port write-back). 237 tests green. Live-verified on macOS: `--detach` registers the daemon, `devproxy status`/`stop` find it, and the system proxy is restored on both graceful API stop and SIGTERM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename Parity.Tests to Integration.Tests post-cut-over With Titanium removed there is no second engine to compare against, so the "parity" framing no longer applies. The suite is now a black-box integration test that boots the Kestrel engine over a real socket against a FakeOrigin. Renamed the project, csproj, namespaces, *ParityTests classes/files, and the solution entry (project GUID unchanged); reworded "parity" comments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add per-plugin integration tests: harness infra + behavior/manipulation/mocking plugins Adds a CapturingLoggerFactory (collects engine-emitted RequestLogs), an inline PluginConfig section builder, and extends KestrelProxyHarness to accept a logger factory and expose its urls-to-watch set. Covers 9 hermetic, HTTP-observable plugins end-to-end through the Kestrel engine: GenericRandomError, Latency, RateLimiting, RetryAfter, Rewrite, MockResponse, Auth (reject + allow), and GraphRandomError. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CrudApi + MockRequest integration tests with InitializeAsync infra Adds PluginTestHost (a minimal DI ServiceProvider supplying HttpClient/ILogger<>/ IProxyConfiguration so plugins whose InitializeAsync builds file loaders can run in-process), TestDefaults (shared HttpClient), a FakeOrigin request recorder + /json endpoint, and a settable ConfigFile on TestProxyConfiguration. Mirrors the host's runtime-only package assets in the test csproj so plugin runtime dependencies (OpenIdConnect, JWT, OpenApi, OpenTelemetry, EF Sqlite, Newtonsoft, Azure.Identity) resolve. Covers MockRequestPlugin (outbound fire) and CrudApiPlugin (in-memory REST served through the engine, origin never contacted). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add guidance plugin integration tests (7 hermetic plugins) Adds TestExchange, which constructs the engine's real canonical session/request/ response types so guidance plugins gated on a fixed upstream host (graph.microsoft.com) can be driven at their hooks with production-fidelity inputs the loopback origin can't impersonate. Covers GraphBetaSupport, GraphClientRequestId, GraphSdk, ODataPaging, ODSPSearch, GraphConnector, and CachingGuidance, asserting the advisory log each emits. GraphSelectGuidancePlugin is documented as non-hermetic (needs the Graph metadata DB) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add reporting plugin integration tests (UrlDiscovery, ExecutionSummary, MinimalPermissions) Drives reporter plugins through AfterRecordingStopAsync exactly as the host's recording controller does: feeds engine-shaped RequestLogs (TestExchange.AsRequestLog) into RecordingArgs with a seeded Reports GlobalData slot, then asserts the structured report stored under the plugin name. MinimalPermissions runs against a temp OpenAPI spec fixture to verify offline permission analysis. GraphMinimalPermissions(+Guidance), MinimalCsom, and ApiCenter reporters are documented as live-backend (Bucket 3) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add generation plugin integration tests (HAR, mock, .http, OpenAPI, TypeSpec) Drives each generator through AfterRecordingStopAsync with an InterceptedResponse log, asserting the stored report and the generated artifact. Generators write to the process CWD, so tests redirect it to a temp dir; assembly test parallelization is disabled to keep that global swap safe. A DisabledLanguageModelClient keeps the OpenAPI/TypeSpec generators hermetic (LM is cosmetic-only; they fall back to deterministic generation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add process-level smoke test for config-driven plugin loading Spawns the real DevProxy host with a devproxyrc.json + mocks file, waits for the Kestrel engine to report its bound port (--port 0), routes an HTTP request through it as an explicit proxy, and asserts MockResponsePlugin short-circuits with the configured body. Proves end-to-end that config-driven plugin discovery + the engine wire survive the Titanium -> Kestrel cut-over. Uses a plain-HTTP watched URL so no CA trust is needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unwired body-mode subsystem; fix misleading body docs The BodyModeResolver + BodyMode/BodyCapabilities/BodyContext/BodyDirection types (319 LOC, 12 tests) were a spike-era design that the Kestrel engine never wired: Resolve() is never called in production, no plugin declares BodyCapabilities, and IHttpMessage exposes no capability property. The engine uses a fixed 4 MiB in-memory inspection cap instead. Worse, the IHttpMessage doc claimed body buffering was 'driven by plugin BodyCapabilities', misleading plugin authors toward an API that does not exist. - Delete BodyModeResolver.cs, BodyHandling.cs, and BodyModeResolverTests.cs. - Inline the one used constant as ProxyConnectionHandler.InMemoryInspectionCapBytes. - Correct the IHttpMessage body-visibility doc to describe actual behavior. - Drop stale 'every engine adapter' language (abandoned multi-engine design) from ForwardingInvariants and HeaderCollection docs. The design is preserved in git and can be restored from this parent commit if/when capability-based body handling is actually needed. A code comment + plan.md follow-up track the genuine, separate unbounded-buffered-body memory risk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename IProxyLogger.cs to LoggingContext.cs to match contents The file's 'I' prefix implied an interface, but it contains no interface — only the MessageType enum and the LoggingContext / StdioLoggingContext classes. Rename to match its primary type and theme. (StdioEvents.cs was left as-is: it correctly groups the stdio event-arg family, mirroring the existing ProxyEvents.cs convention.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add WebSocketMockResponsePlugin (reactive WS frame mocking) Net-new capability unlocked by the Kestrel engine: mock WebSocket frames end-to-end. The engine owns transport (101 handshake + framing via the framework's WebSocket.CreateFromStream); the plugin owns behavior (on-connect scripted sends + match incoming client messages by equals/contains/regex/json and reply with mapped messages, plus optional close). A watched WS upgrade is dispatched to the plugin responder (instead of the origin relay) when the plugin calls IProxySession.HandleWebSocket(handler), leaving the session in the Watched phase. The plugin must NOT set ResponseState.HasBeenSet, which would short-circuit to the mocked-HTTP ResponseWriter path before the WS dispatch branch runs. DRY scheme-normalization footgun fix: the engine reports an intercepted WS upgrade with an http(s) scheme, but authors naturally write wss://host/* in urlsToWatch and mock URLs. A shared ProxyUtils.NormalizeWebSocketScheme normalizes wss->https / ws->http and is applied at both the watch-regex build site (PluginServiceExtensions.ConvertToRegex) and the plugin's mock-URL match, so either form works and a wss:// watch entry is no longer silently relayed to origin. Tests: responder loopback unit tests, matcher unit tests, 2 e2e ClientWebSocket-over-wss-MITM integration tests, and NormalizeWebSocketScheme unit tests. Full suite green (284). Live-verified through the host binary with urlsToWatch ["wss://ws.example.test/*"]: OPEN -> welcome (onConnect) -> pong (reactive ping->pong); proxy logged "Mocking WebSocket" + "WebSocket mock established". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align WebSocket mock matching with HTTP/stdio mock vocabulary The WebSocketMockResponsePlugin invented a `matchType: equals|contains|regex| json` enum, diverging from every other Dev Proxy mock plugin, which matches with the `bodyFragment` (case-insensitive contains) + `bodyRegex` field pair. Replace the enum with the conventional vocabulary so authors familiar with the HTTP/stdio mocks reach for the same fields: - `bodyFragment` — OrdinalIgnoreCase substring - `bodyRegex` — regular expression (1s timeout) - `bodyJson` — structural JSON equality (JsonNode.DeepEquals); accepts a JSON value directly in the mocks file, matching how the HTTP response body accepts object/array/string Precedence when more than one is set: bodyJson > bodyRegex > bodyFragment (mirrors stdio's "bodyRegex wins over bodyFragment"). Null match or no criteria remains a catch-all. The genuinely WS-specific structure (url-keyed mock, onConnect pushes, reactive rules, multi-message responses) is unchanged — only the match expression is aligned. Updates the mocks-file schema and matcher unit tests accordingly. Full suite green (283). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore interactive console (hotkeys + --record) dropped in Kestrel cut-over The hard cut-over to the Kestrel engine deleted ProxyEngine.cs, which owned the interactive key loop (r/s/c/w) and the --record auto-start. KestrelProxyEngine never re-implemented them, so the documented keyboard shortcuts (technical-reference/keyboard-shortcuts.md) stopped working and `devproxy --record` no longer started recording at launch. Restore both as a host-side InteractiveConsoleService (BackgroundService), keeping the Kestrel engine headless: the engine depends only on the proxy abstractions and must not reach into host concerns like IProxyStateController. The service waits for ApplicationStarted so the banner prints after the engine's startup logs, honors --record, prints hotkeys (text mode) or API instructions (JSON mode), and dispatches keys via a testable ConsoleHotkeyHandler over an ISystemConsole seam. Adds DevProxy.Tests (the host had no unit test project) with 10 tests covering key dispatch and both banners. Full suite green (293 tests). Live-verified: --record starts recording at launch, JSON banner prints, proxy serves 200 through the new service. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Windows verification checklist for the Kestrel engine The Windows-specific runtime paths (WinINET system proxy, current-user root-store cert trust, netstat-based process filter) are unit-tested but were never live-verified on a real Windows host after the Kestrel cut-over. Add a self-contained, clone-and-run checklist covering build/tests, cert trust, system proxy on/off, detached daemon lifecycle, process filter, the restored interactive console, and a core proxy + plugin smoke pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Preserve Content-Length on HEAD responses A HEAD response carries the Content-Length a GET would return but no body (RFC 9110 §9.3.2). The engine was recomputing Content-Length from the (empty) HEAD body, so clients saw Content-Length: 0 instead of the real resource size. ResponseWriter and UpstreamForwarder are now method-aware: HEAD preserves the origin's Content-Length, never takes the streaming path, and never writes a body. * Fix API URL banner showing port 0 with --api-port 0 Resolve the OS-assigned API port from the bound server address before printing the JSON-mode banner so the curl commands reference the actual port instead of the literal 0. * Return 504 on upstream timeout instead of silently dropping the connection The forward catch excluded all OperationCanceledException so client cancellation would bubble up as a silent teardown. But an HttpClient request timeout also throws TaskCanceledException (an OperationCanceledException), so a stalled origin was misclassified as a client disconnect: the client got a dropped/reset connection after the default 100s timeout instead of a gateway error. Add a pure UpstreamFailure.Classify helper (mirroring ConnectionTeardown) that tells the two apart by the connection token: OperationCanceledException with the token NOT cancelled is an upstream timeout -> 504 Gateway Timeout; with the token cancelled it is a genuine client teardown -> silent; anything else (TLS handshake failure on an invalid upstream cert, DNS, connection refused) -> 502 Bad Gateway. Wire it into the single forward catch in ProxyConnectionHandler and add the 504 reason phrase. Closes the "upstream timeout" and "invalid upstream cert" parity rows in code with 7 unit tests on the classifier. Full suite green (306 tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clear CA2201 warning in ConnectionTeardownTests The test deliberately constructs a NullReferenceException to prove the teardown classifier does not mask a latent bug as an expected connection close. CA2201 flags constructing reserved exception types, so scope a justified suppression to that single line instead of weakening the test. Solution now builds with 0 warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump version to 4.0.0 (breaking: Kestrel engine + canonical plugin API) This is a major version bump because the migration from Titanium/Unobtanium to the Kestrel-based proxy engine introduces breaking changes: - Plugin API surface moved from Titanium types (SessionEventArgs/Request/ Response/HttpHeader) to the canonical model (IProxySession/IHttpRequest/ IHttpResponse/HeaderCollection). Third-party plugins must recompile. - Behavioral changes around HTTP/2 handling (h2-only/gRPC blind-tunnel). Copies schemas/v3.1.0 -> schemas/v4.0.0 verbatim (41 files) and updates <Version> in all four shipping projects (incl. DevProxy.Proxy.Kestrel), schema URLs in config JSON + MockResponsePlugin, installer/script/Dockerfile version strings, and refreshes packages.lock.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
waldekmastykarz
added a commit
that referenced
this pull request
Sep 14, 2026
* Add throwaway Kestrel proxy spike (Phase 0B de-risking) Validates selective decrypt, non-destructive ClientHello ALPN peek, h2-only/gRPC blind-tunnel, keep-alive isolation, and unbuffered streaming. Not part of DevProxy.sln. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add canonical HTTP proxy model to DevProxy.Abstractions (Phase 1) Introduce engine-agnostic Dev Proxy model that will replace the leaked Unobtanium/Titanium types in the plugin SDK: - IHttpHeader/IHeaderCollection/IHttpMessage/IHttpRequest/IHttpResponse and IProxySession (logical SessionId, Respond() mocking primitive), with concrete HttpHeader + HeaderCollection. - Bodies exposed as ReadOnlyMemory<byte> (decompressed-body contract). - Body strategy: BodyMode, BodyCapabilities, BodyModeResolver (pure, reconciles streaming vs full-body plugin access). - ForwardingInvariants documenting the forwarding contract + shared HopByHopHeaders set. - New DevProxy.Abstractions.Tests (xUnit), 33 tests. Purely additive: existing Titanium-based code is untouched and the full solution builds clean. Adapter wiring follows in Phase 2/3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Titanium adapter mapping engine types onto canonical model (Phase 2) Introduces DevProxy.Proxy.Titanium, the sole project referencing Unobtanium.Web.Proxy, which projects Titanium request/response/header/session types onto the engine-agnostic canonical model in DevProxy.Abstractions. - TitaniumHeaderCollection, TitaniumHttpMessageAdapter (shared DRY base over RequestResponseBase), TitaniumRequestAdapter, TitaniumResponseAdapter, TitaniumProxySession. - Body access guarded by HasBody (Titanium throws BodyNotFoundException otherwise); mutation routed through the session's SetRequestBody/ SetResponseBody since Titanium exposes no public body setter. - DevProxy.Proxy.Titanium.Tests: 38 adapter-fidelity unit tests. Purely additive and ship-safe: the existing engine and plugins are untouched. Full solution builds 0/0; 71 tests green. Cross-engine parity tests are deferred to Phase 4 (require the Kestrel engine to exist). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wire canonical IProxySession through ProxyEvents; migrate first plugin (Phase 3 slice 1) Introduces the dual-field strangler that lets plugins migrate off Titanium types one at a time while staying green: - ProxyHttpEventArgsBase exposes IProxySession ProxySession alongside the existing Titanium SessionEventArgs Session. HasRequestUrlMatch now reads the canonical request. - ProxyEngine builds a TitaniumProxySession (keyed by the existing per-session GetHashCode key) and threads it through the request/response arg construction sites; DevProxy host references DevProxy.Proxy.Titanium. - CachingGuidancePlugin fully migrated to e.ProxySession.Request and is now Titanium-free. Verified live: the cache-window warning fires through the real proxy pipeline reading IHttpRequest.RequestUri via the adapter. Full solution builds 0/0; 71 tests green. The other 21 Titanium-touching plugins are untouched and keep compiling against Session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 Wave 1: add IHttpRequest overloads to ProxyUtils Add canonical IHttpRequest overloads alongside the existing Titanium Request overloads for IsGraphRequest/IsSdkRequest/IsGraphBetaRequest/BuildGraphResponseHeaders. Extract a shared BuildGraphResponseHeadersCore to keep both overloads DRY. Additive only; Titanium overloads remain until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 guidance wave: migrate 4 guidance plugins off Titanium Migrate ODSPSearchGuidancePlugin, GraphSelectGuidancePlugin, GraphSdkGuidancePlugin, and GraphClientRequestIdGuidancePlugin to the canonical model: read request/response via e.ProxySession, helpers take IHttpRequest, header checks via Contains. Each plugin is now Titanium-free in source (LoggingContext still takes e.Session until the final wave). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 mocking wave: migrate 5 mocking plugins off Titanium Migrate AuthPlugin, CrudApiPlugin, GraphMockResponsePlugin, MockResponsePlugin, and OpenAIMockResponsePlugin to the canonical model: request/response via e.ProxySession, GenericResponse -> IProxySession.Respond, canonical HttpHeader, header methods via the canonical IHeaderCollection. Helper signatures and the CrudApi action-handler delegate now use IProxySession/IHttpRequest/ProxyRequestArgs. LoggingContext still takes e.Session until the final wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 inspection-lite: migrate HttpUtils + OpenAI usage/telemetry streaming off Titanium Change HttpUtils.IsStreamingResponse/GetBodyFromStreamingResponse to take the canonical IHttpResponse. Migrate OpenAIUsageDebuggingPlugin fully to e.ProxySession (status comparisons cast HttpStatusCode to int). Point OpenAITelemetryPlugin's streaming call site at e.ProxySession.Response. DevToolsPlugin and the OpenAITelemetry RequestLog.Context.Session readers remain for the final LoggingContext wave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 3 throttling group: migrate ThrottlerInfo + 5 throttling plugins off Titanium Flip ThrottlerInfo.ShouldThrottle delegate from Func<Request,...> to Func<IHttpRequest,...> (PluginEvents) and migrate the coordinated consumers: GraphUtils.BuildThrottleKey, RateLimitingPlugin, GraphRandomErrorPlugin, GenericRandomErrorPlugin, LanguageModelRateLimitingPlugin, RetryAfterPlugin. RetryAfter's GenericResponse becomes IProxySession.Respond with canonical HttpHeader. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Migrate plugins off Titanium onto canonical HTTP model (Phase 3 final wave) Complete the strangler migration: flip LoggingContext and the proxy event args to the engine-agnostic IProxySession model, migrate all remaining plugins off Titanium.Web.Proxy types, and drop the Unobtanium.Web.Proxy package reference from DevProxy.Abstractions and DevProxy.Plugins. - LoggingContext/RequestLog now carry IProxySession (was SessionEventArgs); ProxyHttpEventArgsBase no longer exposes the Titanium Session. - All Behavior/Generation/Guidance/Inspection/Manipulation/Mocking/Reporting plugins read e.ProxySession.Request/Response and the canonical header API. - DevToolsPlugin re-keys _responseBody/CDP requestId on the phase-stable ProxySession.SessionId instead of the Titanium Request hash code. - Add Url setter and response HttpVersion to the canonical model (+ Titanium adapters) to preserve Rewrite and HAR behavior. - Remove dead Titanium FuncExtensions helper. DevProxy.Abstractions and DevProxy.Plugins are now Titanium-free; only the host engine and the Titanium adapter retain the dependency. Build 0/0, 33+38 tests green, live smoke test verified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Kestrel proxy engine vertical slice (Phase 4) Introduce DevProxy.Proxy.Kestrel: a forward-proxy engine built on ASP.NET Core Kestrel that runs the Dev Proxy plugin pipeline against the canonical HTTP model, as the eventual replacement for the Titanium engine. Slice-1 scope (dev-toggle via DEV_PROXY_ENGINE=kestrel, for golden-output comparison against Titanium; not a shipped fallback): - KestrelProxyEngine hosts a raw TCP endpoint via UseConnectionHandler. - ProxyConnectionHandler: HTTP/1.1 parse, CONNECT->SslStream MITM, forward, write-back; catches client disconnect/cancel as normal teardown. - PluginPipeline mirrors ProxyEngine semantics (watched/mocked/notwatched, Before/After request+response, per-exchange SessionData) and emits request logs inside the requestId scope so the console formatter groups plugin and engine lines identically to Titanium. - UpstreamForwarder/ResponseWriter apply ForwardingInvariants (hop-by-hop stripping, decompressed bodies, Content-Length/Encoding fix-up). - In-memory CertificateAuthority + DuplexPipeStream ported from the POC. Engine selection wired via DEV_PROXY_ENGINE in DI; ProxyConsoleFormatter suppresses the engine-name prefix for KestrelProxyEngine (as for ProxyEngine). Deferred hardening (tracked): keep-alive, blind-tunnel for non-watched and h2-only/gRPC (ClientHello peek), WebSocket relay, chunked/trailers/100-continue, body-mode streaming, process filter, persistent CA, host-watch DRY consolidation. Full solution builds 0/0; 71 existing tests green; live-verified plain HTTP passthrough, HTTPS MITM, GenericRandomError/RetryAfter mocking with correct console output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Kestrel proxy engine test project (Phase 4) Add DevProxy.Proxy.Kestrel.Tests with 28 unit tests covering the HTTP/1.1 parser, host watch list (including order-dependent exclusion semantics), response writer, and the mutable HTTP model / CanonicalProxySession. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add selective decrypt + ALPN blind-tunnel to Kestrel engine (Phase 4) At CONNECT time the engine now non-destructively peeks the TLS ClientHello (SNI + ALPN) and decides whether to intercept: - non-watched hosts are blind-tunnelled (encrypted bytes relayed, never MITM'd) - watched hosts whose client is h2-only (gRPC, no http/1.1 fallback) are blind-tunnelled so they stop failing - otherwise the engine terminates TLS advertising http/1.1 only, so h2-capable clients downgrade and are intercepted as HTTP/1.1 Adds a minimal tolerant TlsClientHello parser, wires HostWatchList into the connection handler, and relays blind tunnels bidirectionally with a linked cancellation token so both copy tasks finish before the streams are disposed. Covered by 9 new TlsClientHello unit tests (37 Kestrel tests total). Verified live: watched host downgrades h2->http/1.1 and is intercepted; example.com (non-watched) validates its real cert without -k, proving it is never MITM'd. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add keep-alive + per-request state isolation to Kestrel engine (Phase 4 slice 3) Replace the static one-shot HTTP/1.1 reader with a stateful per-connection Http1ConnectionReader that owns the inter-request buffer, so bytes read past one request's body (a pipelined next request) are retained instead of dropped. This makes client-side keep-alive correct. - Http1ConnectionReader: stateful ReadHeadAsync/ReadBodyAsync owning _pending. - Http1RequestReader: reduced to stateless parse helpers (ParseHead, GetContentLength, IndexOfDoubleCrlf). - ProxyConnectionHandler.ServeConnectionAsync: keep-alive loop over both plain HTTP and decrypted CONNECT streams; each exchange builds a fresh CanonicalProxySession (new GUID + Interlocked requestId) so plugin state never leaks across requests on a reused connection. - ShouldKeepAlive (RFC 9112 9.3): persistent by default on HTTP/1.1, opt-in on HTTP/1.0; forces Connection: close for Connection: close, Transfer-Encoding (chunked, not yet reframable) and Expect: 100-continue (unsupported), so an unframable body can never corrupt the next request. - ResponseWriter.WriteAsync: keepAlive flag drives the Connection header. Tests: +15 Kestrel (stateful reader incl. pipelining/surplus-retention, ShouldKeepAlive matrix, keep-alive/close response-writer). 123 green total. Verified live: two URLs in one curl invocation reuse one TCP connection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add WebSocket transparent relay to Kestrel engine (Phase 4 slice 4) Watched-host WebSocket upgrades (ws:// plain + wss:// after MITM) now relay through the Kestrel engine for parity with Titanium, which relays WS opaquely (no plugin sees frames). HttpClient can't carry a 101 upgrade, so WebSocketRelay opens its own raw socket to the origin (TLS w/ http/1.1 ALPN for wss), replays the handshake in origin-form (preserving Upgrade/Connection/Sec-WebSocket-*, stripping Proxy-*), writes the origin's 101 back to the client verbatim, then splices frames raw via a shared StreamRelay (also now used by BlindTunnelAsync, DRY). Critical ordering: the response pipeline (which flushes the buffered req-log) runs in the onHandshakeResponse callback, right after the 101 and before the splice blocks, else a watched WS never logs until the socket closes. A handshakeObserved fallback flush covers origin-connect failure. Tests: ParseResponseHead unit tests, StreamRelay bidirectional copy, and an end-to-end RelayAsync test against a loopback fake origin (asserts origin-form handshake replay, header preservation, 101 verbatim write-back, frame splicing, and onHandshakeResponse invocation). 128 tests green (+5 Kestrel). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add chunked request bodies + 100-continue to Kestrel engine (Phase 4 slice 5) Closes the biggest remaining HTTP/1.1 correctness gap on the request read path. Previously a chunked request body was silently dropped (Content-Length defaulted to 0, so an empty body was forwarded) and Expect: 100-continue hung (the proxy never sent the interim 100). Response-side chunked already worked via HttpClient. - Http1ConnectionReader.ReadChunkedBodyAsync decodes chunked framing (size lines, chunk extensions, CRLF delimiters) into one buffer, consumes and drops trailers, and retains surplus for the next pipelined request. Shared _pending fill primitives (PullAsync/ReadExactlyAsync/ReadLineAsync/TakeFromPending), reused by ReadBodyAsync. - Http1RequestReader.DetectBodyFraming (None/ContentLength/Chunked/Conflicting) and HasExpectContinue. A request declaring both Content-Length and chunked Transfer-Encoding is Conflicting and refused with 400 (RFC 9112 6.3.3 smuggling guard). - ResponseWriter.WriteContinueAsync sends 100 Continue; ExchangeAsync answers Expect: 100-continue itself before reading the body (the buffering model can't relay the origin's interim 100). Expect is stripped on forward. - ShouldKeepAlive no longer forces close for chunked/Expect; malformed chunked framing replies 400 and closes. 146 tests green (+18). Verified live through the MITM proxy: a chunked upload reached httpbin with the body intact and reframed to Content-Length (no Transfer-Encoding), the client got 100 Continue, two chunked POSTs reused one TCP connection, and a raw Content-Length+Transfer-Encoding request got 400. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Kestrel engine: incremental SSE streaming with capped tee (Slice 6) Forward text/event-stream responses to the client piece-by-piece (re-framed as HTTP/1.1 chunked) instead of buffering the whole body. Buffering withheld every event until the stream ended and hung the engine on unbounded streams. - UpstreamForwarder.ForwardAsync returns an OriginResponse: buffered for normal responses, or a live body stream for text/event-stream (message stays open). - StreamingResponseWriter pumps origin->client one chunk per read (flushed so events arrive live), accumulating a capped copy (4 MiB, reusing BodyModeResolver.DefaultInMemoryLimitBytes); beyond the cap accumulation is dropped but relaying continues, so the engine never hangs or OOMs. - PluginPipeline.RunStreamingResponseAsync runs the body pump between BeforeResponse and AfterResponse, exposing the accumulated body to read-only AfterResponse inspectors (OpenAITelemetry/OpenAIUsageDebugging) for parity. - Keep-alive preserved (chunked is self-delimiting). Tests: +12 (StreamingResponseWriter chunk framing/accumulation/cap, forwarder streaming detection). 158 green. Live-verified: local finite SSE streamed ~0.5s per event (not buffered), Transfer-Encoding: chunked, Connection: keep-alive, connection reused across two streamed responses, non-watched passthrough intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 4 Slice 7: process filtering at CONNECT (--watch-pids/--watch-process-names) Implements the last functional parity gap with the Titanium engine: restricting MITM to specific processes via --watch-pids / --watch-process-names. Matches Titanium semantics exactly — filtering applies ONLY at the CONNECT decision (IsProxiedProcess in ProxyEngine.cs), so plain HTTP stays unfiltered (documented parity quirk). - ProcessFilter resolves the client's PID from its source port and matches against WatchPids then WatchProcessNames (ordinal/case-sensitive HashSet to mirror Titanium's default Contains comparer). No filter configured -> watch all (IsEmpty short-circuit, zero overhead). - ConnectionProcessResolver shells out per-platform: lsof -i on Unix, netstat -ano on Windows. LsofParser/NetstatParser are pure, fully unit-tested partial classes (GeneratedRegex). Chosen over Windows P/Invoke (GetExtendedTcpTable) for testability and boring-tech symmetry. NOTE: the Windows netstat path is parser-tested but not live-verifiable on macOS. - Wired into ProxyConnectionHandler.HandleConnectAsync (after the h2-only check): unresolved/non-matching process -> blind-tunnel (never MITM'd). KestrelProxyEngine constructs the filter from configuration.WatchPids/WatchProcessNames. - Updated the CONNECT-flow doc diagram. Tests: +21 Kestrel (12 ProcessFilter + 4 Lsof + 5 Netstat) -> 179 green (33 Abstractions + 38 Titanium + 108 Kestrel). Live-verified on macOS: matching process name -> MITM intercept + req-log; non-matching -> blind tunnel (real cert, no -k, zero req-log lines). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 5 Slice 5a: persistent certificate authority (disk root + leaf cache) The Kestrel engine previously minted a brand-new in-memory root CA on every start, so the OS would distrust it after each restart. This makes the root (and per-host leaves) persistent on disk, mirroring the Titanium engine's CertificateDiskCache layout exactly: root at <configDir>/rootCert.pfx (overridable via DEV_PROXY_CERT_PATH), leaves at <configDir>/crts/<host>.pfx, both PKCS#12 with an empty password. configDir matches Titanium's resolution (ApplicationData/dev-proxy on mac/Linux, the executable dir on Windows). EUREKA: because the format + path match, the persistent CA loads a root that the Titanium engine ALREADY created and the OS ALREADY trusts — existing users get interception with zero new trust code (verified live: curl WITHOUT -k against a watched HTTPS host returned 200 + full MITM req-log). - LoadOrCreateRoot: load + validate (isCA, has private key, not expired) the on-disk root; regenerate + re-save (and purge the now-stale leaf cache) if absent/invalid/ expired. Per decision #5 there is no cross-version compatibility contract. - Leaves are minted on demand, cached in memory + on disk, and reused across restarts. Leaf validity is clamped to the issuer's window so reusing a near-expiry root can't produce a leaf that outlives it (CertificateRequest.Create rejects that). Filename- unsafe hosts (IPv6 ':', wildcard '*', '/') are sanitized for the disk cache key. - Disk writes are best-effort: an unwritable location degrades to in-memory-only for the run instead of failing interception. - KestrelProxyEngine now builds the CA via CertificateAuthority.CreateDefault(logger). OS-trust install (mac keychain / Windows root store) + the first-run prompt remain Slice 5b (host-side, reusing the existing trust helpers). This slice only makes the root persistent. Tests: +12 Kestrel (CertificateAuthorityTests: first-run create+persist, reuse across instances, invalid/expired/non-CA regeneration, stale-leaf purge, leaf mint+sign+ persist+reuse, leaf-not-outliving-root clamp, IP-literal SAN, filename sanitization) -> 191 green (33 + 38 + 120). Live-verified on macOS: existing root reused (fingerprint unchanged, "Loaded persisted root certificate"), leaf cached + reused across restarts, MITM works without -k. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OS-trust install + first-run wiring for the Kestrel engine (Slice 5b) Wire root-certificate OS trust for the Kestrel engine via a new IRootCertificateTrust abstraction (DevProxy.Abstractions/Proxy), implemented host-side by RootCertificateTrust (DevProxy/Proxy). The engine mints/persists its root (Slice 5a) then calls EnsureTrusted; the host performs the platform install: - macOS: trust in the login keychain via MacCertificateHelper, gated by the first-run flow (InstallCert + NoFirstRun + HasRunFlag + Y/n prompt, auto-"y" under CI/redirected stdin) — mirrors Titanium's FirstRunSetup. - Windows: install the public-only root into CurrentUser/Root, idempotent by identity (CA5380 suppressed — installing the Dev Proxy root is the proxy's explicit, user-consented purpose). NOT live-verifiable on macOS. - Linux: warn to trust manually. The trust decision is factored into a pure RootTrustPolicy.Decide table in Abstractions so it's exhaustively unit-testable without platform/console I/O; the host impl is just the thin I/O around it (option X — new interface, leave Titanium's FirstRunSetup untouched for low blast radius). CertCommand (ensure/remove) already works for Kestrel mode unchanged thanks to Slice 5a's on-disk path parity. The Kestrel engine ctor takes an optional IRootCertificateTrust (null = no trust, e.g. tests); the DEV_PROXY_ENGINE=kestrel DI toggle registers and injects RootCertificateTrust. 207 tests green (49 Abstractions incl. +16 RootTrustPolicy, 38 Titanium, 120 Kestrel). Live-verified on macOS: with an already-trusted persisted root and --no-first-run, a watched HTTPS request returned 200 WITHOUT -k plus a full MITM req-log — trust wiring doesn't regress the existing-root happy path. Fresh-machine keychain trust is covered by the unit-tested decision table (not live-exercised to avoid keychain GUI pollution); Windows store install is parser/logic-only on macOS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Graceful-teardown audit: DRY connection-close classifier (Slice 5c) Extract a single ConnectionTeardown.IsExpected(exception) classifier and use it at every client/origin read/copy/write boundary, replacing the per-site hardcoded `ex is IOException or OperationCanceledException or ...` lists. An exception is "expected" (treat as EOF/close, don't log as error) when it's a ConnectionResetException, ConnectionAbortedException, OperationCanceledException, IOException, or SocketException — the normal result of a peer disconnecting, resetting, or the connection being cancelled mid-exchange. Boundaries updated to the shared classifier: - ProxyConnectionHandler.OnConnectedAsync outer catch (keep-alive read, body read, response write, streaming pump, blind tunnel) — now also covers SocketException/ConnectionAbortedException. - StreamRelay bidirectional-copy teardown. - WriteErrorAsync best-effort write to a possibly-gone client. Fixes the one genuine noise gap: the WebSocket relay catch logged a client disconnect mid-handshake / verbatim-101-write as an ERROR. It now logs at Debug when IsExpected and reserves Error for genuine faults. 216 tests green (49 Abstractions, 38 Titanium, 129 Kestrel; +9 ConnectionTeardown covering OCE/TaskCanceled/IOException/SocketException/ConnectionReset/ ConnectionAborted true and InvalidOperation/NullReference/unrelated false). Live-verified on macOS: aborting curl mid-stream (httpbin /drip, --max-time 1) produced ZERO error/exception log lines and the engine kept serving (a following request returned 200). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden CONNECT authority parsing: IPv6, ports, malformed targets (Slice 5d) Replace the naive "split on the last colon" SplitHostPort with a pure, explicit ConnectAuthorityParser that correctly handles: - IPv6 literals: [::1] and [2001:db8::1]:8443 — the bare host (no brackets) is used for socket connect / cert minting, and a UrlHost property re-adds brackets for composing the absolute URL (https://[::1]:8443/path). - Explicit ports, including validation (reject port 0 / >65535 / non-numeric / empty). - Malformed targets: empty host, unterminated bracket, junk after ], bare (unbracketed) IPv6, invalid host names — all rejected. The authority is now parsed + validated BEFORE the proxy writes "200 Connection Established", so a malformed CONNECT is refused with a 400 instead of establishing an unusable tunnel. The decrypted-tunnel URL is composed from UrlHost so IPv6 targets produce valid absolute URLs. 237 tests green (49 Abstractions, 38 Titanium, 150 Kestrel; +21 parser tests covering reg-name/IPv4/punycode/IPv6 with and without ports plus 15 malformed cases). Live-verified on macOS: normal + explicit-:443 watched MITM return 200 (no -k); malformed CONNECT targets (port 0, bare IPv6 ::1:443, unterminated [::1) each return 400 Bad Request while a valid target returns 200 Connection Established; zero error-log noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Slice 5e: --port 0 actual-port logging + DRY host extraction Log the OS-assigned port (not the configured 0) for the Kestrel engine by reading the bound ListenOptions endpoint after StartAsync, so users running with --port 0 can connect. Extract the duplicated urlsToWatch host-derivation logic into a shared WatchedHostExtractor.ToHostRegex in DevProxy.Abstractions, used by both the Titanium ProxyEngine.LoadHostNamesFromUrls and the Kestrel HostWatchList. One implementation, one test suite (11 new tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 6 slice: engine-agnostic system-proxy on/off (no Titanium) Re-implement turning the OS system-proxy on and off without Titanium, behind a new ISystemProxyManager abstraction (Abstractions) so the Kestrel engine can use it without referencing the host (mirrors IRootCertificateTrust): - SystemProxyManager (host): Windows via WinINET (registry ProxyServer/ProxyEnable + InternetSetOption settings-changed/refresh broadcast), macOS via the existing toggle-proxy.sh, Linux warns. DllImport (not LibraryImport) to avoid enabling AllowUnsafeBlocks project-wide. - SystemProxyAddress (Abstractions, pure): normalizes wildcard/empty bind addresses to loopback for the client-facing proxy address; unit-tested. - KestrelProxyEngine: enable on start (using the actually-bound port, matters for --port 0), disable on stop. Previously the Kestrel engine set no system proxy at all — a parity gap. - StopCommand --force crash-cleanup now uses the shared manager, so a force-killed daemon restores the system proxy on Windows too (was macOS-only before). 259 tests green (+11 SystemProxyAddress). Windows WinINET path is parser/logic-tested but needs verification on a Windows host (cannot run on macOS). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Phase 6 slice: replace Titanium RunTime with System.OperatingSystem Swap the Titanium RunTime OS-check helper for the .NET built-in equivalents (RunTime.IsWindows → OperatingSystem.IsWindows(), RunTime.IsMac → OperatingSystem.IsMacOS()) in ProxyEngine and CertCommand, and drop the now-unused `using Titanium.Web.Proxy.Helpers;` import from both. Strictly equivalent behavior; reduces Titanium API surface ahead of the cut-over. Left untouched (intentionally): - CertificateDiskCache.RunTime.IsUwpOnWindows — no built-in equivalent, and the file (a Titanium ICertificateCache impl) is deleted at cut-over anyway. - The Titanium.Web.Proxy.* log filters — still suppress real Titanium engine logs while it remains the default engine; their removal belongs in the cut-over commit. 259 tests green; Titanium engine boot-smoked clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add hermetic Kestrel parity-test gate (controllable rows) Build the DevProxy.Parity.Tests project that boots the real KestrelProxyEngine on a random localhost port against a deterministic FakeOrigin and asserts the observable proxy contract for every controllable parity-matrix row: - plain HTTP: GET/POST (small/large), large download, status relay, 204 empty-body framing, request-header passthrough - chunked request body reframing + SSE chunked response streaming - keep-alive reuse + Connection: close - plugin Respond() mocking short-circuit (origin never contacted) - Content-Length + Transfer-Encoding smuggling -> 400 guard 15 hermetic rows green. External/OS rows (HTTPS real-cert MITM, h2/gRPC, WebSocket, system-proxy, process filter) remain the scripted live pass and are already slice-verified; see files/parity-results.md for the full matrix. The harness boots only the Kestrel engine (the Titanium engine's process-global static ProxyServer + IServer make in-process co-hosting flaky, and it is deleted at cut-over); it asserts the reference behavior Titanium already exhibits. Total: 274 tests green (71 Abstractions + 38 Titanium + 150 Kestrel + 15 Parity). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Hard cut-over to the Kestrel proxy engine; remove Titanium Make the ASP.NET Core Kestrel engine the sole proxy engine and remove the Unobtanium/Titanium.Web.Proxy adapter, dependency, and dev toggle in one change. - Remove the DEV_PROXY_ENGINE toggle; AddProxyEngine always wires Kestrel, the shared CertificateAuthority, IRootCertificateTrust, and ISystemProxyManager. - Introduce a single shared CertificateAuthority DI singleton (Kestrel AddKestrelCertificateAuthority) with the root X509Certificate2 derived from it; the engine, CertCommand, ProxyController, and EntraMockResponsePlugin all resolve the same root cert. The engine no longer self-creates/disposes its CA. - Add Trust/Untrust to IRootCertificateTrust and implement them host-side (mac keychain / Windows root store / Linux warn); CertCommand uses them. - Delete ProxyEngine, CertificateDiskCache, the DevProxy.Proxy.Titanium and DevProxy.Proxy.Titanium.Tests projects, the Unobtanium PackageReference, the Titanium.Web.Proxy.* log filters, and the THIRD PARTY NOTICES Unobtanium entry. - Update docs (CONTRIBUTING, copilot-instructions) and stale comments to Kestrel. Build 0/0; 236 tests green (71 Abstractions + 150 Kestrel + 15 Parity). Live-verified on macOS: watched HTTPS MITM without -k + req-log, cert-download API, and `cert ensure` trust path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix detached-daemon state registration and bound graceful-shutdown drain Two cut-over regressions surfaced by the macOS system-proxy live test: 1. Detached mode (`--detach`) was broken. The deleted Titanium ProxyEngine wrote the initial ProxyInstanceState (gated on IsInternalDaemon); the Kestrel engine never did, so the daemon orphaned, the parent's readiness poll timed out, and the system proxy was left on. The host's UpdateStateWithApiUrlAsync only updated an existing state file, so it silently no-opped. - KestrelProxyEngine now publishes its actually-bound port back to the shared IProxyConfiguration after binding (resolves --port 0 to the OS-assigned port). - DevProxyCommand.WriteDaemonStateAsync writes the full ProxyInstanceState (Pid, ApiUrl, LogFile, ConfigFile, resolved Port, AsSystemProxy) once the engine has published its port, satisfying the parent's `Port > 0` readiness contract used by `devproxy stop` and `devproxy status`. 2. Graceful shutdown stalled after real traffic. The engine awaited app.StopAsync(CancellationToken.None), which waited unbounded for lingering client keep-alive connections to drain — fast when idle, but >10s after traffic, making `devproxy stop` falsely report "did not stop in time". The drain is now bounded (5s) so remaining connections are force-closed. Adds EnginePortPublishingTests regression coverage (verified failing without the port write-back). 237 tests green. Live-verified on macOS: `--detach` registers the daemon, `devproxy status`/`stop` find it, and the system proxy is restored on both graceful API stop and SIGTERM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename Parity.Tests to Integration.Tests post-cut-over With Titanium removed there is no second engine to compare against, so the "parity" framing no longer applies. The suite is now a black-box integration test that boots the Kestrel engine over a real socket against a FakeOrigin. Renamed the project, csproj, namespaces, *ParityTests classes/files, and the solution entry (project GUID unchanged); reworded "parity" comments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add per-plugin integration tests: harness infra + behavior/manipulation/mocking plugins Adds a CapturingLoggerFactory (collects engine-emitted RequestLogs), an inline PluginConfig section builder, and extends KestrelProxyHarness to accept a logger factory and expose its urls-to-watch set. Covers 9 hermetic, HTTP-observable plugins end-to-end through the Kestrel engine: GenericRandomError, Latency, RateLimiting, RetryAfter, Rewrite, MockResponse, Auth (reject + allow), and GraphRandomError. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CrudApi + MockRequest integration tests with InitializeAsync infra Adds PluginTestHost (a minimal DI ServiceProvider supplying HttpClient/ILogger<>/ IProxyConfiguration so plugins whose InitializeAsync builds file loaders can run in-process), TestDefaults (shared HttpClient), a FakeOrigin request recorder + /json endpoint, and a settable ConfigFile on TestProxyConfiguration. Mirrors the host's runtime-only package assets in the test csproj so plugin runtime dependencies (OpenIdConnect, JWT, OpenApi, OpenTelemetry, EF Sqlite, Newtonsoft, Azure.Identity) resolve. Covers MockRequestPlugin (outbound fire) and CrudApiPlugin (in-memory REST served through the engine, origin never contacted). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add guidance plugin integration tests (7 hermetic plugins) Adds TestExchange, which constructs the engine's real canonical session/request/ response types so guidance plugins gated on a fixed upstream host (graph.microsoft.com) can be driven at their hooks with production-fidelity inputs the loopback origin can't impersonate. Covers GraphBetaSupport, GraphClientRequestId, GraphSdk, ODataPaging, ODSPSearch, GraphConnector, and CachingGuidance, asserting the advisory log each emits. GraphSelectGuidancePlugin is documented as non-hermetic (needs the Graph metadata DB) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add reporting plugin integration tests (UrlDiscovery, ExecutionSummary, MinimalPermissions) Drives reporter plugins through AfterRecordingStopAsync exactly as the host's recording controller does: feeds engine-shaped RequestLogs (TestExchange.AsRequestLog) into RecordingArgs with a seeded Reports GlobalData slot, then asserts the structured report stored under the plugin name. MinimalPermissions runs against a temp OpenAPI spec fixture to verify offline permission analysis. GraphMinimalPermissions(+Guidance), MinimalCsom, and ApiCenter reporters are documented as live-backend (Bucket 3) and deferred. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add generation plugin integration tests (HAR, mock, .http, OpenAPI, TypeSpec) Drives each generator through AfterRecordingStopAsync with an InterceptedResponse log, asserting the stored report and the generated artifact. Generators write to the process CWD, so tests redirect it to a temp dir; assembly test parallelization is disabled to keep that global swap safe. A DisabledLanguageModelClient keeps the OpenAPI/TypeSpec generators hermetic (LM is cosmetic-only; they fall back to deterministic generation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add process-level smoke test for config-driven plugin loading Spawns the real DevProxy host with a devproxyrc.json + mocks file, waits for the Kestrel engine to report its bound port (--port 0), routes an HTTP request through it as an explicit proxy, and asserts MockResponsePlugin short-circuits with the configured body. Proves end-to-end that config-driven plugin discovery + the engine wire survive the Titanium -> Kestrel cut-over. Uses a plain-HTTP watched URL so no CA trust is needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unwired body-mode subsystem; fix misleading body docs The BodyModeResolver + BodyMode/BodyCapabilities/BodyContext/BodyDirection types (319 LOC, 12 tests) were a spike-era design that the Kestrel engine never wired: Resolve() is never called in production, no plugin declares BodyCapabilities, and IHttpMessage exposes no capability property. The engine uses a fixed 4 MiB in-memory inspection cap instead. Worse, the IHttpMessage doc claimed body buffering was 'driven by plugin BodyCapabilities', misleading plugin authors toward an API that does not exist. - Delete BodyModeResolver.cs, BodyHandling.cs, and BodyModeResolverTests.cs. - Inline the one used constant as ProxyConnectionHandler.InMemoryInspectionCapBytes. - Correct the IHttpMessage body-visibility doc to describe actual behavior. - Drop stale 'every engine adapter' language (abandoned multi-engine design) from ForwardingInvariants and HeaderCollection docs. The design is preserved in git and can be restored from this parent commit if/when capability-based body handling is actually needed. A code comment + plan.md follow-up track the genuine, separate unbounded-buffered-body memory risk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename IProxyLogger.cs to LoggingContext.cs to match contents The file's 'I' prefix implied an interface, but it contains no interface — only the MessageType enum and the LoggingContext / StdioLoggingContext classes. Rename to match its primary type and theme. (StdioEvents.cs was left as-is: it correctly groups the stdio event-arg family, mirroring the existing ProxyEvents.cs convention.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add WebSocketMockResponsePlugin (reactive WS frame mocking) Net-new capability unlocked by the Kestrel engine: mock WebSocket frames end-to-end. The engine owns transport (101 handshake + framing via the framework's WebSocket.CreateFromStream); the plugin owns behavior (on-connect scripted sends + match incoming client messages by equals/contains/regex/json and reply with mapped messages, plus optional close). A watched WS upgrade is dispatched to the plugin responder (instead of the origin relay) when the plugin calls IProxySession.HandleWebSocket(handler), leaving the session in the Watched phase. The plugin must NOT set ResponseState.HasBeenSet, which would short-circuit to the mocked-HTTP ResponseWriter path before the WS dispatch branch runs. DRY scheme-normalization footgun fix: the engine reports an intercepted WS upgrade with an http(s) scheme, but authors naturally write wss://host/* in urlsToWatch and mock URLs. A shared ProxyUtils.NormalizeWebSocketScheme normalizes wss->https / ws->http and is applied at both the watch-regex build site (PluginServiceExtensions.ConvertToRegex) and the plugin's mock-URL match, so either form works and a wss:// watch entry is no longer silently relayed to origin. Tests: responder loopback unit tests, matcher unit tests, 2 e2e ClientWebSocket-over-wss-MITM integration tests, and NormalizeWebSocketScheme unit tests. Full suite green (284). Live-verified through the host binary with urlsToWatch ["wss://ws.example.test/*"]: OPEN -> welcome (onConnect) -> pong (reactive ping->pong); proxy logged "Mocking WebSocket" + "WebSocket mock established". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align WebSocket mock matching with HTTP/stdio mock vocabulary The WebSocketMockResponsePlugin invented a `matchType: equals|contains|regex| json` enum, diverging from every other Dev Proxy mock plugin, which matches with the `bodyFragment` (case-insensitive contains) + `bodyRegex` field pair. Replace the enum with the conventional vocabulary so authors familiar with the HTTP/stdio mocks reach for the same fields: - `bodyFragment` — OrdinalIgnoreCase substring - `bodyRegex` — regular expression (1s timeout) - `bodyJson` — structural JSON equality (JsonNode.DeepEquals); accepts a JSON value directly in the mocks file, matching how the HTTP response body accepts object/array/string Precedence when more than one is set: bodyJson > bodyRegex > bodyFragment (mirrors stdio's "bodyRegex wins over bodyFragment"). Null match or no criteria remains a catch-all. The genuinely WS-specific structure (url-keyed mock, onConnect pushes, reactive rules, multi-message responses) is unchanged — only the match expression is aligned. Updates the mocks-file schema and matcher unit tests accordingly. Full suite green (283). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore interactive console (hotkeys + --record) dropped in Kestrel cut-over The hard cut-over to the Kestrel engine deleted ProxyEngine.cs, which owned the interactive key loop (r/s/c/w) and the --record auto-start. KestrelProxyEngine never re-implemented them, so the documented keyboard shortcuts (technical-reference/keyboard-shortcuts.md) stopped working and `devproxy --record` no longer started recording at launch. Restore both as a host-side InteractiveConsoleService (BackgroundService), keeping the Kestrel engine headless: the engine depends only on the proxy abstractions and must not reach into host concerns like IProxyStateController. The service waits for ApplicationStarted so the banner prints after the engine's startup logs, honors --record, prints hotkeys (text mode) or API instructions (JSON mode), and dispatches keys via a testable ConsoleHotkeyHandler over an ISystemConsole seam. Adds DevProxy.Tests (the host had no unit test project) with 10 tests covering key dispatch and both banners. Full suite green (293 tests). Live-verified: --record starts recording at launch, JSON banner prints, proxy serves 200 through the new service. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Windows verification checklist for the Kestrel engine The Windows-specific runtime paths (WinINET system proxy, current-user root-store cert trust, netstat-based process filter) are unit-tested but were never live-verified on a real Windows host after the Kestrel cut-over. Add a self-contained, clone-and-run checklist covering build/tests, cert trust, system proxy on/off, detached daemon lifecycle, process filter, the restored interactive console, and a core proxy + plugin smoke pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Preserve Content-Length on HEAD responses A HEAD response carries the Content-Length a GET would return but no body (RFC 9110 §9.3.2). The engine was recomputing Content-Length from the (empty) HEAD body, so clients saw Content-Length: 0 instead of the real resource size. ResponseWriter and UpstreamForwarder are now method-aware: HEAD preserves the origin's Content-Length, never takes the streaming path, and never writes a body. * Fix API URL banner showing port 0 with --api-port 0 Resolve the OS-assigned API port from the bound server address before printing the JSON-mode banner so the curl commands reference the actual port instead of the literal 0. * Return 504 on upstream timeout instead of silently dropping the connection The forward catch excluded all OperationCanceledException so client cancellation would bubble up as a silent teardown. But an HttpClient request timeout also throws TaskCanceledException (an OperationCanceledException), so a stalled origin was misclassified as a client disconnect: the client got a dropped/reset connection after the default 100s timeout instead of a gateway error. Add a pure UpstreamFailure.Classify helper (mirroring ConnectionTeardown) that tells the two apart by the connection token: OperationCanceledException with the token NOT cancelled is an upstream timeout -> 504 Gateway Timeout; with the token cancelled it is a genuine client teardown -> silent; anything else (TLS handshake failure on an invalid upstream cert, DNS, connection refused) -> 502 Bad Gateway. Wire it into the single forward catch in ProxyConnectionHandler and add the 504 reason phrase. Closes the "upstream timeout" and "invalid upstream cert" parity rows in code with 7 unit tests on the classifier. Full suite green (306 tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clear CA2201 warning in ConnectionTeardownTests The test deliberately constructs a NullReferenceException to prove the teardown classifier does not mask a latent bug as an expected connection close. CA2201 flags constructing reserved exception types, so scope a justified suppression to that single line instead of weakening the test. Solution now builds with 0 warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump version to 4.0.0 (breaking: Kestrel engine + canonical plugin API) This is a major version bump because the migration from Titanium/Unobtanium to the Kestrel-based proxy engine introduces breaking changes: - Plugin API surface moved from Titanium types (SessionEventArgs/Request/ Response/HttpHeader) to the canonical model (IProxySession/IHttpRequest/ IHttpResponse/HeaderCollection). Third-party plugins must recompile. - Behavioral changes around HTTP/2 handling (h2-only/gRPC blind-tunnel). Copies schemas/v3.1.0 -> schemas/v4.0.0 verbatim (41 files) and updates <Version> in all four shipping projects (incl. DevProxy.Proxy.Kestrel), schema URLs in config JSON + MockResponsePlugin, installer/script/Dockerfile version strings, and refreshes packages.lock.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
garrytrinder
pushed a commit
that referenced
this pull request
Sep 17, 2026
* Migrates Dev Proxy to Kestrel (#1735)
* Add throwaway Kestrel proxy spike (Phase 0B de-risking)
Validates selective decrypt, non-destructive ClientHello ALPN peek,
h2-only/gRPC blind-tunnel, keep-alive isolation, and unbuffered
streaming. Not part of DevProxy.sln.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add canonical HTTP proxy model to DevProxy.Abstractions (Phase 1)
Introduce engine-agnostic Dev Proxy model that will replace the leaked
Unobtanium/Titanium types in the plugin SDK:
- IHttpHeader/IHeaderCollection/IHttpMessage/IHttpRequest/IHttpResponse
and IProxySession (logical SessionId, Respond() mocking primitive),
with concrete HttpHeader + HeaderCollection.
- Bodies exposed as ReadOnlyMemory<byte> (decompressed-body contract).
- Body strategy: BodyMode, BodyCapabilities, BodyModeResolver (pure,
reconciles streaming vs full-body plugin access).
- ForwardingInvariants documenting the forwarding contract + shared
HopByHopHeaders set.
- New DevProxy.Abstractions.Tests (xUnit), 33 tests.
Purely additive: existing Titanium-based code is untouched and the full
solution builds clean. Adapter wiring follows in Phase 2/3.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Titanium adapter mapping engine types onto canonical model (Phase 2)
Introduces DevProxy.Proxy.Titanium, the sole project referencing
Unobtanium.Web.Proxy, which projects Titanium request/response/header/session
types onto the engine-agnostic canonical model in DevProxy.Abstractions.
- TitaniumHeaderCollection, TitaniumHttpMessageAdapter (shared DRY base over
RequestResponseBase), TitaniumRequestAdapter, TitaniumResponseAdapter,
TitaniumProxySession.
- Body access guarded by HasBody (Titanium throws BodyNotFoundException
otherwise); mutation routed through the session's SetRequestBody/
SetResponseBody since Titanium exposes no public body setter.
- DevProxy.Proxy.Titanium.Tests: 38 adapter-fidelity unit tests.
Purely additive and ship-safe: the existing engine and plugins are untouched.
Full solution builds 0/0; 71 tests green. Cross-engine parity tests are
deferred to Phase 4 (require the Kestrel engine to exist).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Wire canonical IProxySession through ProxyEvents; migrate first plugin (Phase 3 slice 1)
Introduces the dual-field strangler that lets plugins migrate off Titanium
types one at a time while staying green:
- ProxyHttpEventArgsBase exposes IProxySession ProxySession alongside the
existing Titanium SessionEventArgs Session. HasRequestUrlMatch now reads the
canonical request.
- ProxyEngine builds a TitaniumProxySession (keyed by the existing per-session
GetHashCode key) and threads it through the request/response arg construction
sites; DevProxy host references DevProxy.Proxy.Titanium.
- CachingGuidancePlugin fully migrated to e.ProxySession.Request and is now
Titanium-free. Verified live: the cache-window warning fires through the real
proxy pipeline reading IHttpRequest.RequestUri via the adapter.
Full solution builds 0/0; 71 tests green. The other 21 Titanium-touching plugins
are untouched and keep compiling against Session.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 Wave 1: add IHttpRequest overloads to ProxyUtils
Add canonical IHttpRequest overloads alongside the existing Titanium Request
overloads for IsGraphRequest/IsSdkRequest/IsGraphBetaRequest/BuildGraphResponseHeaders.
Extract a shared BuildGraphResponseHeadersCore to keep both overloads DRY.
Additive only; Titanium overloads remain until the final wave.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 guidance wave: migrate 4 guidance plugins off Titanium
Migrate ODSPSearchGuidancePlugin, GraphSelectGuidancePlugin, GraphSdkGuidancePlugin,
and GraphClientRequestIdGuidancePlugin to the canonical model: read request/response
via e.ProxySession, helpers take IHttpRequest, header checks via Contains. Each plugin
is now Titanium-free in source (LoggingContext still takes e.Session until the final wave).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 mocking wave: migrate 5 mocking plugins off Titanium
Migrate AuthPlugin, CrudApiPlugin, GraphMockResponsePlugin, MockResponsePlugin,
and OpenAIMockResponsePlugin to the canonical model: request/response via
e.ProxySession, GenericResponse -> IProxySession.Respond, canonical HttpHeader,
header methods via the canonical IHeaderCollection. Helper signatures and the
CrudApi action-handler delegate now use IProxySession/IHttpRequest/ProxyRequestArgs.
LoggingContext still takes e.Session until the final wave.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 inspection-lite: migrate HttpUtils + OpenAI usage/telemetry streaming off Titanium
Change HttpUtils.IsStreamingResponse/GetBodyFromStreamingResponse to take the
canonical IHttpResponse. Migrate OpenAIUsageDebuggingPlugin fully to e.ProxySession
(status comparisons cast HttpStatusCode to int). Point OpenAITelemetryPlugin's
streaming call site at e.ProxySession.Response. DevToolsPlugin and the OpenAITelemetry
RequestLog.Context.Session readers remain for the final LoggingContext wave.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 throttling group: migrate ThrottlerInfo + 5 throttling plugins off Titanium
Flip ThrottlerInfo.ShouldThrottle delegate from Func<Request,...> to
Func<IHttpRequest,...> (PluginEvents) and migrate the coordinated consumers:
GraphUtils.BuildThrottleKey, RateLimitingPlugin, GraphRandomErrorPlugin,
GenericRandomErrorPlugin, LanguageModelRateLimitingPlugin, RetryAfterPlugin.
RetryAfter's GenericResponse becomes IProxySession.Respond with canonical HttpHeader.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Migrate plugins off Titanium onto canonical HTTP model (Phase 3 final wave)
Complete the strangler migration: flip LoggingContext and the proxy event
args to the engine-agnostic IProxySession model, migrate all remaining
plugins off Titanium.Web.Proxy types, and drop the Unobtanium.Web.Proxy
package reference from DevProxy.Abstractions and DevProxy.Plugins.
- LoggingContext/RequestLog now carry IProxySession (was SessionEventArgs);
ProxyHttpEventArgsBase no longer exposes the Titanium Session.
- All Behavior/Generation/Guidance/Inspection/Manipulation/Mocking/Reporting
plugins read e.ProxySession.Request/Response and the canonical header API.
- DevToolsPlugin re-keys _responseBody/CDP requestId on the phase-stable
ProxySession.SessionId instead of the Titanium Request hash code.
- Add Url setter and response HttpVersion to the canonical model (+ Titanium
adapters) to preserve Rewrite and HAR behavior.
- Remove dead Titanium FuncExtensions helper.
DevProxy.Abstractions and DevProxy.Plugins are now Titanium-free; only the
host engine and the Titanium adapter retain the dependency. Build 0/0,
33+38 tests green, live smoke test verified.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Kestrel proxy engine vertical slice (Phase 4)
Introduce DevProxy.Proxy.Kestrel: a forward-proxy engine built on ASP.NET
Core Kestrel that runs the Dev Proxy plugin pipeline against the canonical
HTTP model, as the eventual replacement for the Titanium engine.
Slice-1 scope (dev-toggle via DEV_PROXY_ENGINE=kestrel, for golden-output
comparison against Titanium; not a shipped fallback):
- KestrelProxyEngine hosts a raw TCP endpoint via UseConnectionHandler.
- ProxyConnectionHandler: HTTP/1.1 parse, CONNECT->SslStream MITM, forward,
write-back; catches client disconnect/cancel as normal teardown.
- PluginPipeline mirrors ProxyEngine semantics (watched/mocked/notwatched,
Before/After request+response, per-exchange SessionData) and emits request
logs inside the requestId scope so the console formatter groups plugin and
engine lines identically to Titanium.
- UpstreamForwarder/ResponseWriter apply ForwardingInvariants (hop-by-hop
stripping, decompressed bodies, Content-Length/Encoding fix-up).
- In-memory CertificateAuthority + DuplexPipeStream ported from the POC.
Engine selection wired via DEV_PROXY_ENGINE in DI; ProxyConsoleFormatter
suppresses the engine-name prefix for KestrelProxyEngine (as for ProxyEngine).
Deferred hardening (tracked): keep-alive, blind-tunnel for non-watched and
h2-only/gRPC (ClientHello peek), WebSocket relay, chunked/trailers/100-continue,
body-mode streaming, process filter, persistent CA, host-watch DRY consolidation.
Full solution builds 0/0; 71 existing tests green; live-verified plain HTTP
passthrough, HTTPS MITM, GenericRandomError/RetryAfter mocking with correct
console output.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Kestrel proxy engine test project (Phase 4)
Add DevProxy.Proxy.Kestrel.Tests with 28 unit tests covering the HTTP/1.1
parser, host watch list (including order-dependent exclusion semantics),
response writer, and the mutable HTTP model / CanonicalProxySession.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add selective decrypt + ALPN blind-tunnel to Kestrel engine (Phase 4)
At CONNECT time the engine now non-destructively peeks the TLS ClientHello
(SNI + ALPN) and decides whether to intercept:
- non-watched hosts are blind-tunnelled (encrypted bytes relayed, never MITM'd)
- watched hosts whose client is h2-only (gRPC, no http/1.1 fallback) are
blind-tunnelled so they stop failing
- otherwise the engine terminates TLS advertising http/1.1 only, so h2-capable
clients downgrade and are intercepted as HTTP/1.1
Adds a minimal tolerant TlsClientHello parser, wires HostWatchList into the
connection handler, and relays blind tunnels bidirectionally with a linked
cancellation token so both copy tasks finish before the streams are disposed.
Covered by 9 new TlsClientHello unit tests (37 Kestrel tests total). Verified
live: watched host downgrades h2->http/1.1 and is intercepted; example.com
(non-watched) validates its real cert without -k, proving it is never MITM'd.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add keep-alive + per-request state isolation to Kestrel engine (Phase 4 slice 3)
Replace the static one-shot HTTP/1.1 reader with a stateful per-connection
Http1ConnectionReader that owns the inter-request buffer, so bytes read past one
request's body (a pipelined next request) are retained instead of dropped. This
makes client-side keep-alive correct.
- Http1ConnectionReader: stateful ReadHeadAsync/ReadBodyAsync owning _pending.
- Http1RequestReader: reduced to stateless parse helpers (ParseHead,
GetContentLength, IndexOfDoubleCrlf).
- ProxyConnectionHandler.ServeConnectionAsync: keep-alive loop over both plain
HTTP and decrypted CONNECT streams; each exchange builds a fresh
CanonicalProxySession (new GUID + Interlocked requestId) so plugin state never
leaks across requests on a reused connection.
- ShouldKeepAlive (RFC 9112 9.3): persistent by default on HTTP/1.1, opt-in on
HTTP/1.0; forces Connection: close for Connection: close, Transfer-Encoding
(chunked, not yet reframable) and Expect: 100-continue (unsupported), so an
unframable body can never corrupt the next request.
- ResponseWriter.WriteAsync: keepAlive flag drives the Connection header.
Tests: +15 Kestrel (stateful reader incl. pipelining/surplus-retention,
ShouldKeepAlive matrix, keep-alive/close response-writer). 123 green total.
Verified live: two URLs in one curl invocation reuse one TCP connection.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add WebSocket transparent relay to Kestrel engine (Phase 4 slice 4)
Watched-host WebSocket upgrades (ws:// plain + wss:// after MITM) now relay
through the Kestrel engine for parity with Titanium, which relays WS opaquely
(no plugin sees frames).
HttpClient can't carry a 101 upgrade, so WebSocketRelay opens its own raw
socket to the origin (TLS w/ http/1.1 ALPN for wss), replays the handshake in
origin-form (preserving Upgrade/Connection/Sec-WebSocket-*, stripping Proxy-*),
writes the origin's 101 back to the client verbatim, then splices frames raw
via a shared StreamRelay (also now used by BlindTunnelAsync, DRY).
Critical ordering: the response pipeline (which flushes the buffered req-log)
runs in the onHandshakeResponse callback, right after the 101 and before the
splice blocks, else a watched WS never logs until the socket closes. A
handshakeObserved fallback flush covers origin-connect failure.
Tests: ParseResponseHead unit tests, StreamRelay bidirectional copy, and an
end-to-end RelayAsync test against a loopback fake origin (asserts origin-form
handshake replay, header preservation, 101 verbatim write-back, frame splicing,
and onHandshakeResponse invocation). 128 tests green (+5 Kestrel).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add chunked request bodies + 100-continue to Kestrel engine (Phase 4 slice 5)
Closes the biggest remaining HTTP/1.1 correctness gap on the request read path.
Previously a chunked request body was silently dropped (Content-Length defaulted
to 0, so an empty body was forwarded) and Expect: 100-continue hung (the proxy
never sent the interim 100). Response-side chunked already worked via HttpClient.
- Http1ConnectionReader.ReadChunkedBodyAsync decodes chunked framing (size lines,
chunk extensions, CRLF delimiters) into one buffer, consumes and drops trailers,
and retains surplus for the next pipelined request. Shared _pending fill
primitives (PullAsync/ReadExactlyAsync/ReadLineAsync/TakeFromPending), reused by
ReadBodyAsync.
- Http1RequestReader.DetectBodyFraming (None/ContentLength/Chunked/Conflicting) and
HasExpectContinue. A request declaring both Content-Length and chunked
Transfer-Encoding is Conflicting and refused with 400 (RFC 9112 6.3.3 smuggling
guard).
- ResponseWriter.WriteContinueAsync sends 100 Continue; ExchangeAsync answers
Expect: 100-continue itself before reading the body (the buffering model can't
relay the origin's interim 100). Expect is stripped on forward.
- ShouldKeepAlive no longer forces close for chunked/Expect; malformed chunked
framing replies 400 and closes.
146 tests green (+18). Verified live through the MITM proxy: a chunked upload
reached httpbin with the body intact and reframed to Content-Length (no
Transfer-Encoding), the client got 100 Continue, two chunked POSTs reused one TCP
connection, and a raw Content-Length+Transfer-Encoding request got 400.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Kestrel engine: incremental SSE streaming with capped tee (Slice 6)
Forward text/event-stream responses to the client piece-by-piece (re-framed as
HTTP/1.1 chunked) instead of buffering the whole body. Buffering withheld every
event until the stream ended and hung the engine on unbounded streams.
- UpstreamForwarder.ForwardAsync returns an OriginResponse: buffered for normal
responses, or a live body stream for text/event-stream (message stays open).
- StreamingResponseWriter pumps origin->client one chunk per read (flushed so
events arrive live), accumulating a capped copy (4 MiB, reusing
BodyModeResolver.DefaultInMemoryLimitBytes); beyond the cap accumulation is
dropped but relaying continues, so the engine never hangs or OOMs.
- PluginPipeline.RunStreamingResponseAsync runs the body pump between
BeforeResponse and AfterResponse, exposing the accumulated body to read-only
AfterResponse inspectors (OpenAITelemetry/OpenAIUsageDebugging) for parity.
- Keep-alive preserved (chunked is self-delimiting).
Tests: +12 (StreamingResponseWriter chunk framing/accumulation/cap, forwarder
streaming detection). 158 green. Live-verified: local finite SSE streamed ~0.5s
per event (not buffered), Transfer-Encoding: chunked, Connection: keep-alive,
connection reused across two streamed responses, non-watched passthrough intact.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 4 Slice 7: process filtering at CONNECT (--watch-pids/--watch-process-names)
Implements the last functional parity gap with the Titanium engine: restricting
MITM to specific processes via --watch-pids / --watch-process-names. Matches
Titanium semantics exactly — filtering applies ONLY at the CONNECT decision
(IsProxiedProcess in ProxyEngine.cs), so plain HTTP stays unfiltered (documented
parity quirk).
- ProcessFilter resolves the client's PID from its source port and matches against
WatchPids then WatchProcessNames (ordinal/case-sensitive HashSet to mirror
Titanium's default Contains comparer). No filter configured -> watch all
(IsEmpty short-circuit, zero overhead).
- ConnectionProcessResolver shells out per-platform: lsof -i on Unix, netstat -ano
on Windows. LsofParser/NetstatParser are pure, fully unit-tested partial classes
(GeneratedRegex). Chosen over Windows P/Invoke (GetExtendedTcpTable) for
testability and boring-tech symmetry. NOTE: the Windows netstat path is
parser-tested but not live-verifiable on macOS.
- Wired into ProxyConnectionHandler.HandleConnectAsync (after the h2-only check):
unresolved/non-matching process -> blind-tunnel (never MITM'd). KestrelProxyEngine
constructs the filter from configuration.WatchPids/WatchProcessNames.
- Updated the CONNECT-flow doc diagram.
Tests: +21 Kestrel (12 ProcessFilter + 4 Lsof + 5 Netstat) -> 179 green
(33 Abstractions + 38 Titanium + 108 Kestrel). Live-verified on macOS: matching
process name -> MITM intercept + req-log; non-matching -> blind tunnel (real cert,
no -k, zero req-log lines).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 5 Slice 5a: persistent certificate authority (disk root + leaf cache)
The Kestrel engine previously minted a brand-new in-memory root CA on every start,
so the OS would distrust it after each restart. This makes the root (and per-host
leaves) persistent on disk, mirroring the Titanium engine's CertificateDiskCache
layout exactly: root at <configDir>/rootCert.pfx (overridable via DEV_PROXY_CERT_PATH),
leaves at <configDir>/crts/<host>.pfx, both PKCS#12 with an empty password. configDir
matches Titanium's resolution (ApplicationData/dev-proxy on mac/Linux, the executable
dir on Windows).
EUREKA: because the format + path match, the persistent CA loads a root that the
Titanium engine ALREADY created and the OS ALREADY trusts — existing users get
interception with zero new trust code (verified live: curl WITHOUT -k against a
watched HTTPS host returned 200 + full MITM req-log).
- LoadOrCreateRoot: load + validate (isCA, has private key, not expired) the on-disk
root; regenerate + re-save (and purge the now-stale leaf cache) if absent/invalid/
expired. Per decision #5 there is no cross-version compatibility contract.
- Leaves are minted on demand, cached in memory + on disk, and reused across restarts.
Leaf validity is clamped to the issuer's window so reusing a near-expiry root can't
produce a leaf that outlives it (CertificateRequest.Create rejects that). Filename-
unsafe hosts (IPv6 ':', wildcard '*', '/') are sanitized for the disk cache key.
- Disk writes are best-effort: an unwritable location degrades to in-memory-only for
the run instead of failing interception.
- KestrelProxyEngine now builds the CA via CertificateAuthority.CreateDefault(logger).
OS-trust install (mac keychain / Windows root store) + the first-run prompt remain
Slice 5b (host-side, reusing the existing trust helpers). This slice only makes the
root persistent.
Tests: +12 Kestrel (CertificateAuthorityTests: first-run create+persist, reuse across
instances, invalid/expired/non-CA regeneration, stale-leaf purge, leaf mint+sign+
persist+reuse, leaf-not-outliving-root clamp, IP-literal SAN, filename sanitization)
-> 191 green (33 + 38 + 120). Live-verified on macOS: existing root reused (fingerprint
unchanged, "Loaded persisted root certificate"), leaf cached + reused across restarts,
MITM works without -k.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add OS-trust install + first-run wiring for the Kestrel engine (Slice 5b)
Wire root-certificate OS trust for the Kestrel engine via a new
IRootCertificateTrust abstraction (DevProxy.Abstractions/Proxy), implemented
host-side by RootCertificateTrust (DevProxy/Proxy). The engine mints/persists
its root (Slice 5a) then calls EnsureTrusted; the host performs the platform
install:
- macOS: trust in the login keychain via MacCertificateHelper, gated by the
first-run flow (InstallCert + NoFirstRun + HasRunFlag + Y/n prompt, auto-"y"
under CI/redirected stdin) — mirrors Titanium's FirstRunSetup.
- Windows: install the public-only root into CurrentUser/Root, idempotent by
identity (CA5380 suppressed — installing the Dev Proxy root is the proxy's
explicit, user-consented purpose). NOT live-verifiable on macOS.
- Linux: warn to trust manually.
The trust decision is factored into a pure RootTrustPolicy.Decide table in
Abstractions so it's exhaustively unit-testable without platform/console I/O;
the host impl is just the thin I/O around it (option X — new interface, leave
Titanium's FirstRunSetup untouched for low blast radius).
CertCommand (ensure/remove) already works for Kestrel mode unchanged thanks to
Slice 5a's on-disk path parity. The Kestrel engine ctor takes an optional
IRootCertificateTrust (null = no trust, e.g. tests); the DEV_PROXY_ENGINE=kestrel
DI toggle registers and injects RootCertificateTrust.
207 tests green (49 Abstractions incl. +16 RootTrustPolicy, 38 Titanium,
120 Kestrel). Live-verified on macOS: with an already-trusted persisted root and
--no-first-run, a watched HTTPS request returned 200 WITHOUT -k plus a full MITM
req-log — trust wiring doesn't regress the existing-root happy path. Fresh-machine
keychain trust is covered by the unit-tested decision table (not live-exercised to
avoid keychain GUI pollution); Windows store install is parser/logic-only on macOS.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Graceful-teardown audit: DRY connection-close classifier (Slice 5c)
Extract a single ConnectionTeardown.IsExpected(exception) classifier and use it
at every client/origin read/copy/write boundary, replacing the per-site
hardcoded `ex is IOException or OperationCanceledException or ...` lists. An
exception is "expected" (treat as EOF/close, don't log as error) when it's a
ConnectionResetException, ConnectionAbortedException, OperationCanceledException,
IOException, or SocketException — the normal result of a peer disconnecting,
resetting, or the connection being cancelled mid-exchange.
Boundaries updated to the shared classifier:
- ProxyConnectionHandler.OnConnectedAsync outer catch (keep-alive read, body
read, response write, streaming pump, blind tunnel) — now also covers
SocketException/ConnectionAbortedException.
- StreamRelay bidirectional-copy teardown.
- WriteErrorAsync best-effort write to a possibly-gone client.
Fixes the one genuine noise gap: the WebSocket relay catch logged a client
disconnect mid-handshake / verbatim-101-write as an ERROR. It now logs at Debug
when IsExpected and reserves Error for genuine faults.
216 tests green (49 Abstractions, 38 Titanium, 129 Kestrel; +9 ConnectionTeardown
covering OCE/TaskCanceled/IOException/SocketException/ConnectionReset/
ConnectionAborted true and InvalidOperation/NullReference/unrelated false).
Live-verified on macOS: aborting curl mid-stream (httpbin /drip, --max-time 1)
produced ZERO error/exception log lines and the engine kept serving (a following
request returned 200).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Harden CONNECT authority parsing: IPv6, ports, malformed targets (Slice 5d)
Replace the naive "split on the last colon" SplitHostPort with a pure, explicit
ConnectAuthorityParser that correctly handles:
- IPv6 literals: [::1] and [2001:db8::1]:8443 — the bare host (no brackets) is
used for socket connect / cert minting, and a UrlHost property re-adds brackets
for composing the absolute URL (https://[::1]:8443/path).
- Explicit ports, including validation (reject port 0 / >65535 / non-numeric /
empty).
- Malformed targets: empty host, unterminated bracket, junk after ], bare
(unbracketed) IPv6, invalid host names — all rejected.
The authority is now parsed + validated BEFORE the proxy writes
"200 Connection Established", so a malformed CONNECT is refused with a 400
instead of establishing an unusable tunnel. The decrypted-tunnel URL is composed
from UrlHost so IPv6 targets produce valid absolute URLs.
237 tests green (49 Abstractions, 38 Titanium, 150 Kestrel; +21 parser tests
covering reg-name/IPv4/punycode/IPv6 with and without ports plus 15 malformed
cases). Live-verified on macOS: normal + explicit-:443 watched MITM return 200
(no -k); malformed CONNECT targets (port 0, bare IPv6 ::1:443, unterminated
[::1) each return 400 Bad Request while a valid target returns 200 Connection
Established; zero error-log noise.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Slice 5e: --port 0 actual-port logging + DRY host extraction
Log the OS-assigned port (not the configured 0) for the Kestrel engine by
reading the bound ListenOptions endpoint after StartAsync, so users running
with --port 0 can connect.
Extract the duplicated urlsToWatch host-derivation logic into a shared
WatchedHostExtractor.ToHostRegex in DevProxy.Abstractions, used by both the
Titanium ProxyEngine.LoadHostNamesFromUrls and the Kestrel HostWatchList.
One implementation, one test suite (11 new tests).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 6 slice: engine-agnostic system-proxy on/off (no Titanium)
Re-implement turning the OS system-proxy on and off without Titanium, behind a
new ISystemProxyManager abstraction (Abstractions) so the Kestrel engine can use
it without referencing the host (mirrors IRootCertificateTrust):
- SystemProxyManager (host): Windows via WinINET (registry ProxyServer/ProxyEnable
+ InternetSetOption settings-changed/refresh broadcast), macOS via the existing
toggle-proxy.sh, Linux warns. DllImport (not LibraryImport) to avoid enabling
AllowUnsafeBlocks project-wide.
- SystemProxyAddress (Abstractions, pure): normalizes wildcard/empty bind addresses
to loopback for the client-facing proxy address; unit-tested.
- KestrelProxyEngine: enable on start (using the actually-bound port, matters for
--port 0), disable on stop. Previously the Kestrel engine set no system proxy at
all — a parity gap.
- StopCommand --force crash-cleanup now uses the shared manager, so a force-killed
daemon restores the system proxy on Windows too (was macOS-only before).
259 tests green (+11 SystemProxyAddress). Windows WinINET path is parser/logic-tested
but needs verification on a Windows host (cannot run on macOS).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 6 slice: replace Titanium RunTime with System.OperatingSystem
Swap the Titanium RunTime OS-check helper for the .NET built-in equivalents
(RunTime.IsWindows → OperatingSystem.IsWindows(), RunTime.IsMac →
OperatingSystem.IsMacOS()) in ProxyEngine and CertCommand, and drop the now-unused
`using Titanium.Web.Proxy.Helpers;` import from both. Strictly equivalent behavior;
reduces Titanium API surface ahead of the cut-over.
Left untouched (intentionally):
- CertificateDiskCache.RunTime.IsUwpOnWindows — no built-in equivalent, and the file
(a Titanium ICertificateCache impl) is deleted at cut-over anyway.
- The Titanium.Web.Proxy.* log filters — still suppress real Titanium engine logs while
it remains the default engine; their removal belongs in the cut-over commit.
259 tests green; Titanium engine boot-smoked clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add hermetic Kestrel parity-test gate (controllable rows)
Build the DevProxy.Parity.Tests project that boots the real KestrelProxyEngine
on a random localhost port against a deterministic FakeOrigin and asserts the
observable proxy contract for every controllable parity-matrix row:
- plain HTTP: GET/POST (small/large), large download, status relay, 204
empty-body framing, request-header passthrough
- chunked request body reframing + SSE chunked response streaming
- keep-alive reuse + Connection: close
- plugin Respond() mocking short-circuit (origin never contacted)
- Content-Length + Transfer-Encoding smuggling -> 400 guard
15 hermetic rows green. External/OS rows (HTTPS real-cert MITM, h2/gRPC,
WebSocket, system-proxy, process filter) remain the scripted live pass and are
already slice-verified; see files/parity-results.md for the full matrix.
The harness boots only the Kestrel engine (the Titanium engine's process-global
static ProxyServer + IServer make in-process co-hosting flaky, and it is deleted
at cut-over); it asserts the reference behavior Titanium already exhibits.
Total: 274 tests green (71 Abstractions + 38 Titanium + 150 Kestrel + 15 Parity).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Hard cut-over to the Kestrel proxy engine; remove Titanium
Make the ASP.NET Core Kestrel engine the sole proxy engine and remove the
Unobtanium/Titanium.Web.Proxy adapter, dependency, and dev toggle in one change.
- Remove the DEV_PROXY_ENGINE toggle; AddProxyEngine always wires Kestrel,
the shared CertificateAuthority, IRootCertificateTrust, and ISystemProxyManager.
- Introduce a single shared CertificateAuthority DI singleton (Kestrel
AddKestrelCertificateAuthority) with the root X509Certificate2 derived from it;
the engine, CertCommand, ProxyController, and EntraMockResponsePlugin all resolve
the same root cert. The engine no longer self-creates/disposes its CA.
- Add Trust/Untrust to IRootCertificateTrust and implement them host-side
(mac keychain / Windows root store / Linux warn); CertCommand uses them.
- Delete ProxyEngine, CertificateDiskCache, the DevProxy.Proxy.Titanium and
DevProxy.Proxy.Titanium.Tests projects, the Unobtanium PackageReference, the
Titanium.Web.Proxy.* log filters, and the THIRD PARTY NOTICES Unobtanium entry.
- Update docs (CONTRIBUTING, copilot-instructions) and stale comments to Kestrel.
Build 0/0; 236 tests green (71 Abstractions + 150 Kestrel + 15 Parity).
Live-verified on macOS: watched HTTPS MITM without -k + req-log, cert-download
API, and `cert ensure` trust path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix detached-daemon state registration and bound graceful-shutdown drain
Two cut-over regressions surfaced by the macOS system-proxy live test:
1. Detached mode (`--detach`) was broken. The deleted Titanium ProxyEngine wrote
the initial ProxyInstanceState (gated on IsInternalDaemon); the Kestrel engine
never did, so the daemon orphaned, the parent's readiness poll timed out, and
the system proxy was left on. The host's UpdateStateWithApiUrlAsync only updated
an existing state file, so it silently no-opped.
- KestrelProxyEngine now publishes its actually-bound port back to the shared
IProxyConfiguration after binding (resolves --port 0 to the OS-assigned port).
- DevProxyCommand.WriteDaemonStateAsync writes the full ProxyInstanceState
(Pid, ApiUrl, LogFile, ConfigFile, resolved Port, AsSystemProxy) once the
engine has published its port, satisfying the parent's `Port > 0` readiness
contract used by `devproxy stop` and `devproxy status`.
2. Graceful shutdown stalled after real traffic. The engine awaited
app.StopAsync(CancellationToken.None), which waited unbounded for lingering
client keep-alive connections to drain — fast when idle, but >10s after
traffic, making `devproxy stop` falsely report "did not stop in time". The
drain is now bounded (5s) so remaining connections are force-closed.
Adds EnginePortPublishingTests regression coverage (verified failing without the
port write-back). 237 tests green. Live-verified on macOS: `--detach` registers
the daemon, `devproxy status`/`stop` find it, and the system proxy is restored on
both graceful API stop and SIGTERM.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename Parity.Tests to Integration.Tests post-cut-over
With Titanium removed there is no second engine to compare against, so the
"parity" framing no longer applies. The suite is now a black-box integration
test that boots the Kestrel engine over a real socket against a FakeOrigin.
Renamed the project, csproj, namespaces, *ParityTests classes/files, and the
solution entry (project GUID unchanged); reworded "parity" comments.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add per-plugin integration tests: harness infra + behavior/manipulation/mocking plugins
Adds a CapturingLoggerFactory (collects engine-emitted RequestLogs), an inline
PluginConfig section builder, and extends KestrelProxyHarness to accept a logger
factory and expose its urls-to-watch set. Covers 9 hermetic, HTTP-observable
plugins end-to-end through the Kestrel engine: GenericRandomError, Latency,
RateLimiting, RetryAfter, Rewrite, MockResponse, Auth (reject + allow), and
GraphRandomError.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add CrudApi + MockRequest integration tests with InitializeAsync infra
Adds PluginTestHost (a minimal DI ServiceProvider supplying HttpClient/ILogger<>/
IProxyConfiguration so plugins whose InitializeAsync builds file loaders can run
in-process), TestDefaults (shared HttpClient), a FakeOrigin request recorder +
/json endpoint, and a settable ConfigFile on TestProxyConfiguration. Mirrors the
host's runtime-only package assets in the test csproj so plugin runtime
dependencies (OpenIdConnect, JWT, OpenApi, OpenTelemetry, EF Sqlite, Newtonsoft,
Azure.Identity) resolve. Covers MockRequestPlugin (outbound fire) and CrudApiPlugin
(in-memory REST served through the engine, origin never contacted).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add guidance plugin integration tests (7 hermetic plugins)
Adds TestExchange, which constructs the engine's real canonical session/request/
response types so guidance plugins gated on a fixed upstream host (graph.microsoft.com)
can be driven at their hooks with production-fidelity inputs the loopback origin can't
impersonate. Covers GraphBetaSupport, GraphClientRequestId, GraphSdk, ODataPaging,
ODSPSearch, GraphConnector, and CachingGuidance, asserting the advisory log each emits.
GraphSelectGuidancePlugin is documented as non-hermetic (needs the Graph metadata DB)
and deferred.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add reporting plugin integration tests (UrlDiscovery, ExecutionSummary, MinimalPermissions)
Drives reporter plugins through AfterRecordingStopAsync exactly as the host's recording
controller does: feeds engine-shaped RequestLogs (TestExchange.AsRequestLog) into
RecordingArgs with a seeded Reports GlobalData slot, then asserts the structured report
stored under the plugin name. MinimalPermissions runs against a temp OpenAPI spec fixture
to verify offline permission analysis. GraphMinimalPermissions(+Guidance), MinimalCsom,
and ApiCenter reporters are documented as live-backend (Bucket 3) and deferred.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add generation plugin integration tests (HAR, mock, .http, OpenAPI, TypeSpec)
Drives each generator through AfterRecordingStopAsync with an InterceptedResponse log,
asserting the stored report and the generated artifact. Generators write to the process
CWD, so tests redirect it to a temp dir; assembly test parallelization is disabled to
keep that global swap safe. A DisabledLanguageModelClient keeps the OpenAPI/TypeSpec
generators hermetic (LM is cosmetic-only; they fall back to deterministic generation).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add process-level smoke test for config-driven plugin loading
Spawns the real DevProxy host with a devproxyrc.json + mocks file, waits for the Kestrel
engine to report its bound port (--port 0), routes an HTTP request through it as an
explicit proxy, and asserts MockResponsePlugin short-circuits with the configured body.
Proves end-to-end that config-driven plugin discovery + the engine wire survive the
Titanium -> Kestrel cut-over. Uses a plain-HTTP watched URL so no CA trust is needed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove unwired body-mode subsystem; fix misleading body docs
The BodyModeResolver + BodyMode/BodyCapabilities/BodyContext/BodyDirection types
(319 LOC, 12 tests) were a spike-era design that the Kestrel engine never wired:
Resolve() is never called in production, no plugin declares BodyCapabilities, and
IHttpMessage exposes no capability property. The engine uses a fixed 4 MiB in-memory
inspection cap instead. Worse, the IHttpMessage doc claimed body buffering was 'driven
by plugin BodyCapabilities', misleading plugin authors toward an API that does not exist.
- Delete BodyModeResolver.cs, BodyHandling.cs, and BodyModeResolverTests.cs.
- Inline the one used constant as ProxyConnectionHandler.InMemoryInspectionCapBytes.
- Correct the IHttpMessage body-visibility doc to describe actual behavior.
- Drop stale 'every engine adapter' language (abandoned multi-engine design) from
ForwardingInvariants and HeaderCollection docs.
The design is preserved in git and can be restored from this parent commit if/when
capability-based body handling is actually needed. A code comment + plan.md follow-up
track the genuine, separate unbounded-buffered-body memory risk.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename IProxyLogger.cs to LoggingContext.cs to match contents
The file's 'I' prefix implied an interface, but it contains no interface — only the
MessageType enum and the LoggingContext / StdioLoggingContext classes. Rename to match
its primary type and theme. (StdioEvents.cs was left as-is: it correctly groups the
stdio event-arg family, mirroring the existing ProxyEvents.cs convention.)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add WebSocketMockResponsePlugin (reactive WS frame mocking)
Net-new capability unlocked by the Kestrel engine: mock WebSocket frames
end-to-end. The engine owns transport (101 handshake + framing via the
framework's WebSocket.CreateFromStream); the plugin owns behavior (on-connect
scripted sends + match incoming client messages by equals/contains/regex/json
and reply with mapped messages, plus optional close).
A watched WS upgrade is dispatched to the plugin responder (instead of the
origin relay) when the plugin calls IProxySession.HandleWebSocket(handler),
leaving the session in the Watched phase. The plugin must NOT set
ResponseState.HasBeenSet, which would short-circuit to the mocked-HTTP
ResponseWriter path before the WS dispatch branch runs.
DRY scheme-normalization footgun fix: the engine reports an intercepted WS
upgrade with an http(s) scheme, but authors naturally write wss://host/* in
urlsToWatch and mock URLs. A shared ProxyUtils.NormalizeWebSocketScheme
normalizes wss->https / ws->http and is applied at both the watch-regex build
site (PluginServiceExtensions.ConvertToRegex) and the plugin's mock-URL match,
so either form works and a wss:// watch entry is no longer silently relayed to
origin.
Tests: responder loopback unit tests, matcher unit tests, 2 e2e
ClientWebSocket-over-wss-MITM integration tests, and NormalizeWebSocketScheme
unit tests. Full suite green (284). Live-verified through the host binary with
urlsToWatch ["wss://ws.example.test/*"]: OPEN -> welcome (onConnect) -> pong
(reactive ping->pong); proxy logged "Mocking WebSocket" + "WebSocket mock
established".
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Align WebSocket mock matching with HTTP/stdio mock vocabulary
The WebSocketMockResponsePlugin invented a `matchType: equals|contains|regex|
json` enum, diverging from every other Dev Proxy mock plugin, which matches
with the `bodyFragment` (case-insensitive contains) + `bodyRegex` field pair.
Replace the enum with the conventional vocabulary so authors familiar with the
HTTP/stdio mocks reach for the same fields:
- `bodyFragment` — OrdinalIgnoreCase substring
- `bodyRegex` — regular expression (1s timeout)
- `bodyJson` — structural JSON equality (JsonNode.DeepEquals); accepts a JSON
value directly in the mocks file, matching how the HTTP response body accepts
object/array/string
Precedence when more than one is set: bodyJson > bodyRegex > bodyFragment
(mirrors stdio's "bodyRegex wins over bodyFragment"). Null match or no criteria
remains a catch-all. The genuinely WS-specific structure (url-keyed mock,
onConnect pushes, reactive rules, multi-message responses) is unchanged — only
the match expression is aligned.
Updates the mocks-file schema and matcher unit tests accordingly. Full suite
green (283).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restore interactive console (hotkeys + --record) dropped in Kestrel cut-over
The hard cut-over to the Kestrel engine deleted ProxyEngine.cs, which owned the
interactive key loop (r/s/c/w) and the --record auto-start. KestrelProxyEngine
never re-implemented them, so the documented keyboard shortcuts
(technical-reference/keyboard-shortcuts.md) stopped working and `devproxy --record`
no longer started recording at launch.
Restore both as a host-side InteractiveConsoleService (BackgroundService), keeping
the Kestrel engine headless: the engine depends only on the proxy abstractions and
must not reach into host concerns like IProxyStateController. The service waits for
ApplicationStarted so the banner prints after the engine's startup logs, honors
--record, prints hotkeys (text mode) or API instructions (JSON mode), and dispatches
keys via a testable ConsoleHotkeyHandler over an ISystemConsole seam.
Adds DevProxy.Tests (the host had no unit test project) with 10 tests covering key
dispatch and both banners. Full suite green (293 tests). Live-verified: --record
starts recording at launch, JSON banner prints, proxy serves 200 through the new
service.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Windows verification checklist for the Kestrel engine
The Windows-specific runtime paths (WinINET system proxy, current-user root-store cert
trust, netstat-based process filter) are unit-tested but were never live-verified on a
real Windows host after the Kestrel cut-over. Add a self-contained, clone-and-run
checklist covering build/tests, cert trust, system proxy on/off, detached daemon
lifecycle, process filter, the restored interactive console, and a core proxy + plugin
smoke pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Preserve Content-Length on HEAD responses
A HEAD response carries the Content-Length a GET would return but no
body (RFC 9110 §9.3.2). The engine was recomputing Content-Length from
the (empty) HEAD body, so clients saw Content-Length: 0 instead of the
real resource size. ResponseWriter and UpstreamForwarder are now
method-aware: HEAD preserves the origin's Content-Length, never takes
the streaming path, and never writes a body.
* Fix API URL banner showing port 0 with --api-port 0
Resolve the OS-assigned API port from the bound server address before printing the JSON-mode banner so the curl commands reference the actual port instead of the literal 0.
* Return 504 on upstream timeout instead of silently dropping the connection
The forward catch excluded all OperationCanceledException so client cancellation
would bubble up as a silent teardown. But an HttpClient request timeout also throws
TaskCanceledException (an OperationCanceledException), so a stalled origin was
misclassified as a client disconnect: the client got a dropped/reset connection after
the default 100s timeout instead of a gateway error.
Add a pure UpstreamFailure.Classify helper (mirroring ConnectionTeardown) that tells
the two apart by the connection token: OperationCanceledException with the token NOT
cancelled is an upstream timeout -> 504 Gateway Timeout; with the token cancelled it is
a genuine client teardown -> silent; anything else (TLS handshake failure on an invalid
upstream cert, DNS, connection refused) -> 502 Bad Gateway. Wire it into the single
forward catch in ProxyConnectionHandler and add the 504 reason phrase.
Closes the "upstream timeout" and "invalid upstream cert" parity rows in code with 7
unit tests on the classifier. Full suite green (306 tests).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clear CA2201 warning in ConnectionTeardownTests
The test deliberately constructs a NullReferenceException to prove the teardown
classifier does not mask a latent bug as an expected connection close. CA2201 flags
constructing reserved exception types, so scope a justified suppression to that single
line instead of weakening the test. Solution now builds with 0 warnings.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bump version to 4.0.0 (breaking: Kestrel engine + canonical plugin API)
This is a major version bump because the migration from Titanium/Unobtanium
to the Kestrel-based proxy engine introduces breaking changes:
- Plugin API surface moved from Titanium types (SessionEventArgs/Request/
Response/HttpHeader) to the canonical model (IProxySession/IHttpRequest/
IHttpResponse/HeaderCollection). Third-party plugins must recompile.
- Behavioral changes around HTTP/2 handling (h2-only/gRPC blind-tunnel).
Copies schemas/v3.1.0 -> schemas/v4.0.0 verbatim (41 files) and updates
<Version> in all four shipping projects (incl. DevProxy.Proxy.Kestrel),
schema URLs in config JSON + MockResponsePlugin, installer/script/Dockerfile
version strings, and refreshes packages.lock.json.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Removes unnecessary spikes
* Adds support for alpha releases
* Change log level from Information to Debug for loading persisted root certificate (#1736)
* Add WebSocket traffic support to HAR generator and per-message mocking (#1740)
* Add WebSocket message capture to HAR generator
Replace raw byte WebSocket relay with frame-aware relay using
WebSocket.CreateFromStream on both client and origin sides. Each
relayed message is captured as a WebSocketMessageRecord and stored
on the session.
The HAR generator now detects WebSocket upgrade entries and embeds
captured messages using the _resourceType and _webSocketMessages
custom HAR fields, following the Chrome DevTools / mitmproxy
convention.
Changes:
- Add WebSocketMessageRecord abstraction (direction, type, data,
timestamp)
- Add IProxySession.WebSocketMessages property
- Convert WebSocketRelay from raw byte splice to message-level
relay with PrefixedStream for leftover handshake bytes
- Add HarWebSocketMessage model and _resourceType/_webSocketMessages
fields to HarEntry
- Update HarGeneratorPlugin to populate WebSocket fields
- Update WebSocketRelayTests for frame-level relay and message
capture verification
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Add per-message WebSocket interception with origin passthrough
Convert WebSocket mocking from full-connection takeover to per-message
interception: the proxy connects to the origin and relays traffic, but
each client message is offered to the interceptor first. Matched
messages get mock responses; unmatched ones pass through to the origin.
When the origin is unreachable but an interceptor is registered, the
proxy falls back to mock-only mode — same as the old HandleWebSocket
behavior.
- Add InterceptWebSocketMessages to IProxySession with interceptor and
onConnected callbacks
- Split WebSocket relay into client-to-origin and origin-to-client tasks
with thread-safe client sends via SemaphoreSlim
- Add InterceptorClientConnection wrapper with send serialization and
message capture for HAR
- Convert WebSocketMockResponsePlugin from HandleWebSocket to
InterceptWebSocketMessages
- Add origin-unreachable fallback in ProxyConnectionHandler
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address PR review: HAR opcodes, capture scoping, docs, and tests
- Map WebSocketMessageType to RFC 6455 opcodes (1=text, 2=binary,
8=close) in HAR output instead of the framework enum values (0/1/2)
- Only capture WebSocket messages for watched requests to avoid
unbounded memory growth on long-lived/high-volume sockets
- Capture interceptor/onConnected responses in the interceptor-only
fallback so mock-only HAR output includes receive entries
- Update IProxySession.WebSocketMessages doc to reflect interception
and mock-only fallback capture
- Add HarGenerator_WritesWebSocketMessages test asserting correct
type/time/opcode/data formatting
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address PR review: forward non-101 leftover bytes, record close frames
- Forward bytes read past the response head when the origin declines the
WebSocket upgrade (non-101), so clients see the full response body
- Record interceptor-initiated closes (InterceptorClientConnection and
CapturingWebSocketConnection) as Receive/Close, matching how the relay
loops record origin->client closes
- Record the client->proxy close in the interceptor-only fallback loop as
Send/Close so WebSocketMessages/HAR output is complete
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Updates lockfile
* Fixes hot reload (#1744)
* Migrates Dev Proxy to Kestrel (#1735)
* Add throwaway Kestrel proxy spike (Phase 0B de-risking)
Validates selective decrypt, non-destructive ClientHello ALPN peek,
h2-only/gRPC blind-tunnel, keep-alive isolation, and unbuffered
streaming. Not part of DevProxy.sln.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add canonical HTTP proxy model to DevProxy.Abstractions (Phase 1)
Introduce engine-agnostic Dev Proxy model that will replace the leaked
Unobtanium/Titanium types in the plugin SDK:
- IHttpHeader/IHeaderCollection/IHttpMessage/IHttpRequest/IHttpResponse
and IProxySession (logical SessionId, Respond() mocking primitive),
with concrete HttpHeader + HeaderCollection.
- Bodies exposed as ReadOnlyMemory<byte> (decompressed-body contract).
- Body strategy: BodyMode, BodyCapabilities, BodyModeResolver (pure,
reconciles streaming vs full-body plugin access).
- ForwardingInvariants documenting the forwarding contract + shared
HopByHopHeaders set.
- New DevProxy.Abstractions.Tests (xUnit), 33 tests.
Purely additive: existing Titanium-based code is untouched and the full
solution builds clean. Adapter wiring follows in Phase 2/3.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Titanium adapter mapping engine types onto canonical model (Phase 2)
Introduces DevProxy.Proxy.Titanium, the sole project referencing
Unobtanium.Web.Proxy, which projects Titanium request/response/header/session
types onto the engine-agnostic canonical model in DevProxy.Abstractions.
- TitaniumHeaderCollection, TitaniumHttpMessageAdapter (shared DRY base over
RequestResponseBase), TitaniumRequestAdapter, TitaniumResponseAdapter,
TitaniumProxySession.
- Body access guarded by HasBody (Titanium throws BodyNotFoundException
otherwise); mutation routed through the session's SetRequestBody/
SetResponseBody since Titanium exposes no public body setter.
- DevProxy.Proxy.Titanium.Tests: 38 adapter-fidelity unit tests.
Purely additive and ship-safe: the existing engine and plugins are untouched.
Full solution builds 0/0; 71 tests green. Cross-engine parity tests are
deferred to Phase 4 (require the Kestrel engine to exist).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Wire canonical IProxySession through ProxyEvents; migrate first plugin (Phase 3 slice 1)
Introduces the dual-field strangler that lets plugins migrate off Titanium
types one at a time while staying green:
- ProxyHttpEventArgsBase exposes IProxySession ProxySession alongside the
existing Titanium SessionEventArgs Session. HasRequestUrlMatch now reads the
canonical request.
- ProxyEngine builds a TitaniumProxySession (keyed by the existing per-session
GetHashCode key) and threads it through the request/response arg construction
sites; DevProxy host references DevProxy.Proxy.Titanium.
- CachingGuidancePlugin fully migrated to e.ProxySession.Request and is now
Titanium-free. Verified live: the cache-window warning fires through the real
proxy pipeline reading IHttpRequest.RequestUri via the adapter.
Full solution builds 0/0; 71 tests green. The other 21 Titanium-touching plugins
are untouched and keep compiling against Session.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 Wave 1: add IHttpRequest overloads to ProxyUtils
Add canonical IHttpRequest overloads alongside the existing Titanium Request
overloads for IsGraphRequest/IsSdkRequest/IsGraphBetaRequest/BuildGraphResponseHeaders.
Extract a shared BuildGraphResponseHeadersCore to keep both overloads DRY.
Additive only; Titanium overloads remain until the final wave.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 guidance wave: migrate 4 guidance plugins off Titanium
Migrate ODSPSearchGuidancePlugin, GraphSelectGuidancePlugin, GraphSdkGuidancePlugin,
and GraphClientRequestIdGuidancePlugin to the canonical model: read request/response
via e.ProxySession, helpers take IHttpRequest, header checks via Contains. Each plugin
is now Titanium-free in source (LoggingContext still takes e.Session until the final wave).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 mocking wave: migrate 5 mocking plugins off Titanium
Migrate AuthPlugin, CrudApiPlugin, GraphMockResponsePlugin, MockResponsePlugin,
and OpenAIMockResponsePlugin to the canonical model: request/response via
e.ProxySession, GenericResponse -> IProxySession.Respond, canonical HttpHeader,
header methods via the canonical IHeaderCollection. Helper signatures and the
CrudApi action-handler delegate now use IProxySession/IHttpRequest/ProxyRequestArgs.
LoggingContext still takes e.Session until the final wave.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 inspection-lite: migrate HttpUtils + OpenAI usage/telemetry streaming off Titanium
Change HttpUtils.IsStreamingResponse/GetBodyFromStreamingResponse to take the
canonical IHttpResponse. Migrate OpenAIUsageDebuggingPlugin fully to e.ProxySession
(status comparisons cast HttpStatusCode to int). Point OpenAITelemetryPlugin's
streaming call site at e.ProxySession.Response. DevToolsPlugin and the OpenAITelemetry
RequestLog.Context.Session readers remain for the final LoggingContext wave.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Phase 3 throttling group: migrate ThrottlerInfo + 5 throttling plugins off Titanium
Flip ThrottlerInfo.ShouldThrottle delegate from Func<Request,...> to
Func<IHttpRequest,...> (PluginEvents) and migrate the coordinated consumers:
GraphUtils.BuildThrottleKey, RateLimitingPlugin, GraphRandomErrorPlugin,
GenericRandomErrorPlugin, LanguageModelRateLimitingPlugin, RetryAfterPlugin.
RetryAfter's GenericResponse becomes IProxySession.Respond with canonical HttpHeader.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Migrate plugins off Titanium onto canonical HTTP model (Phase 3 final wave)
Complete the strangler migration: flip LoggingContext and the proxy event
args to the engine-agnostic IProxySession model, migrate all remaining
plugins off Titanium.Web.Proxy types, and drop the Unobtanium.Web.Proxy
package reference from DevProxy.Abstractions and DevProxy.Plugins.
- LoggingContext/RequestLog now carry IProxySession (was SessionEventArgs);
ProxyHttpEventArgsBase no longer exposes the Titanium Session.
- All Behavior/Generation/Guidance/Inspection/Manipulation/Mocking/Reporting
plugins read e.ProxySession.Request/Response and the canonical header API.
- DevToolsPlugin re-keys _responseBody/CDP requestId on the phase-stable
ProxySession.SessionId instead of the Titanium Request hash code.
- Add Url setter and response HttpVersion to the canonical model (+ Titanium
adapters) to preserve Rewrite and HAR behavior.
- Remove dead Titanium FuncExtensions helper.
DevProxy.Abstractions and DevProxy.Plugins are now Titanium-free; only the
host engine and the Titanium adapter retain the dependency. Build 0/0,
33+38 tests green, live smoke test verified.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Kestrel proxy engine vertical slice (Phase 4)
Introduce DevProxy.Proxy.Kestrel: a forward-proxy engine built on ASP.NET
Core Kestrel that runs the Dev Proxy plugin pipeline against the canonical
HTTP model, as the eventual replacement for the Titanium engine.
Slice-1 scope (dev-toggle via DEV_PROXY_ENGINE=kestrel, for golden-output
comparison against Titanium; not a shipped fallback):
- KestrelProxyEngine hosts a raw TCP endpoint via UseConnectionHandler.
- ProxyConnectionHandler: HTTP/1.1 parse, CONNECT->SslStream MITM, forward,
write-back; catches client disconnect/cancel as normal teardown.
- PluginPipeline mirrors ProxyEngine semantics (watched/mocked/notwatched,
Before/After request+response, per-exchange SessionData) and emits request
logs inside the requestId scope so the console formatter groups plugin and
engine lines identically to Titanium.
- UpstreamForwarder/ResponseWriter apply ForwardingInvariants (hop-by-hop
stripping, decompressed bodies, Content-Length/Encoding fix-up).
- In-memory CertificateAuthority + DuplexPipeStream ported from the POC.
Engine selection wired via DEV_PROXY_ENGINE in DI; ProxyConsoleFormatter
suppresses the engine-name prefix for KestrelProxyEngine (as for ProxyEngine).
Deferred hardening (tracked): keep-alive, blind-tunnel for non-watched and
h2-only/gRPC (ClientHello peek), WebSocket relay, chunked/trailers/100-continue,
body-mode streaming, process filter, persistent CA, host-watch DRY consolidation.
Full solution builds 0/0; 71 existing tests green; live-verified plain HTTP
passthrough, HTTPS MITM, GenericRandomError/RetryAfter mocking with correct
console output.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Kestrel proxy engine test project (Phase 4)
Add DevProxy.Proxy.Kestrel.Tests with 28 unit tests covering the HTTP/1.1
parser, host watch list (including order-dependent exclusion semantics),
response writer, and the mutable HTTP model / CanonicalProxySession.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add selective decrypt + ALPN blind-tunnel to Kestrel engine (Phase 4)
At CONNECT time the engine now non-destructively peeks the TLS ClientHello
(SNI + ALPN) and decides whether to intercept:
- non-watched hosts are blind-tunnelled (encrypted bytes relayed, never MITM'd)
- watched hosts whose client is h2-only (gRPC, no http/1.1 fallback) are
blind-tunnelled so they stop failing
- otherwise the engine terminates TLS advertising http/1.1 only, so h2-capable
clients downgrade and are intercepted as HTTP/1.1
Adds a minimal tolerant TlsClientHello parser, wires HostWatchList into the
connection handler, and relays blind tunnels bidirectionally with a linked
cancellation token so both copy tasks finish before the streams are disposed.
Covered by 9 new TlsClientHello unit tests (37 Kestrel tests total). Verified
live: watched host downgrades h2->http/1.1 and is intercepted; example.com
(non-watched) validates its real cert without -k, proving it is never MITM'd.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add keep-alive + per-request state isolation to Kestrel engine (Phase 4 slice 3)
Replace the static one-shot HTTP/1.1 reader with a stateful per-connection
Http1ConnectionReader that owns the inter-request buffer, so bytes read past one
request's body (a pipelined next request) are retained instead of dropped. This
makes client-side keep-alive correct.
- Http1ConnectionReader: stateful ReadHeadAsync/ReadBodyAsync owning _pending.
- Http1RequestReader: reduced to stateless parse helpers (ParseHead,
GetContentLength, IndexOfDoubleCrlf).
- ProxyConnectionHandler.ServeConnectionAsync: keep-alive loop over both plain
HTTP and decrypted CONNECT streams; each exchange builds a fresh
CanonicalProxySession (new GUID + Interlocked requestId) so plugin state never
leaks across requests on a reused connection.
- ShouldKeepAlive (RFC 9112 9.3): persistent by default on HTTP/1.1, opt-in on
HTTP/1.0; forces Connection: close for Connection: close, Transfer-Encoding
(chunked, not yet reframable) and Expect: 100-continue (unsupported), so an
unframable body can never corrupt the next request.
- ResponseWriter.WriteAsync: keepAlive flag drives the Connection header.
Tests: +15 Kestrel (stateful reader incl. pipelining/surplus-retention,
ShouldKeepAlive matrix, keep-alive/close response-writer). 123 green total.
Verified live: two URLs in one curl invocation reuse one TCP connection.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add WebSocket transparent relay to Kestrel engine (Phase 4 slice 4)
Watched-host WebSocket upgrades (ws:// plain + wss:// after MITM) now relay
through the Kestrel engine for parity with Titanium, which relays WS opaquely
(no plugin sees frames).
HttpClient can't carry a 101 upgrade, so WebSocketRelay opens its own raw
socket to the origin (TLS w/ http/1.1 ALPN for wss), replays the handshake in
origin-form (preserving Upgrade/Connection/Sec-WebSocket-*, stripping Proxy-*),
writes the origin's 101 back to the client verbatim, then splices frames raw
via a shared StreamRelay (also now used by BlindTunnelAsync, DRY).
Critical ordering: the response pipeline (which flushes the buffered req-log)
runs in the onHandshakeResponse callback, right after the 101 and before the
splice blocks, else a watched WS never logs until the socket closes. A
handshakeObserved fallback flush covers origin-connect failure.
Tests: ParseResponseHead unit tests, StreamRelay bidirectional copy, and an
end-to-end RelayAsync test against a loopback fake origin (asserts origin-form
handshake replay, header preservation, 101 verbatim write-back, frame splicing,
and onHandshakeResponse invocation). 128 tests green (+5 Kestrel).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add chunked request bodies + 100-continue to Kestrel engine (Phase 4 slice 5)
Closes the biggest remaining HTTP/1.1 correctness gap on the request read path.
Previously a chunked request body was silently dropped (Content-Length defaulted
to 0, so an empty body was forwarded) and Expect: 100-continue hung (the proxy
never sent the interim 100). Response-side chunked already worked via HttpClient.
- Http1ConnectionReader.ReadChunkedBodyAsync decodes chunked framing (size lines,
chunk extensions, CRLF delimiters) into one buffer, consumes and drops trailers,
and retains surplus for the next pipelined request. Shared _pending fill
primitives (PullAsync/ReadExactlyAsync/ReadLineAsync/TakeFromPending), reused by
ReadBodyAsync.
- Http1RequestReader.DetectBodyFraming (None/ContentLength/Chunked/Conflicting) and
HasExpectContinue. A request declaring both Content-Length and chunked
Transfer-Encoding is Conflicting and refused with 400 (RFC 9112 6.3.3 smuggling
guard).
- ResponseWriter.WriteContinueAsync sends 100 Continue; ExchangeAsync answers
Expect: 100-continue itself before reading the body (the buffering model can't
relay the origin's interim 100). Expect is stripped on forward.
- ShouldKeepAlive no longer forces close for chunked/Expect; malformed chunked
framing replies 400 and closes.
146 tests green (+18). Verified live through the MITM proxy: a chunked upload
reached httpbin with the body intact and reframed to Content-Length (no
Transfer-Encoding), the client got 100 Continue, two chunked POSTs reused one TCP
connection, and a raw Content-Length+Transfer-Encoding request got 400.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Kestrel engine: incremental SSE streaming with capped tee (Slice 6)
Forward text/event-stream responses to the client piece-by-piece (re-framed as
HTTP/1.1 chunked) instead of buffering the whole body. Buffering withheld every
event until the stream ended and hung the engine on unbounded streams.
- UpstreamForwarder.ForwardAsync returns an OriginResponse: buffered for normal
responses, or a live body stream for text/event-stream (message stays open).
- StreamingResponseWriter …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.