feat: run HTTP plugins around MCP requests

_context/wiki/architecture.md

@@ -16,6 +16,8 @@ TCP/TLS listener
   -> CORS layer
   -> mcp_header_limits_layer    bounds MCP headers (431)
   -> virtual_host_id_layer      inserts VirtualHostId from path (400)
+  -> HTTP pre hook              global plugins, permitted request-header edits
+  -> mcp_header_limits_layer    rechecks edited MCP headers (431)
   -> claims_layer               verifies JWT, inserts AuthorizationClaims (401)
   -> PrincipalExtractorLayer   inserts AuthorizedPrincipal (401)
   -> user_config_store_layer    loads UserConfig (400 missing, 500 decode/error)
@@ -26,7 +28,9 @@ TCP/TLS listener
 
 Origin is checked before authentication. The optional Host allowlist is checked
 at the RMCP boundary, so earlier middleware may return first. MCP header budgets
-apply before JWT verification, configuration reads, and body parsing. See
+apply before JWT verification, user-configuration reads, and body parsing. The
+HTTP post hook runs as the response returns through the HTTP plugin layer,
+before headers are sent; streaming MCP work may still be running. See
 [Security](security.md#mcp-origin-and-host-validation).
 
 Health is registered in every build outside the MCP auth/config layers, while
@@ -89,7 +93,8 @@ into successful response hooks. See [Routing](routing.md) and
 | Principal, claims, virtual-host ID, config snapshot | HTTP request extensions | One request. |
 | Backend RMCP service | Routed operation | One request; explicitly closed after the call. |
 | Tool progress-token mapping | Request's backend client | While the tool call is in flight. |
-| Active CPEX runtime | Registry | Reloadable; in-flight hook state pins its selected runtime. |
+| Active CPEX policies | Registry | Reloadable; each HTTP request pins the full policy snapshot. |
+| Plugin request context | Request-owned `PluginRequest` | Shared CPEX state and permitted extension updates across HTTP and MCP hooks, including streaming work. |
 | RMCP session manager | RMCP service (`LocalSessionManager`) | Transport implementation detail; no session is required by the supported modern request contract. |
 
 The library no longer has `BackendTransports`, `SessionId` middleware, or a
@@ -114,8 +119,8 @@ Locks have specific scopes:
   shared state; do not assume all network I/O is globally lock-free.
 - Tool progress tracking holds a write guard while enqueuing the backend call
   so an early notification cannot race registration.
-- Tool hook state uses a mutex to serialize progress and final-response plugin
-  context updates. Prompt/resource state belongs to one request.
+- A request-scoped plugin mutex serializes HTTP/MCP hook state updates, including
+  tool progress and final responses. No registry lock is held during execution.
 
 There is no shared map of live backend transports to lock during routing.
 
@@ -138,11 +143,15 @@ post-hooks can inspect stream events, and a denied notification is dropped.
 Prompt/resource operations should not be assumed to have the same explicit
 cancellation relay.
 
-Pre-hooks select a runtime before backend I/O. Typed hook state retains both
-that runtime and whether a post-hook was enabled. Reloads affect subsequent
-requests; they cannot add a hook or change policy halfway through a call.
-Invalid reloads mark the registry failed for new calls while already pinned
-requests can finish. Details are in [Plugin Config](config.md#plugin-config-redis-key-contextforgegatewayruntimepluginconfig).
+The HTTP boundary pins the full policy snapshot in one `PluginRequest`. HTTP
+hooks use the global policy; tool hooks resolve published tool/team contexts,
+and prompt/resource hooks use the global policy. Authentication and user-config
+lookup attach verified identity; routing adds target metadata. Each operation
+retains its selected runtime and matched pre/post hooks. Reloads affect new HTTP
+requests; in-flight work keeps its policy and state, including streaming events.
+Invalid reloads fail new requests while already pinned requests can finish.
+See [HTTP hooks and shared request state](config.md#http-hooks-and-shared-request-state)
+for ordering, failure, timeout, cancellation and streaming behavior.
 
 ## Startup
 

_context/wiki/config.md

@@ -244,16 +244,16 @@ Tools select the resolved configuration using
 the tool's canonical `id`, gateway `name`, and owning `team_id`. Aliases select
 the same upstream tool context. With plugins enabled, a missing tool identity or
 policy context fails the call before backend I/O; it never falls back to global
-policy. Prompts and resources use `global`, matching the built-in
+policy. Prompts, resources and HTTP hooks use `global`, matching the built-in
 resolver, which only applies database bindings to team/tool context keys.
 All targets use the same policy resolver and execution engine. Only `enabled: false` explicitly
 bypasses the published policies.
 
 The existing watcher checks Redis every 30 seconds, matching the built-in
 manager's default cache lifetime. Publication adds its own interval. A reload
 builds all policies before swapping the active set; invalid, expired or missing
-configuration fails new requests. Each MCP operation pins its selected policy, so a reload cannot change it
-halfway through. Operations retain their selected
+configuration fails new requests. Each HTTP request pins the full policy snapshot,
+so its MCP operations cannot pick up a newer policy halfway through. Operations retain their selected
 runtime, matched pre/post hooks, CPEX local/shared state, and permitted extension
 updates, including post-only hooks and tool progress/logging events.
 
@@ -268,7 +268,7 @@ a policy or establish identity.
 
 Within each execution mode, hooks run in ascending numeric priority (1 before
 90). Equal priorities retain configuration order. This order is compiled into
-each policy snapshot and applies to all MCP pre/post hooks.
+each policy snapshot and applies to both HTTP and MCP pre/post hooks.
 
 CPEX filters each plugin's view by its declared capabilities. The gateway guards
 every extension write-back before the next plugin executes: identity remains
@@ -285,12 +285,51 @@ five seconds per plugin in both pre and post hooks. CPEX enforces this with
 `on_error` policy. This does not bound backend work or the total time for a chain
 of plugins. Like other asynchronous timeouts, handlers must yield to be cancelled.
 
-Supported gateway hooks are tool, prompt and resource pre/post hooks. Unsupported
+Supported gateway hooks are HTTP, tool, prompt and resource pre/post hooks. Unsupported
 hooks, route-based CPEX configuration, plugin directories and missing factories
 fail configuration loading. Redis policy cannot load Python modules or add new
 Rust factories. Authentication-resolution/permission hooks and the broader
 execution-pool and plugin-load-error parity work remain outside this adapter.
 
+### HTTP hooks and shared request state
+
+`http_pre_request` runs after the existing Origin/header-budget checks and before
+credential verification. `http_post_request` runs when response headers are
+available, including authentication failures. Both use the published global
+configuration. The native `HttpHook` payload provides method, original path,
+client address when available, and response status on the post hook. Headers
+are CPEX extensions, gated by `read_headers`, `write_headers`, and the published
+hook's `headers` write policy. Use `GatewayPluginFactory` with `.with_http_hooks()`
+and/or `.with_cmf_hooks()` to register handlers on the same plugin instance.
+
+Header changes are merged before the next plugin runs, preserving existing
+request/response headers even when the writer lacks `read_headers`. Method,
+path, host and scheme remain host-provided metadata. Invalid header updates
+are ignored. Existing auth headers are protected unless
+`plugins_can_override_auth_headers` is true. The
+before hook may add a previously absent credential header, which normal
+authentication must still verify. MCP header budgets are checked again after
+plugin edits. HTTP hooks do not modify request bodies, response bodies or status.
+
+One `PluginRequest` carries the policy snapshot, correlation ID, CPEX state table
+and permitted extension changes across all stages. Scoped/global managers give
+plugins different runtime IDs; request-local state follows the configured plugin
+kind and name when crossing that boundary. Verified identity comes only from the
+host after authentication and user-configuration lookup, even if that lookup fails.
+MCP routing adds target metadata to this existing state; it does not reconstruct
+transport data or identity. A request-scoped mutex serializes hook state updates;
+no registry lock is held while plugins run.
+
+As in the built-in HTTP middleware, HTTP hook errors, denials and timeouts are
+logged and the request continues. The configured CPEX timeout bounds execution;
+missing required configuration still fails the request. Dropping the request
+future cancels foreground hook work. Invocation spans use `gateway_plugin_invoke`
+with the hook name; logs omit header values.
+
+Streaming responses are not buffered. The HTTP post hook runs once before headers
+are sent, so it cannot observe MCP work that completes later in the stream.
+Those later MCP hooks retain the same request state and policy snapshot.
+
 Compile bundled factories with `--features plugins` and enable execution with
 `--runtime-plugins-enabled true`. A valid document must exist before startup;
 a missing document fails initialization. The
@@ -360,7 +399,7 @@ docker compose -f docker/docker-compose-local.yaml exec -T redis \
 
 This example uses the `quickstart-counter` policy key from the quick-start
 routes. For other tools, publish a matching `contexts[context_id]` entry;
-`global` alone applies to prompt and resource hooks.
+`global` alone applies to prompt, resource and HTTP hooks.
 
 For local testing only, build and run with demo factories, `with_tools` helpers,
 and runtime execution enabled:

_context/wiki/deployment.md

@@ -60,7 +60,7 @@ configuration-cache freshness and dependency availability.
 - A missing entry or Redis GET error currently produces HTTP `400`; undecodable
   configuration produces `500`. See [Failure Modes](failure-modes.md).
 - Enabled CPEX also needs its initial plugin document and checks for reloads
-  every 30 seconds. An invalid reload fails new plugin calls closed.
+  every 30 seconds. An invalid reload fails new HTTP requests closed.
 
 ## Builds and Images
 

_context/wiki/failure-modes.md

@@ -54,11 +54,12 @@ modern routing. A configured backend alone does not make its objects callable.
 | --- | --- |
 | MCP pre-hook denies | MCP error; no upstream call. |
 | MCP post-hook denies | MCP error; backend operation may already have completed. |
+| HTTP hook error, denial or timeout | Logged; request/response processing continues. |
 | Tool identity or resolved policy context missing | MCP error before backend I/O; no fallback to global policy. |
 | Plugin supplies an error code | That code is used; a denial without one defaults to invalid request `-32600`. |
 | Soft plugin error | Logged; execution can continue under the runtime's soft-error behavior. |
 | Missing or invalid initial plugin config | Runtime initialization fails; gateway startup does not complete. |
-| Invalid plugin reload | New plugin calls fail closed until valid configuration is loaded; already pinned requests keep their runtime. |
+| Invalid plugin reload | New HTTP requests fail with `500` until valid configuration is loaded; already pinned requests keep their policies. |
 | Plugin edits cannot be represented faithfully as the operation's MCP result | MCP error; no fallback to the original unredacted response. |
 
 See [Configuration](config.md#plugin-config-redis-key-contextforgegatewayruntimepluginconfig)

_context/wiki/routing.md

@@ -66,14 +66,12 @@ validation.
 tool/prompt arguments or the resource URI. Resource URI edits must resolve through
 the caller's published routes. Post-hooks can rewrite or reject the response.
 
-Tools select their published tool/team policy before backend I/O; prompts and
-resources select the global policy. The host supplies verified identity and
-canonical route metadata to each hook. It returns typed request state
-whose `after_*` method runs the post-hook on that same runtime, even after a reload
-or a reload failure. A request that started without post-hooks never gains one
-mid-flight. Tool state is shared under a mutex so progress notifications and the
-final response use the same correlation ID and serialize plugin-context updates.
-Prompt and resource state is owned by a single request and needs no mutex.
+The HTTP layer pins the policy snapshot before authentication. Tool hooks select
+the routed backend's published tool policy context; prompt, resource and HTTP
+hooks use the global policy. Typed operation state runs pre/post hooks on the
+same runtime, even after a reload or reload failure. One request-scoped mutex
+serializes shared plugin state across HTTP/MCP hooks and tool progress events.
+A request that started without post-hooks never gains one mid-flight.
 
 The internal CPEX crate separates these responsibilities:
 

crates/contextforge-data-plane-cpex/Cargo.toml

@@ -26,6 +26,7 @@ serde_json.workspace = true
 thiserror.workspace = true
 tokio.workspace = true
 tracing.workspace = true
+uuid.workspace = true
 
 [lints]
 workspace = true

crates/contextforge-data-plane-cpex/src/cmf.rs

@@ -1,48 +1,13 @@
 //! Common CMF envelopes and the response contract used by the hook runner.
 //! Conversion rules stay with each operation because their MCP semantics differ.
 
+use crate::hooks::Operation;
 use cpex::cpex_core::{
     cmf::{ContentPart, Message, MessagePayload, Role, constants::SCHEMA_VERSION},
     executor::PipelineResult,
-    hooks::types::cmf_hook_names,
 };
 use rmcp::{ErrorData, model::ErrorCode};
 
-#[derive(Clone, Debug, Copy)]
-pub(crate) enum Operation {
-    Tool,
-    Prompt,
-    Resource,
-}
-
-impl Operation {
-    pub(crate) const ALL: [Self; 3] = [Self::Tool, Self::Prompt, Self::Resource];
-
-    pub(crate) fn hooks(self) -> [&'static str; 2] {
-        match self {
-            Self::Tool => [cmf_hook_names::TOOL_PRE_INVOKE, cmf_hook_names::TOOL_POST_INVOKE],
-            Self::Prompt => [cmf_hook_names::PROMPT_PRE_FETCH, cmf_hook_names::PROMPT_POST_FETCH],
-            Self::Resource => [cmf_hook_names::RESOURCE_PRE_FETCH, cmf_hook_names::RESOURCE_POST_FETCH],
-        }
-    }
-
-    pub(crate) fn subject(self) -> &'static str {
-        match self {
-            Self::Tool => "tool call",
-            Self::Prompt => "prompt",
-            Self::Resource => "resource",
-        }
-    }
-
-    pub(crate) fn id_prefix(self) -> &'static str {
-        match self {
-            Self::Tool => "gateway-tool-call",
-            Self::Prompt => "gateway-prompt-request",
-            Self::Resource => "gateway-resource-request",
-        }
-    }
-}
-
 /// Only the projection and application differ between response types.
 /// The runner owns invocation, unchanged payloads, context, and denial handling.
 pub(crate) trait CmfResponse: Sized {

crates/contextforge-data-plane-cpex/src/extension_guard.rs

@@ -13,18 +13,26 @@ use cpex::cpex_core::{
     registry::{AnyHookHandler, HookEntry},
 };
 
+#[derive(Clone, Copy, PartialEq, Eq)]
+pub(crate) enum HeaderWritePolicy {
+    ReadOnly,
+    PreserveCredentials,
+    AllowNewCredentials,
+    All,
+}
+
 struct ExtensionGuard {
     entry: HookEntry,
     canonical: Arc<Mutex<Extensions>>,
     payload_writable: bool,
-    auth_headers_writable: bool,
+    header_policy: HeaderWritePolicy,
 }
 
 pub(crate) fn guarded_entries(
     entries: &[HookEntry],
     extensions: &Extensions,
     payload_writable: bool,
-    auth_headers_writable: bool,
+    header_policy: HeaderWritePolicy,
 ) -> Vec<HookEntry> {
     let canonical = Arc::new(Mutex::new(extensions.clone()));
     entries
@@ -35,7 +43,7 @@ pub(crate) fn guarded_entries(
                 entry: entry.clone(),
                 canonical: Arc::clone(&canonical),
                 payload_writable,
-                auth_headers_writable,
+                header_policy,
             }),
         })
         .collect()
@@ -66,7 +74,8 @@ impl AnyHookHandler for ExtensionGuard {
             if canonical.validate_immutable(&modified) {
                 let caps = &self.entry.plugin_ref.trusted_config().capabilities;
                 let mut permitted = canonical.cow_copy();
-                if caps.contains("write_headers")
+                if self.header_policy != HeaderWritePolicy::ReadOnly
+                    && caps.contains("write_headers")
                     && let Some(http) = modified.http
                 {
                     let mut http = http.into_inner();
@@ -78,8 +87,12 @@ impl AnyHookHandler for ExtensionGuard {
                         &mut http.response_headers,
                         canonical.http.as_ref().map(|http| &http.response_headers),
                     )?;
-                    if !self.auth_headers_writable {
-                        preserve_auth_headers(&mut http.request_headers, canonical.http.as_deref());
+                    if self.header_policy != HeaderWritePolicy::All {
+                        preserve_auth_headers(
+                            &mut http.request_headers,
+                            canonical.http.as_deref(),
+                            self.header_policy == HeaderWritePolicy::AllowNewCredentials,
+                        );
                     }
                     // A write-only plugin cannot return headers or metadata it never saw.
                     // Merge header edits while retaining the host's transport metadata.
@@ -145,13 +158,16 @@ fn normalize_headers(
 fn preserve_auth_headers(
     headers: &mut std::collections::HashMap<String, String>,
     original: Option<&cpex::cpex_core::extensions::HttpExtension>,
+    allow_new: bool,
 ) {
     fn protected(name: &str) -> bool {
         ["authorization", "proxy-authorization", "cookie", "x-api-key"]
             .iter()
             .any(|header| name.eq_ignore_ascii_case(header))
     }
-    headers.retain(|name, _| !protected(name));
+    headers.retain(|name, _| {
+        !protected(name) || (allow_new && !original.is_some_and(|http| http.has_request_header(name)))
+    });
     if let Some(original) = original {
         headers.extend(
             original