Re-run OAuth flow on 403 insufficient_scope (step-up)

## Motivation and Context

The MCP authorization specification (2025-11-25 scope-selection-strategy)
and RFC 6750 Section 3.1 describe OAuth 2.0 step-up authentication:
the client already holds a valid access token, but the server returns a 403
with `WWW-Authenticate: Bearer error="insufficient_scope", scope="..."`
to demand a higher-privileged scope for a specific operation.
The client is expected to run a fresh authorization request with
the escalated scope and retry.

Until now the Ruby SDK only ran the OAuth flow on 401 responses.
A 403 surfaced unchanged as a `RequestHandlerError` and the operation
that needed the escalated scope failed permanently, even with a valid token
in storage. This lined up with the `auth/scope-step-up` conformance
scenario being listed in `conformance/expected_failures.yml`.

The change adds:

- `MCP::Client::HTTP#send_request` recognises a 403 carrying
  an `insufficient_scope` Bearer challenge as a step-up signal,
  re-runs a full authorization request with the scope from the challenge,
  and retries the original request. Retries are capped at
  one per `send_request` call so a server that repeatedly demands more
  scope cannot loop the transport.
- `run_step_up_flow!` deliberately bypasses the refresh-token path that
  `run_oauth_flow!` uses on 401. Refreshing would re-issue the same
  scope set the existing token already has and the server already
  rejected. Only a fresh `authorization_code` grant with the escalated
  scope can succeed.
- A plain 403 with no `insufficient_scope` challenge continues to
  surface as `RequestHandlerError` with `error_type: :forbidden`.
- `conformance/client.rb` invokes a tool for the `auth/scope-step-up`
  scenario; the scenario's authorization middleware only requires
  the escalated scope on `tools/call`, so the previous `tools/list`-only
  client could never trigger the second authorization.
- `auth/scope-step-up` is removed from `conformance/expected_failures.yml`.

## How Has This Been Tested?

New `HTTPOAuthTest` cases cover: the 403 insufficient_scope path
re-authorizes with the escalated scope from the challenge and retries
the original request with the new bearer token; a plain 403 without
the `insufficient_scope` challenge is surfaced as
`error_type: :forbidden` with no OAuth side effects; repeated
`insufficient_scope` responses retry at most once and then surface the 403;
a provider with a saved `refresh_token` still goes through
the full authorization request rather than exchanging the refresh token,
because the existing scope set is precisely what was rejected.

Conformance: `auth/scope-step-up` now passes 22/22 with no warnings.
`bundle exec rake test`, `bundle exec rake rubocop`, and
`bundle exec rake conformance` are all green.

## Breaking Changes

None. Behaviour on 401 is unchanged. A 403 without the `insufficient_scope`
Bearer challenge still raises the same `RequestHandlerError` as before.
The new retry only fires when both the WWW-Authenticate header signals step-up
and the transport was constructed with an OAuth provider.

Author: koic

README.md

@@ -1913,6 +1913,9 @@ pass an `MCP::Client::OAuth::Provider` to the transport instead of a static `Aut
 - On a `401 Unauthorized`, parse the `WWW-Authenticate` header, discover the authorization server (Protected Resource Metadata + RFC 8414 Authorization Server Metadata),
   perform Dynamic Client Registration if needed, run the OAuth 2.1 Authorization Code flow with PKCE (S256), and retry the failed request with the acquired token.
 - On subsequent 401s with a saved `refresh_token`, exchange it at the token endpoint before falling back to the full interactive flow (RFC 6749 Section 6).
+- On a `403 Forbidden` whose `WWW-Authenticate` header carries `error="insufficient_scope"` (OAuth 2.0 step-up, RFC 6750 Section 3.1 and the MCP scope-selection-strategy),
+  run a fresh authorization request for the union of the currently granted scope and the scope named in the challenge, then retry the failed request once.
+  The refresh path is bypassed because refreshing would re-issue the same scope set the server just rejected. A `403` without that challenge is surfaced unchanged.
 - Request the `offline_access` scope when `client_metadata[:grant_types]` includes `refresh_token` and the authorization server advertises `offline_access` in its metadata
   `scopes_supported` (SEP-2207). This is what lets the server issue the `refresh_token` used above. As an SDK-level safeguard, when the authorization server does not advertise
   `offline_access` the scope is also stripped from any other source (challenge, PRM, or provider-supplied scope) so a server that does not support it never receives it.

conformance/client.rb

@@ -116,7 +116,16 @@ def build_oauth_provider(context, scenario:)
   # Auth-only scenarios: the protocol-level checks (PRM/AS metadata, DCR, PKCE, token usage)
   # are observed by the conformance server during `connect` and the subsequent request below.
   # Listing tools forces a second authenticated MCP request so the bearer token usage check fires.
-  client.tools
+  tools = client.tools
+
+  # `auth/scope-step-up` only fires its escalation 403 on `tools/call`, not `tools/list`,
+  # so the client must actually invoke a tool to drive the second authorization request
+  # the scenario asserts on.
+  if scenario == "auth/scope-step-up"
+    tool = tools.find { |t| t.name == "test-tool" } || tools.first
+    abort("No tool exposed by conformance server for #{scenario}") unless tool
+    client.call_tool(tool: tool, arguments: {})
+  end
 else
   abort("Unknown or unsupported scenario: #{scenario}")
 end

conformance/expected_failures.yml

@@ -5,7 +5,6 @@ client:
   # TODO: Elicitation not implemented in Ruby client.
   - elicitation-sep1034-client-defaults
   # TODO: Remaining OAuth/auth scenarios not yet implemented in Ruby client.
-  - auth/scope-step-up
   - auth/2025-03-26-oauth-metadata-backcompat
   - auth/2025-03-26-oauth-endpoint-fallback
   - auth/client-credentials-jwt

lib/mcp/client/http.rb

@@ -185,6 +185,7 @@ def send_request(request:)
         method = request[:method] || request["method"]
         params = request[:params] || request["params"]
         oauth_retried = false
+        step_up_retried = false
 
         begin
           response = client.post("", request, session_headers)
@@ -217,6 +218,19 @@ def send_request(request:)
             original_error: e,
           )
         rescue Faraday::ForbiddenError => e
+          # OAuth 2.0 step-up: a 403 carrying `error="insufficient_scope"` in
+          # the Bearer challenge means the existing access token is valid
+          # but lacks scopes the server now requires for this operation.
+          # Re-run the full authorization flow with the escalated scope from
+          # the challenge and retry once. A plain 403 without the challenge is
+          # surfaced unchanged.
+          if @oauth && !step_up_retried && insufficient_scope_challenge?(e)
+            step_up_retried = true
+            run_step_up_flow!(forbidden_error: e)
+
+            retry
+          end
+
           raise RequestHandlerError.new(
             "You are forbidden to make #{method} requests",
             { method: method, params: params },
@@ -337,16 +351,63 @@ def session_headers
       #
       # https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#error-handling
       def run_oauth_flow!(unauthorized_error:)
-        response = unauthorized_error.response || {}
-        response_headers = response[:headers] || {}
-        www_authenticate = response_headers["www-authenticate"] || response_headers["WWW-Authenticate"]
-        params = MCP::Client::OAuth::Discovery.parse_www_authenticate(www_authenticate)
+        params = parse_www_authenticate_from_error(unauthorized_error)
+        flow = MCP::Client::OAuth::Flow.new(provider: @oauth)
+        return if attempt_refresh(flow: flow, resource_metadata_url: params["resource_metadata"])
 
+        run_full_authorization_flow!(flow: flow, params: params)
+      end
+
+      # Drives a full Authorization Code + PKCE flow without first attempting
+      # to refresh the access token. Used for the MCP scope-selection-strategy
+      # step-up path: the provider already holds a valid access token,
+      # but the server returned a 403 with
+      # `WWW-Authenticate: ... error="insufficient_scope", scope="..."`
+      # per RFC 6750 Section 3.1. Refreshing the existing token would re-issue
+      # the same scope set the server already rejected, so the SDK must run
+      # a fresh authorization request. The request asks for the union of
+      # the currently granted scope and the newly demanded scope; otherwise
+      # the caller would lose previously held scopes and trigger another step-up
+      # on the next operation that needs them.
+      # https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#scope-selection-strategy
+      def run_step_up_flow!(forbidden_error:)
+        params = parse_www_authenticate_from_error(forbidden_error)
         flow = MCP::Client::OAuth::Flow.new(provider: @oauth)
-        if attempt_refresh(flow: flow, resource_metadata_url: params["resource_metadata"])
-          return
-        end
+        params = params.merge("scope" => escalated_step_up_scope(params["scope"]))
+
+        run_full_authorization_flow!(flow: flow, params: params)
+      end
+
+      # Returns the space-separated union of the currently granted scope (read
+      # from the stored token response per RFC 6749 Section 5.1) and the scope
+      # demanded by the step-up challenge. Duplicates are collapsed; order
+      # follows first appearance so existing scopes precede the newly added
+      # ones. Returns `nil` when neither side carries a scope so
+      # `build_authorization_url` omits the `scope` parameter entirely.
+      def escalated_step_up_scope(challenge_scope)
+        tokens = @oauth.tokens
+        granted = tokens.is_a?(Hash) ? (tokens["scope"] || tokens[:scope]) : nil
+        scopes = [granted, challenge_scope].compact.flat_map { |scope| scope.to_s.split }.uniq
+
+        scopes.empty? ? nil : scopes.join(" ")
+      end
+
+      # True when the response on `forbidden_error` carries a Bearer challenge
+      # with `error="insufficient_scope"` per RFC 6750 Section 3.1 and the MCP
+      # scope-selection-strategy section. A 403 without that signal is not a
+      # step-up challenge and must not trigger re-authorization.
+      def insufficient_scope_challenge?(forbidden_error)
+        parse_www_authenticate_from_error(forbidden_error)["error"] == "insufficient_scope"
+      end
+
+      def parse_www_authenticate_from_error(error)
+        response = error.response || {}
+        response_headers = response[:headers] || {}
+        header = response_headers["www-authenticate"] || response_headers["WWW-Authenticate"]
+        MCP::Client::OAuth::Discovery.parse_www_authenticate(header)
+      end
 
+      def run_full_authorization_flow!(flow:, params:)
         # Use the URL snapshotted at `initialize` time so a post-construction
         # mutation of `@url` cannot redirect PRM/AS discovery and the authorize
         # URL to an attacker-controlled host.