Support mantle endpoint + responses API for OAI models in bedrock#330
Conversation
…support-mantle-responses-oai-models-bedrock
Erin McNulty (erin2722)
requested review from
Alex Z (CLowbrow) and
Ken Jiang (knjiang)
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb480e35ea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| if is_aws_runtime && !has_openai { | ||
| segments.push("openai"); |
There was a problem hiding this comment.
This changes every bedrock-runtime Chat Completions request to /openai/v1/chat/completions, but the router can route non-OpenAI Bedrock models through this path too (for example the existing bedrock_openai_catalog_format_uses_chat_completions_transport scenario with an Anthropic Bedrock model). AWS documents the generic Bedrock Runtime Chat Completions endpoint as https://bedrock-runtime.{region}.amazonaws.com/v1/chat/completions (docs), so Claude/Nova-style Bedrock chat-completions calls will now be sent to the OpenAI-on-Bedrock path instead of the runtime path unless this is gated on the model family or configured endpoint.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
there are no configs that will hit this now
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b73e4e4d3c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| Error::InvalidRequest("Bedrock region required for mantle endpoint".into()) | ||
| })?; | ||
| Url::parse(&format!( | ||
| "https://bedrock-mantle.{region}.api.aws/openai/v1/{suffix}" |
There was a problem hiding this comment.
When responses_url() or the xAI chat-completions path uses this helper, it now sends requests to https://bedrock-mantle.{region}.api.aws/openai/v1/.... AWS documents the Mantle base as https://bedrock-mantle.<region>.api.aws/v1, with requests going to /v1/chat/completions and /v1/responses (https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-mantle.html), so every new Mantle route added by this change will hit the wrong path instead of the Bedrock Mantle API.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
the docs are terrible but this isn't a real issue, all of the relevant models have /openai in their endpoints
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aefcd47b83
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| if self.is_aws_managed_endpoint() { | ||
| return self.mantle_openai_url("responses"); |
There was a problem hiding this comment.
When this branch is taken with AwsSignatureV4 credentials, the request is sent to bedrock-mantle, but send_signed still calls sign_request, which uses the provider-wide service (bedrock by default, or a single user override). AWS documents Mantle SigV4 auth as using service name bedrock-mantle (https://docs.aws.amazon.com/bedrock/latest/userguide/count-tokens.html), so IAM-authenticated Responses calls will be signed for the wrong service and rejected unless callers manually break the runtime paths by changing the whole provider service.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This doesn't seem to be a problem, I tested IAM with these requests and it is working
2 participants
https://github.com/braintrustdata/braintrust/pull/16666