diff --git a/en/docs/ai-gateway/1.2.0/README.md b/en/docs/ai-gateway/1.2.0/README.md index 0c57b2885..3946abb72 100644 --- a/en/docs/ai-gateway/1.2.0/README.md +++ b/en/docs/ai-gateway/1.2.0/README.md @@ -49,6 +49,8 @@ An LLM Provider represents a connection to an AI backend service such as OpenAI, Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend. +To connect the gateway to AWS Bedrock, see [Configure an AWS Bedrock LLM Provider](llm-proxy/configure-aws-bedrock-provider.md). The guide covers both Bedrock bearer API keys and AWS Signature Version 4 (SigV4) authentication. + ### LLM Proxy An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., `/assistant`) and can have its own policies applied. This enables: diff --git a/en/docs/ai-gateway/1.2.0/llm-proxy/configure-aws-bedrock-provider.md b/en/docs/ai-gateway/1.2.0/llm-proxy/configure-aws-bedrock-provider.md new file mode 100644 index 000000000..5cbf79bd0 --- /dev/null +++ b/en/docs/ai-gateway/1.2.0/llm-proxy/configure-aws-bedrock-provider.md @@ -0,0 +1,486 @@ +--- +title: "Configure an AWS Bedrock Large Language Model provider" +description: "Connect API Platform AI Gateway to AWS Bedrock using a bearer API key or AWS Signature Version 4 authentication, then invoke a model through the gateway." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/configure-aws-bedrock-provider/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/configure-aws-bedrock-provider.md +tags: + - ai-gateway + - llm + - aws-bedrock + - authentication +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-04 +content_type: "guide" +--- + +# Configure an AWS Bedrock Large Language Model provider + +Connect API Platform AI Gateway directly to the regional AWS Bedrock Runtime endpoint. You can authenticate the gateway to Bedrock in either of these ways: + +- **Bearer authentication** with an AWS Bedrock API key +- **AWS Signature Version 4 (SigV4)** with IAM credentials or a workload role + +Both methods expose the native Bedrock `Converse` and `ConverseStream` operations through the gateway. They use the regional Bedrock Runtime endpoint, not the Bedrock Mantle endpoint. For a base model ID, choose an AWS Region where that model is available, and use the same Region in the Bedrock endpoint, SigV4 policy, and model ID. For an inference profile ID, invoke Bedrock through the source Region endpoint where the profile is supported; the profile may route requests to destination Regions. In this case, the SigV4 Region must match the source Region in the endpoint. + +## Before you begin + +Make sure that: + +- The AI Gateway is running and its management API is available at `http://localhost:9090/api/management/v1`. +- You have the gateway controller administrator username and password. +- Your model is available to your AWS account in the selected Region. +- You have either an [AWS Bedrock API key](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html) or an IAM identity that can invoke the model. +- `curl` is installed. Install `jq` as well if you want the commands to capture the generated gateway API key automatically. + +Set the gateway administrator credentials: + +```bash +export ADMIN_USERNAME="admin" +export ADMIN_PASSWORD="" +export AWS_REGION="" +``` + +For example, set `AWS_REGION` to `us-east-1`, `us-west-2`, or another Region where your selected Bedrock model is available. + +## Choose an authentication method + +Use the following table to choose the authentication method that matches your gateway environment. + +| Method | Recommended use | +|--------|-----------------| +| Bedrock bearer API key | Getting started and development environments | +| SigV4 with the default credential chain | Production gateways running with an Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), or Amazon Elastic Kubernetes Service (EKS) Pod Identity role | +| SigV4 with IAM Roles for Service Accounts (IRSA) | Gateways running on EKS with IRSA | +| SigV4 with AWS Security Token Service (STS) AssumeRole | Cross-account access or a gateway that must assume a dedicated Bedrock role | +| SigV4 with an IAM access key | Local testing when a workload role is unavailable | + +!!! note "Two different API keys" + A Bedrock bearer API key authenticates the gateway to AWS. Later in this guide, you create a gateway consumer API key that authenticates your application to the gateway. Do not use one in place of the other. + +## Option 1: Bearer authentication + +AWS Bedrock API keys are sent to the Bedrock Runtime endpoint in the `Authorization: Bearer ` header. AWS recommends short-term keys for production and long-term keys only for exploration. + +### Step 1: Generate a Bedrock API key + +Follow the AWS instructions to [generate an AWS Bedrock API key](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys-generate.html): + +1. Open the AWS Bedrock console in the Region where you will invoke the model. +2. Generate a short-term or long-term API key. +3. Copy the key when AWS displays it. + +Short-term keys last for the current AWS session, up to 12 hours, and are recommended for production. Long-term keys have a configurable expiration and are intended for exploration and development. + +The IAM identity associated with the key must be allowed to invoke the selected model. An organization-level deny on `bedrock:CallWithBearerToken` prevents the key from being used with the Bedrock endpoint. + +### Step 2: Store the Bedrock API key + +```bash +export AWS_BEARER_TOKEN_BEDROCK="" +``` + +Do not use an AWS access key ID or secret access key as the bearer token. + +Store the key as an encrypted gateway secret: + +```bash +curl --fail-with-body -X POST \ + http://localhost:9090/api/management/v1/secrets \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + -H "Content-Type: application/yaml" \ + --data-binary @- < +``` + +The service and Region must match the Bedrock Runtime endpoint. The policy supports four credential acquisition modes. + +### IAM access key + +Use `iam-user-access-key` for local testing when the gateway cannot use a workload role. Store the credentials as gateway secrets instead of embedding them in the provider definition. + +Set the AWS credentials in your current shell: + +```bash +export AWS_ACCESS_KEY_ID="" +export AWS_SECRET_ACCESS_KEY="" +``` + +Create a secret for each value: + +```bash +curl --fail-with-body -X POST \ + http://localhost:9090/api/management/v1/secrets \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + -H "Content-Type: application/yaml" \ + --data-binary @- < + authenticationType: default-credential-chain +``` + +Do not set an access key, secret access key, session token, or role Amazon Resource Name (ARN) for this mode. The AWS SDK resolves credentials from the gateway runtime environment. + +### IAM roles for service accounts + +Use `irsa` when the gateway workload already has the required IAM role through IRSA. The gateway must run on EKS with an OpenID Connect (OIDC) provider and a Kubernetes service account associated with an IAM role. The EKS Pod Identity Webhook must inject `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE` into the gateway pod. + +Replace the `bedrock-aws-auth` parameters with: + +```yaml +params: &bedrock-aws-auth + service: bedrock + region: + authenticationType: irsa + awsRoleSessionName: bedrock-gateway-session +``` + +You can add `awsRoleARN` explicitly. If you omit it, the policy uses the injected `AWS_ROLE_ARN` value: + +```yaml +awsRoleARN: arn:aws:iam:::role/ +``` + +### STS AssumeRole + +Use `sts-assume-role` when the gateway must assume a dedicated or cross-account Bedrock role. The gateway first needs a source identity that is allowed to call `sts:AssumeRole`; the target role must trust that source identity. + +Replace the `bedrock-aws-auth` parameters with: + +```yaml +params: &bedrock-aws-auth + service: bedrock + region: + authenticationType: sts-assume-role + awsRoleARN: arn:aws:iam:::role/ + awsRoleSessionName: bedrock-gateway-session +``` + +By default, the policy uses the gateway's AWS SDK credential chain as the source identity. You can instead provide `awsAccessKeyID`, `awsSecretAccessKey`, and, for temporary source credentials, `awsSessionToken` through gateway secrets. + +For a cross-account role that requires an external ID, also add: + +```yaml +awsRoleExternalID: +``` + +## Configure IAM permissions for SigV4 + +Attach a least-privilege policy to the IAM user or role that invokes Bedrock. The following policy allows the non-streaming and streaming operations used in this guide: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "InvokeBedrockModels", + "Effect": "Allow", + "Action": [ + "bedrock:InvokeModel", + "bedrock:InvokeModelWithResponseStream" + ], + "Resource": "*" + } + ] +} +``` + +For production, replace `*` with the model or inference profile Amazon Resource Name (ARN) values the gateway is allowed to invoke. When you scope access to an inference profile, authorize both the inference profile ARN and the corresponding foundation model ARN in each destination Region that the profile can route to. See [Prerequisites for running model inference](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-prereq.html) for the complete AWS guidance. + +For `sts-assume-role`, the source identity also needs `sts:AssumeRole` permission for the target role, and the target role's trust policy must trust the source identity. + +## Verify the provider + +Retrieve the deployed provider: + +```bash +curl --fail-with-body \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + http://localhost:9090/api/management/v1/llm-providers/bedrock-provider +``` + +## Create a gateway consumer API key + +The provider examples protect the exposed routes with `api-key-auth`. Create a consumer key for the application that will call Bedrock through the gateway: + +```bash +export GATEWAY_API_KEY=$(curl --fail-with-body -s -X POST \ + http://localhost:9090/api/management/v1/llm-providers/bedrock-provider/api-keys \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + -H "Content-Type: application/json" \ + -d '{"name":"bedrock-client"}' \ + | jq -r '.apiKey.apiKey') +``` + +Confirm that a key was returned: + +```bash +test -n "$GATEWAY_API_KEY" && test "$GATEWAY_API_KEY" != "null" +``` + +The key value is returned only when it is created or regenerated. Store it securely. + +## Invoke Bedrock through the gateway + +Set a base model ID that is available in the selected AWS Region, or an inference profile ID that is supported by the source Region configured in the Bedrock Runtime endpoint: + +```bash +export BEDROCK_MODEL_ID="" +``` + +Call the native Bedrock `Converse` operation through the gateway: + +```bash +export CA_CERT_PATH="" + +curl --fail-with-body -X POST \ + "https://localhost:8443/bedrock/v1.0/model/${BEDROCK_MODEL_ID}/converse" \ + -H "Content-Type: application/json" \ + -H "X-API-Key: $GATEWAY_API_KEY" \ + -d '{ + "messages": [ + { + "role": "user", + "content": [ + {"text": "Reply with a short hello from AWS Bedrock."} + ] + } + ], + "inferenceConfig": { + "maxTokens": 64, + "temperature": 0.2 + } + }' \ + --cacert "$CA_CERT_PATH" +``` + +URL-encode the model ID if it contains characters that are not safe in a URL path. + +If the gateway uses a certificate signed by a public or locally trusted certificate authority (CA), omit `--cacert`. For a local self-signed certificate, add the issuing CA certificate to your trust store or pass it with `--cacert`. + +## Troubleshooting + +### Bedrock returns `AccessDeniedException` + +Confirm that the IAM identity can invoke the selected model in the configured Region. For streaming requests, it also needs `bedrock:InvokeModelWithResponseStream`. + +### The gateway returns `502 Bad Gateway` + +For SigV4, this usually means the policy could not retrieve AWS credentials or sign the request. Check the gateway logs and verify the selected credential mode. For `default-credential-chain`, confirm that the gateway workload actually has a usable AWS credential source. + +### AWS reports a signature mismatch + +Confirm that `service` is `bedrock` and that the policy Region matches the Region in `https://bedrock-runtime..amazonaws.com`. + +### IRSA fails during policy initialization + +Confirm that the gateway pod has both `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`, and that the projected token file is readable. Also verify the role trust policy and its OIDC subject condition. + +### STS returns `AccessDenied` + +Check both sides of the relationship: the source identity must be allowed to call `sts:AssumeRole`, and the target role's trust policy must trust the source identity. + +## Security recommendations + +- Do not commit AWS access keys, session tokens, Bedrock API keys, gateway passwords, or external IDs. +- Prefer the default credential chain, IRSA, or STS AssumeRole over long-lived IAM user access keys. +- Use short-term Bedrock API keys for production bearer authentication. +- Restrict IAM permissions to the required model and inference profile resources. +- Expose only the required Bedrock operations through `accessControl`. +- Use HTTPS for Bedrock and production gateway endpoints. diff --git a/en/docs/ai-gateway/1.2.0/llm-proxy/llm-templates.md b/en/docs/ai-gateway/1.2.0/llm-proxy/llm-templates.md index 81b224e73..9b8ad62c3 100644 --- a/en/docs/ai-gateway/1.2.0/llm-proxy/llm-templates.md +++ b/en/docs/ai-gateway/1.2.0/llm-proxy/llm-templates.md @@ -8,7 +8,7 @@ tags: - llm - reference author: WSO2 API Platform Documentation Team -last_updated: 2026-06-16 +last_updated: 2026-08-04 content_type: "reference" --- @@ -33,7 +33,7 @@ The API Platform Gateway ships with the following pre-configured LLM provider te | `anthropic` | Anthropic | Anthropic Claude Provider | | `gemini` | Gemini | Google Gemini Provider | | `mistralai` | MistralAI | Mistral AI Provider | -| `awsbedrock` | AWS Bedrock | Amazon Bedrock Provider | +| `awsbedrock` | AWS Bedrock | AWS Bedrock Provider | | `azureai-foundry` | Azure AI Foundry | Microsoft Azure AI Foundry Provider | These templates are automatically loaded when the gateway starts and are immediately available for use when creating LLM providers. @@ -43,39 +43,43 @@ These templates are automatically loaded when the gateway starts and are immedia Each LLM provider template follows a standard YAML structure: ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: spec: displayName: + groupId: + managedBy: + version: # Provider characteristics promptTokens: - location: + location: identifier: completionTokens: - location: + location: identifier: totalTokens: - location: + location: identifier: remainingTokens: - location: + location: identifier: requestModel: - location: + location: identifier: responseModel: - location: + location: identifier: ``` ### Metadata Extraction Patterns -Templates support three types of extraction locations: +Templates support four types of extraction locations: - **`payload`**: Extract from JSON response body using JSONPath expressions (e.g., `$.usage.prompt_tokens`) - **`header`**: Extract from HTTP response headers using header name (e.g., `x-ratelimit-remaining-tokens`) +- **`queryParam`**: Extract from a URL query parameter - **`pathParam`**: Extract from URL path using regular expressions (e.g., `(?<=models/)[a-zA-Z0-9.\-]+`) ## Template Details @@ -85,12 +89,15 @@ Templates support three types of extraction locations: The OpenAI template extracts metadata from OpenAI API responses. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: openai spec: displayName: OpenAI + groupId: wso2-openai + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.prompt_tokens @@ -109,6 +116,15 @@ spec: responseModel: location: payload identifier: $.model + resourceMappings: + resources: + - resource: /responses + promptTokens: + location: payload + identifier: $.usage.input_tokens + completionTokens: + location: payload + identifier: $.usage.output_tokens ``` ### Azure OpenAI @@ -116,18 +132,21 @@ spec: The Azure OpenAI template is compatible with Microsoft's Azure OpenAI Service API. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: azure-openai spec: displayName: Azure OpenAI + groupId: wso2-azure-openai + managedBy: wso2 + version: v1.0 promptTokens: location: payload - identifier: $.usage.input_tokens + identifier: $.usage.prompt_tokens completionTokens: location: payload - identifier: $.usage.output_tokens + identifier: $.usage.completion_tokens totalTokens: location: payload identifier: $.usage.total_tokens @@ -140,6 +159,15 @@ spec: responseModel: location: payload identifier: $.model + resourceMappings: + resources: + - resource: /responses + promptTokens: + location: payload + identifier: $.usage.input_tokens + completionTokens: + location: payload + identifier: $.usage.output_tokens ``` ### Anthropic @@ -147,12 +175,15 @@ spec: The Anthropic template extracts metadata from Anthropic Claude API responses. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: anthropic spec: displayName: Anthropic + groupId: wso2-anthropic + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.input_tokens @@ -175,12 +206,15 @@ spec: The Gemini template is designed for Google's Gemini API. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: gemini spec: displayName: Gemini + groupId: wso2-gemini + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usageMetadata.promptTokenCount @@ -206,12 +240,15 @@ spec: The MistralAI template supports Mistral AI's API. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: mistralai spec: displayName: MistralAI + groupId: wso2-mistralai + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.prompt_tokens @@ -234,15 +271,18 @@ spec: ### AWS Bedrock -The AWS Bedrock template is designed for Amazon Bedrock's unified API. +The AWS Bedrock template is designed for the AWS Bedrock unified API. To configure and deploy a provider with bearer or AWS Signature Version 4 (SigV4) authentication, see [Configure an AWS Bedrock Large Language Model provider](configure-aws-bedrock-provider.md). ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: awsbedrock spec: displayName: AWS Bedrock + groupId: wso2-awsbedrock + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.inputTokens @@ -254,10 +294,10 @@ spec: identifier: $.usage.totalTokens requestModel: location: pathParam - identifier: (?<=model/)[a-zA-Z0-9.:-]+(?=/) + identifier: model/([A-Za-z0-9.:-]+)/ responseModel: location: pathParam - identifier: (?<=model/)[a-zA-Z0-9.:-]+(?=/) + identifier: model/([A-Za-z0-9.:-]+)/ ``` ### Azure AI Foundry @@ -265,12 +305,15 @@ spec: The Azure AI Foundry template supports Microsoft's Azure AI Foundry platform. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: azureai-foundry spec: displayName: Azure AI Foundry + groupId: wso2-azureai-foundry + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.prompt_tokens @@ -289,6 +332,15 @@ spec: responseModel: location: payload identifier: $.model + resourceMappings: + resources: + - resource: /responses + promptTokens: + location: payload + identifier: $.usage.input_tokens + completionTokens: + location: payload + identifier: $.usage.output_tokens ``` ## Creating an LLM Provider with a Template @@ -296,11 +348,11 @@ spec: To create an LLM provider using any of the out-of-the-box templates: ```bash -curl -X POST http://localhost:9090/api/management/v0.9/llm-providers \ +curl -X POST http://localhost:9090/api/management/v1/llm-providers \ -H "Content-Type: application/yaml" \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProvider metadata: name: @@ -308,6 +360,7 @@ spec: displayName: version: v1.0 template: + context: / upstream: url: https://api.openai.com/v1 auth: @@ -330,6 +383,7 @@ Replace the placeholders: - ``: Unique identifier for your provider (e.g., `my-openai-provider`) - ``: Human-readable name (e.g., `My OpenAI Provider`) - ``: One of the supported template IDs (`openai`, `azure-openai`, `anthropic`, `gemini`, `mistralai`, `awsbedrock`, `azureai-foundry`) +- ``: Base path for the provider routes (e.g., `openai/latest`) - ``: Authentication header name (e.g., `Authorization` for most providers) - ``: Your API key with appropriate prefix (e.g., `Bearer sk-...` for OpenAI) @@ -346,7 +400,7 @@ The gateway automatically uses the template's metadata extraction patterns to: To list all available LLM provider templates: ```bash -curl -X GET http://localhost:9090/api/management/v0.9/llm-provider-templates \ +curl -X GET http://localhost:9090/api/management/v1/llm-provider-templates \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" ``` @@ -355,7 +409,7 @@ curl -X GET http://localhost:9090/api/management/v0.9/llm-provider-templates \ To retrieve details of a specific template: ```bash -curl -X GET http://localhost:9090/api/management/v0.9/llm-provider-templates/openai \ +curl -X GET http://localhost:9090/api/management/v1/llm-provider-templates/openai \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" ``` @@ -364,16 +418,19 @@ curl -X GET http://localhost:9090/api/management/v0.9/llm-provider-templates/ope Platform administrators can create custom templates for LLM providers not covered by the out-of-the-box templates: ```bash -curl -X POST http://localhost:9090/api/management/v0.9/llm-provider-templates \ +curl -X POST http://localhost:9090/api/management/v1/llm-provider-templates \ -H "Content-Type: application/yaml" \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: custom-provider spec: displayName: Custom Provider + groupId: custom-provider + managedBy: customer + version: v1.0 totalTokens: location: payload identifier: $.tokens.total @@ -385,16 +442,19 @@ EOF To update an existing custom template: ```bash -curl -X PUT http://localhost:9090/api/management/v0.9/llm-provider-templates/custom-provider \ +curl -X PUT http://localhost:9090/api/management/v1/llm-provider-templates/custom-provider \ -H "Content-Type: application/yaml" \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: custom-provider spec: displayName: Custom Provider Updated + groupId: custom-provider + managedBy: customer + version: v1.0 promptTokens: location: payload identifier: $.usage.input_tokens @@ -407,7 +467,7 @@ EOF To delete a custom template: ```bash -curl -X DELETE http://localhost:9090/api/management/v0.9/llm-provider-templates/custom-provider \ +curl -X DELETE http://localhost:9090/api/management/v1/llm-provider-templates/custom-provider \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" ``` @@ -415,18 +475,24 @@ curl -X DELETE http://localhost:9090/api/management/v0.9/llm-provider-templates/ ## Template Field Reference +Use this table to configure an `LlmProviderTemplate` resource. + | Field | Type | Required | Description | |-------|------|----------|-------------| -| `apiVersion` | string | Yes | API version, must be `gateway.api-platform.wso2.com/v1alpha1` | +| `apiVersion` | string | Yes | API version, must be `gateway.api-platform.wso2.com/v1` | | `kind` | string | Yes | Resource kind, must be `LlmProviderTemplate` | | `metadata.name` | string | Yes | Unique identifier for the template (used as template ID) | | `spec.displayName` | string | Yes | Human-readable name for the template | +| `spec.groupId` | string | No | Stable family identifier shared by versions of the same template; defaults to `metadata.name` | +| `spec.managedBy` | string | No | Template owner; built-in templates use `wso2`, while custom templates default to `customer` | +| `spec.version` | string | No | Template content version; defaults to `v1.0` | | `spec.promptTokens` | object | No | Configuration for extracting prompt/input token count | | `spec.completionTokens` | object | No | Configuration for extracting completion/output token count | | `spec.totalTokens` | object | No | Configuration for extracting total token count | | `spec.remainingTokens` | object | No | Configuration for extracting remaining token allowance | | `spec.requestModel` | object | No | Configuration for extracting request model identifier | | `spec.responseModel` | object | No | Configuration for extracting response model identifier | +| `spec.resourceMappings` | object | No | Resource-specific extraction overrides, such as token paths for `/responses` | ### Extraction Configuration Object @@ -434,5 +500,5 @@ Each extraction configuration object has the following structure: | Field | Type | Values | Description | |-------|------|--------|-------------| -| `location` | string | `payload`, `header`, `pathParam` | Where to extract the value from | -| `identifier` | string | - | JSONPath (for payload), header name (for header), or regex pattern (for pathParam) | +| `location` | string | `payload`, `header`, `queryParam`, `pathParam` | Where to extract the value from | +| `identifier` | string | - | JSONPath (for payload), header or query parameter name, or regex pattern (for pathParam) | diff --git a/en/docs/ai-gateway/1.2.0/overview.md b/en/docs/ai-gateway/1.2.0/overview.md index da2199c23..4ca2ff349 100644 --- a/en/docs/ai-gateway/1.2.0/overview.md +++ b/en/docs/ai-gateway/1.2.0/overview.md @@ -49,6 +49,8 @@ An LLM Provider represents a connection to an AI backend service such as OpenAI, Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend. +To connect the gateway to AWS Bedrock, see [Configure an AWS Bedrock LLM Provider](llm-proxy/configure-aws-bedrock-provider.md). The guide covers both Bedrock bearer API keys and AWS Signature Version 4 (SigV4) authentication. + ### LLM Proxy An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., `/assistant`) and can have its own policies applied. This enables: diff --git a/en/docs/ai-gateway/next/README.md b/en/docs/ai-gateway/next/README.md index 0c57b2885..3946abb72 100644 --- a/en/docs/ai-gateway/next/README.md +++ b/en/docs/ai-gateway/next/README.md @@ -49,6 +49,8 @@ An LLM Provider represents a connection to an AI backend service such as OpenAI, Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend. +To connect the gateway to AWS Bedrock, see [Configure an AWS Bedrock LLM Provider](llm-proxy/configure-aws-bedrock-provider.md). The guide covers both Bedrock bearer API keys and AWS Signature Version 4 (SigV4) authentication. + ### LLM Proxy An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., `/assistant`) and can have its own policies applied. This enables: diff --git a/en/docs/ai-gateway/next/llm-proxy/configure-aws-bedrock-provider.md b/en/docs/ai-gateway/next/llm-proxy/configure-aws-bedrock-provider.md new file mode 100644 index 000000000..5cbf79bd0 --- /dev/null +++ b/en/docs/ai-gateway/next/llm-proxy/configure-aws-bedrock-provider.md @@ -0,0 +1,486 @@ +--- +title: "Configure an AWS Bedrock Large Language Model provider" +description: "Connect API Platform AI Gateway to AWS Bedrock using a bearer API key or AWS Signature Version 4 authentication, then invoke a model through the gateway." +canonical_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/configure-aws-bedrock-provider/ +md_url: https://wso2.com/api-platform/docs/ai-gateway/llm-proxy/configure-aws-bedrock-provider.md +tags: + - ai-gateway + - llm + - aws-bedrock + - authentication +author: WSO2 API Platform Documentation Team +last_updated: 2026-08-04 +content_type: "guide" +--- + +# Configure an AWS Bedrock Large Language Model provider + +Connect API Platform AI Gateway directly to the regional AWS Bedrock Runtime endpoint. You can authenticate the gateway to Bedrock in either of these ways: + +- **Bearer authentication** with an AWS Bedrock API key +- **AWS Signature Version 4 (SigV4)** with IAM credentials or a workload role + +Both methods expose the native Bedrock `Converse` and `ConverseStream` operations through the gateway. They use the regional Bedrock Runtime endpoint, not the Bedrock Mantle endpoint. For a base model ID, choose an AWS Region where that model is available, and use the same Region in the Bedrock endpoint, SigV4 policy, and model ID. For an inference profile ID, invoke Bedrock through the source Region endpoint where the profile is supported; the profile may route requests to destination Regions. In this case, the SigV4 Region must match the source Region in the endpoint. + +## Before you begin + +Make sure that: + +- The AI Gateway is running and its management API is available at `http://localhost:9090/api/management/v1`. +- You have the gateway controller administrator username and password. +- Your model is available to your AWS account in the selected Region. +- You have either an [AWS Bedrock API key](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html) or an IAM identity that can invoke the model. +- `curl` is installed. Install `jq` as well if you want the commands to capture the generated gateway API key automatically. + +Set the gateway administrator credentials: + +```bash +export ADMIN_USERNAME="admin" +export ADMIN_PASSWORD="" +export AWS_REGION="" +``` + +For example, set `AWS_REGION` to `us-east-1`, `us-west-2`, or another Region where your selected Bedrock model is available. + +## Choose an authentication method + +Use the following table to choose the authentication method that matches your gateway environment. + +| Method | Recommended use | +|--------|-----------------| +| Bedrock bearer API key | Getting started and development environments | +| SigV4 with the default credential chain | Production gateways running with an Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), or Amazon Elastic Kubernetes Service (EKS) Pod Identity role | +| SigV4 with IAM Roles for Service Accounts (IRSA) | Gateways running on EKS with IRSA | +| SigV4 with AWS Security Token Service (STS) AssumeRole | Cross-account access or a gateway that must assume a dedicated Bedrock role | +| SigV4 with an IAM access key | Local testing when a workload role is unavailable | + +!!! note "Two different API keys" + A Bedrock bearer API key authenticates the gateway to AWS. Later in this guide, you create a gateway consumer API key that authenticates your application to the gateway. Do not use one in place of the other. + +## Option 1: Bearer authentication + +AWS Bedrock API keys are sent to the Bedrock Runtime endpoint in the `Authorization: Bearer ` header. AWS recommends short-term keys for production and long-term keys only for exploration. + +### Step 1: Generate a Bedrock API key + +Follow the AWS instructions to [generate an AWS Bedrock API key](https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys-generate.html): + +1. Open the AWS Bedrock console in the Region where you will invoke the model. +2. Generate a short-term or long-term API key. +3. Copy the key when AWS displays it. + +Short-term keys last for the current AWS session, up to 12 hours, and are recommended for production. Long-term keys have a configurable expiration and are intended for exploration and development. + +The IAM identity associated with the key must be allowed to invoke the selected model. An organization-level deny on `bedrock:CallWithBearerToken` prevents the key from being used with the Bedrock endpoint. + +### Step 2: Store the Bedrock API key + +```bash +export AWS_BEARER_TOKEN_BEDROCK="" +``` + +Do not use an AWS access key ID or secret access key as the bearer token. + +Store the key as an encrypted gateway secret: + +```bash +curl --fail-with-body -X POST \ + http://localhost:9090/api/management/v1/secrets \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + -H "Content-Type: application/yaml" \ + --data-binary @- < +``` + +The service and Region must match the Bedrock Runtime endpoint. The policy supports four credential acquisition modes. + +### IAM access key + +Use `iam-user-access-key` for local testing when the gateway cannot use a workload role. Store the credentials as gateway secrets instead of embedding them in the provider definition. + +Set the AWS credentials in your current shell: + +```bash +export AWS_ACCESS_KEY_ID="" +export AWS_SECRET_ACCESS_KEY="" +``` + +Create a secret for each value: + +```bash +curl --fail-with-body -X POST \ + http://localhost:9090/api/management/v1/secrets \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + -H "Content-Type: application/yaml" \ + --data-binary @- < + authenticationType: default-credential-chain +``` + +Do not set an access key, secret access key, session token, or role Amazon Resource Name (ARN) for this mode. The AWS SDK resolves credentials from the gateway runtime environment. + +### IAM roles for service accounts + +Use `irsa` when the gateway workload already has the required IAM role through IRSA. The gateway must run on EKS with an OpenID Connect (OIDC) provider and a Kubernetes service account associated with an IAM role. The EKS Pod Identity Webhook must inject `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE` into the gateway pod. + +Replace the `bedrock-aws-auth` parameters with: + +```yaml +params: &bedrock-aws-auth + service: bedrock + region: + authenticationType: irsa + awsRoleSessionName: bedrock-gateway-session +``` + +You can add `awsRoleARN` explicitly. If you omit it, the policy uses the injected `AWS_ROLE_ARN` value: + +```yaml +awsRoleARN: arn:aws:iam:::role/ +``` + +### STS AssumeRole + +Use `sts-assume-role` when the gateway must assume a dedicated or cross-account Bedrock role. The gateway first needs a source identity that is allowed to call `sts:AssumeRole`; the target role must trust that source identity. + +Replace the `bedrock-aws-auth` parameters with: + +```yaml +params: &bedrock-aws-auth + service: bedrock + region: + authenticationType: sts-assume-role + awsRoleARN: arn:aws:iam:::role/ + awsRoleSessionName: bedrock-gateway-session +``` + +By default, the policy uses the gateway's AWS SDK credential chain as the source identity. You can instead provide `awsAccessKeyID`, `awsSecretAccessKey`, and, for temporary source credentials, `awsSessionToken` through gateway secrets. + +For a cross-account role that requires an external ID, also add: + +```yaml +awsRoleExternalID: +``` + +## Configure IAM permissions for SigV4 + +Attach a least-privilege policy to the IAM user or role that invokes Bedrock. The following policy allows the non-streaming and streaming operations used in this guide: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "InvokeBedrockModels", + "Effect": "Allow", + "Action": [ + "bedrock:InvokeModel", + "bedrock:InvokeModelWithResponseStream" + ], + "Resource": "*" + } + ] +} +``` + +For production, replace `*` with the model or inference profile Amazon Resource Name (ARN) values the gateway is allowed to invoke. When you scope access to an inference profile, authorize both the inference profile ARN and the corresponding foundation model ARN in each destination Region that the profile can route to. See [Prerequisites for running model inference](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-prereq.html) for the complete AWS guidance. + +For `sts-assume-role`, the source identity also needs `sts:AssumeRole` permission for the target role, and the target role's trust policy must trust the source identity. + +## Verify the provider + +Retrieve the deployed provider: + +```bash +curl --fail-with-body \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + http://localhost:9090/api/management/v1/llm-providers/bedrock-provider +``` + +## Create a gateway consumer API key + +The provider examples protect the exposed routes with `api-key-auth`. Create a consumer key for the application that will call Bedrock through the gateway: + +```bash +export GATEWAY_API_KEY=$(curl --fail-with-body -s -X POST \ + http://localhost:9090/api/management/v1/llm-providers/bedrock-provider/api-keys \ + -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ + -H "Content-Type: application/json" \ + -d '{"name":"bedrock-client"}' \ + | jq -r '.apiKey.apiKey') +``` + +Confirm that a key was returned: + +```bash +test -n "$GATEWAY_API_KEY" && test "$GATEWAY_API_KEY" != "null" +``` + +The key value is returned only when it is created or regenerated. Store it securely. + +## Invoke Bedrock through the gateway + +Set a base model ID that is available in the selected AWS Region, or an inference profile ID that is supported by the source Region configured in the Bedrock Runtime endpoint: + +```bash +export BEDROCK_MODEL_ID="" +``` + +Call the native Bedrock `Converse` operation through the gateway: + +```bash +export CA_CERT_PATH="" + +curl --fail-with-body -X POST \ + "https://localhost:8443/bedrock/v1.0/model/${BEDROCK_MODEL_ID}/converse" \ + -H "Content-Type: application/json" \ + -H "X-API-Key: $GATEWAY_API_KEY" \ + -d '{ + "messages": [ + { + "role": "user", + "content": [ + {"text": "Reply with a short hello from AWS Bedrock."} + ] + } + ], + "inferenceConfig": { + "maxTokens": 64, + "temperature": 0.2 + } + }' \ + --cacert "$CA_CERT_PATH" +``` + +URL-encode the model ID if it contains characters that are not safe in a URL path. + +If the gateway uses a certificate signed by a public or locally trusted certificate authority (CA), omit `--cacert`. For a local self-signed certificate, add the issuing CA certificate to your trust store or pass it with `--cacert`. + +## Troubleshooting + +### Bedrock returns `AccessDeniedException` + +Confirm that the IAM identity can invoke the selected model in the configured Region. For streaming requests, it also needs `bedrock:InvokeModelWithResponseStream`. + +### The gateway returns `502 Bad Gateway` + +For SigV4, this usually means the policy could not retrieve AWS credentials or sign the request. Check the gateway logs and verify the selected credential mode. For `default-credential-chain`, confirm that the gateway workload actually has a usable AWS credential source. + +### AWS reports a signature mismatch + +Confirm that `service` is `bedrock` and that the policy Region matches the Region in `https://bedrock-runtime..amazonaws.com`. + +### IRSA fails during policy initialization + +Confirm that the gateway pod has both `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`, and that the projected token file is readable. Also verify the role trust policy and its OIDC subject condition. + +### STS returns `AccessDenied` + +Check both sides of the relationship: the source identity must be allowed to call `sts:AssumeRole`, and the target role's trust policy must trust the source identity. + +## Security recommendations + +- Do not commit AWS access keys, session tokens, Bedrock API keys, gateway passwords, or external IDs. +- Prefer the default credential chain, IRSA, or STS AssumeRole over long-lived IAM user access keys. +- Use short-term Bedrock API keys for production bearer authentication. +- Restrict IAM permissions to the required model and inference profile resources. +- Expose only the required Bedrock operations through `accessControl`. +- Use HTTPS for Bedrock and production gateway endpoints. diff --git a/en/docs/ai-gateway/next/llm-proxy/llm-templates.md b/en/docs/ai-gateway/next/llm-proxy/llm-templates.md index 81b224e73..9b8ad62c3 100644 --- a/en/docs/ai-gateway/next/llm-proxy/llm-templates.md +++ b/en/docs/ai-gateway/next/llm-proxy/llm-templates.md @@ -8,7 +8,7 @@ tags: - llm - reference author: WSO2 API Platform Documentation Team -last_updated: 2026-06-16 +last_updated: 2026-08-04 content_type: "reference" --- @@ -33,7 +33,7 @@ The API Platform Gateway ships with the following pre-configured LLM provider te | `anthropic` | Anthropic | Anthropic Claude Provider | | `gemini` | Gemini | Google Gemini Provider | | `mistralai` | MistralAI | Mistral AI Provider | -| `awsbedrock` | AWS Bedrock | Amazon Bedrock Provider | +| `awsbedrock` | AWS Bedrock | AWS Bedrock Provider | | `azureai-foundry` | Azure AI Foundry | Microsoft Azure AI Foundry Provider | These templates are automatically loaded when the gateway starts and are immediately available for use when creating LLM providers. @@ -43,39 +43,43 @@ These templates are automatically loaded when the gateway starts and are immedia Each LLM provider template follows a standard YAML structure: ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: spec: displayName: + groupId: + managedBy: + version: # Provider characteristics promptTokens: - location: + location: identifier: completionTokens: - location: + location: identifier: totalTokens: - location: + location: identifier: remainingTokens: - location: + location: identifier: requestModel: - location: + location: identifier: responseModel: - location: + location: identifier: ``` ### Metadata Extraction Patterns -Templates support three types of extraction locations: +Templates support four types of extraction locations: - **`payload`**: Extract from JSON response body using JSONPath expressions (e.g., `$.usage.prompt_tokens`) - **`header`**: Extract from HTTP response headers using header name (e.g., `x-ratelimit-remaining-tokens`) +- **`queryParam`**: Extract from a URL query parameter - **`pathParam`**: Extract from URL path using regular expressions (e.g., `(?<=models/)[a-zA-Z0-9.\-]+`) ## Template Details @@ -85,12 +89,15 @@ Templates support three types of extraction locations: The OpenAI template extracts metadata from OpenAI API responses. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: openai spec: displayName: OpenAI + groupId: wso2-openai + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.prompt_tokens @@ -109,6 +116,15 @@ spec: responseModel: location: payload identifier: $.model + resourceMappings: + resources: + - resource: /responses + promptTokens: + location: payload + identifier: $.usage.input_tokens + completionTokens: + location: payload + identifier: $.usage.output_tokens ``` ### Azure OpenAI @@ -116,18 +132,21 @@ spec: The Azure OpenAI template is compatible with Microsoft's Azure OpenAI Service API. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: azure-openai spec: displayName: Azure OpenAI + groupId: wso2-azure-openai + managedBy: wso2 + version: v1.0 promptTokens: location: payload - identifier: $.usage.input_tokens + identifier: $.usage.prompt_tokens completionTokens: location: payload - identifier: $.usage.output_tokens + identifier: $.usage.completion_tokens totalTokens: location: payload identifier: $.usage.total_tokens @@ -140,6 +159,15 @@ spec: responseModel: location: payload identifier: $.model + resourceMappings: + resources: + - resource: /responses + promptTokens: + location: payload + identifier: $.usage.input_tokens + completionTokens: + location: payload + identifier: $.usage.output_tokens ``` ### Anthropic @@ -147,12 +175,15 @@ spec: The Anthropic template extracts metadata from Anthropic Claude API responses. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: anthropic spec: displayName: Anthropic + groupId: wso2-anthropic + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.input_tokens @@ -175,12 +206,15 @@ spec: The Gemini template is designed for Google's Gemini API. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: gemini spec: displayName: Gemini + groupId: wso2-gemini + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usageMetadata.promptTokenCount @@ -206,12 +240,15 @@ spec: The MistralAI template supports Mistral AI's API. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: mistralai spec: displayName: MistralAI + groupId: wso2-mistralai + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.prompt_tokens @@ -234,15 +271,18 @@ spec: ### AWS Bedrock -The AWS Bedrock template is designed for Amazon Bedrock's unified API. +The AWS Bedrock template is designed for the AWS Bedrock unified API. To configure and deploy a provider with bearer or AWS Signature Version 4 (SigV4) authentication, see [Configure an AWS Bedrock Large Language Model provider](configure-aws-bedrock-provider.md). ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: awsbedrock spec: displayName: AWS Bedrock + groupId: wso2-awsbedrock + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.inputTokens @@ -254,10 +294,10 @@ spec: identifier: $.usage.totalTokens requestModel: location: pathParam - identifier: (?<=model/)[a-zA-Z0-9.:-]+(?=/) + identifier: model/([A-Za-z0-9.:-]+)/ responseModel: location: pathParam - identifier: (?<=model/)[a-zA-Z0-9.:-]+(?=/) + identifier: model/([A-Za-z0-9.:-]+)/ ``` ### Azure AI Foundry @@ -265,12 +305,15 @@ spec: The Azure AI Foundry template supports Microsoft's Azure AI Foundry platform. ```yaml -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: azureai-foundry spec: displayName: Azure AI Foundry + groupId: wso2-azureai-foundry + managedBy: wso2 + version: v1.0 promptTokens: location: payload identifier: $.usage.prompt_tokens @@ -289,6 +332,15 @@ spec: responseModel: location: payload identifier: $.model + resourceMappings: + resources: + - resource: /responses + promptTokens: + location: payload + identifier: $.usage.input_tokens + completionTokens: + location: payload + identifier: $.usage.output_tokens ``` ## Creating an LLM Provider with a Template @@ -296,11 +348,11 @@ spec: To create an LLM provider using any of the out-of-the-box templates: ```bash -curl -X POST http://localhost:9090/api/management/v0.9/llm-providers \ +curl -X POST http://localhost:9090/api/management/v1/llm-providers \ -H "Content-Type: application/yaml" \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProvider metadata: name: @@ -308,6 +360,7 @@ spec: displayName: version: v1.0 template: + context: / upstream: url: https://api.openai.com/v1 auth: @@ -330,6 +383,7 @@ Replace the placeholders: - ``: Unique identifier for your provider (e.g., `my-openai-provider`) - ``: Human-readable name (e.g., `My OpenAI Provider`) - ``: One of the supported template IDs (`openai`, `azure-openai`, `anthropic`, `gemini`, `mistralai`, `awsbedrock`, `azureai-foundry`) +- ``: Base path for the provider routes (e.g., `openai/latest`) - ``: Authentication header name (e.g., `Authorization` for most providers) - ``: Your API key with appropriate prefix (e.g., `Bearer sk-...` for OpenAI) @@ -346,7 +400,7 @@ The gateway automatically uses the template's metadata extraction patterns to: To list all available LLM provider templates: ```bash -curl -X GET http://localhost:9090/api/management/v0.9/llm-provider-templates \ +curl -X GET http://localhost:9090/api/management/v1/llm-provider-templates \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" ``` @@ -355,7 +409,7 @@ curl -X GET http://localhost:9090/api/management/v0.9/llm-provider-templates \ To retrieve details of a specific template: ```bash -curl -X GET http://localhost:9090/api/management/v0.9/llm-provider-templates/openai \ +curl -X GET http://localhost:9090/api/management/v1/llm-provider-templates/openai \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" ``` @@ -364,16 +418,19 @@ curl -X GET http://localhost:9090/api/management/v0.9/llm-provider-templates/ope Platform administrators can create custom templates for LLM providers not covered by the out-of-the-box templates: ```bash -curl -X POST http://localhost:9090/api/management/v0.9/llm-provider-templates \ +curl -X POST http://localhost:9090/api/management/v1/llm-provider-templates \ -H "Content-Type: application/yaml" \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: custom-provider spec: displayName: Custom Provider + groupId: custom-provider + managedBy: customer + version: v1.0 totalTokens: location: payload identifier: $.tokens.total @@ -385,16 +442,19 @@ EOF To update an existing custom template: ```bash -curl -X PUT http://localhost:9090/api/management/v0.9/llm-provider-templates/custom-provider \ +curl -X PUT http://localhost:9090/api/management/v1/llm-provider-templates/custom-provider \ -H "Content-Type: application/yaml" \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \ --data-binary @- <<'EOF' -apiVersion: gateway.api-platform.wso2.com/v1alpha1 +apiVersion: gateway.api-platform.wso2.com/v1 kind: LlmProviderTemplate metadata: name: custom-provider spec: displayName: Custom Provider Updated + groupId: custom-provider + managedBy: customer + version: v1.0 promptTokens: location: payload identifier: $.usage.input_tokens @@ -407,7 +467,7 @@ EOF To delete a custom template: ```bash -curl -X DELETE http://localhost:9090/api/management/v0.9/llm-provider-templates/custom-provider \ +curl -X DELETE http://localhost:9090/api/management/v1/llm-provider-templates/custom-provider \ -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" ``` @@ -415,18 +475,24 @@ curl -X DELETE http://localhost:9090/api/management/v0.9/llm-provider-templates/ ## Template Field Reference +Use this table to configure an `LlmProviderTemplate` resource. + | Field | Type | Required | Description | |-------|------|----------|-------------| -| `apiVersion` | string | Yes | API version, must be `gateway.api-platform.wso2.com/v1alpha1` | +| `apiVersion` | string | Yes | API version, must be `gateway.api-platform.wso2.com/v1` | | `kind` | string | Yes | Resource kind, must be `LlmProviderTemplate` | | `metadata.name` | string | Yes | Unique identifier for the template (used as template ID) | | `spec.displayName` | string | Yes | Human-readable name for the template | +| `spec.groupId` | string | No | Stable family identifier shared by versions of the same template; defaults to `metadata.name` | +| `spec.managedBy` | string | No | Template owner; built-in templates use `wso2`, while custom templates default to `customer` | +| `spec.version` | string | No | Template content version; defaults to `v1.0` | | `spec.promptTokens` | object | No | Configuration for extracting prompt/input token count | | `spec.completionTokens` | object | No | Configuration for extracting completion/output token count | | `spec.totalTokens` | object | No | Configuration for extracting total token count | | `spec.remainingTokens` | object | No | Configuration for extracting remaining token allowance | | `spec.requestModel` | object | No | Configuration for extracting request model identifier | | `spec.responseModel` | object | No | Configuration for extracting response model identifier | +| `spec.resourceMappings` | object | No | Resource-specific extraction overrides, such as token paths for `/responses` | ### Extraction Configuration Object @@ -434,5 +500,5 @@ Each extraction configuration object has the following structure: | Field | Type | Values | Description | |-------|------|--------|-------------| -| `location` | string | `payload`, `header`, `pathParam` | Where to extract the value from | -| `identifier` | string | - | JSONPath (for payload), header name (for header), or regex pattern (for pathParam) | +| `location` | string | `payload`, `header`, `queryParam`, `pathParam` | Where to extract the value from | +| `identifier` | string | - | JSONPath (for payload), header or query parameter name, or regex pattern (for pathParam) | diff --git a/en/docs/ai-gateway/next/overview.md b/en/docs/ai-gateway/next/overview.md index da2199c23..4ca2ff349 100644 --- a/en/docs/ai-gateway/next/overview.md +++ b/en/docs/ai-gateway/next/overview.md @@ -49,6 +49,8 @@ An LLM Provider represents a connection to an AI backend service such as OpenAI, Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend. +To connect the gateway to AWS Bedrock, see [Configure an AWS Bedrock LLM Provider](llm-proxy/configure-aws-bedrock-provider.md). The guide covers both Bedrock bearer API keys and AWS Signature Version 4 (SigV4) authentication. + ### LLM Proxy An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., `/assistant`) and can have its own policies applied. This enables: diff --git a/en/mkdocs.yml b/en/mkdocs.yml index b3eccf065..829b8ff0b 100644 --- a/en/mkdocs.yml +++ b/en/mkdocs.yml @@ -595,6 +595,11 @@ nav: - AI Gateway runtime with two CPUs: ai-gateway/next/performance/ai-gateway-runtime-with-two-cpus.md - AI Gateway runtime with four CPUs: ai-gateway/next/performance/ai-gateway-runtime-with-four-cpus.md - About this release: ai-gateway/next/about-this-release.md + - "1.2.0": + - Overview: ai-gateway/1.2.0/overview.md + - LLM Proxy: + - LLM Provider Templates: ai-gateway/1.2.0/llm-proxy/llm-templates.md + - Configure an AWS Bedrock LLM Provider: ai-gateway/1.2.0/llm-proxy/configure-aws-bedrock-provider.md - "1.1.0": - Overview: ai-gateway/1.1.0/overview.md - LLM Proxy: @@ -829,18 +834,20 @@ nav: - Policy Hub: - Overview: policy-hub/overview.md - Guides: - - AI and MCP: - - Convert a REST API into an MCP Tool and Use It in Claude Desktop: guides/ai-and-mcp/convert-rest-api-to-mcp-server.md + - LLM Proxies: - Build an AI App with Claude Code that Calls Governed Backend APIs: guides/ai-and-mcp/build-ai-app-with-claude-code.md - Set up a Governed Multi-Model LLM Proxy with Cost Controls and Failover: guides/ai-and-mcp/set-up-a-governed-multi-model-llm-proxy-with-cost-controls-and-failover.md - - Find and Connect to an Enterprise MCP Server from the MCP Hub: guides/ai-and-mcp/find-and-connect-to-an-enterprise-mcp-server-from-the-mcp-hub.md - Enforce Token-Based Rate Limiting on an LLM Proxy: guides/ai-and-mcp/enforce-token-based-rate-limiting-on-an-llm-proxy.md - - Build an AI agent That Uses Aggregated MCP Tools from Multiple APIs: guides/ai-and-mcp/build-ai-agent-with-multiple-mcp-servers.md - Enforce a Consistent AI Persona with the Prompt Decorator Policy: guides/ai-and-mcp/using-prompt-decorator-policy.md + - Configure an AWS Bedrock LLM Provider: ai-gateway/next/llm-proxy/configure-aws-bedrock-provider.md - AI Coding Assistants: - Configure Claude Code with AI Gateway: guides/ai-and-mcp/ai-coding-assistants/claude-code-configuration-with-ai-gateway.md - Configure Google Gemini CLI with AI Gateway: guides/ai-and-mcp/ai-coding-assistants/gemini-cli-configuration-with-ai-gateway.md - Configure OpenAI Codex CLI with AI Gateway: guides/ai-and-mcp/ai-coding-assistants/codex-configuration-with-ai-gateway.md + - MCP Proxies: + - Convert a REST API into an MCP Tool and Use It in Claude Desktop: guides/ai-and-mcp/convert-rest-api-to-mcp-server.md + - Find and Connect to an Enterprise MCP Server from the MCP Hub: guides/ai-and-mcp/find-and-connect-to-an-enterprise-mcp-server-from-the-mcp-hub.md + - Build an AI Agent That Uses Aggregated MCP Tools from Multiple APIs: guides/ai-and-mcp/build-ai-agent-with-multiple-mcp-servers.md - Developer Portal: - Go from Zero to a Working API Call Using the Developer Portal: guides/developer-portal/api-discovery-and-tryout.md - Monetization: