diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md
index 721c951..569664e 100644
--- a/.agents/AGENTS.md
+++ b/.agents/AGENTS.md
@@ -37,7 +37,7 @@
- Comments on fields in `api/v1alpha1/` are parsed by `controller-gen` into CRD OpenAPI schema descriptions. Keep them user-facing and precise.
4. **Update Docs on Architecture Changes**:
- - When an architecture boundary or CRD field changes, update `docs/agentrax.md` and `.agents/skills/agentrax-context/SKILL.md` in the same commit.
+ - When an architecture boundary or CRD field changes, update `docs/ARCHITECTURE.md` and `.agents/skills/agentrax-context/SKILL.md` in the same commit.
## Tooling & Developer Environment
@@ -52,16 +52,17 @@ Configure your MCP client with `--project-from-cwd` (for example, `serena start-
**Use Serena tools instead of text search for the following tasks:**
-| Task | Use instead of |
-| ---- | -------------- |
-| Find where a type, function, or constant is defined | `find_symbol` / `find_declaration` rather than `grep` |
-| Find all usages/call sites of a symbol across packages | `find_referencing_symbols` rather than `grep -r` |
-| Understand what symbols a file or package exports | `get_symbols_overview` rather than skimming the file |
-| Rename a symbol consistently across all packages | `rename_symbol` rather than manual multi-file sed |
-| Navigate to where an interface is implemented | `find_implementations` rather than text search |
-| Check diagnostics/type errors before proposing a fix | `get_diagnostics_for_file` |
+| Task | Use instead of |
+| ------------------------------------------------------ | ----------------------------------------------------- |
+| Find where a type, function, or constant is defined | `find_symbol` / `find_declaration` rather than `grep` |
+| Find all usages/call sites of a symbol across packages | `find_referencing_symbols` rather than `grep -r` |
+| Understand what symbols a file or package exports | `get_symbols_overview` rather than skimming the file |
+| Rename a symbol consistently across all packages | `rename_symbol` rather than manual multi-file sed |
+| Navigate to where an interface is implemented | `find_implementations` rather than text search |
+| Check diagnostics/type errors before proposing a fix | `get_diagnostics_for_file` |
**When NOT to use Serena:**
+
- Simple single-file reads — `view_file` is faster.
- Writing or replacing file content — use the standard edit tools.
- Searching for plain string literals (log messages, YAML values) — `grep` is fine.
diff --git a/.agents/skills/agentrax-context/SKILL.md b/.agents/skills/agentrax-context/SKILL.md
index 8c4cedd..88671d0 100644
--- a/.agents/skills/agentrax-context/SKILL.md
+++ b/.agents/skills/agentrax-context/SKILL.md
@@ -3,7 +3,9 @@ name: agentrax-context
description: Project context and settled architecture decisions for the Agentrax Kubernetes operator (module agentrax.io/v1alpha1, repo agentrax). Always consult this before writing, reviewing, or reasoning about any code in this repository — CRD types, the reconciler, the rollout controller, the autoscaler, the quota webhook, or the MCP registry — so implementation stays consistent with the design doc instead of drifting or re-deriving decisions that are already settled. Trigger on any mention of AgentDeployment, TenantQuota, canary rollout, or this repo's controllers, even if the user doesn't name the skill directly.
---
-> When uncertain about any architecture decision, defer to `docs/agentrax.md` rather than improvising. Don't guess when the doc has the answer.
+# Agentrax Context Skill
+
+> When uncertain about any architecture decision, defer to `docs/ARCHITECTURE.md` rather than improvising. Don't guess when the doc has the answer.
## Non-negotiable terminology
@@ -23,22 +25,24 @@ description: Project context and settled architecture decisions for the Agentrax
## Package map
-| Package | Responsibility |
-| ---------------------- | ----------------------------------------------------------------------------------------- |
-| `api/v1alpha1/` | CRD Go types, validation markers, defaulting. No business logic. |
-| `internal/controller/` | Reconcile loops. Only code that calls the Kubernetes API for core owned resources. |
-| `internal/rollout/` | Canary state machine and PromQL threshold evaluation. |
-| `internal/scaling/` | HPA generation and quota-capped scaling logic. |
-| `internal/registry/` | MCP registrar, registry HTTP handler, TTL sweep. |
-| `internal/quota/` | Quota arithmetic and in-flight reservation. Shared by webhook and TenantQuota reconciler. |
+| Package | Responsibility |
+| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
+| `api/v1alpha1/` | CRD Go types, validation markers, defaulting. No business logic. |
+| `internal/controller/` | Reconcile loops. Only code that calls the Kubernetes API for core-owned resources. |
+| `internal/rollout/` | Canary state machine and PromQL threshold evaluation. |
+| `internal/scaling/` | HPA generation and quota-capped scaling logic. |
+| `internal/registry/` | MCP registrar, registry HTTP handler, TTL sweep. |
+| `internal/quota/` | Quota arithmetic and in-flight reservation. Shared by webhook and TenantQuota reconciler. |
| `internal/webhook/` | Validating and mutating admission webhooks. Lives here (not `api/`) to import `internal/quota` without creating an import cycle. |
-| `internal/metrics/` | Shared Prometheus client plumbing used by rollout and scaling. |
+| `internal/metrics/` | Shared Prometheus client plumbing used by rollout and scaling. |
## Where the hard logic lives
-- **`internal/rollout/`** — never evaluate `rollback` thresholds against a sample smaller than `minRequestSample`. A 10%-weight canary at low traffic produces statistically meaningless error rates; gate on sample size first.
+- **`internal/rollout/`** — never evaluate `rollback` thresholds against a sample smaller than `minRequestSample`. A 10%-weight canary at low traffic produces statistically meaningless error rates; gate on sample size first. Canary steps must include at least one terminal `setWeight: 100` step for full promotion. Range query windows must format to canonical Prometheus syntax (`5m`, `1h`, `30s`, no trailing `0s`).
- **`internal/quota/`** — two concurrent near-limit creates can individually pass a read-then-write quota check but combined exceed it. Use an in-flight reservation (short-lived in-memory map, keyed by tenant), not a naive status read.
- **`internal/registry/`** — registration requires a successful MCP-level `initialize` handshake, not just Kubernetes readiness. Entries carry a TTL/heartbeat; ungraceful termination (OOM-kill, node failure) skips the deletion event path entirely, so don't rely on it.
+- **`internal/metrics/`** — all Prometheus HTTP responses must be read with `io.LimitReader` (1 MiB ceiling) to protect against memory exhaustion.
+- **`internal/controller/`** — reconcilers consume MCP registry operations via the `AgentRegistrar` interface (`Register`, `Deregister`, `Heartbeat`) for test isolation without polluting production structs.
- Finalizer ordering: deregister from MCP _before_ the `Service` is garbage collected. Controller-runtime's foreground deletion via finalizer is the enforcement mechanism, not best-effort.
- Quota reduction: lowering `TenantQuota` below current usage sets an `OverQuota` condition and blocks new creates/scale-ups. Never forcibly delete existing resources.
diff --git a/.coderabbit.yaml b/.coderabbit.yaml
index 145380a..8fcbfdc 100644
--- a/.coderabbit.yaml
+++ b/.coderabbit.yaml
@@ -65,7 +65,7 @@ reviews:
- Finalizer constant must be `AgentDeploymentFinalizer` = `"agentrax.io/mcp-deregister"`.
Never hardcode the string directly; always use the constant.
- Validation markers (`+kubebuilder:validation:*`) on spec fields must
- match the rules in docs/agentrax.md section 6.3. Pay special attention
+ match the rules in docs/ARCHITECTURE.md. Pay special attention
to enum values for `spec.replicas.metric` and `spec.rollout.strategy`.
# Reconciler — enforce controller-runtime patterns strictly.
@@ -222,7 +222,7 @@ reviews:
instructions: |
- Any change to a CRD field, architecture boundary, or package
responsibility must be reflected here in the same PR.
- - `docs/agentrax.md` is the source of truth for architecture decisions.
+ - `docs/ARCHITECTURE.md` is the source of truth for architecture decisions.
Flag any PR that changes architecture without updating it.
# ── Custom review instructions (global) ──────────────────────────────────
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 428f396..75c57d6 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -49,10 +49,27 @@ jobs:
name: coverage
path: cover.out
+ helm-lint:
+ name: Helm Lint & Dry-run
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Set up Helm
+ uses: azure/setup-helm@v4
+ with:
+ version: v3.14.0
+
+ - name: Lint Helm Chart
+ run: helm lint charts/agentrax/
+
+ - name: Template Helm Chart
+ run: helm template test charts/agentrax/ --debug
+
build:
name: Docker Build
runs-on: ubuntu-latest
- needs: [lint, test]
+ needs: [lint, test, helm-lint]
steps:
- uses: actions/checkout@v4
@@ -71,3 +88,27 @@ jobs:
tags: ghcr.io/gitcommitankit/agentrax:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
+
+ e2e:
+ name: End-to-End Tests (kind)
+ runs-on: ubuntu-latest
+ needs: [build]
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Set up Go
+ uses: actions/setup-go@v5
+ with:
+ go-version: "1.23"
+ cache: true
+
+ - name: Create kind Cluster
+ uses: helm/kind-action@v1.10.0
+ with:
+ cluster_name: agentrax-e2e
+
+ - name: Install cluster dependencies
+ run: make deploy-deps
+
+ - name: Run E2E Tests
+ run: make test-e2e
diff --git a/.github/workflows/soak.yml b/.github/workflows/soak.yml
new file mode 100644
index 0000000..11ceba4
--- /dev/null
+++ b/.github/workflows/soak.yml
@@ -0,0 +1,35 @@
+name: Autoscaling Soak Test
+
+on:
+ workflow_dispatch:
+ inputs:
+ soak_duration_minutes:
+ description: "Duration to run autoscaling soak in minutes"
+ default: "10"
+ required: false
+
+jobs:
+ soak:
+ name: Autoscaling Soak Test (kind)
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Set up Go
+ uses: actions/setup-go@v5
+ with:
+ go-version: "1.23"
+ cache: true
+
+ - name: Create kind cluster
+ uses: helm/kind-action@v1.10.0
+ with:
+ cluster_name: agentrax-soak
+
+ - name: Install cluster dependencies
+ run: make deploy-deps
+
+ - name: Run Autoscaling Soak Suite
+ run: make test-e2e-soak
+ env:
+ SOAK_DURATION_MINUTES: ${{ github.event.inputs.soak_duration_minutes }}
diff --git a/Makefile b/Makefile
index 9d98e18..1a7320a 100644
--- a/Makefile
+++ b/Makefile
@@ -68,6 +68,10 @@ test: manifests generate fmt vet envtest ## Run tests.
test-e2e:
go test ./test/e2e/ -v -ginkgo.v
+.PHONY: test-e2e-soak
+test-e2e-soak: ## Run the long-running autoscaling soak test suite.
+ go test ./test/e2e/... -tags e2e -v --timeout 25m -run TestE2E
+
.PHONY: lint
lint: golangci-lint ## Run golangci-lint linter
$(GOLANGCI_LINT) run
diff --git a/README.md b/README.md
index 0d0592d..57f817e 100644
--- a/README.md
+++ b/README.md
@@ -1,119 +1,324 @@
-# agentrax
+# Agentrax
+
+[](https://github.com/gitcommitankit/agentrax/actions/workflows/ci.yml)
+[](https://goreportcard.com/report/github.com/gitcommitankit/agentrax)
+[](LICENSE)
+[](https://kubernetes.io/)
+
+**Agentrax** is a declarative, cloud-native Kubernetes operator designed for managing the full lifecycle of AI and LLM Agent workloads. It delivers multi-tenant resource quota admission, metrics-driven horizontal pod autoscaling, statistically gated canary progressive rollouts with automated rollback, and native Model Context Protocol (MCP) service discovery.
+
+---
+
+## Architecture Overview
+
+```mermaid
+flowchart TB
+ subgraph ControlPlane["Kubernetes Control Plane & API Server"]
+ AD["AgentDeployment CR
agentrax.io/v1alpha1"]
+ TQ["TenantQuota CR
agentrax.io/v1alpha1"]
+ WH["Validating & Mutating
Webhook Server"]
+ end
+
+ subgraph Operator["Agentrax Controller Manager"]
+ direction TB
+ REC["AgentDeployment
Reconciler"]
+ TQC["TenantQuota
Reconciler"]
+ ROLL["Canary Rollout
Controller"]
+ AUTO["Autoscaler
Manager"]
+ REG["MCP Discovery Registry
(HTTP :9090)"]
+ end
+
+ subgraph ManagedResources["Managed Workload Resources"]
+ DEP_STABLE["Stable Deployment"]
+ DEP_CANARY["Canary Deployment"]
+ SVC["ClusterIP Service"]
+ HPA["HorizontalPodAutoscaler"]
+ HTTP_ROUTE["Gateway API HTTPRoute
(Traffic Weight Split)"]
+ SM["Prometheus ServiceMonitor"]
+ end
+
+ subgraph DiscoveryLayer["Service Discovery & Tenancy"]
+ CM["ConfigMap Store
agentrax-registry"]
+ CLIENT["External MCP Clients / Agents
GET /agents"]
+ end
+
+ AD --> WH
+ TQ --> WH
+ WH --> REC
+ WH --> TQC
+ REC --> DEP_STABLE
+ REC --> SVC
+ REC --> SM
+ REC --> AUTO
+ AUTO --> HPA
+ REC --> ROLL
+ ROLL --> DEP_CANARY
+ ROLL --> HTTP_ROUTE
+ REC --> REG
+ REG <--> CM
+ CLIENT --> REG
+```
+
+---
-// TODO(user): Add simple overview of use/purpose
+## Key Features
-## Description
+| Feature | Description | Key Mechanism |
+| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |
+| **Declarative Agent Lifecycle** | Complete deployment management, self-healing, status tracking, and graceful teardown. | `AgentDeployment` reconciler + controller runtime finalizers |
+| **Multi-Tenant Quotas** | Atomic quota admission preventing concurrent over-commit of agent instances, GPU counts, and total replicas. | In-flight reservations + Validating Webhook + `TenantQuota` reconciler |
+| **Inference Autoscaling** | Scale-out and scale-in driven by custom agent metrics (`queueDepth`, `gpuUtilization`) with quota ceiling enforcement. | Prometheus Adapter + native `HorizontalPodAutoscaler` |
+| **Statistical Canary Rollouts** | Automated progressive traffic shifting with sample-size gating and Prometheus error/latency threshold rollback. | Gateway API `HTTPRoute` weights + PromQL evaluation |
+| **Native MCP Discovery** | In-cluster registry with JSON-RPC 2.0 handshake validation, tool capability aggregation, and TTL heartbeat sweeps. | In-operator HTTP Server (`:9090`) + ConfigMap persistence + TTL sweeper |
-// TODO(user): An in-depth paragraph about your project and overview of use
+---
## Getting Started
### Prerequisites
-- go version v1.22.0+
-- docker version 17.03+.
-- kubectl version v1.11.3+.
-- Access to a Kubernetes v1.11.3+ cluster.
+- **Go**: `v1.22+`
+- **Docker**: `v20.10+`
+- **Kubernetes Cluster**: `v1.28+` (e.g., `kind`, `minikube`, or cloud provider)
+- **kubectl**: `v1.28+`
+- **Helm**: `v3.12+` (optional, for chart installation)
-### To Deploy on the cluster
+---
-**Build and push your image to the location specified by `IMG`:**
+### Quick Installation (using Kustomize)
-```sh
-make docker-build docker-push IMG=/agentrax:tag
-```
+1. **Clone the repository:**
-**NOTE:** This image ought to be published in the personal registry you specified.
-And it is required to have access to pull the image from the working environment.
-Make sure you have the proper permission to the registry if the above commands don’t work.
+ ```bash
+ git clone https://github.com/gitcommitankit/agentrax.git
+ cd agentrax
+ ```
-**Install the CRDs into the cluster:**
+2. **Install cluster dependencies** (cert-manager, Prometheus Operator, Gateway API CRDs, and Prometheus Adapter):
-```sh
-make install
-```
+ ```bash
+ make deploy-deps
+ ```
-**Deploy the Manager to the cluster with the image specified by `IMG`:**
+ Note: Prometheus Adapter installation instructions are printed by `make deploy-deps`. Follow the displayed guidance to complete the metrics pipeline setup.
-```sh
-make deploy IMG=/agentrax:tag
-```
+3. **Install Agentrax CRDs:**
+
+ ```bash
+ make install
+ ```
+
+4. **Deploy the Agentrax Controller Manager:**
+
+ ```bash
+ make deploy IMG=ghcr.io/gitcommitankit/agentrax:latest
+ ```
-> **NOTE**: If you encounter RBAC errors, you may need to grant yourself cluster-admin
-> privileges or be logged in as admin.
+5. **Verify the operator is running:**
-**Create instances of your solution**
-You can apply the samples (examples) from the config/sample:
+ ```bash
+ kubectl get pods -n agentrax-system
+ ```
-```sh
-kubectl apply -k config/samples/
+---
+
+### Installation via Helm
+
+```bash
+# Install the Helm chart
+helm install agentrax ./charts/agentrax \
+ --namespace agentrax-system \
+ --create-namespace \
+ --set prometheus.url="http://prometheus-operated.monitoring.svc:9090"
```
-> **NOTE**: Ensure that the samples has default values to test it out.
+---
-### To Uninstall
+## Usage Guide & Workload Examples
-**Delete the instances (CRs) from the cluster:**
+### 1. Define Tenant Quota
-```sh
-kubectl delete -k config/samples/
+Create a tenant quota in the target tenant namespace:
+
+```yaml
+apiVersion: agentrax.io/v1alpha1
+kind: TenantQuota
+metadata:
+ name: team-search
+ namespace: tenant-search
+spec:
+ maxAgents: 5
+ maxGPUs: 4
+ maxTotalReplicas: 15
+ maxReplicasPerAgent: 5
```
-**Delete the APIs(CRDs) from the cluster:**
+```bash
+kubectl apply -f config/samples/agentrax_v1alpha1_tenantquota.yaml
+```
-```sh
-make uninstall
+---
+
+### 2. Deploy an AI Agent with Autoscaling and MCP Discovery
+
+```yaml
+apiVersion: agentrax.io/v1alpha1
+kind: AgentDeployment
+metadata:
+ name: search-agent
+ namespace: tenant-search
+spec:
+ image: ghcr.io/my-org/search-agent:v1.0.0
+ port: 8080
+ tenantRef: team-search
+ replicas:
+ min: 1
+ max: 4
+ metric: queueDepth
+ target: 25
+ rollout:
+ strategy: Recreate
+ mcp:
+ expose: true
+ tools:
+ - webSearch
+ - documentRetriever
```
-**UnDeploy the controller from the cluster:**
+```bash
+kubectl apply -f config/samples/agentrax_v1alpha1_agentdeployment_mcp.yaml
+```
-```sh
-make undeploy
+---
+
+### 3. Progressive Canary Rollout with Automated Rollback
+
+```yaml
+apiVersion: agentrax.io/v1alpha1
+kind: AgentDeployment
+metadata:
+ name: search-agent
+ namespace: tenant-search
+spec:
+ image: ghcr.io/my-org/search-agent:v2.0.0
+ tenantRef: team-search
+ replicas:
+ min: 1
+ max: 4
+ metric: queueDepth
+ target: 25
+ rollout:
+ strategy: Canary
+ steps:
+ - setWeight: 20
+ - pause: 60s
+ - setWeight: 50
+ - pause: 120s
+ - setWeight: 100
+ rollback:
+ maxErrorRate: "1%" # Max 1% error rate
+ maxP99LatencyMs: 450 # Max 450ms P99 latency
+ minRequestSample: 100 # Minimum sample size before evaluating
```
-## Project Distribution
+---
+
+## MCP Discovery REST API
-Following are the steps to build the installer and distribute this project to users.
+The operator manager serves an in-cluster REST discovery API on port `9090` exposed by the `agentrax-registry` Service in `agentrax-system`.
-1. Build the installer for the image built and published in the registry:
+### Query Registered Agents
-```sh
-make build-installer IMG=/agentrax:tag
+```bash
+# Port-forward to local machine
+kubectl port-forward svc/agentrax-registry 9090:9090 -n agentrax-system
+
+# List all active agents
+curl -s http://localhost:9090/agents | jq .
```
-NOTE: The makefile target mentioned above generates an 'install.yaml'
-file in the dist directory. This file contains all the resources built
-with Kustomize, which are necessary to install this project without
-its dependencies.
+**Example Response:**
+
+```json
+[
+ {
+ "namespace": "tenant-search",
+ "name": "search-agent",
+ "endpoint": "http://search-agent.tenant-search.svc:8080",
+ "tools": ["webSearch", "documentRetriever", "calculator"],
+ "registeredAt": "2026-08-16T10:00:00Z",
+ "heartbeatAt": "2026-08-16T10:05:00Z",
+ "ttl": 90000000000
+ }
+]
+```
-2. Using the installer
+The `ttl` field is expressed in nanoseconds (e.g., `90000000000` = 90 seconds).
-Users can just run kubectl apply -f to install the project, i.e.:
+### Endpoints
-```sh
-kubectl apply -f https://raw.githubusercontent.com//agentrax//dist/install.yaml
-```
+| Method | Path | Description |
+| -------- | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
+| `GET` | `/agents` | List all active, non-expired registered agents. |
+| `GET` | `/agents/{namespace}/{name}` | Get details and discovered tool capabilities of a specific agent. |
+| `POST` | `/agents` | Directly register or update an agent entry (bypasses MCP handshake; for administrative use or testing, not normal operation). |
+| `DELETE` | `/agents/{namespace}/{name}` | Deregister an agent from the registry store. |
-## Contributing
+---
-// TODO(user): Add detailed information on how you would like others to contribute to this project
+## Configuration Reference
-**NOTE:** Run `make help` for more information on all potential `make` targets
+### Command-Line Arguments
-More information can be found via the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html)
+| Flag | Default | Description |
+| ----------------------------- | ------------------ | ------------------------------------------------------------------- |
+| `--metrics-bind-address` | `0` | Metrics HTTP endpoint address (`:8443` or `:8080`, `0` to disable). |
+| `--health-probe-bind-address` | `:8081` | Address for `/healthz` and `/readyz` probes. |
+| `--leader-elect` | `false` | Enable leader election for active-standby controller HA. |
+| `--registry-bind-address` | `:9090` | Address for the embedded MCP discovery HTTP server. |
+| `--gpu-resource-name` | `nvidia.com/gpu` | Resource name used for GPU quota accounting. |
+| `--prometheus-url` | `""` | Prometheus API base URL for canary metric queries. |
+| `--gateway-name` | `agentrax-gateway` | Gateway API object name for canary traffic splits. |
+| `--gateway-namespace` | `agentrax-system` | Gateway API object namespace. |
-## License
+### Environment Variables
+
+| Variable | Default | Description |
+| ------------------------------ | ------- | ----------------------------------------------------- |
+| `ENABLE_WEBHOOKS` | `true` | Set to `false` to disable admission webhook servers. |
+| `AGENTRAX_MCP_HEALTH_INTERVAL` | `30s` | Frequency of background MCP initialize health probes. |
+| `AGENTRAX_REGISTRY_TTL` | `90s` | Expiration window for unrefreshed registry entries. |
+
+---
-Copyright 2026.
+## Development & Testing
+
+### Git Commit Barriers
+
+Install traditional `pre-commit` and `pre-push` hooks:
+
+```bash
+make setup-git-hooks
+```
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
+### Run Tests and Linters
- http://www.apache.org/licenses/LICENSE-2.0
+```bash
+# Run golangci-lint
+make lint
+
+# Run all unit and integration tests (envtest)
+make test
+
+# Generate CRD manifests and DeepCopy methods
+make manifests generate
+
+# Build local manager binary
+make build
+```
+
+---
+
+## License
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
+Copyright 2026. Licensed under the [Apache License, Version 2.0](LICENSE).
diff --git a/charts/agentrax/crds/agentrax.io_agentdeployments.yaml b/charts/agentrax/crds/agentrax.io_agentdeployments.yaml
new file mode 100644
index 0000000..1282a25
--- /dev/null
+++ b/charts/agentrax/crds/agentrax.io_agentdeployments.yaml
@@ -0,0 +1,489 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+ annotations:
+ controller-gen.kubebuilder.io/version: v0.16.1
+ name: agentdeployments.agentrax.io
+spec:
+ group: agentrax.io
+ names:
+ kind: AgentDeployment
+ listKind: AgentDeploymentList
+ plural: agentdeployments
+ singular: agentdeployment
+ scope: Namespaced
+ versions:
+ - additionalPrinterColumns:
+ - jsonPath: .status.phase
+ name: Phase
+ type: string
+ - jsonPath: .status.currentReplicas
+ name: Replicas
+ type: integer
+ - jsonPath: .status.stableVersion
+ name: Stable
+ type: string
+ - jsonPath: .status.registered
+ name: Registered
+ type: boolean
+ - jsonPath: .metadata.creationTimestamp
+ name: Age
+ type: date
+ name: v1alpha1
+ schema:
+ openAPIV3Schema:
+ description: |-
+ AgentDeployment is the Schema for the agentdeployments API.
+ It manages the full lifecycle of a model or autonomous agent workload on Kubernetes,
+ including autoscaling, canary rollout, and MCP-based service discovery.
+ properties:
+ apiVersion:
+ description: |-
+ APIVersion defines the versioned schema of this representation of an object.
+ Servers should convert recognized schemas to the latest internal value, and
+ may reject unrecognized values.
+ More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+ type: string
+ kind:
+ description: |-
+ Kind is a string value representing the REST resource this object represents.
+ Servers may infer this from the endpoint the client submits requests to.
+ Cannot be updated.
+ In CamelCase.
+ More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+ type: string
+ metadata:
+ type: object
+ spec:
+ description: AgentDeploymentSpec defines the desired state of an AgentDeployment.
+ properties:
+ args:
+ description: Args overrides the container entrypoint arguments.
+ items:
+ type: string
+ type: array
+ env:
+ description: Env allows passing environment variables to the agent
+ container.
+ items:
+ description: EnvVar represents an environment variable present in
+ a Container.
+ properties:
+ name:
+ description: Name of the environment variable. Must be a C_IDENTIFIER.
+ type: string
+ value:
+ description: |-
+ Variable references $(VAR_NAME) are expanded
+ using the previously defined environment variables in the container and
+ any service environment variables. If a variable cannot be resolved,
+ the reference in the input string will be unchanged. Double $$ are reduced
+ to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e.
+ "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)".
+ Escaped references will never be expanded, regardless of whether the variable
+ exists or not.
+ Defaults to "".
+ type: string
+ valueFrom:
+ description: Source for the environment variable's value. Cannot
+ be used if value is not empty.
+ properties:
+ configMapKeyRef:
+ description: Selects a key of a ConfigMap.
+ properties:
+ key:
+ description: The key to select.
+ type: string
+ name:
+ default: ""
+ description: |-
+ Name of the referent.
+ This field is effectively required, but due to backwards compatibility is
+ allowed to be empty. Instances of this type with an empty value here are
+ almost certainly wrong.
+ More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+ type: string
+ optional:
+ description: Specify whether the ConfigMap or its key
+ must be defined
+ type: boolean
+ required:
+ - key
+ type: object
+ x-kubernetes-map-type: atomic
+ fieldRef:
+ description: |-
+ Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['']`, `metadata.annotations['']`,
+ spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs.
+ properties:
+ apiVersion:
+ description: Version of the schema the FieldPath is
+ written in terms of, defaults to "v1".
+ type: string
+ fieldPath:
+ description: Path of the field to select in the specified
+ API version.
+ type: string
+ required:
+ - fieldPath
+ type: object
+ x-kubernetes-map-type: atomic
+ resourceFieldRef:
+ description: |-
+ Selects a resource of the container: only resources limits and requests
+ (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported.
+ properties:
+ containerName:
+ description: 'Container name: required for volumes,
+ optional for env vars'
+ type: string
+ divisor:
+ anyOf:
+ - type: integer
+ - type: string
+ description: Specifies the output format of the exposed
+ resources, defaults to "1"
+ pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+ x-kubernetes-int-or-string: true
+ resource:
+ description: 'Required: resource to select'
+ type: string
+ required:
+ - resource
+ type: object
+ x-kubernetes-map-type: atomic
+ secretKeyRef:
+ description: Selects a key of a secret in the pod's namespace
+ properties:
+ key:
+ description: The key of the secret to select from. Must
+ be a valid secret key.
+ type: string
+ name:
+ default: ""
+ description: |-
+ Name of the referent.
+ This field is effectively required, but due to backwards compatibility is
+ allowed to be empty. Instances of this type with an empty value here are
+ almost certainly wrong.
+ More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+ type: string
+ optional:
+ description: Specify whether the Secret or its key must
+ be defined
+ type: boolean
+ required:
+ - key
+ type: object
+ x-kubernetes-map-type: atomic
+ type: object
+ required:
+ - name
+ type: object
+ type: array
+ image:
+ description: Image is the container image serving the model or agent.
+ minLength: 1
+ type: string
+ mcp:
+ description: MCP controls whether this agent registers itself for
+ MCP-based discovery.
+ properties:
+ expose:
+ description: Expose, when true, registers this agent in the MCP
+ registry upon reaching a stable state.
+ type: boolean
+ tools:
+ description: Tools is the list of tool names advertised by this
+ agent in the MCP registry.
+ items:
+ type: string
+ type: array
+ type: object
+ port:
+ default: 8080
+ description: Port is the container port the model/agent listens on.
+ format: int32
+ maximum: 65535
+ minimum: 1
+ type: integer
+ replicas:
+ description: Replicas defines the autoscaling policy.
+ properties:
+ max:
+ description: Max is the maximum number of replicas. Must be at
+ least 1.
+ format: int32
+ minimum: 1
+ type: integer
+ metric:
+ description: Metric selects the custom metric used for autoscaling
+ decisions.
+ enum:
+ - queueDepth
+ - gpuUtilization
+ type: string
+ min:
+ description: Min is the minimum number of replicas. Must be at
+ least 1.
+ format: int32
+ minimum: 1
+ type: integer
+ target:
+ description: Target is the desired value of the chosen metric
+ per replica.
+ format: int32
+ minimum: 1
+ type: integer
+ required:
+ - max
+ - metric
+ - min
+ - target
+ type: object
+ resources:
+ description: Resources are passed through to the underlying pod template.
+ properties:
+ claims:
+ description: |-
+ Claims lists the names of resources, defined in spec.resourceClaims,
+ that are used by this container.
+
+ This is an alpha field and requires enabling the
+ DynamicResourceAllocation feature gate.
+
+ This field is immutable. It can only be set for containers.
+ items:
+ description: ResourceClaim references one entry in PodSpec.ResourceClaims.
+ properties:
+ name:
+ description: |-
+ Name must match the name of one entry in pod.spec.resourceClaims of
+ the Pod where this field is used. It makes that resource available
+ inside a container.
+ type: string
+ request:
+ description: |-
+ Request is the name chosen for a request in the referenced claim.
+ If empty, everything from the claim is made available, otherwise
+ only the result of this request.
+ type: string
+ required:
+ - name
+ type: object
+ type: array
+ x-kubernetes-list-map-keys:
+ - name
+ x-kubernetes-list-type: map
+ limits:
+ additionalProperties:
+ anyOf:
+ - type: integer
+ - type: string
+ pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+ x-kubernetes-int-or-string: true
+ description: |-
+ Limits describes the maximum amount of compute resources allowed.
+ More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
+ type: object
+ requests:
+ additionalProperties:
+ anyOf:
+ - type: integer
+ - type: string
+ pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+ x-kubernetes-int-or-string: true
+ description: |-
+ Requests describes the minimum amount of compute resources required.
+ If Requests is omitted for a container, it defaults to Limits if that is explicitly specified,
+ otherwise to an implementation-defined value. Requests cannot exceed Limits.
+ More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
+ type: object
+ type: object
+ rollout:
+ description: Rollout defines how new versions are shipped. Optional;
+ defaults to Recreate.
+ properties:
+ abort:
+ description: Abort, when set to true, triggers an immediate rollback
+ of any in-progress canary.
+ type: boolean
+ rollback:
+ description: |-
+ Rollback defines the automatic rollback thresholds.
+ Required when strategy is Canary.
+ properties:
+ maxErrorRate:
+ description: MaxErrorRate is the maximum acceptable error
+ rate, expressed as a percentage string (e.g. "2%").
+ type: string
+ maxP99LatencyMs:
+ description: MaxP99LatencyMs is the maximum acceptable p99
+ latency in milliseconds.
+ format: int32
+ minimum: 1
+ type: integer
+ minRequestSample:
+ description: |-
+ MinRequestSample is the minimum number of requests that must be observed before
+ threshold evaluation occurs. Guards against false positives at low traffic weight.
+ format: int32
+ minimum: 1
+ type: integer
+ type: object
+ steps:
+ description: |-
+ Steps lists the ordered canary traffic-shift and pause steps.
+ Required when strategy is Canary.
+ items:
+ description: |-
+ CanaryStep represents a single step in a canary rollout.
+ Exactly one of SetWeight or Pause must be set per step.
+ properties:
+ pause:
+ description: |-
+ Pause is the duration to wait before evaluating rollback thresholds.
+ Mutually exclusive with SetWeight.
+ type: string
+ setWeight:
+ description: SetWeight is the percentage of traffic to send
+ to the canary (0–100).
+ format: int32
+ maximum: 100
+ minimum: 0
+ type: integer
+ type: object
+ type: array
+ strategy:
+ default: Recreate
+ description: Strategy determines the rollout approach. Defaults
+ to Recreate.
+ enum:
+ - Recreate
+ - Canary
+ type: string
+ type: object
+ tenantRef:
+ description: TenantRef names the owning TenantQuota object in the
+ same namespace.
+ minLength: 1
+ type: string
+ required:
+ - image
+ - replicas
+ - tenantRef
+ type: object
+ status:
+ description: AgentDeploymentStatus defines the observed state of an AgentDeployment.
+ properties:
+ canaryStepIndex:
+ description: |-
+ CanaryStepIndex is the index of the currently executing rollout step.
+ Persisted so the state machine survives operator restarts.
+ type: integer
+ canaryVersion:
+ description: CanaryVersion is the container image tag of the canary
+ deployment, if one is in progress.
+ type: string
+ canaryWeight:
+ description: CanaryWeight is the current percentage of traffic routed
+ to the canary (0–100).
+ format: int32
+ type: integer
+ conditions:
+ description: Conditions holds the conditions for the AgentDeployment.
+ items:
+ description: Condition contains details for one aspect of the current
+ state of this API Resource.
+ properties:
+ lastTransitionTime:
+ description: |-
+ lastTransitionTime is the last time the condition transitioned from one status to another.
+ This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
+ format: date-time
+ type: string
+ message:
+ description: |-
+ message is a human readable message indicating details about the transition.
+ This may be an empty string.
+ maxLength: 32768
+ type: string
+ observedGeneration:
+ description: |-
+ observedGeneration represents the .metadata.generation that the condition was set based upon.
+ For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+ with respect to the current state of the instance.
+ format: int64
+ minimum: 0
+ type: integer
+ reason:
+ description: |-
+ reason contains a programmatic identifier indicating the reason for the condition's last transition.
+ Producers of specific condition types may define expected values and meanings for this field,
+ and whether the values are considered a guaranteed API.
+ The value should be a CamelCase string.
+ This field may not be empty.
+ maxLength: 1024
+ minLength: 1
+ pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+ type: string
+ status:
+ description: status of the condition, one of True, False, Unknown.
+ enum:
+ - "True"
+ - "False"
+ - Unknown
+ type: string
+ type:
+ description: type of condition in CamelCase or in foo.example.com/CamelCase.
+ maxLength: 316
+ pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+ type: string
+ required:
+ - lastTransitionTime
+ - message
+ - reason
+ - status
+ - type
+ type: object
+ type: array
+ currentReplicas:
+ description: CurrentReplicas is the number of replicas currently running.
+ format: int32
+ type: integer
+ pauseStartedAt:
+ description: |-
+ PauseStartedAt records when the current pause step began.
+ Used to enforce maximum pause extensions and the fail-safe rollback timeout.
+ format: date-time
+ type: string
+ phase:
+ description: Phase is the high-level lifecycle phase of this deployment.
+ enum:
+ - Pending
+ - Running
+ - RolloutInProgress
+ - RolloutFailed
+ - Degraded
+ type: string
+ promUnreachableSince:
+ description: |-
+ PromUnreachableSince records when Prometheus last became unreachable.
+ When non-nil and age exceeds FailSafeTimeout, a fail-safe rollback fires.
+ format: date-time
+ type: string
+ registered:
+ description: Registered is true when this agent is currently registered
+ in the MCP registry.
+ type: boolean
+ stableVersion:
+ description: StableVersion is the container image tag of the currently
+ stable deployment.
+ type: string
+ type: object
+ type: object
+ served: true
+ storage: true
+ subresources:
+ status: {}
diff --git a/charts/agentrax/crds/agentrax.io_tenantquotas.yaml b/charts/agentrax/crds/agentrax.io_tenantquotas.yaml
new file mode 100644
index 0000000..19b0827
--- /dev/null
+++ b/charts/agentrax/crds/agentrax.io_tenantquotas.yaml
@@ -0,0 +1,173 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+ annotations:
+ controller-gen.kubebuilder.io/version: v0.16.1
+ name: tenantquotas.agentrax.io
+spec:
+ group: agentrax.io
+ names:
+ kind: TenantQuota
+ listKind: TenantQuotaList
+ plural: tenantquotas
+ singular: tenantquota
+ scope: Namespaced
+ versions:
+ - additionalPrinterColumns:
+ - jsonPath: .spec.maxAgents
+ name: MaxAgents
+ type: integer
+ - jsonPath: .status.usedAgents
+ name: UsedAgents
+ type: integer
+ - jsonPath: .spec.maxGPUs
+ name: MaxGPUs
+ type: integer
+ - jsonPath: .status.usedGPUs
+ name: UsedGPUs
+ type: integer
+ - jsonPath: .metadata.creationTimestamp
+ name: Age
+ type: date
+ name: v1alpha1
+ schema:
+ openAPIV3Schema:
+ description: |-
+ TenantQuota is the Schema for the tenantquotas API.
+ It declares and enforces per-tenant resource ceilings for AgentDeployment objects.
+ properties:
+ apiVersion:
+ description: |-
+ APIVersion defines the versioned schema of this representation of an object.
+ Servers should convert recognized schemas to the latest internal value, and
+ may reject unrecognized values.
+ More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+ type: string
+ kind:
+ description: |-
+ Kind is a string value representing the REST resource this object represents.
+ Servers may infer this from the endpoint the client submits requests to.
+ Cannot be updated.
+ In CamelCase.
+ More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+ type: string
+ metadata:
+ type: object
+ spec:
+ description: TenantQuotaSpec defines resource ceilings for a tenant.
+ properties:
+ maxAgents:
+ description: MaxAgents is the maximum number of AgentDeployment objects
+ allowed in this tenant.
+ format: int32
+ minimum: 1
+ type: integer
+ maxGPUs:
+ description: |-
+ MaxGPUs is the total number of GPU units that can be allocated across all agents.
+ GPU count is derived from spec.resources.limits["nvidia.com/gpu"] × spec.replicas.max.
+ format: int32
+ minimum: 0
+ type: integer
+ maxReplicasPerAgent:
+ description: MaxReplicasPerAgent is the maximum spec.replicas.max
+ value any single agent may request.
+ format: int32
+ minimum: 1
+ type: integer
+ maxTotalReplicas:
+ description: MaxTotalReplicas is the ceiling on the sum of spec.replicas.max
+ across all agents in this tenant.
+ format: int32
+ minimum: 1
+ type: integer
+ required:
+ - maxAgents
+ - maxGPUs
+ - maxReplicasPerAgent
+ - maxTotalReplicas
+ type: object
+ status:
+ description: TenantQuotaStatus reports current usage against the spec
+ ceilings.
+ properties:
+ conditions:
+ description: Conditions holds the conditions for the TenantQuota (e.g.
+ OverQuota).
+ items:
+ description: Condition contains details for one aspect of the current
+ state of this API Resource.
+ properties:
+ lastTransitionTime:
+ description: |-
+ lastTransitionTime is the last time the condition transitioned from one status to another.
+ This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
+ format: date-time
+ type: string
+ message:
+ description: |-
+ message is a human readable message indicating details about the transition.
+ This may be an empty string.
+ maxLength: 32768
+ type: string
+ observedGeneration:
+ description: |-
+ observedGeneration represents the .metadata.generation that the condition was set based upon.
+ For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+ with respect to the current state of the instance.
+ format: int64
+ minimum: 0
+ type: integer
+ reason:
+ description: |-
+ reason contains a programmatic identifier indicating the reason for the condition's last transition.
+ Producers of specific condition types may define expected values and meanings for this field,
+ and whether the values are considered a guaranteed API.
+ The value should be a CamelCase string.
+ This field may not be empty.
+ maxLength: 1024
+ minLength: 1
+ pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+ type: string
+ status:
+ description: status of the condition, one of True, False, Unknown.
+ enum:
+ - "True"
+ - "False"
+ - Unknown
+ type: string
+ type:
+ description: type of condition in CamelCase or in foo.example.com/CamelCase.
+ maxLength: 316
+ pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+ type: string
+ required:
+ - lastTransitionTime
+ - message
+ - reason
+ - status
+ - type
+ type: object
+ type: array
+ usedAgents:
+ description: UsedAgents is the number of AgentDeployment objects currently
+ in this tenant.
+ format: int32
+ type: integer
+ usedGPUs:
+ description: UsedGPUs is the total GPU units currently allocated across
+ all agents in this tenant.
+ format: int32
+ type: integer
+ usedTotalReplicas:
+ description: UsedTotalReplicas is the sum of spec.replicas.max across
+ all agents currently in this tenant.
+ format: int32
+ type: integer
+ type: object
+ type: object
+ served: true
+ storage: true
+ subresources:
+ status: {}
diff --git a/charts/agentrax/templates/_helpers.tpl b/charts/agentrax/templates/_helpers.tpl
new file mode 100644
index 0000000..3d2c73d
--- /dev/null
+++ b/charts/agentrax/templates/_helpers.tpl
@@ -0,0 +1,63 @@
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "agentrax.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+If release name contains chart name it will be used as a full name.
+*/}}
+{{- define "agentrax.fullname" -}}
+{{- if .Values.fullnameOverride }}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- $name := default .Chart.Name .Values.nameOverride }}
+{{- if contains $name .Release.Name }}
+{{- .Release.Name | trunc 63 | trimSuffix "-" }}
+{{- else }}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
+{{- end }}
+{{- end }}
+{{- end }}
+
+{{/*
+Create chart name and version as used by the chart label.
+*/}}
+{{- define "agentrax.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
+{{- end }}
+
+{{/*
+Common labels
+*/}}
+{{- define "agentrax.labels" -}}
+helm.sh/chart: {{ include "agentrax.chart" . }}
+{{ include "agentrax.selectorLabels" . }}
+{{- if .Chart.AppVersion }}
+app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
+{{- end }}
+app.kubernetes.io/managed-by: {{ .Release.Service }}
+{{- end }}
+
+{{/*
+Selector labels
+*/}}
+{{- define "agentrax.selectorLabels" -}}
+app.kubernetes.io/name: {{ include "agentrax.name" . }}
+app.kubernetes.io/instance: {{ .Release.Name }}
+control-plane: controller-manager
+{{- end }}
+
+{{/*
+Create the name of the service account to use
+*/}}
+{{- define "agentrax.serviceAccountName" -}}
+{{- if .Values.serviceAccount.create }}
+{{- default (include "agentrax.fullname" .) .Values.serviceAccount.name }}
+{{- else }}
+{{- default "default" .Values.serviceAccount.name }}
+{{- end }}
+{{- end }}
diff --git a/charts/agentrax/templates/clusterrole.yaml b/charts/agentrax/templates/clusterrole.yaml
new file mode 100644
index 0000000..2c87021
--- /dev/null
+++ b/charts/agentrax/templates/clusterrole.yaml
@@ -0,0 +1,136 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+ name: {{ include "agentrax.fullname" . }}-manager-role
+ labels:
+ {{- include "agentrax.labels" . | nindent 4 }}
+rules:
+- apiGroups:
+ - ""
+ resources:
+ - configmaps
+ verbs:
+ - create
+ - get
+ - list
+ - patch
+ - update
+ - watch
+- apiGroups:
+ - agentrax.io
+ resources:
+ - agentdeployments
+ verbs:
+ - create
+ - delete
+ - get
+ - list
+ - patch
+ - update
+ - watch
+- apiGroups:
+ - agentrax.io
+ resources:
+ - agentdeployments/finalizers
+ verbs:
+ - update
+- apiGroups:
+ - agentrax.io
+ resources:
+ - agentdeployments/status
+ - tenantquotas/status
+ verbs:
+ - get
+ - patch
+ - update
+- apiGroups:
+ - agentrax.io
+ resources:
+ - tenantquotas
+ verbs:
+ - get
+ - list
+ - watch
+- apiGroups:
+ - apiextensions.k8s.io
+ resources:
+ - customresourcedefinitions
+ verbs:
+ - get
+ - list
+ - watch
+- apiGroups:
+ - apps
+ resources:
+ - deployments
+ verbs:
+ - create
+ - delete
+ - get
+ - list
+ - patch
+ - update
+ - watch
+- apiGroups:
+ - autoscaling
+ resources:
+ - horizontalpodautoscalers
+ verbs:
+ - create
+ - delete
+ - get
+ - list
+ - patch
+ - update
+ - watch
+- apiGroups:
+ - ""
+ resources:
+ - events
+ verbs:
+ - create
+ - patch
+- apiGroups:
+ - ""
+ resources:
+ - pods
+ verbs:
+ - get
+ - list
+ - watch
+- apiGroups:
+ - ""
+ resources:
+ - services
+ verbs:
+ - create
+ - delete
+ - get
+ - list
+ - patch
+ - update
+ - watch
+- apiGroups:
+ - gateway.networking.k8s.io
+ resources:
+ - httproutes
+ verbs:
+ - create
+ - delete
+ - get
+ - list
+ - patch
+ - update
+ - watch
+- apiGroups:
+ - monitoring.coreos.com
+ resources:
+ - servicemonitors
+ verbs:
+ - create
+ - delete
+ - get
+ - list
+ - patch
+ - update
+ - watch
diff --git a/charts/agentrax/templates/clusterrolebinding.yaml b/charts/agentrax/templates/clusterrolebinding.yaml
new file mode 100644
index 0000000..288b4b7
--- /dev/null
+++ b/charts/agentrax/templates/clusterrolebinding.yaml
@@ -0,0 +1,14 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+ name: {{ include "agentrax.fullname" . }}-manager-rolebinding
+ labels:
+ {{- include "agentrax.labels" . | nindent 4 }}
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: {{ include "agentrax.fullname" . }}-manager-role
+subjects:
+- kind: ServiceAccount
+ name: {{ include "agentrax.serviceAccountName" . }}
+ namespace: {{ .Release.Namespace }}
diff --git a/charts/agentrax/templates/deployment.yaml b/charts/agentrax/templates/deployment.yaml
new file mode 100644
index 0000000..e99bffb
--- /dev/null
+++ b/charts/agentrax/templates/deployment.yaml
@@ -0,0 +1,109 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ include "agentrax.fullname" . }}
+ namespace: {{ .Release.Namespace }}
+ labels:
+ {{- include "agentrax.labels" . | nindent 4 }}
+spec:
+ replicas: {{ .Values.replicaCount }}
+ selector:
+ matchLabels:
+ {{- include "agentrax.selectorLabels" . | nindent 6 }}
+ template:
+ metadata:
+ annotations:
+ kubectl.kubernetes.io/default-container: manager
+ {{- with .Values.podAnnotations }}
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ labels:
+ {{- include "agentrax.selectorLabels" . | nindent 8 }}
+ spec:
+ {{- with .Values.imagePullSecrets }}
+ imagePullSecrets:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ serviceAccountName: {{ include "agentrax.serviceAccountName" . }}
+ securityContext:
+ {{- toYaml .Values.podSecurityContext | nindent 8 }}
+ terminationGracePeriodSeconds: 30
+ containers:
+ - name: manager
+ securityContext:
+ {{- toYaml .Values.securityContext | nindent 12 }}
+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+ command:
+ - /manager
+ args:
+ {{- if .Values.manager.leaderElect }}
+ - --leader-elect
+ {{- end }}
+ - --health-probe-bind-address=:{{ .Values.manager.healthProbeBindPort | default 8081 }}
+ - --metrics-bind-address={{ .Values.manager.metricsBindAddress | default "0" }}
+ {{- if hasKey .Values.manager "metricsSecure" }}
+ - --metrics-secure={{ .Values.manager.metricsSecure }}
+ {{- end }}
+ - --registry-bind-address=:{{ .Values.registry.bindPort | default 9090 }}
+ - --gpu-resource-name={{ .Values.manager.gpuResourceName | default "nvidia.com/gpu" }}
+ {{- if .Values.prometheus.url }}
+ - --prometheus-url={{ .Values.prometheus.url }}
+ {{- end }}
+ {{- if .Values.gateway.name }}
+ - --gateway-name={{ .Values.gateway.name }}
+ {{- end }}
+ {{- if .Values.gateway.namespace }}
+ - --gateway-namespace={{ .Values.gateway.namespace }}
+ {{- end }}
+ env:
+ - name: POD_NAMESPACE
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.namespace
+ - name: AGENTRAX_REGISTRY_TTL
+ value: {{ .Values.registry.ttl | default "90s" | quote }}
+ - name: AGENTRAX_MCP_HEALTH_INTERVAL
+ value: {{ .Values.mcp.healthInterval | default "30s" | quote }}
+ {{- range $key, $val := .Values.env }}
+ - name: {{ $key }}
+ value: {{ $val | quote }}
+ {{- end }}
+ ports:
+ - name: registry
+ containerPort: {{ .Values.registry.bindPort | default 9090 }}
+ protocol: TCP
+ - name: health
+ containerPort: {{ .Values.manager.healthProbeBindPort | default 8081 }}
+ protocol: TCP
+ {{- if and .Values.manager.metricsBindAddress (ne .Values.manager.metricsBindAddress "0") }}
+ - name: {{ if eq (toString .Values.manager.metricsSecure) "false" }}http{{ else }}https{{ end }}
+ containerPort: {{ if eq .Values.manager.metricsBindAddress ":8080" }}8080{{ else if eq .Values.manager.metricsBindAddress ":8443" }}8443{{ else if eq (toString .Values.manager.metricsSecure) "false" }}8080{{ else }}8443{{ end }}
+ protocol: TCP
+ {{- end }}
+ livenessProbe:
+ httpGet:
+ path: /healthz
+ port: {{ .Values.manager.healthProbeBindPort | default 8081 }}
+ initialDelaySeconds: 15
+ periodSeconds: 20
+ readinessProbe:
+ httpGet:
+ path: /readyz
+ port: {{ .Values.manager.healthProbeBindPort | default 8081 }}
+ initialDelaySeconds: 5
+ periodSeconds: 10
+ resources:
+ {{- toYaml .Values.resources | nindent 12 }}
+ {{- with .Values.nodeSelector }}
+ nodeSelector:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ {{- with .Values.affinity }}
+ affinity:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
+ {{- with .Values.tolerations }}
+ tolerations:
+ {{- toYaml . | nindent 8 }}
+ {{- end }}
diff --git a/charts/agentrax/templates/leader-election-role.yaml b/charts/agentrax/templates/leader-election-role.yaml
new file mode 100644
index 0000000..06f3c3a
--- /dev/null
+++ b/charts/agentrax/templates/leader-election-role.yaml
@@ -0,0 +1,39 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+ name: {{ include "agentrax.fullname" . }}-leader-election-role
+ namespace: {{ .Release.Namespace }}
+ labels:
+ {{- include "agentrax.labels" . | nindent 4 }}
+rules:
+- apiGroups:
+ - ""
+ resources:
+ - configmaps
+ verbs:
+ - get
+ - list
+ - watch
+ - create
+ - update
+ - patch
+ - delete
+- apiGroups:
+ - coordination.k8s.io
+ resources:
+ - leases
+ verbs:
+ - get
+ - list
+ - watch
+ - create
+ - update
+ - patch
+ - delete
+- apiGroups:
+ - ""
+ resources:
+ - events
+ verbs:
+ - create
+ - patch
diff --git a/charts/agentrax/templates/leader-election-rolebinding.yaml b/charts/agentrax/templates/leader-election-rolebinding.yaml
new file mode 100644
index 0000000..2e976a0
--- /dev/null
+++ b/charts/agentrax/templates/leader-election-rolebinding.yaml
@@ -0,0 +1,15 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+ name: {{ include "agentrax.fullname" . }}-leader-election-rolebinding
+ namespace: {{ .Release.Namespace }}
+ labels:
+ {{- include "agentrax.labels" . | nindent 4 }}
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: Role
+ name: {{ include "agentrax.fullname" . }}-leader-election-role
+subjects:
+- kind: ServiceAccount
+ name: {{ include "agentrax.serviceAccountName" . }}
+ namespace: {{ .Release.Namespace }}
diff --git a/charts/agentrax/templates/metrics-service.yaml b/charts/agentrax/templates/metrics-service.yaml
new file mode 100644
index 0000000..b707e4a
--- /dev/null
+++ b/charts/agentrax/templates/metrics-service.yaml
@@ -0,0 +1,18 @@
+{{- if and .Values.manager.metricsBindAddress (ne .Values.manager.metricsBindAddress "0") -}}
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "agentrax.fullname" . }}-metrics-service
+ namespace: {{ .Release.Namespace }}
+ labels:
+ {{- include "agentrax.labels" . | nindent 4 }}
+spec:
+ type: ClusterIP
+ selector:
+ {{- include "agentrax.selectorLabels" . | nindent 4 }}
+ ports:
+ - name: {{ if eq (toString .Values.manager.metricsSecure) "false" }}http{{ else }}https{{ end }}
+ port: {{ if eq .Values.manager.metricsBindAddress ":8080" }}8080{{ else if eq .Values.manager.metricsBindAddress ":8443" }}8443{{ else if eq (toString .Values.manager.metricsSecure) "false" }}8080{{ else }}8443{{ end }}
+ targetPort: {{ if eq .Values.manager.metricsBindAddress ":8080" }}8080{{ else if eq .Values.manager.metricsBindAddress ":8443" }}8443{{ else if eq (toString .Values.manager.metricsSecure) "false" }}8080{{ else }}8443{{ end }}
+ protocol: TCP
+{{- end }}
diff --git a/charts/agentrax/templates/registry-service.yaml b/charts/agentrax/templates/registry-service.yaml
new file mode 100644
index 0000000..8fd8dc0
--- /dev/null
+++ b/charts/agentrax/templates/registry-service.yaml
@@ -0,0 +1,17 @@
+apiVersion: v1
+kind: Service
+metadata:
+ name: agentrax-registry
+ namespace: {{ .Release.Namespace }}
+ labels:
+ {{- include "agentrax.labels" . | nindent 4 }}
+ app.kubernetes.io/component: registry
+spec:
+ type: {{ .Values.registry.service.type | default "ClusterIP" }}
+ selector:
+ {{- include "agentrax.selectorLabels" . | nindent 4 }}
+ ports:
+ - name: registry
+ port: {{ .Values.registry.service.port | default 9090 }}
+ targetPort: {{ .Values.registry.bindPort | default 9090 }}
+ protocol: TCP
diff --git a/charts/agentrax/templates/serviceaccount.yaml b/charts/agentrax/templates/serviceaccount.yaml
new file mode 100644
index 0000000..cb0618a
--- /dev/null
+++ b/charts/agentrax/templates/serviceaccount.yaml
@@ -0,0 +1,13 @@
+{{- if .Values.serviceAccount.create -}}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: {{ include "agentrax.serviceAccountName" . }}
+ namespace: {{ .Release.Namespace }}
+ labels:
+ {{- include "agentrax.labels" . | nindent 4 }}
+ {{- with .Values.serviceAccount.annotations }}
+ annotations:
+ {{- toYaml . | nindent 4 }}
+ {{- end }}
+{{- end }}
diff --git a/charts/agentrax/templates/tests/test-smoke.yaml b/charts/agentrax/templates/tests/test-smoke.yaml
new file mode 100644
index 0000000..9ab66d3
--- /dev/null
+++ b/charts/agentrax/templates/tests/test-smoke.yaml
@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: Pod
+metadata:
+ name: "{{ include "agentrax.fullname" . }}-test-connection"
+ labels:
+ {{- include "agentrax.labels" . | nindent 4 }}
+ annotations:
+ "helm.sh/hook": test
+ "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
+spec:
+ containers:
+ - name: wget
+ image: busybox:1.36
+ command: ['wget']
+ args: ['-qO-', 'http://agentrax-registry:9090/agents']
+ restartPolicy: Never
diff --git a/charts/agentrax/values.yaml b/charts/agentrax/values.yaml
index 1f59dc8..1efa56d 100644
--- a/charts/agentrax/values.yaml
+++ b/charts/agentrax/values.yaml
@@ -31,6 +31,8 @@ podAnnotations: {}
# -- Pod security context.
podSecurityContext:
runAsNonRoot: true
+ seccompProfile:
+ type: RuntimeDefault
# -- Container security context.
securityContext:
@@ -40,8 +42,6 @@ securityContext:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
- seccompProfile:
- type: RuntimeDefault
resources:
limits:
@@ -60,20 +60,50 @@ tolerations: []
# -- Affinity rules for the manager pod.
affinity: {}
-# -- Manager flags
+# -- Manager flags & options
manager:
# -- Enable leader election (set to true for HA / multiple replicas).
leaderElect: false
- # -- Bind address for metrics. Use ":8443" for HTTPS, ":8080" for HTTP.
+ # -- Bind address for metrics. Use ":8443" for HTTPS, ":8080" for HTTP, "0" to disable.
metricsBindAddress: "0"
- # -- Bind address for health/readiness probes.
- healthProbeBindAddress: ":8081"
+ # -- Serve metrics securely over HTTPS. Set to false for HTTP.
+ metricsSecure: true
+ # -- Bind port for health/readiness probes (used for container port, probes, and manager bind address).
+ healthProbeBindPort: 8081
# -- GPU resource name used for quota calculation.
gpuResourceName: "nvidia.com/gpu"
+# -- Prometheus configuration for canary rollout metrics
+prometheus:
+ # -- Prometheus HTTP API endpoint (e.g. "http://prometheus-operated.monitoring.svc:9090").
+ # If empty, canary rollout threshold evaluations will fall back to recreate strategy.
+ url: ""
+
+# -- Gateway API configuration for canary traffic splitting
+gateway:
+ # -- Name of the Gateway API Gateway object.
+ name: "agentrax-gateway"
+ # -- Namespace of the Gateway API Gateway object.
+ namespace: "agentrax-system"
+
+# -- MCP discovery registry configuration
+registry:
+ # -- Service configuration
+ service:
+ # -- Registry Service port
+ port: 9090
+ # -- Registry Service type
+ type: ClusterIP
+ # -- Bind port for the registry HTTP server (used for container port, service targetPort, and manager bind address).
+ bindPort: 9090
+ # -- Registry entry TTL (e.g. "90s")
+ ttl: "90s"
+
+# -- MCP client configuration
+mcp:
+ # -- Periodic health check interval for registered agents (e.g. "30s")
+ healthInterval: "30s"
+
# -- Environment variables injected into the manager container.
-env:
- # -- MCP health-check interval (e.g. "30s").
- AGENTRAX_MCP_HEALTH_INTERVAL: "30s"
- # -- Registry entry TTL (e.g. "90s").
- AGENTRAX_REGISTRY_TTL: "90s"
+# These values are automatically populated from registry.ttl and mcp.healthInterval above.
+env: {}
diff --git a/cmd/main.go b/cmd/main.go
index 376fc5b..6e1d2e8 100644
--- a/cmd/main.go
+++ b/cmd/main.go
@@ -147,7 +147,7 @@ func main() {
flag.StringVar(®istryAddr, "registry-bind-address", ":9090",
"The address the MCP discovery registry HTTP endpoint binds to.")
opts := zap.Options{
- Development: true,
+ Development: false,
}
opts.BindFlags(flag.CommandLine)
flag.Parse()
@@ -169,10 +169,15 @@ func main() {
tlsOpts = append(tlsOpts, disableHTTP2)
}
- // Resolve the webhook-enabled flag once so both the server creation and
- // handler registration use the same value. Log it explicitly so operators
- // can confirm the resolved state at startup.
- enableWebhooks := os.Getenv("ENABLE_WEBHOOKS") != "false"
+ // Resolve the webhook-enabled flag. Enable webhook server only if explicitly set
+ // to "true" or if TLS certificates are present in /tmp/k8s-webhook-server/serving-certs.
+ enableWebhooks := os.Getenv("ENABLE_WEBHOOKS") == "true"
+ if _, err := os.Stat("/tmp/k8s-webhook-server/serving-certs/tls.crt"); err == nil {
+ enableWebhooks = true
+ }
+ if os.Getenv("ENABLE_WEBHOOKS") == "false" {
+ enableWebhooks = false
+ }
setupLog.Info("webhook state resolved", "enabled", enableWebhooks)
var webhookServer webhook.Server
@@ -212,6 +217,45 @@ func main() {
registryNamespace = "agentrax-system"
}
+ registryTTL := registry.DefaultTTL
+ if v := os.Getenv("AGENTRAX_REGISTRY_TTL"); v != "" {
+ parsed, err := time.ParseDuration(v)
+ if err != nil {
+ setupLog.Error(err, "failed to parse AGENTRAX_REGISTRY_TTL, using default",
+ "value", v, "default", registry.DefaultTTL)
+ } else if parsed < time.Second {
+ setupLog.Error(errors.New("duration must be at least 1s"),
+ "sub-second AGENTRAX_REGISTRY_TTL rejected, using default",
+ "value", v, "default", registry.DefaultTTL)
+ } else {
+ registryTTL = parsed
+ }
+ }
+
+ mcpHealthInterval := 60 * time.Second
+ if v := os.Getenv("AGENTRAX_MCP_HEALTH_INTERVAL"); v != "" {
+ parsed, err := time.ParseDuration(v)
+ if err != nil {
+ setupLog.Error(err, "failed to parse AGENTRAX_MCP_HEALTH_INTERVAL, using default",
+ "value", v, "default", mcpHealthInterval)
+ } else if parsed < time.Second {
+ setupLog.Error(errors.New("duration must be at least 1s"),
+ "sub-second AGENTRAX_MCP_HEALTH_INTERVAL rejected, using default",
+ "value", v, "default", mcpHealthInterval)
+ } else {
+ mcpHealthInterval = parsed
+ }
+ }
+
+ if mcpHealthInterval >= registryTTL {
+ adjusted := registryTTL / 2
+ setupLog.Info("AGENTRAX_MCP_HEALTH_INTERVAL must be strictly less than AGENTRAX_REGISTRY_TTL",
+ "configuredInterval", mcpHealthInterval,
+ "registryTTL", registryTTL,
+ "adjustedInterval", adjusted)
+ mcpHealthInterval = adjusted
+ }
+
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
Metrics: metricsServerOptions,
@@ -267,7 +311,7 @@ func main() {
}
// Initialize MCP discovery registry and registrar.
- mcpRegistry := registry.NewRegistry(mgr.GetClient(), registryNamespace, registry.DefaultTTL)
+ mcpRegistry := registry.NewRegistry(mgr.GetClient(), registryNamespace, registryTTL)
mcpRegistrar := registry.NewRegistrar(mcpRegistry, registry.NewHTTPMCPClient())
if registryAddr != "" && registryAddr != "0" {
@@ -282,11 +326,12 @@ func main() {
}
agentDeploymentReconciler := &controller.AgentDeploymentReconciler{
- Client: mgr.GetClient(),
- Scheme: mgr.GetScheme(),
- GPUResourceName: gpuResourceName,
- CanaryController: canaryController,
- Registrar: mcpRegistrar,
+ Client: mgr.GetClient(),
+ Scheme: mgr.GetScheme(),
+ GPUResourceName: gpuResourceName,
+ CanaryController: canaryController,
+ Registrar: mcpRegistrar,
+ MCPHealthInterval: mcpHealthInterval,
}
if canaryController != nil {
canaryController.Registrar = mcpRegistrar
diff --git a/config/manager/kustomization.yaml b/config/manager/kustomization.yaml
index 5eb2c3f..3912106 100644
--- a/config/manager/kustomization.yaml
+++ b/config/manager/kustomization.yaml
@@ -5,5 +5,5 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
images:
- name: controller
- newName: controller
- newTag: v0.1.0
+ newName: ghcr.io/gitcommitankit/agentrax
+ newTag: latest
diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml
index 70712bd..facb231 100644
--- a/config/manager/manager.yaml
+++ b/config/manager/manager.yaml
@@ -50,23 +50,33 @@ spec:
# - linux
securityContext:
runAsNonRoot: true
- # TODO(user): For common cases that do not require escalating privileges
- # it is recommended to ensure that all your Pods/Containers are restrictive.
- # More info: https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted
- # Please uncomment the following code if your project does NOT have to work on old Kubernetes
- # versions < 1.19 or on vendors versions which do NOT support this field by default (i.e. Openshift < 4.11 ).
- # seccompProfile:
- # type: RuntimeDefault
+ seccompProfile:
+ type: RuntimeDefault
containers:
- command:
- /manager
args:
- --leader-elect
- --health-probe-bind-address=:8081
+ - --registry-bind-address=:9090
+ - --gpu-resource-name=nvidia.com/gpu
+ env:
+ - name: AGENTRAX_MCP_HEALTH_INTERVAL
+ value: "30s"
+ - name: AGENTRAX_REGISTRY_TTL
+ value: "90s"
+ ports:
+ - containerPort: 9090
+ name: registry
+ protocol: TCP
+ - containerPort: 8081
+ name: health
+ protocol: TCP
image: controller:latest
name: manager
securityContext:
allowPrivilegeEscalation: false
+ readOnlyRootFilesystem: true
capabilities:
drop:
- "ALL"
@@ -92,4 +102,4 @@ spec:
cpu: 10m
memory: 64Mi
serviceAccountName: controller-manager
- terminationGracePeriodSeconds: 10
+ terminationGracePeriodSeconds: 30
diff --git a/config/manager/registry_service.yaml b/config/manager/registry_service.yaml
index 1a57013..3575a4b 100644
--- a/config/manager/registry_service.yaml
+++ b/config/manager/registry_service.yaml
@@ -1,7 +1,7 @@
apiVersion: v1
kind: Service
metadata:
- name: agentrax-registry
+ name: registry
namespace: system
labels:
app.kubernetes.io/name: agentrax
diff --git a/config/samples/agentrax_v1alpha1_agentdeployment_canary.yaml b/config/samples/agentrax_v1alpha1_agentdeployment_canary.yaml
new file mode 100644
index 0000000..461d625
--- /dev/null
+++ b/config/samples/agentrax_v1alpha1_agentdeployment_canary.yaml
@@ -0,0 +1,33 @@
+apiVersion: agentrax.io/v1alpha1
+kind: AgentDeployment
+metadata:
+ name: canary-agent
+ namespace: tenant-search
+ labels:
+ app.kubernetes.io/part-of: agentrax-demo
+spec:
+ image: gcr.io/google-containers/echoserver:1.4
+ port: 8080
+ tenantRef: team-search
+ replicas:
+ min: 1
+ max: 4
+ metric: queueDepth
+ target: 50
+ rollout:
+ strategy: Canary
+ steps:
+ - setWeight: 20
+ - pause: 60s
+ - setWeight: 50
+ - pause: 120s
+ - setWeight: 100
+ rollback:
+ maxErrorRate: "1%"
+ maxP99LatencyMs: 450
+ minRequestSample: 100
+ mcp:
+ expose: true
+ tools:
+ - webSearch
+ - summarizer
diff --git a/config/samples/agentrax_v1alpha1_agentdeployment_mcp.yaml b/config/samples/agentrax_v1alpha1_agentdeployment_mcp.yaml
new file mode 100644
index 0000000..6e25d01
--- /dev/null
+++ b/config/samples/agentrax_v1alpha1_agentdeployment_mcp.yaml
@@ -0,0 +1,23 @@
+apiVersion: agentrax.io/v1alpha1
+kind: AgentDeployment
+metadata:
+ name: search-agent
+ namespace: tenant-search
+ labels:
+ app.kubernetes.io/part-of: agentrax-demo
+spec:
+ image: gcr.io/google-containers/echoserver:1.4
+ port: 8080
+ tenantRef: team-search
+ replicas:
+ min: 1
+ max: 3
+ metric: queueDepth
+ target: 50
+ rollout:
+ strategy: Recreate
+ mcp:
+ expose: true
+ tools:
+ - search
+ - summarizer
diff --git a/config/samples/kustomization.yaml b/config/samples/kustomization.yaml
index 8ceb6de..5ac496c 100644
--- a/config/samples/kustomization.yaml
+++ b/config/samples/kustomization.yaml
@@ -1,5 +1,7 @@
## Append samples of your project ##
resources:
- agentrax_v1alpha1_agentdeployment.yaml
+- agentrax_v1alpha1_agentdeployment_mcp.yaml
+- agentrax_v1alpha1_agentdeployment_canary.yaml
- agentrax_v1alpha1_tenantquota.yaml
# +kubebuilder:scaffold:manifestskustomizesamples
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
new file mode 100644
index 0000000..0080652
--- /dev/null
+++ b/docs/ARCHITECTURE.md
@@ -0,0 +1,366 @@
+# Agentrax — System Architecture Blueprint & Design Document
+
+> **Status**: Authoritative System Architecture Reference
+> **API Group**: `agentrax.io/v1alpha1`
+> **Maintainers**: Agentrax Engineering & Architecture
+> **Notice**: This document serves as the permanent system design reference and architectural contract for the Agentrax Kubernetes Operator. Any architectural changes, CRD modifications, or invariant updates must be reflected here.
+
+---
+
+## 1. System Overview & Core Philosophy
+
+**Agentrax** is a specialized, cloud-native Kubernetes operator purpose-built to manage the end-to-end lifecycle of AI and Large Language Model (LLM) Agent workloads.
+
+Unlike standard microservices, autonomous agents and LLM inference endpoints possess unique operational profiles:
+
+1. **Asymmetric Resource Footprints**: Agents consume varying ratios of CPU, GPU units, and external tool resources.
+2. **Low-Traffic Statistical Vulnerability**: Canaries on internal agent services often handle lower request volumes where standard error-rate percentages produce statistical noise.
+3. **Dynamic Tool Capabilities**: AI agents expose dynamic tool sets via the **Model Context Protocol (MCP)** that external clients and multi-agent orchestrators need to discover in real time.
+
+### Core Architectural Principles
+
+- **Re-entrant State Machines**: All rollout and lifecycle workflows persist their operational state in the CRD `status` subresource, ensuring immediate recovery and zero state drift across controller manager restarts.
+- **Idempotent Reconciliation**: Every managed child resource is reconciled via `controllerutil.CreateOrUpdate` with isolated `MutateFn` closures, preserving API-server-defaulted fields.
+- **Atomic Multi-Tenancy**: Quota admission uses in-flight memory reservations to guarantee atomic concurrency safety and prevent Time-Of-Check to Time-Of-Use (TOCTOU) budget exhaustion.
+- **Fail-Safe Self-Healing**: Telemetry or monitoring outages trigger deterministic, fail-safe rollbacks rather than halting or hanging production rollouts.
+- **Native Kubernetes Idioms**: Built natively on top of Gateway API (`HTTPRoute`), standard `HorizontalPodAutoscaler` (via Prometheus Adapter), and Kubernetes finalizer foreground garbage collection.
+
+---
+
+## 2. High-Level System Architecture
+
+```mermaid
+flowchart TB
+ subgraph ControlPlane["Kubernetes Control Plane & API Server"]
+ AD["AgentDeployment CR
agentrax.io/v1alpha1"]
+ TQ["TenantQuota CR
agentrax.io/v1alpha1"]
+ WH["Validating & Mutating
Webhook Server"]
+ end
+
+ subgraph Operator["Agentrax Controller Manager Process"]
+ direction TB
+ REC["AgentDeployment Reconciler
internal/controller"]
+ TQC["TenantQuota Reconciler
internal/controller"]
+ ROLL["Canary Rollout Engine
internal/rollout"]
+ AUTO["Autoscaling Engine
internal/scaling"]
+ QUOTA["Quota Enforcer
internal/quota"]
+ REG["MCP Discovery Registry Server
internal/registry (HTTP :9090)"]
+ end
+
+ subgraph ManagedResources["Managed Workload Resources (Tenant Namespace)"]
+ DEP_STABLE["Stable Deployment
(spec.replicas.min)"]
+ DEP_CANARY["Canary Deployment
(1 replica)"]
+ SVC_STABLE["Stable Service
agentrax.io/variant=stable"]
+ SVC_CANARY["Canary Service
agentrax.io/variant=canary"]
+ HPA["HorizontalPodAutoscaler
(Quota-Capped maxReplicas)"]
+ HTTP_ROUTE["Gateway API HTTPRoute
(Weighted Backend Traffic Split)"]
+ SM["Prometheus ServiceMonitor"]
+ end
+
+ subgraph ExternalTelemetry["Monitoring & Persistence Layer"]
+ PROM["Prometheus Server
internal/metrics"]
+ CM["ConfigMap Storage
agentrax-registry"]
+ CLIENT["External MCP Clients & Orchestrators
GET /agents"]
+ end
+
+ AD --> WH
+ TQ --> WH
+ WH --> QUOTA
+ WH --> REC
+ WH --> TQC
+ REC --> DEP_STABLE
+ REC --> SVC_STABLE
+ REC --> SM
+ REC --> AUTO
+ AUTO --> HPA
+ REC --> ROLL
+ ROLL --> DEP_CANARY
+ ROLL --> SVC_CANARY
+ ROLL --> HTTP_ROUTE
+ ROLL --> PROM
+ REC --> REG
+ REG <--> CM
+ CLIENT --> REG
+```
+
+---
+
+## 3. Package Architecture & Separation of Concerns
+
+The repository enforces strict directional boundaries to prevent circular dependencies and isolate business logic from Kubernetes plumbing:
+
+| Package | Scope & Responsibility | Key Invariants |
+| ---------------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
+| `api/v1alpha1/` | CRD type definitions, OpenAPI markers, schema validation rules, and status condition constants. | **Zero business logic**; only struct declarations and generated deep-copy methods. |
+| `internal/controller/` | Controller-runtime reconcile loops (`AgentDeployment`, `TenantQuota`). | Only layer that executes write calls against the Kubernetes API for core-owned resources (Deployments, Services, HPAs, HTTPRoutes). Consumes subsystems via interfaces. |
+| `internal/quota/` | Quota arithmetic and concurrency-safe in-flight reservation cache. | Pure arithmetic; mutex-guarded state map; zero direct API server network calls in calculation paths. |
+| `internal/webhook/` | Validating and Mutating admission webhooks. | Shared with `internal/quota` to enforce admission rules before objects are persisted. |
+| `internal/scaling/` | HPA synthesis, velocity rules, and dynamic quota ceiling headroom. | Calculates `QuotaHeadroom()` to cap HPA `maxReplicas` and applies stabilization windows. |
+| `internal/rollout/` | Canary state machine, PromQL query construction, and threshold evaluation. | Re-entrant state machine; sample-size gating; fail-safe timeout evaluation. |
+| `internal/registry/` | MCP registrar, JSON-RPC 2.0 handshake, TTL sweeper, and discovery REST API. | In-memory registry with ConfigMap write-through for persistence; background health probes and TTL sweep. Explicitly allowed to write the `agentrax-registry` ConfigMap for state recovery. |
+| `internal/metrics/` | Bounded HTTP Prometheus query client. | Wraps all responses with `io.LimitReader` (1 MiB ceiling) to prevent memory exhaustion. |
+
+---
+
+## 4. Core Subsystems Deep Dive
+
+### 4.1 Multi-Tenancy & Atomic Quota Admission
+
+Multi-tenancy in Agentrax is enforced at the namespace level via the `TenantQuota` CRD. Each tenant namespace (`tenant-`) contains exactly one `TenantQuota` defining upper bounds on:
+
+- `maxAgents`: Total number of `AgentDeployment` instances permitted.
+- `maxGPUs`: Total number of GPU units across all pods in the namespace.
+- `maxTotalReplicas`: Maximum sum of pod replicas across all agents in the namespace.
+- `maxReplicasPerAgent`: Ceiling on `spec.replicas.max` for any single agent.
+
+```mermaid
+sequenceDiagram
+ autonumber
+ actor User as kubectl / GitOps
+ participant APIServer as K8s API Server
+ participant Webhook as Validating Webhook
+ participant Enforcer as Quota Enforcer (In-Flight Map)
+ participant Reconciler as TenantQuota Reconciler
+
+ User->>APIServer: POST /apis/agentrax.io/v1alpha1/agentdeployments
+ APIServer->>Webhook: AdmissionReview Request
+ Webhook->>Enforcer: AdmitAndReserve(tenant, desiredDemand)
+ Note over Enforcer: Mutex Lock Held
Sums Active Replicas + In-Flight Reservations
+ alt Quota Exceeded
+ Enforcer-->>Webhook: Rejected (OverQuota)
+ Webhook-->>APIServer: Admission Denied (403 Forbidden)
+ APIServer-->>User: Error: Quota exceeded
+ else Within Budget
+ Enforcer->>Enforcer: Store In-Flight Reservation (TTL 10s)
+ Enforcer-->>Webhook: Admitted
+ Webhook-->>APIServer: Admission Allowed
+ APIServer-->>User: Created 201
+ Note over Reconciler: Periodic Reconcile (5 min)
Synchronizes real usage & clears stale reservations
+ end
+```
+
+#### Key Invariants:
+
+1. **Atomic `AdmitAndReserve`**: Prevents TOCTOU race conditions where two simultaneous creation requests each pass naive read checks but collectively blow the quota ceiling.
+2. **Dry-Run Awareness**: Admission webhooks evaluate `AdmissionRequest.DryRun`; dry-run requests never write to the in-flight reservation map.
+3. **Non-Destructive Over-Quota Handling**: If an administrator lowers a `TenantQuota` below active usage, the reconciler sets the `OverQuota` condition on the quota object but **never forcibly terminates running workloads**.
+
+---
+
+### 4.2 Metrics-Driven Autoscaling & Dynamic Quota Headroom
+
+Agentrax dynamically synthesizes and manages a `HorizontalPodAutoscaler` (autoscaling/v2) for each `AgentDeployment`.
+
+```
+ ┌────────────────────────┐
+ │ TenantQuota │
+ │ (maxTotalReplicas: 20) │
+ └───────────┬────────────┘
+ │
+ ┌───────────────────────┴───────────────────────┐
+ ▼ ▼
+┌───────────────────────────────┐ ┌───────────────────────────────┐
+│ Agent A (Active) │ │ Agent B (Scaling) │
+│ Current: 8 pods │ │ spec.replicas: min 2, max 10 │
+│ spec.replicas: min 2, max 10 │ │ Used by others: 8 │
+└───────────────────────────────┘ │ Total budget remaining: 12 │
+ │ QuotaHeadroom() -> 10 (Max) │
+ └───────────────────────────────┘
+```
+
+#### Dynamic Quota Ceiling Math:
+
+Before writing the HPA, the reconciler computes the available headroom:
+$$\text{Headroom} = \min(\text{spec.replicas.max}, \, \text{maxReplicasPerAgent}, \, \text{maxTotalReplicas} - \text{ActiveReplicasOtherAgents})$$
+
+If $\text{Headroom} < \text{spec.replicas.min}$, HPA `maxReplicas` is clamped to `spec.replicas.min` and the condition `QuotaLimited: Capped` is surfaced on the `AgentDeployment`.
+
+#### Stabilization Windows & Velocity Control:
+
+To prevent flapping during bursty LLM agent inference loads:
+
+- **Scale-Up**: Stabilization window of `60s`; scale-up rate-limited to **4 pods per 60s** (`autoscalingv2.PodsScalingPolicy`, `Value: 4, PeriodSeconds: 60`).
+- **Scale-Down**: Stabilization window of `300s` (5 minutes); scale-down rate-limited to **1 pod per 60s** (`autoscalingv2.PodsScalingPolicy`, `Value: 1, PeriodSeconds: 60`).
+
+---
+
+### 4.3 Progressive Canary Rollout & Statistical Auto-Rollback
+
+When an `AgentDeployment` has `strategy: Canary` and the image changes, Agentrax enters the `RolloutInProgress` state machine:
+
+```mermaid
+stateDiagram-v2
+ [*] --> Idle: spec.image == status.stableVersion
+ Idle --> StartCanary: spec.image != status.stableVersion
+
+ state StartCanary {
+ [*] --> PauseHPA: Delete stable HPA
+ PauseHPA --> CreateCanary: Deploy canary pod (weight 0)
+ CreateCanary --> ApplyStep: Fetch step[index]
+ }
+
+ state StepExecution {
+ ApplyStep --> SetWeight: step has setWeight
+ SetWeight --> UpdateHTTPRoute: Gateway API weight split
+ UpdateHTTPRoute --> NextStep: Advance index
+
+ ApplyStep --> PauseWindow: step has pause
+ PauseWindow --> QueryPrometheus: Query sampleCount, errorRate, p99
+ }
+
+ state Evaluation {
+ QueryPrometheus --> SampleGateCheck: Check sampleCount >= minRequestSample
+ SampleGateCheck --> ExtendPause: sampleCount < minRequestSample
+ ExtendPause --> PauseWindow: Wait (max 3x or 15m)
+
+ SampleGateCheck --> ThresholdCheck: Sample sufficient
+ ThresholdCheck --> NextStep: errorRate <= max & p99 <= max
+ ThresholdCheck --> TriggerRollback: Threshold Breached
+ QueryPrometheus --> CheckPromTimeout: Prometheus Error
+ CheckPromTimeout --> TriggerRollback: Unreachable > 60s
+ }
+
+ NextStep --> StepExecution: More steps remain
+ NextStep --> Promote: Reached stepWeight 100
+
+ state Promote {
+ UpdateStableDeployment: Update image on stable
+ DeleteCanaryResources: Delete canary Deployment/Service/Route
+ RestoreHPA: Recreate stable HPA
+ ReRegisterMCP: Update MCP registry with new tools
+ }
+
+ state TriggerRollback {
+ ResetRoute: Shift 100% traffic to stable
+ CleanupCanary: Delete canary Deployment
+ RestoreHPA_RB: Recreate stable HPA
+ SetFailedPhase: phase=RolloutFailed
+ }
+
+ Promote --> Idle: status.phase=Running
+ TriggerRollback --> Idle: Manual fix required
+```
+
+#### Statistical Sample-Gating Invariant:
+
+Evaluating error percentages over small sample sizes (e.g., 2 errors out of 3 requests = 66% error rate) causes catastrophic false-positive rollbacks on low-traffic agents. Agentrax strictly enforces:
+
+1. **Sample Gate**: Thresholds are **never evaluated** until $\text{ObservedRequests} \ge \text{minRequestSample}$.
+2. **Pause Extension**: If the sample size is insufficient when a pause timer expires, the pause is extended in increments up to $\min(3 \times \text{step.pause}, 15\text{ minutes})$.
+3. **Canonical Duration Syntax**: PromQL duration selectors are formatted into canonical Prometheus syntax (`5m`, `1h`, `30s`, no trailing `0s`).
+4. **Fail-Safe Prometheus Timeout**: If Prometheus is completely unreachable for $>60\text{ seconds}$ (`promUnreachableSince`), Agentrax triggers an automated fail-safe rollback.
+
+---
+
+### 4.4 MCP Service Discovery & Lifecycle Management
+
+Agentrax features an embedded **Model Context Protocol (MCP)** discovery server served on port `:9090` and exposed via the `agentrax-registry` Service in `agentrax-system`.
+
+The `AgentDeploymentReconciler` in `internal/controller` consumes MCP lifecycle operations via the `AgentRegistrar` interface (`Register`, `Deregister`, `Heartbeat`) — enabling clean dependency injection and test isolation via `mockAgentRegistrar`. The canary rollout `Controller` in `internal/rollout` holds a concrete `*registry.Registrar` to trigger re-registration on promotion.
+
+```
+┌────────────────────────────────────────────────────────────────────────────┐
+│ Agentrax Operator Manager Process │
+│ │
+│ ┌─────────────────────────────────┐ ┌────────────────────────────────┐ │
+│ │ AgentDeploymentReconciler │ │ MCP Discovery Registry │ │
+│ │ Registrar: AgentRegistrar │──▶│ (HTTP :9090) │ │
+│ │ (interface — test injectable) │ │ • GET /agents │ │
+│ └─────────────────────────────────┘ │ • GET /agents/{ns}/{name} │ │
+│ │ • Background TTL Sweeper │ │
+│ ┌─────────────────────────────────┐ └───────────────┬────────────────┘ │
+│ │ Canary rollout.Controller │ │ │
+│ │ Registrar: *registry.Registrar│ │ Write-Through │
+│ │ (concrete — post-promotion │ ▼ │
+│ │ re-registration) │ ┌────────────────────────────────┐ │
+│ └─────────────────────────────────┘ │ ConfigMap: `agentrax-registry` │ │
+│ │ (Cold Startup Recovery Target) │ │
+│ └────────────────────────────────┘ │
+└────────────────────────────────────────────────────────────────────────────┘
+```
+
+#### Registration Handshake Protocol:
+
+1. When an agent pod becomes `Ready` and `phase=Running`, the reconciler calls `AgentRegistrar.Register()`.
+2. The registrar sends an HTTP `POST` containing a JSON-RPC 2.0 `initialize` payload (protocol version `2024-11-05`) to `http://{name}.{namespace}.svc:{port}/initialize`.
+3. Discovered tools from `result.capabilities.tools` are merged with `spec.mcp.tools` (deduplicated).
+4. The registration is written to the in-memory map and persisted to the `agentrax-registry` ConfigMap with `retry.RetryOnConflict`.
+
+#### Heartbeat, TTL Sweeper & 3-Strike Rule:
+
+- Every active agent record carries a `TTL` (default `90s`) and a `HeartbeatAt` timestamp.
+- While the agent remains healthy in `Running` phase, the reconciler calls `AgentRegistrar.Heartbeat()`.
+- A background ticker running every `30s` sweeps the registry and purges entries whose heartbeats have lapsed (handling node crashes and ungraceful pod termination).
+- If an agent fails 3 consecutive heartbeat probes, it is automatically deregistered with `ErrHeartbeatDeregistered` and `status.registered` is set to `false`.
+
+---
+
+### 4.5 Garbage Collection & Finalizer Ordering
+
+When an `AgentDeployment` is deleted, Kubernetes sets `metadata.deletionTimestamp`. The reconciler executes the following strict sequence:
+
+```
+[AgentDeployment Deletion]
+ │
+ ▼
+1. Fetch object & verify DeletionTimestamp != nil
+ │
+ ▼
+2. Invoke AgentRegistrar.Deregister(ctx, ad) <--- Child Service & Deployment STILL ALIVE
+ • Remove entry from memory & ConfigMap
+ │
+ ▼
+3. Remove finalizer: `agentrax.io/mcp-deregister`
+ │
+ ▼
+4. Update object on API Server
+ │
+ ▼
+5. Kubernetes GC cascade deletes child resources (Deployment, Service, HPA, Route)
+```
+
+**Invariant**: MCP deregistration MUST complete _before_ the child `Service` is garbage collected, ensuring external clients never encounter dead routing endpoints.
+
+---
+
+## 5. Architectural Decision Records (ADRs) & Trade-Offs
+
+| Decision | Alternative Considered | Trade-Off & Rationale for Agentrax |
+| --------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Gateway API (`HTTPRoute`)** | Istio `VirtualService` / Ingress Annotations | Istio requires a heavy service-mesh control plane and sidecar injection. Ingress annotations lack standardized multi-backend weighted traffic splits. Gateway API provides a lightweight, vendor-neutral standard for traffic shifting. |
+| **Custom Canary Rollout Engine** | Argo Rollouts / Flagger | Generic rollout tools treat metric anomalies as pure percentages without low-traffic statistical gating (`minRequestSample`). Building an embedded, re-entrant state machine allowed us to guarantee sample-size gating and MCP tool re-registration upon promotion. |
+| **Native HPA via Custom Metrics** | KEDA (`ScaledObject`) | KEDA is powerful but adds external CRD dependencies. Generating native Kubernetes `HorizontalPodAutoscaler` objects tied to the Prometheus Adapter custom metrics pipeline minimized dependencies while giving full control over stabilization windows. |
+| **Embedded Registry + ConfigMap Store** | Dedicated etcd / Redis / Database | Adding a dedicated database for service discovery increases operator operational complexity. The in-operator HTTP server with ConfigMap write-through store provides simple, robust storage for hundreds of agent services with cold-restart recovery. |
+| **Go (`controller-runtime`)** | Python (`Kopf`) | Go provides native compile-time safety, seamless alignment with Kubernetes upstream libraries, and access to `setup-envtest` for isolated in-process integration testing. |
+
+---
+
+## 6. Canonical Integration Validation Contract
+
+The following top-level integration scenarios represent the non-negotiable correctness guarantees validated by the automated test suite:
+
+```mermaid
+gantt
+ title Canonical Lifecycle Scenarios
+ dateFormat X
+ axisFormat %s
+ section Self-Healing
+ Delete Child Deployment :active, a1, 0, 5
+ Reconciler Recreates Deployment :crit, a2, 5, 10
+ section Quota Collisions
+ Concurrent Near-Limit Creates :active, b1, 0, 2
+ In-Flight Map Admits 1, Rejects 2 :crit, b2, 2, 4
+ section Canary Rollback
+ Inject 500ms Latency Spike :active, c1, 0, 5
+ Threshold Breached Rollback Fire :crit, c2, 5, 10
+ Traffic Restored 100% Stable :c3, 10, 12
+ section MCP Expiration
+ Ungraceful Pod Termination :active, d1, 0, 10
+ TTL Sweeper Purges Expired Entry :crit, d2, 10, 15
+```
+
+1. **Self-Healing Invariant**: Out-of-band manual deletion of any child resource (Deployment, Service, HPA, HTTPRoute) is detected and recreated on the next reconcile cycle with owned state intact.
+2. **Quota Barrier Invariant**: Under high-concurrency requests, total admitted replicas and GPU allocations across tenants never exceed the configured `TenantQuota`.
+3. **Canary Safety Invariant**: Injected error rates or latency anomalies trigger automated rollback before traffic promotion reaches 100%, and stable traffic is never dropped.
+4. **Ungraceful Termination Invariant**: Dead or crashed agent pods that bypass the deletion finalizer are swept from the MCP discovery registry within one TTL cycle ($90\text{s}$).
+5. **Foreground Finalizer Invariant**: Deleting an `AgentDeployment` always removes the agent from external discovery _before_ tearing down cluster networking.
diff --git a/docs/agentrax.md b/docs/agentrax.md
deleted file mode 100644
index 94ee94e..0000000
--- a/docs/agentrax.md
+++ /dev/null
@@ -1,78 +0,0 @@
-# Agentrax — Architecture Reference
-
-> Permanent architecture reference. Update this file in the same commit as any architecture boundary or CRD change.
-
----
-
-## Alternatives Considered (§4.3)
-
-| Decision | Alternative | Why not chosen (v1) |
-| ------------------------------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Custom CRD + controller | KEDA (`ScaledObject`) for autoscaling | KEDA is excellent for scaling in isolation, but Agentrax's scope is the full lifecycle (rollout + tenancy + discovery). KEDA is a credible v2 backend alternative. |
-| Custom rollout controller | Argo Rollouts / Flagger | Both are mature progressive-delivery tools and the closest prior art. Not adopted because the statistical-significance handling for low-traffic agent canaries needed to be owned explicitly, not inherited as a black box. |
-| Custom model-serving management | KServe / Seldon Core | Both solve model serving but don't address multi-tenant quota admission or MCP-native discovery — Agentrax's actual differentiators. |
-| Traffic splitting | Istio VirtualService | Istio's weighting works but requires full mesh sidecar deployment; Gateway API achieves the same with a lighter footprint. |
-| Language | Python (Kopf) | Smaller testing ecosystem (no `envtest` equivalent) and weaker typing for CRD schemas at this complexity. |
-
----
-
-## E2E Validation Scenarios (§9.4)
-
-The canonical list of top-level end-to-end test scenarios. Do not add new top-level e2e scenarios without first checking this list.
-
-| Feature | Key scenario validated end-to-end |
-| ------------------- | ----------------------------------------------------------------------------------------------------- |
-| Core reconciliation | Manual deletion of child `Deployment` is self-healed within one reconcile interval |
-| Autoscaling | Synthetic queue-depth load produces bounded, non-flapping scale-out and scale-in |
-| Canary rollout | Injected latency regression triggers rollback before reaching 100% weight; stable traffic never drops |
-| Multi-tenancy | Concurrent near-limit creates never both succeed when only one fits in quota |
-| MCP registry | Registry entry expires after ungraceful pod termination without waiting on an explicit delete event |
-
----
-
-## Development Roadmap (§8)
-
-| Phase | Focus | Status | Milestone |
-| ----- | ------------------- | :---------: | ---------------------------------------------------------------------------------------- |
-| 0 | Scaffolding | **Done** | `make install` applies CRDs; `make run` starts both controllers |
-| 1 | Core reconciliation | **Done** | `AgentDeployment` → Deployment/Service/ServiceMonitor; status/conditions; finalizer stub |
-| 2 | Multi-tenancy | **Done** | `TenantQuota`, validating + mutating webhook, quota enforcement |
-| 3 | Autoscaling | **Done** | Prometheus Adapter integration, managed HPA |
-| 4 | Canary rollout | **Done** | Rollout state machine, Gateway API traffic shifting, PromQL threshold evaluation |
-| 5 | MCP registry | **Done** | Registrar, registry HTTP handler, discovery API, TTL/heartbeat, ConfigMap persistence |
-| 6 | Hardening & demo | Pending | E2e tests in CI, Helm chart, README, recorded demo |
-
-Phases 2, 3, and 5 are independent of each other and can run in parallel once Phase 1 is complete. Phase 4 depends on both 2 and 3.
-
-### Phase 4 — Canary Rollout (§6)
-
-The canary controller (`internal/rollout.Controller`) is a pure helper driven by the `AgentDeploymentReconciler` each reconcile cycle. When `spec.rollout.strategy: Canary` and `spec.image` differs from `status.stableVersion`, the reconciler transitions to `RolloutInProgress` and calls `Step()` on every subsequent reconcile.
-
-**State machine**: `setWeight` steps create the canary Deployment and upsert a Gateway API `HTTPRoute` with the target weight split. `pause` steps query Prometheus for request count (sample gate), error rate, and p99 latency via `internal/metrics.Client`. If Prometheus is unreachable for longer than a fixed 60-second timeout, a fail-safe rollback fires. Sample-size gating extends the pause window up to the lesser of `3×pause_duration` or an absolute 15-minute maximum before forcing evaluation. Promotion updates the stable Deployment image and restores the HPA; rollback reverts everything and sets `phase=RolloutFailed`.
-
-**New operator flags**: `--prometheus-url` (required for Canary), `--gateway-name`, `--gateway-namespace`.
-
-**New status fields**: `canaryStepIndex`, `pauseStartedAt`, `promUnreachableSince` — all persisted so the state machine is re-entrant across operator restarts.
-
-### Phase 5 — MCP Registration & Discovery (§7)
-
-The MCP registry is an embedded HTTP service inside the operator manager process, served on `--registry-bind-address` (default `:9090`) and exposed to the cluster as the `agentrax-registry` ClusterIP Service in `agentrax-system`.
-
-**Registration flow**: When `spec.mcp.expose: true` and the agent's underlying Deployment reports `rolloutComplete` (`phase=Running`), the reconciler calls `registry.Registrar.Register()`, which performs an MCP-level `initialize` JSON-RPC 2.0 handshake by posting to the `/initialize` sub-path at `http://{name}.{namespace}.svc:{port}/initialize`. Discovered tools are merged with `spec.mcp.tools` and persisted to the in-memory registry map and `agentrax-registry` ConfigMap. On failure, `MCPHandshakeFailed` condition is set and `status.registered` remains `false`.
-
-**TTL & Heartbeat**: Every registry entry carries a `TTL` (default 90s) and `HeartbeatAt` timestamp. When an agent is `status.registered: true` and healthy in `Running` phase, the reconciler triggers `Registrar.Heartbeat()`. A background sweeper running every 30s removes expired entries if heartbeats lapse (e.g. following an ungraceful pod termination). After 3 consecutive heartbeat probe failures, the agent is automatically deregistered.
-
-**Deregistration & Finalizers**: Clean deregistration occurs when: (1) an `AgentDeployment` is deleted (via the `agentrax.io/mcp-deregister` finalizer before child services are garbage-collected), (2) `spec.mcp.expose` is toggled to `false`, or (3) 3 consecutive heartbeat probes fail.
-
-**State Recovery**: On operator startup/restart, existing registrations are reloaded from the `agentrax-registry` ConfigMap.
-
-**REST API Endpoints**:
-- `GET /agents`: List all active, non-expired registered agents.
-- `POST /agents`: Register / update an agent. This endpoint requires a successful MCP initialize handshake and currently requires no authentication. It bypasses the handshake verification when called directly.
-- `GET /agents/{namespace}/{name}`: Get metadata and tools for a specific agent.
-- `DELETE /agents/{namespace}/{name}`: Deregister an agent. This endpoint currently requires no authentication.
-- *(Legacy aliases `POST /register` and `DELETE /deregister` are supported for backward compatibility)*.
-
-**New operator flags**: `--registry-bind-address` (default `:9090`).
-
-
diff --git a/hack/setup-git-hooks.sh b/hack/setup-git-hooks.sh
index fc3bf7e..5291b69 100755
--- a/hack/setup-git-hooks.sh
+++ b/hack/setup-git-hooks.sh
@@ -66,6 +66,13 @@ echo "🧪 [pre-push] Running full test suite and manifest verification..."
echo " -> make manifests generate..."
make manifests generate
+# Verify no uncommitted manifest/codegen drift
+if ! git diff --quiet config/ api/; then
+ echo "❌ [pre-push] Generated manifests or code have uncommitted changes!"
+ echo " Please commit the generated changes before pushing."
+ exit 1
+fi
+
# Run test suite
echo " -> make test..."
make test
@@ -76,6 +83,12 @@ if [ -f "./bin/kustomize" ]; then
./bin/kustomize build config/default > /dev/null
fi
+# Verify Helm chart if helm is available
+if command -v helm >/dev/null 2>&1 && [ -d "./charts/agentrax" ]; then
+ echo " -> helm lint charts/agentrax/..."
+ helm lint charts/agentrax/ > /dev/null
+fi
+
echo "✅ [pre-push] All pre-push checks passed!"
EOF
diff --git a/internal/controller/agentdeployment_controller.go b/internal/controller/agentdeployment_controller.go
index 1827406..e44dfec 100644
--- a/internal/controller/agentdeployment_controller.go
+++ b/internal/controller/agentdeployment_controller.go
@@ -21,7 +21,6 @@ import (
"context"
"errors"
"fmt"
- "sync"
"time"
appsv1 "k8s.io/api/apps/v1"
@@ -70,7 +69,16 @@ const (
quotaStateSkipped
)
+// AgentRegistrar defines the interface for managing agent lifecycle in the MCP discovery registry.
+type AgentRegistrar interface {
+ Register(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error
+ Deregister(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error
+ Heartbeat(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error
+}
+
// AgentDeploymentReconciler reconciles an AgentDeployment object.
+// It manages the complete lifecycle of AI agent workloads including Deployments,
+// Services, ServiceMonitors, HPAs, Canary rollouts, and MCP registry registration.
type AgentDeploymentReconciler struct {
client.Client
Scheme *runtime.Scheme
@@ -87,29 +95,16 @@ type AgentDeploymentReconciler struct {
// Registrar manages registration and discovery of this agent in the MCP registry.
// When nil, MCP registration features are disabled.
- Registrar *registry.Registrar
+ Registrar AgentRegistrar
+
+ // MCPHealthInterval is the requeue interval used to drive periodic MCP
+ // heartbeat probes. Must stay shorter than the registry TTL so heartbeats
+ // land before entries expire. Defaults to 60s when zero.
+ MCPHealthInterval time.Duration
// hasServiceMonitorCRD is set once during SetupWithManager and determines
// whether ServiceMonitor reconciliation is attempted at all.
hasServiceMonitorCRD bool
-
- // deregisterMu guards the Deregister field so test goroutines can safely
- // inject and clear the hook while the reconciler goroutine reads it.
- deregisterMu sync.Mutex
-
- // Deregister is an optional hook called during deletion cleanup before the
- // finalizer is removed. Phase 5 will set this to a real MCP deregistration
- // function. In tests it can be used to assert ordering invariants.
- // Always access through SetDeregister / the mutex-protected load in runDeletionCleanup.
- Deregister func(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error
-}
-
-// SetDeregister safely replaces the Deregister hook under the mutex.
-// Use this instead of direct field assignment to avoid data races.
-func (r *AgentDeploymentReconciler) SetDeregister(fn func(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error) {
- r.deregisterMu.Lock()
- defer r.deregisterMu.Unlock()
- r.Deregister = fn
}
// +kubebuilder:rbac:groups=agentrax.io,resources=agentdeployments,verbs=get;list;watch;create;update;patch;delete
@@ -240,22 +235,11 @@ func (r *AgentDeploymentReconciler) Reconcile(ctx context.Context, req ctrl.Requ
}
// runDeletionCleanup performs pre-deletion tasks before the finalizer is removed.
-// If a Deregister hook is set on the reconciler, it is called here so that
-// deregistration happens while child resources (Service, Deployment) still exist.
-// Phase 5 will set Deregister to the real MCP deregistration implementation.
+// It deregisters the agent from the MCP discovery registry while child resources
+// (Service, Deployment) still exist.
func (r *AgentDeploymentReconciler) runDeletionCleanup(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error {
logger := log.FromContext(ctx)
- // Load the hook under the mutex so test goroutines can safely inject/clear it
- // without a data race against this reconciler goroutine.
- r.deregisterMu.Lock()
- deregister := r.Deregister
- r.deregisterMu.Unlock()
-
- if deregister != nil {
- if err := deregister(ctx, ad); err != nil {
- return fmt.Errorf("deregistering agent: %w", err)
- }
- } else if r.Registrar != nil {
+ if r.Registrar != nil {
if err := r.Registrar.Deregister(ctx, ad); err != nil {
return fmt.Errorf("deregistering agent: %w", err)
}
@@ -639,8 +623,11 @@ func (r *AgentDeploymentReconciler) reconcileMCPRegistration(ctx context.Context
return 0
}
- // Requeue interval shorter than TTL (90s) to ensure heartbeats occur before expiry.
- requeueInterval := 60 * time.Second
+ // Requeue interval shorter than the registry TTL to ensure heartbeats occur before expiry.
+ requeueInterval := r.MCPHealthInterval
+ if requeueInterval <= 0 {
+ requeueInterval = 60 * time.Second
+ }
// If already registered, perform heartbeat probe.
if ad.Status.Registered {
diff --git a/internal/controller/agentdeployment_controller_test.go b/internal/controller/agentdeployment_controller_test.go
index 3b6577b..b56e3ff 100644
--- a/internal/controller/agentdeployment_controller_test.go
+++ b/internal/controller/agentdeployment_controller_test.go
@@ -204,16 +204,19 @@ var _ = Describe("AgentDeployment Controller", func() {
return k8sClient.Get(ctx, key, &corev1.Service{})
}, testTimeout, testInterval).Should(Succeed(), "child Service should exist before deletion")
- // Inject a Deregister hook. A buffered channel is used so the hook
+ // Inject a mock Registrar. A buffered channel is used so the hook
// (called on the reconciler goroutine) can pass its observation to the
// test goroutine without a data race on plain booleans.
resultCh := make(chan bool, 1)
- testReconciler.SetDeregister(func(hctx context.Context, had *agentraxv1alpha1.AgentDeployment) error {
- err := k8sClient.Get(hctx, key, &corev1.Service{})
- resultCh <- (err == nil)
- return nil
- })
- DeferCleanup(func() { testReconciler.SetDeregister(nil) })
+ mockReg := &mockAgentRegistrar{
+ deregisterFn: func(hctx context.Context, had *agentraxv1alpha1.AgentDeployment) error {
+ err := k8sClient.Get(hctx, key, &corev1.Service{})
+ resultCh <- (err == nil)
+ return nil
+ },
+ }
+ testRegistrarProxy.SetDelegate(mockReg)
+ DeferCleanup(func() { testRegistrarProxy.SetDelegate(testRegistrar) })
// Delete the object — the reconciler must call Deregister, then remove the finalizer.
Expect(k8sClient.Delete(ctx, ad)).To(Succeed())
diff --git a/internal/controller/mcp_registration_test.go b/internal/controller/mcp_registration_test.go
index 893ed40..b6683b2 100644
--- a/internal/controller/mcp_registration_test.go
+++ b/internal/controller/mcp_registration_test.go
@@ -306,16 +306,19 @@ var _ = Describe("MCP Registration Integration", func() {
g.Expect(fetched.Status.Registered).To(BeTrue())
}, timeout, interval).Should(Succeed())
- // Install a Deregister hook that verifies Service still exists.
+ // Install a mock Registrar that verifies Service still exists during Deregister.
var serviceExistedDuringDeregister atomic.Bool
- testReconciler.SetDeregister(func(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error {
- svc := &corev1.Service{}
- svcKey := types.NamespacedName{Name: adName, Namespace: ns}
- err := k8sClient.Get(ctx, svcKey, svc)
- serviceExistedDuringDeregister.Store(err == nil)
- return testRegistrar.Deregister(ctx, ad)
- })
- defer testReconciler.SetDeregister(nil)
+ mockReg := &mockAgentRegistrar{
+ deregisterFn: func(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error {
+ svc := &corev1.Service{}
+ svcKey := types.NamespacedName{Name: adName, Namespace: ns}
+ err := k8sClient.Get(ctx, svcKey, svc)
+ serviceExistedDuringDeregister.Store(err == nil)
+ return testRegistrar.Deregister(ctx, ad)
+ },
+ }
+ testRegistrarProxy.SetDelegate(mockReg)
+ defer func() { testRegistrarProxy.SetDelegate(testRegistrar) }()
// Delete the AgentDeployment
fetched := &agentraxv1alpha1.AgentDeployment{}
diff --git a/internal/controller/suite_test.go b/internal/controller/suite_test.go
index 9280829..40c7f7f 100644
--- a/internal/controller/suite_test.go
+++ b/internal/controller/suite_test.go
@@ -68,6 +68,10 @@ var mgrDone chan struct{}
// and clear it in AfterEach to avoid contaminating other tests.
var testReconciler *AgentDeploymentReconciler
+// testRegistrarProxy is a thread-safe proxy installed in testReconciler.Registrar
+// before manager start. Tests can safely swap the delegate via SetDelegate.
+var testRegistrarProxy *registrarProxy
+
// testEnforcer is the shared quota Enforcer used by the TenantQuota reconciler
// and the validating webhook in integration tests.
var testEnforcer *quota.Enforcer
@@ -201,11 +205,15 @@ var _ = BeforeSuite(func() {
testMockMCP = &testMockMCPClient{}
testRegistrar = registry.NewRegistrar(testRegistry, testMockMCP)
+ // Install a thread-safe proxy so tests can swap the registrar delegate
+ // without data races after the manager starts.
+ testRegistrarProxy = newRegistrarProxy(testRegistrar)
+
testReconciler = &AgentDeploymentReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
GPUResourceName: quota.DefaultGPUResourceName,
- Registrar: testRegistrar,
+ Registrar: testRegistrarProxy,
}
Expect(testReconciler.SetupWithManager(mgr)).To(Succeed())
diff --git a/internal/controller/test_helpers_test.go b/internal/controller/test_helpers_test.go
index 9eea77c..ecca511 100644
--- a/internal/controller/test_helpers_test.go
+++ b/internal/controller/test_helpers_test.go
@@ -17,12 +17,16 @@ limitations under the License.
package controller
import (
+ "context"
+ "sync"
"time"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/client"
+
+ agentraxv1alpha1 "github.com/gitcommitankit/agentrax/api/v1alpha1"
)
const (
@@ -48,3 +52,71 @@ func namespaceObject(name string) *corev1.Namespace {
func inNamespace(ns string) client.ListOption {
return client.InNamespace(ns)
}
+
+// mockAgentRegistrar is a test double implementing AgentRegistrar.
+type mockAgentRegistrar struct {
+ registerFn func(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error
+ deregisterFn func(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error
+ heartbeatFn func(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error
+}
+
+func (m *mockAgentRegistrar) Register(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error {
+ if m.registerFn != nil {
+ return m.registerFn(ctx, ad)
+ }
+ return nil
+}
+
+func (m *mockAgentRegistrar) Deregister(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error {
+ if m.deregisterFn != nil {
+ return m.deregisterFn(ctx, ad)
+ }
+ return nil
+}
+
+func (m *mockAgentRegistrar) Heartbeat(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error {
+ if m.heartbeatFn != nil {
+ return m.heartbeatFn(ctx, ad)
+ }
+ return nil
+}
+
+// registrarProxy is a thread-safe proxy that wraps an AgentRegistrar and
+// allows per-test swapping of the delegate without data races.
+// This is installed once in the reconciler before manager start, then tests
+// can safely swap the delegate using SetDelegate.
+type registrarProxy struct {
+ mu sync.RWMutex
+ delegate AgentRegistrar
+}
+
+func newRegistrarProxy(initial AgentRegistrar) *registrarProxy {
+ return ®istrarProxy{delegate: initial}
+}
+
+func (p *registrarProxy) SetDelegate(d AgentRegistrar) {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ p.delegate = d
+}
+
+func (p *registrarProxy) Register(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error {
+ p.mu.RLock()
+ d := p.delegate
+ p.mu.RUnlock()
+ return d.Register(ctx, ad)
+}
+
+func (p *registrarProxy) Deregister(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error {
+ p.mu.RLock()
+ d := p.delegate
+ p.mu.RUnlock()
+ return d.Deregister(ctx, ad)
+}
+
+func (p *registrarProxy) Heartbeat(ctx context.Context, ad *agentraxv1alpha1.AgentDeployment) error {
+ p.mu.RLock()
+ d := p.delegate
+ p.mu.RUnlock()
+ return d.Heartbeat(ctx, ad)
+}
diff --git a/internal/metrics/prometheus.go b/internal/metrics/prometheus.go
index 2a9e794..44bdf87 100644
--- a/internal/metrics/prometheus.go
+++ b/internal/metrics/prometheus.go
@@ -35,6 +35,9 @@ import (
// defaultTimeout is the per-request HTTP timeout used when no custom timeout is set.
const defaultTimeout = 10 * time.Second
+// maxResponseBytes is the maximum allowed response size from Prometheus (1 MiB) to guard against memory exhaustion.
+const maxResponseBytes = 1 << 20
+
// Client is a lightweight HTTP client for the Prometheus query API.
// Create one with NewClient; the zero value is not usable.
type Client struct {
@@ -94,7 +97,7 @@ func (c *Client) QueryScalar(ctx context.Context, query string) (float64, error)
}
defer resp.Body.Close() //nolint:errcheck
- body, err := io.ReadAll(resp.Body)
+ body, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes))
if err != nil {
return 0, fmt.Errorf("reading Prometheus response: %w", err)
}
@@ -135,7 +138,7 @@ func (c *Client) QueryRange(ctx context.Context, query string, start, end time.T
}
defer resp.Body.Close() //nolint:errcheck
- body, err := io.ReadAll(resp.Body)
+ body, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes))
if err != nil {
return 0, fmt.Errorf("reading Prometheus range response: %w", err)
}
diff --git a/internal/registry/registry.go b/internal/registry/registry.go
index 7e4888a..f84bd05 100644
--- a/internal/registry/registry.go
+++ b/internal/registry/registry.go
@@ -358,7 +358,7 @@ func (r *Registry) loadFromConfigMap(ctx context.Context) error {
// ── HTTP Handler ──────────────────────────────────────────────────────────────
// Handler returns an http.Handler implementing the registry REST API.
-// It exposes standard RESTful endpoints under /agents as well as legacy aliases (/register, /deregister).
+// It exposes standard RESTful endpoints under /agents.
func (r *Registry) Handler() http.Handler {
mux := http.NewServeMux()
@@ -368,10 +368,6 @@ func (r *Registry) Handler() http.Handler {
mux.HandleFunc("GET /agents/{namespace}/{name}", r.handleGetAgent)
mux.HandleFunc("DELETE /agents/{namespace}/{name}", r.handleDeregisterAgentPath)
- // Backward-compatible RPC action aliases
- mux.HandleFunc("POST /register", r.handleRegister)
- mux.HandleFunc("DELETE /deregister", r.handleDeregister)
-
return mux
}
@@ -411,37 +407,6 @@ func (r *Registry) handleDeregisterAgentPath(w http.ResponseWriter, req *http.Re
_ = json.NewEncoder(w).Encode(map[string]string{"status": "deregistered"})
}
-// handleDeregister handles legacy HTTP DELETE requests targeting /deregister.
-func (r *Registry) handleDeregister(w http.ResponseWriter, req *http.Request) {
- namespace := req.URL.Query().Get("namespace")
- name := req.URL.Query().Get("name")
-
- if namespace == "" || name == "" {
- // Also support JSON body if query params are missing
- var body struct {
- Namespace string `json:"namespace"`
- Name string `json:"name"`
- }
- if err := json.NewDecoder(req.Body).Decode(&body); err == nil {
- namespace = body.Namespace
- name = body.Name
- }
- }
-
- if namespace == "" || name == "" {
- http.Error(w, "namespace and name are required", http.StatusBadRequest)
- return
- }
-
- if err := r.Deregister(req.Context(), namespace, name); err != nil {
- http.Error(w, fmt.Sprintf("deregistration failed: %v", err), http.StatusInternalServerError)
- return
- }
-
- w.WriteHeader(http.StatusOK)
- _ = json.NewEncoder(w).Encode(map[string]string{"status": "deregistered"})
-}
-
// handleListAgents handles HTTP GET requests to list all active, non-expired registered agents.
func (r *Registry) handleListAgents(w http.ResponseWriter, req *http.Request) {
agents := r.List()
diff --git a/internal/registry/registry_test.go b/internal/registry/registry_test.go
index 26d80ba..a56b09f 100644
--- a/internal/registry/registry_test.go
+++ b/internal/registry/registry_test.go
@@ -300,22 +300,6 @@ func TestRegistry_HTTPHandler(t *testing.T) {
if _, ok := reg.Get("tenant-1", "agent-http"); ok {
t.Fatalf("expected agent-http to be deregistered")
}
-
- // 5. Test legacy aliases (POST /register and DELETE /deregister)
- req = httptest.NewRequest(http.MethodPost, "/register", bytes.NewBufferString(regPayload))
- req.Header.Set("Content-Type", "application/json")
- w = httptest.NewRecorder()
- handler.ServeHTTP(w, req)
- if w.Code != http.StatusOK {
- t.Fatalf("POST /register legacy alias returned %d", w.Code)
- }
-
- req = httptest.NewRequest(http.MethodDelete, "/deregister?namespace=tenant-1&name=agent-http", nil)
- w = httptest.NewRecorder()
- handler.ServeHTTP(w, req)
- if w.Code != http.StatusOK {
- t.Fatalf("DELETE /deregister legacy alias returned %d", w.Code)
- }
}
func TestHTTPMCPClient_Initialize(t *testing.T) {
diff --git a/internal/rollout/promql.go b/internal/rollout/promql.go
index f767f3a..9c5d726 100644
--- a/internal/rollout/promql.go
+++ b/internal/rollout/promql.go
@@ -23,6 +23,7 @@ package rollout
import (
"context"
"fmt"
+ "strings"
"time"
agentraxv1alpha1 "github.com/gitcommitankit/agentrax/api/v1alpha1"
@@ -36,42 +37,81 @@ import (
// The label selectors match the canary Deployment's pod labels:
// - app.kubernetes.io/name=
// - agentrax.io/variant=canary (propagated from pod labels via ServiceMonitor)
-func requestCountQuery(adName, namespace string, window time.Duration) string {
+func requestCountQuery(adName, namespace string, window time.Duration) (string, error) {
+ d, err := promDuration(window)
+ if err != nil {
+ return "", err
+ }
return fmt.Sprintf(
`sum(increase(http_requests_total{namespace=%q,app_kubernetes_io_name=%q,agentrax_io_variant="canary"}[%s])) or vector(0)`,
- namespace, adName, promDuration(window),
- )
+ namespace, adName, d,
+ ), nil
}
// errorRateQuery returns a PromQL expression computing the fraction of 5xx
// responses out of total requests for the canary, over the given window.
// Returns 0 if no requests have been received (safe division).
-func errorRateQuery(adName, namespace string, window time.Duration) string {
- d := promDuration(window)
+func errorRateQuery(adName, namespace string, window time.Duration) (string, error) {
+ d, err := promDuration(window)
+ if err != nil {
+ return "", err
+ }
return fmt.Sprintf(
`(sum(increase(http_requests_total{namespace=%q,app_kubernetes_io_name=%q,agentrax_io_variant="canary",code=~"5.."}[%s])) or vector(0))`+
` / on() group_left `+
`clamp_min(sum(increase(http_requests_total{namespace=%q,app_kubernetes_io_name=%q,agentrax_io_variant="canary"}[%s])) or vector(0), 1)`,
namespace, adName, d,
namespace, adName, d,
- )
+ ), nil
}
// p99LatencyQuery returns a PromQL expression for the 99th-percentile request
// latency in milliseconds for the canary pods over the given window.
-func p99LatencyQuery(adName, namespace string, window time.Duration) string {
+func p99LatencyQuery(adName, namespace string, window time.Duration) (string, error) {
+ d, err := promDuration(window)
+ if err != nil {
+ return "", err
+ }
return fmt.Sprintf(
`histogram_quantile(0.99, sum by (le) (`+
`rate(http_request_duration_milliseconds_bucket{namespace=%q,app_kubernetes_io_name=%q,agentrax_io_variant="canary"}[%s])))`,
- namespace, adName, promDuration(window),
- )
+ namespace, adName, d,
+ ), nil
}
-// promDuration formats a time.Duration into the Prometheus duration string
+// promDuration formats a time.Duration into canonical Prometheus duration syntax
// understood by range selectors and the rate()/increase() functions.
-// e.g. 5*time.Minute → "5m0s".
-func promDuration(d time.Duration) string {
- return d.String()
+// e.g. 5*time.Minute → "5m", 1*time.Hour → "1h", 30*time.Second → "30s".
+// Returns an error for durations with sub-millisecond precision, which cannot
+// be accurately represented in PromQL range selectors.
+func promDuration(d time.Duration) (string, error) {
+ if d <= 0 {
+ return "0s", nil
+ }
+ if d%time.Second != 0 {
+ if d%time.Millisecond == 0 {
+ return fmt.Sprintf("%dms", d/time.Millisecond), nil
+ }
+ return "", fmt.Errorf("duration %v has sub-millisecond precision, which PromQL range selectors cannot represent", d)
+ }
+
+ hours := d / time.Hour
+ d -= hours * time.Hour
+ mins := d / time.Minute
+ d -= mins * time.Minute
+ secs := d / time.Second
+
+ var b strings.Builder
+ if hours > 0 {
+ b.WriteString(fmt.Sprintf("%dh", hours))
+ }
+ if mins > 0 {
+ b.WriteString(fmt.Sprintf("%dm", mins))
+ }
+ if secs > 0 {
+ b.WriteString(fmt.Sprintf("%ds", secs))
+ }
+ return b.String(), nil
}
// ── Evaluation ────────────────────────────────────────────────────────────────
@@ -114,7 +154,11 @@ func Evaluate(
policy := ad.Spec.Rollout.Rollback
// ── 1. Sample-size gate ───────────────────────────────────────────────────
- sampleCount, err := promClient.QueryScalar(ctx, requestCountQuery(name, ns, window))
+ reqCountQuery, err := requestCountQuery(name, ns, window)
+ if err != nil {
+ return EvaluationResult{}, fmt.Errorf("building request count query: %w", err)
+ }
+ sampleCount, err := promClient.QueryScalar(ctx, reqCountQuery)
if err != nil {
return EvaluationResult{}, fmt.Errorf("querying request count: %w", err)
}
@@ -132,7 +176,11 @@ func Evaluate(
}
// ── 2. Error rate threshold ───────────────────────────────────────────────
- errorRate, err := promClient.QueryScalar(ctx, errorRateQuery(name, ns, window))
+ errRateQuery, err := errorRateQuery(name, ns, window)
+ if err != nil {
+ return EvaluationResult{}, fmt.Errorf("building error rate query: %w", err)
+ }
+ errorRate, err := promClient.QueryScalar(ctx, errRateQuery)
if err != nil {
return EvaluationResult{}, fmt.Errorf("querying error rate: %w", err)
}
@@ -162,7 +210,11 @@ func Evaluate(
}
// ── 3. p99 latency threshold ──────────────────────────────────────────────
- p99Ms, err := promClient.QueryScalar(ctx, p99LatencyQuery(name, ns, window))
+ p99Query, err := p99LatencyQuery(name, ns, window)
+ if err != nil {
+ return EvaluationResult{}, fmt.Errorf("building p99 latency query: %w", err)
+ }
+ p99Ms, err := promClient.QueryScalar(ctx, p99Query)
if err != nil {
return EvaluationResult{}, fmt.Errorf("querying p99 latency: %w", err)
}
diff --git a/internal/rollout/promql_test.go b/internal/rollout/promql_test.go
index 003a5ae..f46593e 100644
--- a/internal/rollout/promql_test.go
+++ b/internal/rollout/promql_test.go
@@ -106,9 +106,18 @@ func TestQueryTemplates(t *testing.T) {
ad := makeAD("my-agent", "tenant-prod", "1%", 200)
window := 2 * time.Minute
- reqQuery := requestCountQuery(ad.Name, ad.Namespace, window)
- errQuery := errorRateQuery(ad.Name, ad.Namespace, window)
- p99Query := p99LatencyQuery(ad.Name, ad.Namespace, window)
+ reqQuery, err := requestCountQuery(ad.Name, ad.Namespace, window)
+ if err != nil {
+ t.Fatalf("requestCountQuery returned unexpected error: %v", err)
+ }
+ errQuery, err := errorRateQuery(ad.Name, ad.Namespace, window)
+ if err != nil {
+ t.Fatalf("errorRateQuery returned unexpected error: %v", err)
+ }
+ p99Query, err := p99LatencyQuery(ad.Name, ad.Namespace, window)
+ if err != nil {
+ t.Fatalf("p99LatencyQuery returned unexpected error: %v", err)
+ }
for name, q := range map[string]string{
"requestCount": reqQuery,
@@ -124,8 +133,8 @@ func TestQueryTemplates(t *testing.T) {
if !strings.Contains(q, `agentrax_io_variant="canary"`) {
t.Errorf("%s query missing agentrax_io_variant=canary selector: %q", name, q)
}
- if !strings.Contains(q, "[2m0s]") {
- t.Errorf("%s query missing duration window [2m0s]: %q", name, q)
+ if !strings.Contains(q, "[2m]") {
+ t.Errorf("%s query missing duration window [2m]: %q", name, q)
}
}
}
@@ -352,20 +361,45 @@ func TestEvaluate_ZeroErrors_Absent5xxSeries(t *testing.T) {
// ── promDuration helper ───────────────────────────────────────────────────────
-// TestPromDuration_Format verifies that promDuration formats durations correctly.
+// TestPromDuration_Format verifies that promDuration formats durations correctly into canonical Prometheus syntax.
func TestPromDuration_Format(t *testing.T) {
cases := []struct {
in time.Duration
want string
}{
- {5 * time.Minute, "5m0s"},
- {time.Hour, "1h0m0s"},
+ {5 * time.Minute, "5m"},
+ {time.Hour, "1h"},
{30 * time.Second, "30s"},
+ {90 * time.Second, "1m30s"},
+ {time.Hour + 15*time.Minute, "1h15m"},
+ {time.Hour + 30*time.Minute + 10*time.Second, "1h30m10s"},
+ {500 * time.Millisecond, "500ms"},
+ {0, "0s"},
}
for _, tc := range cases {
- got := promDuration(tc.in)
+ got, err := promDuration(tc.in)
+ if err != nil {
+ t.Errorf("promDuration(%v) returned unexpected error: %v", tc.in, err)
+ }
if got != tc.want {
t.Errorf("promDuration(%v) = %q, want %q", tc.in, got, tc.want)
}
}
}
+
+// TestPromDuration_RejectsSubMillisecond verifies that promDuration rejects
+// sub-millisecond durations (e.g. 100ns, 500µs) which cannot be accurately
+// represented in PromQL range selectors.
+func TestPromDuration_RejectsSubMillisecond(t *testing.T) {
+ cases := []time.Duration{
+ 100 * time.Nanosecond,
+ 500 * time.Microsecond,
+ 1*time.Millisecond + 100*time.Nanosecond,
+ }
+ for _, d := range cases {
+ got, err := promDuration(d)
+ if err == nil {
+ t.Errorf("promDuration(%v) = %q, expected an error for sub-millisecond precision", d, got)
+ }
+ }
+}
diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go
index bd5aa9a..31bbcb4 100644
--- a/test/e2e/e2e_test.go
+++ b/test/e2e/e2e_test.go
@@ -19,6 +19,7 @@ package e2e
import (
"fmt"
"os/exec"
+ "strings"
"time"
. "github.com/onsi/ginkgo/v2"
@@ -27,96 +28,318 @@ import (
"github.com/gitcommitankit/agentrax/test/utils"
)
-const namespace = "agentrax-system"
+const (
+ managerNamespace = "agentrax-system"
+ e2eNamespace = "tenant-e2e"
+ e2eQuotaName = "e2e-quota"
+)
-var _ = Describe("controller", Ordered, func() {
- BeforeAll(func() {
- By("installing prometheus operator")
- Expect(utils.InstallPrometheusOperator()).To(Succeed())
+var _ = Describe("Agentrax Operator End-to-End Suite", Ordered, func() {
+ var projectimage = "example.com/agentrax:v0.1.0"
- By("installing the cert-manager")
+ BeforeAll(func() {
+ By("installing cert-manager for webhook TLS")
Expect(utils.InstallCertManager()).To(Succeed())
By("creating manager namespace")
- cmd := exec.Command("kubectl", "create", "ns", namespace)
+ cmd := exec.Command("kubectl", "create", "ns", managerNamespace)
+ _, _ = utils.Run(cmd)
+
+ By("building the manager container image")
+ cmd = exec.Command("make", "docker-build", fmt.Sprintf("IMG=%s", projectimage))
+ _, err := utils.Run(cmd)
+ Expect(err).NotTo(HaveOccurred())
+
+ By("loading the manager image into the Kind cluster")
+ err = utils.LoadImageToKindClusterWithName(projectimage)
+ Expect(err).NotTo(HaveOccurred())
+
+ By("installing CRDs")
+ cmd = exec.Command("make", "install")
+ _, err = utils.Run(cmd)
+ Expect(err).NotTo(HaveOccurred())
+
+ By("deploying the controller-manager")
+ cmd = exec.Command("make", "deploy", fmt.Sprintf("IMG=%s", projectimage))
+ _, err = utils.Run(cmd)
+ Expect(err).NotTo(HaveOccurred())
+
+ By("waiting for controller-manager pod to be in Running phase")
+ Eventually(func() error {
+ cmd := exec.Command("kubectl", "get", "pods",
+ "-l", "control-plane=controller-manager",
+ "-n", managerNamespace,
+ "-o", "jsonpath={.items[*].status.phase}",
+ )
+ out, err := utils.Run(cmd)
+ if err != nil {
+ return err
+ }
+ if !strings.Contains(string(out), "Running") {
+ return fmt.Errorf("manager pod not yet running, status: %s", string(out))
+ }
+ return nil
+ }, 3*time.Minute, 2*time.Second).Should(Succeed())
+
+ By("creating tenant namespace for E2E tests")
+ cmd = exec.Command("kubectl", "create", "ns", e2eNamespace)
_, _ = utils.Run(cmd)
+
+ By("creating baseline TenantQuota")
+ quotaYAML := fmt.Sprintf(`
+apiVersion: agentrax.io/v1alpha1
+kind: TenantQuota
+metadata:
+ name: %s
+ namespace: %s
+spec:
+ maxAgents: 3
+ maxGPUs: 2
+ maxTotalReplicas: 10
+ maxReplicasPerAgent: 4
+`, e2eQuotaName, e2eNamespace)
+ Expect(kubectlApply(quotaYAML)).To(Succeed())
})
AfterAll(func() {
- By("uninstalling the Prometheus manager bundle")
- utils.UninstallPrometheusOperator()
+ By("cleaning up tenant namespace")
+ cmd := exec.Command("kubectl", "delete", "ns", e2eNamespace, "--ignore-not-found")
+ _, _ = utils.Run(cmd)
- By("uninstalling the cert-manager bundle")
- utils.UninstallCertManager()
+ By("undeploying controller-manager")
+ cmd = exec.Command("make", "undeploy")
+ _, _ = utils.Run(cmd)
By("removing manager namespace")
- cmd := exec.Command("kubectl", "delete", "ns", namespace)
+ cmd = exec.Command("kubectl", "delete", "ns", managerNamespace, "--ignore-not-found")
_, _ = utils.Run(cmd)
})
- Context("Operator", func() {
- It("should run successfully", func() {
- var controllerPodName string
- var err error
-
- // projectimage stores the name of the image used in the example
- var projectimage = "example.com/agentrax:v0.0.1"
-
- By("building the manager(Operator) image")
- cmd := exec.Command("make", "docker-build", fmt.Sprintf("IMG=%s", projectimage))
- _, err = utils.Run(cmd)
- ExpectWithOffset(1, err).NotTo(HaveOccurred())
-
- By("loading the the manager(Operator) image on Kind")
- err = utils.LoadImageToKindClusterWithName(projectimage)
- ExpectWithOffset(1, err).NotTo(HaveOccurred())
-
- By("installing CRDs")
- cmd = exec.Command("make", "install")
- _, err = utils.Run(cmd)
- ExpectWithOffset(1, err).NotTo(HaveOccurred())
-
- By("deploying the controller-manager")
- cmd = exec.Command("make", "deploy", fmt.Sprintf("IMG=%s", projectimage))
- _, err = utils.Run(cmd)
- ExpectWithOffset(1, err).NotTo(HaveOccurred())
-
- By("validating that the controller-manager pod is running as expected")
- verifyControllerUp := func() error {
- // Get pod name
-
- cmd = exec.Command("kubectl", "get",
- "pods", "-l", "control-plane=controller-manager",
- "-o", "go-template={{ range .items }}"+
- "{{ if not .metadata.deletionTimestamp }}"+
- "{{ .metadata.name }}"+
- "{{ \"\\n\" }}{{ end }}{{ end }}",
- "-n", namespace,
- )
-
- podOutput, err := utils.Run(cmd)
- ExpectWithOffset(2, err).NotTo(HaveOccurred())
- podNames := utils.GetNonEmptyLines(string(podOutput))
- if len(podNames) != 1 {
- return fmt.Errorf("expect 1 controller pods running, but got %d", len(podNames))
+ Context("Scenario 1: Core Reconciliation & Self-Healing", func() {
+ const agentName = "self-heal-agent"
+
+ It("creates child Deployment and self-heals upon out-of-band deletion", func() {
+ adYAML := fmt.Sprintf(`
+apiVersion: agentrax.io/v1alpha1
+kind: AgentDeployment
+metadata:
+ name: %s
+ namespace: %s
+spec:
+ image: gcr.io/google-containers/echoserver:1.4
+ port: 8080
+ tenantRef: %s
+ replicas:
+ min: 1
+ max: 2
+ metric: queueDepth
+ target: 50
+ rollout:
+ strategy: Recreate
+`, agentName, e2eNamespace, e2eQuotaName)
+
+ Expect(kubectlApply(adYAML)).To(Succeed())
+
+ By("waiting for child Deployment to be created")
+ Eventually(func() error {
+ cmd := exec.Command("kubectl", "get", "deployment", agentName, "-n", e2eNamespace)
+ _, err := utils.Run(cmd)
+ return err
+ }, time.Minute, time.Second).Should(Succeed())
+
+ By("waiting for child Service to be created")
+ Eventually(func() error {
+ cmd := exec.Command("kubectl", "get", "service", agentName, "-n", e2eNamespace)
+ _, err := utils.Run(cmd)
+ return err
+ }, 30*time.Second, time.Second).Should(Succeed())
+
+ By("deleting child Deployment out-of-band")
+ deleteCmd := exec.Command("kubectl", "delete", "deployment", agentName, "-n", e2eNamespace)
+ _, err := utils.Run(deleteCmd)
+ Expect(err).NotTo(HaveOccurred())
+
+ By("verifying operator self-heals by recreating the Deployment")
+ Eventually(func() error {
+ cmd := exec.Command("kubectl", "get", "deployment", agentName, "-n", e2eNamespace)
+ _, err := utils.Run(cmd)
+ return err
+ }, 30*time.Second, time.Second).Should(Succeed())
+ })
+ })
+
+ Context("Scenario 2: Multi-Tenancy Quota Enforcement", func() {
+ It("caps HPA max replicas and sets QuotaLimited condition when exceeding tenant ceiling", func() {
+ overQuotaAgent := fmt.Sprintf(`
+apiVersion: agentrax.io/v1alpha1
+kind: AgentDeployment
+metadata:
+ name: over-quota-agent
+ namespace: %s
+spec:
+ image: gcr.io/google-containers/echoserver:1.4
+ tenantRef: %s
+ replicas:
+ min: 1
+ max: 8
+ metric: queueDepth
+ target: 50
+`, e2eNamespace, e2eQuotaName)
+
+ Expect(kubectlApply(overQuotaAgent)).To(Succeed())
+
+ By("waiting for HPA to be created with quota-capped maxReplicas")
+ Eventually(func() string {
+ cmd := exec.Command("kubectl", "get", "hpa", "over-quota-agent", "-n", e2eNamespace,
+ "-o", "jsonpath={.spec.maxReplicas}")
+ out, err := utils.Run(cmd)
+ if err != nil {
+ return ""
}
- controllerPodName = podNames[0]
- ExpectWithOffset(2, controllerPodName).Should(ContainSubstring("controller-manager"))
-
- // Validate pod status
- cmd = exec.Command("kubectl", "get",
- "pods", controllerPodName, "-o", "jsonpath={.status.phase}",
- "-n", namespace,
- )
- status, err := utils.Run(cmd)
- ExpectWithOffset(2, err).NotTo(HaveOccurred())
- if string(status) != "Running" {
- return fmt.Errorf("controller pod in %s status", status)
+ return strings.TrimSpace(string(out))
+ }, 30*time.Second, time.Second).Should(Equal("4"), "HPA maxReplicas should be capped to maxReplicasPerAgent (4)")
+
+ By("verifying QuotaLimited condition is set to True")
+ Eventually(func() string {
+ cmd := exec.Command("kubectl", "get", "agentdeployment", "over-quota-agent", "-n", e2eNamespace,
+ "-o", `jsonpath={.status.conditions[?(@.type=="QuotaLimited")].status}`)
+ out, err := utils.Run(cmd)
+ if err != nil {
+ return ""
}
- return nil
- }
- EventuallyWithOffset(1, verifyControllerUp, time.Minute, time.Second).Should(Succeed())
+ return strings.TrimSpace(string(out))
+ }, 30*time.Second, time.Second).Should(Equal("True"))
+ })
+ })
+
+ Context("Scenario 3: Canary Rollout & Abort Rollback", func() {
+ const canaryAgentName = "canary-test-agent"
+
+ It("transitions through rollout and cleans up on manual abort", func() {
+ initialYAML := fmt.Sprintf(`
+apiVersion: agentrax.io/v1alpha1
+kind: AgentDeployment
+metadata:
+ name: %s
+ namespace: %s
+spec:
+ image: gcr.io/google-containers/echoserver:1.4
+ port: 8080
+ tenantRef: %s
+ replicas:
+ min: 1
+ max: 2
+ metric: queueDepth
+ target: 50
+ rollout:
+ strategy: Canary
+ steps:
+ - setWeight: 20
+ - pause: 60s
+ - setWeight: 100
+ rollback:
+ maxErrorRate: "1%%"
+ maxP99LatencyMs: 500
+ minRequestSample: 100
+`, canaryAgentName, e2eNamespace, e2eQuotaName)
+
+ Expect(kubectlApply(initialYAML)).To(Succeed())
+
+ By("waiting for initial stable Deployment to be created")
+ Eventually(func() error {
+ cmd := exec.Command("kubectl", "get", "deployment", canaryAgentName, "-n", e2eNamespace)
+ _, err := utils.Run(cmd)
+ return err
+ }, time.Minute, time.Second).Should(Succeed())
+ By("triggering canary rollout with image update and abort")
+ abortYAML := fmt.Sprintf(`
+apiVersion: agentrax.io/v1alpha1
+kind: AgentDeployment
+metadata:
+ name: %s
+ namespace: %s
+spec:
+ image: gcr.io/google-containers/echoserver:1.5
+ port: 8080
+ tenantRef: %s
+ replicas:
+ min: 1
+ max: 2
+ metric: queueDepth
+ target: 50
+ rollout:
+ strategy: Canary
+ abort: true
+ steps:
+ - setWeight: 20
+ - pause: 60s
+ - setWeight: 100
+ rollback:
+ maxErrorRate: "1%%"
+ maxP99LatencyMs: 500
+ minRequestSample: 100
+`, canaryAgentName, e2eNamespace, e2eQuotaName)
+
+ Expect(kubectlApply(abortYAML)).To(Succeed())
+
+ By("verifying stable Deployment remains available after abort")
+ Eventually(func() error {
+ cmd := exec.Command("kubectl", "get", "deployment", canaryAgentName, "-n", e2eNamespace)
+ _, err := utils.Run(cmd)
+ return err
+ }, 30*time.Second, time.Second).Should(Succeed())
+ })
+ })
+
+ Context("Scenario 4: Graceful Deletion & Finalizer Cleanup", func() {
+ const deleteAgentName = "delete-test-agent"
+
+ It("removes finalizer and cleans up all owned child resources", func() {
+ adYAML := fmt.Sprintf(`
+apiVersion: agentrax.io/v1alpha1
+kind: AgentDeployment
+metadata:
+ name: %s
+ namespace: %s
+spec:
+ image: gcr.io/google-containers/echoserver:1.4
+ port: 8080
+ tenantRef: %s
+ replicas:
+ min: 1
+ max: 2
+ metric: queueDepth
+ target: 50
+`, deleteAgentName, e2eNamespace, e2eQuotaName)
+
+ Expect(kubectlApply(adYAML)).To(Succeed())
+
+ By("waiting for AgentDeployment to exist")
+ Eventually(func() error {
+ cmd := exec.Command("kubectl", "get", "agentdeployment", deleteAgentName, "-n", e2eNamespace)
+ _, err := utils.Run(cmd)
+ return err
+ }, time.Minute, time.Second).Should(Succeed())
+
+ By("deleting the AgentDeployment")
+ cmd := exec.Command("kubectl", "delete", "agentdeployment", deleteAgentName, "-n", e2eNamespace)
+ _, err := utils.Run(cmd)
+ Expect(err).NotTo(HaveOccurred())
+
+ By("verifying AgentDeployment is fully deleted")
+ Eventually(func() bool {
+ cmd := exec.Command("kubectl", "get", "agentdeployment", deleteAgentName, "-n", e2eNamespace)
+ _, err := utils.Run(cmd)
+ return err != nil
+ }, time.Minute, 2*time.Second).Should(BeTrue())
})
})
})
+
+func kubectlApply(yaml string) error {
+ cmd := exec.Command("kubectl", "apply", "-f", "-")
+ cmd.Stdin = strings.NewReader(yaml)
+ _, err := utils.Run(cmd)
+ return err
+}
diff --git a/test/utils/utils.go b/test/utils/utils.go
index d49f72e..1bf115d 100644
--- a/test/utils/utils.go
+++ b/test/utils/utils.go
@@ -110,6 +110,13 @@ func LoadImageToKindClusterWithName(name string) error {
cluster := "kind"
if v, ok := os.LookupEnv("KIND_CLUSTER"); ok {
cluster = v
+ } else if ctxCmd := exec.Command("kubectl", "config", "current-context"); ctxCmd != nil {
+ if out, err := ctxCmd.Output(); err == nil {
+ ctxStr := strings.TrimSpace(string(out))
+ if strings.HasPrefix(ctxStr, "kind-") {
+ cluster = strings.TrimPrefix(ctxStr, "kind-")
+ }
+ }
}
kindOptions := []string{"load", "docker-image", name, "--name", cluster}
cmd := exec.Command("kind", kindOptions...)