docs(self-managed): publish 0.6.1-rc snapshot

docs/v0.6.1-rc/api.md

@@ -0,0 +1,205 @@
+# API
+
+This page provides a brief overview of the NVCF API. All API endpoints are served through your gateway. See [gateway-routing](./gateway-routing.md) for details on configuring your gateway domain and DNS.
+
+## OpenAPI Specification
+
+This page does not cover all endpoints.
+
+Please refer to the [OpenAPI Spec](https://api.nvcf.nvidia.com/v3/openapi) for the latest API information.
+
+<Note>
+The OpenAPI spec linked above documents the full NVCF API surface. Replace the hosted domain with your own gateway domain when making requests. See [gateway-routing](./gateway-routing.md) for your deployment's base URL.
+
+</Note>
+
+The NVCF API is divided into the following sets of APIs:
+
+| APIs                  | Usage                                                                                 |
+| --------------------- | ------------------------------------------------------------------------------------- |
+| Function Invocation   | Execution of a function that runs on a worker node. Usually an inference call.        |
+| Cluster Groups & GPUs | Defines endpoints to list Cluster Groups and GPUs as targets for function deployment. |
+| Function Management   | The creation, modification and deletion of functions                                  |
+| Function Deployment   | Endpoints for creating and managing function deployments.                             |
+| Task Management       | GPU-backed batch jobs via NVIDIA Cloud Tasks (NVCT).                                  |
+
+**API Versioning**
+
+All API endpoints include versioning in the path prefix.
+
+```bash
+/v2/nvcf
+```
+
+## Authorization
+
+Self-hosted NVCF uses three bearer credential types:
+
+| Credential | CLI source | Typical use |
+| --- | --- | --- |
+| `NVCF_TOKEN` | `nvcf-cli init` | Default CLI credential for management operations and self-hosted cluster management |
+| `NVCF_API_KEY` | `nvcf-cli api-key generate` | Default CLI credential for function invocation, function discovery, and queue status |
+| `NVCF_NVCT_API_KEY` | `nvcf-cli api-key generate` | Credential for task commands (`task create`, `task list`, etc.) |
+
+All three credential types are sent as bearer credentials:
+
+```bash
+Authorization: Bearer <credential>
+```
+
+Use the CLI for normal credential management:
+
+```bash
+nvcf-cli init
+nvcf-cli api-key generate
+```
+
+`nvcf-cli init` calls the API Keys admin endpoint and mints the JWT used as
+`NVCF_TOKEN`. The CLI saves the token in `~/.nvcf-cli.state` for the default
+configuration and reads it automatically on later commands. Export it only when
+you are making direct API calls or using tooling that reads environment
+variables:
+
+```bash
+export NVCF_TOKEN=$(jq -r .token ~/.nvcf-cli.state)
+```
+
+The default self-hosted admin issuer role gives this token the following
+scopes:
+
+- `register_function`
+- `list_functions`
+- `list_functions_details`
+- `deploy_function`
+- `update_function`
+- `update_secrets`
+- `delete_function`
+- `manage_telemetries`
+- `manage_registry_credentials`
+- `cluster-management`
+
+The function API key (`NVCF_API_KEY`) is created with these default scopes:
+
+- `invoke_function`
+- `list_functions`
+- `queue_details`
+- `list_functions_details`
+
+The task API key (`NVCF_NVCT_API_KEY`) is created with these default scopes:
+
+- `launch_task`
+- `list_tasks`
+- `task_details`
+- `cancel_task`
+- `delete_task`
+- `list_events`
+- `list_results`
+- `update_secrets`
+
+NVCF API authorization is scope-based. For the NVCF API endpoints below, either
+`NVCF_TOKEN` or `NVCF_API_KEY` can be used if that bearer includes the required
+scope. The CLI prefers `NVCF_API_KEY` for read, invoke, and queue commands. It
+prefers `NVCF_TOKEN` for management commands when both credentials are
+configured.
+
+Pass `--scopes` when you need to narrow a key or create a key with management
+scopes for testing.
+
+### Scope reference
+
+The OpenAPI spec describes the permission checked for each endpoint. "Accepted
+bearer" means which saved CLI credential type can be sent as
+`Authorization: Bearer <credential>` when it includes the listed scope.
+
+| Category | Method and endpoint | Accepted bearer | Scope |
+| --- | --- | --- | --- |
+| Function invocation | `POST /v2/nvcf/pexec/functions/{functionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `invoke_function` |
+| Function invocation | `POST /v2/nvcf/pexec/functions/{functionId}/versions/{versionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `invoke_function` |
+| Function invocation | `GET /v2/nvcf/pexec/status/{requestId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `invoke_function` |
+| Queue details | `GET /v2/nvcf/queues/{requestId}/position` | `NVCF_TOKEN` or `NVCF_API_KEY` | `queue_details` |
+| Queue details | `GET /v2/nvcf/queues/functions/{functionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `queue_details` |
+| Queue details | `GET /v2/nvcf/queues/functions/{functionId}/versions/{versionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `queue_details` |
+| Function management | `GET /v2/nvcf/functions` | `NVCF_TOKEN` or `NVCF_API_KEY` | `list_functions` or `list_functions_details` |
+| Function management | `GET /v2/nvcf/functions/ids` | `NVCF_TOKEN` or `NVCF_API_KEY` | `list_functions` or `list_functions_details` |
+| Function management | `POST /v2/nvcf/functions` | `NVCF_TOKEN` or `NVCF_API_KEY` | `register_function` |
+| Function management | `GET /v2/nvcf/functions/{functionId}/versions` | `NVCF_TOKEN` or `NVCF_API_KEY` | `list_functions` or `list_functions_details` |
+| Function management | `POST /v2/nvcf/functions/{functionId}/versions` | `NVCF_TOKEN` or `NVCF_API_KEY` | `register_function` |
+| Function management | `GET /v2/nvcf/functions/{functionId}/versions/{functionVersionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `list_functions` or `list_functions_details` |
+| Function management | `PUT /v2/nvcf/functions/{functionId}/versions/{functionVersionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `update_function` |
+| Function management | `DELETE /v2/nvcf/functions/{functionId}/versions/{functionVersionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `delete_function` |
+| Function management | `PUT /v2/nvcf/metadata/functions/{functionId}/versions/{functionVersionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `update_function` |
+| Function deployment | `GET /v2/nvcf/deployments/functions/{functionId}/versions/{functionVersionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `deploy_function` |
+| Function deployment | `POST /v2/nvcf/deployments/functions/{functionId}/versions/{functionVersionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `deploy_function` |
+| Function deployment | `PUT /v2/nvcf/deployments/functions/{functionId}/versions/{functionVersionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `deploy_function` |
+| Function deployment | `DELETE /v2/nvcf/deployments/functions/{functionId}/versions/{functionVersionId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `deploy_function` |
+| Function deployment | `GET /v2/nvcf/deployments/{deploymentId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `deploy_function` |
+| Function deployment | `PATCH /v2/nvcf/deployments/{deploymentId}/gpu-specifications/{gpuSpecId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `deploy_function` |
+| Registry credential management | `GET /v2/nvcf/registry-credentials` | `NVCF_TOKEN` or `NVCF_API_KEY` | `manage_registry_credentials` |
+| Registry credential management | `POST /v2/nvcf/registry-credentials` | `NVCF_TOKEN` or `NVCF_API_KEY` | `manage_registry_credentials` |
+| Registry credential management | `GET /v2/nvcf/registry-credentials/{registryCredentialId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `manage_registry_credentials` |
+| Registry credential management | `PATCH /v2/nvcf/registry-credentials/{registryCredentialId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `manage_registry_credentials` |
+| Registry credential management | `DELETE /v2/nvcf/registry-credentials/{registryCredentialId}` | `NVCF_TOKEN` or `NVCF_API_KEY` | `manage_registry_credentials` |
+| Registry credential management | `GET /v2/nvcf/recognized-registries` | `NVCF_TOKEN` or `NVCF_API_KEY` | `manage_registry_credentials` |
+
+Self-hosted cluster management uses SIS endpoints from the Spot API. In
+self-hosted CLI workflows, use `NVCF_TOKEN` with the `cluster-management` scope
+for these endpoints.
+
+| Method and endpoint | Accepted bearer | Scope |
+| --- | --- | --- |
+| `GET /v1/accounts/{ncaId}/clusters` | `NVCF_TOKEN` | `cluster-management` |
+| `POST /v1/accounts/{ncaId}/clusters` | `NVCF_TOKEN` | `cluster-management` |
+| `GET /v1/accounts/{ncaId}/clusters/{clusterId}` | `NVCF_TOKEN` | `cluster-management` |
+| `PUT /v1/accounts/{ncaId}/clusters/{clusterId}` | `NVCF_TOKEN` | `cluster-management` |
+| `DELETE /v1/accounts/{ncaId}/clusters/{clusterId}` | `NVCF_TOKEN` | `cluster-management` |
+| `GET /v1/accounts/{ncaId}/clusterVersions` | `NVCF_TOKEN` | `cluster-management` |
+
+Task management uses NVCT endpoints served through your gateway. All task
+endpoints require `NVCF_NVCT_API_KEY` with the listed scope.
+
+| Method and endpoint | Accepted bearer | Scope |
+| --- | --- | --- |
+| `POST /v1/nvct/tasks` | `NVCF_NVCT_API_KEY` | `launch_task` |
+| `GET /v1/nvct/tasks` | `NVCF_NVCT_API_KEY` | `list_tasks` |
+| `POST /v1/nvct/tasks/bulk` | `NVCF_NVCT_API_KEY` | `list_tasks` |
+| `GET /v1/nvct/tasks/{taskId}` | `NVCF_NVCT_API_KEY` | `task_details` |
+| `DELETE /v1/nvct/tasks/{taskId}` | `NVCF_NVCT_API_KEY` | `delete_task` |
+| `POST /v1/nvct/tasks/{taskId}/cancel` | `NVCF_NVCT_API_KEY` | `cancel_task` |
+| `GET /v1/nvct/tasks/{taskId}/events` | `NVCF_NVCT_API_KEY` | `list_events` |
+| `GET /v1/nvct/tasks/{taskId}/results` | `NVCF_NVCT_API_KEY` | `list_results` |
+| `PUT /v1/nvct/secrets/tasks/{taskId}` | `NVCF_NVCT_API_KEY` | `update_secrets` |
+
+### Direct API key creation
+
+The CLI calls the API Keys service for you. If you need to call it directly,
+set `expires_at` and the `scopes` array explicitly.
+
+```bash
+GATEWAY_ADDR=<your-gateway-address>
+EXPIRES_AT=$(date -u -v+1d '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -u -d '+1 day' '+%Y-%m-%dT%H:%M:%SZ')
+SERVICE_ID="nvidia-cloud-functions-ncp-service-id-aketm"
+
+curl -s -X POST "http://${GATEWAY_ADDR}/v1/keys" \
+  -H "Host: api-keys.${GATEWAY_ADDR}" \
+  -H "Content-Type: application/json" \
+  -H "Key-Issuer-Service: nvcf-api" \
+  -H "Key-Issuer-Id: ${SERVICE_ID}" \
+  -H "Key-Owner-Id: test@nvcf-api.local" \
+  -d '{
+    "description": "invocation key",
+    "expires_at": "'"${EXPIRES_AT}"'",
+    "authorizations": {
+      "policies": [{
+        "aud": "'"${SERVICE_ID}"'",
+        "auds": ["'"${SERVICE_ID}"'"],
+        "product": "nv-cloud-functions",
+        "resources": [
+          {"id": "*", "type": "account-functions"},
+          {"id": "*", "type": "authorized-functions"}
+        ],
+        "scopes": ["invoke_function", "list_functions", "queue_details", "list_functions_details"]
+      }]
+    },
+    "audience_service_ids": ["'"${SERVICE_ID}"'"]
+  }' | jq .
+```

docs/v0.6.1-rc/autoscaling/architecture.md

@@ -0,0 +1,64 @@
+# Function Autoscaler Architecture
+
+The function autoscaler is a Rust service deployed as a horizontally scaled Kubernetes Deployment. It reads utilization and request metrics from a Prometheus-compatible timeseries database, stores discovered functions and coordination state in Cassandra, and writes desired instance counts to the NVCF API. Cassandra lightweight transactions handle leader election and short-lived per-function locks.
+
+The work is split into two loops. A leader-elected discovery loop scans the timeseries database for active function versions and upserts them into Cassandra. A scaling loop runs on every replica, but each replica only handles the functions whose IDs hash into its assigned buckets, so the active set is sharded across replicas.
+
+## Sequence Diagram
+
+```mermaid
+sequenceDiagram
+    participant Workers as Workers / Invocation Services
+    participant TSDB as Time Series DB
+    participant Autoscaler as Function Autoscaler
+    participant Cassandra as Cassandra
+    participant NVCF as NVCF Service
+
+    Workers->>TSDB: Emit utilization and instance metrics
+
+    Note over Autoscaler,Cassandra: Discovery loop (~15s, leader-elected)
+    Autoscaler->>TSDB: Query active functions
+    TSDB-->>Autoscaler: Function set
+    Autoscaler->>Cassandra: Upsert newly discovered functions
+
+    Note over Autoscaler,NVCF: Scaling loop (~30s, per-bucket)
+    Autoscaler->>Cassandra: Read active functions for this node's buckets
+    Autoscaler->>TSDB: Query current instances and utilization history
+    TSDB-->>Autoscaler: Metrics
+    Autoscaler->>TSDB: Check recent invocations (scale-to-zero guard, PromQL)
+    Note over Autoscaler: Compute desired instance count
+    Autoscaler->>NVCF: PUT requiredNumberOfInstances
+    NVCF-->>Autoscaler: OK or error
+    Autoscaler->>Cassandra: Write predicted count, refresh function TTL
+```
+
+The discovery loop runs on one leader-elected replica. The scaling loop runs on every replica, but each replica only processes the function buckets assigned to it.
+
+## Timeseries Database
+
+The function autoscaler is a read-only client of a Prometheus-compatible timeseries store. It calls the `/api/v1/query_range` HTTP endpoint and uses PromQL for every metric query, so any backend that implements that interface works: upstream Prometheus, Thanos, Grafana Mimir, or VictoriaMetrics. The reference NVCF deployments point at VictoriaMetrics via the `timeseries_db_url` setting.
+
+The function autoscaler does not run a scrape config of its own and does not write samples. Before it can do anything useful, the rest of the data plane has to be feeding the same store:
+
+- Worker pods export utilization and instance count metrics (`nvcf_worker_service_worker_thread_busy_seconds_total`, `nvcf_worker_service_worker_thread_count_total`, instance gauges).
+- Invocation services and the gRPC proxy export request counters (`function_request`, `function_request_total`) labeled by `function_id`, `function_version_id`, and `nca_id`. These labels are how the discovery loop finds active function versions.
+
+For a self-hosted control plane, you need three things in place before bringing the function autoscaler online:
+
+1. A Prometheus-compatible store reachable from the function autoscaler pod.
+2. A scrape configuration (or remote-write feed) covering the worker pods and the invocation-plane services.
+3. The resulting query endpoint passed in as `timeseries_db_url`. The function autoscaler reports `not ready` on its readiness probe until that endpoint responds.
+
+## Coordination and Self-Healing
+
+Coordination relies on Cassandra TTLs to recover from failures without operator intervention:
+
+- The discovery lock self-expires if the leader replica crashes or is partitioned, so a new leader takes over on the next loop iteration.
+- Bucket ownership is recomputed when replicas join or leave. During a reshuffle a function may be skipped for a single scaling cycle or briefly picked up by a different replica, and a short-lived per-function lock prevents two replicas from racing on the same function in that window.
+- Each active function row carries a TTL refreshed by every scaling cycle, so functions that stop emitting metrics age out of the active set automatically.
+
+## See Also
+
+- [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds, factors, thresholds, and stickiness via the NVCF API.
+- [Function Autoscaler Operations](./operations.md) for health endpoints and common issues.
+- [Function Autoscaler Observability](./observability.md) for emitted metrics, traces, and logs.

docs/v0.6.1-rc/autoscaling/index.md

@@ -0,0 +1,37 @@
+# Function Autoscaling
+
+The NVCF Function Autoscaler is a distributed Rust service that monitors function utilization and uses it to determine the ideal instance count per function on the NVCF control plane. It runs as a horizontally scaled deployment on the same Kubernetes cluster as the rest of the control plane.
+
+On an interval, the function autoscaler reads metrics from the timeseries database, decides how many instances each function should have, and calls the NVCF API to apply that decision.
+
+The function autoscaler depends on a Prometheus-compatible timeseries database fed by the worker pods and invocation-plane services. Without it, the service reports `not ready` and makes no scaling decisions. See [Timeseries database](./architecture.md#timeseries-database) for the required metrics and endpoints.
+
+## Function Autoscaler vs Horizontal Pod Autoscaler
+
+Function autoscaling is distinct from Kubernetes horizontal pod autoscaling (HPA). HPA scales pods within a single cluster, so it cannot reach NVCF worker pods that are spread across multiple clusters. Function autoscaling orchestrates scaling across clusters using global load patterns.
+
+## Key Functionality
+
+- Discovers active functions from invocation and worker metrics in the timeseries database and persists the active set in Cassandra.
+- Periodically computes a desired instance count per function from recent utilization and the function's scaling policy.
+- Applies the desired count by calling the NVCF API's predictions endpoint.
+- Coordinates work across replicas using hash-based bucket assignment and Cassandra Lightweight Transaction (LWT) distributed locks.
+
+## Architecture Overview
+
+```mermaid
+flowchart LR
+    Workers[Workers / Invocation Services] --> TSDB[(Time Series DB)]
+    TSDB --> Autoscaler[Function Autoscaler]
+    Autoscaler <--> Cassandra[(Cassandra)]
+    Autoscaler --> NVCF[NVCF API]
+```
+
+See [Architecture](./architecture.md#sequence-diagram) for the end-to-end sequence diagram and the bucket model.
+
+## See Also
+
+- [Architecture](./architecture.md) for components, data flow, and the Cassandra LWT lock behavior that elects the discovery leader.
+- [Configure Autoscaling](../configure-autoscaling.md) for setting per-function scaling bounds, factors, thresholds, and stickiness via the NVCF API.
+- [Function Autoscaler Operations](./operations.md) for health endpoints and operational guidance.
+- [Function Autoscaler Observability](./observability.md) for the metrics, traces, and logs emitted by the service.