From a1989589fefdfb3647491a9181394a15c00490dd Mon Sep 17 00:00:00 2001 From: steiler Date: Mon, 11 May 2026 15:18:08 +0200 Subject: [PATCH] System Architecture --- README.md | 2 +- docs/index.md | 2 +- docs/system-architecture/architecture.md | 195 ++++++++ docs/system-architecture/cache.md | 159 +++++++ docs/system-architecture/config-server.md | 435 +++++++++++++++++- docs/system-architecture/data-server.md | 513 ++++++++++++++++++++++ docs/system-architecture/schema-server.md | 234 ++++++++++ 7 files changed, 1518 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 3417205..a064be0 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ Home of the code of the project homepage. ## Features - Schema's: [YANG][yang], others TBD -- Targets: Physical devices (`PNF`), Containers (`CNF`), Virtual Machines (`VNF`) and Dummy test target (`NOOP`) +- Targets: Physical devices (`PNF`), Containers (`CNF`), Virtual Machines (`VNF`) - Vendor agnostic - Flexible deployments: Small, medium or large-scale scaled-out deployments - Target Protocols: [gNMI][gnmi], [Netconf][netconf] diff --git a/docs/index.md b/docs/index.md index 0571cd5..4e8b924 100644 --- a/docs/index.md +++ b/docs/index.md @@ -47,7 +47,7 @@ The config-server is a Kubernetes-based Operator and comprises of several contro ## Features - Schema's: [YANG][yang], others TBD -- Targets: Physical devices (`PNF`), Containers (`CNF`), Virtual Machines (`VNF`) and `NOOP` (No Operation) +- Targets: Physical devices (`PNF`), Containers (`CNF`), Virtual Machines (`VNF`) - Vendor agnostic - Flexible deployments: Small, medium or large scale scaled out deployments - Target Protocols: [gNMI][gnmi], [Netconf][netconf] diff --git a/docs/system-architecture/architecture.md b/docs/system-architecture/architecture.md index e69de29..e6be61d 100644 --- a/docs/system-architecture/architecture.md +++ b/docs/system-architecture/architecture.md @@ -0,0 +1,195 @@ +# System Architecture + +## Overview + +SDC is a Kubernetes-native, model-driven network configuration system. It bridges the +Kubernetes resource model to real network devices by translating Kubernetes Custom Resources +into vendor-specific southbound protocol operations (gNMI, NETCONF). Operators define +desired device configuration as Kubernetes objects; SDC resolves intent priority conflicts, +validates all changes against YANG schemas, and pushes the resulting configuration to +devices using a confirmed-commit pattern. + +The system is YANG-first: every configuration path and value is validated against a parsed +YANG schema before it reaches a device. Schemas are loaded from Git repositories and stored +in an embedded schema store (persistent or in-memory mode), then served to all components +that need type or namespace information. This validation layer makes it impossible to push +schema-invalid configuration to a device. + +SDC is delivered as a full Kubernetes-native platform, with the control plane exposing a +standard Kubernetes API surface where operators use `kubectl` and familiar KRM workflows. +Some components (for example `data-server`) can also run standalone via gRPC for +non-Kubernetes integrations. Under the hood, config-server translates KRM operations into +gRPC calls to data-server, which in turn drives devices over gNMI or NETCONF. + +--- + +## High-Level Architecture + +```mermaid +flowchart TD + subgraph Operator + kubectl["kubectl / KRM client"] + end + + subgraph K8s["Kubernetes Control Plane"] + kapi["kube-apiserver\n(etcd)"] + apiserver["api-server\n(config.sdcio.dev resources\nBadger store)"] + controller_central["controller\n(central deployment)\nDiscoveryRule · ConfigSet\nWorkspace · Rollout"] + controller_col["controller\n(colocated StatefulSet)\nSchema · Target · TargetDatastore\nTargetConfig · TargetRecovery"] + end + + subgraph DataPlane["Data-Server Pod (StatefulSet, one per target group)"] + dataserver["data-server\n:56000\nDataServer + SchemaServer gRPC"] + schemalib["schema-server lib\n(embedded, Badger /schemadb)"] + cachelib["cache lib\n(embedded, emptyDir /cached/caches)"] + sbi["SBI Drivers\ngNMI · NETCONF"] + dataserver --- schemalib + dataserver --- cachelib + dataserver --- sbi + end + + subgraph Devices["Network Devices"] + dev1["Device A\n(gNMI)"] + dev2["Device B\n(NETCONF)"] + end + + kubectl -->|"kubectl apply\nConfig / ConfigSet"| apiserver + kubectl -->|"kubectl apply\nSchema / DiscoveryRule / Target"| kapi + apiserver -->|"TransactionSet / Confirm"| dataserver + kapi <-->|"List/Watch CRDs"| controller_central + kapi <-->|"List/Watch CRDs"| controller_col + controller_col -->|"CreateDataStore\nTransactionSet / Confirm"| dataserver + controller_col -->|"CreateSchema\nGetSchemaDetails"| dataserver + sbi -->|"gNMI SetRequest"| dev1 + sbi -->|"NETCONF edit-config"| dev2 +``` + +--- + +## Northbound Interface + +Operators interact with SDC entirely through Kubernetes resources. There are two API +groups: + +**`inv.sdcio.dev`** — inventory and infrastructure resources stored in etcd via standard +CRDs. These include [`Schema`](../user-guide/configuration/schemas.md) (YANG model references), +[`DiscoveryRule`](../user-guide/configuration/discovery/introduction.md) (IP scanning +configuration), +[`TargetConnectionProfile`](../user-guide/configuration/target-profiles/connection-profile.md), +[`TargetSyncProfile`](../user-guide/configuration/target-profiles/sync-profile.md), +[`Subscription`](../user-guide/configuration/subscription/subscription.md), +[`Workspace`](config-server.md#workspace-reconciler), and [`Rollout`](config-server.md#rollout-reconciler). + +**`config.sdcio.dev`** — configuration resources stored in a local Badger database via an +aggregated API server. This group contains [`Target`](../user-guide/configuration/target/target.md) +(managed device endpoint), +[`Config`](../user-guide/configuration/config/config.md) and +[`ConfigSet`](../user-guide/configuration/config/configset.md) (desired device configuration +blobs), [`SensitiveConfig`](../user-guide/configuration/config/sensitiveconfig.md) +(credential-aware intent), [`Deviation`](../user-guide/deviation.md) (detected drift from intended state), +[`RunningConfig`](../getting-started/basic-usage.md) (read-only live device state), and +[`ConfigBlame`](../getting-started/basic-usage.md) (per-path intent ownership). The +aggregated server bypasses etcd deliberately: individual Config objects can be megabytes in +size when representing a full device configuration, exceeding etcd's 1.5 MB per-object +limit. + +The Kubernetes-native design means operators get familiar tools for free: `kubectl apply`, +`kubectl diff`, `kubectl dry-run`, RBAC, admission webhooks, and GitOps workflows all work +without modification. + +### Embedded vs Standalone Ports + +In the default deployment, schema-server and cache are embedded libraries inside +`data-server`, so config-server talks to both `DataServer` and `SchemaServer` APIs through +the data-server endpoint (`:56000` in this documentation set). The `:55000` (schema-server) +and `:50100` (cache) ports are standalone service defaults used only when those components are +deployed out-of-process. + +--- + +## Southbound Interface + +Devices are reached through one of two embedded SBI drivers compiled directly into the +`data-server` binary. There is no external driver process or plugin system. + +| Driver | Protocol | Library | +|--------|----------|---------| +| `gnmi` | gNMI over gRPC/TLS | `github.com/openconfig/gnmic` | +| `netconf` | NETCONF over SSH (RFC 6241) | `github.com/scrapli/scrapligo` | + +The driver type is selected per-target at datastore creation time from the +[`TargetConnectionProfile.protocol`](../user-guide/configuration/target-profiles/connection-profile.md) +field. Both drivers implement the same [`Target`](data-server.md) interface (`Get`, +`Set`, `AddSyncs`, `Status`, `Close`), making the rest of data-server protocol-agnostic. + +Sync goroutines continuously update the in-memory `syncTree` with live device state. +For gNMI targets, data-server can use streaming subscriptions (ON_CHANGE, SAMPLE), +periodic `Get` polling, and repeated ONCE subscriptions. For NETCONF targets, it polls +state using periodic `` requests. This synchronized live view is used to +compute configuration deviations, provide validation context for subsequent +[`TransactionSet`](data-server.md) operations, and serve +[`RunningConfig`](../getting-started/basic-usage.md#retrieve-configuration) data through +the API. + +--- + +## Data Flow: Configuration Change Lifecycle + +The sequence below traces a `kubectl apply` of a `Config` object through to a device ACK. + +```mermaid +sequenceDiagram + participant Op as kubectl + participant AS as api-server
(config.sdcio.dev) + participant DS as data-server
(:56000) + participant SC as schema-server lib
(embedded) + participant CA as cache lib
(embedded) + participant Dev as Device
(gNMI / NETCONF) + + Op->>AS: kubectl apply Config + Note over AS: Create/Update handler (dry-run aware) + AS->>DS: TransactionSet(intents, dryRun=false) + DS->>DS: Lock datastore mutex + DS->>CA: Load existing intents + DS->>SC: Validate paths and values + Note over DS: Resolve intent priorities + DS->>Dev: Set() — edit-config / SetRequest + Dev-->>DS: ACK + DS->>CA: Persist new intent blob + DS->>CA: Optimistic writeback of "running" snapshot (cache) + DS->>DS: Start rollback timer (default 5 min) + DS-->>AS: TransactionSetResponse (success) + AS->>DS: TransactionConfirm + DS->>DS: Cancel rollback timer + DS-->>AS: TransactionConfirmResponse + AS->>AS: Persist Config in Badger
(update status.appliedConfig) + AS-->>Op: 201 Created / 200 OK +``` + +If `TransactionConfirm` is not sent within the auto-cancel timeout, data-server +automatically rolls back by re-running `TransactionSet` with the previous intent content, +returning the device to its prior state. + +In the default deployment, this persisted `"running"` snapshot lives in the cache on an +ephemeral `emptyDir` volume, so it is rebuilt after restart by replaying intents. + +The `"running"` snapshot written here is data-server's local post-transaction view, not a +fresh state report pulled from the device. data-server applies the same accepted edits to +its in-memory running tree and persists that expected result. This gives subsequent +`TransactionSet` operations and deviation calculations an immediate baseline without +waiting for a fresh device poll/stream update. Normal sync then continues, and any drift is +reconciled on subsequent device sync cycles. + +--- + +## Component Summary + +| Component | Role | Stateful? | Key Protocol | +|-----------|------|-----------|-------------| +| **api-server** | Aggregated K8s API server for `config.sdcio.dev`; stores Config blobs in Badger; calls data-server on every write | Yes (Badger) | HTTPS (K8s aggregated API), gRPC to data-server | +| **controller (central)** | Runs DiscoveryRule, ConfigSet, Workspace, Rollout reconcilers | No | Kubernetes watch, gNMI (discovery) | +| **controller (colocated)** | Runs Schema, Target, TargetDatastore, TargetConfig, TargetRecovery, Subscription reconcilers; co-deployed with data-server | No | Kubernetes watch, gRPC to data-server and schema-server | +| **data-server** | Southbound agent: owns one Datastore per target; embeds schema-server and cache libraries; pushes config to devices | Yes (Badger schema DB, ephemeral intent cache) | gRPC (DataServer + SchemaServer), gNMI, NETCONF | +| **schema-server** (embedded) | Parses YANG modules; stores schema objects in Badger; serves type/namespace info | Yes (Badger) | In-process (or gRPC `:55000` if split out) | +| **cache** (embedded) | Intent-blob persistence; one instance per target device | Ephemeral (`emptyDir`) | In-process (or gRPC `:50100` if split out) | +| **SBI drivers** | Protocol adapters (gNMI, NETCONF) compiled into data-server | No | gNMI over gRPC, NETCONF over SSH | diff --git a/docs/system-architecture/cache.md b/docs/system-architecture/cache.md index e69de29..f0b2014 100644 --- a/docs/system-architecture/cache.md +++ b/docs/system-architecture/cache.md @@ -0,0 +1,159 @@ +# Cache + +## Overview + +The cache is a multi-instance, intent-blob store. Each cache instance corresponds to one datastore (target device) and holds a set of named intent blobs — serialised [`tree_persist.Intent`](https://github.com/sdcio/sdc-protos/blob/main/tree_persist.proto) protobuf messages. + +The cache is a **pure persistence layer**. It has no understanding of config tree structure, priority resolution, or candidate/running separation. All of that logic lives in data-server's `pkg/tree` package. In production, data-server embeds the cache directly via the `LocalCache` adapter (an in-process Go call), making the standalone gRPC server unnecessary for the current deployment. + +A standalone gRPC server is provided for out-of-process or experimental deployments but is not used by default. + +--- + +## Data Model: Config Trees + +### Cache instances + +One cache instance maps to one datastore. The instance name equals the datastore name. Instances are managed by `pkg/cache.Cache` and kept in an internal `map[string]*cacheInstance`. + +### Intent blobs + +Each named intent is stored as a raw `[]byte` — the result of `proto.Marshal` applied to a `tree_persist.Intent` message. A blob contains: + +- `intent_name` — string key (e.g. `"my-interface-config"`, `"running"`) +- `priority` — int32; lower value = higher precedence +- All path+value updates belonging to this intent, encoded as a tree + +There is no partial update: every write replaces the entire blob for that intent name. + +--- + +## Intent Ownership & Priority + +Priority is an `int32` embedded in the `tree_persist.Intent` blob. The cache stores this value as-is without interpretation. Intent ownership, priority semantics, and conflict resolution are managed by data-server, not the cache. + +--- + +## Write Path + +All writes originate in data-server. + +1. data-server computes new intent content in `pkg/tree` using a deep-copied candidate tree derived from `syncTree`. +2. Calls `ops.TreeExport(root.Entry, intentName, priority, onlyIntended)` to serialise the intent subtree to a `tree_persist.Intent` proto. +3. Calls `cacheClient.IntentModify(ctx, intent)`. +4. `LocalCache.InstanceIntentModify` marshals the proto: `proto.Marshal(intent)` → `[]byte`. +5. `cache.InstanceIntentModify(ctx, name, bytes)` → `store.IntentModify(ctx, intentName, bytes)` → `os.WriteFile(...)`. + + +### Running-config writeback + +After every successful `TransactionSet`, data-server calls `writeBackSyncTree`: + +1. Exports the entire `syncTree` as intent `"running"` via `ops.TreeExport`. +2. Writes it to cache with `IntentModify("running", ...)`. + +This keeps the persisted `"running"` blob aligned with the committed in-memory tree for fast follow-up operations (for example replace-intent baselining); cold-start still rebuilds state from user intents and excludes `"running"` from bulk load. + +--- + +## Read Path & Subscriptions + +### Cold-start / datastore startup + +On datastore initialisation, data-server calls `LoadAllButRunningIntents`: + +1. `cacheClient.IntentGetAll(ctx, []string{"running"}, intentChan, errChan)` — streams all intent blobs except `"running"`. +2. Each blob is unmarshalled (`proto.Unmarshal` → `tree_persist.Intent`) and loaded into a working tree (deep-copied from `syncTree`) via `root.ImportConfig()` using `treeproto.NewProtoTreeImporter`. + +The `"running"` blob is excluded from the bulk load because the running config is recomputed from the deep-copied `syncTree` after the loaded user intents have been imported into it. + +### Replace-intent transactions + +Before applying a replace-intent, data-server calls `cacheClient.IntentGet(ctx, "running")` to retrieve the current running snapshot. This is used to populate the base tree so the replace diff can be computed correctly. + +--- + +## Persistence + +### Filesystem store (default) + +Implemented in `pkg/store/filesystem/filesystem.go`. + +``` +/ + / + intents/ + / + data ← raw proto bytes (tree_persist.Intent) +``` + +- One directory per intent; the file is always named `data`. +- Writes use `os.WriteFile` under a mutex. There is no write-ahead log. +- Delete removes the entire `` directory via `os.RemoveAll`. + + +### Volume and durability + +The `cache` volume in the data-server StatefulSet is an **`emptyDir`**. All persisted intent blobs are lost on pod restart. + +On cold start, config-server detects the empty state and re-applies all `Config` and `ConfigSet` resources. This drives data-server to rebuild every intent blob, restoring the cache to the pre-restart state. + +--- + +## Configuration & Deployment + +### Embedded (data-server) configuration + +| Key | Default | Description | +|---|---|---| +| `cache.type` | `local` | `local` — embedded via `LocalCache`; `remote` — standalone gRPC server | +| `cache.store-type` | `filesystem` | `filesystem` or `badgerdb` | +| `cache.dir` | `./cached/caches` | Base path for the filesystem store | +| `cache.address` | `:50100` | Address of the remote cache server (only used when `type: remote`) | + +### Standalone server flags + +Relevant when running the cache as an independent process: + +| Flag | Default | Description | +|---|---|---| +| `--config` / `-c` | `cache.yaml` | Path to config file | +| `--debug` / `-d` | — | Enable debug log level | +| `--trace` / `-t` | — | Enable trace log level | +| `--version` / `-v` | — | Print version and exit | + +The standalone gRPC service listens on `:50100` (overridable via `grpc-server.address`). + +### Adapter layer (data-server) + +data-server never calls `pkg/cache.Cache` directly. It uses two adapter types: + +- **`LocalCache`** (`data-server/pkg/cache/local.go`) — implements the `Client` interface by calling `cache.Cache` in-process. Responsible for `proto.Marshal` / `proto.Unmarshal` of [`tree_persist.Intent`](https://github.com/sdcio/sdc-protos/blob/main/tree_persist.proto). +- **`CacheClientBound`** (`data-server/pkg/cache/cacheClientBound.go`) — wraps `Client` with a fixed `cacheName`, exposing short-form methods (`IntentGet`, `IntentModify`, etc.) that datastore code calls directly. + +--- + +## Interactions + +| Direction | Component | Mechanism | Purpose | +|---|---|---|---| +| Inbound | data-server | Direct Go calls via `LocalCache` → `cache.Cache` | All intent read / write / delete operations | +| Inbound (standalone) | External clients | gRPC `:50100` | Standalone mode — not used in current deployment | +| Outbound | Filesystem | `os.WriteFile` / `os.ReadFile` | Durable intent storage at `/cached/caches/` | +| Outbound (optional) | BadgerDB | `badger.DB.*` | Alternative storage backend — not default | + +### gRPC API reference ([`cache.proto`](https://github.com/sdcio/cache/blob/main/proto/cache.proto)) + +| RPC | Description | +|---|---| +| `InstanceCreate` | Create a new cache instance | +| `InstanceDelete` | Delete a cache instance and all its intents | +| `InstancesList` | List all cache instance names | +| `InstanceIntentsList` | List all intent names in an instance | +| `InstanceIntentGet` | Retrieve a single intent blob by name | +| `InstanceIntentModify` | Write or overwrite a single intent blob | +| `InstanceIntentDelete` | Delete a single intent by name | +| `InstanceIntentExists` | Check whether an intent exists | +| `InstanceIntentsGetAll` | Stream all intents, optionally excluding named ones | + +`InstanceClose` and `InstanceExists` exist in data-server's `Client` interface but are adapter-only methods — they have no corresponding gRPC RPC definition. diff --git a/docs/system-architecture/config-server.md b/docs/system-architecture/config-server.md index f3a9be7..bc67a83 100644 --- a/docs/system-architecture/config-server.md +++ b/docs/system-architecture/config-server.md @@ -1,34 +1,429 @@ -# Config server system architecture +# Config Server -The config-server comprises 6 essential components: +## Overview -* `Schema Reconciler`: Manages the lifecycle of a `yang` schema via the `schema` Custom Resource Definition (CRD). -* `DiscoveryRule Reconciler`: Oversees the lifecycle of the `discoveryRule` CRD. -* `TargetDatastore Reconciler`: Controls the lifecycle of the `datastore` within the data-server. -* `TargetConfigServer Reconciler`: Orchestrates the lifecycle of the `config` KRM resource in response to target state changes. -* `TargetConfigSet Server Reconciler`: Orchestrates the lifecycle of the `configSet` KRM resource in response to target state changes. -* `Config Server`: Coordinates the lifecycle of the `config` and `configSet` KRM resources in the data-server +The config-server is the Kubernetes-native northbound entry point for sdcio. It exposes network configuration via Kubernetes Custom Resources and an aggregated API server, drives discovery of network devices, manages YANG schema ingestion, and orchestrates the application of configuration intents to data-server targets. -## Schema Reconciler +The config-server does **not** communicate directly with network devices. All device interaction is delegated to the [data-server](data-server.md) over gRPC. -The Schema Reconciler is tasked with managing `yang` schemas in the schema-server through the `schema` CRD. This reconciler handles the addition and deletion of `yang` schemas. Notably, the schema CRD remains immutable to simplify updates. It assumes that yang schemas are validated offline before integration into the system. Upon adding a `schema` CR, the reconciler downloads the referenced git repository, extracts the referenced Schema files and loads the Schema into the schema-server. Deleting a schema CR results in the deletion of the corresponding schema from the schema-server. The reconciler employs the `READY` condition to signal the reconciliation status of the `schema` CR. +## Architecture -## Discovery Reconciler +The config-server consists of two binaries compiled from the same repository, deployed as three distinct Kubernetes workloads. -The Discovery Reconciler is responsible for managing the lifecycle of the `discoveryRule` CRD. It monitors the availability and alterations of referenced profiles in the `discoveryRule` CR. Additionally, it initiates or halts a discovery goroutine for each `discoveryRule` CR, regardless of whether discovery is `enabled` or `disabled`. Based on the discovery outcomes, it manages the lifecycle of the respective `target` CR. A successful discovery results in the creation of a `target` CR with a `READY` condition set to `True`. +| Binary | K8s Workload | Role | Key env vars | +|--------|-------------|------|--------------| +| `controller` | `Deployment/controller` | Runs DiscoveryRule, ConfigSet, Workspace, Rollout reconcilers | `ENABLE_DISCOVERYRULE`, `ENABLE_CONFIGSET`, `ENABLE_WORKSPACE`, `ENABLE_ROLLOUT` | +| `controller` | `StatefulSet/data-server-controller` (colocated with data-server) | Runs Schema, Target, TargetDatastore, TargetConfig, TargetRecoveryConfig, Subscription reconcilers | `LOCAL_DATASERVER=true`, `ENABLE_SCHEMA`, `ENABLE_TARGET`, `ENABLE_TARGETDATASTORE`, `ENABLE_TARGETCONFIG`, `ENABLE_TARGETRECOVERYCONFIG`, `ENABLE_SUBSCRIPTION` | +| `api-server` | `Deployment/api-server` | Aggregated Kubernetes API server for `config.sdcio.dev` resources stored in Badger | `--tls-cert-file`, `--tls-private-key-file`, `--secure-port=6443` | -## TargetDatastore Reconciler +**Reconciler enabling mechanism:** At startup, the controller binary calls `IsReconcilerEnabled(name)`, which reads `ENABLE_`. Any value other than `"false"` activates the reconciler. -The TargetDatastore Reconciler oversees the lifecycle of the datastore of a target within the [`Data-Server`](../system-architecture/data-server.md). It creates or deletes a datastore in the `Data-Server` based on updates to the `target` CR or changes in the target state within the datastore. Its status is reflected in the `DATASTORE` condition (`DSReady` in yaml / json) of the `target` CR and should be `READY` in case of normal operation. +**`LOCAL_DATASERVER` mode:** When `LOCAL_DATASERVER=true`, the controller binary creates a `TargetManager` that maintains gRPC connections to the local data-server sidecar on `localhost:`. This is required for the TargetDatastore, TargetConfig, TargetRecoveryConfig, and Subscription reconcilers. -## TargetConfigServer Reconciler +**Kubernetes API extension strategy:** Two complementary extension points are used: -The TargetConfigServer Reconciler manages the lifecycle of the `config` KRM resources based on `target` transitions. To ensure consistent results when a target transitions from `NotReady` to `Ready`, it reapplies the original configurations before handling new ones. Consequently, the reconciler reapplies previously applied configs, and upon successful completion, declares the `CONFIG` condition (`ConfigReady` in json / yaml output) state with reason `Ready` as `True` in the `target` CR . +- **Aggregated API Server** (`api-server` binary, `config.sdcio.dev` group): backed by Badger (`.WithoutEtcd()`), registered via an `APIService` object. Used because Config blobs can be megabytes in size and etcd enforces a 1.5 MB per-object limit. +- **Standard CRD controller** (`controller` binary, `inv.sdcio.dev` group): controller-runtime watches on CRDs backed by etcd in the normal way. -## TargetConfigSetServer Reconciler +**Persistent volume layout (by workload):** -The TargetConfigSetServer Reconciler oversees the lifecycle of the `configSet` KRM resources based on `target` transitions. It updates the status of the `configSet` KRM resource when a `target` CR changes state. +| PVC | Mount path | Consumer workload | Contents | +|-----|-----------|-------------------|----------| +| `pvc-schema-store` | `/schemas` | colocated controller + data-server StatefulSet | Raw YANG files git-cloned by the Schema Reconciler | +| `pvc-schema-db` | `/schemadb` | data-server StatefulSet | Badger DB of parsed schema proto objects | +| `pvc-config-store` | `/config` | api-server Deployment | Badger DB storing Config, ConfigSet, Target, Deviation objects | +| `pvc-workspace-store` | `/workspace` | central controller Deployment | Git workspace for Workspace / Rollout reconcilers | -## Config Resources +The following diagram shows the high-level component topology: -The Config resources are implemented as an aggregated API server, as Config resources may exceed the constraints of the etcd storage limits. It manages the `config` and `configSet` KRM resources based on the `target` `READY` conditions and communicates with the data-server through the intent RPC(s). \ No newline at end of file +```mermaid +flowchart TD + subgraph k8s["Kubernetes Cluster"] + kapi["kube-apiserver\n(etcd-backed CRDs)"] + + subgraph central["Deployment: controller"] + dr["DiscoveryRule Reconciler"] + cs["ConfigSet Reconciler"] + ws["Workspace Reconciler"] + ro["Rollout Reconciler"] + end + + subgraph apiserver["Deployment: api-server"] + agg["Aggregated API Server\nconfig.sdcio.dev\n(Badger)"] + end + + subgraph colocated["StatefulSet: data-server-controller"] + schema["Schema Reconciler"] + tgt["Target Reconciler"] + tds["TargetDatastore Reconciler"] + tc["TargetConfig Reconciler"] + trc["TargetRecovery Reconciler"] + sub["Subscription Reconciler"] + ds["data-server\n(gRPC)"] + end + end + + kubectl["kubectl / client"] -->|"config.sdcio.dev resources"| agg + kubectl -->|"inv.sdcio.dev CRDs"| kapi + kapi -->|"List/Watch"| central + kapi -->|"List/Watch"| colocated + agg -->|"TransactionSet / Confirm"| ds + schema -->|"CreateSchema / DeleteSchema"| ds + tds -->|"CreateDataStore / DeleteDataStore"| ds + tc -->|"TransactionSet / Confirm"| ds + trc -->|"TransactionSet / Confirm"| ds + dr -->|"gNMI CapabilityRequest / GetRequest"| devices["Network Devices"] + ds -->|"gNMI / NETCONF"| devices +``` + +## CRDs & KRM Resources + +### `inv.sdcio.dev` — Standard CRDs (etcd-backed) + +| Kind | Namespaced | Purpose | Key spec fields | +|------|-----------|---------|-----------------| +| `Schema` | Yes | YANG model reference | `provider`, `version`, `repositories[]{repoURL, ref, dirs[], schema{models, includes, excludes}}` | +| `DiscoveryRule` | Yes | IP-range device discovery | `prefixes[]`, `addresses[]`, `defaultSchema`, `discoveryProfile{connectionProfiles[]}`, `period`, `concurrentScans` | +| `DiscoveryVendorProfile` | Yes | gNMI discovery response parsing | `gnmi{organization, modelMatch, paths[]{key, path, script, regex}, encoding}` | +| `TargetConnectionProfile` | Yes | Protocol and auth parameters for a device | `protocol`, `port`, `encoding`, `preferredNetconfVersion`, `commitCandidate`, `insecure`, `skipVerify` | +| `TargetSyncProfile` | Yes | Sync subscription configuration | `validate`, `buffer`, `workers`, `sync[]{name, protocol, paths[], mode, interval}` | +| `Subscription` | Yes | Telemetry subscriptions | `target.targetSelector`, `protocol`, `subscriptions[]{paths[], mode, interval}` | +| `Workspace` | Yes | Git repository for config templates | `repoURL`, `ref`, `credentials` | +| `Rollout` | Yes | Bulk config push from a workspace | `repoURL`, `strategy`, `skipUnavailableTarget` | + +### `config.sdcio.dev` — Aggregated API Server (Badger-backed) + +| Kind | Namespaced | Purpose | Key spec fields | +|------|-----------|---------|-----------------| +| `Target` | Yes | A managed network device | `provider`, `address`, `credentials`, `tlsSecret`, `connectionProfile`, `syncProfile` | +| `Config` | Yes | Desired device configuration intent | `lifecycle`, `priority` (int32), `revertive`, `config[]{path, value}` | +| `SensitiveConfig` | Yes | Config intent containing credentials | Same as `Config` (credential-aware RBAC) | +| `ConfigSet` | Yes | Config template applied to multiple targets via label selector | `target.targetSelector`, `priority`, `config[]{path, value}` | +| `Deviation` | Yes | Detected config drift (computed, not user-supplied) | — | +| `RunningConfig` | Yes | Live device state snapshot (read-only) | — | +| `ConfigBlame` | Yes | Per-path intent ownership (read-only) | — | + +## Reconcilers + +All reconcilers are implemented using `sigs.k8s.io/controller-runtime`. Each reconciler registers itself via `reconcilers.Register(name, &reconciler{})` in an `init()` function. The controller binary iterates `reconcilers.Reconcilers` at startup and calls `SetupWithManager` only if `IsReconcilerEnabled(name)` returns true. + +### Schema Reconciler + +**Source:** `pkg/reconcilers/schema/reconciler.go` + +**Watches:** `invv1alpha1.Schema`, `corev1.Secret` + +**Purpose:** Downloads YANG files from Git repositories, then loads them into the schema-server via gRPC. + +On reconcile, the reconciler first ensures a finalizer is present. It calls `schemaLoader.GetRef()` to check whether the YANG directory already exists on the shared PVC. If it does not, it sets a `Loading` condition and calls `schemaLoader.Load()`, which iterates `spec.repositories`, uses `github.com/go-git/go-git/v5` to clone each repository to a temporary directory, then copies the extracted paths to `///`. + +Once files are present, the reconciler creates an ephemeral gRPC client to the schema-server. It calls `GetSchemaDetails` to check the current state. If the schema is missing or in a `FAILED` state, it calls `CreateSchema` with the model, include, and exclude path lists. On deletion, it calls `GetSchemaDetails` + `DeleteSchema` and removes the YANG files from disk via `schemaLoader.DelRef()`. + +The schema-server address resolves in order: `SDC_SCHEMA_SERVER` env → `SDC_DATA_SERVER` env → `schema-server.sdc-system.svc.cluster.local:56000`. The schema base path is set by `SDC_SCHEMA_SERVER_BASE_DIR`, defaulting to `/schemas`. + +**External calls:** +- `SchemaServer/GetSchemaDetails` → schema-server +- `SchemaServer/CreateSchema{Schema, File[], Directory[], Exclude[]}` → schema-server +- `SchemaServer/DeleteSchema` → schema-server + +**Conditions updated:** `Ready` (true on success), `Failed` (on error), `Loading` (transient during download) — all on the `Schema` CR. + +--- + +### DiscoveryRule Reconciler + +**Source:** `pkg/reconcilers/discoveryrule/reconciler.go` + +**Watches:** `invv1alpha1.DiscoveryRule`, `TargetConnectionProfile`, `TargetSyncProfile`, `corev1.Secret` + +**Purpose:** Manages a long-running discovery goroutine per `DiscoveryRule` CR that periodically scans IP ranges and creates or updates `Target` CRs based on probe results. + +On reconcile, the reconciler validates the spec and resolves referenced profiles into a normalized `DiscoveryRuleConfig`. If a goroutine is already running and `HasChanged()` returns false, the reconciler returns immediately. If the config has changed, the old goroutine is stopped and a new `dr.Run(baseCtx)` goroutine is started. `dr.Run()` loops at `spec.period` intervals across all configured prefixes and addresses. + +Per-host discovery flow: + +- **Static (discovery disabled):** If `spec.defaultSchema != nil`, the reconciler uses `Protocol_NONE` and creates a `Target` CR directly without probing the device. +- **gNMI:** Dials the target, sends a `CapabilityRequest`, and matches the returned `organization` string against `DiscoveryVendorProfile.gnmi.organization`. Then sends a `GetRequest` for each configured path (version, hostname, platform, serial, MAC). Raw values are processed by applying regex patterns and optionally a Starlark `transform(value)` function defined in `DiscoveryVendorProfile.gnmi.paths[].script` (using `go.starlark.net`). +- **NETCONF discovery:** Not yet implemented; only the static path is available. + +**External calls:** +- `gNMI/CapabilityRequest` → network device +- `gNMI/GetRequest` → network device + +**Conditions updated:** `Ready` on `DiscoveryRule` CR; `TargetDiscoveryReady=True` on created/updated `Target` CRs. + +--- + +### Target Reconciler + +**Source:** `pkg/reconcilers/target/reconciler.go` + +**Watches:** `configv1alpha1.Target` + +**Purpose:** Roll-up reconciler that computes the aggregate `Ready` condition on a `Target` CR by evaluating four sub-conditions in sequence. + +`GetOverallStatus(target)` evaluates the conditions in priority order: `TargetDiscoveryReady` → `TargetDatastoreReady` → `TargetConfigRecoveryReady` → `TargetConnectionReady`. The first condition that is false sets `Ready=False` with a specific message indicating which gate failed. If the aggregate condition changed and the target is not ready, the reconciler requeues after 5 seconds. + +On deletion, the reconciler removes the `Deviation` CR associated with the target and removes the finalizer. + +**External calls:** None. + +**Conditions updated:** `Ready` (aggregate) on `Target` CR. + +--- + +### TargetDatastore Reconciler + +**Source:** `pkg/reconcilers/targetdatastore/reconciler.go` + +**Watches:** `configv1alpha1.Target`, `TargetConnectionProfile`, `TargetSyncProfile`, `corev1.Secret`, `invv1alpha1.Schema` + +**Purpose:** Creates and destroys the data-server `Datastore` object for each target. Requires `LOCAL_DATASERVER=true`. + +The reconciler first checks whether `TargetDiscoveryReady` is true. If not, it calls `targetMgr.ClearDesired()` and returns. It then checks whether the referenced `Schema` CR is in a `Ready` state; if not, it requeues after 10 seconds. + +When ready to proceed, it builds a `CreateDataStoreRequest` from the target spec, resolved profiles, and referenced secrets, then computes a deterministic hash over the request. `targetMgr.ApplyDesired(ctx, targetKey, dsReq, usedRefs, hash)` drives the datastore lifecycle asynchronously. The reconciler reads runtime status back from `targetMgr.GetOrCreate(targetKey).Status()` to set the condition. + +On deletion, it calls `targetMgr.ClearDesired()` + `targetMgr.Delete()` and removes the finalizer. + +**External calls:** +- `DataServer/CreateDataStore` → data-server +- `DataServer/DeleteDataStore` → data-server +- `DataServer/GetDataStore` → data-server + +**Conditions updated:** `TargetDatastoreReady` on `Target` CR. + +--- + +### TargetConfig Reconciler + +**Source:** `pkg/reconcilers/targetconfig/reconciler.go` + +**Watches:** `configv1alpha1.Target`, `configv1alpha1.Config` + +**Purpose:** Applies all `Config` CRs for a target whenever the target is ready and the datastore context reports it has been recovered. + +The reconciler checks `target.IsReady()` — if false, it sets `TargetForConfigReady=Failed` on all `Config` CRs for this target and requeues after 5 seconds. It then checks `dsctx.Status.Recovered` — if false, it requeues after 5 seconds. + +When both conditions are met, `transactor.Transact()` lists all `Config` CRs in the same namespace carrying the label `config.sdcio.dev/targetName`. Each config is classified as `update` (spec changed or not yet applied), `delete` (deletion timestamp present), or `noChange` (applied shasum matches spec shasum). A `TransactionSet` is issued with all intents, followed by `TransactionConfirm` on success. Each `Config` CR's status is updated with `appliedConfig` and `lastKnownGoodSchema`. + +On target deletion, the reconciler sets `TargetForConfigReady=Failed` on all associated `Config` CRs. + +**External calls:** +- `DataServer/TransactionSet` → data-server +- `DataServer/TransactionConfirm` → data-server + +**Conditions updated:** `TargetForConfigReady` on `Target` CR; `ConfigReady` on each `Config` CR. + +--- + +### TargetRecoveryConfig Reconciler + +**Source:** `pkg/reconcilers/targetrecovery/reconciler.go` + +**Watches:** `configv1alpha1.Target` + +**Purpose:** Re-applies all previously-committed `Config` intents after a data-server restart, before normal reconciliation resumes. + +The reconciler retrieves the `dsctx` from `targetMgr`. If it is nil, it requeues after 5 seconds. If `dsctx.Status.Recovered == true`, there is nothing to do and the reconciler returns. + +`transactor.RecoverConfigs()` lists all `Config` CRs with `status.appliedConfig != nil` and builds `TransactionIntent` entries with `PreviouslyApplied: true` for each. It issues a `TransactionSet` with `transactionID="recovery"`, `DryRun=false`, and `Timeout=120s`, which re-installs all intents both onto the device and into the data-server cache. On success, `TransactionConfirm` is sent and `dsctx.MarkRecovered(true)` is called. + +**External calls:** +- `DataServer/TransactionSet` (recovery transaction) → data-server +- `DataServer/TransactionConfirm` → data-server + +**Conditions updated:** `TargetConfigRecoveryReady` on `Target` CR; `ConfigReady` on recovered `Config` CRs. + +--- + +### ConfigSet Reconciler + +**Source:** `pkg/reconcilers/configset/reconciler.go` + +**Watches:** `configv1alpha1.ConfigSet`, `configv1alpha1.Target` + +**Purpose:** Expands a `ConfigSet` into individual `Config` CRs — one per target matching the label selector — keeping them synchronized as targets come and go. + +`unrollDownstreamTargets()` lists all `Target` CRs in the namespace matching `spec.target.targetSelector`. `ensureConfigs()` creates or updates a `Config` CR named `.` for each matched target, carrying the same `spec.config` payload and `spec.priority`. Config CRs for targets that no longer match are deleted. + +On deletion of a `ConfigSet`, the reconciler lists and deletes all owned child `Config` CRs and removes the finalizer only once all children have been garbage-collected. The per-target reconciliation status is surfaced in `ConfigSetStatus.Targets[]`. + +**External calls:** None. + +**Conditions updated:** `Ready` on `ConfigSet` CR; per-target `TargetForConfigReady` entries in `status.targets[]`. + +--- + +### Subscription Reconciler + +**Source:** `pkg/reconcilers/subscription/reconciler.go` + +**Watches:** `invv1alpha1.Subscription`, `configv1alpha1.Target` + +**Purpose:** Registers and deregisters telemetry subscriptions on data-server targets based on `Subscription` CRs. + +On delete, `targetMgr.RemoveSubscription()` is called. Otherwise, `targetMgr.ApplySubscription(ctx, subscription)` registers the subscription across all targets matching `spec.target.targetSelector`. + +**External calls:** Indirect — via `TargetManager` which manages gRPC connections to the local data-server. + +**Conditions updated:** `Ready` on `Subscription` CR. + +--- + +### Workspace & Rollout Reconcilers + +**Workspace Reconciler** + +**Source:** `pkg/reconcilers/workspace/reconciler.go` + +**Watches:** `invv1alpha1.Workspace` + +**Purpose:** Clones a Git repository to a local workspace directory (`///`) for use by the Rollout reconciler. Uses `github.com/go-git/go-git/v5`. Tracks the deployed commit in `status.deployedRef`. + +**External calls:** Git clone/fetch over HTTPS or SSH. + +**Conditions updated:** `Ready` on `Workspace` CR. + +--- + +**Rollout Reconciler** + +**Source:** `pkg/reconcilers/rollout/reconciler.go` + +**Watches:** `invv1alpha1.Rollout` + +**Purpose:** Applies a set of `Config` CRs sourced from a Workspace Git repository to a set of targets in a coordinated transaction. Iterates Config files from the cloned workspace, builds `TransactionIntent` entries, and applies them via the Transactor. The `networkWideTransaction` strategy is defined but not yet fully implemented. + +**External calls:** +- `DataServer/TransactionSet` → data-server (via Transactor) +- `DataServer/TransactionConfirm` → data-server (via Transactor) + +**Conditions updated:** `Ready` on `Rollout` CR; `status.targets[]{name, conditions}`. + +## Config & ConfigSet: Aggregated API Server + +`Config`, `SensitiveConfig`, and related read-only resources (`RunningConfig`, `ConfigBlame`, `Deviation`) are served by the `api-server` binary using `github.com/henderiw/apiserver-builder` with Badger v4 as the backing store (`github.com/henderiw/apiserver-store`). The storage root is set by `SDC_CONFIG_DIR`, defaulting to `/config`. + +### Dry-run hooks + +For `Config` and `SensitiveConfig`, `DryRunCreateFn` / `DryRunUpdateFn` / `DryRunDeleteFn` hooks in `apis/config/handlers/confighandler.go` execute **before** writing to Badger: + +- **Normal apply** (`dryrun=false`): calls `TransactionSet` + `TransactionConfirm` on the data-server, applying the intent to the device immediately. On success, `AppliedConfig` and `LastKnownGoodSchema` are populated in the status. +- **Dry-run** (`kubectl apply --dry-run=server`): runs a validation transaction on the data-server with no device change. + +### Intent mapping + +| Config field | Maps to in `TransactionSetRequest` | +|-------------|-----------------------------------| +| `metadata.namespace + metadata.name` | `TransactionIntent.Intent` = `"."` | +| `spec.priority` | `TransactionIntent.Priority` | +| `spec.config[]{path, value}` | `TransactionIntent.Update[]` | +| `spec.revertive == false` | `TransactionIntent.NonRevertive = true` | +| label `config.sdcio.dev/targetName` on `Config` CR | `TransactionSetRequest.DatastoreName` | + +### Delete intent + +`DryRunDeleteFn` sends `TransactionIntent{Intent: name, Delete: true}` to the data-server. The data-server removes the intent blob from its cache and recomputes the device's effective running configuration. + +## Target Lifecycle + +### Conditions + +| Condition | Set by | True means | +|-----------|--------|-----------| +| `TargetDiscoveryReady` | DiscoveryRule Reconciler | Device discovered; `Target` CR populated with `discoveryInfo` | +| `TargetDatastoreReady` | TargetDatastore Reconciler | Datastore created on data-server; device connection being established | +| `TargetConfigRecoveryReady` | TargetRecoveryConfig Reconciler | All previously-applied `Config` intents re-applied after restart | +| `TargetConnectionReady` | `TargetRuntime.pushConnIfChanged()` in `pkg/sdc/target/manager/runtime.go` | data-server has an active protocol connection to the device | +| `Ready` | Target Reconciler (roll-up) | All four conditions above are true | + +### State transitions + +```mermaid +stateDiagram-v2 + [*] --> Discovered : DiscoveryRule probes device\n(gNMI CapabilityRequest matches) + + Discovered --> DatastoreCreating : TargetDiscoveryReady=True\nSchema CR Ready + + DatastoreCreating --> DatastoreReady : CreateDataStore gRPC succeeds\n(TargetDatastoreReady=True) + + DatastoreReady --> Connecting : data-server dials device + + Connecting --> Connected : Protocol connection established\n(TargetConnectionReady=True) + + Connected --> Recovering : No prior AppliedConfig\nOR data-server restarted\n(TargetConfigRecoveryReady=False) + + Recovering --> Ready : TransactionSet(PreviouslyApplied=true)\n+ TransactionConfirm\n(TargetConfigRecoveryReady=True) + + Ready --> ConfigApplied : TargetConfig Reconciler\ntransacts all Config CRs + + ConfigApplied --> Ready : Config CRs in sync + + Ready --> NotReady : Any condition becomes False\n(Ready=False, requeue 5s) + + NotReady --> Recovering : Target becomes ready again\ndsctx.Recovered=false + + NotReady --> DatastoreCreating : TargetDatastoreReady lost\n(e.g. data-server restart) +``` + +**Fresh start sequence:** + +1. DiscoveryRule probes device → `TargetDiscoveryReady=True` → `Target` CR created with `discoveryInfo`. +2. TargetDatastore checks Schema CR ready → `CreateDataStore` gRPC → `TargetDatastoreReady=True`. +3. data-server establishes protocol connection → `TargetRuntime.pushConnIfChanged()` → `TargetConnectionReady=True`. +4. No prior `AppliedConfig` → TargetRecoveryConfig marks recovered immediately → `TargetConfigRecoveryReady=True`. +5. All four conditions true → `Ready=True` → TargetConfig reconciler applies all `Config` CRs. + +**After data-server restart:** + +1. `dsctx.Status.Recovered=false` → `TargetConfigRecoveryReady=False` → `Ready=False`. +2. TargetRecoveryConfig: `TransactionSet(PreviouslyApplied=true)` with all `Config` CRs where `status.appliedConfig != nil`. +3. `TransactionConfirm` → `dsctx.MarkRecovered(true)` → `TargetConfigRecoveryReady=True` → `Ready=True`. + +**When a target goes NotReady:** + +The TargetConfig Reconciler sets `TargetForConfigReady=Failed` on all associated `Config` CRs but does **not** delete them. When the target becomes ready again, the reconciler re-transacts all intents. + +## Configuration & Deployment + +### `api-server` binary + +| Setting | Source | +|---------|--------| +| Badger config store path | `SDC_CONFIG_DIR` env, default `/config` | +| TLS certificate | `--tls-cert-file` flag | +| TLS private key | `--tls-private-key-file` flag | +| Secure port | `--secure-port=6443` flag | +| Metrics port | `METRIC_PORT` env | +| pprof port | `PPROF_PORT` env → binds to `127.0.0.1:` | + +### `controller` binary + +| Setting | Source | +|---------|--------| +| Schema base directory | `SDC_SCHEMA_SERVER_BASE_DIR` env, default `/schemas` | +| Workspace directory | `SDC_WORKSPACE_DIR` env, default `/workspace` | +| Reconciler enable flags | `ENABLE_` env per reconciler | +| Local data-server mode | `LOCAL_DATASERVER=true` env | +| Data-server address | `SDC_DATA_SERVER` env, or service DNS | +| Schema-server address | `SDC_SCHEMA_SERVER` → `SDC_DATA_SERVER` → `schema-server.sdc-system.svc.cluster.local:56000` | +| Max concurrent reconciles | 16 (hardcoded) | +| Metrics port | `METRIC_PORT` env | + +## Interactions + +| Direction | Peer | gRPC Service / RPC | Purpose | +|-----------|------|--------------------|---------| +| controller → schema-server | schema-server (in data-server pod) | `SchemaServer/CreateSchema` | Load YANG files after git download | +| controller → schema-server | schema-server | `SchemaServer/GetSchemaDetails` | Check if schema is already loaded | +| controller → schema-server | schema-server | `SchemaServer/DeleteSchema` | Remove schema when `Schema` CR is deleted | +| controller → data-server | data-server (local sidecar) | `DataServer/CreateDataStore` | Create datastore when target becomes ready | +| controller → data-server | data-server | `DataServer/DeleteDataStore` | Remove datastore when target is deleted | +| controller → data-server | data-server | `DataServer/TransactionSet` | Apply `Config` intents to device | +| controller → data-server | data-server | `DataServer/TransactionConfirm` | Commit transaction | +| api-server → data-server | data-server | `DataServer/TransactionSet` | Dry-run validation and actual apply on `kubectl create/apply` | +| api-server → data-server | data-server | `DataServer/TransactionConfirm` | Commit after successful apply | +| controller ← kube-apiserver | kube-apiserver | Kubernetes List/Watch | React to CRD and resource changes | +| api-server ← kube-apiserver | kube-apiserver | Aggregated API forwarding | `kubectl` calls for `config.sdcio.dev` resources | +| controller → network devices | via openconfig/gnmic | `gNMI/CapabilityRequest`, `gNMI/GetRequest` | Discovery of network devices | \ No newline at end of file diff --git a/docs/system-architecture/data-server.md b/docs/system-architecture/data-server.md index e69de29..090b842 100644 --- a/docs/system-architecture/data-server.md +++ b/docs/system-architecture/data-server.md @@ -0,0 +1,513 @@ +# Data Server + +## Overview + +`data-server` is the central southbound agent in the SDC stack. It owns one `Datastore` per managed target device and is responsible for: + +- **Intent priority resolution** — merging multiple [Config](../user-guide/configuration/config/config.md)/[ConfigSet](../user-guide/configuration/config/configset.md) resources into a single, ordered config view +- **YANG validation** — enforcing schema constraints before any change reaches a device +- **Device push** — applying the resolved configuration via an embedded SBI driver (gNMI or NETCONF) +- **Deviation detection** — continuously comparing the intended state with the running state on the device + +In the default production deployment, `data-server` embeds both the schema-server library (YANG parsing + Badger persistence) and the cache library (intent blob store) in-process, making it the single runtime process for all three components. + +### Statefulness + +| State | Owner | Persistent? | +|-------|-------|-------------| +| `syncTree` (`*tree.RootEntry`) | `Datastore` struct, in-memory | No — lost on pod restart | +| Intent blobs | cache library (filesystem, `emptyDir` volume) | No — lost on pod restart | +| Running intent blob | cache (written as `"running"` after each `TransactionSet`) | No — lost on pod restart | +| Parsed YANG schema objects | schema-server library (Badger, `pvc-schema-db` PVC) | Yes | +| Active gRPC server state | `server.Server` struct | No | +| Active transaction (at most one per datastore) | `TransactionManager` | No | + +On pod restart, the cache (`emptyDir`) is empty and `syncTree` contains no intents. The schema DB on `pvc-schema-db` survives the restart; YANG schemas do not need re-parsing. The config-server (colocated controller) detects the empty state via target status conditions and re-applies all [Config](../user-guide/configuration/config/config.md)/[ConfigSet](../user-guide/configuration/config/configset.md) resources, driving [`TransactionSet`](https://github.com/sdcio/sdc-protos/blob/main/data.proto) calls that rebuild every intent blob. + +--- + +## System-Reserved Intents and Priority Model + +Every intent carries a priority value (int32); **lower numeric value = higher precedence**. When multiple intents set the same leaf path, the one with the lowest priority wins. Intents are merged in-memory by the `syncTree` and exported to cache as serialised [`tree_persist.Intent`](https://github.com/sdcio/sdc-protos/blob/main/tree_persist.proto) blobs. + +### User Intents + +User intents are created via [Config](../user-guide/configuration/config/config.md) and [ConfigSet](../user-guide/configuration/config/configset.md) Kubernetes resources. Their priority can be set explicitly; if omitted, it defaults to a system-assigned value. User intents must have a priority ≤ `UserSettableMax` (`MaxInt32 - 500 = 2147483147`), ensuring they remain in the user-reserved range below system-reserved priorities. + +### System-Reserved Intent Names + +The following intent names are reserved for data-server use: + +| Intent Name | Priority | Purpose | Lifecycle | +|---|---|---|---| +| `"running"` | `MaxInt32 - 100` | Aggregate running config snapshot, recalculated after every `TransactionSet` | Temporary; not loaded on cold-start | +| `"replace"` | `MaxInt32 - 101` | Placeholder for replace-intent transactions (temporary delta) | Transaction-scoped | +| `"revrun"` | `MaxInt32 - 102` | Revert-running: captures the running config state at transaction start for rollback | Transaction-scoped | +| `"default"` | `MaxInt32 - 103` | YANG defaults supplied by the schema-server | Persistent; loaded on cold-start | + +All system intents cluster near `MaxInt32`, so they act as low-precedence fallback values and lose to user intents on path conflicts. + +### Priority Resolution at Read Time + +When a leaf is read from the `syncTree`, the `LeafVariants.GetHighestPrecedence()` method compares all values across all intents and returns the one with the **lowest numeric priority number**. The caller never sees competing values; only the winning intent's data is exposed. + +--- + +## Architecture Diagram (Mermaid) + +```mermaid +graph TD + CS[config-server] -->|gRPC DataServer :56000| DS[data-server] + CS -->|gRPC SchemaServer :56000| DS + + subgraph data-server process + DS --> SRV[pkg/server\ngRPC listener] + SRV --> DST[pkg/datastore\nDatastore map] + DST --> TM[TransactionManager] + DST --> TREE[pkg/tree\nsyncTree RootEntry] + DST --> TGT[pkg/datastore/target\nTarget interface] + DST --> SCH[pkg/schema\nSchemaClientBound] + DST --> CACHE[pkg/cache\nCacheClientBound] + + SCH -->|local| SCHLIB[schema-server lib\nBadger /schemadb] + CACHE -->|local| CACHELIB[cache lib\nfilesystem /cached/caches] + end + + TGT -->|gNMI over gRPC/TLS| GNMI[Device gNMI] + TGT -->|NETCONF over SSH| NC[Device NETCONF] + + SCHLIB -->|pvc-schema-db| PVC[(PVC)] + CACHELIB -->|emptyDir| EMPTYDIR[(emptyDir)] +``` + +--- + +## Datastore Lifecycle + +A `Datastore` instance maps 1:1 to a managed target device. It is created by the `CreateDataStore` RPC and torn down by `DeleteDataStore`. + +### `CreateDataStore` + +1. Validate the request (name, schema reference, SBI type/address/port). +2. Build `*config.DatastoreConfig` from the proto fields. +3. Call `datastore.New(ctx, cfg, schemaClient, cacheClient)`: + - Creates `SchemaClientBound` bound to the schema's vendor/name/version. + - Allocates a `TreeContext` and `syncTreeRoot` (`tree.NewTreeRoot`). + - Creates `CacheClientBound` bound to the cache instance named after the datastore. + - Creates `TransactionManager` with a `DatastoreRollbackAdapter`. + - Calls `initCache(ctx)` (blocking) — creates the cache instance on disk if it does not yet exist. + - Starts a goroutine `connectSBI(ctx)` — blocking connect loop; once the SBI connection succeeds it starts the `DeviationMgr` goroutine. +4. Registers the datastore in `DatastoreMap`. + +**`Datastore` key fields:** + +```go +type Datastore struct { + config *config.DatastoreConfig + cacheClient cache.CacheClientBound + sbi target.Target + schemaClient schemaClient.SchemaClientBound + syncTree *tree.RootEntry + syncTreeMutex *sync.RWMutex + dmutex *sync.Mutex // serialises transactions (one at a time) + deviationClients map[DataServer_WatchDeviationsServer]string + transactionManager *types.TransactionManager + taskPool *pool.SharedTaskPool +} +``` + +### `DeleteDataStore` + +1. Look up the datastore in `DatastoreMap`. +2. Call `ds.Delete(ctx)` → `d.cacheClient.InstanceDelete(ctx)` — removes all intent files from disk. +3. Call `ds.Stop(ctx)` — cancels the datastore context and closes the SBI target connection. +4. Remove the entry from `DatastoreMap`. + +--- + +## Configuration Intent Lifecycle + +The intent lifecycle follows a **confirmed-commit** pattern: every `TransactionSet` call arms a rollback timer. The operator (config-server) must send `TransactionConfirm` before the timer expires, or the change is automatically reverted. + +### Phase A — `TransactionSet` + +Entry point: `pkg/server/transaction.go` → `pkg/datastore/transaction_rpc.go` + +```mermaid +sequenceDiagram + participant CS as config-server + participant SRV as pkg/server + participant DS as Datastore + participant SYNC as syncTree (committed) + participant CAND as candidate tree (deep copy) + participant VAL as validation + participant SBI as Target (device) + participant CACHE as CacheClient + + CS->>SRV: TransactionSet(intents) + SRV->>DS: dmutex.TryLock() + DS->>DS: Register transaction in TransactionManager + loop For each TransactionIntent + DS->>DS: ExpandAndConvertIntent (schema validation) + DS->>SYNC: Deep-copy syncTree (RLock) + DS->>CACHE: LoadAllButRunningIntents → import into candidate tree + DS->>CAND: Mark old owner leaves with OwnerDeleteMarker + DS->>CAND: Add new updates with flagNew + end + DS->>CAND: FinishInsertionPhase (defaults, keys, leaf-list order) + DS->>VAL: Validate (YANG semantic + type/mandatory checks) + alt dry_run == true + DS-->>CS: Return computed diff, no device write + else + DS->>SBI: sbi.Set(ctx, source) + SBI-->>DS: ACK + DS->>CACHE: IntentModify (write intent blobs) + DS->>SYNC: writeBackSyncTree (apply accepted deltas to syncTree) + write "running" blob + DS->>DS: StartRollbackTimer (transaction-timeout) + DS-->>CS: TransactionSetResponse + end +``` + +**Step-by-step:** + +1. **Lock datastore** via `d.dmutex.TryLock()`. Returns `ErrDatastoreLocked` if another transaction is active. +2. **Register transaction** in `TransactionManager` (at most one active at a time). +3. **For each `TransactionIntent`:** + - Convert sdcpb → `types.TransactionIntent`; expand and validate updates against the schema via `treetypes.ExpandAndConvertIntent`. + - Take a deep copy of `syncTree` under `syncTreeMutex.RLock`. + - `LoadAllButRunningIntents` — stream all intent blobs from cache (excluding `"running"`) into the tree copy. + - Mark old owner's leaves with `OwnerDeleteMarker`; save old intent content for rollback; add new updates with `flagNew`. + - All merge and validation steps run on the candidate tree copy, not on `syncTree`. +4. **`FinishInsertionPhase`** — resolve defaults, key presence, and leaf-list ordering across the combined tree. +5. **Validation** (`validation.Validate`) — YANG semantic and type checks (including mandatory children, pattern/range/type constraints). +6. **Dry-run exit** — if `dry_run=true`, return the computed diff without touching the device or cache. +7. **Apply to device** — `d.sbi.Set(ctx, source)` blocks until the device ACKs. +8. **Write intent blobs** — `ops.TreeExport` → `tree_persist.Intent` → `cacheClient.IntentModify`. +9. **`writeBackSyncTree`** — applies accepted deltas from the candidate result back to `syncTree` and exports the resulting committed tree as the `"running"` blob. +10. **Start rollback timer** — armed for `transaction-timeout` (default 5 min). On expiry, `lowlevelTransactionSet` is called automatically with the captured old intent content. + +**Validation context (what is checked and when):** + +- **Path and schema expansion (early):** each incoming update is first normalized through `treetypes.ExpandAndConvertIntent`, which resolves schema-aware paths (including list key structure) and rejects paths that do not exist in the bound vendor/model/version schema. +- **Tree-shape correctness (mid-phase):** `FinishInsertionPhase` enforces structural correctness before full semantic validation, including key completeness for list entries and deterministic leaf-list ordering. +- **YANG semantic validation (pre-device):** `validation.Validate` runs after all intents are merged into one candidate view; this catches supported cross-field constraints and type-level issues (ranges, patterns, enums, mandatory children) using schema lookups via `SchemaClientBound`. +- **Error surfacing:** validation outcomes are returned in `TransactionSetResponse.Intents` for per-intent reporting. Requests with schema/validation failures do not call `Target.Set`, so invalid state never reaches the device. + +**Implemented validation checks (current behavior):** + +- Unknown or non-expandable schema paths are rejected during intent expansion. +- YANG list key completeness is enforced before semantic validation. +- Leaf-list ordering is normalized deterministically during insertion finalization. +- Mandatory children and required structural elements are enforced. +- Leaf type constraints are validated (including schema-driven type checks such as ranges/patterns/enums where defined). +- YANG `must` expressions are evaluated against the merged candidate tree. + +### Phase B — `TransactionConfirm` + +1. Find the active transaction by ID. +2. `transaction.Confirm()` — stops the rollback timer. +3. Transaction removed; device state stands. + +### Phase C — `TransactionCancel` (or timer fires) + +1. Build rollback transaction from `transaction.GetRollbackTransaction()` — uses `oldIntents` captured during Phase A. +2. Re-run `lowlevelTransactionSet` with the rollback transaction, pushing old content back to both the device and cache. +3. Transaction removed. + +### `WatchDeviations` + +The `DeviationMgr` goroutine starts once per datastore after the SBI connects. It runs on a ticker (default 30 s, overridable via `DEVIATION_INTERVAL`): + +1. Emit `DeviationEvent_START` to all connected streaming clients. +2. `calculateDeviations(ctx)`: + - Deep-copy `syncTree` to a candidate analysis tree. + - `LoadAllButRunningIntents` — load all user intents into that copy. + - `FinishInsertionPhase` — resolve the copied tree. + - Emit `DeviationReasonIntentExists` per loaded intent name. + - `ops.GetDeviations` — walk every leaf; compare the highest-priority intent value against the running value; emit `DeviationReasonOverruled` for any divergence. + - Emit `DeviationReasonUnhandled` for paths present on the device but not covered by any intent. +3. Emit `DeviationEvent_END`. + +### `BlameConfig` + +1. Deep-copy `syncTree` to an analysis tree. +2. `LoadAllButRunningIntents` into that copy. +3. Create `processors.NewBlameConfigProcessor({IncludeDefaults: includeDefaults})`. +4. `bcp.Run(ctx, root.Entry, taskPool)` — annotates each leaf with the owning intent name and priority. +5. Return `BlameConfigResponse{Tree: *BlameTreeElement}` — a recursive tree where every leaf identifies which intent owns it. + +--- + +## The Config Tree (`pkg/tree`) + +`pkg/tree` implements the YANG-aware in-memory tree model used across data-server workflows: intent merge, validation, deviation detection, and blame attribution. Each datastore keeps one committed tree instance (`syncTree`) as its live baseline, while transaction and analysis logic runs on temporary deep copies and only applies accepted results back to the committed tree. The cache stores only serialised intent blobs; tree semantics and conflict handling remain in `pkg/tree`. + +### Node Structure + +The tree is composed of `Entry` nodes, each backed by a `sharedEntryAttributes` struct. The `String()` / `StringIndent()` methods produce an indented representation that reflects the actual in-memory layout. For a YANG list such as `interface`, there is a **list node** level (`interface`) and below it a **key-value node** (`eth0`) that carries the list key as its `pathElemName`: + +``` +interfaces + interface ← list node (schema = SchemaElem_List) + eth0 ← key-value node (schema = nil, pathElemName = key value) + config + mtu ← leaf node + -> Owner: network-team, Priority: 100, Value: 9000, New: false, Delete: false, DeleteIntendedOnly: false, ... + -> Owner: platform-team, Priority: 200, Value: 1500, New: false, Delete: false, DeleteIntendedOnly: false, ... + -> Owner: running, Priority: MaxInt32-100, Value: 9000, New: false, Delete: false, DeleteIntendedOnly: false, ... +``` + +This is an illustrative snapshot of the `String` / `StringIndent` output style: each level is indented by two spaces, and leaf variants are appended on `->` lines with owner, priority, value, and per-entry state flags (`New`, `Delete`, `Update`, `DeleteIntendedOnly`, `ExplicitDelete`, `Non-Revertive`). These are the complete leaf-entry flags currently rendered by the formatter. + +Every `Entry` holds: + +| Field | Type | Purpose | +|---|---|---| +| `pathElemName` | `string` | The YANG path element this node represents | +| `childs` | `*api.ChildMap` | Children map (mutually exclusive with `leafVariants`) | +| `leafVariants` | `*api.LeafVariants` | Per-owner leaf values at this path (leaf nodes only) | +| `schema` | `*sdcpb.SchemaElem` | Pointer to the YANG schema element — enables constraint checking without separate lookups | +| `choicesResolvers` | `api.ChoiceResolvers` | Tracks YANG `choice/case` selections to enforce mutual exclusion | +| `treeContext` | `api.TreeContext` | Shared context across all nodes (see below) | + +### Multi-Owner Coexistence and Priority Resolution + +Multiple intents can set the same leaf path simultaneously. All values are stored as `LeafEntry` records inside `LeafVariants` — one per owner. Nothing is discarded at write time. + +At read time, `LeafVariants.GetHighestPrecedence()` scans the slice and picks the entry with the **lowest numeric priority value**: + +```go +if highest.Priority() > e.Priority() { + secondHighest = highest + highest = e +} +``` + +The second-highest candidate is retained as a fallback in case the winner is marked for deletion. When the winning entry is removed (e.g. its intent is deleted), the next-highest candidate is automatically promoted — no re-import required. + +`LeafVariants` also drives **deviation detection**: after computing the highest-precedence winner, it compares that value against the `running` variant (device state). Any mismatch produces a `DeviationEntry` with reason `DeviationReasonOverruled` or `DeviationReasonMissing`. + +### TreeContext + +A single `TreeContext` instance is shared across all nodes in a tree. It carries: + +| Field | Purpose | +|---|---| +| `SchemaClientBound` | YANG schema access — passed once at tree creation, available to every node | +| `ExplicitDeletes` | Set of paths explicitly deleted by the current transaction | +| `NonRevertiveInfo` | Tracks which owners opt into non-revertive mode (winner stays even if a higher-priority intent appears) | +| `TaskPool` | Worker pool used by parallel operations in `pkg/tree/ops` | + +### Lifecycle Phases + +A tree is used in two distinct phases: + +**Insertion phase** — intents are loaded via `AddUpdatesRecursive`. Entries are created or updated; leaf variants are appended. During this phase the tree may be partially inconsistent (e.g. list keys incomplete, YANG defaults unresolved). + +**`FinishInsertionPhase`** — called once after all intents are loaded. This pass: +- Resolves YANG default values and injects them as `LeafEntry` records owned by `"default"`. +- Verifies list key completeness. +- Orders leaf-lists according to YANG ordering rules. + +No read, validation, or export operation should be performed before `FinishInsertionPhase` completes. + +The `syncTree` held by each datastore is always in finished state. Transactional mutations work on a `DeepCopy` of `syncTree`, apply changes, call `FinishInsertionPhase`, run validation, and only swap the copy into `syncTree` on success. + +### Sub-packages + +| Package | Key Types / Functions | Role | +|---|---|---| +| `pkg/tree/api` | `LeafVariants`, `LeafEntry`, `EntryOutputAdapter`, `IntentResponseAdapter` | Core tree API types; priority resolution; adapter bridge between tree results and gRPC response types | +| `pkg/tree/ops` | `TreeExport`, `GetDeviations`, `GetHighestPrecedence`, `LeafsOfOwner`, `validation.Validate` | Batch read operations; run concurrently via `TaskPool` | +| `pkg/tree/processors` | `OwnerDeleteMarker`, `BlameConfigProcessor`, `ExplicitDeleteProcessor`, `RemoveDeletedProcessor`, `ResetFlagsProcessor` | Processor pattern — each implements a tree-walk with side effects; composable pipeline | +| `pkg/tree/importer` | `proto/`, `xml/`, `json/` | Deserialise intent blobs (from cache), NETCONF XML, or JSON into tree nodes | +| `pkg/tree/types` | `Update`, `UpdateSlice`, `DeviationEntry`, `NonRevertiveInfo`, `DeleteEntriesList` | Shared value types passed between tree, datastore, and gRPC layers | +| `pkg/tree/consts` | Intent name constants (`RunningIntentName`, `DefaultsIntentName`, …) | Avoids magic strings throughout the tree | + +--- + +## Target & SBI Abstraction + +All southbound communication goes through the `Target` interface defined in `pkg/datastore/target/target.go`: + +```go +type Target interface { + Get(ctx context.Context, req *sdcpb.GetDataRequest) (*sdcpb.GetDataResponse, error) + Set(ctx context.Context, source types.TargetSource) (*sdcpb.SetDataResponse, error) + AddSyncs(ctx context.Context, sps ...*config.SyncProtocol) error + Status() *types.TargetStatus + Close(ctx context.Context) error +} +``` + +The factory function `target.New(ctx, name, cfg, schemaClient, runningStore, syncConfigs, taskpoolFactory)` dispatches on `cfg.Type`: + +| `cfg.Type` | Driver | Package | +|------------|--------|---------| +| `"gnmi"` | `gnmiTarget` | `pkg/datastore/target/gnmi/` | +| `"netconf"` | `ncTarget` | `pkg/datastore/target/netconf/` | + +### gNMI Driver + +- **Library:** `github.com/openconfig/gnmic/pkg/api` +- **Connection:** gRPC with TLS and keepalive (ping every 10 s, timeout 2 s). Sends a `Capabilities` RPC on connect; rejects the configured encoding if the device does not advertise it. +- **`Set()`:** Builds a gNMI `SetRequest` from `TargetSource`. Encoding selection: `"json"` → `TypedValue_JsonVal`, `"json_ietf"` → `TypedValue_JsonIetfVal`, `"proto"` → per-leaf updates. Deletes use `ToProtoDeletes()`. +- **Sync modes:** `"stream"` (gNMI STREAM subscription), `"get"` (periodic Get), `"once"` (repeated ONCE subscriptions). + +### NETCONF Driver + +- **Library:** `github.com/scrapli/scrapligo/driver/netconf` — SSH-based NETCONF 1.0/1.1 +- **Connection:** SSH with NETCONF hello exchange on `d.Open()`. Health is checked via `driver.IsAlive()`; a reconnect loop triggers on EOF. +- **`Set()` flow:** + 1. `source.ToXML(ctx, true, includeNS, operationWithNS, useOperationRemove)` — produces a single XML document; deletes are emitted inline with `operation="delete"` (or `"remove"`) attributes. + 2. `driver.EditConfig(commitDatastore, xmlDoc)` — sends ``. + 3. If `commitDatastore == "candidate"`: `driver.Commit()` sends ``. + 4. RPC errors: `"warning"` severity → collected in response; `"error"` severity → discard + reconnect on EOF. +- **Sync:** Polling only via periodic ``. NETCONF notification subscriptions are not used. +- **Note:** Lock/Unlock are present on the driver interface but are not called during `Set()` — there is no explicit lock-commit-unlock cycle. + +**XML ↔ schemapb translation** (NETCONF only): + +- `XMLConfigBuilder` — converts sdcpb path+value updates/deletes into NETCONF `` XML, calling `schemaClient.GetSchemaSdcpbPath()` per path element for namespace URI resolution. +- `XML2sdcpbConfigAdapter` — converts an `*etree.Document` (`` response) into `[]*sdcpb.Notification`, querying the schema-server for leaf types and following leafref chains. + +Testing-only target stubs exist in the codebase but are intentionally not documented as +supported southbound options. + +--- + +## Schema Validation + +`data-server` calls the schema-server for every operation that touches the YANG model: + +| Operation | Call | When | +|-----------|------|------| +| Path validation / expansion | `ToPath`, `ExpandPath` | Processing `TransactionSet` updates | +| Schema node lookup | `GetSchema`, `GetSchemaDetails` | YANG validation (`validation.Validate`) — types, ranges, patterns, `must` expressions | +| List key completeness | `GetSchemaElements` | `FinishInsertionPhase` | +| Namespace resolution | `GetSchema` | XML construction for NETCONF `` | + +All calls go through `schemaClient.SchemaClientBound`, which is bound to a specific vendor/name/version. In **local mode** (default), these are direct Go calls to `schema.LocalClient` sharing the same process. In **remote mode**, they are gRPC calls to a standalone schema-server pod. + +--- + +## Configuration & Deployment + +### Config File (`data-server.yaml`) + +```yaml +grpc-server: + address: :56000 + max-recv-msg-size: 4194304 + rpc-timeout: 60s + tls: + ca: + cert: + key: + skip-verify: false + schema-server: + enabled: true + schemas-directory: /schemas + data-server: + enabled: true + +schema-store: # mutually exclusive with schema-server + type: persistent + path: /schemadb + read-only: false + +cache: + type: local + store-type: filesystem + dir: ./cached/caches + +transaction-timeout: 5m + +deviation-defaults: + interval: 30s +``` + +**`schema-store` vs `schema-server`:** The presence of a `schema-store` section activates the embedded Badger store (local mode). Replacing it with a `schema-server` section (pointing to a remote address) switches to gRPC calls against a standalone schema-server pod. + +**`cache.type`:** `local` embeds the cache in-process (default). `remote` points to a standalone cache-server pod over gRPC. + +### CLI Flags + +| Flag | Default | Description | +|------|---------|-------------| +| `--config` / `-c` | `` | Config file path (YAML) | +| `--debug` / `-d` | `false` | Enable debug log level | +| `--trace` / `-t` | `false` | Enable trace log level | +| `--version` / `-v` | `false` | Print version and exit | + +### Environment Variables + +| Variable | Description | +|----------|-------------| +| `EXTRA_LOG_FILE` | Optional additional log file path | +| `DEVIATION_INTERVAL` | Override the deviation polling interval (Go duration string, e.g. `60s`) | + +### Ports + +| Port | Protocol | Purpose | +|------|----------|---------| +| `:56000` | gRPC | `DataServer` + `SchemaServer` services | +| `:55555` | HTTP | Prometheus metrics | +| `localhost:6060` | HTTP | pprof profiling | + +### Volumes + +| Mount path | Source | Contents | +|------------|--------|---------| +| `/schemadb` | `pvc-schema-db` (PVC) | Badger database of parsed YANG schema objects | +| `/cached/caches` | `emptyDir` | Intent blobs (cache library filesystem store) | +| `/schemas` | ConfigMap / PVC | Raw YANG files loaded by the schema-server library | + +### gRPC Services + +Both the `DataServer` and `SchemaServer` services are registered on the same listener (`:56000`). + +**`DataServer` RPCs:** + +| RPC | Streaming | Description | +|-----|-----------|-------------| +| `CreateDataStore` | — | Create a datastore for a target device | +| `DeleteDataStore` | — | Remove a datastore | +| `GetDataStore` | — | Get datastore details | +| `ListDataStore` | — | List all datastores | +| `TransactionSet` | — | Start a transaction (write + apply intents) | +| `TransactionConfirm` | — | Commit a transaction; cancel the rollback timer | +| `TransactionCancel` | — | Roll back a transaction | +| `ListIntent` | — | List intent names and priorities in a datastore | +| `GetIntent` | — | Get the full content of an intent | +| `WatchDeviations` | Server-streaming | Stream deviation events (config drift detection) | +| `BlameConfig` | — | Return per-path intent ownership annotation | + +--- + +## Interactions + +| Direction | Component | Mechanism | Purpose | +|-----------|-----------|-----------|---------| +| Inbound | config-server | gRPC `DataServer` (`:56000`) | `TransactionSet/Confirm/Cancel`, `CreateDataStore`, `DeleteDataStore`, `WatchDeviations`, `BlameConfig`, `ListIntent`, `GetIntent` | +| Inbound | config-server | gRPC `SchemaServer` (`:56000`) | `CreateSchema` — schema reconciler registers YANG schemas on startup | +| Outbound | Device (gNMI) | gNMI over gRPC/TLS | `Set`, `Get`, `Subscribe` via openconfig/gnmic | +| Outbound | Device (NETCONF) | NETCONF 1.0/1.1 over SSH | `EditConfig`, `GetConfig`, `Commit` via scrapligo | +| Internal | schema-server lib | Direct Go calls (`LocalClient`) | Schema lookup, path expansion, validation, namespace resolution | +| Internal | cache lib | Direct Go calls (`LocalCache`) | Intent blob read / write / delete | + +### Internal Package Map + +| Package | Responsibility | +|---------|---------------| +| `pkg/server` | gRPC server; registers `DataServer` + `SchemaServer` on one port; interceptors | +| `pkg/datastore` | Core per-target logic: `Datastore` struct, transaction, deviations, intent CRUD | +| `pkg/datastore/target` | `Target` interface + factory; `gnmi/` and `netconf/` protocol packages | +| `pkg/tree` | YANG-aware in-memory config tree; priority resolution; validation | +| `pkg/tree/ops` | Batch tree operations: `TreeExport`, `GetDeviations`, `Validate` | +| `pkg/tree/processors` | Named tree processors: blame, delete-marking, flag reset | +| `pkg/tree/importer` | Import adapters: proto, XML, JSON | +| `pkg/schema` | Schema client abstraction: `LocalClient`, `RemoteClient`, `SchemaClientBound` | +| `pkg/cache` | Cache client abstraction: `LocalCache`, `RemoteCache`, `CacheClientBound` | +| `pkg/pool` | `SharedTaskPool` — bounded worker goroutine pool for parallel tree operations | +| `pkg/config` | All config structs, defaults, validation | diff --git a/docs/system-architecture/schema-server.md b/docs/system-architecture/schema-server.md index e69de29..0fc111d 100644 --- a/docs/system-architecture/schema-server.md +++ b/docs/system-architecture/schema-server.md @@ -0,0 +1,234 @@ +# Schema Server + +## Overview + +Schema Server is a Go library and optional standalone gRPC service that parses YANG modules from disk using the [SDC goyang fork](https://github.com/sdcio/goyang), serialises the resulting schema objects into a persistent embedded database (Badger v4), and serves schema metadata to other SDC components on demand. It is the single source of truth for all type, namespace, and constraint information derived from YANG. + +**Deployment model.** The `schema-server` Go module (`github.com/sdcio/schema-server`) is imported directly by `data-server`. In the current production deployment, data-server embeds the schema store **in-process** — no separate schema-server pod is required. The standalone gRPC binary exists for future split-out deployments. The `cache` component follows the same pattern: it can be embedded (`type: local`) or remote (`type: remote`). + +**Statefulness.** + +| Store type | Backing | Persistence | Notes | +|------------|---------|-------------|-------| +| `persistent` (default) | Badger v4 embedded KV at `schema-store.path` | Survives restarts | Keys: `@@/`; values: `proto.Marshal(sdcpb.SchemaElem)`. If a key already exists on startup it is **not** re-parsed, so restarts are fast. | +| `memory` | `map[SchemaKey]*Schema` behind `sync.RWMutex` | Lost on restart | Useful for testing or ephemeral environments. | + +A background goroutine runs `bdb.RunValueLogGC(0.7)` every minute to reclaim Badger value-log space. + +`persistStore` can optionally wrap Badger lookups with a `ttlcache.Cache[cacheKey, *sdcpb.GetSchemaResponse]` (capacity + TTL configurable) to avoid repeated protobuf unmarshalling on hot paths. + +--- + +## Architecture Diagram (Mermaid) + +```mermaid +graph TD + subgraph k8s["Kubernetes StatefulSet Pod"] + DS["Data Server\n(embeds schema-server in-process)"] + SS_GRPC["SchemaServer RPCs\nserved by data-server listener\n:56000 (embedded default)"] + BADGER[("Badger DB\npvc-schema-db → /schemadb")] + YANG_VOL[("Raw YANG files\npvc-schema-store → /schemas")] + + DS -- "localClient\n(direct Go call)" --> SCHEMA_STORE["persistStore / memStore\n(pkg/store)"] + SCHEMA_STORE -- "read/write\nproto.Marshal(SchemaElem)" --> BADGER + SS_GRPC -- "delegates RPCs" --> SCHEMA_STORE + SCHEMA_STORE -- "parse .yang on CreateSchema\nor UploadSchema" --> YANG_PARSER["YANG parser\npkg/schema\n(goyang fork)"] + YANG_PARSER -- "reads" --> YANG_VOL + end + + CTRL["Config Server\n(controller)"] -- "gRPC: CreateSchema\nDeleteSchema\nGetSchemaDetails" --> SS_GRPC + CTRL -- "git clone → YANG files" --> YANG_VOL +``` + +For split-out deployments, a standalone schema-server binary can expose the same RPCs on +its own listener (default `:55000`). + +--- + +## Responsibilities + +- **Parse YANG.** Consume `.yang` files (plus include/exclude globs), build a `yang.Entry` tree via the [SDC goyang fork](https://github.com/sdcio/goyang), resolve cross-module `leafref`, `augment`, and `uses` references. +- **Persist schemas.** Serialise each `yang.Entry` node as a `sdcpb.SchemaElem` proto into Badger, keyed by vendor/name/version and path. Skip already-persisted keys on restart for fast cold-start. +- **Serve schema metadata.** Answer `GetSchema`, `GetSchemaElements`, `ExpandPath`, and related RPCs used by data-server to validate paths and values, resolve XML namespaces, and expand wildcarded paths. +- **Lifecycle management.** Register, reload, and delete schemas at runtime without restarting the process. +- **Upload gateway (future).** Accept raw YANG file bytes over a client-streaming RPC (`UploadSchema`), write them to disk, then parse and persist. + +--- + +## Schema Ingestion + +### Static configuration at startup + +Schemas listed under `schema-store.schemas` in the YAML config are parsed **concurrently at startup** (one goroutine per schema entry). If the `SchemaKey{Name, Vendor, Version}` already exists in Badger the schema is skipped — restarts are idempotent and fast. + +### `CreateSchema` RPC + +The caller provides the vendor, name, version, and paths to YANG files already on disk. The store parses them immediately and writes the result to the configured schema store; in persistent mode that store is Badger. + +### `UploadSchema` RPC (three-phase client streaming) + +1. **First message** — `UploadSchemaRequest_CreateSchema`: identifies the schema (vendor, version, exclude patterns). +2. **Subsequent messages** — `UploadSchemaRequest_SchemaFile`: chunked file bytes with an optional integrity hash (MD5, SHA-256, or SHA-512). +3. Files are written to `/__/`. +4. On stream close the server parses the uploaded files and stores them to Badger. + +> `UploadSchema` is not actively used in the current K8s deployment; YANG files are supplied via a shared PVC cloned by the controller. + +### YANG parsing (`pkg/schema/schema.go`) + +1. `schema.NewSchema(cfg)` calls the SDC goyang fork to parse all `.yang` files matching the configured `files` + `directories` paths, minus any `excludes` patterns. +2. A root `yang.Entry` tree (`__root__`) is built where top-level children are individual YANG modules. +3. `buildReferencesAnnotation()` walks the tree and resolves cross-module `leafref`, `uses`, and `augment` references, annotating each `yang.Entry` node. +4. Each node is serialised as `proto.Marshal(sdcpb.SchemaElem)` and written to Badger with the key `@@/__root__//`. + +### Disk layout + +``` +/ # Badger DB (SSTs, value log, MANIFEST) +/ + __/ # one directory per uploaded schema + *.yang # raw YANG source files +``` + +### K8s deployment flow + +``` +controller (Schema Reconciler) + → git clone YANG files → pvc-schema-store (/schemas) + → gRPC CreateSchema (paths on /schemas) +data-server (embedded schema-server) + → parse YANG from /schemas + → write Badger DB → pvc-schema-db (/schemadb) +``` + +--- + +## Schema Serving + +**`GetSchema` request flow:** + +1. Client calls `GetSchema(GetSchemaRequest{Schema: {Vendor, Version}, Path: })`. +2. `persistStore.getSchema` constructs a Badger key from the schema key + path elements and performs a point lookup. +3. **Module prefix resolution:** if the first path element contains `:` the prefix selects the correct top-level module entry. +4. The raw bytes are unmarshalled from `proto.Marshal(sdcpb.SchemaElem)` and returned. +5. **TTL response cache:** if configured, `persistStore` checks a `ttlcache` (keyed by `{SchemaKey, path-as-string}`) before hitting Badger, avoiding repeated unmarshalling on hot paths. + +**Hot-reload (`ReloadSchema`):** re-parses YANG files from their original configured paths. `Schema.Reload()` calls `NewSchema(sc.config)` to produce a fresh `Schema` value and then `store.AddSchema` replaces the Badger entries atomically. + +**Per-datastore schema path cache (`SchemaClientBound`):** each data-server datastore holds a `SchemaClientBoundImpl` that wraps the schema `Client` and binds it to a fixed `{vendor, name, version}` triple. It maintains a `sync.Map` indexed by keyless path string → `*SchemaIndexEntry`. On the first lookup of a path the response is stored in the index; subsequent lookups return the cached entry without touching the store. This is the primary per-datastore hot-path cache. + +--- + +## Configuration & Deployment + +### CLI flags + +| Flag | Default | Description | +|------|---------|-------------| +| `--config` / `-c` | `schema-server.yaml` | Path to YAML config file | +| `--debug` / `-d` | `false` | Set log level to DEBUG | +| `--trace` / `-t` | `false` | Set log level to TRACE | +| `--version` / `-v` | `false` | Print version and commit SHA | + +### Full YAML config reference + +```yaml +grpc-server: + address: ":55000" + max-recv-msg-size: 4194304 # bytes + rpc-timeout: 1m + tls: + ca: + cert: + key: + skip-verify: false + schema-server: + schemas-directory: ./schemas-dir # landing dir for UploadSchema + +schema-store: + type: persistent # "persistent" | "memory" + path: ./schema-store # Badger DB directory + read-only: false + cache: + ttl: 5m + capacity: 10000 + with-description: false + schemas: + - name: srl + vendor: Nokia + version: 23.7.1 + files: + - ./yang/srl/srl_nokia/models + directories: + - ./yang/srl/ietf + excludes: + - .*tools.* + +prometheus: + address: ":9090" +``` + +### gRPC interface + +**Service:** `SchemaServer` (proto package `sdcpb`, defined in [`sdc-protos/schema.proto`](https://github.com/sdcio/sdc-protos/blob/main/schema.proto)) +**Default standalone listen address:** `:55000` + +In the default embedded deployment, `SchemaServer` RPCs are served by the `data-server` +process endpoint (commonly `:56000`), not by a separate `:55000` listener. + +| RPC | Streaming | Description | +|-----|-----------|-------------| +| `GetSchema` | Unary | Returns one `SchemaElem` for a gNMI-style path | +| `GetSchemaElements` | Server-streaming | Streams all `SchemaElem`s along a path | +| `GetSchemaDetails` | Unary | Full details of a registered schema | +| `ListSchema` | Unary | List known schemas (name, vendor, version, status) | +| `CreateSchema` | Unary | Register schema metadata and trigger parse | +| `ReloadSchema` | Unary | Re-parse YANG files for an existing schema | +| `DeleteSchema` | Unary | Remove a schema from the store | +| `UploadSchema` | Client-streaming | Upload raw YANG file bytes then trigger parse | +| `ToPath` | Unary | Convert path element strings to a structured `Path` | +| `ExpandPath` | Unary | Enumerate all concrete sub-paths under a wildcarded path | + +All RPC handlers in `pkg/server/rpcs.go` are thin wrappers that delegate directly to the `store.Store` interface. + +### Startup behaviour + +`main.go` wraps `server.Serve` in a `goto START` loop with a 1-second back-off, so the process automatically recovers from transient initialisation errors without an external supervisor. + +External dependencies at startup: **none** — the schema-server dials no external gRPC services; all communication is inbound. + +### Volume layout (K8s StatefulSet) + +| Volume | PVC | Mount path | Contents | +|--------|-----|------------|----------| +| `schema-store` | `pvc-schema-store` | `/schemas` (controller + data-server) | Raw `.yang` files cloned from Git | +| `schema-db` | `pvc-schema-db` | `/schemadb` (data-server only) | Badger DB of parsed `SchemaElem` protos | + +--- + +## Interactions + +| Direction | Peer component | Mechanism | Purpose | +|-----------|---------------|-----------|---------| +| Inbound (embedded, current) | Data Server | Direct Go call via `localClient` (`pkg/schema/local.go`) | Validate paths and values, resolve XML namespaces, convert paths. No gRPC hop — in-process. | +| Inbound (remote, future) | Data Server | gRPC `SchemaServer` `:55000` | Same as above when data-server is configured with a `schema-server` address instead of a local `schema-store` | +| Inbound (gRPC) | Config Server (controller) | gRPC `SchemaServer` to the schema-server K8s Service (resolves to the data-server pod) | Schema Reconciler calls `GetSchemaDetails`, `CreateSchema`, `DeleteSchema` | +| None | Cache | — | Schema-server has no dependency on the cache component | + +--- + +## Internal Package Tour + +| Package | Purpose | Key types / functions | +|---------|---------|-----------------------| +| `pkg/server` | gRPC server init and lifecycle | `Server`, `NewServer()`, `Serve()`, `Stop()` | +| `pkg/server/rpcs.go` | Thin RPC handlers delegating to `store.Store` | all `SchemaServer` RPC methods | +| `pkg/config` | YAML config unmarshalling, TLS setup, validation | `Config`, `SchemaStoreConfig`, `SchemaConfig`, `GRPCServer` | +| `pkg/store` | `Store` interface and `SchemaKey` type | `Store`, `SchemaKey`, `Key()` | +| `pkg/store/persiststore` | Badger-backed store with optional TTL response cache | `persistStore`, `New()` | +| `pkg/store/memstore` | In-memory store | `memStore`, `New()` | +| `pkg/schema` | YANG parsing, `yang.Entry` → `sdcpb.SchemaElem` conversion | `Schema`, `NewSchema()`, `SchemaElemFromYEntry()` | +| `pkg/schema/container.go` | Converts container/list nodes | `containerFromYEntry()` | +| `pkg/schema/leaf.go` | Converts leaf/leaf-list nodes | `leafFromYEntry()` | +| `pkg/schema/references.go` | Resolves leafref / augment / uses cross-module references | `buildReferencesAnnotation()` | +| `pkg/schema/expand.go` | Path expansion for `ExpandPath` RPC | `Expand()` |