Skip to content

feat(go): native Go policy evaluator for Kubernetes SSI targeting#71

Open
iamluc wants to merge 4 commits into
mainfrom
luc/wls-go-policy-evaluator
Open

feat(go): native Go policy evaluator for Kubernetes SSI targeting#71
iamluc wants to merge 4 commits into
mainfrom
luc/wls-go-policy-evaluator

Conversation

@iamluc

@iamluc iamluc commented Jun 19, 2026

Copy link
Copy Markdown

Description

Adds a self-contained, dependency-free Go policy evaluator to this repo
(go/policies), mirroring the semantics of the C engine: tri-state
TRUE/FALSE/ABSTAIN over an AND/OR/NOT tree of leaf evaluators, plus a dd-wls
JSON decoder (the JSON projection of the FlatBuffers policy schema).

This is the Go evaluator the Cluster Agent will use to target Kubernetes
Single Step Instrumentation (SSI) workloads natively (no CGO at runtime).

To make "shares the same test suite" real, this PR also adds a cross-engine
conformance suite
: a single go test runs a shared corpus through both the
Go engine and the real C engine (libpolicies, linked via cgo) in one
process and asserts they agree. The K8s string evaluators are added to the
schema (and to the C engine's public enum) so policies can target
pods/namespaces on both sides.

Scope is deliberately limited to the engine + schema + conformance. The
target -> policy compiler and the Cluster Agent wiring (RC subscription,
mutator) stay in datadog-agent and land in follow-up PRs.

What's in here

Go engine (go/policies)

  • model + node constructors, tri-state evaluator with C-parity semantics
    (boolean folding, signed/unsigned numeric leaves, key=value/key= label
    leaves, depth limit mirroring PLCS_MAX_EVAL_DEPTH);
  • dd-wls JSON decoder (ParsePolicies);
  • Evaluate (rule tree -> tri-state Result) and Decide (folds the outcomes
    of all matching policies in document order, matching the C engine's
    no-first-match-short-circuit behavior — not first-match-wins).

Cross-engine conformance

  • go/policies/testdata/conformance/vectors.json: shared corpus (boolean
    composition, string comparators, labels, numeric, ABSTAIN propagation).
  • conformance_test.go: portable Go-only runner (no cgo, runs everywhere).
  • conformance_cgo.go + conformance_cross_test.go: cgo bridge to the real C
    engine plus host evaluators that mirror what the Go engine bakes in (constant
    ALWAYS_* leaves, the key=value/key= label evaluator, and numeric
    evaluators that abstain on an unset source). Each vector is serialized to a
    FlatBuffers NodeTypeWrapper buffer and run through libpolicies; the result
    must equal both the Go engine's result and the expected value. Gated behind
    the conformance_cgo build tag, so the default CGO_ENABLED=0 builds are
    unaffected. Run with make -C go conformance-cross.

Schema + C engine parity

  • fbs-schema/evaluator_ids.fbs: NAMESPACE_NAME, NAMESPACE_LABEL,
    POD_LABEL, POD_ANNOTATION (append-only); regenerated Go schema.
  • c/include/dd/policies/evaluator_types.h: extend the public
    string-evaluator enum with the same 4 K8s ids so the engine reaches schema
    parity (the _Static_assert against the schema count now matches);
    regenerate evaluator_ids_reader.h.

CI

  • .github/workflows/dev.yml: new conformance-cross-engine job running the
    cross-engine suite in the build container on Linux amd64 + arm64.

Related Issue

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Refactor / cleanup
  • CI / tooling
  • Documentation

Checklist

  • Documentation updated if needed (package doc)
  • Tests added or updated
  • No unintended breaking changes (schema additions are append-only)

How Has This Been Tested?

In the build container:

  • CGO_ENABLED=0 go vet ./policies/... and the Go-only conformance test are
    green; the cgo-gated file is correctly excluded from default builds.
  • make -C go conformance-cross: all corpus vectors pass through both the
    Go engine and the real C engine in one process (semantic parity with
    libpolicies proven on the same inputs, not just ported expectations).
  • make -C c test: the full C unit-test suite still passes after the public
    enum extension.
  • TestDecoderIdentifiersMatchSchema pins every identifier the decoder accepts
    to the .fbs source shared with the C engine, so the two cannot drift apart.

Additional Notes

  • Numeric "unset" semantics: the default C numeric evaluator compares a
    missing fact against the NOT_SET sentinel (FALSE), whereas the Go model
    abstains. Rather than change the engine (and break existing C tests), the
    conformance host registers numeric evaluators that abstain on NOT_SET,
    reproducing the Go semantics at the host level — where the generic C engine
    expects such decisions to live.
  • JSON schema diff: policy.schema.json was regenerated with
    flatc 25.2.10; the committed file lagged the .fbs, so the regen also
    resyncs it in addition to the 4 new K8s ids.
  • Follow-ups (in datadog-agent): target -> policy compiler + golden
    tests; Cluster Agent wiring (facts, mutator, RC apm-wls subscription) with a
    differential test vs the current TargetMutator.

iamluc added 2 commits June 19, 2026 14:47
Add NAMESPACE_NAME, NAMESPACE_LABEL, POD_LABEL and POD_ANNOTATION to the
StringEvaluators enum (append-only) so policies can target Kubernetes
workloads, shared with the C engine.

Regenerated the Go schema and JSON schema with flatc 25.2.10. The committed
policy.schema.json had been produced by an older flatc and lagged the .fbs,
so the regeneration also resyncs it: the *_UNKNOWN zero sentinels, the
evaluator ids 34-51 (IIS_APPLICATION_POOL..CONTAINER_LABEL) and the
dd_wls_UUID / Policy.version definitions, on top of the four new ids.
Add a self-contained, dependency-free Go package that evaluates policies
natively (no CGO), mirroring the dd-policy-engine C evaluator: tri-state
TRUE/FALSE/ABSTAIN over an AND/OR/NOT tree of leaf evaluators, plus a
dd-wls JSON decoder (the JSON projection of the FlatBuffers policy schema).

This is the Go evaluator the Cluster Agent will use for Kubernetes SSI
targeting; the target->policy compiler and the wiring stay in datadog-agent
and land in follow-up PRs.

Tests reuse what is common with the C implementation:
- the wildcard and tri-state conformance vectors are ported from
  c/src/test/test_evaluator.c to guarantee semantic parity with libpolicies;
- TestDecoderIdentifiersMatchSchema pins every identifier the decoder
  accepts to the .fbs source shared with the C engine, so the two cannot
  drift apart silently.
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jun 19, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 21.71%

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 37fc60f | Docs | Datadog PR Page | Give us feedback!

iamluc added a commit to DataDog/datadog-agent that referenced this pull request Jun 19, 2026
…licies

Add the shared github.com/DataDog/dd-policy-engine/go evaluator as a
dependency and lower the static "targets" SSI configuration into its policy
model (policiesFromTargets). This is the static-configuration counterpart of
the remote-config dd-wls parser: both produce policies.Policy, the single
shape the matcher understands, so Kubernetes SSI targeting shares one rule
semantics with the host injector.

The dependency is pinned to a pseudo-version of the dd-policy-engine PR
branch (DataDog/dd-policy-engine#71); repin to the released version once that
PR merges. golang.org/x/sys is bumped 0.45.0 -> 0.46.0 as required by the
dependency's go.mod.

Only the target->policy compiler lands here; the mutator wiring and the RC
subscription follow in a separate PR.
iamluc added a commit to DataDog/datadog-agent that referenced this pull request Jun 19, 2026
…licies

Add the shared github.com/DataDog/dd-policy-engine/go evaluator as a
dependency and lower the static "targets" SSI configuration into its policy
model (policiesFromTargets). This is the static-configuration counterpart of
the remote-config dd-wls parser: both produce policies.Policy, the single
shape the matcher understands, so Kubernetes SSI targeting shares one rule
semantics with the host injector.

The dependency is pinned to a pseudo-version of the dd-policy-engine PR
branch (DataDog/dd-policy-engine#71); repin to the released version once that
PR merges. golang.org/x/sys is bumped 0.45.0 -> 0.46.0 as required by the
dependency's go.mod.

Only the target->policy compiler lands here; the mutator wiring and the RC
subscription follow in a separate PR.
Add a single `go test` (make -C go conformance-cross) that runs the shared
vectors.json corpus through both engines in one process and asserts they
agree: the Go engine via the public ParsePolicies + Evaluate path, and the
real C engine (libpolicies, linked via cgo) fed the same vector serialized
to a FlatBuffers NodeTypeWrapper buffer.

- go/policies/conformance_cgo.go: cgo bridge to the C engine plus host
  evaluators that mirror the Go engine's baked-in semantics (constant
  ALWAYS_* leaves, "key=value"/"key=" label evaluator, numeric evaluators
  that abstain on an unset source). Serializes vectors to FlatBuffers.
- go/policies/conformance_cross_test.go: the cross-engine test, gated behind
  the conformance_cgo build tag so default CGO_ENABLED=0 builds and the
  portable Go-only corpus test are unaffected.
- go/policies/conformance_test.go + testdata/conformance/vectors.json: the
  shared conformance corpus and its Go-only runner.
- c: extend the public string-evaluator enum with the Kubernetes SSI ids
  (NAMESPACE_NAME/NAMESPACE_LABEL/POD_LABEL/POD_ANNOTATION) to reach schema
  parity; regenerate evaluator_ids_reader.h.
- go engine alignment (model/evaluator/dd_wls): full tri-state parity with
  the C engine (boolean folding, numeric + label semantics, depth limit).
- go/Makefile: conformance-cross target.
- .github/workflows/dev.yml: run the cross-engine suite on Linux amd64+arm64.
Move the cross-engine conformance suite (the cgo bridge to libpolicies, the
cross-engine test, and the Go-only corpus runner) out of the importable
"policies" package into a dedicated "policies/conformance" package.

The cgo bridge must live in a regular, non-test source file (cgo is not allowed
in _test.go files), so its FlatBuffers and go/schema imports were part of the
"policies" package's dependency set. `go mod tidy` resolves imports across all
build tags, so even though the bridge is gated behind the "conformance_cgo"
tag, those imports leaked into every consumer's module graph (e.g. the Cluster
Agent pulled github.com/google/flatbuffers and go/schema, plus the matching
LICENSE-3rdparty entries) despite never being linked into any binary.

The bridge no longer reuses the unexported dd-wls decoder types; it parses the
dd-wls JSON with its own local structs, so the conformance package depends only
on the public policies API. The "policies" package thus has no non-test
dependency on FlatBuffers/schema and consumers stop pulling them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant