Skip to content

Add Kubernetes AgentInterface E2E backend - #24639

Merged
nubtron merged 57 commits into
masterfrom
nubtron/kubernetes-agent-interface-e2e
Aug 10, 2026
Merged

Add Kubernetes AgentInterface E2E backend#24639
nubtron merged 57 commits into
masterfrom
nubtron/kubernetes-agent-interface-e2e

Conversation

@nubtron

@nubtron nubtron commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds a KubernetesAgent implementation of ddev's existing AgentInterface for running integration E2E checks from inside a disposable Kubernetes cluster.

Today we have DockerAgent and VagrantAgent, this PR adds a third choice.

Usage:

@pytest.fixture(scope='session')
def dd_environment():
    with kind_run(conditions=[setup_integration]) as kubeconfig:
        instance = {'openmetrics_endpoint': METRICS_ENDPOINT}
        metadata = {
            'agent_type': 'kubernetes',
            'kubernetes': {
                'kubeconfig': kubeconfig,
            },
        }

        yield instance, metadata

Status Quo:
2026-07-24_13-43-19

New Option:
2026-07-24_13-44-26

Conversion example

#24642 converts Kuma to KubernetesAgent.

Motivation

Running the Agent outside the cluster requires port forwarding, introduces additional startup races, and doesn't allow us to test autodiscovery. Creating the Agent container in a pod inside the cluster solves these problems.

Limitations

he backend is not intended for shared or pre-existing clusters. Support for that could be added easily but I wanted to keep this PR small.

Validation

  • integrations-core pre-commit linters
  • focused Kubernetes Agent and environment lifecycle tests: 33 passed
  • ddev Ruff formatting/checks and mypy
  • coverage for manifest creation, RBAC, package/configuration synchronization, invocation, reload, shell, logs, no-op Agent teardown, and failed fixture-teardown state retention
  • full Kind E2E validation for the converted Kuma, Argo CD, KEDA, Kyverno, Calico, Argo Rollouts, and Tekton integrations

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

@nubtron nubtron added the qa/skip-qa Automatically skip this PR for the next QA label Jul 22, 2026
@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Tests  Code Coverage

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 95.19%
Overall Coverage: 88.51%

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a7d9d7b | Docs | Datadog PR Page | Give us feedback!

@nubtron nubtron changed the title Add Kubernetes AgentInterface backend for discovery E2E tests Add Kubernetes AgentInterface E2E backend Jul 22, 2026
@nubtron
nubtron force-pushed the nubtron/kubernetes-agent-interface-e2e branch from 2a64043 to f21cf4c Compare August 10, 2026 09:28
@temporal-github-worker-1
temporal-github-worker-1 Bot dismissed HadhemiDD’s stale review August 10, 2026 09:28

Review from HadhemiDD is dismissed. Related teams and files:

  • agent-integrations
    • datadog_checks_dev/changelog.d/24639.added
    • datadog_checks_dev/datadog_checks/dev/plugin/pytest.py
    • ddev/changelog.d/24639.added
    • ddev/src/ddev/cli/env/init.py
    • ddev/src/ddev/cli/env/agent.py
    • ddev/src/ddev/cli/env/logs.py
    • ddev/src/ddev/cli/env/reload.py
    • ddev/src/ddev/cli/env/shell.py
    • ddev/src/ddev/cli/env/show.py
    • ddev/src/ddev/cli/env/start.py
    • ddev/src/ddev/cli/env/stop.py
    • ddev/src/ddev/e2e/agent/init.py
    • ddev/src/ddev/e2e/agent/docker.py
    • ddev/src/ddev/e2e/agent/image.py
    • ddev/src/ddev/e2e/agent/interface.py
    • ddev/src/ddev/e2e/agent/kubernetes.py
    • ddev/src/ddev/e2e/agent/vagrant.py
    • ddev/tests/cli/env/test_agent.py
    • ddev/tests/cli/env/test_logs.py
    • ddev/tests/cli/env/test_start.py
    • ddev/tests/cli/env/test_stop.py
    • ddev/tests/e2e/agent/test_kubernetes.py
    • docs/developer/ddev/plugins.md
    • docs/developer/ddev/test.md
@dd-octo-sts

dd-octo-sts Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 21 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

@nubtron
nubtron merged commit 861bd5a into master Aug 10, 2026
374 of 377 checks passed
@nubtron
nubtron deleted the nubtron/kubernetes-agent-interface-e2e branch August 10, 2026 17:43
@dd-octo-sts dd-octo-sts Bot added this to the 7.83.0 milestone Aug 10, 2026
@cit-pr-commenter-54b7da

Copy link
Copy Markdown

evalya-impact-summary

evalya impact analysis
Impact analysis: RUN-ALL — every test task will run
Trigger:         empty diff (default branch, scheduled run, or shallow-clone fallback)
Test tasks:      0 (all selected)
Publish tasks:   0 (always emitted)
Diff:            empty (no diff information)

Learn more about CI impact filtering

github-actions Bot pushed a commit that referenced this pull request Aug 10, 2026
* Add Kubernetes Agent E2E backend

* Use draft PR number for changelogs

* Remove unused Velero Kind port mapping

* Use shared named-port discovery helper

* Test generated discovery behavior

* Test generated discovery through public contract

* Extract Velero migration into stacked PR

* Encapsulate Agent backend startup policy

* Add Kubernetes discovery candidate stability helper

* Document Kubernetes candidate stability support

* Clean up unsupported CI environments

* Clarify Agent interface creation

* refactor(e2e): move Kubernetes discovery helper to consumer PR

* refactor(e2e): avoid caching Kubernetes Agent properties

* Restore unsupported CI backend state handling

* Clarify Kubernetes Agent constraints

* refactor(ddev): use singular Agent build config keys

* Remove Kubernetes Agent pod label option

* Remove custom Kubernetes Agent namespace option

* Deduplicate Agent log commands

* Simplify Kubernetes metadata access

* refactor(ddev): remove Kubernetes Agent owner ID

* Simplify Kubernetes Agent cluster ownership

* Update logs tests to Python 3.13

* Refine Kubernetes command contract tests

* Test Kubernetes topology command directly

* Reject non-Kind Kubernetes Agent contexts

* Relax Kubernetes orchestration assertions

* Update Kubernetes Agent description for E2E tests

* Simplify Kubernetes local package metadata

* Restore environment state cleanup ordering

* Wait for Kubernetes Agent before restart

* Fail fast on unsupported Agent supervision

* Simplify Kubernetes Agent restart checks

* Keep Kubernetes Agent restarts repeatable

* Detect Kubernetes Agent container state loss

* Rename Kubernetes Agent module constants

* Update Kubernetes node support description

Clarified the implementation details regarding Kubernetes node support.

* Decouple Kubernetes Agent tests from commands

* Expose shared Agent image normalization

* Require a Kubernetes metadata mapping

Reading self.metadata['kubernetes'] unguarded turned a missing metadata
block into a bare KeyError, so an environment that selects agent_type
'kubernetes' without the accompanying mapping failed with 'Unable to
start the Agent: kubernetes' and gave a raw traceback from ddev env
shell, which only handles CalledProcessError.

Validate the mapping where it is read so the failure names the missing
contract instead.

* Stamp the prepared marker before restarting the Agent

start() stamped PREPARED_MARKER after _restart_agent_process(), so a
container replaced during the restart was marked prepared. The marker
lives in the container filesystem and the pod declares no volumes, so a
replacement loses the copied conf.yaml, auto_conf.yaml and the editable
installs, while _wait_for_agent() still succeeds against the fresh
container. Every later _require_prepared() check then passed and tests
silently exercised the Agent-shipped integration with no configuration.

Stamp the marker once preparation is complete and assert it after the
restart, matching the invariant restart() already relies on. Checking
after the stamp instead would only observe the gap between the two
commands.

* Share Agent image normalization through the interface

* Clarify the prepared marker invariant

* Move Agent image normalization to a shared module

* Fail fast on unsupported Kubernetes Agent inputs

* Clarify shared-log backend limitation

* Preserve Agent check failures during config cleanup

* Clarify current Kubernetes Agent limitations

* Keep kubectl warnings out of structured output 861bd5a
github-actions Bot pushed a commit to ConnectionMaster/integrations-core that referenced this pull request Aug 10, 2026
* Add Kubernetes Agent E2E backend

* Use draft PR number for changelogs

* Remove unused Velero Kind port mapping

* Use shared named-port discovery helper

* Test generated discovery behavior

* Test generated discovery through public contract

* Extract Velero migration into stacked PR

* Encapsulate Agent backend startup policy

* Add Kubernetes discovery candidate stability helper

* Document Kubernetes candidate stability support

* Clean up unsupported CI environments

* Clarify Agent interface creation

* refactor(e2e): move Kubernetes discovery helper to consumer PR

* refactor(e2e): avoid caching Kubernetes Agent properties

* Restore unsupported CI backend state handling

* Clarify Kubernetes Agent constraints

* refactor(ddev): use singular Agent build config keys

* Remove Kubernetes Agent pod label option

* Remove custom Kubernetes Agent namespace option

* Deduplicate Agent log commands

* Simplify Kubernetes metadata access

* refactor(ddev): remove Kubernetes Agent owner ID

* Simplify Kubernetes Agent cluster ownership

* Update logs tests to Python 3.13

* Refine Kubernetes command contract tests

* Test Kubernetes topology command directly

* Reject non-Kind Kubernetes Agent contexts

* Relax Kubernetes orchestration assertions

* Update Kubernetes Agent description for E2E tests

* Simplify Kubernetes local package metadata

* Restore environment state cleanup ordering

* Wait for Kubernetes Agent before restart

* Fail fast on unsupported Agent supervision

* Simplify Kubernetes Agent restart checks

* Keep Kubernetes Agent restarts repeatable

* Detect Kubernetes Agent container state loss

* Rename Kubernetes Agent module constants

* Update Kubernetes node support description

Clarified the implementation details regarding Kubernetes node support.

* Decouple Kubernetes Agent tests from commands

* Expose shared Agent image normalization

* Require a Kubernetes metadata mapping

Reading self.metadata['kubernetes'] unguarded turned a missing metadata
block into a bare KeyError, so an environment that selects agent_type
'kubernetes' without the accompanying mapping failed with 'Unable to
start the Agent: kubernetes' and gave a raw traceback from ddev env
shell, which only handles CalledProcessError.

Validate the mapping where it is read so the failure names the missing
contract instead.

* Stamp the prepared marker before restarting the Agent

start() stamped PREPARED_MARKER after _restart_agent_process(), so a
container replaced during the restart was marked prepared. The marker
lives in the container filesystem and the pod declares no volumes, so a
replacement loses the copied conf.yaml, auto_conf.yaml and the editable
installs, while _wait_for_agent() still succeeds against the fresh
container. Every later _require_prepared() check then passed and tests
silently exercised the Agent-shipped integration with no configuration.

Stamp the marker once preparation is complete and assert it after the
restart, matching the invariant restart() already relies on. Checking
after the stamp instead would only observe the gap between the two
commands.

* Share Agent image normalization through the interface

* Clarify the prepared marker invariant

* Move Agent image normalization to a shared module

* Fail fast on unsupported Kubernetes Agent inputs

* Clarify shared-log backend limitation

* Preserve Agent check failures during config cleanup

* Clarify current Kubernetes Agent limitations

* Keep kubectl warnings out of structured output 861bd5a
AAraKKe added a commit that referenced this pull request Aug 17, 2026
Review follow-ups on the planning package:

- `resolve_platforms` parsed every `Supported OS` classifier tag before collapsing to
  Windows-exclusivity, so the eight manifests advertising AIX raised and aborted the whole
  run. `ci_matrix.py` collapses first and never validates those values, so this was a
  regression against existing data. The manifest list now only decides Windows-exclusivity
  and strict parsing is kept for the hand-written CI override, which also now rejects a
  repeated platform instead of planning two identically named jobs.
- Fix a test importing `_normalize_agent_image_name`, renamed in master by #24639. The
  restack did not conflict, and the resulting collection error aborted the whole suite.
- Planning failures all raise `PlanningError`. Agent-image errors are wrapped at the
  boundary in `jobs.py`, naming the job, rather than made to subclass it.
- Drop the `default_python_version` argument nothing read; `EnvironmentProvider` owns it.
- Record the `ci_matrix.py` relationship and the per-target hatch subprocess cost in the
  package `AGENTS.md`, and correct its docstring and protocol rules.
- Drop a test of pydantic's own frozen behaviour, and fold the dependency-bump case into
  the parametrized set it duplicated.
pull Bot pushed a commit to ConnectionMaster/integrations-core that referenced this pull request Aug 18, 2026
DataDog#24687)

* Build deterministic Dispatcher test batching plans

- Add changed-file target discovery, concrete job expansion, configurable batching strategies, and partition validation.
- Introduce stable batch identities and use them for runner/gatherer correlation.
- Preserve standalone CI matrix behavior while adding focused batching, configuration, message, and execution tests.

* Carry Python version and Agent image through Dispatcher test plans

Each BatchJob now records the major.minor Python its runner must set up and,
when it runs E2E tests, the Agent image to run them against. The version comes
from Hatch's own `python` value rather than the environment name, so repos that
do not follow the `pyX.Y-` naming convention resolve correctly.

- Add `ddev.e2e.agent_images`, mapping a Python version and platform to an
  Agent image. Superseded Agent lines are pinned to their final release; the
  current line tracks the development build. Base tags only, since ddev already
  appends `-jmx` per environment at E2E runtime.
- Add `[dispatcher] default_python_version`, used where a Hatch environment
  declares no Python and for targets that define no environments.

Also applies review findings on the planning layer:

- Collapse `TestUnit` to a single environment and drop `split_environments`.
  Both paths produced the same job set and differed only in where the job name
  was built, so job naming now has one implementation.
- Freeze `BatchJob` and validate batch coverage by value instead of `id()`, so
  a strategy may rebuild equal jobs rather than pass the originals through.
- Derive capacity from the config inside the strategy and validator instead of
  passing it alongside the config it already lives on.
- Rename the `units.Platform` namedtuple to `PlatformSpec`, which no longer
  collides with `messages.Platform`.
- Drop `RepositoryWideRule.exempt_files`. It cancelled the whole repository-wide
  expansion whenever `agent_requirements.in` appeared in a change set, which
  never fires for dependency bumps (they do not touch the base package sources)
  and only fires when a change genuinely does warrant the full set.
- Keep git diagnostics off stdout so they cannot reach the strict diff parser.
- Move duplicated test builders into `tests.helpers.batching`.

* Give platform identity a single home in ddev.utils.platform

Adds `PlatformName` to the module that already owns platform-name vocabulary
(`get_platform_name`, `normalize_platform_name`) and uses it for every platform
value across messages, planning, and Agent image selection, replacing the
`messages.Platform` enum and the string constants in `agent_images`.

`normalize_platform_name` keeps returning a plain string: `platform.system()`
can report systems ddev does not target, and `Platform.linux` is defined as
"neither Windows nor macOS" on purpose. Raw platform strings from
`.ddev/config.toml` and `manifest.json` are parsed into `PlatformName` once, in
`resolve_platforms`, which names the offending target when a value is not
recognized. Everything downstream holds the enum, so the unsupported-platform
check no longer sits in the middle of unit expansion.

Also adds `find_unpublished_images`, built on `ddev.utils.docker_registry`, so a
mistyped or withdrawn tag can be caught before any job runs instead of failing
every E2E job. It queries each distinct image once. `get_agent_image` stays pure
and offline so plans remain deterministic and reproducible; the registry check is
explicitly called rather than implied by a lookup. A `requires_ci` test uses it
to assert every image in the table is served by registry.datadoghq.com.

* Trim planning comments and warn on targets with no testable environment

- Cut module and class docstrings to what is not already in the code, drop Sphinx roles and double backticks
- Add batching/AGENTS.md describing the package for future readers
- Drop a redundant str() around the platform in artifact_name
- Log a warning when a target resolves no testable environment at all

* Use ddev's git module for change detection instead of a batching-local one

- Delete batching/git.py; ChangedFile and the diff parser now come from ddev.utils.git
- Move comparison-base selection to cli/ci/tests/changes.py, outside the batching package
- changes.get_changed_files takes a GitRepository rather than a bespoke GitProvider

* Fold message construction into the build module

assembly.py held one function with one call site and no test file of its own; building the
TestBatch messages is the last step of build_test_batches. The shared jobs() test builder moves
to tests.helpers.batching so both test modules can use it.

* Rebalance the planning test suite toward its lower layers

- Drop build-level tests that re-assert edge cases owned by targets, units, jobs or strategy
- Split validate_batches tests out of test_strategy.py into test_validation.py
- Share make_job, FakeIntegration, FakeRegistry, drain_queue and the changed-file builders
  through tests.helpers.batching instead of four near-identical copies
- Collapse repeated single-case tests into parametrized tables
- Cover three untested paths: an oversized integration spilling from an open batch, platforms
  read from manifest classifier tags, and the repository-wide rule gated on the repo name

* Select rename sources through the shared affected_paths

* Reconcile the gatherer tests with the batch progress aggregate

Master's progress aggregate replaced the expected-batches counter and kept Platform in messages. The correlation tests now register the batch they report, the local job and queue builders defer to the shared helpers, and the new progress tests use PlatformName.

* Fix repository-wide target selection and validate planned Python versions

- eligible_targets asked the registry with an empty selection, which ddev resolves to changed, so a shared-framework change planned only the already changed targets
- the repository-wide rule matched only a change's destination, missing a shared file renamed away
- a unit-only environment's Python version reached the plan unchecked, since only E2E resolves an Agent image
- FakeRegistry now honours the selection, so the first bug fails a test
- the batching test helpers move to the subtree that uses them

* Run unconstrained environments on every platform a target declares

An environment that names no platform was routed only to the first one the target listed, and the remaining platforms fell through to a synthesised environment claiming the default Python and no E2E. That silently dropped Windows E2E for disk, dns_check, ibm_mq, network and tcp_check, and made the order of the configured platform list load-bearing.

Unconstrained environments now run on every platform the target declares, so each job carries a real environment with its own Python version and facets. A target that resolves no environment at all is dropped with a warning when definitions are built rather than planned with an invented one, and a platform no environment covers is warned about and planned empty.

* Stop the planner failing on platforms it has no runner for

Review follow-ups on the planning package:

- `resolve_platforms` parsed every `Supported OS` classifier tag before collapsing to
  Windows-exclusivity, so the eight manifests advertising AIX raised and aborted the whole
  run. `ci_matrix.py` collapses first and never validates those values, so this was a
  regression against existing data. The manifest list now only decides Windows-exclusivity
  and strict parsing is kept for the hand-written CI override, which also now rejects a
  repeated platform instead of planning two identically named jobs.
- Fix a test importing `_normalize_agent_image_name`, renamed in master by DataDog#24639. The
  restack did not conflict, and the resulting collection error aborted the whole suite.
- Planning failures all raise `PlanningError`. Agent-image errors are wrapped at the
  boundary in `jobs.py`, naming the job, rather than made to subclass it.
- Drop the `default_python_version` argument nothing read; `EnvironmentProvider` owns it.
- Record the `ci_matrix.py` relationship and the per-target hatch subprocess cost in the
  package `AGENTS.md`, and correct its docstring and protocol rules.
- Drop a test of pydantic's own frozen behaviour, and fold the dependency-bump case into
  the parametrized set it duplicated.

* Record why the unit and E2E flags are intent rather than a decision

* Mark the unit and E2E facet flags for per-integration configuration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants