CADC-15011: Kueue testing updates for lifecycle management and control-plane observability

[shinybrar]

Overview

This PR introduces a comprehensive overhaul of the kueuer testing framework for production-ready Kueue validation system with full lifecycle management, control-plane observability, and automated analysis.

Key Improvements

Lifecycle

• New lifecycle module: Complete workflow automation from preflight checks to teardown
• Scenario support: control and backlog scenarios for testing under different queue pressures
• Queue validation: Automated Kueue health checks and queue readiness verification
• Artifact consolidation: All outputs (benchmarks, plots, observations, reports) organized under unified artifacts/<run_id>/ structure

Observability

• Real-time observe module to gather control-plane metrics during benchmark runs
• Tracks Kueue controller resources, API server latency, and queue pressure
• Visualization of controller memory/CPU, API server p95 latency, and queue wait times

Metrics

• Better metrics which emphasize decision-critical signals (throughput, completion ratio, tail turnaround, overhead)
• Single-view summaries for performance, evictions, and observations
• Clear metric definitions and interpretation guidelines

Streamlined CLI

• Unified under kr benchmark e2e workflow
• Add basic profiles, local-safe and cluster-scale for different testing scenarios
• Consistent cli patterns across all commands
• Removed duplicated code and cleaner command structure

Documentation

• High-level introduction to concepts and architecture
• Complete command documentation with all options

What This Enables

The tool now answers two critical production questions:

1. Scale handling: Can Kueue admit, queue, preempt, and complete large numbers of jobs correctly?
2. Control-plane load: Does Kueue add unacceptable pressure to the Kubernetes control plane?

Example Usage

# Quick end-to-end validation
RUN_ID="$(date -u +%Y%m%d-%H%M%S)"
uv run kr preflight --run-id "$RUN_ID"
uv run kr benchmark e2e \
  --run-id "$RUN_ID" \
  --profile local-safe \
  --counts 2,4,8,16,32,64

# Inspect results
find "artifacts/$RUN_ID" -type f | sort

[github-actions]

✅ All pre-commit checks passed

Thanks for keeping the repo tidy! ✨

[SharonGoliath]

1. It's too big to read individually, so I downloaded and installed it, and did a pylint (default configuration) and pytest --cov on it.
   pylint output that I think is important enough to raise:

2. Bug: access before definition (src/kueuer/utils/k8s_config.py)Pylint: E0203: Access to member '_initialized' before its definition line 42 at line 39.
3. Wrong number of arguments (src/kueuer/benchmarks/track.py)
   Pylint: E1121: Too many positional arguments for method call at lines 293, 326, 340 (e.g. status(item, "Complete") / status(item, "Failed")).
4. Cyclic import (pylint R0401)
   Chain: kueuer.benchmarks.benchmark → kueuer.lifecycle.commands → kueuer.lifecycle.suite (and back).

3. There's no login host, so I don't know how this will change execution instructions. Shaun has set up capsule for multi-tenant (CADC and RCS) user management. Use an OIDC claim for kubectl.
4. Comments on the documentation:
5. benchmark-walkthrough.md - line 68 mentions "manifest state". A "manifest" shows up in a lot of places in the code, but this is the only mention in the docs. Either remove the single mention, or explain it if it's important enough.
6. metrics-semantics.md - lines 90 - 94 read more like they belong in a change log
7. the docs do not mention the kueue occupation restrictions. For my own knowledge, do these restrictions affect this testing?
8. What would happen if testing was done with milli-cores, for example?