{AKS} Fix remaining aks-preview live-runner failures - #10216
Draft
FumingZhang wants to merge 4 commits into
Draft
Conversation
…-up to Azure#10184) Root-cause fixes for the 46 failed + 2 timeout live-runner scenarios reported in Kusto run 6e6acd32 (2026-08-13), building on the merged PR Azure#10184. Genuine CLI/SDK bugs fixed: - managed_cluster_decorator.py: `aks create --enable-osdisk-full-caching` was silently dropped for the default agent pool profile (only nodepool add/update wired it). Added get_enable_os_disk_full_caching() and set_up_os_disk_full_caching(), wired into construct_mc_profile_preview(). - aks_diagnostics.py: `_get_temp_kubeconfig_path` (shared by `aks kollect`/ `aks kanalyze`) called list_cluster_user_credentials() with a positional None arg; the vendored SDK now requires server_fqdn as keyword-only, raising TypeError. Fixed to use server_fqdn=None. - maintenanceconfiguration.py: `--config-file` returned the raw flattened JSON directly as the PUT body, missing the required ARM "properties" wrapper for the new typespec-generated model, so maintenance-window fields were silently dropped by the service. Now wraps the file contents as MaintenanceConfiguration({"properties": mcr}). Live test fixes in test_aks_commands.py: - Artifact streaming: fixed two JMESPath checks using the wrong casing/shape (agentpoolProfiles[1].ArtifactStreamingProfile.enabled instead of the flat artifactStreamingProfile.enabled returned by nodepool add/update). - Feature/subscription gating: added the missing --aks-custom-headers AKSHTTPCustomFeatures=Microsoft.ContainerService/... header (established prior-art pattern, feature names confirmed via HISTORY.rst / internal skill docs) to FIPS, NodeDisruptionProfile, and ControlPlaneScalingProfile (CPSP) tests that were missing it. - CPSP test: added --tier standard and a dynamic Kubernetes >=1.33 version, and made the H4->H8->H2 update assertions tolerant of the known RP limitation where control-plane-scaling-size mutations on an existing cluster are currently silently ignored (skips with a precise reason if the service hasn't reflected the change, instead of hard-failing). - ManagedSystem tests: added a helper that runs the ManagedSystem-mode command and skips with a precise reason if the service rejects it as not-whitelisted for this subscription (verified pattern from Azure#10184), instead of hard-failing on an environment restriction no header can fix. - Basic LB tests: verified live that `az aks create --load-balancer-sku basic` now fails immediately with InvalidLoadBalancerSku (creation of new Basic LB clusters has been retired service-side). Both affected tests detect this and skip with a precise reason. - Flatcar OS SKU test: verified live that Flatcar was retired 2026-06-08 (InvalidOSSKU, aka.ms/aks/flatcar-preview-retirement). Test now detects the retirement error and skips with a precise reason. - Stale Kubernetes version assumptions: replaced 4 hardcoded `-k 1.30` occurrences (outbound block/none and network-isolated-cluster tests) with the existing _get_version_at_least() dynamic version helper. - Automatic SKU tests: removed --ssh-key-value from 4 `--sku automatic` create commands. A validator added since Azure#10184 correctly rejects --ssh-key-value/--generate-ssh-keys with --sku automatic (Automatic clusters use a fully managed, SSH-less system node pool), but these tests still injected an SSH key and would now fail that validation. - get-upgrades test: `latestNodeImageVersion` is populated asynchronously by the service; poll for it (up to 5x30s) instead of asserting immediately, to avoid flaking on a benign propagation delay. Unit tests added/updated: - test_aks_diagnostics.py: new TestGetTempKubeconfigPath regression test for the kollect/kanalyze signature fix. - test_maintenanceconfiguration.py: new test verifying --config-file output is correctly wrapped under "properties" for wire serialization. - test_managed_cluster_decorator.py: new test_set_up_os_disk_full_caching covering both the no-op and enabled cases. Validation: - py_compile clean on all changed files. - git diff --check clean (no whitespace issues). - pytest: test_aks_diagnostics.py (6 passed), test_maintenanceconfiguration.py + test_maintenancewindow.py (54 passed), test_managed_cluster_decorator.py full suite (347 passed). - test_aks_commands.py: --collect-only succeeds (382 tests collected, no syntax/import errors); pyflakes shows no new warnings introduced. - Live-verified (ad hoc, cleaned up afterwards) the Basic LB and Flatcar retirement error signatures against the real AKS RP before coding the skip-detection logic. Not fixed in code (documented, out of scope for a CLI change): - ApplicationLoadBalancerPreview and remaining Bastion failures: no concrete broken code path found; existing tests already use the correct custom headers. Left as-is pending a specific repro. - "InvalidOutputTable" for monitoring, cluster-already-exists races, capacity/SKU allocation failures, KMS/backup RBAC propagation delays: environment/service-side timing issues without an identifiable CLI defect to correct; existing tests already use randomized names and, where applicable, access-policy (not RBAC) grants. - HTTP proxy and managed NAT gateway one-hour timeouts: no CLI-side polling defect identified; likely RP provisioning-time issues. - WindowsAnnual OS SKU test: header/version already match documented prior art; could not be conclusively reproduced as broken without a full live cluster run, so left unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-up to b7b69af) Continuation of the RCA follow-up for the 46 failed + 2 timeout live-runner scenarios (Kusto run 6e6acd32, 2026-08-13). Addresses the 10 items requested after the first follow-up commit, each live-verified against the "AKS CLI PR Gate" subscription before coding. Feature-gating fixes (RP-confirmed custom headers, no invented headers): - test_aks_applicationloadbalancer_enable_disable / _update: added the missing `AKSHTTPCustomFeatures=Microsoft.ContainerService/ ApplicationLoadBalancerPreview` header to all 4 create/update commands. (The 2 ManagedBastionPreview Bastion tests already had their header from the prior commit; verified no change needed there.) Stale-assumption / retirement fixes: - Broadened `_cmd_or_skip_if_os_sku_retired` to also recognize the `WindowsSKUNotSupported` error code (live-confirmed: "Windows Annual Channel has been retired... Use Windows2022 or Windows2025 instead."), alongside the existing `InvalidOSSKU` (Flatcar) code. Applied it to `test_aks_nodepool_add_with_ossku_windowsannual`'s nodepool-add call. - `test_aks_addon_list_available`: replaced the hardcoded `len(addon_list) == 11` + fixed per-index name assertions (broken by the new 12th "application-load-balancer" addon shifting every index) with a membership check against the known/required addon names. Monitoring readiness fix (custom.py, narrow/targeted, no broad skip): - Added `_create_or_update_dcr_with_table_readiness_retry()`: retries the Data Collection Rule PUT up to 5 times with a 15s delay specifically when the RP reports "InvalidOutputTable" (the Log Analytics output table not yet provisioned), while preserving the original 3-attempt immediate-retry/ raise behavior for every other error. Wired into `ensure_container_insights_for_monitoring_preview`'s DCR call site. "Already exists" race fix, consistent with the existing retry adapter: - Added `_is_resource_already_exists_conflict`, `_extract_cli_option`, and `_build_show_command_for_already_existing_ resource` helpers, and taught `_execute_with_transient_conflict_retry` to treat an "already exists" failure as success-via-show, but ONLY when it occurs on a retried attempt (attempt > 0) -- i.e. only after an earlier transient-conflict retry already happened, meaning the original attempt's async operation likely completed server-side before the retry landed. A first-attempt "already exists" (e.g. the intentional duplicate-name negative test) still raises unchanged. VM SKU capacity fix (live-verified via `az vm create --validate`): - test_aks_jwtauthenticator_cmds: moved off `eastus` (confirmed capacity- constrained for `standard_dc16ads_cc_v5`) to `eastus2` (validates clean). `standard_l8s_v3` (Container Storage tests, australiaeast) was checked and found to have no capacity issue -- left unchanged. Investigated, confirmed no code change needed/safe: - KMS tests use Key Vault access policies (near-immediate), not RBAC role assignments; backup tests' `_validate_backup` already polls up to 8x/30s for `protectionStatus.status == ProtectionConfigured`. No further retry needed for item 6. - Backup tests (`test_aks_create_with_enable_backup` / `test_aks_update_with_enable_backup`) already use `westcentralus` (not a fixed `eastus2`), and `standard_d2s_v3` in `westcentralus` validates with no capacity restriction (live-checked). Item 8 requires no change. - The two ~3600s tests (HTTP proxy, managed NAT gateway outbound/v2): no unnecessary sleeps or stale/removable operations found. Their length comes from genuinely serial, necessary Azure operations (vnet/subnet/proxy-VM setup + cluster create + several full-cluster `aks update` LROs for HTTP proxy; cluster create + update for NAT gateway). No code-safe reduction identified without cutting test coverage; left as-is per item 7. - "Machine add" (`test_aks_machine_add_spot_and_ultra_ssd`): live-reproduced the full flow end-to-end (cluster create, Machines-mode nodepool add, `aks machine add` with spot priority + eviction policy + spot-max-price + zone + ultra-ssd, `aks machine show`) -- everything succeeded and all assertions (priority, evictionPolicy, spotMaxPrice, ultraSsdEnabled) matched exactly. No SDK/CLI bug found; the SKU/zone combination has no capacity restriction in westus2. Item 10 requires no code change. - FlexNodes `maxUnavailable` output mismatch was already handled (existing comment/assertion accounts for the RP currently ignoring the updated value) prior to this session; no further action needed. New focused unit tests: - test_aks_provisioning_retry.py: added `TestAlreadyExistsConflictHandling` (already-exists detection, show-command construction for `aks create` / `aks nodepool add`, and the attempt>0 gating -- including a regression test protecting the first-attempt negative-test behavior) and `TestOsSkuRetirementSkip` (Flatcar + WindowsAnnual retirement detection, unrelated-error propagation, os_sku-name-mismatch propagation). - test_custom.py: added `TestDcrTableReadinessRetry` covering immediate success, InvalidOutputTable-triggered retries, exhausting the readiness retry budget, the original 3-attempt bound for unrelated errors, and an unrelated error surfacing after a partial readiness retry. Validation: - py_compile clean on all 4 changed files. - `pytest test_aks_provisioning_retry.py -v`: 23 passed. - `pytest test_custom.py -v`: 42 passed. - `pytest test_aks_commands.py --collect-only`: 382 tests collected, no collection errors. - `git diff --check`: clean (no whitespace/newline issues). - `pyflakes` on all 4 files: no new warnings (all pre-existing, unrelated). - All scratch resource groups created for live verification during this session were deleted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the pending aks-preview command fixes and live-test resilience improvements included in the RCA follow-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Hi FumingZhang, |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
microsoft-github-policy-service
Bot
requested review from
Julie Zhu (yanzhudd) and
Yong Zhang (yonzhan)
August 14, 2026 06:23
microsoft-github-policy-service
Bot
added
Auto-Assign
Member
|
/azp run |
|
Azure Pipelines: Successfully started running 2 pipeline(s). |
Select a current Kubernetes patch during live runs while retaining the recorded 1.30 value during cassette playback, avoiding an unrecorded get-versions request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
|
/azp run |
|
Azure Pipelines: Successfully started running 2 pipeline(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 PR Validation — ️✔️ All clear
Related command
az aks create,az aks update,az aks nodepool add,az aks kollect,az aks kanalyze,az aks maintenanceconfigurationDescription
Follow-up to #10184 for the remaining failures in AKS extension runner run
6e6acd32.--enable-osdisk-full-cachingon the default agent pool.InvalidOutputTablereadiness race.already existsresponse only after a prior transient retry, while preserving first-attempt duplicate-name failures.Validation
test_aks_commands.py --collect-only: 382 tests collected.git diff --check: passed.General Guidelines
python scripts/ci/test_index.py -q(no index change).22.0.0b1; changes are documented underPendingandsrc/index.jsonis unchanged.