Skip to content

fix(ci): let the arm-ttk gate decide the azure package job, not the error stream - #309

Merged
cevheri merged 1 commit into
mainfrom
fix/azure-armttk-warning-gate
Aug 7, 2026
Merged

fix(ci): let the arm-ttk gate decide the azure package job, not the error stream#309
cevheri merged 1 commit into
mainfrom
fix/azure-armttk-warning-gate

Conversation

@cevheri

@cevheri cevheri commented Aug 7, 2026

Copy link
Copy Markdown
Member

What

The first real dispatch of the Azure Marketplace Package workflow after #307 merged failed — and not on anything wrong with the package. arm-ttk reported 35 tests, 0 failed, 1 warning, which is the expected result.

arm-ttk's individual tests report through the error stream, and Test-AzTemplate collects that stream into each result's .Errors / .Warnings (. $myModule $TheTest @testInput 2>&1 3>&1). GitHub's shell: pwsh runs with $ErrorActionPreference = 'Stop', so the first such write became a terminating error inside the toolkit at Test-AzTemplate.ps1:253, and the job's own gate —

$failed = $results | Where-Object { $_.Errors }
if ($failed) { … exit 1 }

— never executed. A finding arm-ttk itself classifies as a warning was therefore indistinguishable from a failure: this job could not have passed while any warning existed.

Why the fix is in the job, not the template

The warning is URIs Should Be Properly Constructed on applicationUrl. Silencing it at the template is not the right move:

  • the test's own source carries the comment # commenting out the test due to # 417;
  • it reports an empty function name (Function '') — the regex does not even capture it;
  • format is on its $FunctionNotAllowedInUri list alongside concat, so rewriting the output as format(...) would not clear it either;
  • arm-ttk classifies the result as a warning with Errors empty. The job should agree with it.

The change

$ErrorActionPreference = 'Continue' before the call, so the toolkit finishes and the declared gate decides. Plus two things the original step lacked:

  • each finding is surfaced as a GitHub annotation (::warning / ::error) with the test name;
  • an empty result set now fails. A wrong -TemplatePath or a half-imported module would otherwise sail through the gate as "nothing failed".

Verification

The step's run: block was extracted from the workflow file itself — not retyped — and executed under the same shell semantics ($ErrorActionPreference = 'stop' prepended, pwsh -command ". 'file'") against the pinned toolkit release:

package exit code output
the real one 0 arm-ttk: 35 tests, 0 failed, 1 warned
with a planted securestring defaultValue 1 ::error … Secure String Parameters Cannot Have Default

The second row matters more than the first: a gate that was just loosened has to be shown to still fail.

Then dispatched for real on this branch — run 31170797677: success, every step green, arm-ttk: 35 tests, 0 failed, 1 warned, artifact uploaded (15 KB, the two-file package zip).

Note on the earlier verification

The local check in #307 reported arm-ttk's verdict accurately but ran in a plain container, where $ErrorActionPreference defaults to Continue — so it could not predict the job's outcome. Reproducing the tool is not the same as reproducing the job; this PR's check does the latter.

…rror stream

The first real dispatch of the Azure Marketplace Package workflow failed, and not
on anything wrong with the package: arm-ttk reported 35 tests, 0 failed, 1 warning,
which is the expected result.

arm-ttk's individual tests report through the ERROR stream and Test-AzTemplate
collects that stream into each result's .Errors / .Warnings. GitHub's `shell: pwsh`
runs with $ErrorActionPreference = 'Stop', so the first such write became a
TERMINATING error inside the toolkit at Test-AzTemplate.ps1:253 and the job's own
gate - `$failed = $results | Where-Object { $_.Errors }` - never executed. A finding
arm-ttk itself classifies as a warning was therefore indistinguishable from a
failure, and this job could not have passed while any warning existed.

The warning is `URIs Should Be Properly Constructed` on `applicationUrl`. It is not
worth silencing at the template: the test's own source is annotated "commenting out
the test due to # 417", it reports an empty function name (`Function ''`), and
`format` is on its disallowed list alongside `concat`, so rewriting the output would
not clear it. arm-ttk classifies it as a warning; the job should too.

So the fix is in the job: set $ErrorActionPreference = 'Continue' before the call,
let the toolkit finish, and gate on `.Errors` as originally intended. Also surfaces
each finding as a GitHub annotation, and fails when the result set is EMPTY - a wrong
-TemplatePath would otherwise sail through the gate as "nothing failed".

Verified by extracting the step's `run:` block from the workflow itself and executing
it under the same shell semantics ($ErrorActionPreference = 'stop' prepended, pwsh
-command ". 'file'"), against the pinned toolkit release:

- real package     -> exit 0, "35 tests, 0 failed, 1 warned"
- package with a planted securestring defaultValue -> exit 1, error annotated

The second run matters more than the first: a gate that was just loosened has to be
shown to still fail.
@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2026

Copy link
Copy Markdown

@cevheri
cevheri merged commit 5324f93 into main Aug 7, 2026
19 checks passed
@cevheri
cevheri deleted the fix/azure-armttk-warning-gate branch August 7, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant