fix(ci): let the arm-ttk gate decide the azure package job, not the error stream - #309
Merged
Conversation
…rror stream
The first real dispatch of the Azure Marketplace Package workflow failed, and not
on anything wrong with the package: arm-ttk reported 35 tests, 0 failed, 1 warning,
which is the expected result.
arm-ttk's individual tests report through the ERROR stream and Test-AzTemplate
collects that stream into each result's .Errors / .Warnings. GitHub's `shell: pwsh`
runs with $ErrorActionPreference = 'Stop', so the first such write became a
TERMINATING error inside the toolkit at Test-AzTemplate.ps1:253 and the job's own
gate - `$failed = $results | Where-Object { $_.Errors }` - never executed. A finding
arm-ttk itself classifies as a warning was therefore indistinguishable from a
failure, and this job could not have passed while any warning existed.
The warning is `URIs Should Be Properly Constructed` on `applicationUrl`. It is not
worth silencing at the template: the test's own source is annotated "commenting out
the test due to # 417", it reports an empty function name (`Function ''`), and
`format` is on its disallowed list alongside `concat`, so rewriting the output would
not clear it. arm-ttk classifies it as a warning; the job should too.
So the fix is in the job: set $ErrorActionPreference = 'Continue' before the call,
let the toolkit finish, and gate on `.Errors` as originally intended. Also surfaces
each finding as a GitHub annotation, and fails when the result set is EMPTY - a wrong
-TemplatePath would otherwise sail through the gate as "nothing failed".
Verified by extracting the step's `run:` block from the workflow itself and executing
it under the same shell semantics ($ErrorActionPreference = 'stop' prepended, pwsh
-command ". 'file'"), against the pinned toolkit release:
- real package -> exit 0, "35 tests, 0 failed, 1 warned"
- package with a planted securestring defaultValue -> exit 1, error annotated
The second run matters more than the first: a gate that was just loosened has to be
shown to still fail.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What
The first real dispatch of the Azure Marketplace Package workflow after #307 merged failed — and not on anything wrong with the package. arm-ttk reported
35 tests, 0 failed, 1 warning, which is the expected result.arm-ttk's individual tests report through the error stream, and
Test-AzTemplatecollects that stream into each result's.Errors/.Warnings(. $myModule $TheTest @testInput 2>&1 3>&1). GitHub'sshell: pwshruns with$ErrorActionPreference = 'Stop', so the first such write became a terminating error inside the toolkit atTest-AzTemplate.ps1:253, and the job's own gate —— never executed. A finding arm-ttk itself classifies as a warning was therefore indistinguishable from a failure: this job could not have passed while any warning existed.
Why the fix is in the job, not the template
The warning is
URIs Should Be Properly ConstructedonapplicationUrl. Silencing it at the template is not the right move:# commenting out the test due to # 417;Function '') — the regex does not even capture it;formatis on its$FunctionNotAllowedInUrilist alongsideconcat, so rewriting the output asformat(...)would not clear it either;Errorsempty. The job should agree with it.The change
$ErrorActionPreference = 'Continue'before the call, so the toolkit finishes and the declared gate decides. Plus two things the original step lacked:::warning/::error) with the test name;-TemplatePathor a half-imported module would otherwise sail through the gate as "nothing failed".Verification
The step's
run:block was extracted from the workflow file itself — not retyped — and executed under the same shell semantics ($ErrorActionPreference = 'stop'prepended,pwsh -command ". 'file'") against the pinned toolkit release:arm-ttk: 35 tests, 0 failed, 1 warnedsecurestringdefaultValue::error … Secure String Parameters Cannot Have DefaultThe second row matters more than the first: a gate that was just loosened has to be shown to still fail.
Then dispatched for real on this branch — run 31170797677: success, every step green,
arm-ttk: 35 tests, 0 failed, 1 warned, artifact uploaded (15 KB, the two-file package zip).Note on the earlier verification
The local check in #307 reported arm-ttk's verdict accurately but ran in a plain container, where
$ErrorActionPreferencedefaults toContinue— so it could not predict the job's outcome. Reproducing the tool is not the same as reproducing the job; this PR's check does the latter.