Skip to content

Commit 9ce6d72

Browse files
authored
ci(release): fix OIDC trusted publishing, add Slack alert, allow manual re-runs (#2275)
## Summary Four ci-only changes to `release.yml` that together unblock npm OIDC trusted publishing, route future failures to Slack, and give us a manual retry path: 1. **Upgrade npm to `latest` in `check_changesets`** so `npm publish` can perform the OIDC handshake. Runner ships Node 22 + npm 10.9.4 by default, but client-side OIDC trusted publishing landed in **npm 11.5.1** (July 2025). Without this upgrade `npm publish` has no OIDC code path, falls back to looking for a `NODE_AUTH_TOKEN` (deliberately not set, to avoid conflicting with OIDC), and exits `ENEEDAUTH`. 2. **Drop `continue-on-error: true`** from the `Create Release Pull Request or Publish to npm` step so a future publish failure fails the workflow run instead of leaving it green with no releases shipped. 3. **Add `slack-notify-failure` job** mirroring the pattern in `release-nightly.yml`: gated on `if: failure() && always()`, waits on every major job, lists the failed ones via `listJobsForWorkflowRun`, and posts a danger-colored Slack message to `SLACK_WEBHOOK_URL_ENG_NOTIFS` with the commit SHA + author. Combined with (2), the next release.yml failure surfaces on Slack instead of going silent. 4. **Add `workflow_dispatch:` trigger** so we can manually re-run the workflow against an arbitrary `main` SHA (`gh workflow run release.yml --ref main` or the UI "Run workflow" button). Needed for the recovery path below — `gh run rerun <run-id>` re-uses the original commit's workflow file, so it can't test fixes added in later commits. ## Why we didn't catch this on the April-23 OIDC switch The April-23 release that introduced OIDC ([commit 3ae1b85](3ae1b85d8)) only **appeared** to work — OIDC never actually fired. Sequence: 1. [Run 24862921978](https://github.com/hyperdxio/hyperdx/actions/runs/24862921978) (2026-04-23, on the release-PR commit before the OIDC switch) tried to publish `@hyperdx/cli@0.4.0` and failed with `E404 Not Found - PUT /@HyperDX%2fcli` — same `continue-on-error: true` mask, workflow green, no GH Releases created. 2. Between 22:55 and 23:30 UTC, `@hyperdx/cli@0.4.0` was published out of band (manual `npm publish`). 3. [Run 24864061403](https://github.com/hyperdxio/hyperdx/actions/runs/24864061403), kicked off by the OIDC commit itself, re-ran `changeset publish` which saw the package was already on npm and emitted: ``` 🦋 warn @hyperdx/cli is not being published because version 0.4.0 is already published on npm 🦋 warn No unpublished projects to publish ``` `changeset publish` then exited zero, `changesets/action` proceeded with its `createRelease` loop, and the five `@hyperdx/{api,app,cli,common-utils,otel-collector}` GH Releases got created. OIDC was never exercised — the publish was a no-op. Today's 2.25.0 release ([run 25827737203](https://github.com/hyperdxio/hyperdx/actions/runs/25827737203)) had no out-of-band publish to fall back on, so the underlying `ENEEDAUTH` surfaced as a real failure for `@hyperdx/cli@0.4.1`. `changesets/action` aborts inside `runPublish` (the `execWithOutput` call sits above the `createGithubReleases` loop, and it re-throws on non-zero exit), so neither the cli nor the four private-package GH Releases (`@hyperdx/api@2.25.0`, `@hyperdx/app@2.25.0`, `@hyperdx/common-utils@0.19.0`, `@hyperdx/otel-collector@2.25.0`) ever got created. The lone `cli-v0.4.1` entry on the Releases page came from the separate `release-cli` job using `softprops/action-gh-release` — not from changesets. The npm trusted-publisher config also had a `release.yaml` → `release.yml` filename typo (fixed in the npm UI). Real bug, but **independent** — when the npm client never sends an OIDC token in the first place, registry-side trusted-publisher matching is moot. ## Recovery for the stuck 2.25.0 / cli@0.4.1 cycle After merging this PR: ```bash gh workflow run release.yml --ref main ``` The run will: - `check_changesets`: install npm `latest`, see no changesets, `yarn release` → genuine OIDC `npm publish` of `@hyperdx/cli@0.4.1` (with provenance) → push the five 2.25.0 / 0.19.0 git tags → `changesets/action` creates all five GH Releases. - `check_version`: `manifest inspect` finds the existing 2.25.0 image tag → `should_release=false` → all Docker build / publish / downstream-notify jobs short-circuit. - `release-cli`: `gh release view cli-v0.4.1` succeeds → `exists=true` → compile + create-release steps are skipped. - `slack-notify-failure`: only runs if any of the above fail — silent on success.
1 parent 46fe675 commit 9ce6d72

1 file changed

Lines changed: 87 additions & 1 deletion

File tree

‎.github/workflows/release.yml‎

Lines changed: 87 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ name: Release
22
on:
33
push:
44
branches: [main]
5+
workflow_dispatch:
56
permissions:
67
contents: write
78
packages: write
@@ -26,6 +27,11 @@ jobs:
2627
cache: 'yarn'
2728
# Note: Do NOT set registry-url here — it creates a NODE_AUTH_TOKEN
2829
# that conflicts with npm OIDC trusted publishing
30+
# npm OIDC trusted publishing requires npm >= 11.5.1 client-side. Node 22
31+
# LTS ships npm 10.x, which has no OIDC code path and falls back to
32+
# NODE_AUTH_TOKEN, failing ENEEDAUTH.
33+
- name: Upgrade npm for OIDC trusted publishing
34+
run: npm install -g npm@latest
2935
- name: Configure npm registry
3036
run: npm config set registry https://registry.npmjs.org/
3137
- name: Install root dependencies
@@ -39,7 +45,6 @@ jobs:
3945
run: make ci-build
4046
- name: Create Release Pull Request or Publish to npm
4147
if: always()
42-
continue-on-error: true
4348
id: changesets
4449
uses: changesets/action@v1
4550
with:
@@ -687,6 +692,86 @@ jobs:
687692
}
688693
});
689694
695+
# ---------------------------------------------------------------------------
696+
# Failure notification
697+
# ---------------------------------------------------------------------------
698+
slack-notify-failure:
699+
needs:
700+
[
701+
check_changesets,
702+
publish-app,
703+
publish-otel-collector,
704+
publish-local,
705+
publish-all-in-one,
706+
release-cli,
707+
notify_helm_charts,
708+
notify_ch,
709+
notify_clickhouse_clickstack,
710+
]
711+
runs-on: ubuntu-24.04
712+
if: failure()
713+
steps:
714+
- name: Get failed steps
715+
id: get_failed_steps
716+
uses: actions/github-script@v9
717+
with:
718+
github-token: ${{ secrets.GITHUB_TOKEN }}
719+
script: |
720+
const { data } = await github.rest.actions.listJobsForWorkflowRun({
721+
owner: context.repo.owner,
722+
repo: context.repo.repo,
723+
run_id: context.runId,
724+
});
725+
726+
const failedSteps = data.jobs
727+
.filter(job => job.conclusion === 'failure')
728+
.flatMap(job => {
729+
const stepNames = (job.steps ?? [])
730+
.filter(step => step.conclusion === 'failure')
731+
.map(step => `${job.name} → ${step.name}`);
732+
return stepNames.length > 0 ? stepNames : [`${job.name} → (unknown step)`];
733+
})
734+
.join(', ');
735+
736+
core.setOutput('failed_steps', failedSteps || 'unknown');
737+
- name: Slack Notification
738+
uses: 8398a7/action-slack@v3
739+
with:
740+
status: custom
741+
fields: repo,workflow
742+
custom_payload: |
743+
{
744+
"text": "Release workflow failed!",
745+
"attachments": [{
746+
"color": "danger",
747+
"fields": [
748+
{
749+
"title": "Failed step(s)",
750+
"value": "${{ steps.get_failed_steps.outputs.failed_steps }}",
751+
"short": false
752+
},
753+
{
754+
"title": "Commit",
755+
"value": "<https://github.com/${{ github.repository }}/commit/${{ github.sha }}|${{ github.sha }}>",
756+
"short": false
757+
},
758+
{
759+
"title": "Actor",
760+
"value": "${{ github.actor }}",
761+
"short": true
762+
},
763+
{
764+
"title": "Run",
765+
"value": "<https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|${{ github.run_id }}>",
766+
"short": true
767+
}
768+
]
769+
}]
770+
}
771+
env:
772+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL_ENG_NOTIFS }}
773+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
774+
690775
otel-cicd-action:
691776
if: always()
692777
name: OpenTelemetry Export Trace
@@ -702,6 +787,7 @@ jobs:
702787
notify_helm_charts,
703788
notify_ch,
704789
notify_clickhouse_clickstack,
790+
slack-notify-failure,
705791
]
706792
steps:
707793
- name: Export workflow

0 commit comments

Comments
 (0)