Skip to content

fix: replace postinstall download with optional platform packages#17

Open
smorimoto wants to merge 1 commit into
upstash:masterfrom
smorimoto:fix/optional-platform-packages
Open

fix: replace postinstall download with optional platform packages#17
smorimoto wants to merge 1 commit into
upstash:masterfrom
smorimoto:fix/optional-platform-packages

Conversation

@smorimoto
Copy link
Copy Markdown

@smorimoto smorimoto commented May 6, 2026

Summary

  • Replaces the unreliable postinstall-based binary download (which silently fails under bunx / pnpx because their dlx hooks are disabled by default for security — see issue running the cli using bunx #6 and bug: Bus error when using pnpm on Linux #9) with the optional platform packages pattern used by esbuild, biome, and similar tooling.
  • Adds a small CommonJS launcher at bin/qstash that resolves the platform-specific binary via require.resolve and execs it, propagating the exit code and signal.
  • Adds six platform packages (@upstash/qstash-cli-<platform>-<arch>) under packages/ with os / cpu filters and their own publishConfig, so access and provenance behaviour stays consistent locally and in CI.
  • Rewrites the release workflow to publish all six platform packages in a matrix, then publish the main package with optionalDependencies pinned to the same version.
  • Drops install.ts, tsconfig.json, bun.lockb, bin/.gitkeep, and the now-unused tar / unzipper dependencies.

Closes #6
Closes #9

The previous `postinstall` approach silently fails under `bunx` and `pnpx`
(issues upstash#6 and upstash#9), where dlx-style install hooks are disabled by default
for security. Migrate to the optional platform packages pattern used by
esbuild, biome, and similar tooling.

- Add a small CommonJS launcher at `bin/qstash` that resolves the
  platform-specific binary via `require.resolve` and execs it,
  propagating the exit code and any terminating signal.
- Introduce six platform packages under `packages/` with `os` / `cpu`
  filters so that npm installs only the matching binary. Each declares
  its own `publishConfig`, so access and provenance behaviour stays
  consistent whether published from CI or locally.
- Rewrite the release workflow to publish all six platform packages in
  a matrix, then publish the main package with `optionalDependencies`
  pinned to the same version.
- Drop `install.ts`, `tsconfig.json`, `bun.lockb`, and `bin/.gitkeep`
  along with the now-unused `tar` and `unzipper` dependencies.

Closes upstash#6
Closes upstash#9
@smorimoto smorimoto force-pushed the fix/optional-platform-packages branch from 1aef881 to ef556d7 Compare May 6, 2026 16:19
@tombeckenham
Copy link
Copy Markdown

Hi @smorimoto — really nice work here. I opened #18 a few hours ago after running into the same bunx breakage from a different angle, but yours predates mine and takes the same fundamental approach. Closing #18 in favor of this one to avoid splintering review attention. Posting the security-hardening I did on top of the same pattern below in case any of it is useful here.

Three things this PR does better than what I built, worth keeping regardless of any other changes:

  1. Signal forwarding in the shimprocess.kill(process.pid, result.signal) correctly re-raises Ctrl-C / SIGTERM so the parent shell sees the right exit code. I missed this; real correctness fix.
  2. publishConfig: { access: "public", provenance: true } in each subpackage's package.json — cleaner than passing those at the CLI.
  3. Richer missing-subpackage error that names --no-optional / --omit=optional as the likely cause.

Suggestions below, roughly priority order. None are blockers; all mechanically simple if you want them.


1. Verify the downloaded binary against the upstream SHA-256 manifest

artifacts.upstash.com already publishes a goreleaser-style checksums file at https://artifacts.upstash.com/qstash/versions/<VERSION>/qstash-server_<VERSION>_checksums.txt in standard sha256sum -c format. Closes the asymmetric-swap case (binary replaced on the artifact host, manifest untouched). Currently the only integrity check is TLS.

curl --fail --silent --show-error --location --output checksums.txt \
  "https://artifacts.upstash.com/qstash/versions/${VERSION}/qstash-server_${VERSION}_checksums.txt"
grep -F "  archive.${{ matrix.ext }}" checksums.txt > expected.sha256 \
  || { echo "Archive not listed in checksums.txt" >&2; exit 1; }
sha256sum -c expected.sha256

2. Sanity-check the extracted binary's file-type matches the slot

This PR closes #9 ("Bus error when using pnpm on Linux"), which is the classic wrong-binary-in-wrong-slot symptom. The checksum from (1) catches "binary swapped" but not "binary uploaded to the wrong filename." A file --brief substring check on the extracted binary catches that and would prevent #9 from recurring:

case "${node_os}-${node_arch}" in
  darwin-arm64) family=Mach-O arch=arm64   ;;
  darwin-x64)   family=Mach-O arch=x86_64  ;;
  linux-arm64)  family=ELF    arch=aarch64 ;;
  linux-x64)    family=ELF    arch=x86-64  ;;
  win32-arm64)  family=PE32   arch=Aarch64 ;;
  win32-x64)    family=PE32   arch=x86-64  ;;
esac
out=$(file --brief "bin/$bin_name")
[[ "$out" == *"$family"* && "$out" == *"$arch"* ]] \
  || { echo "wrong binary in slot: $out" >&2; exit 1; }

3. Pin every actions/* use to a 40-char commit SHA

@v4 is mutable. Concrete SHAs (latest stable, verified round-trip via the GitHub API):

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd          # v6.0.2
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e        # v6.4.0
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a   # v7.0.1
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1

4. Split the workflow into build → publish-platform → publish-wrapper

Current shape does setup → download → extract → publish in one matrix step. If extraction fails on matrix entry #4, entries #1-3 may have already published, and the registry can't be reverted without a version bump. A build job that runs all 6 download+verify+extract steps and uploads each subpackage as a workflow artifact, then a publish-platform job (matrix, needs: build) that downloads and publishes, gives a real validation gate before the first npm publish runs.

5. Concurrency guard

concurrency:
  group: release-${{ github.ref }}
  cancel-in-progress: false

6. Poll npm for each optionalDependency before publishing the wrapper

Brief propagation window after publish where npm view ...@<v> can 404 from some POPs. A user installing in that window lands on the shim's "could not locate" error.

deps=$(jq -r '.optionalDependencies | keys[]' package.json)
for dep in $deps; do
  attempt=0
  until npm view "${dep}@${VERSION}" version >/dev/null 2>&1; do
    attempt=$((attempt + 1))
    [ "$attempt" -ge 6 ] && { echo "timed out: $dep@${VERSION}" >&2; exit 1; }
    sleep $((attempt * 5))
  done
done

7. Keep OIDC for publish (don't reintroduce NPM_TOKEN)

#15 moved publishing to tokenless OIDC in April. This PR re-adds:

env:
  NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

which is a regression — it reintroduces a long-lived secret. With permissions: id-token: write already declared at the top of this workflow and publishConfig: { provenance: true } in each package, npm publish authenticates via OIDC and doesn't need the token.

8. (Minor) Workflow expression injection hardening

${{ matrix.X }} reaching run: blocks works fine here because matrix values are static, but indirecting through env: is the recommended pattern. Same for GITHUB_REF:

env:
  REF: ${{ github.ref }}
run: |
  VERSION="${REF#refs/tags/v}"
  echo "VERSION=$VERSION" >> "$GITHUB_ENV"

9. (Tiny) tar --no-same-owner --no-same-permissions

Defense in depth. Modern tar already rejects .. paths and the target dir is controlled, but flags cost nothing.


Happy to open a PR against your branch with any subset of these if it'd help.

Disclosure: this work and the comment were developed with Claude Code (Opus 4.7) assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Bus error when using pnpm on Linux issue running the cli using bunx

2 participants