Release-job partial failure leaves main package-lock.json drifted (self-heal skipped) — data-access 4.0.0 release

[rainer-friederich]

Summary

main's root package-lock.json is currently drifted and every open PR fails the Test job at the Verify package-lock.json is in sync step with a diff it did not author. Root cause: the 4.0.0 release of spacecat-shared-data-access published successfully and pushed its release commit, but the Semantic Release step then failed later in the run, which caused the release job's self-heal step (Sync package-lock.json after release) to be skipped. So the lock was left pinning data-access@3.81.0 while package.json says 4.0.0.

This is the exact "main-wide breakage that first surfaces on unrelated PRs" the drift guard warns about — but the mechanism that hit us is not the one docs/RELEASE-RUNBOOK.md Failure mode 7 documents. FM-7 covers "the sync step itself fails (red ✗)"; here the sync step never ran at all because the upstream Semantic Release step failed after already pushing the version bump.

Current broken state (verify)

On origin/main (a7b72cc7 chore(release): 4.0.0 [skip ci]):

• packages/spacecat-shared-data-access/package.json → 4.0.0
• root package-lock.json entry for packages/spacecat-shared-data-access → still 3.81.0
• There is no chore(release): sync package-lock.json after release [skip ci] commit after the 4.0.0 release (contrast: the previous 1.124.1 release has one at 0a692a45).

Timeline / evidence

• Release run: 28669792110 (commit a7b72cc7, Release job conclusion = failure).
• Release job step conclusions:
  • Semantic Release → failure
  • Sync package-lock.json after release → skipped
  • Surface failure pointer to release runbook → success (if: failure())
• Semantic Release published data-access@4.0.0 (OIDC token exchange succeeded, GitHub release created, tag @adobe/spacecat-shared-data-access-v4.0.0 pushed), then aborted with:

  ✘ An error occurred while running semantic-release: TypeError: Cannot read properties of undefined (reading 'map')
  AggregateError:
      SemanticReleaseError: No npm token specified.
  TypeError: Cannot read properties of undefined (reading 'map')
  

  i.e. the failure happened after the release commit + tag were already pushed, so the drift was created but the sync step never ran.

Why the self-heal didn't heal (the design gap)

In .github/workflows/main.yaml, the Release job:

• Semantic Release (~line 225) has no continue-on-error, so its failure fails the job and short-circuits later if: success() steps.
• Sync package-lock.json after release (~line 230) is gated if: success() (line 243) with continue-on-error: true (line 244).

The continue-on-error: true only protects against the sync step's own failure. It does nothing when the failure is in the upstream Semantic Release step — in that case if: success() simply skips the sync step. And because semantic-release pushes the release commit (the thing that drifts the lock) and can fail afterwards, the single scenario that most needs the lock re-sync is exactly the one where it is skipped. The documented backstop ("the next PR drift guard will surface it") then converts a release-job failure into a main-wide contributor tax — every PR is red until someone manually files a lockfile PR.

What needs to be done

1. Immediate — unblock main (lockfile-only PR, no republish). Per RELEASE-RUNBOOK FM-7 recovery:

git checkout main && git pull
npm install --package-lock-only --ignore-scripts   # only change: data-access 3.81.0 -> 4.0.0
git checkout -b fix/sync-package-lock
git add package-lock.json
git commit -m "fix: sync package-lock.json with released workspace versions (data-access 4.0.0)"
# open PR; drift guard on the PR confirms the fix

(Verified locally that npm install --package-lock-only produces exactly this one-line change.)

2. Systemic — make the self-heal resilient to a partial Semantic Release failure. The sync step must run even when Semantic Release fails after pushing a release commit. Options to evaluate:

• Change the sync step gate from if: success() to if: always() (or if: success() || failure()), keeping continue-on-error: true, so a partial SR failure still triggers the lock re-sync. Guard it so it only commits when the lock actually drifted (the step already does a git diff --quiet check).
• Alternatively, add the lockfile to the @semantic-release/git assets list so the lock is committed atomically with the version bump in the SAME release commit (removes the whole "separate sync step" failure surface). Assess why it was kept out originally (the workflow comment says the assets list is package.json + CHANGELOG.md only).

3. Diagnose the underlying Semantic Release failure so releases stop half-failing. data-access@4.0.0 published fine via OIDC, yet the run still aggregated SemanticReleaseError: No npm token specified. + TypeError: ... reading 'map'. Determine which plugin/package instance raised "No npm token specified" after a successful OIDC publish (only data-access had commits to release this run — every other package reported "no release"), and whether the reading 'map' TypeError is a secondary crash in semantic-release-monorepo's error handling masking the real cause. See RELEASE-RUNBOOK for the OIDC publish path.

4. Document this failure mode. RELEASE-RUNBOOK FM-7 only describes "the sync step itself fails." Add the "SR fails after pushing the release commit → sync step skipped → main drifted" variant and its recovery, so on-call recognizes it.

References

• Workflow: .github/workflows/main.yaml (drift guard ~line 120; Release job sync step ~line 230)
• Runbook: docs/RELEASE-RUNBOOK.md Failure mode 7 (line 415)
• Failing release run: https://github.com/adobe/spacecat-shared/actions/runs/28669792110
• Broken release commit: a7b72cc7 chore(release): 4.0.0 [skip ci]
• Example unrelated PR blocked by this: feat(project-engine-client): re-vendor Semrush swagger; retire CR9/CR10, slim CR16 #1772 (worked around with a per-branch lock-sync commit)

[rainer-friederich]

Reproduced a second time on the very next release. The project-engine-client@1.9.0 release (merge of #1772, run 28773336313) followed the identical pattern:

• 1.9.0 published to npm + tag + GitHub release created successfully (via OIDC), THEN
• Semantic Release failed: ✘ Failed step "verifyConditions" of plugin "@semantic-release/npm" → ENONPMTOKEN No npm token specified. + TypeError: Cannot read properties of undefined (reading 'map'), so
• Sync package-lock.json after release was skipped again → main's root lock is drifted again (project-engine-client pinned 1.8.0 in the lock vs 1.9.0 in package.json).

New signal narrowing the SR root cause (task 3): the failure is specifically verifyConditions of @semantic-release/npm firing for a package instance after the first package already published successfully via OIDC — i.e. the npm OIDC auth context appears to be established only for the first publishing package, and a subsequent package's verifyConditions falls back to token auth (ENONPMTOKEN). Two consecutive releases (data-access 4.0.0, project-engine-client 1.9.0) have now failed this way, so it is systematic, not a one-off — every release currently drifts main's lock. Prioritizing tasks 1 (unblock main again) and 2 (make the self-heal run on partial SR failure) accordingly.