Skip to content

Stepping back from 1.9 stalls the RustFS Deployment on its deleted ServiceAccount #352

Open

Description

@teemow

Problem

A Helm upgrade from 1.9.0-rc.1 back to a 1.8.x release stalls on the bundled RustFS Deployment. 1.9 renders a rustfs ServiceAccount and sets serviceAccountName: rustfs on the Deployment's pod template; 1.8.x renders neither. On the step back Helm deletes the ServiceAccount and drops serviceAccountName, but the API server keeps the deprecated mirror field spec.template.spec.serviceAccount: rustfs, which re-fills serviceAccountName. The new ReplicaSet cannot create a pod:

pods "rustfs-…" is forbidden: error looking up service account ate-system/rustfs: serviceaccount "rustfs" not found

The release waits until its timeout and every component that depends on Substrate stays not ready. The forward step (1.8.x to 1.9) is not affected; the same stall follows any later release that removes or renames that ServiceAccount.

Proposed solution

The chart renders the deprecated serviceAccount field next to serviceAccountName on every pod template that names its own ServiceAccount (RustFS first), so a later release that changes the name also replaces the mirror field; or the RustFS ServiceAccount stays rendered (and unchanged in name) from 1.9 on. Removing both fields from the Deployment by hand (kubectl patch --type json with remove on both paths) unblocks a stalled release.

Acceptance criteria

  • An upgrade from the 1.9 line to a release without the rustfs ServiceAccount (or with another name) rolls the RustFS pod without a manual step.
  • A chart test asserts the field the fix relies on.

Found during the 1.8.x → 1.9.0-rc.1 step proof of #343 (returning the test cluster to 1.8.4 afterwards).

Activity

  1. teemow commented on Oct 13, 2026

    @teemow
    MemberAuthor

    The same stall appears on the way back from Substrate 1.10 to 1.8.x: 1.10 adds the ServiceAccount ate-system/rustfs and sets it on the rustfs Deployment. The 1.8.5 upgrade removes the ServiceAccount but the Deployment keeps serviceAccountName: rustfs, so the pod is refused (serviceaccount "rustfs" not found), the 1.8.5 release hangs in pending-upgrade, and the next golden boot fails its snapshot upload (S3 PutObject). Recreating the ServiceAccount by hand unblocks it. A fix here (the chart keeping or removing the reference consistently across versions) should cover 1.10 down to 1.8.x as well.

    Written by an agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions