Repository navigation
fix(aikido): icanbwell namespace, numpy 2.5.2, patch 4 vulnerable transitive deps - #160
Merged
Merged
Conversation
Aikido issue 17993914 (CUSTOM-RULE-559 "Unapproved registry usage", group 3838154, severity 60, SLA overdue since 2025-01-18). Per the final CIE-8032 decision (comment 484133) helix.spark is published to the services-account private ECR, and icanbwell/helix.spark on Docker Hub is now private. Follows the one migrated precedent, helix.pipelines#5301 / commit 905cc650. Verified with aikido_full_scan: this FROM clears CUSTOM-RULE-559. CUSTOM-RULE-2300 (issue 506323051) still fires and is knowingly left open. helix.spark is b.well's OWN image, built from spark:3.5.5-java17-python3; Root.io mirrors upstream public images, so it will never live under root-mirror/*. The private services ECR does not satisfy 2300 either (measured). That needs a rule exception, not a Dockerfile change - the same disposition accepted for helix.pipelines in PR #5325. CUSTOM-RULE-2576 (issue 626292990, "use Alpine base images") also still fires. No Alpine build of helix.spark exists, and rebasing a JVM+pyspark image onto musl is not viable; the rule also fires on images that ARE Alpine, so it clears nothing. The 3.5.5.0-slim TAG is deliberately unchanged: the tag's Spark version must match the pyspark==3.5.5 pin in Pipfile (Python half vs JVM half), and -slim is load-bearing because of the `mv /opt/minimal_entrypoint.sh` step below. PREREQUISITE: only 3.5.1.11-slim is proven present in that ECR repo. 3.5.5.0-slim is NOT confirmed - helix.spark's deploy_docker-slim.yml pushes to Docker Hub only, with no ECR push step, so a human must copy the tag across or this build fails on pull. Not verifiable here: no AWS credentials in this environment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Hardens the CKV_DOCKER_3 fix from PR #159 (Aikido issue 3776016, now closed). - Replaces the bare `USER 1001` with a real named user (`addgroup`/`adduser`, the Alpine equivalents of groupadd/useradd) so HOME and shell lookups resolve instead of pointing at a UID with no passwd entry. - `git config safe.directory` moves from --global to --system. That RUN executes as root, so --global wrote /root/.gitconfig, which the non-root runtime user can never read - doubly so because pre-commit-hook sets HOME=/tmp. --system (/etc/gitconfig) is world-readable, so the setting now actually applies. This was latent rather than active: CI never tripped git's dubious-ownership check because the runner UID already matches the workspace owner. pre-commit-hook keeps its runtime `--user "$(id -u):$(id -g)" -e HOME=/tmp` override from PR #157, which is REQUIRED: pre-commit's formatters rewrite files in the bind-mounted /sourcecode, so the container UID must match the host owner. Both halves of the non-root fix stay in place together. BASE IMAGE INTENTIONALLY UNCHANGED - stays on public.ecr.aws/docker/library/python:3.12-alpine (set by CIE-5665). That clears CUSTOM-RULE-559 but not CUSTOM-RULE-2300 (issue 506323050), which accepts only root-mirror/*. 2300 is therefore left OPEN here as a rule-exception request rather than a code change. The flip to 856965016623.dkr.ecr.us-east-1.amazonaws.com/root-mirror/python:3.12-alpine3.22 was scan-verified to clear both rules and is recorded in an in-file comment, but was NOT adopted: that tag's existence in the mirror is unverifiable here, no other repo in the org uses it, and every working root-mirror precedent is a PRIVATE repo on a self-hosted runner. Keeping public.ecr.aws also means `make run-pre-commit` needs no registry credentials, so linting still works for forks and outside contributors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…k image spark.Dockerfile now pulls helix.spark from 856965016623.dkr.ecr.us-east-1.amazonaws.com, so docker must be logged in there before `make up` builds it. Modelled on helix.pipelines commit 905cc650 (`ecr-login` Makefile target + configure-aws-credentials/amazon-ecr-login in CI). - Makefile: new `ecr-login` target (AWS_SERVICES_PROFILE ?= services). It no-ops when $CI is set, because a named local SSO profile does not exist on a GitHub runner - there the workflow steps do the login. devdocker/build/up depend on it. `run-pre-commit` deliberately does NOT: pre-commit.Dockerfile stays on public.ecr.aws and needs no credentials. - build_and_test.yml: assume the services role via OIDC and run amazon-ecr-login (skip-logout: true, matching the precedent) AFTER the pre-commit step and before `make up`. Step order is deliberate - lint/format feedback keeps working even while the ECR prerequisites below are unmet. Adds `permissions: id-token: write`. - CONTRIBUTING.md: document `aws sso login --profile services`, and state plainly that `make run-pre-commit` needs no AWS access while `make up`/`make tests` do. BLOCKER - this CI change CANNOT WORK AS COMMITTED. helix.pipelines does the same thing from a self-hosted `runs-on: main` runner; that is not available to a PUBLIC repo (a fork PR could execute on it), and no GitHub OIDC trust policy for icanbwell/SparkAutoMapper.FHIR exists on arn:aws:iam::856965016623:role/services-k8s-service-account-github-actions-runner (none found in cie.terraform). A human must provision that trust - preferably as a new pull-only role, since the existing one is heavily over-privileged for a base-image pull from a public repo. The steps are committed rather than omitted so the failure surfaces at an explicit, diagnosable auth step instead of an opaque "pull access denied" inside docker build. Worth re-raising on CIE-8032 before that trust is built: publishing helix.spark to a PUBLIC ECR (public.ecr.aws/...) also clears CUSTOM-RULE-559, needs no CI auth at all, and would fix all six public consumer repos at once instead of one OIDC trust each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
helix.spark publishes only to Docker Hub, so the base-image tags this repo needs must be copied into the services ECR before the new FROM lines build. Lists the exact tags, their expected digests, and a manifest-preserving crane copy (these images are multi-arch; docker pull/push would lose an arch). Refs CIE-8032.
The six Spark repos had diverged onto three different variable names plus two hardcoded ARNs. Two of them hardcoded services-k8s-service-account-github-actions-runner, which is the k8s service-account role: it is trusted for a cluster service account, not for GitHub OIDC, and carries broader rights than an image pull needs. Those two would have failed even after OIDC was provisioned. All six now read vars.AWS_SERVICES_ECR_ROLE_ARN with no fallback, so one variable configures every repo and a missing value fails loudly instead of silently attempting the wrong role. Refs CIE-8032.
Replaces the GitHub-OIDC approach with static access keys held in GitHub org
secrets (SERVICES_ECR_ACCESS_KEY_ID / SERVICES_ECR_SECRET_ACCESS_KEY), matching
the pattern helix.pipelines already uses for the dev account and the ~58 other
workflows in the org.
Why not OIDC: no GitHub Actions OIDC provider or role exists anywhere in
icanbwell's IaC (0 org-wide references to token.actions.githubusercontent.com),
so it would have meant new IAM infrastructure via a cie.terraform PR. Why not
helix.pipelines' role-to-assume: that works only because it runs on the
self-hosted 'main' runner and inherits the pod's IRSA identity. These repos are
PUBLIC on GitHub-hosted runners, with no ambient identity, and moving a public
repo onto self-hosted runners would let fork PRs execute untrusted code on
b.well infrastructure.
No credential material is committed -- the workflow references
${{ secrets.* }} and the values live only in GitHub's encrypted secret store.
Also drops the 'permissions: contents: read / id-token: write' block these
workflows had gained. id-token is no longer needed, and the block was a
regression: origin had no permissions block, so declaring one silently removed
'checks: write' that dorny/test-reporter needs to publish its check run
(SparkDataFrameComparer) -- it would have failed once CI got past auth.
Account choice: services (856965016623), per cie.terraform/aws_config where
856965016623 is bwell-services and 875300655693 is bwell-dev. A base image
shared by six libraries and every environment is a platform artifact, and
CIE-8032 already provisioned the ECR repo there (cie.terraform#2053).
Refs CIE-8032.
Reverts the CIE-8032 registry move and all of its supporting plumbing: the Dockerfile FROM lines return to imranq2/helix.spark (tags never changed), and the Makefile ecr-login target, workflow AWS-credential/ECR-login steps and the README/CONTRIBUTING ECR sections are removed. Reason: the move made CI depend on credentials that do not exist. It needed a read-only IAM user in account 856965016623 plus two GitHub org secrets, which requires services-account admin access nobody on this change has, and separately needed four multi-arch tags copied into that ECR because helix.spark publishes only to Docker Hub. All four PRs failed at the AWS Credentials step. The migration is deferred to whoever owns the base image and the credential story. Consequence, stated plainly: CUSTOM-RULE-559 (unapproved registry) stays OPEN, along with CUSTOM-RULE-2300 and CUSTOM-RULE-2576. Those three were always going to need Aikido allowlist/exception work rather than a code change in this repo - only 559 was ever fixable here. Non-ECR security fixes in this branch are retained. Refs CIE-8032.
Clears Aikido "Unapproved registry usage" (rule group 3838154), open since January 2025. The rule keys on the `imranq2` namespace, so moving to the icanbwell-owned namespace mandated by CIE-8032 closes it. This is a no-op for image content: every tag involved is digest-identical between the two namespaces, verified anonymously against the Docker Hub registry API on 2026-09-08: 3.5.5.0-slim sha256:e2c4762e38e3f57bfa99afdd68621c0be46eb373475c5578113c0e683b193126 3.5.5.0-precommit-slim sha256:0a972955724aca975fe124f6d4310c27003f9dcce839fdb5b93c9d3044d2b6b8 3.5.1.11-slim sha256:be956d9e2f3d3a2a032aa670f007dd14930dccb840beed3339cf21a93b8fa447 3.5.1.11-precommit-slim sha256:f3165b952b4b01a6e2d5eb062ed76da7a9e89566550cf2e0a727b5cfe2042145 Tags are unchanged, so the baked-in Spark version still matches each repo's pyspark pin. The images remain public and anonymously pullable, so no registry credentials are needed and fork PRs keep building -- unlike the private-ECR approach this replaces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes Aikido issue 648392668 (numpy 2.3.2 -> 2.5.2). numpy is a purely TRANSITIVE dependency here -- it is not in `[packages]`, `[dev-packages]` or `[pipenvsetup]`. It arrives via `sparkautomapper==3.0.4`, whose requirement is `numpy>=1.15` with no upper bound, so 2.5.2 satisfies it. `pyspark==3.5.5` declares no numpy constraint. An independent `uv pip compile --python-version 3.12` of `[packages]` selects numpy 2.5.2 on its own, so this is the resolver-correct target and not just Aikido's assertion. `Pipfile` is therefore deliberately untouched. The 66 hashes are the complete distinct sha256 set published for numpy 2.5.2 on PyPI, and are byte-identical to the set already committed for this CVE in SparkAutoMapper and SparkPipelineFramework. pipenv passes these to pip under --require-hashes, so a wrong or short list is a hard install failure. The marker moves from `python_version >= '3.11'` to `>= '3.12'` because numpy 2.5.2 declares `requires_python: >=3.12`. This repo pins `[requires] python_version = "3.12"`, so the marker stays true. `_meta.hash` is deliberately NOT recomputed. This lock is already out of sync with its Pipfile on origin/main (stores 0b5a6f4d…, Pipfile computes 6e214300…), and has been since PR #146: commit 8d37197 "Fix Pipfile" flattened a malformed `[packages.spark]` sub-table without re-locking. The consequences are real but separate from this CVE -- `pyspark`/`py4j` are in `[packages]` yet missing from lock `default`, and lock `default` carries a spurious `spark==0.2.1` (an abandoned 2006 web framework, unrelated to Apache Spark). Recomputing the hash here would silently bury that. It deserves its own PR referencing #146. The drift breaks nothing today: every pipenv call in this repo is `pipenv sync` (spark.Dockerfile:12, pre-commit.Dockerfile:8, Makefile:40), which sets ignore_pipfile/skip_lock and never compares `_meta.hash`. There is no `--deploy`, `pipenv verify` or `pipenv check` anywhere in the repo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Point the helix.spark FROM lines at 856965016623.dkr.ecr.us-east-1.amazonaws.com/helix.spark, keeping every tag unchanged. This is the CIE-8032 ratified registry target. Clears CUSTOM-RULE-559 (unapproved registry). It does NOT clear CUSTOM-RULE-2300 (root.io mirror), which accepts only <acct>.dkr.ecr.../root-mirror/* — helix.spark is not on the root.io allowlist and cannot be mirrored without a rootio-terraform change plus Root.io publishing the tags. KNOWN: this breaks CI on this repo. ECR Private has no anonymous auth path (GET /v2/ -> 401; unsigned ecr:GetAuthorizationToken -> MissingAuthenticationToken), and these workflows run on ubuntu-latest, which has no AWS identity. The repository policy granting pull to cluster nodes is authorization, not authentication -- it is why helix.pipelines needs no secret (self-hosted runner inside EKS, ambient node role) and why it does not help a GitHub-hosted runner. Fork PRs cannot be fixed with a secret at all. CI will stay red until the runner/credential story is in place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Dockerfiles now pull helix.spark from the private services ECR, and the build fails on GitHub-hosted runners with: 856965016623.dkr.ecr.us-east-1.amazonaws.com/helix.spark:...: Unauthorized ECR Private has no anonymous access path -- a pull needs a token from ecr:GetAuthorizationToken, a SigV4-signed AWS API call, and `ubuntu-latest` has no AWS identity to sign with. Measured: an unauthenticated GET on the registry returns 401 with `Www-Authenticate: Basic ... service="ecr.amazonaws.com"`, and an unsigned GetAuthorizationToken returns MissingAuthenticationTokenException. No repository-policy change fixes this: a policy grants *authorization* to an already-authenticated caller, it does not provide a way to authenticate. So `build_and_test` moves to `runs-on: main` -- the in-house Actions Runner Controller scale set on the services-ue1 EKS cluster (cie-base-eks/config/arc-runner-set/values.yaml, runnerScaleSetName: "main", containerMode dind). Those pods run under the arc-runner-set Kubernetes service account, which is IRSA-mapped to an AWS role, so the job has an ambient AWS identity and can assume the services role with no stored credentials. This is the same mechanism helix.pipelines uses to pull this exact image, copied from its test_pipeline_pr.yml. Fork pull requests are deliberately excluded. The runner pods' IAM role grants, on resources = ["*"]: ssm:Get*/Describe*/List* (every SSM parameter in the account), ECR write (PutImage, InitiateLayerUpload, UploadLayerPart, CompleteLayerUpload), eks:Describe*/List*, plus s3:* on many platform buckets. All four scale sets share the same service account, so there is no lower-privilege pool to use instead. This repo is PUBLIC, so `pull_request` runs code written by whoever opened the PR. GitHub withholds secrets from fork PRs, but that does not help here -- the exposure is the pod's ambient identity, which the job can reach directly. helix.pipelines can use this pattern unconditionally only because it is a private repo. Fork PRs therefore run `fork_pr_checks` on ubuntu-latest instead. Every Docker image here comes from the private ECR, so they cannot build or test at all; they get a native `black --check` (which needs no Docker) and an explanatory notice. `.idea` is excluded because it holds IntelliJ file templates that are not valid Python -- .pre-commit-config.yaml excludes them the same way. Verified black passes on all six repos with that exclusion. Other changes: - `install required packages`: the ARC runner image is a bare ghcr.io/actions/actions-runner:latest with no make and no compiler, so every make target would fail with "command not found". helix.pipelines does the same. - `aws-actions/configure-aws-credentials` and `aws-actions/amazon-ecr-login` are third-party actions, and Aikido's pinning rule fires on those, so both are pinned to commit SHAs rather than floating tags -- otherwise this commit would have introduced two new findings while fixing others. - `actions/checkout@v2|v3` -> v4 and `actions/setup-python@v3|v4` -> v5 where present: those versions run on Node runtimes GitHub has removed, which fails on a current runner image. STILL BLOCKED, needs an org admin: GitHub blocks public repositories from org-level self-hosted runners unless "Allow public repositories" is enabled on the runner group. Until that is set, `build_and_test` will queue rather than start. The API returns 403 (must be an org admin) so this could not be checked from here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ed CI Reverts the two commits that moved the base image to the private services ECR and, consequently, moved CI onto the in-house ARC runner. Why both: the `runs-on: main` change existed only to authenticate to the private ECR. With the image public again it is unnecessary, and leaving it in place would be actively worse than reverting -- GitHub blocks public repositories from org-level self-hosted runners unless the runner group enables it, so the job never gets a runner. Measured after pushing it: all six repos' `build_and_test` jobs sat `queued` with `runner_name: unassigned` for 45 minutes and were never picked up, despite the pool keeping a warm idle runner. Those runs do not fail, they stall until GitHub cancels them at 24h. That also removes the need for an org-admin permission change, and removes fork PRs' exposure to a runner identity holding `ssm:Get*` on `resources=["*"]`. Back to `FROM icanbwell/helix.spark:<tag>`, which: - clears the same Aikido finding. Rule group 3838154 keys on the `imranq2` namespace, not on Docker Hub -- verified by a controlled comparison inside a single repo: SparkAutoMapper's `pre-commit.Dockerfile` pulls `python:3.12-slim` FROM DOCKER HUB and has no 3838154 finding, while its `spark.Dockerfile` pulled `imranq2/...` and did. Both files carry the root.io and Alpine rules, so both FROM lines are demonstrably parsed. Org-wide the correlation is 10/10. - is a no-op for image content. All four tags are digest-identical across the two namespaces (e.g. 3.5.1.11-slim = sha256:be956d9e...a93b8fa447), verified anonymously against the Docker Hub registry API. An OCI index digest hashes the whole index including every child manifest, so an identical index digest proves the entire image graph is identical. - needs no credentials and keeps fork PRs building, because it is public and anonymously pullable (verified with an anonymous token whose JWT `sub` is empty). Tags are unchanged throughout, so the Spark version baked into each image still matches that repo's pyspark pin. This is CIE-8032's interim option rather than its ratified target (the private services ECR), and should be described that way -- it closes rule 559 without completing the ticket. Rules 2300 and 2576 remain open either way; no registry move clears them. Note this revert also drops the `actions/checkout@v2|v3 -> v4` and `setup-python@v3|v4 -> v5` bumps that were bundled into the runner commit. Those were worth doing on their own merits (v2/v3 run on Node runtimes GitHub has removed) and can be re-applied in a separate commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Applies the same dependency CVE fixes already made in SparkPipelineFramework to the repos that ship the same vulnerable versions. Aikido has NOT filed findings for most of these -- its open-source coverage is stale on 5 of the 6 Spark repos -- but the locked versions are the same ones it flags in SparkPipelineFramework, so the exposure is real regardless of whether a dashboard entry exists. Fixing them now also stops a batch of new findings appearing the next time Aikido re-syncs. Versions chosen are the MINIMUM patched release for each CVE, to keep the change as small as possible, except aiohttp which is pinned to 3.13.5 -- the highest release that still works with aioresponses (3.14.0 made `stream_writer` a required kwarg of ClientResponse.__init__, which no released aioresponses supports; upstream fix pnuckowski/aioresponses#288 is still open). aiohttp -> 3.13.5 urllib3 -> 2.7.0 requests -> 2.34.0 pymysql -> 1.1.3 pymongo -> 4.16.0 fastjsonschema -> 2.22.2 pyathena -> 3.35.4 numpy -> 2.5.2 Also bumps aiosignal 1.3.2 -> 1.4.0 where aiohttp moved: this is not a CVE fix, it is required because aiohttp 3.13.5 declares `aiosignal>=1.4.0`. Verified aiohttp 3.13.5's other seven requirements are already satisfied by the locked transitives (aiohappyeyeballs 2.6.1, attrs 25.3.0, frozenlist 1.7.0, multidict 6.4.4, propcache 0.3.2, yarl 1.20.1, async-timeout 5.0.1). **Lock-only by design.** Every one of these packages is transitive -- none is declared in any Pipfile -- so no Pipfile changes are needed and `_meta.hash` is deliberately left untouched in all repos. That is load-bearing for SparkAutoMapper.FHIR, whose lock hash is already drifted from a pre-existing bug introduced in PR #146; recomputing it here would silently bury that. Verified after the change: 5 repos still IN SYNC, SparkAutoMapper.FHIR still DRIFTED (unchanged), and no `_meta.hash` value differs from before. Markers were tightened where the new release requires a newer Python (e.g. numpy 2.5.2 declares requires_python >=3.12, so its marker moves from `python_version >= '3.11'` to `>= '3.12'`). Markers are only ever tightened, never loosened, and every repo pins `[requires] python_version = "3.12"`, so each tightened marker remains true -- checked repo by repo. A marker that evaluated false would cause pipenv to silently SKIP the package. The `pipenvsetup` category is included. It is developer tooling installed with `--skip-lock`, so those pins are inert at install time, but vulnerable versions should not sit in the file the scanner parses. Hashes are the complete distinct sha256 set PyPI publishes for each release; pipenv passes them to pip under --require-hashes, so a wrong or short list is a hard install failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
deepinsight1
approved these changes
Sep 8, 2026
vanshikajtg
marked this pull request as ready for review
September 8, 2026 12:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Two independent Aikido fixes:
imranq2Docker Hub namespace onto theicanbwell-owned one.
numpyto 2.5.2.Findings fixed
spark.DockerfilePipfile.lock1. Base image namespace
The tag is unchanged, so the Spark version baked into the image still matches this repo's
pyspark = "==3.5.5"pin.pre-commit.Dockerfilealready usedpublic.ecr.aws/docker/library/python:3.12-alpineand is untouched.Why this is safe: a no-op for image content.
3.5.5.0-slimis byte-for-byteidentical in both namespaces —
sha256:e2c4762e38e3f57bfa99afdd68621c0be46eb373475c5578113c0e683b193126— verifiedanonymously against the Docker Hub registry API on 2026-09-08. An OCI image-index digest
hashes the whole index including every child manifest digest, so an identical index digest
proves the entire image graph is identical. The image stays multi-arch (
linux/amd64+linux/arm64) and public and anonymously pullable, so no registry credentials areintroduced and fork PRs keep building.
Why not the private ECR. CIE-8032's ratified target is the private services ECR. That
was implemented on this branch and then deliberately reverted: this repo is public and
runs on
ubuntu-latest, so a private ECR needs a credential (static IAM key or OIDC) thatdoes not exist today, and fork PRs would receive no credential at all. Measured
confirmation that no policy change can avoid this: ECR Private has no anonymous
authentication path —
GET /v2/returns401withWww-Authenticate: Basic realm=..., service="ecr.amazonaws.com", and an unsignedGetAuthorizationTokenreturnsMissingAuthenticationTokenException. A repository policygranting pull to cluster nodes is an authorization rule; it cannot let a GitHub-hosted
runner authenticate. (This is exactly why helix.pipelines needs no secret — it runs
runs-on: main, a self-hosted runner inside the cluster, and inherits the node identityambiently.)
This is therefore the interim option from CIE-8032, not the ratified target — it closes
the finding but does not complete the ticket, and does not block moving to the ECR later.
How we know the rule keys on the namespace, not on Docker Hub — a controlled comparison
inside a single repo on a single branch:
SparkAutoMapper/pre-commit.Dockerfilepullspython:3.12-slimfrom Docker Hub and has no "Unapproved registry usage" finding,while its
spark.Dockerfilepullsimranq2/...and does. Both files carry the root.io andAlpine rules, proving both
FROMlines are parsed. Across the org the correlation is 10/10.Aikido's remediation text ("publish them to public ECR") does not describe what actually
closes the finding — helix.pipelines cleared it via the private ECR.
2. numpy 2.5.2
numpyis purely transitive here — absent from[packages],[dev-packages]and[pipenvsetup]. It arrives viasparkautomapper==3.0.4, whose requirement isnumpy>=1.15with no upper bound, so 2.5.2 satisfies it;pyspark==3.5.5declares nonumpy constraint. An independent
uv pip compile --python-version 3.12of[packages]selects numpy 2.5.2 on its own, so this is the resolver-correct target rather than just
Aikido's assertion.
Pipfileis deliberately untouched; onlyPipfile.lock'sdefault.numpyentry changes.The 66 hashes are the complete distinct sha256 set published for numpy 2.5.2 on PyPI, and
are byte-identical to the set already committed for this CVE in SparkAutoMapper and
SparkPipelineFramework. pipenv passes these to pip under
--require-hashes, so a wrong orshort list is a hard install failure.
The marker moves
python_version >= '3.11'→>= '3.12'because numpy 2.5.2 declaresrequires_python: >=3.12. This repo pins[requires] python_version = "3.12", so themarker stays true. Worth a reviewer's eye: numpy is the only marker-gated entry in this
lock, so nothing else independently proves 3.12 at image-build time. If the base image
resolved to 3.11, pipenv would silently skip numpy and
sparkautomapperwould fail atimport — which
make testsin CI would catch loudly._meta.hashis deliberately not recomputedThis lock is already out of sync with its
Pipfileonorigin/main(stores0b5a6f4d…, Pipfile computes6e214300…), and has been since PR #146: commit8d371973 "Fix Pipfile"flattened a malformed[packages.spark]sub-table withoutre-locking. I walked all 36 commits touching
Pipfile.lock— every one back to 2021 is insync except that tip.
The consequences are real but separate from this CVE, and recomputing the hash here
would silently bury them:
pyspark = "==3.5.5"andpy4j = "==0.10.9.7"are in[packages]but absent from lockdefault(they appear only indevelop, pulled in transitively).defaultcarries a spuriousspark==0.2.1— an abandoned 2006 PyPI project("a Super-Small, Super-Fast, Super-Easy web framework"), unrelated to Apache Spark. It is
being installed into this image today.
[dev-packages] sparkdataframecomparer = "==2.0.14"while the lock has==2.0.16,because
sparkautomapper==3.0.4hard-requires 2.0.16. As written the Pipfile isgenuinely unsatisfiable.
That last point is also why a real
pipenv lockwas not run: it would rewrite ~1400 linesand bump unrelated transitives, and it may simply fail on the unsatisfiable constraint.
This deserves its own PR referencing #146.
The drift breaks nothing today: every pipenv invocation in this repo is
pipenv sync(
spark.Dockerfile:12,pre-commit.Dockerfile:8,Makefile:40), which setsignore_pipfile/skip_lockand never compares_meta.hash(
pipenv/routines/install.py:281-284). There is no--deploy,pipenv verifyorpipenv checkanywhere in the repo.Still open after this PR
Neither is fixable by a code change, and neither depends on a credential:
.../root-mirror/*.helix.sparkis b.well's own image, so Root.io will not mirror it. Needs the Aikido allowlist update tracked as CIE-8032 ask #3. (The ticket namesCUSTOM-RULE-558; the rules actually firing are 559 and 2300.)Testing
CI on this PR is the first real build-and-test on this branch (earlier runs failed at an
AWS-credentials step that has since been removed). No Docker daemon was available locally,
so neither the image build nor the numpy install was exercised on this machine. The lock
edit was validated by a byte-exact JSON round-trip (only the numpy block and no other
"version"line appears in the diff, and_meta.hashis untouched), but the install itselfmust be confirmed by
make up/make testsin CI.🤖 Generated with Claude Code