Repository navigation
Security fixes from the October 2026 audit - #21
Open
abd-ulbasit wants to merge 42 commits into
Open
abd-ulbasit wants to merge 42 commits into
abd-ulbasit wants to merge 42 commits into
Conversation
Eleven charts rendered an in-chart Secret by default from committed dev
placeholders (a DSN with password 'dev-****' and sslmode=disable). An install
that forgot secrets.existingSecret silently shipped that known credential.
- secrets.create now defaults to false. The Deployment envFroms a Secret named
secrets.existingSecret, falling back to the chart fullname, which is the name
the ExternalSecret for fp-auth already targets. A missing Secret now stops the
pod from starting instead of running with placeholders.
- The secretName helper used `required` on existingSecret, which contradicted
the values comment ("defaults to the chart's fullname when empty"). The
helper now does what the comment says.
- The migrate initContainer gate no longer requires existingSecret to be set
explicitly when create=false.
- make helm-install / helm-upgrade pass --set secrets.create=true, so the
local Kind flow is unchanged.
Side effect worth knowing: with the old default, 11 of 12 charts failed to
render without a jwtSecret, so Trivy's Helm scanner could not render them and
skipped them. They now render with default values.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nine charts pulled migrate/migrate:v4.18.3 from Docker Hub by tag, under a comment calling it "an immutable tag". Docker Hub tags are mutable: whoever controls the repo can re-push v4.18.3, and the next pod start would run different bytes with database credentials in its environment. The tag value is now v4.18.3@sha256:39b5... (the current index digest for that tag), so the kubelet pulls by digest and the tag stays readable for humans. The comments now say why the digest is there. No version change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes six of the HIGH Trivy findings that keep the Security gate red: KSV-0014 on postgres, grafana and ollama, and KSV-0118 (x2) on ollama. - postgres: readOnlyRootFilesystem: true plus an emptyDir at /tmp. As uid 999 the entrypoint skips its root-only chown step, and PGDATA (PVC) and the socket dir (emptyDir) were already writable. The in-file comment claiming the image could not run read-only was wrong, so the Kyverno require-readonly-rootfs exception for postgres is removed with it. - grafana: readOnlyRootFilesystem: true, an emptyDir at /tmp, and GF_LOG_MODE=console so Grafana stops trying to write /var/log/grafana/grafana.log. - ollama: the pod had a completely default security context. It now has seccomp RuntimeDefault, all capabilities dropped, no privilege escalation, a read-only root (writes go to the model PVC and a /tmp emptyDir) and no SA token. It still runs as uid 0, because the existing PVC (including the 0600 id_ed25519 key) is root-owned and local-path ignores fsGroup. The migration steps are in deploy/llm/README.md. None of this was tested against a cluster. postgres is synced by the fp-infra Argo CD app, so check `kubectl -n fp-infra logs postgres-0` after merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The infra manifests ran floating tags (postgres:17, redis:7, nats:2.11, postgres:17-alpine) that a Kyverno comment described as "pinned". Every new node or pod restart could pull different bytes. Each image now names the exact version it resolves to today plus that tag's index digest: postgres:17 -> 17.11@sha256:b8fa... redis:7 -> 7.4.11@sha256:4496... nats:2.11 -> 2.11.17@sha256:c991... postgres:17-alpine -> 17.11-alpine@sha256:eee6... (backup CronJob) ollama/ollama:0.30.10, grafana 11.5.1, tempo 2.7.1, loki 3.4.2, prometheus v3.1.0, otel-collector-contrib 0.119.0 -> same tags + digest None of these is a version change: each digest is what the old tag resolves to on Docker Hub today. Digests were read from the registry API, not from a pull. Not changed: minio/minio and minio/mc. Both Docker Hub repositories now return 404, so their RELEASE tags can no longer be pulled or resolved to a digest. The MinIO deployment needs a replacement image or a different object store, which is a separate decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.terraform.lock.hcl was in the root .gitignore, while versions.tf and environments/dev/versions.tf both said "the committed .terraform.lock.hcl freezes the exact version + checksums". It was never committed, so every `terraform init` could resolve a different provider build inside the ~> constraints, with no hash to check it against. - Drop .terraform.lock.hcl from the root .gitignore and rewrite the note in deploy/terraform/.gitignore to say it is committed and how to regenerate it. - Add environments/dev/.terraform.lock.hcl, generated by `terraform providers lock -platform=linux_amd64 -platform=darwin_arm64`: aws 5.100.0, kubernetes 2.38.0, random 3.9.1, tls 4.4.2. The two versions.tf comments are now true, so they are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eight services logged their NATS and Redis URLs verbatim at startup
("redis connected" url=..., "nats connected" url=...), and pkg/natsutil logged
nc.ConnectedUrl() on every reconnect. In AWS, FP_REDIS_URL comes from Secrets
Manager through External Secrets and embeds the ElastiCache auth token
(rediss://:<token>@host), so each pod start copied that token into the log
pipeline.
- pkg/config.RedactURL replaces the userinfo password with "xxxxx"
(net/url's URL.Redacted), passes a bare host:port through unchanged, and
returns a placeholder rather than echoing a value it cannot parse.
- Startup log lines in ai-gateway, billing, experiment-tracker,
inference-gateway, model-serving, notification, pipeline-orchestrator and
registry use it. ConnectedUrl() becomes nats.go's ConnectedUrlRedacted().
Tests: pkg/config unit tests pass. The pkg/natsutil tests need Docker
(testcontainers) and were not run here. All eight services build and vet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docker-compose published Postgres, Redis, NATS, MinIO, Prometheus, Tempo and
Loki as "port:port", which Docker binds on 0.0.0.0. With a remote engine (this
repo is developed against a docker context over ssh), 0.0.0.0 is the engine
host's LAN and VPN interfaces, and Redis and NATS had no auth at all.
- Every published port binds to ${FP_BIND_ADDR:-127.0.0.1}. Monitoring and UI
ports (8222, 9001, 9090, 3000, 3200, 3100) are always loopback-only. To reach
the stack from another machine, use an ssh tunnel.
- Redis runs with --requirepass and NATS with --user/--pass, from REDIS_PASSWORD
and NATS_PASSWORD (dev default "fp", the same as POSTGRES_PASSWORD). The
defaults are published, so the control is the loopback bind. The .env
passwords exist so that real values take effect end to end.
- Go local defaults carry the same dev credentials (redis://:fp@localhost:6379,
nats://fp:fp@localhost:4222), so `go run` against compose works as before.
Every Helm ConfigMap sets these URLs explicitly, so no deployed default
changes.
- registry and billing passed FP_REDIS_URL straight to Options.Addr, so they
could not take a password or TLS at all, including rediss:// from
ElastiCache. They now use redis.ParseURL when the value has a scheme and
still accept a bare host:port (the Helm default). Unit tests cover both.
- :latest images pinned (prometheus v3.1.0, grafana 11.5.1, tempo 2.7.1,
loki 3.4.2, mc RELEASE.2025-04-16T18-13-26Z, matching deploy/k8s), and
postgres/redis/nats pinned by digest. The MinIO images can no longer be
pulled from Docker Hub (404). See the compose comment.
Tests: new unit tests pass. Packages whose tests need Docker (testcontainers)
were not run. All touched modules build and vet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Trivy AWS-0164 (x3, HIGH) flagged map_public_ip_on_launch = true on the three public subnets, one of the findings keeping the Security gate red. Nothing in those subnets needs it. The NAT gateway's public address is the Elastic IP passed as allocation_id, and internet-facing ALB/NLB nodes get public addresses from AWS whatever this flag says. The only thing the flag did was give a public IP to any EC2 instance later launched into those subnets (a debug box, a mis-placed node group), which is the accident the public/private split exists to prevent. The comment that said the flag was needed is corrected. `terraform validate` passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README says to copy terraform.tfvars.example to terraform.tfvars. That file set eks_public_access = true and eks_public_access_cidrs = ["0.0.0.0/0"], which overrode the secure defaults in variables.tf and put the Kubernetes API server on the internet, guarded only by authentication. Trivy does not parse *.tfvars.example, so no gate saw it. The example now matches variables.tf (private endpoint, empty CIDR list), with a commented recipe for a throwaway cluster that enables the public endpoint for one /32 egress IP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The RDS module generated the master password with random_password, passed it to aws_db_instance and wrote it, plus a full DSN, into a Secrets Manager secret_string. All three persist in Terraform state, which sat in an SSE-S3 bucket. There was also a drift trap: aws_db_instance ignored password changes but the secret version did not, so an out-of-band rotation set up a later apply that would overwrite the secret with a password the DB no longer took. - aws_db_instance uses manage_master_user_password = true with master_user_secret_kms_key_id = the RDS CMK. RDS generates, stores and rotates the password in a secret it owns, and Terraform never sees it. random_password.master and the ignore_changes block are gone. - "<prefix>/rds/postgres" now holds connection info only (host, port, dbname, username, sslmode, master_user_secret_arn). New outputs: master_user_secret_arn (module) and rds_master_user_secret_arn (dev). - The external_secrets IRSA role can read the RDS-managed secret. Both ExternalSecrets now build FP_DATABASE_URL with an ESO template from the managed username/password (by ARN, placeholder <RDS_MASTER_USER_SECRET_ARN>) plus the connection info, URL-escaping the credentials. - ElastiCache cannot do the same: there is no managed password, and provider 5.100 has no write-only auth_token (checked against the provider schema). The token stays in state. The module and README now say so, and list the CMK on the state bucket and IAM-authenticated ElastiCache users as the mitigation and the real fix. - README: the "outputs are marked sensitive" claim was false. It now says why none are (ARNs are not secrets, and `sensitive` does not keep a value out of state). Trade-off: RDS rotates every 7 days by default, and envFrom env vars only change on pod restart, so the cluster needs a reloader or scheduled rollouts. Documented in deploy/secrets/README.md. `terraform validate` passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README says staging and prod are made by copying environments/dev/. That directory hard-coded skip_final_snapshot = true and ECR force_delete = true, both modules hard-coded Secrets Manager recovery_window_in_days = 0, and the module and env defaults had deletion protection off. A copied prod would inherit every one of them. - Module defaults are now the safe values: rds deletion_protection = true, skip_final_snapshot = false, ecr force_delete = false, and a new secret_recovery_window_days (default 30, validated 0 or 7-30) on the rds and elasticache modules. - environments/dev exposes them as variables with the same safe defaults (rds_deletion_protection, rds_skip_final_snapshot, ecr_force_delete, secret_recovery_window_days). - terraform.tfvars.example opts the throwaway dev env into the teardown values, with each prod value next to it. The tfvars header, the variables.tf header and the README say which values are defaults and which are opt-ins. Behaviour change, on purpose: running dev without the example tfvars now yields a protected DB, a final snapshot on delete, non-force-deletable ECR repos and a 30-day secret recovery window. `terraform validate` passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
environments/dev passed the RDS CMK to the s3 module, and the registry and model-serving IRSA roles got kms:Decrypt on that key so they could read SSE-KMS objects. The same key wraps the database secrets, so any artifact reader was one IAM mistake (a broad secretsmanager:GetSecretValue) away from a credential read. - The s3 module creates a dedicated, rotated CMK (alias <prefix>-artifacts) unless a key is passed in, always uses SSE-KMS (the SSE-S3 fallback is gone) and outputs kms_key_arn. - The iam module takes artifacts_kms_key_arn and grants it per role: kms:Decrypt for read, plus kms:GenerateDataKey for readwrite. The old grant only had Decrypt, so a registry PutObject to an SSE-KMS bucket would have been denied. - registry and model-serving no longer hold any grant on the RDS key. - Comments say plainly that no service code uses S3 yet (there is no AWS SDK in any go.mod), so these roles grant nothing until a ServiceAccount is annotated with them. `terraform validate` passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
modules/state-backend was never called by any root, used SSE-S3 (Trivy AWS-0132, one of the findings keeping the gate red), had no TLS-only policy and no prevent_destroy, and locked with a DynamoDB table, which the S3 backend has deprecated. This state also holds the Redis AUTH token. modules/state-backend: - A customer-managed KMS key (rotation on, 30-day deletion window, prevent_destroy). The bucket default is SSE-KMS with that key plus a bucket key. Reading state now needs kms:Decrypt as well as s3:GetObject. - prevent_destroy on the bucket, BucketOwnerEnforced (no ACLs), and a bucket policy that denies non-TLS requests. - Optional state_access_principal_arns: when set, object read/write/delete is denied to everyone else. Bucket-level actions are deliberately not denied, so an admin cannot lock themselves out of the policy. - The DynamoDB lock table is gone. Locking is use_lockfile (S3-native, GA in Terraform 1.11), so required_version is now >= 1.11 in every root. deploy/terraform/bootstrap (new root, local state, committed lock file): calls the module with forgepoint-tfstate-<account_id> and outputs a ready backend_config block (bucket, key, region, encrypt, kms_key_id, use_lockfile). Docs: backend.tf (root + dev) and the README bootstrapping section describe the real flow. The old "no CMK, to avoid a circular dependency" rationale is replaced: key and bucket are created together from local state, and the risk that matters is losing the key, which prevent_destroy covers. dev/backend.tf no longer claims CI runs plan. It refers to the terraform workflow added in the next commit, which runs fmt + validate only. Decision: CMK over SSE-S3. Alternative: keep SSE-S3 and rely on IAM. Why: the state holds a live credential, and a CMK puts a second, separately grantable and logged permission in front of it, for roughly $1/month. `terraform validate` passes for bootstrap/ and environments/dev/. Trivy finds no HIGH/CRITICAL in deploy/terraform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dev runs an EKS control plane, a NAT gateway, interface endpoints, two nodes, RDS and ElastiCache around the clock (roughly $200-250/month idle), and nothing in the stack would ever say so. - aws_budgets_budget "<prefix>-monthly": COST, monthly, default $250 (budget_monthly_limit_usd), alerting at 50/80/100% of actual spend and at 100% of forecast. - budget_alert_emails is required (no default, validated non-empty), so the environment cannot be applied without someone receiving the alerts. The example tfvars and README show it. Decision: account-wide budget. Alternative: filter on the Project tag. Why: a tag filter reads $0 until the tag is activated as a cost allocation tag in Billing, which is exactly the silent failure a budget is meant to prevent. The comment explains how to switch once the tag is active. `terraform validate` passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The ECR module's default service list had 11 entries and no ai-gateway, although services/ai-gateway has a Dockerfile and the Security workflow builds it. It is added, and the description now says the list must match services/*/Dockerfile. - cluster_version defaulted to 1.30 (the module, dev variables and the example tfvars). By the EKS release calendar, 1.30 ended standard support in July 2025 and extended support in July 2026, so EKS will no longer create it. Moved to 1.36, which is in standard support until 2 Aug 2027. I picked 1.36 over the newest (1.37, released 1 Oct 2026) to stay one minor behind the edge. Managed node groups on 1.30+ already default to AL2023 AMIs, so the launch template needs no change. `terraform validate` passes. Nothing has been applied, so no cluster is upgraded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ECR repos had no pusher and Terraform defined no CI identity at all. CI publishes to GHCR with GITHUB_TOKEN, so the gap was latent. Whatever pushes to ECR one day should not be a long-lived access key in repo secrets. modules/github-oidc (new): - the account's token.actions.githubusercontent.com provider (or reuse an existing one via create_oidc_provider = false); - role <prefix>-github-ecr-push. Its trust requires aud = sts.amazonaws.com and sub StringLike one of allowed_subjects. A validation rule rejects any pull_request subject and any bare ":*" wildcard; - ecr:GetAuthorizationToken (account-level by design) plus push/pull actions on the listed repository ARNs only. environments/dev wires it behind enable_github_ecr_push (default false), allowing refs/heads/main and refs/tags/v* only, and outputs github_ecr_push_role_arn for the AWS_ROLE_ARN repository variable. The comment notes that this repo predates GitHub's 15 Jul 2026 immutable sub format, which applies if it is ever renamed or transferred. Decision: opt-in. Alternative: on by default. Why: no workflow pushes to ECR yet, and an unused push role is attack surface with no benefit. `terraform validate` passes. A scratch root confirmed the validation rejects "repo:...:pull_request" and "repo:...:*". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
security.yml passes --build-arg VERSION=${{ github.ref_name }}, but no
Dockerfile declared ARG VERSION. BuildKit ignored the value, and every service
also hard-coded ServiceVersion: "dev", so no image could say which version it
was in its traces or metrics.
- All 12 Dockerfiles declare ARG VERSION=dev (after the dependency layers, so
only the compile step reruns) and link it with -X main.version=${VERSION}.
- Each services/*/cmd/server/main.go has `var version = "dev"` and passes it as
observability.Config.ServiceVersion (OTel service.version). Local `go run`
still reports "dev".
Checked: a local build with -X main.version=v9.9.9-test embeds the string, and
all 12 services build and vet. No image was built here (remote engine).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Trivy filesystem gate had been red on main since 2026-07-27 on HIGH IaC
findings, with no decision recorded. The earlier commits in this branch fix
eight of them (postgres, grafana and ollama KSV-0014/KSV-0118, AWS-0164 x3,
AWS-0132). This commit records the rest and closes two blind spots.
- .trivyignore.yaml (new, read via `trivyignores:`). Two path-scoped entries,
each with a statement and an expiry:
* KSV-0014 on the backup CronJob until 2027-01-31. The job installs tools
at runtime and is broken in other ways too; it needs a rewrite.
* KSV-0109 on the fp-ai-gateway ConfigMap until 2027-04-30. False positive:
FP_TEAM_TOKEN_BUDGET is an integer, not a secret.
- TRIVY_HELM_KUBE_VERSION=1.36.0. Trivy renders charts as Kubernetes v1.20.0
by default, below every chart's kubeVersion, so it skipped all 12 charts
("Failed to render Chart files") while the job reported success.
- tf-vars: terraform.tfvars.example. Trivy does not read *.example files. With
this set, the old example's 0.0.0.0/0 EKS endpoint would have raised
AWS-0040/AWS-0041 (checked locally against the pre-fix file).
- SECURITY.md: the "Open findings" section is now a triage table with a
decision per finding, an "Accepted findings" policy, and the backup job's
defects.
Checked locally with Trivy v0.70.0 (the version trivy-action v0.36.0 installs):
no HIGH/CRITICAL misconfigurations or secrets remain. The gate is still red on
HIGH dependency CVEs in the Go modules and web/. Those are dependency bumps,
which the open Dependabot PRs carry, so this branch does not duplicate them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit said the remaining HIGH dependency CVEs are carried by the open Dependabot PRs. I then checked #17 and #10 against Trivy's fixed versions, and that is only partly true: x/net (needs 0.60.0, #17 gives 0.58.0), klauspost/compress (1.18.7 vs 1.18.6), moby/go-archive (0.3.0, untouched), axios (1.20.0 vs 1.19.0) and follow-redirects (1.16.1, untouched) stay vulnerable, and #17 does not cover cli/, gen/go or services/bff. SECURITY.md now has a table of fixed-in versions against what each PR brings, so nobody merges the Dependabot PRs expecting a green gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e job
On push events, the image job pushed to GHCR inside build-push-action and only
then ran the Trivy image scan, against the first metadata tag (the mutable
:main) rather than the digest it had just built. An image with a HIGH CVE was
already pullable when the gate went red. The job also held packages:write and
id-token:write statically, including on same-repo pull_request runs, and
published under ghcr.io/<owner>/forgepoint-<svc>, a name nothing that deploys
images referenced.
image (every run, contents: read only):
- builds into a local docker-archive tarball (no push, no registry ref);
- Trivy scans that tarball (input:) and Syft makes the SBOM from it;
- on main/v* pushes, hands the tarball and its config digest (build imageid)
to publish as a one-day artifact.
publish (pushes to main or a v* tag only; the only job with packages:write and
id-token:write):
- refuses a v* tag whose commit is not an ancestor of origin/main;
- recomputes the tarball's config digest and requires it to equal the scanned
one, so the check does not depend on the runner's image store;
- pushes all tags, asks the registry for the sha-<40> manifest and requires
its config digest to equal the scanned one;
- cosign signs by digest and attests the SBOM.
Image name: IMAGE_PREFIX = ghcr.io/${{ github.repository }}, giving
ghcr.io/abd-ulbasit/forgepoint/<svc>, the name the charts, Rollouts and
Kyverno already use. SECURITY.md documents the split and the corrected
`cosign verify` command.
Decisions:
- Push on main and v* tags only. Alternative: tags only. Why: Argo CD tracks
main, so main builds are deployable and should exist as scanned, signed
images. Tags must still be on main.
- Sign every published image, not only tags. Alternative: tags only, as
before. Why: an admission policy can then require a signature for every
image from this repo, instead of having to exempt main builds.
- Scan a tarball and push that tarball. Alternative: build with load:true,
scan, then rebuild with push:true from cache. Why: the rebuild is a second
artifact. Here the bytes scanned are the bytes pushed, and both digest checks
prove it.
Trade-off: the docker exporter cannot carry buildx provenance attestations, so
provenance/sbom are off in the build. The Syft SBOM is attested with cosign
instead. Build provenance (actions/attest-build-provenance) is a follow-up.
Not run: no actionlint locally. Reviewed by hand; YAML parses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No workflow ran terraform fmt, validate or plan, although
environments/dev/backend.tf said CI validated offline. The only thing reading
the .tf files was Trivy's misconfiguration scanner.
.github/workflows/terraform.yml:
- fmt: `terraform fmt -check -recursive -diff deploy/terraform`.
- validate, one leg per root (bootstrap, environments/dev):
`init -backend=false -lockfile=readonly`, then `validate`. -lockfile=readonly
makes CI install exactly what the committed .terraform.lock.hcl records, and
fail if a constraint changes without a regenerated lock.
- Terraform pinned to 1.14.8. hashicorp/setup-terraform pinned by SHA (v4.0.1,
the current release). Top-level permissions are {}, each job gets
contents: read, and checkout does not persist credentials.
- No plan: CI has no AWS credentials and nothing has been applied. The header
says how to add one later (a read-only OIDC role).
- No path filter, so the check can be made required without deadlocking
unrelated PRs.
Ran the same commands locally with Terraform 1.14.8: fmt clean, both roots
validate, and the lock files are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dependabot covered gomod, npm and github-actions only. Nothing refreshed the digest-pinned base images in the 12 Dockerfiles, the infra images now pinned by digest in deploy/, the migrate image in the charts, the compose stack or the AWS provider. A digest pin without automation rots silently: the image keeps its CVEs and the pin looks fine. - docker: /services/*, /deploy/k8s/infra/*, /deploy/k8s/observability/*, /deploy/llm, /deploy/backup (Dockerfiles and, per GitHub's docs, Kubernetes manifests). Minor/patch grouped. postgres and redis majors ignored, because a database major is a migration. - helm: /deploy/helm/* (images referenced by the charts). - docker-compose: / (same postgres/redis major ignore). - terraform: deploy/terraform/bootstrap and environments/dev. It updates the constraint and the committed .terraform.lock.hcl. Majors stay separate PRs. GitHub's docs say the docker ecosystem handles Kubernetes manifests and that the helm ecosystem updates images inside charts. Check the first PRs to see whether the tag@digest form is rewritten as expected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t's ECR Nothing in the cluster checked the signatures CI makes, so keyless signing stopped no attack. The registry allowlist also had two holes: - "*.dkr.ecr.*.amazonaws.com/*". Kyverno's `*` matches the account id too, so images from ANY AWS account's ECR were allowed, an attacker's included. The base no longer has an ECR entry (nothing deploys from ECR yet), and the comment says the EKS overlay must add one named account and repo prefix. - migrate/migrate was not allowlisted, although every Postgres-backed chart runs it as an initContainer. Flipping to Enforce would have blocked every schema migration. It is added (both spellings). New base/50-verify-images.yaml, verify-image-signatures (Audit, like every base policy): ghcr.io/abd-ulbasit/forgepoint/* must carry a cosign keyless signature with issuer https://token.actions.githubusercontent.com subject ^https://github\.com/abd-ulbasit/forgepoint/\.github/workflows/security\.yml@refs/(heads/main|tags/v.*)$ which is the identity the new publish job signs with. mutateDigest and verifyDigest are off while auditing (no mutation from an audit policy, and the charts still deploy by tag). The file and README say to turn both on with Enforce. The README table, the allowlist section and a new "Signature verification" section match. The stale "the chart points TODAY" wording is updated now that CI publishes to that path. Not tested against a cluster or the Kyverno CLI. YAML parses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pace The single "forgepoint" AppProject allowed `argocd` as a destination (so the root app-of-apps could create child Applications) and also whitelisted Secret, Role, RoleBinding and CustomResourceDefinition. Any chart or manifest merged to main, auto-synced, could therefore write a cluster/repo credential Secret or a RoleBinding into Argo CD's own namespace, which is privilege escalation through Git. It still did not whitelist the Application kind the root app exists to create, nor batch/Job, which the infra kustomization renders (the MinIO bucket init). - forgepoint-bootstrap (new, used by root.yaml): sourceRepos = this repo, destination = argocd only, no cluster-scoped kinds, one namespaced kind: argoproj.io/Application. - forgepoint (child apps): fp-system/fp-infra/fp-models/fp-jobs only (argocd removed). Cluster kinds: Namespace only (CRD removed). Namespaced kinds: exactly what the charts and infra render (checked with helm template), batch/Job added, Role/RoleBinding removed. - default-project.yaml (new): empties the built-in `default` project. An Application picks its own spec.project, so without this a child app could name `project: default` and bypass both projects. Applied by hand like project.yaml, never synced from Git. - Every apps/fp-*.yaml said "CI overwrites with the built image's immutable tag". No CI step does that (ADR-0002's tag bump was never built). The comments and README now say so and describe the manual, reviewed-PR path. Not applied to a cluster. All manifests parse. Before applying default-project.yaml, check that nothing in the cluster uses `default` (command in the file header). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…heus
The canary path was broken as written:
- Both Rollouts ran :latest images. Kyverno's disallow-latest-tag flags that,
and rollback means nothing when the "previous" version is also :latest.
- model-serving's Rollout ran a model-artifact-fetcher:latest initContainer.
That image was never built (there is no services/model-artifact-fetcher),
so every canary pod would sit in ImagePullBackOff. model-serving fetches
artifacts in-process into the OS temp dir.
- All five AnalysisTemplate queries used
prometheus-operated.fp-infra.svc, the Service name the Prometheus Operator
creates. This repo runs plain Prometheus, whose Service is `prometheus`
(deploy/k8s/observability/prometheus/prometheus.yaml), so every analysis run
would fail and abort the rollout.
Changes:
- Images: ghcr.io/abd-ulbasit/forgepoint/{inference-gateway,model-serving}
at 0.1.0 (Chart.appVersion), with the tag set in one place, an images:
block in kustomization.yaml. Comments say how to bump it.
- model-serving: the fetcher initContainer and the /models volume are
removed. A /tmp emptyDir (sizeLimit 2Gi) matches the Helm chart.
- AnalysisTemplates and the README point at
http://prometheus.fp-infra.svc.cluster.local:9090.
`kubectl kustomize deploy/rollouts` renders the pinned images and the new
address. Not applied to a cluster. Not changed: the ECR lifecycle cap of 20
tagged images also limits how far back a rollback can go, but nothing pushes
to ECR yet, so it is left for when ECR is used.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Trivy's secret scanner only reads the working tree, and there was no pre-commit hook. A secret committed and later removed was invisible to every gate while it stayed in every clone. A dev JWT key was in fact committed at some point. - New `gitleaks` job in security.yml: checkout with fetch-depth 0, build gitleaks v8.30.1 with `go install` (module checksum DB verified, like govulncheck) rather than a binary download or a third-party action, run `gitleaks git --redact --exit-code 1` over every commit. `image` now also needs it. - .gitleaks.toml extends the default rules with a narrow allowlist of published placeholders only: the old dev JWT value (public, and denylisted in the charts), the dev-only-change-me* and changeme-* chart/Secret placeholders, and the pkg/auth unit-test key. - .pre-commit-config.yaml runs the same rules on staged changes, pinned to the v8.30.1 commit SHA. Dependabot's pre-commit ecosystem keeps that pin current. - SECURITY.md: a gitleaks row in the gates table, plus a "Secrets in history" note. Any environment that ever signed tokens with the public dev JWT value must rotate it. A history purge needs a force-push, does not reach existing clones, and is not done here. Not run locally (no gitleaks binary here, and I did not download one). A grep of all 129 commits for high-signal formats (AWS, GitHub, OpenAI, Anthropic, Google, Slack keys, private-key headers) found nothing. The generic-API-key rule may still flag something the allowlist does not cover; if so, the first CI run shows it, redacted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit allowlisted the old dev JWT placeholder by its full value. That value is already public (git history, and the denylist in every chart's _helpers.tpl), but it was a signing key once and should not be repeated in new files. The existing ^dev-only-change-me(-[a-z0-9-]+)?$ pattern already covers it, so the explicit line is removed and the comment moved onto that pattern. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two audit findings are redesigns that should not be half-implemented in a fix branch. Following the repo's ADR convention (immutable once Accepted, new ADRs for changes), each gets a Proposed ADR with problem, options, recommendation and migration: - ADR 0010: replace the one HS256 key shared by 11 services with Ed25519 (EdDSA) signing in auth and a published public key set with `kid`. Today every verifier can mint admin tokens, and rotation means restarting all services in lockstep. The migration ships a dual-accepting validator first, then switches issuance, then removes HS256 and FP_JWT_SECRET from the 11 non-auth charts. - ADR 0011: one owner role per service database (no CREATEROLE/CREATEDB, PUBLIC connect revoked) instead of the RDS master / local superuser DSN in every pod. Master credentials are confined to a provisioning Job, with IAM database auth on RDS as the later step. Both are listed in docs/adr/README.md. No code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lind spots Two audit findings: - The images the cluster runs but this repo does not build (postgres, redis, nats, grafana, prometheus, tempo, loki, otel-collector, ollama, migrate) were never scanned. The image job covers first-party services only. - SECURITY.md described what each gate catches and nothing about what it misses (history secrets, third-party images, *.tfvars.example, runtime signature verification, npm devDependencies...). security.yml: - New `schedule` trigger (Mondays 04:17 UTC) so the existing gates also re-run against a fresh vulnerability DB. - third-party-matrix derives the image list from the repo: every `image: ...@sha256:` in deploy/ and docker-compose.yaml, plus repository/tag@sha256 pairs in chart values. A newly pinned image is covered without editing the workflow. - third-party-images: Trivy per image, HIGH/CRITICAL, exit-code 1, .trivyignore.yaml honoured. Schedule/dispatch only. Decision: third-party scans use ignore-unfixed and run off the PR path. Alternative: same settings as the first-party gates, on every push. Why: these CVEs are fixed by bumping a pin (which Dependabot proposes), not by the change under review. An unfixed upstream CVE cannot be fixed by bumping at all, so reporting it every week would only teach people to ignore the run. SECURITY.md: - "Third-party images" explains the above. - "What the gates do not see" is a catches/misses table for every gate, ending with the gap no workflow can close: main is unprotected, so red checks block nothing. Not run: no actionlint locally. The resolve script was run against the working tree and lists 11 images. YAML parses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ult project The AppProject key is `namespaceResourceWhitelist` (CRD and Go json tag). Every project here spelled it `namespacedResourceWhitelist`, already on main and carried into d04b80e, which built its claims on it. Two outcomes, both bad: - kubectl apply (strict field validation) rejects all three AppProjects with `unknown field "spec.namespacedResourceWhitelist"`, so root.yaml points at a project that does not exist. - Applied with validation off, the key is dropped and the whitelist is nil. Argo CD's IsGroupKindNamePermitted treats a nil namespaced whitelist as "every kind allowed", so forgepoint-bootstrap (destination argocd) could write Secrets, RoleBindings or AppProjects into argocd: the escalation d04b80e says it closes. The Role/RoleBinding removal in `forgepoint` was a no-op for the same reason. Changes: - Rename the key in both projects in project.yaml and in the README. - default-project.yaml: replace the empty whitelist with a `*/*` namespaceResourceBlacklist, the lock-down the Argo CD docs recommend. The field is `omitempty`, so an empty list does not survive a round trip through Argo CD (UI or `argocd proj`), and nil means allow-all. The blacklist denies every kind regardless. - Document the spelling trap and the server-side dry run in project.yaml and the README. Checked against a throwaway kube-apiserver v1.35.0 with the Argo CD v3.5.6 CRDs (since removed): - origin/main and the previous branch head: `kubectl apply --dry-run=server` fails with the strict-decoding error above. - This commit: all three AppProjects, root.yaml and the 11 child Applications pass the server-side dry run. - Stored objects evaluated with the permission logic copied from argo-cd v3.5.6: forgepoint-bootstrap allows only Application; forgepoint allows its listed kinds plus Namespace and denies Role, RoleBinding, AppProject; default allows nothing. The pre-fix spelling allowed every kind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gitleaks job added in c8207ac failed on every run, and `image` needs it, so nothing was ever built, scanned, published or signed. Running the pinned gitleaks v8.30.1 exactly as the job does (`gitleaks git --config .gitleaks.toml --redact --exit-code 1 .`) over all 141 commits found one hit: generic-api-key at docs/diagrams/c4-architecture.md:112 in commit d553f7f. The rule captured the C4 technology label "Traefik/Kong" as a key. It is not a secret. Add a second [[allowlists]] entry with condition = "AND": the value must be exactly "Traefik/Kong" AND the file must be that diagram. A path-and- value pair survives later edits of the line (a .gitleaksignore fingerprint is tied to one commit and line) and still reports the same string anywhere else. SECURITY.md records the finding and why it is allowlisted. Checked locally with gitleaks v8.30.1: - full history with the new config: "no leaks found", exit 0 - the flagged line copied into the diagram path and into another path: only the other path is reported, so the allowlist is scoped as intended Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…/ URL 1899dc1 moved billing and registry to redis.ParseURL so they accept the rediss:// URL from ElastiCache, but the URL the module writes (Secrets Manager property `url`, projected to FP_REDIS_URL by fp-infra-creds) embedded the raw token. override_special is "!&#$^<>-", and Go's net/url, which go-redis ParseURL uses, rejects `#` ("invalid port") and `^ < >` ("invalid userinfo") in the userinfo. The chance that a 48-character token avoids all four is (66/70)^48, about 6%, so roughly 94% of applies would have produced a URL that crashes billing and registry at boot with "invalid FP_REDIS_URL". ai-gateway, inference-gateway, feature-store, notification and model-monitor call ParseURL directly and had the same problem before this branch. Build the URL with urlencode(token). go-redis decodes the userinfo, so clients still send the exact token. This changes only the secret version's `url`; the token itself, the replication group and `auth_token` (raw, for clients that build their own connection) are untouched, so applying it does not rotate anything. Restricting override_special instead would have replaced random_password and rotated the token. Tests: billing and registry redisOptions gain a case with a token containing ! & # $ ^ < > -. It asserts the raw form is rejected and the url.QueryEscape form parses to the exact token with TLS on. Terraform's urlencode and Go's url.QueryEscape give the same output for this alphabet (checked with `terraform console` 1.14.8 and a Go probe). Correction to the 1899dc1 message: "Every Helm ConfigMap sets these URLs explicitly" is not accurate. fp-ai-gateway, fp-feature-store, fp-inference-gateway and fp-model-monitor set FP_REDIS_URL only in their Secret, which is now supplied out of band (secrets.create=false). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rraform roles deploy/terraform/README.md and the ElastiCache module said the state bucket is "readable only by the roles that run Terraform" / "restricted to the Terraform roles", as fact. state_access_principal_arns defaults to [] in both the state-backend module and the bootstrap root, and an empty list adds no deny statement, so until someone sets it any principal in the account with S3 and KMS read access can read state, including the ElastiCache AUTH token. Say what is true: restricted to the roles in state_access_principal_arns once that variable is set. The variable is left optional on purpose: the bootstrap runs before the Terraform roles necessarily exist, and a wrong list would lock the operator out of their own state. Setting it is listed as an owner action in the PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0f0fcb8 set secrets.create=false in every chart, which is the right default, but nothing said what that means for the GitOps apps. Every Deployment still mounts a Secret named after its release via envFrom, and nothing the root app syncs creates one. Only fp-auth has an ExternalSecret in the repo (applied by hand). The other nine apps would sync and then sit in CreateContainerConfigError. That is fail-closed by design, but an operator should not have to discover it from pod events. - deploy/argocd/README.md: a table of all ten apps with Secret name, namespace, the keys the chart expects (from each templates/secret.yaml) and what the repo provides today, plus how to fill the gap (ExternalSecret per app modelled on externalsecret-fp-auth.yaml, or a SealedSecret) and the reminder that FP_JWT_SECRET must match across all ten. - apps/fp-*.yaml: the valueFiles comment still said the committed values carry an "in-chart Secret with placeholder creds". It now says the in-chart Secret is off and the release-named Secret must already exist. Secret names were checked by rendering each chart with its app's release name and namespace (helm template, default values). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
restrict-image-registries matches fp-infra, where deploy/k8s/observability runs grafana/grafana, grafana/loki, grafana/tempo, prom/prometheus and otel/opentelemetry-collector-contrib. None was allowlisted. fd5ffae rewrote the allowlist and d3da918 pinned these images by digest, but only migrate/migrate was called out as an Enforce blocker; applying overlays/enforce would have blocked the whole observability plane. The gap predates this branch. Add each one as a full repo path, in both the short and docker.io- qualified spellings like the other infra images (never "grafana/*" or "docker.io/*"). The deny message and policies/README.md list them, and the README notes that ollama/ollama runs in fp-ml, which these policies do not match. Checked: every container and initContainer image rendered from deploy/k8s/infra, deploy/k8s/observability, deploy/backup, deploy/rollouts and all fp-* charts (24 distinct refs, digests included) matches at least one allowlist glob. overlays/audit and overlays/enforce still render. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- deploy/rollouts/README.md: the paragraph added in 753661d says the AnalysisTemplates now query the plain `prometheus` Service because this repo does not run the Prometheus Operator, but the old line below it still called that address "the kube-prometheus-stack (or Prometheus Operator) address". Removed; kept the "adjust if yours differs" advice. - deploy/rollouts/kustomization.yaml: the comment called newTag "0.1.0" a tag the Security workflow published. Nothing has been published: GHCR returns 404 for forgepoint/inference-gateway and model-serving (checked again today), and the existing v0.1.0 git tag predates the publish job. The comment now says 0.1.0 is a placeholder, that the Rollouts will be in ImagePullBackOff until a release passes a green pipeline, and how to get one (push a new v* tag once the gates pass). The pin is unchanged. - docs/adr/0010: "12 templates/secret.yaml files" is 11. All 11 reference JWT_SECRET (fp-bff has no secret.yaml). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The weekly schedule added in b1e2b19 shared the concurrency group security-<workflow>-<ref> with pushes to main. GitHub keeps at most one pending run per group and cancels it when another queues, regardless of cancel-in-progress. So a scheduled run that queued while a push run was in progress and another push run was pending cancelled that pending push run, and its commit was never published or signed. Add github.event_name to the group. Schedule and workflow_dispatch runs now queue separately and never displace a push. Pull-request runs are unaffected (their ref is already per PR). Push runs to main still queue behind each other as before; a burst of merges still publishes only the newest pending commit, which is the existing trade-off. actionlint passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
govulncheck failed in all 15 modules on GO-2026-6809 (crypto/x509, reachable from every service's http.Server). The workflows asked setup-go for a bare "1.26", which resolved to the runner's cached 1.26.9; the builder images were pinned to a golang:1.26-alpine digest that is Go 1.26.4. Go 1.26.10 fixes it. Set the exact patch release everywhere the version is chosen: - go.work and all 15 go.mod files: `go 1.26.10` (was 1.26.0 / 1.25.0). setup-go and the golang image both set GOTOOLCHAIN=local, so an older toolchain now refuses to build instead of silently compiling on an unpatched standard library. - GO_VERSION "1.26.10" in ci.yml and security.yml. - All 12 Dockerfiles: golang:1.26.10-alpine by multi-arch index digest (sha256:27f2d3b8..., GOLANG_VERSION=1.26.10 in its config). - README prerequisites and ADR 0006 now name the same version. Stayed on 1.26.x rather than 1.27.3: a patch bump fixes the advisory without a minor-version upgrade in a security PR. This clears only the standard-library finding. govulncheck still fails every module on reachable third-party advisories (x/net, grpc, x/text, otel, x/crypto, moby/go-archive), which are dependency bumps left to Dependabot #17 (see SECURITY.md). Checked with GOTOOLCHAIN=go1.26.10: govulncheck v1.1.4 reports no standard-library finding in any module; go build, go vet and go test -race -shuffle=on pass in all 15 modules (Docker-backed tests not run locally); golangci-lint v2.12.2 (release binary) reports 0 issues in the 14 linted modules; GOTOOLCHAIN=go1.26.9 is refused. services/ai-gateway built with buildx on the remote engine: its binary reports go1.26.10 and Trivy (image, os,library) finds no stdlib or OS CVE, only the dependency CVEs above. The other 11 images share the same FROM line; their builds could not finish (engine disk full). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CI web job failed at `npm audit --omit=dev --audit-level=high` on HIGH advisories in two production packages: - axios 1.18.0: GHSA-542g-h47m-68v8 (HTTP/2 session DoS) and eleven others, all fixed in 1.20.0. Dependabot #10 only reaches 1.19.0, which is still in the vulnerable range. - follow-redirects 1.16.0 (axios' transitive dependency): GHSA-q3jh-wrc6-jgq8 and GHSA-8r9p-f939-6h3c, fixed in 1.16.1. `npm install axios@1.20.0` raises the package.json floor to ^1.20.0; `npm update follow-redirects` moves the transitive pin within axios' existing ^1.16.0 range. Both with --package-lock-only --ignore-scripts. The lockfile diff is those two packages and axios' form-data range (already satisfied by the locked 4.0.6). Trivy's fs scan reported the same nine CVEs in web/package-lock.json; it no longer does. react-router-dom stays on 6.30.4. Its two advisories (GHSA-337j-9hxr-rhxg, GHSA-wrjc-x8rr-h8h6) are MODERATE, below the gate, and fixed only in the 7.x major (Dependabot #4). GHSA-337j only affects SSR hydration in Data/Framework mode; this app uses declarative <BrowserRouter> with no SSR. Checked in node:22 (npm 10.9.9, as CI): npm ci --ignore-scripts, npm run typecheck, npm run build, and npm audit --omit=dev --audit-level=high (exit 0, 2 moderate left). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The "Still red" table listed axios and follow-redirects, which the previous commit fixes, and said nothing about govulncheck. After the Go 1.26.10 pin, govulncheck still fails every module, on reachable third-party advisories rather than the standard library. - Drop the axios and follow-redirects rows; note they and GO-2026-6809 are fixed on this branch. - Add otel (GO-2026-6505, fixed in 1.45.0; #17 brings 1.46.0) and note that govulncheck wants x/crypto 0.56.0 for GO-2026-6354/6355, one past the 0.55.0 that Trivy and #17 use. - Note that bff also carries the otel advisory #17 does not reach. Versions are from govulncheck v1.1.4 and Trivy v0.70.0 run on this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
govulncheck failed in all 15 workspace modules and the Trivy filesystem
scan failed on every go.mod, all on third-party module advisories. This
moves each affected module to the lowest release that clears every
advisory, keeping majors unchanged:
golang.org/x/net v0.55.0 -> v0.60.0 GO-2026-6603/6611/6612/6617,
CVE-2026-46600, CVE-2026-78669
google.golang.org/grpc v1.82.1 -> v1.83.2 GO-2026-6348 (fixed 1.83.1),
GO-2026-6443 (1.83.x fix is
1.83.2), CVE-2026-84304/84445
golang.org/x/text v0.39.0 -> v0.42.0 GO-2026-6629, CVE-2026-56851
golang.org/x/crypto v0.52.0 -> v0.57.0 GO-2026-6354/6355,
CVE-2026-56854
go.opentelemetry.io/otel/* v1.44.0 -> v1.45.0 GO-2026-6505 (sdk,
otlptrace, otlptracegrpc)
github.com/moby/go-archive v0.2.0 -> v0.3.0 GO-2026-6253, CVE-2026-17106
github.com/klauspost/compress v1.18.0, v1.18.5 -> v1.18.7 CVE-2026-63209
x/text and x/crypto land one release above their advisory floors
(v0.41.0, v0.56.0) because x/net v0.60.0 itself requires x/text v0.42.0
and x/crypto v0.57.0. The other otel v1 modules pkg lists (exporters,
sdk/metric, trace, metric) move to v1.45.0 with the SDK, since otel-go
releases them as one set. otelgrpc stays at v0.69.0.
One code change falls out of the otel bump, in the same place as the last
two: otel 1.45's resource.Default() carries semconv schema 1.43.0, so
resource.Merge() in observability.Setup() rejected the v1.41.0 resource
with "conflicting Schema URL" and every service would have failed to
start. TestSetup_* caught it; the import moves to semconv/v1.43.0. The
API used there is unchanged.
SECURITY.md's "Still red" section deferred these bumps to Dependabot PR
#17, which clears neither gate. It now records them as fixed here, with
the advisory, fixed-in and resulting version for each.
Verified with GOTOOLCHAIN=go1.26.10 in every module: go build, go vet,
go test -race -shuffle=on pass; govulncheck v1.1.4 exits 0; Trivy v0.70.0
fs with the workflow's settings exits 0. Testcontainers-backed tests
skip locally (no Docker engine reachable); CI runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR 0006 has `go work sync` manage the requires, so this runs it after the advisory bump. It changes no version the workspace resolves. - go work sync moves genproto/googleapis/rpc to the version pkg already resolves (v0.0.0-20260803160001-6ac0973c030d), so each go.mod names the version the workspace builds. - go work sync also drops indirect requires that no package in the module itself needs, from cli, gen/go and the 12 services. In the workspace, and in the minimal go.work each Dockerfile generates, pkg's go.mod supplies them: otel, otel/sdk, otel/trace, otel/metric, auto/sdk, go-logr, xxhash, google/uuid. go.sum loses the matching hashes. - gen/go is tidied as well. It is the only module that imports no sibling module, so `go mod tidy` cannot reach for one. After the sync its go.sum lacked two go.mod hashes, and `GOWORK=off go build` failed. That build passes again. The other modules are not tidied. They import pkg or gen/go, which `go mod tidy` ignores go.work for, so tidy pins a proxy pseudo-version of main for them. ADR 0006 gotcha #1 forbids that. Verified on this tree with GOTOOLCHAIN=go1.26.10: - go build, go vet, go test -race -shuffle=on and golangci-lint v2.12.2 pass in all 15 modules. - govulncheck v1.1.4 exits 0 in all 15. - Trivy fs with the workflow's settings exits 0. - Each service's Dockerfile Go steps were replayed without Docker: go work init with pkg, gen/go and the service, then go mod download all, then a -mod=readonly build of cmd/server. All 12 build. Trivy finds no HIGH/CRITICAL in the 12 binaries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch fixes the findings from the October 2026 security audit across CI, Terraform, Helm, Argo CD, Kyverno and the local compose stack. It then fixes the three majors and seven minors from an independent review of the branch. Nothing here has been applied to a cluster or a cloud account. The last commits also fix the dependency advisories that kept the Security workflow red: the Go toolchain (1.26.10), the Go modules,
axiosandfollow-redirects. Dependabot PR #17 is superseded as a result (see "Not done here").Fixes
From the audit
imagebuilds to a tarball, scans it, makes the SBOM, never pushes. A separatepublishjob (main andv*tags only) checks that the digest it pushes is the digest that was scanned, then signs it with cosign keyless and attests the SBOM.github/workflows/security.yml,SECURITY.mdghcr.io/<owner>/forgepoint-<svc>, but charts, Rollouts and Kyverno expectghcr.io/abd-ulbasit/forgepoint/<svc>IMAGE_PREFIX=ghcr.io/${{ github.repository }}; cosign verify docs fixed.github/workflows/security.yml,SECURITY.mdpackages:writeandid-token:writewere granted to every job, PR runs includedpublishhas them, and it never runs forpull_requestorworkflow_dispatch.github/workflows/security.yml.trivyignore.yaml. Per-finding table in SECURITY.mddeploy/k8s/infra/postgres/,deploy/k8s/observability/grafana/,deploy/llm/,deploy/terraform/modules/{network,state-backend}/,.trivyignore.yaml,SECURITY.md,deploy/policies/terraform.tfvars.exampleTRIVY_HELM_KUBE_VERSION=1.36.0,tf-vars:set.github/workflows/security.yml0.0.0.0/0deploy/terraform/environments/dev/terraform.tfvars.examplemanage_master_user_passwordwith a CMK. ExternalSecret templates build the DSN from the RDS-managed secret. Redis token stays in state (provider 5.100 has no write-onlyauth_token), documenteddeploy/terraform/modules/{rds,elasticache}/,deploy/secrets/prevent_destroy, no TLS-only policyprevent_destroy, TLS-only deny, optional principal deny, S3 lockfile instead of DynamoDB, newbootstrap/rootdeploy/terraform/modules/state-backend/,deploy/terraform/bootstrap/,deploy/terraform/README.md.terraform.lock.hclgitignoredenvironments/devandbootstrap; CI uses-lockfile=readonly.gitignore,deploy/terraform/**/.terraform.lock.hclterraform fmt/validateterraform.yml(fmt, init with read-only lock, validate per root).github/workflows/terraform.ymlaws_budgets_budget, alerts at 50/80/100% actual and 100% forecast;budget_alert_emailsrequireddeploy/terraform/environments/dev/skip_final_snapshot,force_delete, 0-day secret recovery, no deletion protection)deploy/terraform/modules/{rds,elasticache,ecr}/,deploy/terraform/environments/dev/GenerateDataKey)deploy/terraform/modules/{s3,iam}/sensitivedeploy/terraform/README.mddeploy/terraform/modules/{ecr,eks}/,environments/dev/github-oidcmodule: ECR push role trusted only for main andv*tagsdeploy/terraform/modules/github-oidc/secrets.create=trueby default with a committed DSNfalsein every chart;make helm-installstill sets it for local Kinddeploy/helm/fp-*/,Makefile127.0.0.1by default; Redisrequirepass, NATS user/pass from.env; billing and registry parseredis:///rediss://URLsdocker-compose.yaml,.env.example,pkg/config/,services/*/cmd/server/pkg/config.RedactURL; eight services andnatsutillog redacted URLspkg/config/,pkg/natsutil/,services/*/cmd/server/main.goVERSIONbuild arg ignored, every service reporteddevARG VERSIONand stamp-X main.versionservices/*/Dockerfile,services/*/cmd/server/main.godeploy/k8s/,deploy/llm/,deploy/backup/,deploy/helm/fp-*/values.yaml.github/dependabot.ymlverify-image-signatures(Audit); ECR wildcard removed; migrate addeddeploy/policies/argocdforgepoint-bootstrapandforgepoint;defaultproject lockeddeploy/argocd/:latest, a never-built fetcher image and a Prometheus address that does not existprometheus.fp-infraaddressdeploy/rollouts/gitleaksjob over full history, pre-commit hook, narrow allowlist.github/workflows/security.yml,.gitleaks.toml,.pre-commit-config.yamldocs/adr/third-party-imagesscan; "What the gates do not see" table.github/workflows/security.yml,SECURITY.mdFrom the review of this branch
namespacedResourceWhitelist(already on main).kubectl applyrejects it; with validation off the whitelist is nil, which Argo CD treats as "allow every kind"namespaceResourceWhitelist.defaultproject now uses a*/*namespaced blacklist, as the Argo CD docs recommend. Spelling trap and dry-run command documenteddeploy/argocd/project.yaml,deploy/argocd/default-project.yaml,deploy/argocd/README.mdTraefik/Kongin the C4 diagram), soimagenever ran[[allowlists]]entry withcondition = "AND": that exact value in that one file.gitleaks.toml,SECURITY.mdrediss://URL embedded the raw token;# ^ < >break Go'snet/url, so about 94% of applies would crash billing and registry at booturlencode(token)in the URL; tests in billing and registrydeploy/terraform/modules/elasticache/main.tf,services/{billing,registry}/cmd/server/redis_options_test.go,deploy/secrets/README.mdstate_access_principal_arnsis set (default[]adds nothing)deploy/terraform/README.md,deploy/terraform/modules/elasticache/main.tfdeploy/rollouts/README.mdnewTag: "0.1.0"described as published; GHCR has no imagesdeploy/rollouts/kustomization.yamlsecret.yamlfilesdocs/adr/0010-asymmetric-jwt-signing.mdFP_REDIS_URLonly in their Secret, not their ConfigMapdeploy/argocd/README.md,deploy/argocd/apps/fp-*.yamldeploy/policies/base/20-image-policies.yaml,deploy/policies/README.mdgithub.event_nameadded to the group.github/workflows/security.ymlDecisions for you to confirm
secrets.create=falseeverywhere, so apps fail closedtruewith placeholders for convenienceCreateContainerConfigErroruntil you add their ExternalSecrets?v*tag is published and signedload, scan, rebuild withpush(keeps provenance)attest-build-provenanceis added?ignore-unfixed, never on PRsrandom_password(in state)envFrom, so they need a reloader or scheduled restarts. Which one?urlencode()in the URL, no token changeoverride_special, which replacesrandom_passwordand rotates the tokenstate_access_principal_arnsoptional, default[]enable_github_ecr_push=false).gitleaksignorefingerprint (tied to one commit and line)defaultAppProject*/*blacklistproject: default? The check command is in the file header.deploy/llm/README.mdnow or later?Not done here, needs you
GitHub settings
main(branch protection or a ruleset) and require the Security, Terraform and CI checks. Todaymainhas no protection, so red checks block nothing.ghcr.io/abd-ulbasit/forgepoint/*) or give the cluster an image pull secret.Dependencies
No dependency bumps are left for you. These are no longer deferred to Dependabot.
4915ca9andb00771fmove all 15 Go modules to the lowest release that clears every advisory govulncheck and Trivy reported. Majors are unchanged:golang.org/x/netgoogle.golang.org/grpcgolang.org/x/textgolang.org/x/cryptoklauspost/compressmoby/go-archivex/textandx/cryptoend up one release above their fixed-in versions becausex/net0.60.0 requires them.7168eddalready movedaxiosto 1.20.0 andfollow-redirectsto 1.16.1. SECURITY.md has the per-advisory table.Dependabot PR deps: bump the go-minor-and-patch group across 12 directories with 21 updates #17 is now superseded and can be closed, at your discretion; this PR does not touch it. It cleared neither gate: it stopped at
x/net0.58.0,x/crypto0.55.0,klauspost/compress1.18.6 andmoby/go-archive0.2.0, and it skippedcli/,gen/goandservices/bff. It also addsrequirelines with pseudo-versions ofpkgandgen/go, which ADR 0006 rules out. Neither gate needs its other bumps, for examplepgx,nats.go, testcontainers and otel 1.46. Dependabot's next run will propose whatever is still behind against the new go.mod files.Credentials
Cloud (nothing has been applied)
deploy/terraform/bootstrapwithstate_access_principal_arnsset, then migrateenvironments/devstate into the bucket.budget_alert_emailsand, if wanted,enable_github_ecr_push.<prefix>/auth/jwtsecret by hand, and replace the<RDS_MASTER_USER_SECRET_ARN>placeholders in the ExternalSecrets with the Terraform output.Cluster
deploy/argocd/project.yamlanddefault-project.yamlby hand afterkubectl apply --dry-run=serveron the real cluster.deploy/argocd/README.mdbefore enabling them.FP_JWT_SECRETmust match across all ten.kubectl -n fp-infra logs postgres-0(postgres now runs with a read-only root).mutateDigestandverifyDigestwhenverify-image-signaturesgoes to Enforce.v*tag to publish the first signed images, then set that version indeploy/rollouts/kustomization.yamland the Argo CDimage.tagparameters.Design decisions waiting on you
minio/minioandminio/mcnow return 404 on Docker Hub, so they cannot be pinned or pulled. Pick a replacement image or another object store.actions/attest-build-provenance) is a follow-up.Checks run
kubectl apply --dry-run=serverunknown field "spec.namespacedResourceWhitelist"). This branch: all 3 AppProjects,root.yamland 11 child Applications pass. The server was removed afterwards.forgepoint-bootstrap: Application only.forgepoint: its listed kinds plus Namespace; no Role, RoleBinding or AppProject.default: nothing. The old spelling allowed every kind.gitleaks git --config .gitleaks.toml --redact --exit-code 1 .b00771f: 154 commits, no leaks.go build,go vet,go test ./cmd/server/for billing and registryterraform console:urlencode()vs Gourl.QueryEscapeon the token alphabetfmt -check -recursive;init -backend=false -lockfile=readonlyandvalidateforbootstrapandenvironments/devkubectl kustomizefordeploy/rollouts,deploy/policies/overlays/{audit,enforce}security.ymlb00771f: exit 0. 0 HIGH/CRITICAL vulnerabilities in all 15 go.mod files andweb/package-lock.json, 0 misconfigurations, 0 secrets. Before4915ca9: 100 HIGH dependency findings across the 15 go.mod files.GOTOOLCHAIN=go1.26.10, in each of the 15 go.work modulesb00771f: exit 0 in all 15. Before4915ca9: exit 3 in all 15, from GO-2026-6603/6611/6612/6617, 6348, 6443, 6505, 6629, 6354/6355 and 6253. The only module-level note left is GO-2026-5932 (x/crypto/openpgpdeprecated). It has no fix, nothing imports the package, and it does not fail the gate.go build,go vet,go test -race -shuffle=onin all 15 modules at4915ca9andb00771f; golangci-lint v2.12.2 in all 15 atb00771fgo work init,go mod download all,-mod=readonlybuild ofcmd/server, then Trivy on the 12 binariesHow to review
db2f8d1,8e81eb4,a9db793,b4ea7f2,5a9adad,80f812e,251e68b,761b0d0,f545e46,d54ebed, thenbc44e4e(Redis URL encoding) anddc54865(state-bucket wording).0f0fcb8,cb19e95,d636d12,d3da918.005498b,1899dc1,0f189d7.d04a049,7bc38e5,ba1e40d,d35fa3b,e8d76e1,c8207ac,b8e1d59,b1e2b19, then5710920(gitleaks false positive) anda3d222b(concurrency).fd5ffae,bde56b9,d04b80e,aaa484b(whitelist spelling),6b82d5e(Secrets per app),753661d.90f45e7(ADRs),c831592.dc9a4e3(Go 1.26.10),7168edd(web),ece4442(SECURITY.md), then the module bumps:4915ca9: version bumps plus the semconv v1.43.0 import that otel 1.45 needs.b00771f:go work syncandgo mod tidyingen/go. No resolved version changes.🤖 Generated with Claude Code