Skip to content

Lock certifi 2026.10.19 to stop trusting the compromised eMudhra CA certificates - #23

Merged
anth-volk merged 2 commits into
mainfrom
claude/upgrade-certifi
Oct 22, 2026
Merged

anth-volk merged 2 commits into
mainfrom
claude/upgrade-certifi

Conversation

@anth-volk

Copy link
Copy Markdown
Contributor

Fixes #22

Why

On 2026-10-20 it was disclosed that eight eMudhra (emSign) intermediate CA certificates have private keys that anyone can recover from the public certificates (https://emudhra.we-have-a-problem.com). They chain to "emSign ECC Root CA - G3" and, through Sectigo's cross-signatures, to "AAA Certificate Services". The collector locked certifi 2026.7.22, which contains "emSign ECC Root CA - G3", so the deployed collector would accept a forged certificate for huggingface.co when it verifies a Hugging Face token. PolicyEngine/microcosm-emitter#30 and PolicyEngine/calibration-diagnostics#256 made the same upgrade.

Changes

  • uv.lock: certifi 2026.7.22 -> 2026.10.19, produced with uv lock --upgrade-package certifi using uv 0.13.5, the newest release, which CI's setup-uv installs because it sets no version. No other package changed (39 packages resolved). The hashes match microcosm-emitter#30's.
  • tests/test_certificate_authorities.py (new): one test requires the locked certifi to be 2026.10.19 or later; the other builds the SSL context httpx uses by default and requires that it trusts neither "emSign ECC Root CA - G3" nor "AAA Certificate Services". Neither emitter#30 nor calibration-diagnostics#256 added a test.
  • changelog.d/emudhra-ca-trust-update.fixed.md: a "fixed" fragment, so merging releases 0.3.3.

certifi 2026.10.19 removes only "emSign ECC Root CA - G3". It keeps three other eMudhra roots (emSign Root CA - G1, emSign Root CA - C1, emSign ECC Root CA - C3), which the disclosure does not implicate, and neither certifi version contains "AAA Certificate Services".

Where the collector's HTTPS requests get their certificate authorities

  • Hugging Face (telemetry_collector/auth.py, GET https://huggingface.co/api/whoami-v2): httpx.get with the default verify=True. httpx 0.28.1 then calls ssl.create_default_context(cafile=certifi.where()), unless SSL_CERT_FILE or SSL_CERT_DIR is set. Neither the base image, the Dockerfile nor the deployment sets either; the production service's environment holds only TELEMETRY_MAINTENANCE_MODE, DATABASE_URL, TELEMETRY_JWT_SECRET and TELEMETRY_READ_TOKEN. This is the collector's only outbound HTTPS request, and this PR fixes it.
  • Google APIs: the collector's code calls none, and uv.lock contains no Google client library. Cloud Run reads the Secret Manager values and passes them to the container as environment variables before it starts.
  • Database: the deployment attaches the Cloud SQL instance with --add-cloudsql-instances. The instance has no authorized networks and no private network, so the collector can reach it only through Cloud Run's Cloud SQL connector, which psycopg reaches through a Unix socket. The connector authenticates the instance with Cloud SQL's own CA (serverCaMode: GOOGLE_MANAGED_INTERNAL_CA), not with certifi or the operating-system store. I did not read DATABASE_URL, so the socket use is inferred from the instance settings.

Not changed: the base image's operating-system CA store

ghcr.io/astral-sh/uv:python3.13-bookworm-slim was last rebuilt on 2026-02-04. It has Debian's ca-certificates 20230311+deb12u1, whose store contains both "emSign ECC Root CA - G3" and "AAA Certificate Services", and ssl.create_default_context() without a CA file trusts both in that image. No collector code uses that store, because httpx passes certifi's file explicitly. Code added later that uses urllib, ssl.create_default_context() without a CA file, or curl inside the image would trust the affected roots unless the image upgrades ca-certificates.

Checks

  • Before the upgrade (commit 29618aa), both new tests failed: assert (2026, 7, 22) >= (2026, 10, 19) and assert {'emSign ECC Root CA - G3'} == set().
  • After it: pytest tests/test_certificate_authorities.py tests/test_packaging.py tests/test_auth.py passed (9 tests).
  • uv lock --locked (uv 0.13.5) passes. tools/check-quality.sh (Ruff check and format, bash -n on the scripts) passes. tools/check-changelog.sh reports 0.3.2 -> 0.3.3.
  • docker build of this branch, then inside the image: certifi 2026.10.19 under /app/.venv, httpx's default context trusts neither affected root, and no SSL_CERT_*, REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE variable is set.
  • I did not run the full test suite locally; CI runs it.

Merging releases and deploys production

Merging into main starts push.yaml, which commits version 0.3.3 and its changelog, runs CI, tags the commit, builds the image from uv.lock (uv sync --frozen --no-dev), deploys it to staging and then, after staging succeeds, to production. COLLECTOR_DEPLOY_ENABLED is true, and the previous release run (for 9b32902) deployed both. Production currently runs the image built from 9b32902, with certifi 2026.7.22.

Environment variables

This PR changes no workflow, deployment script or Dockerfile, and it adds, removes or renames no environment variable. The new image reads the same ones as before: DATABASE_URL, TELEMETRY_JWT_SECRET and TELEMETRY_READ_TOKEN from Secret Manager through --set-secrets in scripts/deploy.sh; TELEMETRY_MAINTENANCE_MODE through --set-env-vars; and the optional TELEMETRY_HF_ORG, TELEMETRY_JWT_ISSUER, TELEMETRY_JWT_AUDIENCE and TELEMETRY_SESSION_TOKEN_TTL_SECONDS, which production leaves unset so their defaults apply. The new test removes SSL_CERT_FILE and SSL_CERT_DIR from its own process only.

🤖 Generated with Claude Code

anth-volk and others added 2 commits October 22, 2026 21:46
Add tests that fail while uv.lock locks a certifi older than 2026.10.19, or
while the trust store httpx uses by default for the Hugging Face identity
check contains "emSign ECC Root CA - G3" or "AAA Certificate Services". Both
fail against the currently locked certifi 2026.7.22. Refs #22.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
certifi 2026.10.19 removes "emSign ECC Root CA - G3", the root that the
eMudhra intermediates with recoverable private keys chain to. The image
installs the locked versions with `uv sync --frozen`, and httpx verifies the
Hugging Face identity check against certifi's bundle, so the next deploy
stops trusting certificates issued from those intermediates. Produced with
`uv lock --upgrade-package certifi` (uv 0.13.5); no other package changed.

Fixes #22.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review October 22, 2026 18:51
@anth-volk
anth-volk merged commit bd44a21 into main Oct 22, 2026
4 checks passed
@anth-volk
anth-volk deleted the claude/upgrade-certifi branch October 22, 2026 18:51
anth-volk pushed a commit that referenced this pull request Oct 27, 2026
Brings in #6 (the submission contract, released as 0.4.0), #4, #11 and #23.
No conflicts: README.md, pyproject.toml and uv.lock merged automatically, and
the consumed fragment changelog.d/artifact-submission-contract.added.md is
removed, leaving only this branch's artifact-intake.added.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade certifi so the collector stops trusting the compromised eMudhra CA certificates

1 participant