Repository navigation
Lock certifi 2026.10.19 to stop trusting the compromised eMudhra CA certificates - #23
Merged
Merged
Conversation
Add tests that fail while uv.lock locks a certifi older than 2026.10.19, or while the trust store httpx uses by default for the Hugging Face identity check contains "emSign ECC Root CA - G3" or "AAA Certificate Services". Both fail against the currently locked certifi 2026.7.22. Refs #22. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
certifi 2026.10.19 removes "emSign ECC Root CA - G3", the root that the eMudhra intermediates with recoverable private keys chain to. The image installs the locked versions with `uv sync --frozen`, and httpx verifies the Hugging Face identity check against certifi's bundle, so the next deploy stops trusting certificates issued from those intermediates. Produced with `uv lock --upgrade-package certifi` (uv 0.13.5); no other package changed. Fixes #22. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk
marked this pull request as ready for review
October 22, 2026 18:51
anth-volk
pushed a commit
that referenced
this pull request
Oct 27, 2026
Brings in #6 (the submission contract, released as 0.4.0), #4, #11 and #23. No conflicts: README.md, pyproject.toml and uv.lock merged automatically, and the consumed fragment changelog.d/artifact-submission-contract.added.md is removed, leaving only this branch's artifact-intake.added.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #22
Why
On 2026-10-20 it was disclosed that eight eMudhra (emSign) intermediate CA certificates have private keys that anyone can recover from the public certificates (https://emudhra.we-have-a-problem.com). They chain to "emSign ECC Root CA - G3" and, through Sectigo's cross-signatures, to "AAA Certificate Services". The collector locked certifi 2026.7.22, which contains "emSign ECC Root CA - G3", so the deployed collector would accept a forged certificate for
huggingface.cowhen it verifies a Hugging Face token. PolicyEngine/microcosm-emitter#30 and PolicyEngine/calibration-diagnostics#256 made the same upgrade.Changes
uv.lock: certifi 2026.7.22 -> 2026.10.19, produced withuv lock --upgrade-package certifiusing uv 0.13.5, the newest release, which CI'ssetup-uvinstalls because it sets noversion. No other package changed (39 packages resolved). The hashes match microcosm-emitter#30's.tests/test_certificate_authorities.py(new): one test requires the locked certifi to be 2026.10.19 or later; the other builds the SSL context httpx uses by default and requires that it trusts neither "emSign ECC Root CA - G3" nor "AAA Certificate Services". Neither emitter#30 nor calibration-diagnostics#256 added a test.changelog.d/emudhra-ca-trust-update.fixed.md: a "fixed" fragment, so merging releases 0.3.3.certifi 2026.10.19 removes only "emSign ECC Root CA - G3". It keeps three other eMudhra roots (emSign Root CA - G1, emSign Root CA - C1, emSign ECC Root CA - C3), which the disclosure does not implicate, and neither certifi version contains "AAA Certificate Services".
Where the collector's HTTPS requests get their certificate authorities
telemetry_collector/auth.py,GET https://huggingface.co/api/whoami-v2):httpx.getwith the defaultverify=True. httpx 0.28.1 then callsssl.create_default_context(cafile=certifi.where()), unlessSSL_CERT_FILEorSSL_CERT_DIRis set. Neither the base image, the Dockerfile nor the deployment sets either; the production service's environment holds onlyTELEMETRY_MAINTENANCE_MODE,DATABASE_URL,TELEMETRY_JWT_SECRETandTELEMETRY_READ_TOKEN. This is the collector's only outbound HTTPS request, and this PR fixes it.uv.lockcontains no Google client library. Cloud Run reads the Secret Manager values and passes them to the container as environment variables before it starts.--add-cloudsql-instances. The instance has no authorized networks and no private network, so the collector can reach it only through Cloud Run's Cloud SQL connector, which psycopg reaches through a Unix socket. The connector authenticates the instance with Cloud SQL's own CA (serverCaMode: GOOGLE_MANAGED_INTERNAL_CA), not with certifi or the operating-system store. I did not readDATABASE_URL, so the socket use is inferred from the instance settings.Not changed: the base image's operating-system CA store
ghcr.io/astral-sh/uv:python3.13-bookworm-slimwas last rebuilt on 2026-02-04. It has Debian'sca-certificates20230311+deb12u1, whose store contains both "emSign ECC Root CA - G3" and "AAA Certificate Services", andssl.create_default_context()without a CA file trusts both in that image. No collector code uses that store, because httpx passes certifi's file explicitly. Code added later that usesurllib,ssl.create_default_context()without a CA file, orcurlinside the image would trust the affected roots unless the image upgradesca-certificates.Checks
assert (2026, 7, 22) >= (2026, 10, 19)andassert {'emSign ECC Root CA - G3'} == set().pytest tests/test_certificate_authorities.py tests/test_packaging.py tests/test_auth.pypassed (9 tests).uv lock --locked(uv 0.13.5) passes.tools/check-quality.sh(Ruff check and format,bash -non the scripts) passes.tools/check-changelog.shreports 0.3.2 -> 0.3.3.docker buildof this branch, then inside the image: certifi 2026.10.19 under/app/.venv, httpx's default context trusts neither affected root, and noSSL_CERT_*,REQUESTS_CA_BUNDLEorCURL_CA_BUNDLEvariable is set.Merging releases and deploys production
Merging into
mainstartspush.yaml, which commits version 0.3.3 and its changelog, runs CI, tags the commit, builds the image fromuv.lock(uv sync --frozen --no-dev), deploys it to staging and then, after staging succeeds, to production.COLLECTOR_DEPLOY_ENABLEDistrue, and the previous release run (for 9b32902) deployed both. Production currently runs the image built from 9b32902, with certifi 2026.7.22.Environment variables
This PR changes no workflow, deployment script or Dockerfile, and it adds, removes or renames no environment variable. The new image reads the same ones as before:
DATABASE_URL,TELEMETRY_JWT_SECRETandTELEMETRY_READ_TOKENfrom Secret Manager through--set-secretsinscripts/deploy.sh;TELEMETRY_MAINTENANCE_MODEthrough--set-env-vars; and the optionalTELEMETRY_HF_ORG,TELEMETRY_JWT_ISSUER,TELEMETRY_JWT_AUDIENCEandTELEMETRY_SESSION_TOKEN_TTL_SECONDS, which production leaves unset so their defaults apply. The new test removesSSL_CERT_FILEandSSL_CERT_DIRfrom its own process only.🤖 Generated with Claude Code