[vahid-ahmadi]

Summary

hugging_face_auth_headers() reads only HUGGING_FACE_TOKEN, HF_TOKEN and HUGGINGFACE_HUB_TOKEN. A user who follows Hugging Face's own setup (hf auth login) has a valid token saved, but ensure_datasets, managed_microsimulation and materialize_dataset send no token and then report:

DatasetMaterializationError: Could not download UK dataset 'microcosm_uk_2024_25': Hugging Face rejected the configured credentials. Set HUGGING_FACE_TOKEN to a token with access to the repository.

No credentials were configured, so the message points the wrong way, and it does not say how to get access to the private UK data.

Changes:

• Fall back to huggingface_hub.get_token() (the hf auth login token) when none of the environment variables is set. The environment variables keep precedence. huggingface_hub comes in through policyengine-core; the import is guarded.
• One helper, hugging_face_auth_failure(), builds the 401/403 message for both download paths (bundle datasets and explicit hf:// URIs). It says either "no Hugging Face credentials were found" or "rejected the configured credentials", names both ways to supply a token, and gives the access route from the policyengine-uk getting-started guide (email contact@policyengine.org with the Hugging Face username and evidence of UK Data Service access).

Found while reproducing a published think-tank energy-bill targeting analysis on Microcosm UK 2024-25 as a new user with only the public docs.

Evidence

On 6.10.1 with a valid hf auth login and no token environment variables, pe.uk.ensure_datasets(datasets=["microcosm_uk_2024_25"], years=[2024]) fails with the message above. With this branch, hugging_face_auth_headers() returns the saved token, and a HEAD request for the pinned microcosm_uk_2024_25.h5 returns 302 (redirect to the file) instead of 401.

Tests

tests/test_dataset_materialization.py gains four tests: the CLI login is used when no environment variable is set; an environment token takes precedence; a 401 without a token says none was found and gives the access route; a 403 with a token says it was rejected and does not echo the token. The first three fail on main; all 17 tests in the file pass here, as do tests/test_release_manifests.py and tests/test_dataset_source.py. ruff check and ruff format --check pass on the changed files.

🤖 Generated with Claude Code