Skip to content

UK population data: no public path for outside analysts, and no documented bring-your-own-FRS route #711

Open

Description

@vahid-ahmadi

Problem

An outside analyst who wants to reproduce a published UK distributional analysis with policyengine.py has no UK population data to start from, and the docs do not tell them how to get some.

  • Every UK dataset in the 6.10.1 bundle (microcosm_uk_2024_25 and the overlays enhanced_frs_*, frs_*, populace_uk_2023) is in a private Hugging Face repository. The public policyengine/policyengine-uk-data-public holds only geography weights. There is no public or synthetic UK population file, so no UK population example in this repo runs without access.
  • The access route exists only in the policyengine-uk getting-started guide (policyengine-uk#1563): email contact@policyengine.org with a Hugging Face username and evidence of UK Data Service access. policyengine.py's README and docs/microsim.md said only "a token from an account with access" (Point the UK quickstart and examples at the certified dataset #709 adds the route). The request form proposed in policyengine-uk#1115 was not built; that issue was closed after the docs change.
  • An analyst who already holds the FRS under a UK Data Service licence can run managed_microsimulation(dataset=path, allow_unmanaged=True) on a file in policyengine-uk's format, but nothing documents how to produce that file from the raw FRS release they hold, which FRS years the build code supports, or how their results relate to the certified Microcosm build.

Evidence

A replication of a published energy bill targeting analysis on FRS 2024-25, done as a new user with the public docs:

  • without a token the download fails, and on 6.10.1 the error said the "configured credentials" were rejected although none were configured (Use the hf auth login token for dataset downloads #707 fixes the message and accepts the hf auth login token);
  • with access, the replication works: on Microcosm cut a133f37a it lands within a few points of the publication's headline shares (38% vs about 40% passing the £24,000 individual test; 71% vs 75% of the four lowest after-housing-costs deciles; 23% vs about 25% passported on reported receipt);
  • an external replication that rebuilt FRS 2024-25 from raw UK Data Service files needed run-time patches for FRS codes the build code does not know.

Proposal

  1. One page in policyengine.py (linked from the README's UK section): who can get UK data access, how to request it, what to expect, and what runs without it.
  2. Decide whether a small public synthetic UK file, with clear labels that it is not for analysis, can ship so the UK examples and tests run for anyone.
  3. Document the bring-your-own-FRS route: the supported FRS years, the build entry point, and how to run the result through allow_unmanaged=True.

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions