Serve a whole v2 metadata catalog version in one response (GET /v2/model-catalog)

changelog.d/apiv2-1e-metadata-catalog-route.added.md

@@ -0,0 +1 @@
+GET /v2/model-catalog returns every parameter node, parameter (with its current-law values keyed by start date) and variable of a catalog version in one response, for clients that load the whole model at once. Each instance builds the document once per country and catalog version, and the response supports ETag revalidation.

docs/engineering/migration-contracts.md

@@ -8,7 +8,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
 | --- | ---: |
 | route group count | 9 |
 | workflow count | 16 |
-| request count | 89 |
+| request count | 90 |
 | db entity count | 6 |
 | sim flow count | 3 |
 
@@ -18,7 +18,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
 | --- | --- | --- | --- |
 | `health` | `health`, `simulation-gateway-check`, `liveness-check`, `readiness-check` | `none` | `none` |
 | `specification` | `specification` | `none` | `none` |
-| `metadata` | `metadata`, `datasets`, `economy-options`, `parameter-values`, `parameters`, `regions`, `tax-benefit-model-versions`, `tax-benefit-models`, `variables` | `metadata` | `none` |
+| `metadata` | `metadata`, `datasets`, `economy-options`, `model-catalog`, `parameter-values`, `parameters`, `regions`, `tax-benefit-model-versions`, `tax-benefit-models`, `variables` | `metadata` | `none` |
 | `policy` | `policy`, `policies`, `user-policy`, `user-policies` | `policy` | `none` |
 | `household` | `household`, `households`, `user-household`, `user-households`, `calculate`, `calculate-full` | `household` | `household` |
 | `economy` | `economy` | `simulation` | `economy` |
@@ -214,6 +214,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
 | `GET` | `/v2/regions/{region_id}?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.item` |
 | `GET` | `/v2/regions/by-code/state/ca?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.item` |
 | `GET` | `/v2/tax-benefit-models/by-country/us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.model`, `result.model_version` |
+| `GET` | `/v2/model-catalog?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.parameter_nodes`, `result.parameters`, `result.variables` |
 | `GET` | `/v2/economy-options?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.current_law_id`, `result.region`, `result.time_period`, `result.datasets` |
 
 ### `simulation_submit_poll`

docs/generated/migration_contracts.json

@@ -1,7 +1,7 @@
 {
   "metadata": {
     "db_entity_count": 6,
-    "request_count": 89,
+    "request_count": 90,
     "route_group_count": 9,
     "sim_flow_count": 3,
     "workflow_count": 16
@@ -33,6 +33,7 @@
         "metadata",
         "datasets",
         "economy-options",
+        "model-catalog",
         "parameter-values",
         "parameters",
         "regions",
@@ -1599,6 +1600,21 @@
             "result.model_version"
           ]
         },
+        {
+          "expected_status": 200,
+          "headers": [],
+          "method": "GET",
+          "path": "/v2/model-catalog?country_id=us",
+          "route_group": "metadata",
+          "stable_response_fields": [
+            "status",
+            "message",
+            "result.policyengine_version",
+            "result.parameter_nodes",
+            "result.parameters",
+            "result.variables"
+          ]
+        },
         {
           "expected_status": 200,
           "headers": [],

docs/migration/stage-9-v2-metadata.md

@@ -12,8 +12,10 @@ read-only, resource-specific routes from the Cloud Run ASGI application under
 `/v2`. The routes separately serve tax-benefit models, model versions,
 variables, parameters, direct parameter-tree children, canonical parameter
 values, logical input datasets, regions, and compact economy-selection
-options. Collection routes require `country_id`, use bounded `offset` and
-`limit` pagination, and never assemble the complete catalog into one response.
+options. Collection routes require `country_id` and use bounded `offset` and
+`limit` pagination. One later addition, `GET /v2/model-catalog`, does assemble
+a whole catalog version into one cached response for clients that load the
+complete model at start-up (see Preview verification).
 The generated OpenAPI document is available at `GET /v2/openapi.json`.
 
 The unprefixed Flask/API v1 routes remain mounted within the Cloud Run ASGI
@@ -198,6 +200,24 @@ Request `GET /v2/openapi.json` and confirm that the public document contains
 every documented resource path and explicit component schema references for
 success and error responses.
 
+`GET /v2/model-catalog` returns a whole catalog version in one response: every
+parameter node, every parameter with its current-law values (keyed by start
+date), and every variable. It exists for clients such as the app's v3
+workspace that load the whole model at start-up; the paged routes would need
+about 2,550 requests of 500 rows for the US catalog (100,276 `gov` parameters
+and 1,174,586 values). Each API instance builds the serialized document once
+per country and catalog version and keeps the four most recently used. Measured
+locally against the 6.7.5 catalog, the US document takes 5.8 seconds to build
+and is 60.7 MB (12.1 MB gzipped); the UK document takes 1.0 second and is
+1.3 MB. The response carries an ETag naming the country, catalog version and
+document format, and `Cache-Control: public, max-age=86400` when the request
+names `policyengine_version` (`max-age=300` otherwise, because the running
+version changes on deploy). The release workflow checks the UK document,
+including a conditional request that must return 304. The route reads its
+query through the shared `CatalogQuery` model, so an unknown or repeated
+parameter, an unsupported country or an invalid version returns 422 before
+any database read (the paged metadata routes return 400 for the last two).
+
 Also request the unprefixed US and UK metadata endpoints and confirm their
 responses still come from v1. Do not modify route selectors, internal callers,
 or traffic rules to use `/v2` during Stage 9.

policyengine_api/fastapi_routes/dependencies.py

@@ -31,8 +31,12 @@
         AccountPrincipal,
         UserPrincipal,
     )
+    from policyengine_api.services.v2.metadata.catalog_cache import (
+        MetadataCatalogCache,
+    )
     from policyengine_api.services.v2.metadata.types import (
         MetadataCanonicalParameterValue,
+        MetadataCatalogDocument,
         MetadataDataset,
         MetadataDetailResult,
         MetadataEconomyOptionsResult,
@@ -309,6 +313,18 @@ def get_economy_options(
         """
         ...
 
+    def get_model_catalog(
+        self,
+        country_id: str,
+        policyengine_version: str | None = None,
+    ) -> MetadataCatalogDocument:
+        """Return the selected catalog's parameter nodes, parameters and variables.
+
+        Parameters carry their current-law values. The result is the serialized
+        success envelope, built once per country and catalog version.
+        """
+        ...
+
 
 class V2PolicyResourceService(Protocol):
     """Route-independent native policy operations for one request."""
@@ -723,6 +739,16 @@ def _running_policyengine_version() -> str:
     return importlib_metadata.version("policyengine")
 
 
+@lru_cache(maxsize=1)
+def _metadata_catalog_cache() -> MetadataCatalogCache:
+    """Return the catalog documents this process keeps for every request."""
+    from policyengine_api.services.v2.metadata.catalog_cache import (
+        MetadataCatalogCache,
+    )
+
+    return MetadataCatalogCache()
+
+
 def _default_v2_metadata_reader_factory() -> V2MetadataResourceReader:
     from policyengine_api.data.v2.database import get_v2_session_factory
     from policyengine_api.services.v2.metadata.database_session import (
@@ -733,6 +759,7 @@ def _default_v2_metadata_reader_factory() -> V2MetadataResourceReader:
     return V2MetadataService(
         MetadataDatabaseSession(get_v2_session_factory()()),
         running_policyengine_version=_running_policyengine_version(),
+        catalog_cache=_metadata_catalog_cache(),
     )
 
 

policyengine_api/fastapi_routes/v2/metadata/catalog_routes.py

@@ -0,0 +1,107 @@
+"""API v2 route serving a whole metadata catalog version in one response."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from fastapi import APIRouter, Depends, Header
+from starlette.responses import JSONResponse, Response
+
+from policyengine_api.fastapi_routes.query_parameters import query_dependency
+from policyengine_api.fastapi_routes.v2.metadata.common import (
+    ERROR_RESPONSES,
+    run_metadata_read,
+)
+from policyengine_api.fastapi_routes.v2.metadata.response_models import (
+    MetadataCatalogResponse,
+)
+from policyengine_api.query_parameters import CatalogQuery
+
+if TYPE_CHECKING:
+    from policyengine_api.fastapi_routes.dependencies import (
+        NativeRouteDependencies,
+        V2MetadataResourceReader,
+    )
+    from policyengine_api.services.v2.metadata.types import MetadataCatalogDocument
+
+METADATA_CATALOG_FORMAT = 1
+"""Changes whenever the document's shape changes, so validators never match old bodies."""
+
+RUNNING_CATALOG_MAX_AGE_SECONDS = 300
+PINNED_CATALOG_MAX_AGE_SECONDS = 86_400
+
+
+def catalog_entity_tag(document: MetadataCatalogDocument) -> str:
+    """Return the quoted ETag naming a document's country, version and format."""
+    return (
+        f'"{document.country_id}:{document.policyengine_version}:'
+        f'{METADATA_CATALOG_FORMAT}"'
+    )
+
+
+def _matches(if_none_match: str | None, entity_tag: str) -> bool:
+    if if_none_match is None:
+        return False
+    candidates = {candidate.strip() for candidate in if_none_match.split(",")}
+    return "*" in candidates or bool(
+        {entity_tag, f"W/{entity_tag}"}.intersection(candidates)
+    )
+
+
+def catalog_response(
+    document: MetadataCatalogDocument,
+    *,
+    pinned: bool,
+    if_none_match: str | None,
+) -> Response:
+    """Serve the document, or 304 when the client already holds this version.
+
+    A request that names its catalog version may be cached for a day; one that
+    takes the running version is revalidated after five minutes, since the
+    running version changes when the API deploys a new PolicyEngine.py.
+    """
+    max_age = (
+        PINNED_CATALOG_MAX_AGE_SECONDS if pinned else RUNNING_CATALOG_MAX_AGE_SECONDS
+    )
+    entity_tag = catalog_entity_tag(document)
+    headers = {"ETag": entity_tag, "Cache-Control": f"public, max-age={max_age}"}
+    if _matches(if_none_match, entity_tag):
+        return Response(status_code=304, headers=headers)
+    return Response(
+        content=document.body, media_type="application/json", headers=headers
+    )
+
+
+def build_v2_model_catalog_router(
+    dependencies: NativeRouteDependencies,
+) -> APIRouter:
+    """Build the whole-catalog route; building it touches no database."""
+    router = APIRouter(prefix="/v2", responses=ERROR_RESPONSES)
+    catalog_query = query_dependency(CatalogQuery)
+
+    @router.get(
+        "/model-catalog",
+        response_model=MetadataCatalogResponse,
+        summary="Get every parameter node, parameter and variable of a catalog",
+        description=(
+            "Returns the selected PolicyEngine.py catalog's parameter nodes, "
+            "parameters with their current-law values, and variables in one "
+            "response, for clients that load the whole model at once. Name the "
+            "catalog with policyengine_version to make the response cacheable for "
+            "a day; the ETag supports conditional requests."
+        ),
+    )
+    def get_model_catalog(
+        query: CatalogQuery = Depends(catalog_query),
+        if_none_match: str | None = Header(default=None),
+    ) -> Response | JSONResponse:
+        def read(reader: V2MetadataResourceReader) -> Response:
+            return catalog_response(
+                reader.get_model_catalog(query.country_id, query.policyengine_version),
+                pinned=query.policyengine_version is not None,
+                if_none_match=if_none_match,
+            )
+
+        return run_metadata_read(dependencies, read)
+
+    return router

policyengine_api/fastapi_routes/v2/metadata/common.py

@@ -62,6 +62,7 @@
 
 
 ResponseT = TypeVar("ResponseT", bound=BaseModel)
+ReadT = TypeVar("ReadT")
 
 
 def read_resource(
@@ -75,6 +76,20 @@ def read_resource(
     (400, 404 or 503, and 500 for anything unexpected), and the reader is
     closed whatever happens.
     """
+    return run_metadata_read(
+        dependencies, lambda reader: response_type(result=operation(reader))
+    )
+
+
+def run_metadata_read(
+    dependencies: NativeRouteDependencies,
+    operation: Callable[[V2MetadataResourceReader], ReadT],
+) -> ReadT | JSONResponse:
+    """Run one metadata read and return its result unchanged.
+
+    Failures become the typed v2 error responses described by read_resource,
+    and the reader is closed whatever happens.
+    """
     reader = None
     try:
         factory = dependencies.v2_metadata_reader_factory
@@ -85,7 +100,7 @@ def read_resource(
 
             factory = _default_v2_metadata_reader_factory
         reader = factory()
-        return response_type(result=operation(reader))
+        return operation(reader)
     except (InvalidMetadataPageError, InvalidPolicyEngineVersionError) as error:
         return v2_error_response(400, str(error))
     except UnsupportedPreviewCountryError as error:

policyengine_api/fastapi_routes/v2/metadata/response_models.py

@@ -6,6 +6,7 @@
 
 from policyengine_api.services.v2.metadata.types import (
     MetadataCanonicalParameterValue,
+    MetadataCatalogResult,
     MetadataDataset,
     MetadataDetailResult,
     MetadataEconomyOptionsResult,
@@ -139,3 +140,7 @@ class MetadataEconomyOptionsResponse(
     MetadataResourceSuccessResponse[MetadataEconomyOptionsResult]
 ):
     """Success response holding the options for configuring an economy comparison."""
+
+
+class MetadataCatalogResponse(MetadataResourceSuccessResponse[MetadataCatalogResult]):
+    """Success response holding a whole catalog version's parameters and variables."""

policyengine_api/fastapi_routes/v2/routes.py

@@ -12,6 +12,9 @@
     build_v2_household_router,
 )
 from policyengine_api.fastapi_routes.v2.me.routes import build_v2_me_router
+from policyengine_api.fastapi_routes.v2.metadata.catalog_routes import (
+    build_v2_model_catalog_router,
+)
 from policyengine_api.fastapi_routes.v2.metadata.geography_routes import (
     build_v2_metadata_geography_router,
 )
@@ -56,6 +59,7 @@ def build_v2_router(
     router.include_router(build_v2_metadata_model_router(dependencies))
     router.include_router(build_v2_metadata_parameter_router(dependencies))
     router.include_router(build_v2_metadata_geography_router(dependencies))
+    router.include_router(build_v2_model_catalog_router(dependencies))
 
     @router.get(
         "/v2/openapi.json",

policyengine_api/migration_registry.py

@@ -39,6 +39,7 @@ class RouteGroupConfig:
             "metadata",
             "datasets",
             "economy-options",
+            "model-catalog",
             "parameter-values",
             "parameters",
             "regions",