Skip to content

Raw-artifact scans silently skip manifests not named manifest.yaml #337

Open

Description

@MaxGhenis

Problem

chronicle publish-raw, chronicle verify-raw and the artifact inventory (inventory_source_artifacts) scan for files named exactly manifest.yaml by default (chronicle/artifacts.py, manifest_filename="manifest.yaml"). Several source folders use other names, and a default scan of db/data silently skips them:

  • db/data/stats_nz/census_2023_place_summaries/: 18 per-place manifests, manifest-nz.yaml and manifest-rc-XX.yaml (Add Stats NZ 2023 Census household and family counts by regional council #335);
  • at least five other folders on main hold a manifest named other than manifest.yaml: census/pep_2024_age_sex, cms_nhe/historical_service_source, irs_soi/ira_contributions, irs_soi/w2_statistics and usda_snap/fy69_to_current.

The review of #335 measured the effect. inventory_source_artifacts('db/data/stats_nz') finds 4 manifests and none of the 18 census ones. With manifest_filename='manifest-*.yaml' it finds all 18, with 0 checksum mismatches. So a documented verify-raw --root db/data run reports success while never reading those objects back.

Options

  1. Default to a pattern such as manifest*.yaml in all three commands. That changes what publish-raw uploads, and what the inventory counts, for the folders above. Some of their extra files may not be manifests in the usual sense (for example sidecar or legacy manifests), so each needs checking before the default changes.
  2. Keep the default, but report skips. Have each scan count and list YAML files whose names start with manifest but that it skipped, and warn, or fail in verify-raw, when any exist.
  3. Standardise the names, so every folder has one manifest.yaml. Packages that need several artifacts in one folder would need a supported multi-manifest convention instead.

Option 2 is the smallest safe step and makes silent skips visible. Options 1 and 3 need the per-folder check first.

Found while landing the NZ Census package (#335). Related: #332 and #334 (verify-raw).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions