You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Raw-artifact scans silently skip manifests not named manifest.yaml #337
chronicle publish-raw, chronicle verify-raw and the artifact inventory (inventory_source_artifacts) scan for files named exactly manifest.yaml by default (chronicle/artifacts.py, manifest_filename="manifest.yaml"). Several source folders use other names, and a default scan of db/data silently skips them:
at least five other folders on main hold a manifest named other than manifest.yaml: census/pep_2024_age_sex, cms_nhe/historical_service_source, irs_soi/ira_contributions, irs_soi/w2_statistics and usda_snap/fy69_to_current.
The review of #335 measured the effect. inventory_source_artifacts('db/data/stats_nz') finds 4 manifests and none of the 18 census ones. With manifest_filename='manifest-*.yaml' it finds all 18, with 0 checksum mismatches. So a documented verify-raw --root db/data run reports success while never reading those objects back.
Options
Default to a pattern such as manifest*.yaml in all three commands. That changes what publish-raw uploads, and what the inventory counts, for the folders above. Some of their extra files may not be manifests in the usual sense (for example sidecar or legacy manifests), so each needs checking before the default changes.
Keep the default, but report skips. Have each scan count and list YAML files whose names start with manifest but that it skipped, and warn, or fail in verify-raw, when any exist.
Standardise the names, so every folder has one manifest.yaml. Packages that need several artifacts in one folder would need a supported multi-manifest convention instead.
Option 2 is the smallest safe step and makes silent skips visible. Options 1 and 3 need the per-folder check first.
Found while landing the NZ Census package (#335). Related: #332 and #334 (verify-raw).
Activity