Add drift-detection tooling (check-drift) to the release repo

[ideaship]

What

Adds check-drift.py, a plugin-based drift detector for OSISM's image and
version wiring, and wires it into CI. It catches divergence — when it is
introduced — before it breaks a deploy or a gate, across two check groups
(--group image | kolla | all).

Image version pins — follows an image's version down the chain
release base.yml → rendered manager images.yml → role defaults:

• release_vs_manager — the rendered manager images.yml disagrees with the release pin
• role_shadows — a role-default tag shadows the release pin (LIVE = the role default actually deploys; DORMANT = the release pin wins)
• role_unpinned — a role-default tag with no release pin
• image_orphan — a manager image alias is emitted but consumed by no role or playbook

Kolla container-image toolchain — divergence between the OSISM consumer repos
(defaults, release, generics, the container-image repos) and upstream
openstack/kolla / openstack/kolla-ansible:

• kolla_enablement_orphan — OSISM enables a service upstream no longer defines
• kolla_groupvars_missing — an upstream group_vars/all var OSISM's defaults never mirrored, so it is undefined at deploy/upgrade (the add-direction mirror of kolla_enablement_orphan; the class behind the 2025.2 keystone_listen_port break)
• kolla_orphan_config — dead companion vars / image blocks for an orphaned service
• kolla_secrets_orphan — an OSISM kolla secret with no upstream counterpart
• kolla_enablement_build — an enabled, upstream-buildable service missing from the build set
• kolla_version_chain_upstream — an upstream docker/ service with no version-pin line
• kolla_version_chain_inner — a referenced version pin that resolves to nothing (inert pin)
• kolla_inventory — an upstream inventory group missing from the OSISM 50/51-kolla files

The framework lands first (empty registry, flags nothing on its own), then the
plugins in lifecycle order, each reviewable in isolation. Sources resolve
remotely (GitHub) by default, so the detector runs anywhere including CI;
--base-dir reads local checkouts. An allowlist marks intentional exceptions
(each requiring a reason); entries that match nothing are reported stale, so
suppressions can't silently rot. Exit codes: 0 clean / 1 drift or stale / 2 input
error. Full reference: docs/check-drift-image.md, docs/check-drift-kolla.md
(incl. the osism/defaults release/bake model and why the range-aware checks
compare against the union of supported releases).

Commits (each independently green — unit tests pass at every commit)

• drift: add plugin framework and driver
• drift: add image-drift plugins
• drift: add kolla-drift plugins
• drift: add drift-detection docs
• drift: wire tox and Zuul CI
• kolla: read enable flags from all defaults files — make the enable-flag readers layout-agnostic (read enable_* from the union of all/*.yml, not just 099-kolla.yml), so a defaults reorganization can't silently drop a flag
• docs: explain defaults bake model and unions
• kolla: add kolla_groupvars_missing drift check — the add-direction group_vars check (both OSISM delivery paths counted: osism/defaults + the rendered container-image-kolla-ansible versions.yml)
• kolla: allowlist eutron_external_interface typo

CI

• release-tox-drift-test — the unit suite (tox -e drift-test, 248 tests).
  Runs in check (gates this PR) and periodic-daily.
• release-tox-drift — the detector (tox -e drift, check-drift.py --group all). Runs periodic-daily only: it reads live upstream state, so
  it's unfit as a PR gate. It exits non-zero when drift exists, so the daily job
  goes red with the report in its log.

What this finds in practice

Run against the live OSISM repos today, the detector reports 73 actionable
items, 42 allowlisted, 0 stale (115 total):

• image pins — 1 rendered-vs-release disagreement (ara_server), 3 LIVE
  role-default shadows that actually deploy (dnsmasq, scaphandre, stepca),
  9 DORMANT lags, 4 unpinned tags (httpd, httpd_data, kepler,
  thanos_sidecar), and 3 orphaned manager aliases (nginx, osism_netbox,
  registry).
• kolla toolchain — now surfaces real drift (it was reported clean before
  this branch). The layout-agnostic enable-flag reader scans enable_* across
  all osism/defaults all/*.yml files (not just 099-kolla.yml), surfacing
  9 enablement orphans (auditd, chrony, openstack_exporter, karbor,
  qinling, searchlight, three horizon_*) and their 9 dead companion vars
  — some are removed-upstream projects to drop, some are OSISM inventions to
  allowlist. The new add-direction kolla_groupvars_missing surfaces 35
  upstream group_vars/all keys OSISM never mirrored — the osism/defaults
  re-sync backlog behind the 2025.2 keystone_listen_port upgrade break, incl.
  enable_ironic_pxe_filter and the nova_database_*/nova_cell0_database_*
  batch. eutron_external_interface is allowlisted (upstream typo for
  neutron_external_interface, awaiting
  https://review.opendev.org/c/openstack/kolla-ansible/+/985031).

Earlier, the removal-direction kolla checks already drove a wave of cleanups
across the consumer repos (kolla actionable-drift 81 → 0 at the time, all merged);
this branch's layout-agnostic reader and new add-direction check now surface a
further backlog — chiefly the osism/defaults re-sync the keystone_listen_port
break called for:

• Remove orphaned kolla service config defaults#291
• Reconcile versions template with kolla images container-image-kolla-ansible#924
• Build valkey image in 2025.1 and 2025.2 #2551
• Pin valkey via SBOM_IMAGE_TO_VERSION container-images-kolla#745
• inventory: add neutron 2025.2 service groups generics#599
• inventory: add ironic-dnsmasq service group generics#601

It self-audits its allowlist too: an entry that no longer suppresses real drift is
a hard error (exit 1), so suppressions can't silently rot.

Because the detector is daily-only, this output does not appear in this PR's
checks. Full release-tox-drift report against current upstream
(check-drift.py --group all):

Checks follow an image's version pin: release base.yml → rendered manager images.yml → role defaults.

release_vs_manager — 1 image tags in the rendered manager images.yml
disagree with the release base.yml pins:

    ara_server

  Fix: re-render environments/manager/images.yml from the current release,
       or allowlist the entry if the divergence is intentional (e.g. a
       rolling-release image pinned to 'latest').
  Refs: release/latest/base.yml
        testbed/environments/manager/images.yml

role_shadows — 9 DORMANT — overridden by the rendered images.yml; the
release pin wins at deploy:

    adminer_tag (4.7 → 5.4.2)   roles/adminer/defaults/main.yml
    ara_server_mariadb_tag (11.8.3 → 11.8.4)   roles/manager/defaults/main.yml
    manager_redis_tag (7.4.6-alpine → 7.4.7-alpine)   roles/manager/defaults/main.yml
    vault_tag (1.21.0 → 1.21.4)   roles/manager/defaults/main.yml
    netbox_redis_tag (7.4.6-alpine → 7.4.7-alpine)   roles/netbox/defaults/main.yml
    postgres_tag (16.10-alpine → 16.11-alpine)   roles/netbox/defaults/main.yml
    nexus_tag (3.82.0 → 3.93.1)   roles/nexus/defaults/main.yml
    phpmyadmin_tag (5.2 → 5.2.3)   roles/phpmyadmin/defaults/main.yml
    traefik_tag (v3.4.4 → v3.5.0)   roles/traefik/defaults/main.yml

  Fix: lower priority; sync when convenient.
  Refs: release/latest/base.yml

role_shadows — 3 LIVE — no images.yml override; the role default is what
actually deploys:

    dnsmasq_tag (2.90 → 2.91)   roles/dnsmasq/defaults/main.yml
    scaphandre_tag (1.0.0 → 1.0.2)   roles/scaphandre/defaults/main.yml
    stepca_tag (0.28.4 → 0.30.2)   roles/stepca/defaults/main.yml

  Fix: add `<alias>_tag`/`<alias>_image` to the manager render template
       (images.yml.j2) so the latest/base.yml pin governs the deployed
       version.
  Refs: release/latest/base.yml

role_unpinned — 4 <alias>_tag pins in role defaults with no release base.yml
pin:

    httpd_tag (alpine, no release pin)   roles/httpd/defaults/main.yml
    httpd_data_tag (latest, no release pin)   roles/httpd/defaults/main.yml
    kepler_tag (v0.11.2, no release pin)   roles/kepler/defaults/main.yml
    thanos_sidecar_tag (v0.32.5, no release pin)   roles/thanos_sidecar/defaults/main.yml

  Fix: add a pin to release base.yml (and wire <alias>_tag into the manager
       render template) to make it release-managed, or allowlist it if the
       image is intentionally role-managed.
  Refs: release/latest/base.yml

image_orphan — 3 image vars emitted by the manager images.yml that no role
or playbook consumes:

    nginx, osism_netbox, registry

  Fix: remove the orphaned <alias>_tag/<alias>_image (and its etc/images.yml
       + role-default remnants), or allowlist it if the image is genuinely
       consumed in a form this scan misses — it flags an alias only when no
       literal {{ <alias>_image }} reference is found in the scanned
       roles/playbooks.
  Refs: ansible-collection-services/roles/, ansible-playbooks-manager/playbooks/
        generics/environments/manager/images.yml

Checks follow a service's path: enabled → built → version-pinned → deployed.

kolla_enablement_orphan — 9 OSISM enable flags whose service upstream
kolla-ansible no longer defines at any supported release; the service was
removed or renamed upstream, leaving the flag orphaned:

    auditd, chrony, horizon_karbor, horizon_qinling, horizon_searchlight,
    karbor, openstack_exporter, qinling, searchlight

  Fix: remove the stale enable_<name> from osism/defaults, or migrate it to
       the upstream replacement. Some flags (e.g. common, kolla_operations)
       are OSISM inventions with no upstream counterpart — keep those
       allowlisted rather than removed.
  Refs: openstack/kolla-ansible group_vars enable-defaults @ supported refs
        osism/defaults all/*.yml

kolla_groupvars_missing — 35 upstream kolla-ansible group_vars OSISM
defaults never mirrored; each is undefined in the deploy var context and
aborts any role task that uses it:

    ceph_cinder_backup_keyring, ceph_cinder_keyring, ceph_glance_keyring,
    ceph_gnocchi_keyring, ceph_manila_keyring, ceph_nova_keyring,
    distro_python_version_map, enable_cinder_backend_hnas_nfs,
    enable_ironic_pxe_filter, enable_swift_recon, enable_swift_s3api,
    glance_backend_swift, keystone_listen_port, nova_cell0_database_address,
    nova_cell0_database_name, nova_cell0_database_password,
    nova_cell0_database_shard_id, nova_cell0_database_user,
    nova_database_address, nova_database_name, nova_database_user,
    prometheus_msteams_port, prometheus_msteams_webhook_url,
    swift_account_server_port, swift_container_server_port,
    swift_external_fqdn, swift_internal_base_endpoint,
    swift_internal_endpoint, swift_internal_fqdn, swift_object_server_port,
    swift_proxy_server_listen_port, swift_proxy_server_port,
    swift_public_base_endpoint, swift_public_endpoint, swift_rsync_port

  Fix: mirror the missing var into osism/defaults all/*.yml (copying
       upstream's definition — harmless when the service is off, needed when
       an environment enables it), or allowlist it with a reason if OSISM
       deliberately omits it: the related service is not shipped/supported
       by OSISM at all (so the var is never evaluated anywhere), a var OSISM
       supplies another way, or an upstream typo.
  Refs: openstack/kolla-ansible group_vars/all @ supported refs
        osism/defaults all/*.yml

kolla_orphan_config — 2 dead companion/image config vars for services
upstream removed (their enable_<name> flag is reported separately by
kolla_enablement_orphan); these must be removed too:

    chrony_image, chrony_tag

  Fix: remove these vars from the listed osism/defaults file, or allowlist
       any that are intentionally kept (an OSISM invention with no upstream
       service).
  Refs: openstack/kolla-ansible (service removed upstream)
        osism/defaults all/002-images-kolla.yml

kolla_orphan_config — 2 dead companion/image config vars for services
upstream removed (their enable_<name> flag is reported separately by
kolla_enablement_orphan); these must be removed too:

    chrony_allowed_subnets, chrony_servers

  Fix: remove these vars from the listed osism/defaults file, or allowlist
       any that are intentionally kept (an OSISM invention with no upstream
       service).
  Refs: openstack/kolla-ansible (service removed upstream)
        osism/defaults all/099-generic.yml

kolla_orphan_config — 5 dead companion/image config vars for services
upstream removed (their enable_<name> flag is reported separately by
kolla_enablement_orphan); these must be removed too:

    openstack_exporter_clouds_yml_cloud, openstack_exporter_clouds_yml_path,
    openstack_exporter_secure_yml_path,
    openstack_exporter_service_monitor_scrape_interval,
    openstack_exporter_service_monitor_scrape_timeout

  Fix: remove these vars from the listed osism/defaults file, or allowlist
       any that are intentionally kept (an OSISM invention with no upstream
       service).
  Refs: openstack/kolla-ansible (service removed upstream)
        osism/defaults all/099-infrastructure.yml

Summary: 73 to act on, 42 allowlisted, 0 stale allowlist entries (115 total)

How to read it

• image pins — DORMANT role_shadows lag but the release pin governs at
  deploy (low priority); LIVE role_shadows (dnsmasq, scaphandre, stepca)
  actually deploy the role default (wire into images.yml.j2); role_unpinned
  (httpd*, kepler, thanos_sidecar) have no release pin; image_orphan
  (nginx, osism_netbox, registry) are emitted but consumed nowhere.
• kolla removal-direction — kolla_enablement_orphan + kolla_orphan_config:
  stale enable_* flags and their companion vars for services upstream dropped;
  drop the removed projects, allowlist the OSISM inventions.
• kolla add-direction — kolla_groupvars_missing: the osism/defaults
  re-sync backlog. The right fix for most is to mirror the upstream var into
  osism/defaults (harmless when the service is off, correct when an environment
  enables it — e.g. metalbox turns on ironic, so enable_ironic_pxe_filter bites
  there); allowlist only genuine divergences (the eutron typo).

The detector runs unauthenticated against GitHub (no token support in
source.py), same as generics' existing daily job. A heavy run can occasionally
hit API rate limits and fail without a report; adding GITHUB_TOKEN support is a
sensible follow-up.