test(tenant): diagnose data mount visibility after mount-unit restart #117

Context

The disposable Docker/systemd tenant-scenario intermittently fails after online volume growth and a mount-unit restart. This is tracked separately from #93 because the tenant implementation, mount-unit generator, scenario and workflow are unchanged in 79f1dc5..da2d39d.

Observed in the failed tenant job: quota refusal, 100M-to-300M online growth, a successful 150M write, unchanged container start time and online shrink refusal all pass. systemctl restart srv-tenants-t1-vol.mount then returns success, but docker start tenant-t1 refuses the missing bind source /srv/tenants/t1/vol/data.

The green aggregate on 79f1dc5 did not run this scenario (job 129375513672 was skipped). A genuinely successful prior scenario is job 129160563231 on 63cb803; its tenant code and scenario are also unchanged. Both jobs reported runner image 20261101.360.1.

Reproduction and diagnosis

Run ROOTFILES_TEST_BINARY=<Linux binary> bash tests/scenarios/tenant-lifecycle.sh only on a disposable Docker/systemd test host. It installs test configuration, creates loop-backed tenant volumes and containers, and changes test service units.

After the mount-unit restart and before starting the existing container, distinguish:

• The host sees a bare mountpoint or missing data directory: inspect mount-unit state/journal, findmnt, and loop backing.
• The host sees the mounted filesystem but dockerd does not: inspect dockerd's mount namespace and mount propagation.

The #93 follow-up adds fail-closed mount/data checks and host/dockerd namespace diagnostics. It does not create a replacement data directory or change production mount options without evidence. No data deletion has been demonstrated by the current logs.

Acceptance criteria

• Reproduce and identify the remount or namespace failure with the new diagnostics.
• Preserve existing tenant image contents; never format or populate a bare mountpoint as a workaround.
• Make the disposable lifecycle scenario reliably verify data after the restart.
• Add focused regression coverage for the identified cause; keep GPU-host rollout validation separate.