TIER IV universe / split mirror (planning)

This repository split mirrors the planning subtrees of several upstream repositories, used for TIER IV release workflows.

This branch only holds the mirror configuration and its tooling. See the mirror branches for source code.

Published branches

Branch	Contents
awf-latest/universe	autowarefoundation/autoware_universe:main, planning paths
awf-latest/core	autowarefoundation/autoware_core:main, planning paths
feat/v0.64/e2e_scenario_pass	tier4/autoware_universe:feat/v0.64/e2e_scenario_pass, planning paths
feat/v0.64/e2e_scenario_pass-core	tier4/autoware_core:feat/v0.64/e2e_scenario_pass, planning paths
tier4-e2e-combined-latest	feat/v0.64/e2e_scenario_pass and feat/v0.64/e2e_scenario_pass-core replayed into one linear history (no launch)

The combined branch treats the tier4 feat/v0.64/e2e_scenario_pass branches of autoware_universe and autoware_core as the trusted histories (not AWF main). Planning does not mirror launch; root files come from the universe e2e mirror.

tier4-e2e-combined-latest/
├── universe/                   # from feat/v0.64/e2e_scenario_pass (tier4/autoware_universe)
├── core/                       # from feat/v0.64/e2e_scenario_pass-core (tier4/autoware_core)
└── .github/  docs/  LICENSE  NOTICE  README.md  ...

The upstream mirrors live under the awf-latest/ namespace; the combined branch is derived from them rather than from an upstream, so it sits outside it. Note that git cannot hold a branch named awf-latest at the same time as awf-latest/*, so the old flat branch has to be deleted before these can be created.

How it works

.sync/sources.yaml is the single source of truth. It describes every upstream, the paths to retain, the commit-message rewriting and the branch each mirror is published to. .github/workflows/mirror.yaml derives its job matrices from that file, so adding or changing a mirror is a configuration change and never a workflow change.

The pipeline has two stages:

1. Filter. tools/mirror.py mirror SOURCE clones the upstream and runs git-filter-repo with arguments generated from the configuration, then pushes the result to that source's mirror_branch.
2. Combine. tools/mirror.py combine TARGET reads the mirror branches that stage 1 published and appends any not-yet-reflected member commits onto the already published combined tip (or builds from scratch on first publish). It never clones an upstream, so the combined branch cannot disagree with the per-source mirrors.

Determinism and publishing

Per-source mirrors are pure functions of (upstream commit, .sync/sources.yaml):

• git-filter-repo rewrites a given history the same way every time. The version is pinned in the workflow, because a different version may rewrite differently.
• The same upstream tip therefore republishes as a fast-forward. A failed push means reproducibility was lost.

The combined branch is the pure function f(published tip, member tips):

• Member commits not yet reflected in the published tip are appended onto it. The resume point is recovered from the tip tree (each member's renamed subtree oids). No sidecar ref is required.
• Same published tip plus same member tips always yield the same commit ids, so the push is a fast-forward (or unchanged).
• tools/mirror.py combine --verify checks that the tip tree matches what the current member tips compose to, and that a second build from the same inputs reproduces the commit id. The scheduled workflow always passes --verify.

Working on the configuration

python3 -m pip install pyyaml git-filter-repo==2.47.0

tools/sync_config.py validate              # check the configuration
tools/sync_config.py show autoware_core    # the git-filter-repo call it implies
tools/mirror.py combine awf-combined-latest \
    --work /tmp/mirror --downstream https://github.com/OWNER/REPO.git --verify

Without --push, both subcommands are dry runs.

Manual sync

Publishing is fast-forward only (never --force). This is the local equivalent of .github/workflows/mirror.yaml.

git checkout ci && git pull origin ci
python3 -m pip install pyyaml 'git-filter-repo==2.47.0'

PUSH=0 ./tools/manual_sync.sh   # dry run (build only)
PUSH=1 ./tools/manual_sync.sh   # push to origin

Variable	Default	Meaning
PUSH	1	Set 0 for a dry run
DOWNSTREAM	git@github.com:tier4/pilot_auto_planning.git	Git remote to push
WORK	/tmp/pilot_auto_planning-sync	Scratch directory for clones

To refresh one source only:

export PYTHONPATH="$PWD/tools${PYTHONPATH:+:$PYTHONPATH}"
tools/mirror.py mirror SOURCE \
  --work "/tmp/pilot_auto_planning-sync/mirror" \
  --downstream "git@github.com:tier4/pilot_auto_planning.git" \
  --push

Replace SOURCE with a name from tools/mirror.py list-sources. For a combined target, use tools/mirror.py combine TARGET ... --verify --push after its member mirrors are published.