Partial eval-cache load appends regenerated batches to stale ones (ragged per-rank eval)

[xy200303]

setup_eval loads the per-rank eval cache and regenerates it when any rank got nothing:

https://github.com/Tencent/AngelSpec/blob/main/angelspec/controller/eval.py#L195-L207

loaded = train_group.load_eval_cache(eval_cache_path)
if all(n > 0 for n in loaded):
    eval_cache_loaded = True
    ...
else:
    ...  # regenerate cache from inference

The cache save is fire-and-forget (async_save_eval_cache, one eval_rank_<r>.pt file per rank), so an interrupted previous run can leave a partial cache at the same dp_size: some ranks load n > 0, others 0. On the regeneration path, cache_eval_samples appends to whatever was loaded:

https://github.com/Tencent/AngelSpec/blob/main/angelspec/training/trainer.py#L363-L369

def cache_eval_samples(self, count: int) -> int:
    for sample in itertools.islice(self._eval_data_fetcher, count):
        ...
        self._eval_cache.append(cpu_sample)

Ranks that loaded stale shards end up with stale batches + freshly generated ones, so per-rank eval batch counts diverge. This is exactly the failure mode the comment at eval.py L182-L186 warns about ("ragged per-rank eval batch counts -> desynced FSDP all-gathers -> NCCL deadlock") — the dp_size cache key guards the cross-dp_size case but not the partial-save case at the same dp_size. Even without a deadlock, eval silently runs on a mix of stale and fresh hidden states.

Suggested fix: when the load is partial (regeneration path), explicitly clear every rank's loaded eval cache before regenerating, or have load_eval_cache roll back on partial loads.