Skip to content

refactor(dataset): unify memory dataset post-processing - #2140

Merged
Yunnglin merged 1 commit into
mainfrom
refactor/dataset-loader-pipeline
Nov 19, 2026
Merged

Yunnglin merged 1 commit into
mainfrom
refactor/dataset-loader-pipeline

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Summary

  • Centralize record conversion, sample filtering, repetition, reindexing, and choice shuffling in one shared memory-dataset builder.
  • Route the remote, local, and dictionary loaders plus build_dataset_from_records through the same post-processing pipeline.
  • Preserve source-specific loading and shuffling, existing limit resolution and validation, and the public repeat_samples import path.

This is a behavior-preserving follow-up to #2122, intended to prevent the loader entry points from drifting into different stage ordering again.

Test plan

  • 212 dataset contract tests covering loaders, builders, limits, filters, repeats, zero limits, and shuffle determinism.
  • 63 MCP-Atlas, OpenAI MRCR, and generated-media consumer tests.
  • pytest tests/cli/test_all.py::TestRun::test_ci_lite -q -p no:warnings
  • Ruff check and formatting on both changed files.
  • make lint-imports — 5 contracts kept.

Centralize conversion, filtering, repetition, indexing, and choice shuffling so every loader and the record builder share one pipeline.
@github-actions github-actions Bot added area:core Framework and infrastructure needs-triage Waiting for a maintainer to triage scope and evidence labels Nov 19, 2026
@github-actions

Copy link
Copy Markdown

EvalScope AI review

Findings to verify · 7635b40958a6 · 2026-11-19 09:48:48 UTC · Actions run

  • Test Gap · evalscope/api/dataset/loader.py:140 — The refactor funnels four separate loading paths (`_to_memory_dataset`, `LocalDataLoader.load`, `DictDataLoader.load`, and `build_dataset_from_records`) through the new `_build_memory_dataset` helper, yet no tests are added or updated. Subtle default/ordering differences now affect all paths at once, e.g. the helper explicitly passes `shuffled=False` where some loaders previously omitted the argument, and it always invokes `shuffle_choices_if_requested` even in the record-builder path that never did so. A regression test exercising filter + limit + repeats + auto_id on at least the file and dict loaders would make this extraction safe.

@github-actions github-actions Bot added the ai:findings AI review found actionable items for a maintainer to verify label Nov 19, 2026
@Yunnglin
Yunnglin merged commit e3fd1f4 into main Nov 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:findings AI review found actionable items for a maintainer to verify area:core Framework and infrastructure needs-triage Waiting for a maintainer to triage scope and evidence

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant