Skip to content

[VARKA-617] Replay a matrix run's bins against its own measured phase - #1632

Open
MaxGekk wants to merge 2 commits into
vecbricks:masterfrom
MaxGekk:varka-617-matrix-planner-check-r
Open

MaxGekk wants to merge 2 commits into
vecbricks:masterfrom
MaxGekk:varka-617-matrix-planner-check-r

Conversation

@MaxGekk

@MaxGekk MaxGekk commented Nov 22, 2026

Copy link
Copy Markdown

What changes were proposed in this pull request?

The option matrix's planner packs each configuration's suites into bin jobs by per-pair seconds.
Until now, nothing checked the planner against the run it planned.

  • dev/varka_matrix_bins.py replay prints a finished run's measured bin phase beside three replays.
  • The two pair files are rewritten from runs 43642146943 and 44485292430: 10,846 pairs, where October's file held 5,718.

On run 43642146943, "as run" is 32.5 minutes against the measured 34.4, and replanned on its own pairs 30.4.

The three replays, the measured table and the pair refresh

The option matrix (varka-option-matrix.yml) runs every Varka suite under every emit-option configuration, split by dev/varka_matrix_bins.py plan into bin jobs of two lanes of about 480 s each.
The planner weighs each (configuration, suite) pair by the seconds it took in an earlier run, from sql/varka/matrix/config-suite-seconds.tsv.
The only earlier replay was a one-off from October, timed at another run's bin count and slot count.
It cannot say whether the planner is right about its own run.

dev/varka_matrix_bins.py replay RUN_DIR --jobs JOBS --suites MODULE=FILE... reads one run's downloaded bin artifacts (gh run download) and its jobs from the Actions API.
It prints the run's measured bin phase (the first bin job's start to the last one's end) beside three replays.
Each replay starts the jobs in bin order on the run's slots (the most bin jobs it ran at once, 38).
Each job starts after the run's median wait for a runner once its slot is free.

  • jobs as run: the run's own jobs at their measured lengths. This tests the slot model alone.
  • as run: each of the run's own bins as its measured set-up and upload around its units. The units run at their measured seconds on two lanes, in the order the job started them. This tests the bin model too.
  • replanned: the planner, rerun over the run's configurations, suites and reach record with the pair files given (--plan-from REF, or --pairs and --suite-seconds). Each new unit is timed at the run's measured JUnit seconds of its suites plus the run's median JVM overhead. This tests the planner.

The pair files are rewritten from two fresh runs' report artifacts: 43642146943 (14 November, with a reach record) and 44485292430 (19 November, master, every configuration running every test).
config-suite-seconds.tsv goes from October's 5,718 pairs to 10,846, the mean where both runs hold a pair.
suite-seconds.tsv comes from the master run.

The replay was checked before anything was scored with it.
Given the run's configurations, the suites its build listed and its reach record, the planner at the run's head cuts run 43642146943's 155 bins again, with the same units in the same order (73,197 s planned, as the run's own summary line says).
It cuts run 44485292430's 190 bins the same way.

Measured (sql/varka/plans/m8/results/VARKA-617-replay.txt):

run measured jobs as run as run replanned, own pairs replanned, this PR's files
43642146943 34.4 min 32.8 32.5 30.4 (172 bins, 3 over 600 s) 31.0 (175 bins)
44485292430 45.6 min 45.6 45.2 37.2 (217 bins, 0 over 600 s) 38.0 (216 bins)

The model comes within 5.5 per cent of its own run.
Most of the gap is in the slot model: jobs start slightly out of bin order, and one runner wait reached about four minutes.
On the run's own pairs, the replanned phase is within a fifth of the measured one.
That figure is in sample by construction, so the out-of-sample check is the next matrix run on master.

Why are the changes needed?

The pair file was from 30 October, so 51 suites had no pair.
An earlier replay was 22 per cent under its run, and nothing could say if the planner or the file was at fault.

How stale the pairs were

Over the pairs both files hold, a fresh run's seconds range from 0.65 to 1.65 times the committed ones (the middle 80 per cent).
The planner packs bins by these seconds, so a pair at half or twice its real weight makes a bin half or twice its plan.
Before this change there was also no way to tell whether a gap between a plan and its run comes from the weights or from the model of how jobs run.
The replay separates the two, and the refreshed pairs are what the planner plans from.

Does this PR introduce any user-facing change?

No. This is a CI planning script and its data files, and Varka is not in any released Spark.

How was this patch tested?

On the laptop, on head 9032bbbb3d1 (master b4cbee926d1 merged, no conflict):

  • Regeneration, push check, extra gate, the 32 self-tests and ruff passed.
  • plan runs as the workflow calls it. This is a parse check only.
  • No sbt suite is owed, because no JVM code reads the pair files.
  • Not rerun on this head: the replays. Their figures are from 20 November.
  • Not run: a CI Build or an option matrix with the new pairs.
Full test record and what was not run

On the laptop, one sbt at a time, SPARK_LOCAL_IP=127.0.0.1, on 22 November 2026, at head 9032bbbb3d1.
That head is the branch merged with origin/master at b4cbee926d1 (the merge had no conflict).
Passed there:

  • dev/varka_regen_generated.sh --all. It left nothing to commit.
  • python3 dev/varka_push_check.py: precommit range check, plan lint, quote check, census check and size.
  • ~/varka-ops/extra-gate.sh: it passed, and no step applied, since no Java, Scala or catalyst file changed.
  • python3 dev/varka_matrix_bins.py --selftest: 32 tests pass. Two are new. One checks that "as run" and "jobs as run" over a two-bin, one-slot fixture equal its measured 98 s. The other checks that a fixture laid out as the planner cut it, with its pairs read back by whole_runs, replans to its own bins and phase.
  • ruff check and ruff format --check (0.14.8) on dev/varka_matrix_bins.py.
  • dev/varka_matrix.sh --list-suites for both modules, bash -n dev/varka_matrix.sh, and plan with --platforms as the workflow's "Plan the bins" step calls it. It exits 0 and writes valid JSON. Its configuration list was taken from the pair file, not from a build, so its bin count is not comparable with the figures above.
  • The modules that import the planner (dev/varka_lint.py, dev/varka_matrix_report.py) import it.

The branch does not change any file under .github/workflows, and I did not parse the workflow YAML, because no local Python has PyYAML.
The suite VarkaTestReachSuite shares a reach fixture with the self-test, and this branch does not change that fixture or the reach reader.

Run on 20 November on the earlier head and not repeated, because the run artifacts are not on the laptop now:

  • The replays of runs 43642146943 and 44485292430, as recorded in sql/varka/plans/m8/results/VARKA-617-replay.txt.
  • times --reach --per-configuration over the downloaded 43642146943 tree. It reproduced the report artifact's 8,392 pairs and suite seconds exactly.
  • plan over master's configurations on both file versions: 190 bins on October's pairs, 216 on the fresh ones.

Not run:

  • The Build's Python lint job (ruff over dev/), which runs on push.
  • One option matrix dispatch on master after this PR and its follow-up (VARKA-617 step 5, on the owner's word). It is the out-of-sample score: its bin phase against the replanned figure for its own pairs. It also confirms that the rewritten pair file parses into the CI planner.

Review

/code-review high found five nits and no important finding.
The changed lines did not change after the merge, so I did not rerun it.
The reviewed SHA is not recorded.
With the new pairs, the planner cuts 216 bins on master's configurations, against 190 before.
The size is 466 counted lines.

The nit list and the review record

/code-review high ran on the branch before its rebase onto 0e5c1657efe and the merge of origin/master at b4cbee926d1.
The exact SHA it reviewed is not recorded.
The rebased commit 5468fddb639 has the same patch id as c0a78b0cdfe, and its changed lines equal those of f1908404e3f; only the PLAN.md context lines of neighbour rows differ.
After the merge, the diff against origin/master has the same changed lines as 5468fddb639.
So the review was not repeated.

It found five nits and one pre-existing note, and no important finding. The five nits are left as they are:

  • The Actions API's paginated output is not parsed as concatenated objects.
  • A mistyped --plan-from ref reads as an empty planner.
  • A partial download makes the replay crash.
  • The overhead of a job that never reached its run step is counted twice.
  • The planner's per-unit JVM start of 20 s is set against the replay's measured 1.4 s.

One finding is a consequence of the refresh that the reader should know.
With the new pairs, the planner cuts 216 bins on master's configurations and suites, against 190 on October's pairs (results/VARKA-617-replay.txt).
The next part's part-run weighing is meant to bring that back.

The size is 466 counted lines.
The rewritten pair files are data, and the replay text is a results file, so they are not counted.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

t added 2 commits November 22, 2026 09:44
dev/varka_matrix_bins.py replay reads one option matrix run's downloaded bin
artifacts and its jobs from the Actions API, and prints the run's measured bin
phase beside three replays: the run's jobs at their measured lengths on the run's
slots, the run's own bins from their units' measured seconds, and the bins the
planner cuts now, each unit at the run's measured JUnit seconds. Replayed on run
43642146943 it cuts the run's 155 bins back unit for unit; "as run" is 32.5
minutes against the measured 34.4, and replanned on the run's own pairs 30.4.

The pair files are rewritten from runs 43642146943 (14 November) and
44485292430 (19 November, master), 10846 pairs where the committed file held
October's 5718.

Generated-by: Claude Code (Claude Opus 5.5)
Generated-by: Claude Code (Claude Opus 5.5)
@MaxGekk

MaxGekk commented Nov 22, 2026 •

Copy link
Copy Markdown
Author

Option matrix run: https://github.com/MaxGekk/spark/actions/runs/45060246559 (jobs: 10 completed, 8 in_progress, 176 queued)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant