Repository navigation
Conversation
added 2 commits
November 22, 2026 09:44
dev/varka_matrix_bins.py replay reads one option matrix run's downloaded bin artifacts and its jobs from the Actions API, and prints the run's measured bin phase beside three replays: the run's jobs at their measured lengths on the run's slots, the run's own bins from their units' measured seconds, and the bins the planner cuts now, each unit at the run's measured JUnit seconds. Replayed on run 43642146943 it cuts the run's 155 bins back unit for unit; "as run" is 32.5 minutes against the measured 34.4, and replanned on the run's own pairs 30.4. The pair files are rewritten from runs 43642146943 (14 November) and 44485292430 (19 November, master), 10846 pairs where the committed file held October's 5718. Generated-by: Claude Code (Claude Opus 5.5)
Generated-by: Claude Code (Claude Opus 5.5)
Author
|
Option matrix run: https://github.com/MaxGekk/spark/actions/runs/45060246559 (jobs: 10 completed, 8 in_progress, 176 queued) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The option matrix's planner packs each configuration's suites into bin jobs by per-pair seconds.
Until now, nothing checked the planner against the run it planned.
dev/varka_matrix_bins.py replayprints a finished run's measured bin phase beside three replays.On run 43642146943, "as run" is 32.5 minutes against the measured 34.4, and replanned on its own pairs 30.4.
The three replays, the measured table and the pair refresh
The option matrix (
varka-option-matrix.yml) runs every Varka suite under every emit-option configuration, split bydev/varka_matrix_bins.py planinto bin jobs of two lanes of about 480 s each.The planner weighs each (configuration, suite) pair by the seconds it took in an earlier run, from
sql/varka/matrix/config-suite-seconds.tsv.The only earlier replay was a one-off from October, timed at another run's bin count and slot count.
It cannot say whether the planner is right about its own run.
dev/varka_matrix_bins.py replay RUN_DIR --jobs JOBS --suites MODULE=FILE...reads one run's downloaded bin artifacts (gh run download) and its jobs from the Actions API.It prints the run's measured bin phase (the first bin job's start to the last one's end) beside three replays.
Each replay starts the jobs in bin order on the run's slots (the most bin jobs it ran at once, 38).
Each job starts after the run's median wait for a runner once its slot is free.
--plan-from REF, or--pairsand--suite-seconds). Each new unit is timed at the run's measured JUnit seconds of its suites plus the run's median JVM overhead. This tests the planner.The pair files are rewritten from two fresh runs' report artifacts: 43642146943 (14 November, with a reach record) and 44485292430 (19 November, master, every configuration running every test).
config-suite-seconds.tsvgoes from October's 5,718 pairs to 10,846, the mean where both runs hold a pair.suite-seconds.tsvcomes from the master run.The replay was checked before anything was scored with it.
Given the run's configurations, the suites its build listed and its reach record, the planner at the run's head cuts run 43642146943's 155 bins again, with the same units in the same order (73,197 s planned, as the run's own summary line says).
It cuts run 44485292430's 190 bins the same way.
Measured (
sql/varka/plans/m8/results/VARKA-617-replay.txt):The model comes within 5.5 per cent of its own run.
Most of the gap is in the slot model: jobs start slightly out of bin order, and one runner wait reached about four minutes.
On the run's own pairs, the replanned phase is within a fifth of the measured one.
That figure is in sample by construction, so the out-of-sample check is the next matrix run on master.
Why are the changes needed?
The pair file was from 30 October, so 51 suites had no pair.
An earlier replay was 22 per cent under its run, and nothing could say if the planner or the file was at fault.
How stale the pairs were
Over the pairs both files hold, a fresh run's seconds range from 0.65 to 1.65 times the committed ones (the middle 80 per cent).
The planner packs bins by these seconds, so a pair at half or twice its real weight makes a bin half or twice its plan.
Before this change there was also no way to tell whether a gap between a plan and its run comes from the weights or from the model of how jobs run.
The replay separates the two, and the refreshed pairs are what the planner plans from.
Does this PR introduce any user-facing change?
No. This is a CI planning script and its data files, and Varka is not in any released Spark.
How was this patch tested?
On the laptop, on head
9032bbbb3d1(masterb4cbee926d1merged, no conflict):planruns as the workflow calls it. This is a parse check only.Full test record and what was not run
On the laptop, one sbt at a time,
SPARK_LOCAL_IP=127.0.0.1, on 22 November 2026, at head9032bbbb3d1.That head is the branch merged with
origin/masteratb4cbee926d1(the merge had no conflict).Passed there:
dev/varka_regen_generated.sh --all. It left nothing to commit.python3 dev/varka_push_check.py: precommit range check, plan lint, quote check, census check and size.~/varka-ops/extra-gate.sh: it passed, and no step applied, since no Java, Scala or catalyst file changed.python3 dev/varka_matrix_bins.py --selftest: 32 tests pass. Two are new. One checks that "as run" and "jobs as run" over a two-bin, one-slot fixture equal its measured 98 s. The other checks that a fixture laid out as the planner cut it, with its pairs read back bywhole_runs, replans to its own bins and phase.ruff checkandruff format --check(0.14.8) ondev/varka_matrix_bins.py.dev/varka_matrix.sh --list-suitesfor both modules,bash -n dev/varka_matrix.sh, andplanwith--platformsas the workflow's "Plan the bins" step calls it. It exits 0 and writes valid JSON. Its configuration list was taken from the pair file, not from a build, so its bin count is not comparable with the figures above.dev/varka_lint.py,dev/varka_matrix_report.py) import it.The branch does not change any file under
.github/workflows, and I did not parse the workflow YAML, because no local Python has PyYAML.The suite
VarkaTestReachSuiteshares a reach fixture with the self-test, and this branch does not change that fixture or the reach reader.Run on 20 November on the earlier head and not repeated, because the run artifacts are not on the laptop now:
sql/varka/plans/m8/results/VARKA-617-replay.txt.times --reach --per-configurationover the downloaded 43642146943 tree. It reproduced the report artifact's 8,392 pairs and suite seconds exactly.planover master's configurations on both file versions: 190 bins on October's pairs, 216 on the fresh ones.Not run:
dev/), which runs on push.Review
/code-review highfound five nits and no important finding.The changed lines did not change after the merge, so I did not rerun it.
The reviewed SHA is not recorded.
With the new pairs, the planner cuts 216 bins on master's configurations, against 190 before.
The size is 466 counted lines.
The nit list and the review record
/code-review highran on the branch before its rebase onto0e5c1657efeand the merge oforigin/masteratb4cbee926d1.The exact SHA it reviewed is not recorded.
The rebased commit
5468fddb639has the same patch id asc0a78b0cdfe, and its changed lines equal those off1908404e3f; only thePLAN.mdcontext lines of neighbour rows differ.After the merge, the diff against
origin/masterhas the same changed lines as5468fddb639.So the review was not repeated.
It found five nits and one pre-existing note, and no important finding. The five nits are left as they are:
--plan-fromref reads as an empty planner.One finding is a consequence of the refresh that the reader should know.
With the new pairs, the planner cuts 216 bins on master's configurations and suites, against 190 on October's pairs (
results/VARKA-617-replay.txt).The next part's part-run weighing is meant to bring that back.
The size is 466 counted lines.
The rewritten pair files are data, and the replay text is a results file, so they are not counted.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)